Lazy loading and lazy installation of packages

# The problem I might have found a way to make my config more scalable. For context, my config file is a 200LOC litterate config (I know. It's not that bad) and I have a lot of packages that I use every once in a while that clutters my config and increases my startup time (3.61 seconds). Moreover, every time I migrate my emacs config, the initial startup time (`use-package` downloads everything) is annoyingly high (I just checked, it was about 7 minutes). # The solution Lazy load (and install!) everything. nn I wrote a small library that lets you define your packages and their configuration as a dependency graph. A package will be loaded (and installed, if necessary) when a dependent package gets loaded, or when a generated placeholder function gets run. This is what it looks like in practice: ```emacs-lisp (defmodule m:org :dependencies (list m:company m:yasnippet) :init (progn (require 'org) (require 'ox-html) (require 'ox-publish) (defvar org-publish-project-alist nil) ;; hooks (add-to-list 'terra:company-backends-alist '(org-mode . (company-yasnippet)))) :hooks '((org-mode-hook . company-mode) (org-mode-hook . yas-minor-mode))) (defmodule m:org-transclusion :grafts (list m:org) :init (progn (straight-use-package 'org-transclusion) (require 'org-transclusion) (require 'org-transclusion-indent-mode) (add-to-list 'org-transclusion-extensions 'org-transclusion-indent-mode)) :hooks '((org-mode-hook . org-transclusion-mode))) (defmodule m:org-roam :dependencies (list m:org) :init (progn (straight-use-package 'org-roam) (require 'org-roam) (setq-default org-roam-completion-everywhere t org-roam-db-autosync-mode t org-roam-extract-new-file-path "${slug}-%<%Y%m%d%H%M%S>.org") (setq org-roam-directory (file-truename "~/org-roam")) (setq org-roam-v2-ack t org-roam-node-display-template (concat "${title:70}" (propertize "${tags}" 'face 'org-tag))) ;; other configurations (define-key org-mode-map (kbd "C-c n a") #'org-roam-alias-add)) :keys '(("C-c n r" . terra:update-org-roam-state) ("C-c n t" . org-roam-tag-add) ("C-c n i" . org-roam-node-insert) ("C-c n f" . org-roam-node-find))) ``` # Deferred loading is deferred slowness After I converted my whole `init.org` file to use this instead of `use-package`, my startup time (excluding the tangling of `init.org`) decreased to 1.02 seconds. However, this just made the first use of a command in some package very slow. For instance, when `swiper` isn't loaded, `C-s` takes a quarter of a second to run (this number probably gets higher for bigger packages such as magit). This is annoying. I modified my tiny library to log which packages get loaded on each emacs run, then I tinkered with idle timers to, about every 5 seconds of inactivity, load the most commonly used not-loaded package. This mechanism has two very interesting advantages: 1. Makes the whole experience feel much smoother: usually, the most used commands are loaded before their first use, without requiring them to be loaded during startup. 2. Keeps running emacs in daemon mode useful (the whole point of daemon mode emacs is to load packages and run compurations ahead of time.) This was probably my first attempt ate writing "serious" emacs lisp and I feel pretty proud of myself. [here is the src block containing the thing](https://pastebin.com/zQATVnQS). I copied it verbatim from my config.

Have you had a check on Leaf instead of Use Package? I think it would be good to contribute this idea to these packages, mainly to grow the community contributed code instead of splitting it (unless it’s too hard to integrate).

> every time I migrate my emacs config, the initial startup time (use-package downloads everything) is annoyingly high (I just checked, it was about 7 minutes). Why would you leave behind most of your config when "migrating" it? You seem to have caused your own problem, and invented a solution which is still worse then just not causing the problem in the first place. Copy your whole config to start with and... you instantly have your whole config without *any* of the "slowness" you're seeking to avoid.

Also .. how often does someone migrat their emacs (to a new server/account) that eating 7 minutes is a big problem? I mean, hacking on a solution is fun, do it by all means :) .. but if you're not knocking accounts out every few weeks why bother? (Ah, maybe you're sshing into 50 boxes and making new Emacs deployments at each? Not using Tramp mode remote edit or emacsclient over ssh or something?)

To be clear, the 7 minutes loading time is caused by use-package eagerly *downloading* every package in the configuration. By migrating my config, I mean that I copy [`init.org`](https://init.org) to a new computer. >Why would you leave behind most of your config when "migrating" it? I don't understand what you mean by this.

I often try out different versions of emacs (like when I wanted to check out the background transparency coming in emacs 29) and straight.el proceeds to recompile everything. However, I agree that this not a major problem.

> > Why would you leave behind most of your config when "migrating" it? > > I don't understand what you mean by this. I believe that they suggest copying over the packages with your `init.org` instead of downloading them again.

I mean that if you copy your config from one machine to another, then there's *nothing* to download on the new machine -- it's all there, complete, in a known working state. Because you copied it. Your entire problem of needing to download things (which is slow) is caused by you not copying those things in the first place (which would have been fast). My suggestion is that however you're "migrating your config" in a state where most of it gets left behind and needs to be newly downloaded as individual packages from remote sources... don't do that. Instead, just copy your (complete) config. Those packages are *part of your config*. If you mean to migrate your config, migrate all of it.

You can still optimize the startup time a bit. My setup uses 300 packages, and start up time is about 0.5 seconds. My computer is 10 years old, btw. See [https://www.reddit.com/r/emacs/comments/ws017t/comment/ikvob4j/?utm\_source=share&utm\_medium=web2x&context=3](https://www.reddit.com/r/emacs/comments/ws017t/comment/ikvob4j/?utm_source=share&utm_medium=web2x&context=3) which has some tech details. You could use [https://github.com/redguardtoo/elpa-mirror](https://github.com/redguardtoo/elpa-mirror) before migrating your setup. The elpa-mirror creates a local repository from your *installed* packages. Then you can use the local repository to replace remote repository during migration. As its README said, "15 seconds to install 115 packages".

https://github.com/progfolio/elpaca should be significantly faster for the initial install (packages are installed/built asynchronously). For me, installing around 116 packages takes roughly 100 seconds. That's including building the mu system binary, pdf-tools binary, and several other custom pre-build steps. Sans those binaries it would take roughly a minute. Keep in mind the package is still in early development, though.

If I had an issue with config deploying time (I don't, since I use Emacs only on a single machine), I would rewrite `:ensure` keyword to install packages lazily instead, like `:lazy-ensure` or `:defer-ensure`. And \`:defer\` is already there, so my 1600LoC config loads in a second.