Correct way to split configuration into multiple files

I have a question about splitting my configuration into multiple files, for vanilla Emacs, i.e., no Doom/Spacemacs/etc... I currently have my configuration by split theme which I then load using either `(load "my-config.el")` or `(require 'my-config)`. This work well, but it crucially means that it is difficult to compile the configuration files (using the byte compiler or the automatic native compiler). This is because some of these config files have some dependencies on each other. In particular, all my config files assume that the `load-path` is set correctly and `use-package` and `straight` are installed and configured so that packages can be installed or used. When compiling files, especially using the asynchronous native compiler, these files are compiled in batch mode in a vanilla Emacs without any configurations. Hence any statements using unloaded packages would fail. What is the correct way to do this? I know I can put all my config in a single file (literate or otherwise) but I think this will get unwieldy quite quickly for me.

Can't you require the packages you need in each file that needs them? So, for example, if `my-config.el` depends on `use-package` and `straight`, you start that file by requiring them.

I haven't played with native compilation yet, so I'm not sure if that changes anything. But what I've been doing is to have a `~/.emacs.d/init.el` which loads the basics (`use-package`, etc) and then have it loop through other files in alphanumeric order to load them. Then number files in whatever way is required to sort out the dependencies. Not super fancy, but it's been working since 2014, so the stability factor is good :) [Here's my `init.el`](https://github.com/MatthewDarling/.emacs/blob/master/init.el), if you want to see how that works.

You don't need byte compile your own setup. It won't make your emacs start up faster or more efficient. Consider idle loader. Idle loading certain 3rd party packages and its setup code could speed up Emacs startup. But byte compiler need load these packages to do the compiling thing. So that makes idle loading impossible.

Sorry /u/redguardtoo, I know you make lots of positive contributions to the community, but I have to downvote this. You don't need to byte-compile your own config, yes, that's true, but of course byte-compiled code will start up faster and run more efficiently. How could it not? Also, I think I know what you mean by "idle loading", but a newbie definitely won't, because that's not what it's actually called in Emacs (if I'm understanding correctly what you mean). It's called autoloading.

Each file needs to contain either a `(require 'foo)` or an `(autoload #'foo-function "foo")` form to resolve each dependency on such a function as `foo-function`, where `foo-function` is defined in package `foo` in file `foo.el`, and the file `foo.el` is located in a directory that is an element of `load-path`. The `require` or `autoload` form should itself be inside an `eval-and-compile` form (or in the case of a Lisp macro that is definitely only required at compile time, it should be an `eval-when-compile` form instead). If the byte-compiler complains about a function not being available at runtime even with a correct `autoload` form present, you may also have to add a `(define-function foo-function "foo")` form -- don't ask me why.

That's also what I do, but native compilation fails because I think the files are compiled without loading `init.el`.

Hmm.. the problem is that it's not just the packages that are required. It's also some configuration to make these packages work. However, now that I think about, I could write a module that does these configuration and require that module at the start of each configuration file.

"newbie"... ouch :D :D I already use autoloading extensively. My Emacs start time is 0.6 seconds (byte compiling reduces that to 0.45 or so, not sure about native compiling). In any case, I want to compile my files not just for speed but also to resolve any errors/warnings that are a result of my dependency hell.

Ah, okay, so it's a native compilation specific thing that I haven't tackled yet. Could you run the native compilation inside Emacs after config is loaded? Like, inside the body of that loop? Assuming there's an elisp function like for byte compilation, that should be viable.

I have mine set up so my `init.el` itself configures the really foundational packages like `use-package` and `straight`, then it `require`s a "prelude" file containing other shared packages like `general`, `which-key` and `evil`. Once that's done, `init.el` moves on to loading the rest of the files, all of which also `require` the prelude package, and are split up by subject matter like Python, UI, web languages, etc. About the worthiness of compiling config files, that's already being discussed in other threads here, but when I've tried it it only reduces my load time by less than a second. Given that I typically keep Emacs running for weeks on end, the cost of compiling outweighs the benefits for me. That said, I hope the setup I described helps your goal to compile!