Emacs 28 & 29 watching .eln files and causing “too many files open” on M1 Mac

Been trying to figure out a way to get out of this trap… I got a new Mac as a work laptop and I can’t seem to update the file descriptors. I updated it for the system but whenever eMacs is opened, ulimit is still at 1024. I remember someone posted about this before but their issue was that lsp was watching too many files. I wonder if I should not load so many packages before starting? Looking for suggestions and thanks in advance!

sysctl? launchctl limit maxfiles 400000 unlimited

Funny enough sysctl has values for max files to be way above 120k and perproc to be 75k+. It does not impact the ulimit while inside eMacs though sadly.

Will try the launchctl command now though! Edit: unfortunately no change with launchctl command.

There no way to fix this or you can try to send this to mail list.

That is sad BUT good to know. I can relentlessly stop looking for a solution and send this to the mailing list.

What are you doing that needs > 1024 open file descriptors? Emacs generally doesn't keep those open longer than necessary to read in the file contents.

Things like lsp-mode and treemacs will easily hit the limit if you work on large repositories.

I hit this all the time, as I work on a large monorepo with lsp-mode (and sometimes treemacs, which also watches stuff). Whenever it happens I run `M-x file-notify-rm-all-watches` and things go back to normal for a while. Last time I looked into this, you could not work around it by adjusting ulimits or literally anything. It's a core limitation of a low-level API used internally and is not configurable. I will try to find the previous discussion. Edit: - [Relevant StackExchange answer](https://emacs.stackexchange.com/a/70278/18209) with more mitigation strategies - [Previous Reddit discussion](https://www.reddit.com/r/emacs/comments/mq2znn/no_file_descriptors_left/gudivjv/?context=3) Edit 2: I don't think this has anything to do with .eln files, by the way.

Polling via file descriptors isn't a great technique for this very reason. The built in inotify library (im not sure if LSP uses it or not) could be patched to implement recursive watches which would lessen the file descriptor count, but there is a limit to the usefulness of that technique. I believe u/jethroksy proposed such a patch and a similar package on MELPA awhile ago. Not sure on current status of either.

On occasion I need to work in three repos at once (frontend , backend, and a submodule), with lsp active on all three, I can run into the thousands of files in an instant.

It is a combination of the eln files and lsp. On startup, my eMacs is already watching 1000 files and using lsof I looked at which files it was actively watching and it turned out to be .eln files at the start. Then once I start opening some lsp projects this gets out of hand really fast lol.

I don’t necessarily need it, it just happens as others discussed by using lsp. Also when I first open eMacs, for some reason it is watching a ton of eln files at the start to begin with, which causes me to hit the limit pretty fast.

I can't think of a reason offhand why Emacs should be watching .eln files. Might be worth a bug-report.

That sounds more like an lsp issue than anything else. Why does lsp need to open/watch all those files?

I have hit the same issue on my M1 MBA with Emacs 28, especially when using mu4e. Since then I had set a high ulimit to get around this.

Your mileage may vary, but I have this working in Monterey. There is [big gist](https://gist.github.com/tombigel/d503800a282fcadbee14b537735d202c) that has a bunch of comments with the things various folks have tried to get this to work. What the config below does is gets `launchd` (like Linux's init) to change maxfiles *before* any of your user processes (shells, editors, whatever) start up. You should really read the gist to get all the details and historical back-and-forth, but after you get the file and the ownership right you need to: `sudo launchctl load -w /Library/LaunchDaemons/limit.maxfiles.plist` Once you reboot and bet back to your shell, `maxfiles` should be set to whatever you set it to in `limit.maxfiles.plist`. $ ls -l /Library/LaunchDaemons/limit.maxfiles.plist -rw-r--r-- 1 root wheel 545 Aug 4 2021 /Library/LaunchDaemons/limit.maxfiles.plist $ cat /Library/LaunchDaemons/limit.maxfiles.plist <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple/DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>limit.maxfiles</string> <key>ProgramArguments</key> <array> <string>launchctl</string> <string>limit</string> <string>maxfiles</string> <string>65536</string> <string>65536</string> </array> <key>RunAtLoad</key> <true /> </dict> </plist> $ launchctl limit maxfiles maxfiles 65536 65536

What a question to unpack. Depending on the language, LSP will analyse the entire codebase and dependency tree to provide autocompletion and symbol definitions - along with power refactoring capabilities like renaming symbols across *all* references. It needs to watch every source file in order to keep the symbol declarations and references cached and up-to-date as source code and dependencies are changed. Even in a nominally small repo with three or four source files, add a few external dependencies and LSP can easily start watching a few hundred files.

In short: a change in the current buffer could cause a compilation error in some distant (but within the project) file you haven’t opened. It’s expected of a modern IDE to be able to tell you about that in real time.

And it absolutely definitely needs to know that instantaneously? It can't run 'git status' every 5 seconds?

In theory file-notify will watch directories for changes to files within a directory, which means that you then run into the issue once you have more than 1000 directories to watch. On macOS file-notify will use kqueue, which can't watch directories like that, it has to watch each individual file, so you get the problem earlier.

Giving the OS callback code to run when it detects file changes is significantly less resource intensive than running stat() on every tracked file, storing modification times, and then diff checking every 5 seconds. It would be like flying to the factory to get a shipping container of chips because they were sold out online.

How do you know that it's watching those files vs just having them open for some reason? I just restarted my Linux Emacs 28.1 server process and was somewhat surprised to see it has ~300 .eln files loaded (from both eln-cache and from the build native-lisp dirs) just by loading my init.el.

Sorry when I say watching I actually meant having them open. Sometimes flop my words…. But yea that’s what I’m seeing also. Mines is 1000+ at start up. I’m guessing a solution might be to not load so many packages at start up ( been meaning to do this for a while)