Making clangd fast for big projects

I've been fighting with clangd for a couple weeks, which was a big disappointment. For years I saw these interesting plugins to help code completion completely pass me by because converting the project I was on into a format understandable by clang was a huge hassle. Well, now I'm working on an Unreal project, and its build steps let me create a compile\_commands.json file that works with clangd. But the compile\_commands.json file is about 745MB, and I was getting constant blocking stalls in emacs. If I opened up the eglot events buffer, it was scrolling endlessly, presumably because of background generation of semantic data that would be used for all the interesting stuff that you want to use clangd for. Hunting around, I found mention of clangd-indexer, which creates a static index. clangd-indexer has a few drawbacks: * It is not updated live and can easily fall out of date. You will have to manage it. * It takes time to make the index. I wouldn't recommend doing it while you're also trying to use it. * It takes up extra space on disk. * The documentation is absolutely miserable. Trying to figure out how to use it is a nightmare. * Trying to figure out where to even DOWNLOAD it is cryptic; it's not part of the standard package. Where to download it: [https://github.com/clangd/clangd/releases/latest](https://github.com/clangd/clangd/releases/latest) Some info on how to use it: [https://clangd.llvm.org/guides/remote-index](https://clangd.llvm.org/guides/remote-index) (This mostly is attempting to detail how to set up a whole remote server with a static index that team would use. But you can also make a local index.) Some useful info from devs and a user going back and forth. [https://github.com/clangd/clangd/issues/479](https://github.com/clangd/clangd/issues/479) The command line that I used (more or less) `clangd-indexer -extra-arg="-Wno-everything" --executor=all-TUs .\compile_commands.json > clangd.dex` And finally, the contents of my .clangd file: CompileFlags: Remove: -W* Diagnostics Suppress: * Index: Background: Skip File: d:\p4v\Dev\clangd.dex I know this isn't strictly an emacs topic, but this is where I get the majority of my exposure to clangd discussion.

I went from having constant stalls to actually being able to use eglot properly on a large codebase and jump to definitions, see hover-state for variables, etc. It's a completely different experience. I initially threw eglot under the bus, but that absolutely wasn't fair!

Have you tried lsp-mode? Curious how it compares to elgot on big project.

I would reckon about the same, since the bottleneck is `clangd`, not `eglot` or `lsp-mode`.

u might want to try lsp-bridge. I hear it's asynchronous

Definitely try [lsp-bridge](https://github.com/manateelazycat/lsp-bridge) and see if it helps. I still haven't tried it myself, but it should work asynchronously. Clangd should really have an option to communicate with a custom protocol so we could get it to emit s-expressions instead of json. Json is great for javascript applications over wire. For desktop applications written in other languages, it is obviously a waste of resource. First we compile data to a json file, then we decompile it from json to whatever needed. When run locally, which I guess is quite common use case, it would be much better if clangd could compile its ast into user supplied format, maybe with a user loaded plugin or something.

Are you sure your make/ninja/cmake/whatever project is set up correctly? I work day-to-day on a 500kloc codebase with 20+ targets and my `compile_commands.json` file is only 3MB. For me clangd often slows down, but generally works (just about).

This part of the process is out of my control. Unreal allows you to pass an argument to the build process to generate a clang database, and this is what it spits out.

That's not really going to help when the bottleneck isn't the emacs LSP client (lsp mode, eglot, whatever) but **clangd itself** choking on that humongous compilation database. That's why OP's problem was solved by using a separate indexer. With lsp-bridge you will only (hopefully) prevent emacs from freezing while it is waiting for response but wouldn't make anything else faster. Unreal is several GB of source code, so no wonder that that JSON gets big.

Unreal engine has about 2GB of just source code when you check it out of git, if I remember correctly. It is a totally humongous project, with a ton of libraries and dependencies, with thousands of smallish files. Also the build system used to be a "homebrew" C# tool that also acted as a preprocessor and code generator ... I believe they started to switch to CMake in the latest versions but I don't think that has been released yet. So that huge compilation database doesn't surprise me. Lot of common tooling chokes on Unreal.

You may want to postprocess that JSON, though. E.g. filter out lines for files that you don't care about, such as many of the various dependencies or libraries and duplicate entries. Clangd cares only directories where to look for files and compilation switches, so if you have 200 files using the same path and switches, it only consumes space but doesn't bring any new information for the indexer.

You're not wrong, but that just introduces another step in the chain that's time consuming and possibly error-prone. I have to make sure I strip out everything that I don't want and reduce the file without introducing errors; it's not clear to me that it's a more efficient use of my time than allowing the process to run overnight whenever I do a sync and database regeneration. Having briefly scrolled through the file, I know for a fact that some of the flags are being emitted as duplicates, and some of them are probably no-ops, so the 700+MB size is a bit of a misnomer, but it is still a tremendous number of files.

I haven't. I tried to use lsp-mode early on and it didn't work out of the box and I quickly gave up on it. But as has already been mentioned, because the bottleneck is really clangd's background compilation, I don't think that lsp-mode would do any better.

Amazing how many people missed this.

I think you can safely combine all unique paths and skip all compilations flags except `-D` (preprocessor macro definition) and `-I` (where to look for headers). Nothing else is really relevant for LSP. I have solved a similar problem by writing a bit of Python that took the json file, filtered whatever was needed and ran this as part of the build process. Problem solved.

Yeah, I don’t think unique paths are the problem, it’s that there are 346 -DI occurrences and 1565 -D directives PER FILE that needs to be compiled. Like I said, I know there are a lot of duplicates (I counted at least 10 -DINCLUDE_CHAOS flags). Unfortunately, it looks like each of those includes IS unique already.

I did actually find a legit problem with the commands file (it's using clang-cl but passing -std=c++17 which is not valid for clang-cl args) so now I'm using sed to munge the file after it's output. I'll probably hack the source or ask a dev to do it so it generates better commands.