Navigating an enormous code base

​At my day job I work on a 35 year old product with literally a few hundred million lines of codes. It is essentially a mono-repo. There are thousands of directory trees with Makefiles. To be consistent with Emacs concepts, let's call these 'projects'. Luckily projects are rarely nested, and not in the areas I work on. There is no discipline requiring file names be unique across projects, nor even within projects. Furthermore, within any project there can be multiple unittest/ and/or pkgtest/ directories. Each such directory must contain a suite\_registration.cpp. Most often I work within a single project and want to navigate to files in the same project. Less frequently I need to find a file in a sibling project. Unfortunately, there are \~400 siblings, though I very rarely visit more than 10. With recent attempts to work within Emacs' built-in tools, I wonder how I find files in this code base. Or if Emacs lacks adequate built-in tooling, what package(s) might​ help?

how do you define a project?

Emacs native project detects considers the root of a version controlled folder as the root of a project and it has functions to find files in a project,open dired or a shell in the project root, etc.. You can extend it to detect what is "a project" by implementing your own version of `project-files` like described in [this blog post](https://manueluberti.eu/emacs/2020/11/14/extending-project/). You can, for instance, consider any folder with a Makefile as the root of a project.

This is a very blunt instrument, but when I'm dealing with gigantic, tangled codebases, my swiss-army-machete tends to be the 'ag' package (usually directed to "." or ".."...) + linkmarks: https://github.com/Wilfred/ag.el https://github.com/dustinlacewell/linkmarks which I mention mostly because of the suggestion of an issue with multiple identically named files. I have a *lot* of links that are some variant of 'this filename, but in THIS project' for quick jumps, and with linkmarks I can keep clarifying text about them with the rest of my notes. If you aren't already using org for that then this is much less useful and regresses to being on par with the built-in bookmarks. That said I'm curious what others may come up here with in terms of browsing 'nearby' files. I have not seen any packages that do this that I've been completely satisfied with either. The closest setup I've gotten is projectile due to its relative agnosticism as to what a 'project' is, and being able to fall back on making a literal 'no, HERE' .projectile file at the 'project root' I want when needed. https://docs.projectile.mx/projectile/index.html elaborates. This was a notable improvement from the built-in project tooling in my use.

I expect I would maintain a TAGS file for (only) the projects of interest.

I would try a combination of this and darcamo’s suggestion. Could use the tag-file instead of makefile as project-file. Bookmarks are also quite useful

If this is a c/c++ codebase then I would look into using bear, eglot, and clangd. Clangd will dump metadata files in your home directory. You can put a config file that will keep the meta data where you want it if you think you need to keep things separated. This workflow has been the single biggest change in my workflow equivalent to the impact of moving from spinning disks to solid state drives and I highly recommend it. For other codebases, use a different LSP server and configure eglot accordingly

As I work on a large code-base and found I needed t configure things to better suit my needs. - https://melpa.org/#/lsp-mode (for typical IDE features). - https://melpa.org/#/counsel (great package for searching / selecting within a project). Other packages I've written that I find help navigating a large project. - https://melpa.org/#/doc-show-inline (show public doc-strings inline). - https://melpa.org/#/bookmark-in-project (to easily toggle, list & jump between bookmarks). - https://melpa.org/#/sidecar-locals (to have find grained control over configuration in the project). - https://melpa.org/#/counsel-at-point (to use the current context when searching and listing files to switch between). - https://melpa.org/#/diff-at-point (to use my current patch as a kind of mini-map for switching between edits on the working copy or on my branch).

https://github.com/jacktasia/dumb-jump could help, you can set the main directory with dumb jump and let it match definitions and guess in which file a function exists

Usually I use a combination of cscope and universal-ctags. Both are well integrated with emacs ( xcscope extension for cscope). Cscope is effective even with Linux kernel

>There are thousands of directory trees with Makefiles. To be consistent with Emacs concepts, let's call these 'projects'. Luckily projects are rarely nested, and not in the areas I work on. [Projectile](https://projectile.mx/) can learn about each independent project and enable searching in one without results from the other (same for other actions). You can also switch between projects. Multiple empty .projectile files would signal which directories should be considered projects. You may want those in .gitignore or equivalent. root/project1/.projectile root/project2/.projectile root/project3/ (ignored project) root/project4/.projectile ... To switch use something like `projectile-switch-project`, or if you'd prefer, a completion-boosted alternative like [counsel-projectile-switch-project](https://github.com/ericdanan/counsel-projectile#the-counsel-projectile-switch-project-command=). ​ edit: typos

Don't know what programming languages the projects, but cli ctags supports most programming languages. I use `counsel-etags` for code navigation, `company-ctags` for code completion. Both require cli tool ctags. I also use `find-file-in-project` to find files. It requires cli tool `GNU Find`, but `rust-fd` is also supported.

- [rg.el](https://github.com/dajva/rg.el) or [deadgrep](https://github.com/Wilfred/deadgrep): Emacs interfaces to [ripgrep](https://github.com/BurntSushi/ripgrep), a grep-like tool that is very fast. This lets us search across a large number of files for a pattern of text. The disadvantage of searching for text is that if you are looking for the method called `foo` and there are hundreds of them that exist, it can be hard to know which one you really want. On the other hand, at the scale and complexity that you are talking about, I can imagine that more IDE-like tools just start failing. If you prefer built-in tools, maybe `M-x rgrep` can be an appropriate replacement. - [dumb-jump](https://github.com/jacktasia/dumb-jump): another tool based on ripgrep, this one defines regexes for what definitions look like in a bunch of languages. This gives you a primitive jump-to-def functionality without any setup (except installing ripgrep). The pros and cons are roughly the same as rg.el and deadgrep: you might not jump to exactly the thing you want (if there are multiple choices, you can select the definition you prefer), but it requires no setup and is pretty fast.

After a night's sleep and reading initial comments I realize that I could have been clearer about my issues. My goal is not so much navigating by symbols (I have lsp for that). Rather it is more of a UI question. I want to use a modern completion interface to open files by name. Supporting completion requires that the space of possible file names be indexed and supplied to the completion function. As mentioned in my first post, an issue is how should the completion UI present files with duplicate names. Another issue is what should be the scope of file name (paths) fed to the completion UI. Here I can imagine the following possibilities: * The current project * An explicitly specified sibling project * Within the current workspace, all projects in which I have made changes (perhap via after-save-hook) * One of various pre-specified canned sets of projects I could imagine splitting this UI into * Find in current project * Find within a menu of wider contexts

Have you considered treemacs? https://github.com/Alexander-Miller/treemacs

>https://github.com/jacktasia/dumb-jump could help, you can set the main directory with dumb jump and let it match definitions and guess in which file a function exists You just create a `.projectile` file in each directory you want to act as a project. It can be empty, like a .gitkeep file

You could define a wrapper for project-find-file. By default, it searches within the current project. When a universal prefix is given, it lets you select an alternative root of project and search from the root.

I find consult-buffer + vertico + orderless (+ marginalia) great for navigating a long list of recent files and buffers. It has some project support by default, and that's likely soon to grow. It can already show you other buffers and recent files in the current project (just type p space). In the meantime, a small consult extension package I threw together for project jumping + project buffer+file selection is [consult-jump-project](https://github.com/jdtsmith/consult-jump-project) (see also [consult-project-extra](https://github.com/Qkessler/consult-project-extra) which it was inspired by). Be sure to increase your recentf file count to something large, like 1000.