RFC: Literate emacs/system configuration using org-roam

I'm not sure if this is a good idea or if anyone else already does it, but I've been meaning to try it out. So I made a non-functioning demo to see if the graph for a literate emacs/system config with org-roam would be useful like I had hoped. It seems like it is, but requires not linking some things you'd think you want linked to keep things understandable. I'm reminded of advice to take care before linking two nodes in org-roam. I bet there is some smarter or different graph algorithm or settings I could change to do this for me, but here's what I've got for now and I think it's pretty cool. Curious of others thoughts on its utility (or lack of) or ideas to make it more useful.

Some rambling thoughts and lingering questions: Are there ways to filter out connections? I guess I'd need to filter out specific nodes or tags (if I can come up with tag names for certain views). For instance I originally linked "version control with git" from "version control with magit" but that makes it harder to identify emacs workflows from non-emacs workflows in the graph. The same with linking to "emacs" from "emacs packages". Aside: I might want to only link "emacs based workflows" from emacs. I guess i'm asking for general guidance and experiences others have had keeping their graphs useful and understandable.

Image is low resolution

Hm, I'll upload another in a few minutes.

Edit: make sure you click to zoom in if you aren't This one should be better judging by the added filesize: https://imgur.com/a/1CUgq6q

So is the idea here that you have certain nodes, whose inbound links are used to find files which are used in the literate configuration? I have built a similar system over the last few years that is not so much "using" org-roam as it is inspired by it and compatible with it. I found in trying to work within the org-roam DB that there simply wasn't enough context in its cache to be able to reliably generate configurations, and that you have to be too "careful" for my tastes to make it all work, (making sure the file names of the tangle files can be mapped from the slugged file names, dependency ordering was quite chaotic and often broke my init until i added a pile of `with-eval-after-load` and then "priority" keywords and then a topography dependency sort, etc). You could kludge custom links like "this is a dependency of that", "this should be tangled in to that", etc in to `org-link-parameters` at the cost of compatibility with external tools, but I chose not to. So now I have a more complex set up; I have an `emacsql-sqlite` cache database whose design I cribbed from the `org-roam-db` package to extract enough metadata from the files to be able to tangle the files and embed them in to an init.el... It's not terribly clear how the thing functions in aggregate or how others could use or reproduce it, or how to even bootstrap it at the moment but you can get some vague ideas of it [here](https://cce.whatthefuck.computer/arroyo/), [here](https://cce.whatthefuck.computer/arroyo/db-cache), and [here](https://cce.whatthefuck.computer/arroyo/emacs) but it allows me to add a handful of `#+KEYWORD: value` elements to a document like `#+ARROYO_EMACS_MODULE: org-mode` will imply tangling to my "arroyo directory" as `org-mode.el` and `(provide cce/org-mode)`... Save them, and invoke `arroyo-emacs-generate-init` for a new `init.el` with the code included in it. I have a vague goal of brushing this up to the point where someone other than me could use it, but perhaps it's useful to look through. The keywords are cached in to a simple (file, keyword, value) table and then other tables are generated from that: sqlite> .schema CREATE TABLE emacs (file NOT NULL, tangled NOT NULL, module NOT NULL); CREATE TABLE nixos (file NOT NULL, nix_module NOT NULL, role ); CREATE TABLE home_manager (file NOT NULL, home_module NOT NULL, role ); CREATE TABLE keywords (file NOT NULL, keyword NOT NULL, value NOT NULL); I've since expanded this pattern to generating NixOS systems, [home-manager](https://github.com/nix-community/home-manager) configuration, and even my dynamic [org-roam publishing platform](https://engine.arcology.garden/arroyo#arcology/arroyo) This sort of thing is definitely viable and quite nice when you get the whole house of cards stood up, especially if you then expand it to publishing with `ox-hugo` or build your own publishing system like I have. it's so nice to be able to share a direct URL to particular configuration snippets and documentation like I have done here! I encourage you to keep playing and see if you can design something more elegant than what I have described. ref: - [https://code.rix.si/rrix/arroyo](https://code.rix.si/rrix/arroyo) the generators - [https://code.rix.si/rrix/complete-computing-environment/raw/branch/main/org-roam.org](an example module installing and configuring org-roam)

Wow! This is all very interesting and very related to my thoughts. Let me reflect on it all and give you a more detailed answer later. BTW some of the pages are erroring like this one: https://cce.whatthefuck.computer/home/rrix/org/cce/emacs_shell_mode_is_good_enough.org

ahhh yeah, the link generation is all home-brew ... I guess the stuff on that index page is all busted up. I wonder if I can fix that today, thanks for letting me know! edit: Okay those work now! easy enough to ask org-roam to give me the ID of the "level 0" node associated with the file path, but the URL rewriting from those \`id\` links to my crazy multi-domain setup is slow enough that it might generate a 504 if you miss the cache. I'll have to think about how to deal with that soon

Click to zoom on mobile is still v low res. Cool proof of concept!

My graph contains a lot of stuff, as my work these days is primarily knowledge work with a huge amount of social interaction. This means my nodes are highly interconnected into clusters (that usually connect elsewhere also). If I want a graph overview, I usually limit it to 1-2 connections away from where I currently am because it tends to become a rat's nest otherwise. On the flipside, this connection allows me to find associations very fast (for example if I forget what a certain group of people is doing or what a specific person works with and how I understand that). The other part of my job is broadly interacting with a ton of different tools. Now that I've seen you post this, I realise they can have their configs inside of org-roam using tangling (perhaps with a tag or something so I can search for nodes to tangle). This will assist me with migrating between different machines, which I currently use a dotfile setup for I don't love. I've been meaning to test out git annex as well, which will hopefully allow me to annex sensitive stuff like my keys to encrypted remotes and use tangling and other scripting to put them where they go when I pull everything. Essentially I've embraced the chaos. My only concern these days is making that knowledge web as resilient as possible, with incremental backups and a versioning system (along with multiple backup backends). I've found my biggest issue is not the setup but accidentally losing data. That can sometimes mean quite a lot of focused and hard to find info.

You didn't say how or why this would be useful.

Yeah, because i'm not sure if it is, that's why i said: > Curious of others thoughts on its utility (or lack of) or ideas to make it more useful. For me the uses i can think of are: - Knowing the most important and depended on parts of my computing environment - Being able to see which emacs packages serve what purpose or use case - being amenable to visually be "designed to developed in fits and bursts" as [Arroyo systems](https://cce.whatthefuck.computer/arroyo) puts it, which another commenter posted about - i think i find the navigation nicer feeling and more intuitive than an org file To expand on that last point, a blocker for making incremental progress on my ideas automating or improving emacs workflows is having to expensively rebuild context to understand and visualize what's left. Though i didn't demonstrate it in the graph pictured, this could be one way to solve it.

Damn this is really awesome. I am kinda jealous, I might try this myself

NixOS + emacs + org-roam. This is a really awesome combo. Loved it.

Thanks, i'm looking forward to trying to get it functioning after I clear up some unknowns about keeping the graph useful, filtering, and applying other graph algoritms.