Metarosetta: Write translation expressions to extract and sync textual data between code comments, org entries or any other format and convention

Hello dear people of `r/emacs`! As I write code daily, it always frustrated me how advanced our machines have become, the UI, the computational abilities, and yet if I add a `TODO` comment within my code as a pointer to my future self it’s essentially useless as soon as I close the file or exit the code editor or IDE. Such technical ability, and still I’m manually re-typing and re-formatting my comment to a separate “app” so I can have some sort of centralized task management. I’m embarrassed to think of how many orphaned `TODO`s I’ve forgotten in code over the years. As I started using Emacs, and of course configuring it to my needs, I soon realized the understated power and flexibility of this transparent textual OS. So I built a way to synchronize textual information between files and formats. In a nutshell, Metarosetta allows you to define textual conventions for things like code comments, org list items or markdown checklists in a way that makes their semantic data interoperable. So when you update an org list item with a new status or tag for example, the updates get automatically synchronized with defined data-compatible counterparts, like a code comment. This is my very first Emacs package, so I’m really excited and eager to share this with all of you and hopefully get your honest feedback!

Looks interesting; will have to take a look. Thanks for sharing.

Thanks, I really appreciate it! Let me know how it goes if you decide to try it…

[removed]

This is very interesting. I've wondered if something like this would be practical. I figure there must be prior art out there in academic literature, at least, if someone were motivated enough to look for it. :) Looking at some of the code, I'm reminded of Pandoc as well.

Thanks so much for taking a look! I've immediately recognized your username, on account of `makem.sh` and of course `org-ql`, I'm a big fan of linguistic tools like that both in terms of internal mechanics and utility, so it's a pretty humbling experience right now :) Relating to the practicality you'd mentioned, this is what I'm most curious about. If it's usable for people on a daily basis, would it fit their workflows and how. I know IDEs and editors usually have a list-all-project-todos feature/plugin/extension, but that's just a crude lookup and code-only, doesn't allow for syntactic customization and extension, and of course it's not interoperable with other contexts like markdown lists and such. I've also experimented with automatic API endpoints, where any match changes are parsed through and submitted in JSON form, and having a receiving hook for inbound updates. But since I'm only using it for file-to-file local synchronization, I've chosen to make the configuration simple. Also, it's easy enough to define a metalanguage expression which outputs JSON format for example. Whatever the case, it would be exciting to see where the concept might go from here and what applications it might have. I mean, I just created it for my own personal use, as it seemed like an interesting challenge to construct a usable and human-readable metalanguage, as well as a fun way to learn Lisp and its homoiconic qualities. If other people find it useful, that would be pretty cool :)

Thanks for the kind words. I'm glad they're useful to you. I agree with what you said about practicality, but what I meant was on the low level rather than the application level, i.e. I wondered whether a simple A-to-B format-converting domain-specific language would be practical. It seems like, with all the existing tools out there (Coccinelle comes to mind), and all the serious academic research done in software and languages, something like that must be out there, but I haven't spent time looking for it, myself. :) Anyway, your library looks nice! I am curious, why did you go with the plain-language style DSL? It does seem elegant, but being a Lisper, I confess that I would probably prefer a Lisp-style syntax, myself. Also, have you done any benchmarking? e.g. I'd be curious to know how well it could perform to convert a large Org file with a thousand headings to, say, some JSON format, and then back to Org from it. That could be helpful to know whether it could be practically applied to help Org interoperate with other formats and services. Keep up the great work!

Ah, I see! I should've inferred what you meant the moment you referenced Pandoc. :) In terms of the metalanguage itself, writing expressions for complex nested patterns shouldn't be a problem, as it's inherently recursive with expressions acting like fractals. It's funny how these things evolve, the motivation behind the metalanguage was to match, parse and sync one-liners across code comments, org and markdown based on simple syntactic conventions. But, as I started constructing the language, it just organically grew and naturally I added things like support for complex lists which I can't ever imagine someone needing in a one-liner, but the language just felt incomplete without it. Referring to the benchmarking of long-form matches, the aforementioned was the reason I've never really tried it. It was just beyond the scope of this initial "purpose". However, as I'm writing this and listening to what you're saying, it absolutely seems like the most natural way forward. In theory, the matching itself mostly depends on the performance of the underlying regex engine, since all expressions compile to plain regex to match against. The biggest performance issue I suspect would be when frequently updating a tiny piece of a long text. But of course, these things can always be optimized with localized updates and caching for example. In terms of the reasoning behind the plain-language style, I've envisioned it perhaps used by non-programmers as well, and also beyond the Elisp/Emacs context. Elisp was just a perfect playground to experiment and interact with the language. Although the expressions written are actual valid Lisp expressions of symbols and strings, and are treated as such under the hood, I like to look at them as in plain English, including all its contextual inflections, with parentheses "only" used to logically group. Going further down the rabbit hole, for such general purpose use, I'd imagine it would make sense to port the core language engine over to something ultra portable, like Scheme for example? That way, it would be usable as a standalone utility in any context, not just within Emacs. And I suppose there would be a performance benefit, but that's just an educated guess and something to confirm before going this route of course. As you can see, you've really sparked my interest here, and this is something I'd genuinely like to explore. There are a couple of things I can think of right away that I'd modify with the language to make it more adept to long form text, but this reply is getting really long. :) More concretely, I could draft up a short org file for example to list all these modifications and proposals, if you're willing to take a look?

Sure, I'd be glad to look at that. But don't mistake me for an expert of any kind. You already have more experience with this topic than I do. :) Porting it to Scheme or CL would be interesting, but if you still wanted to use it in Emacs, you'd have to interface with them. Performance might improve, but Emacs Lisp has native compilation now, which helps a lot. So I'd suggest doing such a thing last, and only if it seemed necessary.

Awesome, thanks! I'll compose a list of all the things I believe would make the project natively suitable for such agnostic and long-form conversions. These will include slight modifications, or updates, to the metalanguage itself (e.g. support for newlines and indentation as part of the target format syntax description, like YAML has for example), the actual processing and textual regeneration engine, and finally to the configuration and general interface layer. In terms of porting, yes I wholeheartedly agree on this. I had mentioned it mostly for compatibility and usability reasons beyond Emacs, as the end of the rabbit hole, but of course that would become a "concern" only if, or hopefully when, the package is actually widely used and considered useful within Emacs. :) I really find this "evolution" of the project quite exciting, as it will both allow me to further experiment and work on the metalanguage on more serious use-cases, and of course make the project itself much more generally usable. What's the joy of working on an open source project if I'm the only one actually using it. :) I'll send you the list as soon as I compile everything!

Ok /u/github-alphapapa, I'm finally back with the list! :) Instead of directly sending it here, I've opened a GitHub discussion on the matter so anyone and everyone interested can take a look and participate. Here's the link to the discussion: https://github.com/73D1/metarosetta/discussions/1 Needless to say this is a rough draft, and by no means a complete spec on the language extensions, but I believe it's a good start. As soon as I have these up and running, we can try them out and test various formats. From there, we can organically grow the language spec if or when needed! Please take a look when you can and let me know what you think over on GitHub! Thanks again for the suggestions! :)

Thanks, I'll take a look.

Great, thanks!