Programming with Elisp is magic

* this post in Org format: [https://github.com/tonyaldon/posts](https://github.com/tonyaldon/posts) * email me: [aldon.tony@gmail.com](mailto:aldon.tony@gmail.com) * message me: [https://twitter.com/tonyaldon](https://twitter.com/tonyaldon) ─────────────────────────────────────────── BOOM!!! Let's get to the point!!! What's magic when programming Elisp code is that at any time: 1. we can extract a little part of the program, 2. replace some symbols by custom values, 3. send it to the minibuffer with `M-x eval-expression` (or `pp-eval-expression`), press `RET` and, 4. automatically get back some value in the echo area (or in the dedicated buffer `*Pp Eval Output*`). In almost no time, misconceptions about what a program does (or why a program fails) can be spot that way. Those who have already read the post about [link abbrevations and org-open-at-point](https://www.reddit.com/r/emacs/comments/tw3fpu/search_options_in_file_links_link_abbreviations/) might be familiar with the above description. Indeed, today's post is literally a section extracted from this previous post. Why am I publishing it again? Because that post didn't received much success (maybe the wrong topic, maybe too long, maybe I don't know) and so does the section about the magic of programming with Elisp. And this is really unfortunate because it describes a super effective strategy do deal with Elisp code. Beside adding "print statements everywhere" (`message` in Elisp parlance) this is my best tool for working with Elisp code, and I want everyone to know it and use it. I am not a magician who needs to keep his tricks secret, quite the contrary. So, I decided to give that strategy to deal with Elisp code another chance with that post. I hope you find it useful. Here is the context. When we introduced the section "Programming with Elisp is magic" in the post about [link abbrevations and org-open-at-point](https://www.reddit.com/r/emacs/comments/tw3fpu/search_options_in_file_links_link_abbreviations/), we were studying some implementation details of the function `org-element-link-parser` that parses an Org link at point, if any. For instance, in the following org buffer (if you have never used "link abbreviations", I encourage you to read the info node `M-x eval-expression RET (info "(org)Link Abbreviations")`): #+LINK: emacs /tmp/emacs/ [[emacs:lisp/simple.el::(defun next-error (&optional]] with the point at the beginning of the link, calling the function `org-element-link-parser` returns the org object (a list): (link (:type "file" :path "/tmp/emacs/lisp/simple.el" :format bracket :raw-link "/tmp/emacs/lisp/simple.el::(defun next-error (&optional" :application nil :search-option "(defun next-error (&optional" :begin 28 :end 82 :contents-begin nil :contents-end nil :post-blank 0)) We were particularly interested in the computation of the values of the properties `:path` and `:search-option`. The function `org-element-link-parser` is 128 lines long, uses many regexp to do its jobs, mutates several times the local let binded variable `path` that is returned as the value of the property `:path` (the one we are interested in). The first time, I looked at its code, I couldn't understand all the subtleties of the implementation just by reading it. This is not a problem, because when reading is not enough, I always use the same strategy: I break the problem down into pieces until I arrive at simple s-expressions that I can understand. And doing it in Emacs/Elisp is super cheap because you can evaluate ANYTHING, ANYWHERE, ANYTIME, for FREE (you just pay the computation). Think about it! Fast feedback, this is the magic of programming with Elisp. So here we are. Let's say we want to be sure that the following snippet in the function `org-element-link-parser` does what it seems to do: (when (string-match "::\\(.*\\)\\'" path) (setq search-option (match-string 1 path)) (setq path (replace-match "" nil nil path))) In our example, at that point in the function, the local variable `path` has the string value `"/tmp/emacs/lisp/simple.el::(defun next-error (&optional"`. We can test the result of the `when` condition by evaluating the following: (string-match "::\\(.*\\)\\'" "/tmp/emacs/lisp/simple.el::(defun next-error (&optional") ;; 25 By reading the help of `string-match`, we know that it returns the index of the start of the first match or `nil`. Ok, there's a match. But, to me the string `"/tmp/emacs/lisp/simple.el::(defun next-error (&optional"` is to long with to many repetive characters that don't appear in the regexp `"::\\(.*\\)\\'"` to wrap my head around what's going on. So, let's use the good `foo` and `bar` words to simplify our discoveries and gain confidence about this piece of code. In the regexp, the only part "that seems" of interest is `::`, so let's try again with the strings `"/tmp/foo::bar"`, `"/tmp/foo::"` and `"/tmp/foo"`: (string-match "::\\(.*\\)\\'" "/tmp/foo::bar") ;; 8 (string-match "::\\(.*\\)\\'" "/tmp/foo::") ;; 8 (string-match "::\\(.*\\)\\'" "/tmp/foo") ;; nil It become clearer. We start to get a sense of the match. By reading the documentation (`M-x eval-expression RET (info "(elisp)Simple Match Data")`), we learn (or recall): 1. that search functions like `string-match` or `looking-at` set the match data for every successful search, 2. and if the first argument of `match-string` is `0`, we get the entire matching text and if it's `1` we get the first parenthetical subexpression of the given regular expression. So, continuing with the string `"/tmp/foo::bar"`, we have: (let ((path "/tmp/foo::bar")) (when (string-match "::\\(.*\\)\\'" path) (list (match-string 0 path) (match-string 1 path)))) ;; ("::bar" "bar") Reading the help buffer about `replace-match` tells us that this function replaces the text matched by the last search with its first argument. And if we give it an optional fourth argument being a string, the replacement is made on that string. So replacing the entire match with the empty string `""` should remove the matched part of the string: (let ((path "/tmp/foo::bar")) (when (string-match "::\\(.*\\)\\'" path) (replace-match "" nil nil path))) ;; "/tmp/foo" Now putting everything together we can write the following example: (let ((path "/tmp/foo::bar")) (when (string-match "::\\(.*\\)\\'" path) `(:search-option ,(match-string 1 path) :path ,(replace-match "" nil nil path)))) ;; (:search-option "bar" ;; :path "/tmp/foo") And maybe we've removed some misconceptions about this part of the function `org-element-link-parser`. WE ARE DONE!!!

Upvoted for enthusiasm. ;)

Now level up by forever using rx macro (compiler for a parser combinator inspired lispy DSL) instead of ugly regex ;)

Do you really think that regexps are ugly? It is really a question. This is not the first time that I read this statement and I quite don't understand why because I find regexp beautiful. Could you be more specific?

BIG THANK YOU :)

I was actually being facetious, beauty is subjective after all. But personally I really can't really see the beauty in regex. It's terse and utilitarian by design, and one might say there is beauty in that. But then I have to wonder why regex lovers are not also using APL family of languages or something specifically designed for code golfing everywhere and all the time then? I think in practice ugliness of regex doesn't matter because our regexes are usually small enough. But in general after a certain threshold terse/compressed syntax starts to correlate negatively with readability and maintainability, and my definition of ugly is not just driven by aesthetics but also this. But speaking of aesthetics, there is also the matters of regex flavours. It's not like for like comparison but I think PCRE is terse and expressive, whereas POSIX is verbose which is dichotomous to the overarching theme of regex. And then there is Emacs where due to no literal support you have escape at string level the desired escape in regex. Eventually there are backslashes everywhere, which is aesthetically off-putting for me.

Lisp overall is magic. You can do all kinds of crazy stuff. (Writing a lisp yourself is a good exercise in understanding lisp, I highly recommend it.)

+1 for `rx`. It changes your regex from a bunch of opaque or even cryptic syntax -- even for those of us who feel we can claim to understand everything about Emacs regex -- into a semantically transparent and tractable set of Lisp-y forms, the all-purpose solution to, well, everything!

I think it just depends on how comfortable someone is with regexp syntax. If you're really familiar, then regexps which are highly readable to you may be incomprehensible to someone less-familiar; whereas the clear `rx` structure is a leveller of sorts (albeit the regexp expert who doesn't know `rx` will see it as an unnecessary abstraction which makes things *harder* to read at a glance). For *complex* regexps I think `rx` becomes a good idea for readability and maintenance. The meaning of "complex" varies per-person, though.

Indeed. It is so good that sometimes I have the temptation of using Emacs instead of "real" programming languages, for application development.

Hey man, I was just going through your posts and wanted to thank you, it’s super helpful.

Thank you for your detailed answer. >I was actually being facetious Each joke contains a part of truth. I wanted yours. >It's terse and utilitarian by design I haven't thought of regexp that way but it makes sense to me. Maybe I should use another term than "beauty". Let's say that I find regexp fascinating and the fascination is due to the power and effectiveness that can be embedded in only a few characters. I'm fascinating the same way for mathematical notations and music notations. >But then I have to wonder why regex lovers are not also using APL family of languages or something specifically designed for code golfing everywhere and all the time then? Regexp are just another tool in the dev toolbox and we can like regexp without being maximalist. I think part of the job of any professional (developers, engineer, writers, …) is to pick the right tool for the right task and to understand the limit of each one of them. There is always tradeoffs. Which leads to your second paragraph with which I totally agree. >I think in practice ugliness of regex doesn't matter because our regexes are usually small enough. But in general after a certain threshold terse/compressed syntax starts to correlate negatively with readability and maintainability, and my definition of ugly is not just driven by aesthetics but also this. About PCRE and POSIX: >It's not like for like comparison but I think PCRE is terse and expressive, whereas POSIX is verbose which is dichotomous to the overarching theme of regex. It's seems that PCRE comes at some cost. You might be interested in reading the following article by Russ Cox (Go programming language): [Regular Expression Matching Can Be Simple And Fast (but is slow in Java, Perl, PHP, Python, Ruby, …)](https://swtch.com/~rsc/regexp/regexp1.html) >And then there is Emacs where due to no literal support you have escape at string level the desired escape in regex. Totally agree. Having to use 4 backslashes `\\\\` in emacs regexp to match one backslash `\` makes the regexp noisy. side note (as you talk about APL): I've never use APL but I found this talk by Gilad Bracha introducing APL interesting: [A Slice Through the History of Programming Languages](https://www.youtube.com/watch?v=91fjAbsJdUo&ab_channel=GiladBracha) After watching that talk (I don't remember when), I made some "research" on APL and read in [https://en.wikipedia.org/wiki/APL\_(programming\_language)](https://en.wikipedia.org/wiki/APL_(programming_language)) the following: >Richard Stallman, founder of the GNU Project, was an early adopter of APL, using it to write a text editor as a high school student in the summer of 1969. It seems that Richard Stallman was always interested in text editors. Have a nice day

Agreed. Lisp is magic. Did you write your own Lisp? What was the most interesting and insightful in doing it?

Yes really tempting

I'm glad to hear that it is helpful :)

The down-side with `rx` is that it's another thing to have to learn. If you already use other languages/environments that don't have `rx` - you'll need to learn regex whether you like it or not. There are also some corner cases where I programmatically construct regex - so in short. I don't think it's practical to be able to avoid regex literals entirely.

You will love Common Lisp too!

[deleted]

There is no doubt. If I'm not wrong [https://lisp-journey.gitlab.io/](https://lisp-journey.gitlab.io/) is your website. I: 1. read your article ([Python VS Common Lisp, workflow and ecosystem](https://lisp-journey.gitlab.io/pythonvslisp/)) a few months ago, 2. and watched your video ([Common Lisp: how to create a new project: cl-cookieproject demo](https://www.youtube.com/watch?v=XFc513MJjos)) a few minutes ago. You are super productive. I hope your course on Common Lisp will be successful. The day I want to learn Common Lisp, I know where to start :) Have a nice day.

I am currently still working on it, it's nowhere near complete. So far, the biggest discovery was how simple it is. You can literally just implement a lisp in 40 lines of code. Parsing is simple, it's just nested lists, and interpreting is simple as well. I just check if the list should be evaluated (whether it's quoted or not), if it should be, I check the value of the first symbol, if it's a normal function, I evaluate all arguments and call it, if it's what I call a noeval/quoted function (basically a runtime macro), I call the functions with the args as-is, and it can evaluate any of them with the eval function. Some functions are implemented outside of the lisp, such as `when`, which evaluates its first argument, if it's true, then it evaluates its second argument, otherwise it does nothing. The thing would also be fully Turing-complete just with the fn function, which takes the first argument as a list of symbols which are the arguments to the function, the rest of the arguments as the body, and returns the function (lambda calculus). Macros will be fun as well, they are basically functions which take a list and spit out a list, however they are evaluated before anything else. If the lang has a split compile and runtime, they are evaluated at compile time. Another fun thing is if you make the evaluator take a list to evaluate and a function to resolve symbols, as well as make a table/map/assocarray a function which returns a value when called with a key. Then allow this to be controlled within the lisp itself. Now you can change how every single symbol will be resolved, and for example generate functions from their names. This is actually how I implement the classic `let`. It takes a map and any number of lists. The `let` function just overlays the map given in the first argument to the environment, and then with that new environment evaluates the rest of the lists with `do`. `do` just iterates over all its arguments and evaluates them, returning the result of the last one.

I got a great day, thaaanks :) (am I productive, really? Will think this through it might help!^^)

Cool. You seems to have a great time working this lisp implementation. >So far, the biggest discovery was how simple it is. This is maybe one of the best feeling in life. If you don't mind I have a few more questions, I'm curious about how you're doing it and the methodology you're following (or even creating): 1. Do you follow some well known implementations (like Common Lisp, Emacs lisp, Clojure, Scheme...) of Lisp to do your own or you do without "guide lines/models"? 2. What underlying language do you use for its implementation? Have a nice day ;)

I am using clojure as an inspiration probably the most, since it's the lisp I know best, but the implementation is my own. Which actually makes me worry a lot that what I am making is actually not a lisp, since my knowledge of lisp is pretty limited. It's python, since I find it best for prototyping and langdev. The guidelines for the lisp I guess is to be as simple as possible, since I am mostly using it as a platform to experiment with writing various libraries. I already wrote a lib for infinite-precision arithmetic (limited to racionals and add/sub/mul/div so far), which was quite interesting. I use an older implementation with a few modifications as my shell. It has quite a lot of bugs, tho, so it'll need a rewrite. Currently I am stuck on rewriting the evaluator to not be recursive. Another guideline is to have the least amount of required builtins possible. There's many paths to take depending on which functions you choose as builtins.

Thank you for sharing your experience. >The guidelines for the lisp I guess is to be as simple as possible \[...\] Another guideline is to have the least amount of required builtins possible. Thank you for those guidelines/advices, I will keep them in mind in case I try to implement a lisp one day.

Oh those are my own guidelines which I decided on, I don't think other lisps follow them that much. For example common lisp is bloated iirc.

Thank you for the clarification. Anyway it's always interesting to have the opinion of people that have already done something one might want to try to do at some point. But I note that these are your own guidelines.