Quick Intro to Emacs Lisp Regression Testing

A few days back, I started working on a new Emacs library. I always had plans to learn how to use ERT. I finally got around to learning the basics of ERT and cannot believe how easy it was. So here's a post on that. I'd welcome corrections/suggestions/tips and tricks related to using ERT.

Nice post. I don't see any issues with the post... only that in my experience most of the hassles regarding testing came from having to initialize packages, especially if there are indirect dependencies that aren't distributed with emacs. Having to properly load melpa or manually enter a path for a 3rd party package is quite awkward - and I'm not sure if there is a defacto-standard way of dealing with this. Also, simulating keyboard input is something that doesn't seem to be supported out of the box (at least not conveniently), I wrote tests for [evil-numbers](https://github.com/juliapath/evil-numbers/blob/master/tests/evil-numbers-tests.el) where I ended up having to create some macros to handle this.

Thanks for the write-up. Personally I think property-based testing is miles ahead of other approaches, at least for code without side effects. There isn't anything in the elisp world as powerful as Haskell's QuickCheck framework AFAIK but https://github.com/Wilfred/propcheck is a good start.

>most of the hassles regarding testing came from having to initialize packages, especially if there are indirect dependencies that aren't distributed with emacs. I have a generic elisp snippet for downloading dependencies that I use for ox-hugo testing. It basically loops through a list and calls `package-install` on all of them. That wasn't much of a hassle. But it was challenging to figure out how to override the Org installation cleanly on top of the built-in Org version. >Also, simulating keyboard input is something that doesn't seem to be supported out of the box (at least not conveniently) Yes, I think that is a bit difficult; in general, testing interactive calls of functions. I figured out [how to test sections of code in `called-interactively-p` and `use-region-p`][1]. Next, I need to learn how to test a function that take in input interactively via `read-string` from the mini buffer. > I wrote tests for evil-numbers where I ended up having to create some macros to handle this. Thanks for sharing that! I've made a note of its as I could need that in future. I agree that it would be nice to have built-in emacs libraries to do these kinds of test setups that do region selection, simulate user data entry in mini buffer prompts, simulate key bindings, etc. [1]: https://github.com/kaushalmodi/baser/blob/ffcf615b25c018cda2fd8029510f720b2892d886/test/baser-test-lib.el#L22-L33

>Personally I think property-based testing is miles ahead of other approaches Thanks! I wasn't aware of "property-based testing". I think it's probably what's called _formal verification_ in my domain (ASIC verification). There, the idea is that the design under test is provided all sort of random vectors by the tool in quick succession and the assertions or "properties" set for the design must always be true. If it fails, then we likely found a bug. I see the benefit of property based verification because the range of input stimulus is not limited to the values the user thought of when writing the test. The test's robustness comes from the assertions or mathematical rules the user writes. I see the value of these. I also see a value of traditional testing to cover the scenarios that the assertions cannot, like testing some code on how it would behave interactively. > but https://github.com/Wilfred/propcheck is a good start. Thanks! I'll take a look there.

Yes, I have often found edge cases that I hadn't thought of via property-based testing. Sometimes I think I'm "certain" these won't occur in production, but after thinking about it I might realise they are possible - either way, dealing with them can make the code more bomb-proof. I think you're right that property-based testing isn't a good fit for integration testing. It's not necessarily the right way to test anything interactive either, but mocking can help there. The nice thing about QuickCheck is that as well as supplying random values of basic types (numbers, strings etc) it is easy to put constraints on that (numbers in a certain range, strings with a certain property) and to make arbitrary values of your own datatypes (e.g. if you've made a datatype of Sudoku puzzles when writing a solver you can tell the framework how to create lots of arbitrary puzzles then try solving them).

> I'd welcome corrections/suggestions/tips and tricks related to using ERT. The only ERT-related tip I can think of is to read its manual, as it contains helpful information that might otherwise be overlooked. Besides that, I'd suggest also looking at the Buttercup framework. ERT is great and I use it in some projects, but Buttercup also has its own strengths, and I use it in some others.

> The only ERT-related tip I can think of is to read its manual I agree. That's why I mention how to access the ert Info manual in the beginning of the post. _I think I should mention the manual in the Summary too.. will do._ > Besides that, I'd suggest also looking at the Buttercup framework. Buttercup is on my list to learn. btw on its README page, it mentions _"ERT has no good way of being run in a continuous integration environment."_. I didn't think so. All I needed was a 1-liner Makefile target and that's it. I think I haven't yet come across the CI setup issue that the Buttercup author is referring to :).

> Buttercup is on my list to learn. btw on its README page, it mentions "ERT has no good way of being run in a continuous integration environment.". I didn't think so. All I needed was a 1-liner Makefile target and that's it. I think I haven't yet come across the CI setup issue that the Buttercup author is referring to :). I don't know what it is referring to, either. Here's the code I use in `makem.sh` to run ERT tests: https://github.com/alphapapa/makem.sh/blob/7096e02752c28ef6fd8bf1684e8d1069be4c6dd2/makem.sh#L1000 But Buttercup has been around for a while, and ERT has probably seen improvements since Buttercup was first written.