LEM - What If Emacs Was Multithreaded

“were”

[deleted]

With conditional (i.e. "if") either works, which I believe GP was hinting at.

Since we’re discussing a hypothetical we need to use the subjunctive mood, which is “were” even in the singular. > In these cases, the key to understanding when to use was or were in a sentence is determining whether you need to use the subjunctive mood or not. A verb is in the subjunctive mood if it expresses an action or state that is not reality. For example, it might be hypothetical, wished for, or conditional. — https://www.grammarly.com/blog/was-vs-were/

This is interesting but multithreading or its use wasn't showcased, I hope to see it though!

[This is actually a little known bit of English that's a bit confusing.](https://en.wikipedia.org/wiki/English_conditional_sentences#Counterfactual_conditionals) u/subderisorious is correct if pedantic.

The threads I made in the demo e.g. the sleep example are running in the same session as lem. That was the multithreading showcase. They are used for a few things in the internals of the editor but really I wanted to show the power it offers for extending it.

Wow… just… wow… what did I just watch?…

Thanks for the correction ☺️

[deleted]

r/confidentlyincorrect

Ah, cool

> pedantic Guilty as charged.

Very cool, although I'd personally wait until it's a solid replacement for Emacs. Hope they get GUI support eventually (terminals tend to disable bindings I depend on).

It does support multiple backends but at the moment I think that's only the terminal and electron. Not sure if you consider electron a deal breaker. I do hope it becomes more stable.

and opengl: https://github.com/gregcman/lem-opengl ping /u/ideasman_42

That's not really showcasing threading, as waiting for something without blocking doesn't require threads and is easily handled in Emacs. Threads could bring broadly easier and more efficient multi-core parallel execution for speed, but otherwise they just sidestep *one* error for package writers where bad code lead to blocking, while introducing a whole host of other potential error vectors. Writing asynchronous code is often much more straightforward with an event loop than a threaded system because you then rarely need synchronization and locks. Which is also the reason making Emacs multi-threaded would be a huge job - because it hasn't had to have those locks and introducing them at this stage would be a nightmare.

Electron == Not a chance.

Well ya it's not a good usecase I guess if all you did was sleep it was just an example of something that could be replaced with something like a long running elisp computation. Often this could be done with async.el but that loses the entire environment and requires a slightly hacky method to even pass parts of it in. Seems like my example wasn't clear enough.

Oh I almost forgot about that thanks.

Well it's not really the only option but it does seem to be the main GUI option they are considering. As mentioned below there is a OpenGL option.

😱

[deleted]

Wondering if it's unmaintained (last commit was from two years ago).

The real problem with replacing Emacs is not the core. To replace the whole ecosystem and community would be an impossible task. Using Guile was considered in Emacs, but who would port all of the Elisp code?

Sorry, but Electron *is* a deal breaker.

I don't understand what your video has to do with the title? What if Emacs was multitjreaded, and then you are just showing of another text editor. I mean it is nice Lem i implemented in CL, bur your titel is cimpletely missleading. Is Lem itself multithreaded or what? What is the point? Which part(s) of Lem is/are multirhreaded? Observe that none of CL implementations currently have their command loop itself multithreaded. Does Lem have it? I didn't got it from your video. I am not sure if any even have a parallel garbage collector yet Having threads is good, but parallelazing interactive application like a text editor is hard. Games, and resource heavy applications like 3d modelers and similar, have some space for utilizing threads, but an interactive text editor like Emacs is relatively light on resource usage, and much harder to parallelize.

Guile can run elisp

Could you point out any of my inconsistencies? As far as I can tell I didn't make any. I definitely didn't say it's a superior solution just a personal opinion on the language and the possibilities it offered.

I clearly should have been more clear on what is multithreaded and what isn't. Specifically how it can be done in lem. I am sorry if you feel mislead I'll make a follow up going into much more detail. It seems like many are upset I didn't focus on it more. Yes multiple threads are available in the editor and the language it's self supports native os threads. For reference on what is and isn't multithreaded In the editor here are a few uses of threads in the editor https://github.com/lem-project/lem/search?q=make-thread&type= The do not have parallel garbage collectors no. A good example of the power native os threads offers is a way to improve the performance of basically any async elisp transaction. A good example is with org. Due to many functionalities requiring the entire environment (mostly just the global vars) with something like async.el this isn't an option. Multithreading allows for many of these to be optimized to utilize multiple threads with access to the current environment.

Luckily there is an OpenGL option but I have not looked into it myself.

Totally I definitely see this as an issue. I am not trying to say this is a perfect solution or recommendation to use it over emacs. If anything this is just a project to keep an eye on. It's not truely intended to replace emacs and it's ecosystem. I didn't mean to give that impression.

Can you run multiple windows in the same session the way you can with `emacs --daemon` and `emacsclient`?

Unfortunately no. For me this is a big lose. I honestly should have covered.

Some corrections I'd have is on the topic of libraries there are libraries for database interactions. I didn't mean to imply they didn't exist just that there aren't many options and most aren't really intended to cover a wide array of uses. Maybe that redo got added recently since you could always undo an undo or use a package. Maybe I should have been clear that emacs also is heavily (nearly completely written in elisp) though I should have made the effort to fully understand it's support for ffi since I had more knowledge of cl's support for cffi.

Anyways if you have any other corrections besides just blind criticism I'd be happy to add it in a pinned comment with my other points.

[deleted]

[deleted]

People mention how hard it would be to replace Emacs ecosystem. I honestly don't think it would be all that hard. I'd even go as far to say that some packages would benefit from being re-written from scratch. I love Emacs, but if/when a CL Emacs-like editor comes along that's stable enough, I wouldn't hesitate to port everything I've written to it.

Why? Such that you get access to namespaces or don't have to write cl-prefixes to access cl functions? Having no namespaces is actually a feature of Elisp. Multithreading brings its own set of problem as long as you don't introduce strong isolation (like in web workers). The cl language implementation and runtimes are of course better and faster. But is the language itself significantly better than Elisp with lexical scoping at this point?

Why? How is CL significantly better than Elisp?

Why would not having namespaces be a feature?

Nice to know for Emacsers, Lem has a little dired mode too. Very cool video!

re libraries, what I would say is that the elisp and CL libraries don't target the same world: CL is an industrial-grade language, elisp is targetting a text editor. I know, the Emacs world is so big that we have a web server written in elisp, it's crazy, and anything can be done in elisp. But would you build your startup's application with the elisp or the CL one? So, building on CL libraries opens another world of possibilities. That's what I find super attractive and why I look at the two ecosystems differently, maybe similarly to OP.

> But is the language itself significantly better than Elisp with lexical scoping at this point? Yes: - Package system (Have you written much CL? The package system is much more than just not needing prefixes) - Condition system - CLOS and MOP (eieio is something at least, but it's not CLOS, and it seems like a lot of developers avoid it) - Better type system (especially numeric types, see also [coalton](https://github.com/coalton-lang/coalton)) - Symbol macros, read macros, etc. - Real multiple values - Great libraries like [trivia](https://github.com/guicho271828/trivia), iterate/for/alternative loop libraries, alexandria, and a hundred others. Common Lisp is a general purpose programming language with good support for ffi, working with files, databases, images, audio, etc. Just skim [awesome-cl](https://github.com/CodyReichert/awesome-cl) if you haven't. You could argue this doesn't have to do with the language, but a lot of these libraries are so good (or even possible) in part because of language features elisp does not have. - cl-lib is a subset of CL functionality, and CL has first class support for a lot of things that are an afterthought in elisp - etc., there's so much more I've written tens of thousands of lines of elisp at this point and enjoy it, but elisp is a very poorly designed language by comparison.

If they have GL it is fine. I am surprised they didn't choose Skia, since it's C interface should be easy to expose to FFI, and it has OpenGL backend as well. I am not at home, I have only access to my phone, so I can't look at the code. Will take a peek at Lem next week.

Thanks for the link. As it seems threads are used to start external process, in almost all of those gists Emacs actually has threads which you can use in same manner, but Emacs can also run asynchronois processes too whi h is easier. The OS will (hopefully) schedulle those on other cores. When it comes to text Editors and similar applications, accelerating guis and parallellizing interactive user tasks is not trivial. Partly, there is a lot of global state, partly lot of tasks take longer to send over to other cores and collect results than it takes to execute them on a single core, since user would like to have feedback instantenious and actually synchronous. When we press a key, welike to see instant feedbacka nn the screen. The feedback usually depends on the immediate surrounding of the key, so there is not much to parallelize there. With that said, typical use of threads in guis is ti populate long lists for menus and tables and such with data and to display partial results on the screen as a menu for example is populated, so that user get some feedback instead of seeing "stuck" application. In GUIs those threads are usually called worker threads. In Emacs such use case would be for example populating candidate lists in completions in parallel. I am not familiar with Emacs internal implementation of completing read, if they use threads or if it is single threaded, but there would be a n opportunity for example. Menus and gui in Emacs are relatively spartan so there is not much need to multithread those. Another example would be to do search or syntax highlight multithreaded, or grammar checking as I saw in one of your gists, and then to just use results to present data to user. Emacs built in interactive search and syntax highlight (font lock) even spell checker get around this in interactive mode since they really need to just spellcheck, search or colour one screen of data at a time as max. That is not much so multitheading is unnecessary there. It is when we are using completion frameworks like Helm, when we need entire buffers or files to be searched in entirety to populate some candidate list or some buffer or something similar where threading could come handy in Emacs. But for me even helm-ag when searching multiple files is seldom so slow that I notice a delay. But I don't work with multimillion lines of code, so I should not speak too much here 😀. Threads could be used to exec external tasks like check mail, backup your files, copy big files etc. But those are all not really interactive tasks, and Emacs can execute them in async process, already, so we get same effect. So absent of threads is not *that much* of a problem in Emacs. As Monnier put it in a pdf about Lisp development, Elisp is general language, but not for programming in general. I think it make sense, and I think it translates to Emacs in a way that for those tasks we normally do in Emacs, we can do them *mostly* good enough without threads, with async processes. For programming in general you would definitely like to have threads in Emacs, but observe that Elisp purpose is far and foremost to configure and extend Emacs itself not to solve tasks for oil industry or military industry etc. It would be nice if garbage collector run in parallel, without stopping the world, but on newer cpus with more ram, running it more often in shorter bursts works just fine and achieves de facto a sort of cooperative threading, but without threads 😀.

It's a more powerful language. Two fearures off the top of my head: - namespacing - reader macros Package management/declaration is handled better. Im sure there are more points in favor of CL. As someone who writes elisp daily, I find myself (and many of the high quality libraries I study) lean more and more on the Emacs's "cl" emulation library.

Thank you for your answer! It is undeniable that you get access to tons of existing libraries. On the other hand you lose all the existing Elisp code. Overall I remain unconvinced regarding the language argument in this case, but this is probably due to my ignorance. I just don't see a big difference. CL is an old language as is Elisp. As you point out, Elisp is still lagging behind, EIOIO lagging behind of CLOS etc, no reader macros, poor numeric types. I agree that Elisp had many poor design choices, but some of the more serious issues got fixed over time. It is not as if CL is a language based on the newest programming language research either and for its time it was widely considered as overly complex. The points you mention seem like details. I believe that languages do matter and it makes sense to chose them wisely depending on purpose. I just don't see how a switch from Elisp to CL would magically resolve all the issues as some people seem to assume. For example multithreading - how is this supposed to work in a highly extensible, heterogenous system if you don't want to introduce numerous issues? There must be a reason why languages with concurrency push a lot of guarantees into the type system, use immutable data structures or give up on shared data altogether and rely on an actor-style approach. I think one can enjoy programming in every language if one accepts the rules of the game. It can be inspiring to move from language class to language class, e.g., low level ones, Lisp, Smalltalk, SML or Prolog. In the case the of Emacs, the Clojure guys claim that Emacs would shine much brighter if written in Clojure, the CL guys claim the same with regards to CL, the Scheme guys say one should switch to Guile and I don't see the big difference. Of course I am simplifying greatly here and your answer did address some specifics. These specifics certainly matter if you are hitting ugly edge cases all the time. Maybe that's how many people feel about Elisp? But it seems to me that one can still do many things with Elisp and as a Lisp, it is not intentionally lobotomized like Go without macros or generics.

Couldn't have said it better myself. Still love elisp but for me CL is just a more enjoyable experience.

Emacs usually (depends on version and system) has cooperative multithreading but never competitive multithreading, I'm just saying that the premise is false that Emacs lacks multithreading

I totally agree I don't think the language difference would completely change the game. I think the improvments it offers are great for me. I by no means think elisp is a terribly language. It's great just not as powerful as cl. But with great power comes great responsibility.

And some dogs can walk on their hind legs. Guile Emacs is actually a project, but Guile’s Engine is very poor at running Emacs Lisp.

I get it. I’m just pointing this out because many kids think it’s all about coming with new ideas… but big, ongoing communities are the reason we have good and maintained free software. It gets a lot of time from a lot of people to reach Emacs’s actual state. Good ideas come and go.

> It is not as if CL is a language based on the newest programming language research either and for its time it was widely considered as overly complex There are not new programming languages that provide the same experience in terms of iterative/image-based development and expressiveness. In what new non-lisp language can I be in the middle of running some program with complex state, encounter an error, fix the error in the code, and then continue running the program as if the error had never occurred without needing to spend time to restart the program and get to that same situation? Most new programming languages don't even allow for interactive/iterative development as well as elisp. New programming languages are still reinventing worse versions of things that common lisp has had forever. And as the programmable programming language, it can adapt without the need to design a completely new language. > more serious issues got fixed over time > The points you mention seem like details Common lisp's condition/restart system and CLOS are on a completely different level from try/catch and OOP in other languages. And while multiple values, symbol macros, and read macros are more minor, the little things add up. It's also important to remember that part of the reason common lisp has such incredible performance is not just incredible implementations like sbcl but also due to the language design itself, e.g. macros, compiler macros, better utilities related to these, and a compiler that helps you optimize your code. In what other language can you achieve better performance *because* of using higher level abstractions? > the CL guys claim the same with regards to CL, the Scheme guys say one should switch to Guile and I don't see the big difference. The difference is that emacs lisp is very similar to common lisp but quite different from clojure, guile, or anything else, but there's also a pretty big difference between elisp and clojure to me. > I just don't see how a switch from Elisp to CL would magically resolve all the issues as some people seem to assume > On the other hand you lose all the existing Elisp code. > But it seems to me that one can still do many things with Elisp and as a Lisp, it is not intentionally lobotomized like Go without macros or generics I agree. I think Emacs would be a lot better if it had been written in common lisp, but short of going back in time and using common lisp in place of elisp or newly adding the ability to use common lisp in Emacs, I don't see much benefit of switching to CL. And a different language doesn't magically fix design issues like you say. Maybe lem will be great in a few decades if it continues to be developed, but Emacs has so much more momentum. Elisp is good enough for a text editor (though it would be cool if you could do something like nyxt directly in Emacs).

It's always been on my list to look into what Electron is and why I see so much hate for it. I just googled the most popular electron apps and was really surprised to see several big names that I recognize. What about it brings so much ire from the community?

I'd like to write about that. For me it's about language features (namespacing, an interactive debugger, good threads story, rich exception handling system, built-in and rich object system (who uses EEIO, and how is happy about it?), super efficient, access to low-level…) and libraries (libraries targetting "the real world"© and of industrial grade).

> or give up on shared data altogether and rely on an actor-style approach what's nice in CL is that we can choose. We have a nice actor-style library now (https://github.com/mdbergmann/cl-gserver).

These are the argument I collected throughout the years (doesn't mean they are mine): - It uses unnecessary amount of resources because it's a "stripped down" version of chromium, but still being chromium though. - It depends on chromium, if it brokes, electron brokes. If chromium sucks, electron does the same. - Too much layers of abstraction. - No native looking. - No integration in the system (because the app is just a web page after all). - NPM sucks. - NodeJS sucks. - JavaScript sucks. - Google sucks. - Good software is written in real programming language, C and using proper widget toolkit, GTK.

Thanks for collecting those. I did a fair amount of work with Node/JS and used to be a Google fanboy when I was a web-dev many years ago - it was nice break from Java Enterprise + maintaining IE6 support. Since my primary language then was Java, all of my "desktop apps" looked horrendous. If I were to write an app today I might consider using Electron as a UI layer only, or writing a PWA since I need cross-platform support. The Electron API does seem full of awkward object-oriented layers that turn me off. Maybe I should give GTK a closer look.

> Good software is written in real programming language, C and using proper widget toolkit, GTK. I'd rather it be written with a good programming language though, like Common Lisp or Ada (I'll settle for others though), instead of one that's asking for bugs like C. It may use GTK however. But in any case all of those will vastly outperform Electron-based programs on essentially all metrics.

> Since my primary language then was Java, all of my "desktop apps" looked horrendous. The trick is to use binding libraries for better graphical frameworks (e.g. [Qt Jambi](https://github.com/OmixVisualization/qtjambi/wiki)), because most [Swing](https://en.wikipedia.org/wiki/Swing_(Java\)) implementations screw it up.

https://github.com/tauri-apps/tauri

Ya that was my original point behind the statement. There are lots of great packages for emacs but it's not really interested or used in production applications and if it is that's very very rare to see. As you mentioned, this opens up a whole new ecosystem with very different goals and values.

[deleted]