Why not built in lsp and code completion for emacs? [almost rant, sorry]

[deleted]

There are two different issues here. Some performance issues can be solved by implementing performance-critical features in the C core, as opposed to implementing them in elisp. It has been done, for example, with json parsing, which LSP uses a lot, or line numbers. Besides that, although I agree the LSP experience in vscode is impressive, bundling a LSP client with Emacs won't be any different than getting it from Melpa — you could argue it could make things actually worse, since it's harder to contribute to Emacs than to most third-party packages.

Can i ask, why it would be harder to contribute to emacs exactly?

Not exactly harder, but the mailing-list model may be unusual to some, intimidating sometimes, and some people won't/can't sign the copyright waiver form the FSF requests from all contributors.

Well, i mean yea, then it means having it as third-party means faster development, i guess. But there still should be some core parts of emacs, that could be improved to work well with lsp and code completion ui, no? one example i know is overlays (popups) are quite slow in emacs, and thats why using something like corfu or company feels so laggy.

Don't they just use posframes nowadays?

My understanding is that some part of the LSP ecosystem is proprietary, so it would never be included in mainline emacs.

I don't believe that's correct, LSP is just a protocol. The servers can be proprietary, but they aren't meant to be bundled with editors anyways.

oh really? i'd never guess that!

I have an almost complete opposite opinion with you. I think stock Emacs should including as little thing as possible. Even org-mode or tramp should be an external package. Emacs itself should be focus on being a good elisp interpreter. Also, I doubt that rewriting those lsp and completion in C would boost the performance by a large margin. Also, writing those packages in C would sacrifice a lot of the flexibility in configuration. I don’t know where the laggy feeling of pop up as you mentioned comes from. Did you set a delay on completion? Or do you have old hardwares. With eglot and corfu, I am getting a really good code completion. Edit: typo

Its really not that extreme lag, tbh and i compared emacs and vscode completion results by writing same code side by side, and found that vscode guesses, what i want, a bit more that emacs completions, tho i don't have concrete evidence and analysis on why exactly vscode has better completions, so it might as well be personal bias. Another thing is lack of consistency, in emacs, for example, in c/c++ while not using auto completion, (configuration that lets to press tab manually to activate completion ui) i found that tab does not work, cause emacs <tab> is bind to c-indent-line-or-region, which depends on c-tab-always-indent that doesn't even have an option to let you use for tab for complete, but for some reason tab-always-indent does have option that let you use tab for complete, but its only used in lisp modes. There are other example, like in rust-mode i found that completion highlighting didn't even work. there are more examples, but you get the idea, there is a lot of weirdness like this in emacs, that i dont like, and thats why im all for built in lsp and completions.

> vscode guesses, what what i want, a bit more that emacs completions it isn't the editor that does the guessing but the LSP servers, which should be the same on both sides. Either both codes are subtly different, or both editors don't install the same version, or there's some other difference between the two setups, but it's not a difference between *editors*.

Which problem specifically do you want to solve by moving it into the core? Responsiveness by moving it into the C layer? Or rather: how would moving it into the core address the issues you have?

[deleted]

both :P

[deleted]

There are proprietary LSP *servers* but that doesn't really matter for the client side. As long as they don't extend the protocol in nonstandard ways you can use them with whatever client you want.

100% agree. Although my local LSP configuration works quite well, I'm exhausted and out of ideas to even make it work with TRAMP. LSP support with proper TRAMP integration should be a game changer in mass adoption of emacs.

I think emacs should come in two options (with better names than the ones I'm using here, of course), both very clear about what they offer: \- Emacs "barebones", with the minimal setup and being exactly as you describe it. "Recommended for people who like to setup all things their own way. If you're new to emacs, be aware that this version usually needs some tweaking and time investment to get to a state most people feel comfortable using." \- Emacs "ready to go": Comes with an init file with the things people agree benefit most, and making it a version that you could start working immediately with a decent productivity. "Recommended for people who are new to emacs and wants to try a working version of emacs before making their own changes." The "barebones" would be even more barebones than what Vanilla Emacs is now, while the "ready to go" would come with more cool things by default than you have now. IMHO, current Vanilla Emacs tries to fit a middle ground that doesn't satisfy neither camp. I know the second option is somewhat covered by things like Doom, which I very much respect, but I still would like to have an option that is pretty much vanilla and don't need to add layers of complexity just to have some better defaults.

Although eglot is available on GNU ELPA, I haven't found a proposal to include it in upstream Emacs. Maybe you can submit an issue at [https://github.com/joaotavora/eglot](https://github.com/joaotavora/eglot/issues) and persuade the author? I would be happy if that happens, too.

That's why distributions like Doom and Space Emacs exist. They prepackage all of that into the main program so you don't need to deal with it. Are you using native comp? That's helped a ton since it was released.

Have you tried eglot? Eglot+tramp just works out of the box for me

There is no need to persuade them, one of the explicit goals of eglot is to be upstreamed, that is why they have been requiring copyright assignment to the FSF since day 1 even though it is developed in GitHub

There seems to be a perception issue with respect to GNU ELPA packages. A package in GNU ELPA is for all effects and purposes effectively a part of the GNU Emacs project, with a goal of being properly maintained as such. The ultimate goal, as far as i understand it, is to reduce the difference between "include in upstream Emacs" and "package in GNU ELPA" such that it makes little difference to anybody.

I did hear that Eglot had great potential to be upstreamed from times to times, just be surprised that despite 4\~5 years of development and the software is relatively stable, there hasn't been an attempt to promote it to the built-in status.

There’s a lot of noise in the mailing list every once in a while about defaults. The wiki already has a place for starters: https://www.emacswiki.org/emacs/CategoryDotEmacs From there, you can reach Prelude and get better defaults for beginners.

IMO built-in packages and other installable packages are hugely different. Built-in packages are available under any environments: `-q`, `--batch`, etc. while others are not. Also, being a part of GNU Emacs isn't important anymore from Emacs 28, from the end-users' perspectives there are no different between GNU ELPA, and NonGNU ELPA packages as they both installable out of the box. If Eglot isn't upstreamed, to me it's no different than a properly maintained package from MELPA.

eglot is the solution

> IMO built-in packages and other installable packages are hugely different. Built-in packages are available under any environments: -q, --batch, etc. while others are not. True, having certain packages part of the core Emacs distribution is certainly a convenience in some cases. > Also, being a part of GNU Emacs isn't important anymore from Emacs 28, from the end-users' perspectives there are no different between GNU ELPA, and NonGNU ELPA packages as they both installable out of the box. > > If Eglot isn't upstreamed, to me it's no different than a properly maintained package from MELPA. What specific things do you expect from Elot moving into Emacs' core? You have stated the benefits of having packages in all environments, without having to install packages. Anything more? Emacs has a long development cycle, with releases about once a year. GNU ELPA is the projects answer to the problems with that, since it allow something like Eglot to have a faster release cycle. Moving Eglot into emacs' core could be a bad idea, in my opinion. Had it been that way from the start, people running Emacs 27, for example, would have no LSP support at all, or perhaps an old and buggy version, and people running Emacs 28 would have to wait about a year for any bug fixes. It makes me wonder whether you are really looking for Eglot to be "upstreamed" or simply an easier way to discover, install, and configure packages in Emacs.

Not everyone who uses emacs codes! For many users out there, a simple org-mode setup or even less is enough. That's all they want. Unless you are talking about moving these to the C layer, I think bundling them with emacs will not make a difference in their performance.

That is kind of too much for a person that knows very little about emacs and wants to try it, IMHO. Complete beginners are unlikely to even get to that page. I'd much rather have something like that available in the front page of the emacs page, so beginners would only have to install it and give it a go.

I know this is kind of a warzone, but is it really faster than lsp-mode?

One thing to consider: if LSP functionality is moved into core emacs, what happens when later a newer and better way to do code completion rolls around? If code completion had been moved into core emacs 10 years ago, it would have created a barrier to LSP adoption. By having these kinds of things handled by packages, it allows for more innovation and dynamism in the emacs ecosystem.

Do you want a release once per year?

In the webpage of emacs there are several links and resources to get started. In “further information“, you have a link to the wiki. There is a full section on “Learning About Emacs” in there. I don’t know. It doesn’t seem very confusing to me. Maybe it can be enhanced. Adding some “I’M A BEGINNER” button with an intro not written by some emacs hacker wanting users to follow the entire tutorial before actually seeing the Emacs’s capabilities… maybe. There are lots of videos on YouTube too. I still think people just don’t want to put some time on things. Maybe I’m wrong.

Although a built-in option would be great, the current melpa packages (eglot,lsp-mode) work great. The problem with having this built into emacs is that in case of bug fixes or updates, you would have to wait for the next emacs release, which is not so convenient since lsp-servers are still rapidly improving. Have you tried using either lsp-mode or eglot and had a problem?

> LSP support with proper TRAMP integration should be a game changer in mass adoption of emacs. Just like TRAMP was a game changer for the mass adoption of emacs before LSP existed, right? There isn't going to be any mass adoption of Emacs -- it's too different to what the masses are familiar with, and that fact isn't ever likely to change. Emacs will just continue to be a game-changer for some of the individuals who make the effort to try it, on a small scale.

It's not proprietary, but it *is* (by my understanding, going by comments made by people supporting LSP integration for Emacs) something which is *heavily* tied to VSCode, and in flux for that reason -- it was invented *for* VSCode, and the protocol sometimes changes in conjunction with VSCode changes. People supporting LSP for other editors then have to play catch-up. It's been said that there's no point in having LSP in Emacs core at this point, because LSP is not stable enough for the multi-year release cycles of Emacs.

Actually I have spent a lot of time figuring out and eventually solving auto completion causing input lag. And I can safely say my Emacs with js/clangd/rust and any lsp I have tested so far unnoticeable input latency. I have to know what combination of packages and Emacs version are using first because there are multiple pitfalls in configuring those + Emacs itself

I think eglot is better, but not mainly because of speed. I choose eglot over lsp-mode is for its simplicity and tightly integration with the built-in Emacs functions. I like the philosophy of eglot where it does not do things in its own way. Other examples are flycheck vs flymake, company vs corfu, ivy vs vertico. Setting up eglot is simpler than lsp-mode, it does not need to install something like lsp-dart, instead you can just tell eglot what command to launch the lsp server, and it will do the job. lsp-mode has a lot more functionalities, but in my use case, eglot provides everything I need.

I think the reason could be partly because LSP is a moving target. The LSP is an evolving technology, and its specification has been continuously updated. It would be hard for Emacs maintainers to stay updated with it, which could sacrifice development of Emacs in other aspects. Also note that there are three well-known LSP clients for Emacs as of now, all of which are actively developed. It is not easy to get LSP right, and we don't have an answer which one could rule us yet. (IMO, eglot is the most faithful to the Emacs way.) EDIT: Grammar

I'd settle for just TRAMP working reliably by itself.

The lsp-bridge looks really nice but the dependency on python unfortunately won’t work in my work environment. I’ll be sure to try the other options mentioned in this post, thanks for posting!

I don’t think there are big speed differences - eglot works very well via tramp and the question was about tramp. I don’t think it is a war zone either. Just test which one works best for your use case.

Sometimes the completion styles affect things I think. They are definitely different sometimes though even if thats not why.

Between what OSes and shells do you use TRAMP? How does the unreliability manifests for you? I needed a few tweaks to make it work between 2 macOS machines with zsh, where my Clojure CLI command was provided by a nix-shell, activated by direnv. It was far from obvious to figure it out and although I have a Host section in my ~/.ssh/config and I could connect via TRAMP using that single-segment host alias, CIDER couldn't connect to the REPL it just started, because it would require the full hostname. Having LSP to work over TRAMP would be a similar ordeal, I guess. It was very reassuring to see it work finally, but was not exactly "transparent", as its name suggests. I also had higher expectations based on ppl's comments saying "it just works.", "i never had any problems with it"... sure, if you run it between 2 Trisquels with /bin/sh version -0.77 as the default shell, then maybe it works out of the box...

Both the software and the hardware landscape can change drastically, given the recent geopolitical tensions. Emacs wouldn't even need to change to gain sudden adoption, because it's relative position to alternatives might change, if those alternatives have issues. I don't like to think and talk about a gloom future, but I can't help it either, because I'm worried. Have you considered, that JetBrains, who has substantial developer base in Ukraine and Russia, might face serious hurdles, because of the war? Their developers might not be able to receive their salary or god forbid even get killed. If development stalls, people will migrate, some to Emacs. What if a Microsoft reverses its progressive course and do something to VS Code, which chase people away? The internet can be thrown back by decades if a couple of satellites get shot down or some submarine cables got damaged. I remember how painful was it for many days, when Hong Kong had issues around Christmas time with one of its international submarine uplinks a few years ago... Then there is the question of raw materials, chip fabs, supply chain issues. Ukraine is/was a huge supplier for noble gases needed for lasers, used by chip fabs. That's not trivial to substitute. The highest resolution chip fabs are in Taiwan, but China is violating their air space with an increasing frequency... If their fabrication capacity suddenly dries up, then we might be stuck with the hardware we currently have, for quite a few years, before new fabs are built and their output is sufficiently ramped up, to make 5nm resolution chips affordable again. You might be forced to dust off your 10 years old ThinkPad i5 with HDD and 4GB RAM, if your 2022 laptop craps itself and you can't source replacement parts for it. People won't be patient enough to run latest IntelliJ on such machines and would switch to Emacs instead! So, yeah, that's my darkest vision of how Emacs might rein ;)

also, software might implode under its own weight. meaning, it becomes unfeasible to maintain. we saw it to happen to LightTable and recently to Atom too. I'm surprised to see VS Code holding up so well... in fact, I'm worried a bit about Emacs too, because its C core is not the most hospitable code, I've ever read. given how English, the natural language, evolves, Emacs's core code base is becoming archaic and soon it will feel like reading Shakespeare or Beowulf. but still, somehow, I feel it's more likely to outlive the recent alternatives...

I used it between Linux and Linux with bash, so in theory it's as good as it gets. Even then, it's not very stable. Every time I've tried working remotely with it, at some point within a few hours emacs will completely freeze up, so I have to kill -9 the process. That alone is a non-starter. Seems to be related to loss of wifi or VPN, so my guess is that it doesn't handle losing the TCP connection. I could probably do something with TCP keepalives in the ssh settings, but one shouldn't need to know how TCP works to use this. It's bad enough you have to pick from 10 transport protocols. Asides from that, it's also kinda slow. Just opening a file is much slower than the equivalent ssh and cat, so it's not a network issue. I've tried all the tips in the manual and the wiki, and they only help so much. Third party packages are hit and miss, e.g. Helm had very noticeable typing lag for find-file over Tramp. It's not technically Tramp's fault, but it's far from "transparent". All in all it's a cool idea and 90% there, but probably needs more polish and bugfixes to be reliable.

thanks for all the background info! https://www.murilopereira.com/how-to-open-a-file-in-emacs/ have you read / do you remember this article? i wonder if the enhancements proposed there are something, which is/was incorporated to recent TRAMP versions...

Thanks for the article. I think it highlights the issue -- if you need a 5k essay and hours of debugging and programming to get moderate performance, things could definitely be better.