tree-sitter has been merged into master

What is best way to install emacs with tree-sitter on mac?

I'm so glad to see these type of things get merged into Emacs. JSON, Eglot, project.el, and now this – they all signal to me that Emacs is perfectly willing to adapt, and not slowly at all that is. Makes me happy

I'm unfamiliar with tree-sitter, but it looks like it's an alternative to a language server? What does it let you do?

It's not, it parses files (or buffer in Emacs' case), incrementally. This is useful because it'll make font-locking faster and more correct, especially where Emacs was previously using regex. It's not aimed at completion or formatting but it'll provide better syntax highlighting and better code navigation.

Thanks team, it's really cool! I'm hoping to move regex based strategies to tree-sitter, typically things like renaming parameters inside functions or classes.

> Emacs is perfectly willing to adapt, and not slowly at all The pace has picked up in the past few years. In addition to libjansson, Eglot, project.el and tree-sitter, we got native compilation and (upcoming in 29) sublinear-time overlay performance.

“Perfectly” is an overstatement, but I agree

You'd need to compile it yourself, on linux thats quite easy, on mac i don't really know. There are some mac versions, you probably have to check what these guys do

check emacs-plus@head on homebrew,

that is a great explanation! Also you can program boring editing away since you can navigate the abstract syntax tree in a more or less consistent manner: for instance https://ag91.github.io/blog/2021/08/11/moldable-emacs-editing-your-file-via-treesitter-(or-how-i-fixed-my-css-with-a-playground)/

Perfectly willing to become bloatware.

Quick question, I've been using the tree-sitter package from [here](https://github.com/emacs-tree-sitter). Is this duplicate effort at this point that I can remove once I update Emacs?

Emacs update is native support on C level

Pretend I copy pasted the "ed is the standard editor" bit here.

We're talking about things that 90% of people are going to fetch anyway as part of a standard config /initial setup. They are all behind feature flags. Nobody is forcing anything upon you. Pen and paper is not being depricated, and you dont have to downgrade or pin your version to keep using your monochrome display. People come in all shapes and sizes, and with varying degrees of grumpyness. Emacs embrace all of them.

Something I haven't seen discussed anywhere: if I already get syntax highlighting through some combination of rustic-mode and lsp, can I enable tree-sitter and have it work out of the box? Or will it be fighting with the other syntax highlighting mechanism?

However, note that it’s laggy if the buffer is big, remember to disable it if the file has say more than 500 lines

>linear-time overlay performance. ELI5? would this make things like company-mode smoother?

Emacs should be snappier when you have lots of overlays. A company-mode popup is just one overlay, so it's not going to make a difference. I'd expect Avy to be faster when hinting hundreds of matches, as well as buffer navigation and editing in buffers with with lots of images/latex-previews etc. Org-mode folding should be much faster if your Org is pinned at version< 9.0. (The latest org-mode don't use overlays for folding and will be unaffected.)

When is 29 set to release?

If memory servers from my time in Neovim, LSP will overwrite treesitter. But that might be just a quirk of Neovim/the plugin I was using.

So, now we can safely replace them with built-in tree-sitter completely?

From what I can see, the API is not the same. So, one will still need to port third-party major modes. Important built-in major modes should work out of the box though. Tree-sitter support for many core modes is a part of the upcoming Emacs 29 release, AFAIK.

It builds a *concrete* syntax tree\* (AKA parse tree) of source code; hence the name, tree-sitter. Tooling can thus leverage tree-sitter to enable syntax-aware functionality. See: [https://tree-sitter.github.io/tree-sitter/](https://tree-sitter.github.io/tree-sitter/) [https://en.wikipedia.org/wiki/Parse\_tree](https://en.wikipedia.org/wiki/Parse_tree) [https://en.wikipedia.org/wiki/Abstract\_syntax\_tree](https://en.wikipedia.org/wiki/Abstract_syntax_tree) &#x200B; \*as opposed to an abstract syntax tree (AST)

[deleted]

It should work more or less out of the box.

Not really. Most plugins that are built on top of tree-sitter will have to explicitly add support for the built-in one instead of/along with elisp-tree-sitter package.

I believe it depends on the order which you configure the modes. If you have a setup that causes conflicts tree sitter will give you a warning including instructions how to set it up properly. Super helpful. Once set up, you can toggle treesitter hl and it will just fall back to regular highlighting. edit: Well that was the case with the elisp wrapper anyway. I just realized I have no idea how it will work with native support Perhaps it'll be expose some config variables to control precedence and behavior og hl sources

How do I build `—with-tree-sitter’ on Debian stable. There is no native package. :-(

They want to cut the branch at the end of november afaik. So probably early next year.

What can `tree-sitter` be used for if one is already using `lsp-mode` with semantic highlighting enabled?

You know what they say.... "Eight Megabytes And Constantly Swapping"

Yeah and I’m just going to stick with 28 to avoid all this complexity

Is it language agnostic, or does it work only for C / lisp (I.e. emacs source)?

oh thanks for the concrete syntax tree correction: made me look into this https://stackoverflow.com/questions/1888854/what-is-the-difference-between-an-abstract-syntax-tree-and-a-concrete-syntax-tre

I downloaded tree-sitter from the github and put it in my bin, but ./configure says I don't have it installed. I'm on Ubuntu.

How am I going to even use the built-in one? I was using [elisp-tree-sitter](https://github.com/emacs-tree-sitter/elisp-tree-sitter). I know I have to add grammar for different languages, but how? I have been searching for a while and still have no clue.

Exotic modes like paredit, lispy, would be easier to make for all languages. Modes that allow you do all sorts of transformations on the structure of the code. It's easy to make these types of modes for Lisp because the syntax is already parsed out into a tree the moment you write it. Not so easy for mainstream languages, at least not until they are parsed into a tree. Tree-sitter creates that tree for you allowing really advanced modes on all languages to be created pretty easily (in comparison to making them without a tree). Also the syntax highlighting would likely be a bit more performant with tree-sitter as the tree would be in process rather than having to send some json messages to an LSP server and back.

There will be manual section.

It'll work with any language for which it has a grammar. I've been pleasantly surprised that even the more obscure languages I work with have tree-sitter grammars readily available.

If you have rust and cargo already installed you could check [the Arch PKGBUILD](https://github.com/archlinux/svntogit-community/blob/packages/tree-sitter/trunk/PKGBUILD) for inspiration. I'm not sure how easy it is to follow the steps on Debian - you'd probably have to combine the build and package steps to get the correct options for system directories.

Sure. I will try and share.

I'm still a bit sad about the schism of eglot and lsp-mode. Wanna try eglot, I like the idea of minimalism, but it doesn't work with dap-mode and it doesn't look like there is a substitute to debug Unity projects.

Ah I think I'm beginning to understand. So everyone writes a grammer for their language of choice, which acts as an interface to tree-sitter, which parses the language

That sounds great: A way to make language major modes more powerful for things like navigation and transformation would greatly complement LSP.

Ya. It does complement LSP. For some things it's not ideal to send a message to an external server. For example if your point is on a variable name, and you want to highlight all instances of that variable on the screen. That should work best by using a locally maintained tree.

You can still download version 18 somewhere. I am quite sure, it is even less complex than version 28 ;-) :-)

Any idea if Emacs will allow for defining your own grammars in tree-sitter, or it will be only possible via the tree-sitter upstream, or how will all that work when we write our own major modes for DSLs and languages? How are you going to do for org-mode? Continuing with regex based font-lock or writing an org-mode grammar for tree-sitter?

> We're talking about things that 90% of people are going to fetch anyway as part of a standard config /initial setup And letting people fetch them through their config is a superior way to go about this imo > People come in all shapes and sizes, and with varying degrees of grumpyness. Emacs embrace all of them. It would seem that /r/emacs, however, does not lol. Avoiding the pitfalls that lead other programs to become bloatware is an important conversation, and to characterize people who have these concerns as being 'grumpy' is dismissive. I'm not particularly happy when I see non-essential features merged into the main project because I think it hurts the ecosystem at large and makes me worry that the current goals of emacs of a project are to be a competitor to IDE software that I don't like by trying to capture the audiences who prefer those tools (which necessarily requires becoming more like them). If emacs takes a strong opinion on a way of doing things by adopting a package with many competitors as part of the vanilla, it weakens the entire ecosystem by legitimizing that package over the others. If a better way comes along and outstrips that package, more work must be done in the future to remove and replace it. In my opinion, Emacs' strength comes from being focused more on being a programmable productivity environment that is extensible, and efforts to make it a more 'out-of-the-box' experience leave me dismayed. Those kinds of changes will not age well in comparison to focusing on the performance and extensibility of the platform itself and allowing additional functionality to remain separate and come and go as trends change. I very much like the doom / spacemacs approach as an alternative. I think it's much healthier to have separate projects that make a complex ecosystem attainable quickly for new users and which catalogue and support as many relevant packages as possible without bias. This leaves emacs itself as a more clean, manageable codebase with a more precise set of goals. In the context of what I consider good software, scope creep is always a bad thing.

> Avoiding the pitfalls that lead other programs to become bloatware is an important conversation. Absolutely. But just dumping a drive by negative comment as a response to someone expressing their excitement, doesn't feel like an invitation to have a conversation about it. It makes you seem grumpy. To be honest, i actually tend to agree with a lot of the point you present here. It was just that balloon popping, sandcastle stomping, bullshit attitude that made me feel like shitting a bit on the previous commentor.

Many language servers dont support syntax highlighting

That's true. When I wrote the comment I think I was a little miffed that most of the comments that weren't enthusiastic about bringing outside functionality in were downvoted out of visibility, but in retrospect none of those comments were actually constructive

Tree sitter is much more than syntax highlighting. It gives you text objects for different programming languages. For example I have a function to copy a function, it works in C, Rust and Lisp with the same keybinding. Also I got copy parameters of a function etc. Treesitter recognizes the rust, C functions etc and gives you a text object to work with.

Will sub-linear overlays improve show-hide mode? I am using this regularly in json files (folding) and performance is really not that good.

missing dap and the limitation of one ls per buffer are currently the disadvantage of eglot compared to lsp-mode

What is missing though is enlargement of selection (region) by treesitter objects. Its available as a viper? extension bit not for vanila. should be the other way round. That's what I really miss!

The second, every editor would profit. Regexp-based font-locking will be a thing of the past

not yet in emacs afaik?

And works cross-editor ... it's a native compiled blob though which some find off-putting

You are not forced to use the built-in stuff, see following examples: * ido, icomplete vs. ivy, helm * doc-view vs. pdf-tools * auctex vs AucTeX There are many more such examples, You can even prevent the built-in stuff from being loaded.

Well some of them were still reasonable I think, and I understand how you feel. Like the person who got downvoted for saying they'd rather stick to 28. In a top level comment. While it doesn't contribute much to the conversation by itself, it feels much more like an invitation to discuss. They simply voiced their opinion. It's a serious problem that people use downvotes as the polar opposite of upvotes. In most cases not-voting would've been the appropriate response. Imo it shouldn't be seen as a "I disagree"-button, but as a way to help filter out inappropriate or unpleasant behaviour. When you start punishing people for simply voicing opinions that even slightly misalign with the majority, you end up with these infamous Reddit circlejerk echo chambers, and it's super detrimental to the quality of discussions. It's a bit like the software monoculture problem you mentioned earlier, except on the community level.

> Any idea if Emacs will allow for defining your own grammars in tree-sitter You will need to go through the usual tree-sitter workflow: Write the grammer js file and compile to .so file. Then, you will need to tell Emacs where that file is located. It is just how tree-sitter works. > How are you going to do for org-mode? Continuing with regex based font-lock or writing an org-mode grammar for tree-sitter? Org mode is not context-free. It is much easier to express Org grammar as recursive grammar instead of GLR-compatible grammar for tree-sitter. Also, note that Org has its own parser written in Elisp already. And the work to use that existing parser instead of regexps for font-lock is underway. See https://orgmode.org/list/87ee7c9quk.fsf@localhost

In the last couple of months, I find myself tending more toward built-in functionalities and third-party packages that depend upon the built-in APIs, so I started to use project.el instead of projectile, eglot instead of lsp, vertico and its brothers and sisters instead of ivy ecosystem, and so on. For one reason I am enjoying this process, and discovering how default functions fulfill my needs pretty well. Also, and more importantly, I feel that I am a part of a wider trend that started recently with this exact same switch. I have been using Emacs since 2017 and I was never happier.

Not built-in but tree-sitter is available as a package of course

What makes it off-putting? Isn’t it essentially the same as linking to a shared library?

Once you get used to something like lispy for slinging sexps around, delimiter/line/word/indentation/region-based programming in other languages feels like pedaling your big wheel through a wading pool of molasses. I'm incredibly excited to see what UI ingenuities people come up with to work smoothly with the *atoms* of syntax — blocks, functions, expressions, etc. Features in lispy I use a lot: moving sexps, teleporting them elsewhere, raising an sexp to "eat" its parent, slurping/barfing new sexps into/out of the current, quickly selecting a series of sexps to operate on, etc. I mean in python just moving a few lines up a level to be in the block above them is a painful indenting exercise. Faster & prettier font-locking is fine, but I'm really hoping someone uses native tree-sitter to come up with a clean, intuitive, fast, and powerful modal UI for *mogrifying the syntax tree* across languages.

Yeah but with the complexity to features trade is worth it, with tree sitter it is not

I've previously built Emacs 29 on Debian, and I just rebuilt it with tree-sitter support. First, build and install the tree-sitter library (`libtree-sitter.so`) with: git clone https://github.com/tree-sitter/tree-sitter.git cd tree-sitter # NOTE: update version in Makefile to 0.20.7 sed -i 's/^VERSION := 0\.6\.3$/VERSION := 0.20.7/' Makefile make # NOTE: manually correct 'name', 'version', and 'provides' sudo checkinstall --pkgname tree-sitter --pkgversion 0.20.7 --provides tree-sitter --default sudo dpkg -i tree-sitter_0.20.7-1_amd64.deb sudo ldconfig Then recompile Emacs 29, making sure to add the `--with-tree-sitter` argument to the `./configure` line. In my case, this looks like: CC="gcc-10" CFLAGS="-O2" ./configure --with-native-compilation --with-json --with-cairo --with-tree-sitter --prefix=/opt/emacs29

(I haven't compiled any language parsers yet, so major modes such as `bash-ts-mode` don't work for now). **EDIT:** The language parsers provided by [tree-sitter-langs](https://github.com/emacs-tree-sitter/tree-sitter-langs) work with the built-in `treesit` library.

Thanks for these steps.

Interesting. What's the use case for two ls in a single buffer?

Of course, and I do use quite a bit of extra packages (including Ivy instead of Ido because of swiper and counsel) but it is kinda weird to have a built-in functionality using up storage while using an extra package that does "the same". I actually didn't know I could prevent it from being loaded. I thought it wouldn't already unless I have some "require" somewhere.

Editing html with embedded CSS and javascript

> it is kinda weird to have a built-in functionality using up storage while using an extra package that does "the same". Emacs has already multiple redundant mail clients, and multiple games, and completion frameworks (ido, icomplete, ...), and themes, all built-in. Their compile time matters, but otherwise does that disturb you? :-) By today's standards this space waste (ido.el.gz is 43 KB) is neglectible. Do you have a web browser installed, or use snap/appimage packages, or docker or a virtual machine? That's wasteful, because they all bring there own binaries of system shared libraries. And if those few megabytes really matter (embedded system), I imagine it is not difficult to create some minimal Emacs distribution with everything superfluous removed from source. > I thought it wouldn't already unless I have some "require" somewhere. That! But there are autoloads and OS distribution init scripts.

Thanks for the answer. C and C++ are not context free either, but they have grammars :). Anyway, I understand your point, and agree with it. Just wondered if everything and everyone is jumping on the tree-sitter train. I am currently writing a small blog generator and [experimenting with writing HTML as symbolic expressions](https://imgur.com/a/55TZI49), I call it shtml, and wonder if I should use tree-sitter or continue with font-lock. But seems like font-lock is currently the only option considering that I have to implement a shared .so library in tree-sitter case :). It was an interesting read about org parser. There is so much to follow and so little time, so I have missed that. I basically don't follow much of mailing lists anymore. Also have to finish that org-capture thing I started long time ago. Sorry for being lazy, life just happened, and now it is hard to get back to it. but one beautiful day I'll come to it again :).

> C and C++ are not context free either, but they have grammars :) Sure. Implemented as separate supplements in C. It is more practical to keep Org parser in Elisp and hack there rather than forcing Org contributors to learn grammar writing in tree-sitter + its C API. If anything, PEG grammars might be more suitable for Org and a number of other languages. See https://yhetil.org/emacs-devel/877d07a16u.fsf@localhost > Just wondered if everything and everyone is jumping on the tree-sitter train It is handy when a grammar is (a) stable; (b) already maintained by someone else. (c) do not need to be tweaked for Emacs purposes. Basically, less headache for Emacs maintainers. > shtml There is a built-in sexp parser in Emacs. You can call it using `read` ;) You can even interpret html sexp by calling `xml-print'. > I have to implement a shared .so library in tree-sitter case Note that Emacs has a built-in LR parser. Bovine. > I basically don't follow much of mailing lists anymore wrt Org mode, we provide the most important announcements via rss: https://updates.orgmode.org/ Life is life, indeed. In free software community, contributions are appriciated, but not mandatory.

> There is a built-in sexp parser in Emacs. You can call it using read ;) You can even interpret html sexp by calling `xml-print'. Yes, I know, I am using built-in read and list parsing stuff, already :-). Actually I am reusing the entire elisp mode, but, there are few twists, unfortunately. I also took a small opportunity to write slightly more literate code by not requiring comments at the top level. It is just an experiment. So I have to do a bit more, but it is not so complicated, and not too hard to implement it. > https://updates.orgmode.org/ Cool, didn't know about this one. Thank you!

You're completely right. That irky feeling is just that: a feeling. It is not rational, hehe. I can live with some redundant KBs of code. Another issue are the human hours, but it is not like developers are just going to throw away their project to work on the built-in instead. Last time I saw eglot and lsp-mode developers talking it was not super friendly. So, it is what it is.

Update: presidential color codes red trump blue ..also find wiki diagram at cross section ~microsoft,