Living The Eshell Dream: A Reduction in Latency From 70 Seconds to 3 Seconds

That's pretty cool! does this cause any problem at all?

Great job! Any plans to upstream?

I have it in my current build, I haven't observed anything yet, but I wouldn't be surprised if there are more edge cases. I have it implemented right now to only support PTY processes because I have the more experience in using those than the rest. The commit itself should merge cleanly with a cherry-pick if you wanted to add it. The main way it failed before getting most of the bugs worked out was that the notification fd set would get out of sync, which would cause emacs to no longer block in the select loop, which would cause emacs to take 100% of a cpu core, but it'd still stay responsive since it's still receiving user input, so in at least that situation, you wouldn't lose any work. I haven't observed any memory corruption bugs from it.

Yes, I would like to, although right now it's limited to only handling PTY subprocesses. I'd like to make it handle all subprocess types before trying to upstream it.

That's great, sadly I'm not using `eshell` anymore (I switched to `vterm`) but it's always fun to read about the internals.

Interesting find! You seem to have tested this only on MacOS, are you planning to reproduce this test on other OS?

I am planning on trying it out on some distro of linux as well. As far as I know it should build correctly as is, but I haven't set up a build chain for emacs on linux before so it'd take a non-negligible amount of time to test out.

> However, it has the appearance of being slow in comparison to other shells. shells have nothing to do with output speed, I suppose you are talking about terminals Another thing is why people may want to see the whole 10Mb compilation log in realtime? Redirect it to a file, M-x grep the things you need, and you are perfectly fine. There's also https://gitlab.com/niklaseklund/dtache

You are correct in a pedantic way, but eshell is acting like a terminal emulator so the terminology is a bit mixed here.

Nice! Would this also speedup the normal shell mode? ("M-x shell") (I know some people swear by eshell but I've never made the switch...)

Yes, it would. It's a little misleading to say this is specific to eshell because really it improves process output handling for any subprocess in emacs that uses a PTY (which "M-x shell" does.)

Running bash in shell vs running it in iterm with the command being \`time sourcekitten complete\`.: iterm: real 0m0.146s user 0m0.033s sys 0m0.015s ​ emacs M-x shell Before Change: real 0m9.585s user 0m0.038s sys 0m0.022s ​ emacs M-x shell After Change: real 0m0.146s user 0m0.033s sys 0m0.017s So, it looks like with the change, there's about 0.002s of sys overhead in comparison to iterm, and approximately a 98% reduction in latency compared to running it in emacs before the change. Edited to include emacs without the change

Wow, I guess this would speed up all sorts of subprocesses that spew a lot of output (I disabled file change summaries in large git repos because magit would choke updating them). Add me to the people who hope this change can be upstreamed.

wow that is an amazing improvement! I use eshell every day so if you try to merge this upstream I can buy you a coffee

DUDE, this is gunna change my life! Fantastic work. Well, if I'm able to compile emacs with the patch. Let's give it a try... Edit: Compiled w/o issues. Tested by doing "time head -n 1000". The time reported by time was way faster (5s to like .1s), but emacs was still very slow to actually catch up and display all the text, which makes sense I guess. What I really want is a way to make shell mode somehow interact with a large body of text more like emacs does with a file. (Or for emacs to just be faster in general. :-)

Question: one thing that’s nice about existing Emacs behavior is shown in this example: imagine you are on a console tty port - which at 19.2kbps output is the display bottleneck in getting the output of a verbose output compile of a large source package. Right now, in shell mode, the output fills the buffer, but the processing of my input is immediate. I can do c-x o to switch to another buffer and do things, and there’s next to no delay in the processing of that command. Similarly, in the shell buffer I can stop the compilation with C-c, and it is processed immediately. This immediate handling of input given a barrage of output queued up and backlogged by the slow tty port speed is a good and necessary thing. Do your changes impact this behavior in any noticeable way?

If you could provide me a way to emulate your setup, I’d answer with a bit more certainty. However, I’d imagine that it does behave like you’re saying. Stdin processing is not handled by the background thread, and the total size of the buffer surfaced to emacs at a time is limited to read-process-output-max. The responsiveness at that point depends on the how fast your processor can execute elisp

What’s your read-process-output-max set to currently out of curiosity?

4096 (the default, I guess)

We’re in the future man. Set that to at least a megabyte

Hi - I tried cloning your code onto my GNU/Linux system. I ran into a couple of issues in process.c The first was the gcc compiler not liking the definition of PROCESS_OUTPUT_MAX because it is a variable which is determining the size of file global arrays. Changing it’s structure to a #define fixed that. The second is that FD_COPY is a BSD thing, not Linux. I didn’t have time in the brief moment available to check for what FD_COPY does in detail to see what a quick change to build the equivalent logic in Linux would be. I figured mentioning this would be useful if you want to upstream this.

Thanks for the heads up, I appreciate it. Definitely a worthwhile comment, it's legit.

wilco!

Things ive noticed quickly, not a knock on your code.. great work ! ​ FD\_COPY would need to be re-implemented for Linux as I dont think its on Linux. ​ You can probably get away with something like: \#ifndef FD\_COPY \#define FD\_COPY(dest,src) memcpy((dest),(src),sizeof \*(dest)) \#endif ​ Do you want pull requests sent to your git repo ?

Yes I do! No promises on how fast I reply though, tbh.

Will clean up my mess and send pull requests within 15 hours. ​ Edit: my build has a problem connecting to elpa , i think its related to this patch, trying to figure out why.

"You shouldn't want to do that" is rarely a helpful thing to say.

Does this work for linux?

> eshell is acting like a terminal emulator it isn't, try `echo $TERM` in it

> "You shouldn't want to do that" is rarely a helpful thing to say. I didn't say that, I asked, "what for?"

Redirecting is an inconvenience, and it's preferable to not have to ask yourself before each command "is this going to produce enough output to lock up my terminal for minutes?"

>Redirecting is an inconvenience well, there's https://git.sr.ht/~niklaseklund/dtache

I guess I'll counter with the question: What's wrong with wanting something that doesn't need to be slow to not be slow?

[See if this discussion helps](https://stackoverflow.com/questions/2421672/are-there-any-platforms-where-using-structure-copy-on-an-fd-set-for-select-or). Could this work (as suggested in that SX discussion): #ifndef FD_COPY #define FD_COPY(dest,src) memcpy((dest),(src),sizeof *(dest)) #endif If you would also be interested to try the patch, however, I am not interested in cloning another, possibly old Emacs. I have tried to apply [the patch from the commit](https://github.com/tyler-dodge/emacs/commit/e56c55a742c0f0d152afb4958b863cdf7207b4c3.patch) in a recently pulled master (as of today), but the patch can not be applied. But if you manage to get it to work, I'll be glad to tinker more with it. I don't have time to manually port this into my Emacs, not tonight. But if you confirm it works, I could try :).

How old is your fork? I created a patch/diff as described in the comment to jaythebookfan, but it does not apply. Can you create your patch against recent master? Really great work otherwise, I am also very interested to get this patch to work. Kudos for the work and thanks for sharing!

> What's wrong with wanting something that doesn't need to be slow to not be slow? Things don't have to be as fast as possible, they just have to be fast enough. And while there's nothing wrong with wanting them to not be slow, but what if the problem is not with the slowness but with the approach itself? Do you remember the eternal issue with linum being slow on large files? I was used to line numbers for some years before switching to Emacs, thus one of my first wishes was to enable line numbers. On of my PCs was a bit old (P-III in the times of emacs 23-24), so I had to disable linum completely. After some time I realized I didn't miss it at all, one just don't need them. After some time I found `avy` and it became even more clear than there's absolutely no sense in linum. Less distraction, more ergonomics (no need to change eye line, `O(logN)` instead of `O(N)` etc.), more screen space, so even when `display-line-numbers-mode` appeared I was a bit confused why the hell anyone may want to use it. So back to the current problem, what if there's an alternative solution? I asked myself some questions Q: Do I need to see the whole command output? A: Actually, no, I want to inspect it only if something goes bad, and only the part that I need to see to debug the error. Otherwise, the only thing I want to see is the result. The great example here is Magit. Q: Do I need a shell for running commands at all? A: Well, in theory, one may want to do some preparations, like setting environment variables, copying some files here and there and so on. But, usually these preparations are more or less look similar, so why don't just put the stuff in a script? After all, what could be worse than typing routine things again and again?

Superb work, I'm definitely interested to see that.

I tried to solve the same issue (at least I think it's the same issue), but differently. I added this code into `emacs_intr_read` ([commit](https://github.com/geza-herman/emacs/commit/784a9fd3d511b7f6794f713a8d0b1370ab1b2401) in my fork): if (result > 0) { for (;;) { ssize_t r = read(fd, buf + result, nbyte - result); if (r <= 0) break; if (interruptible) maybe_quit (); result += r; } } Together with setting: (setq process-adaptive-read-buffering nil) (setq read-process-output-max (* 1024 1024)) gives a \~60x speedup for `shell` on my linux machine. I've been using this for a long time without any problems.

these kind of questions should be asked more often! changing the approach to a solution might just save you from solving the problem or at least exchange it for a simpler, easier to solve problem. I'm also used to just firing off commands, which spew out a lot of text, because 1. I want feedback while the command is running. 2. I have more ideas what to do with the output if I can see it, as opposed to thinking about it upfront. 3. I might have noticed something and want to scroll back and read its context, while the command is still running. after finished reading the interesting part, I want to jump to the end of the output received so far and continue following it. imagine running an apt dist-upgrade or nix-env -u. I often browse the version numbers while such commands are running, just so I have a rough idea how much each of them has changed since my last update. it helps me to decide which changelogs to read. that being said, I think what would be even better if long output would behave like a toilet roll "beginning" being yanked and dropped on the floor. u see the latest unfolding part, u see a bit of the beginning , but most of the in-between part is compacted. kinda like the last example here, at the 1min mark: https://youtu.be/85WjObNXT7I

line numbers can be useful while pair/mob programming, because they can be used as "audio pointers" by the non-driver person. vi users love to use line numbers, especially the relative ones. maybe you move your hands less, but move your eyes and mind more. your operation is more precise, so you don't have to do corrective motions. but the mental overhead and the lost screen estate, doesn't make it very compelling to me either...

> line numbers can be useful while pair/mob programming, because they can be used as "audio pointers" by the non-driver person. actually, these are not so accurate, you have to name an entity anyway > vi users love to use line numbers, especially the relative ones. maybe you move your hands less, but move your eyes and mind more. your operation is more precise, so you don't have to do corrective motions. but the mental overhead and the lost screen estate, doesn't make it very compelling to me either... there's avy for that

oh, I haven't said that I was referring to remote pair programming. having line numbers always visible, is like having the non-controlling person of a pair constant access to `consult-line` and avy, over audio, without taking keyboard/mouse control over. if you just mention a symbol or keyword on a 2K screen with 3 windows, plus a line number and maybe a buffer name, then that's usually enough for the navigator to quickly look at what you are talking about. we use pop.com for 2 years now. it has a pretty good drawing facility, but we often found that when someone underlines or circles something on the screen, the navigator might have already scrolled away. but line numbers provide a coordinate system, which scrolls with the text... either way, I agree with you, that most of the time it's useless, but remote pair-programming is not an extremely niche situation nowadays, where line numbers might be useful.

That makes sense. The main reason for needing the background thread on osx is because the pipe size is 1024 bytes, so even doing that loop wouldn't result in that large of a buffer. I would not be surprised if there's not much of a benefit from this change on OSs other than MacOS.

The commit that I had made prior broke url-retrieve-synchronously because it changed how input masks behaved when the keyboard wasn't in the fd set, so that's why it couldn't connect to elpa. I made a new commit with a few other bug fixes and also rebased off of master here, that I haven't observed any bugs with yet: https://github.com/tyler-dodge/emacs/commit/b386047f311af495963ad6a25ddda128acc1d461

This one is cherry picked directly off of master, so should be much easier to apply. It also has a few other bug fixes in it as well for handling url-retrieve-synchronously correctly, and handling stop and continue process correctly: https://github.com/tyler-dodge/emacs/commit/b386047f311af495963ad6a25ddda128acc1d461