How to make TRAMP faster?

I'm a heavy user of TRAMP and I'm always looking for ways to improve the speed of my setup. Specifically, the pain points with TRAMP shows up mostly in dired and magit, where the turnaround speed (time spent to open a folder in dired, or time spend in any magit command) is quite high. I suspect this is due to the underlying process that TRAMP uses to connect to my remote server. My remote connection is relatively fast (100Mbps+). Things that I have tried till now are: - For my remote server, set the following flags in `.ssh/config` such that TRAMP re-uses the connections ``` Host myfasthost ... Compression yes ControlPath /tmp/%r@%h:%p ControlMaster auto ControlPersist 90m ``` - I use `scp` protocol to connect to TRAMP in dired. For example, to open a remote folder /path/to/my/folder, I use the following: ``` /scp:myfasthost:/path/to/my/folder ``` Is there any other protocols I can use to ensure faster connections? Or any other tips/hacks to improve the TRAMP dired+magit experience? TIA!

RemindMe! 2 days

I will be messaging you in 2 days on [**2022-10-05 14:47:26 UTC**](http://www.wolframalpha.com/input/?i=2022-10-05%2014:47:26%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/emacs/comments/xul3qm/how_to_make_tramp_faster/iqw0nmt/?context=3) [**7 OTHERS CLICKED THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2Femacs%2Fcomments%2Fxul3qm%2Fhow_to_make_tramp_faster%2Fiqw0nmt%2F%5D%0A%0ARemindMe%21%202022-10-05%2014%3A47%3A26%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%20xul3qm) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)| |-|-|-|-|

HAve you treid TRAMP on a default/empty init? I find that most often it's having too many modes that does file operations all the time is the reason that tramp feels slow most of the time. I guess you could turn on the profiler/debugger and check how much time is spend in the various `file-` functions in your config.

[deleted]

Good point, I haven't tried that! I'll give it a go and see if that improves the speed. In case of profiling, I just ran the Doom profiler, here is the memory profile: https://pastebin.com/WaDuqnvd . It seems the majority of the % is in the call "tramp-accept-process-output" under "tramp-wait-for-regexp". Any thoughts?

Is this related to the OP's question?

AFAIK helm is not slower than any other completion framework.

I mean...before you spend *too* much time trying to optimize the ssh connection, you should make sure that's really the problem. Start with an `emacs -q` and try a remote dired, and see if it's still as slow. If so, then yes, optimizing the connection is probably your best hope, but I'm guessing it won't be. And if it's not, then you need to start bisecting your config to see if you can pin down what *is* causing the slowdown. A lot of packages (e.g. completion frameworks) like to talk to the fs in the background, which can have a huge impact on TRAMP performance. Also, have you tried `/ssh:host:/path` instead of scp? I know that scp is generally deprecated! The actual [OpenSSH release notes say](https://www.openssh.com/txt/release-8.0): "The scp protocol is outdated, inflexible and not readily fixed." I don't know if that would explain the performance issues, but it seems like something you might want to avoid in any case.

[deleted]

[deleted]

Ahh, that makes sense. Thanks!

Which version of Tramp are you using (`C-h v tramp-version`)? Have you tried installing the latest version (currently 2.5.3.3) from GNU ELPA?

There's also [sftp](https://man.openbsd.org/sftp).

Might be worth a try, but I would expect plain ssh to be the best option, since Emacs needs to run remote processes as well as transfer files!

I enjoy a fairly rich set of packages, and some things really like to hunt around the filesystem. I had some trouble preventing searching the filesystem for things like `.git`. My initial debug technique was first to turn up logging, e.g. `(setq tramp-verbose 6)`, and look in the buffer `*debug tramp.../<host>*` to see what it was up to. But even after some recommended settings to reduce remote activity, it wasn't obvious how to stop some of undesired activity. So I broke in with the debugger when it was looking at files I didn't want it to: (defun intercept-tramp-send-command (vec command &rest args) "break into the debugger if it looks for project files" (when (string-match "\\(.projectile\\|/git\\)" command) (debug))) (advice-add 'tramp-send-command :before 'intercept-tramp-send-command) And then I could jump down the stack trace and work out what was going wrong. I can't remember the details, but I see from my commits I set: '(doom-modeline-project-detection nil) '(projectile-auto-update-cache nil) '(projectile-dynamic-mode-line nil) And made my own `lsp` starting function that didn't work if `(file-remote-p default-directory)`. (I've since abandoned fancy modelines. Perhaps some day I'll get off `projectile`)

(I have some significant ssh latency to deal with, sadly it's still irritatingly slow, but faster than it was)

I don't use Tramp myself, but when I searched after seeing your post, I found a stack overflow post that talked about tramp and regex where tramp kept going. They mentioned 2 solutions, on to modify tramp to not continue if the file list would be over 1000 files, and the other to be aggressive in .gitignore.

I prefer to use emacs terminal with tmux on the remote server for this. Have you give it a shot? Don't know why most here are so adverse to running emacs in the terminal.

So you have to compare with a tool doing the same thing. e.g. comparing ls and your graphical file manager to list a huge directory is not fair. Also saying completion is vague, helm covers hundred of "completions" of various things. And for generic completions like completion-at-point or completing-read, helm uses by default a system no other completion tools use, making it much faster.

The only thing missing for me in the terminal is the support for seeing images. Do you have a solution for this?

[deleted]

Depends on what you are doing I guess but for remote images I run a web server on the remote host to see the images in the local web browser.

Currently I'm doing that as a backup. It works well, however I have the following gripes: - Insert mode doesn't change the cursor. I tried to debug this for a bit but couldn't get it working (using Alacritty and Mosh) - I don't know how to map my usual "Cmd+" keymaps over Alacritty. I re-learned the alternative keymaps for remote. - The above two are mostly nitpicky. My main use case for working over TRAMP instead of remote is the ability to view images. I don't see a way to do that over remote.

I checked mine is `2.5.2.28.1`. I'll try to update it and see if it helps, thanks!

Thanks for the tip! I set the tramp verbose to 6 and inspected the logs. It seems like during my magit-status call, there are several "tramp-send-string (1) # Remote file error: Forbidden reentrant call of Tramp" errors. From the TRAMP docs, this error is shown when "Timers, process filters and sentinels, and other event based functions can run at any time, when a remote file operation is still running.". This causes TRAMP to block - for sure I see a noticeable difference of 1 second blocks each time it happened. Any tips on how to disable such timers?

Also, I see many "tramp-send-command (6) # test -e /my/path 2>/dev/null; echo tramp_exit_status $?" and "test -d /my/path 2>/dev/null; echo tramp_exit_status $?" commands in the log, which are also adding 1-2 seconds delay each time. Any ideas what these could be?

I think a whole lot of the `test` calls are just what Tramp does. It'd be nice if it could batch them up to avoid the round-trips. I know there's some refactoring work gone in recently, but I don't know if that's being addressed. I browsed the code a while back in the hope I might be able to work out how to contribute, but I quickly got disheartened. My elisp is not strong. Sorry I have nothing to offer on the timer stuff.

No worries, thanks for the tip! Something to start investigations from!

Thanks for the tip! I tried a minimal installation using `emacs -Q`, installed magit and tested stuff. Dired navigation to my remote is much faster, however `magit-status` is still slow (takes around 20 seconds to return the result, which is same for my own setup). So I suspect the cause is probably due to TRAMP (tested on version 2.5.3)

In a terminal, the cursor is controlled *by* the terminal. Emacs could only change it if the terminal supported some escape code for doing that. FWIW a quick search turned up https://unix.stackexchange.com/questions/49485/escape-code-to-change-cursor-shape

I investigated the escape code issue in detail before. The problem is I use mosh, which kind of filters out these escape codes. There is an open issue for this (https://github.com/mobile-shell/mosh/issues/352).

See also https://debbugs.gnu.org/cgi/bugreport.cgi?bug=56342 which is why I'd suggested you should try updating Tramp (although I don't know whether or not related improvements have landed in a release yet).

Very interesting and relevant bug report! Seems we may have to wait till Emacs 29 to get the proposed changes?

For absolute clarity: The GNU ELPA Tramp package often gets updated in between Emacs releases. Therefore you do *not* need to wait for a new version of Emacs to get a new version of Tramp. You might only need to wait for an Emacs release if the practical improvements depended upon other Emacs changes *outside* of Tramp itself. (I don't know whether any of the improvements made thus far for that particular bug report have landed in a Tramp release, but it would still be worthwhile trying the latest version.)

Did you try this? Did it help?

Yes, it did not make a difference unfortunately!

Ah well. My best suggestion then is to keep an eye on (or get involved with) the upstream bug report I mentioned -- it did sound like there would be at least some improvement resulting from that, so I think one of the upcoming Tramp releases should improve things at least a little. I think Tramp has to work quite hard to guarantee reliability in all of the many supported environments, though, so undoubtedly there are limits to what can be done (i.e. hope for the best, but I would keep expectations in check).

Totally understand! I would love to get involved, I'll need to figure out this bug report system.

You could try "direct-async-process": (add-to-list 'tramp-connection-properties (list "/scp:" "direct-async-process" t)) Or: (add-to-list 'tramp-connection-properties (list (regexp-quote "/scp:user@host:") "direct-async-process" t))

Tried it, magit didn't improve that much over TRAMP - I get the same lag, magit-status on my remote repo returns results in ~20 seconds.