When Exactly can Async Process Output Actually Arrive?

Async programming in Emacs is... challenging. If the modern asyncio/await/async paradigm represents "cooperative" multi-tasking, Emacs' async seems more "semi-cooperative". I'm working with async process output in a comint mode, and I (re-)read the following [docs](https://www.gnu.org/software/emacs/manual/html_node/elisp/Output-from-Processes.html) often (emphasis added): >Output from a subprocess can arrive only while Emacs is waiting: when reading terminal input (see the function waiting-for-user-input-p), in sit-for and sleep-for (see Waiting for Elapsed Time or Input), in accept-process-output (see Accepting Output from Processes), and in **functions which send data to processes** (see Sending Input to Processes). This minimizes the problem of timing errors that usually plague parallel programming. So it's pretty clear that sit-for and accept-process-output are good "yield points" (ala *await*). But what about in *functions which send data*? Does that mean: 1. Only functions which directly call process-send-\*? 2. Any function which calls such a function (and so on up the call stack)? And where exactly in such *functions which send data* can process-output interrupts happen? Anywhere? Only specific places? If you've been down this road and found any enlightenment, I'd be grateful to hear about your travels.

I think there could be a misunderstanding here of what exactly does "output from a process can arrive" mean, in the context of Emacs's communications with async subprocesses. What that means here is that the filter function of the process object can be called at these times. IOW, "output arrives" when it is handed to some Lisp function that your Lisp program can use to get at that output. If, after understanding the above meaning, you still think your questions are relevant, please rephrase them and add more details, because otherwise I think the answers are self-evident and almost trivial. And yes, "functions that send data" are the `process-send-*` functions documented in that section.

I’m also trying to understand eMacs async and threading model. Have rich experience both with the jvm and the JS model. Trying to debug a problem but the stack trace makes no sense to me. Would you recommend some blog or article that explains in more detail emacs concurrency and async?

I'm not aware of any such blog, but maybe someone else could help. If what's in the ELisp manual is not clear or you have questions, feel free to ask here or on help-gnu-emacs mailing list.

reading the timers manual page I'm left with the impression that emacs handles them in event loop, same way as javascript? async processes have two parts: \- the process itself, external to emacs \- the communication with it which is again done in event loop fashion (no multi-threading on emacs side) do I get it right?

Thanks for your response. I certainly understand that in Emacs, asynchronous output from a process means your process' filter function gets called. The question I have is *where precisely* emacs can pause execution and call your filter-function while you are busy doing other things. Let me add detail to the analogy with async/await. In such a system (in say Javascript or Python), the *only* places where another task/coroutine can "take the ball" and do some processing are at the specific "await ..." statements you have *explicitly* included in your code. At no other point can your program be interrupted, so you can 100% rely on continuous execution outside of these specific "opt-in" cooperative async locations. The sit-for and accept-process-output parts are clear I think; in my way of thinking, these function quite similarly to "await" in that they can "yield execution control" at specific locations in the code. The real question I have is where *else* can Emacs take over control of execution and pass fresh process output to your filter function? I put together [this gist](https://gist.github.com/jdtsmith/0c35675ec33be1402fab60fe6cbd4d0c) in an attempt to make this question hopefully clearer.

Yes and yes.

Emacs never "pauses execution" (if you forget about Lisp threads for a moment, and consider only the main Lisp thread). So there's no "yield". When Emacs finishes the command it is running, it returns to the main loop, where it can run timers, check for input from any of the possible sources (including from sub-processes), etc. If any of the possible sources of input has some input ready to be read, Emacs reads the first one, executes whatever is needed to be executed for that input, then goes back to the main loop. `sit-for`, `accept-process-output`, etc. call the function which checks for available input directly, without going through the main loop. There's a chapter called "Command Loop" in the ELisp manual with the above and many more details, which I suggest to read. I hope this answers your question about *where precisely* Emacs calls filter functions: it's when it processes output from any process whose file descriptor indicates that some stuff is ready to be read.

> Emacs never “pauses execution” Right. This is also true of all cooperative single-threaded multi-tasking systems I know of, so speaking of paused execution is merely a useful abstraction. > There’s a chapter called.. Thanks this was helpful. I read through the Command Loop docs which are very useful (though do not discuss subprocess output). Based on this I now think the doc phrase “functions which send data to processes” means simply and only the _literal_ process-send* functions, rather than any generic user function which themselves invoke these. It might be useful to use phrasing like “the following functions” to make this clearer. So, presuming the process-send-* functions also call the same “function to check for process output directly” as e.g. sit-for, relative to the gist I posted, A,E,F and G are the only places input can arrive.

> presuming the process-send-* functions also call the same “function to check for process output directly” Yes, they do, if they cannot send all the output at once, in one go, for some reason.