make-process separating stdout/stderr for api but not for display

Longtime emacs user (20+ years) and developer in other languages, but just now dabbling in elisp beyond my .emacs. I want to run an async process and have both stdout and stderr display to the same buffer (as if it was run in a terminal), but I want to just store stdout to a variable/pass to a lambda for use in code. I don't see an obvious way to do this from the docs. The two options I see are: * I call make-process without the :stderr option, and both stdout/stderr end up in the same buffer, but a process filter can't discern between them. * I call make-process with :stderr, and I have to provide a separate buffer for stderr What I want is the combined output to show in a buffer for informational purposes, but code parses and uses stdout. This is because the commands I'm running are long running and display progress in stderr. Am I missing some obvious way to do this?

In the absence of anything ready-made, I suggest writing a custom output filter for the stderr buffer process, to copy that text to the stdout buffer. Remember that processes have only a single output filter, so your custom filter should incorporate any existing filter that it is replacing, if necessary.

Yep, this exactly.

Ok, so the solution is just to make a throwaway buffer for stderr. Just seems like a waste to create the extra buffer, but not a huge one in the long run.

Not particularly different to shell output redirections going to separate file handles, really. It's fine -- buffers are cheap.

If you really want to reduce waste, you could create a meta buffer that basically keeps track of two buffers internally and pass that to output stream. I’d know how to do in most languages easily but not elisp unfortunately. If you’re familiar with OOP, picture an iterator that iterates two internal iterators. Availability of item would be just an AND of two availability predicates and you just have to prioritize one over the other in some convention to totally order the outputs between two buffers. It would be a hugely wasted effort as hardly any scenario involving humongous (truly humongous to cause memory pressure) stderr buffers has other issues, for instance, making actual use of terabytes worth of logging outputs…

> I have to provide a separate buffer for stderr I haven't worked with separating stderr - is that with `make-pipe-process`? From the doc it does not allow a `nil` for `buffer` as with `make-process`, which I assume is your issue. It's easy with `make-process` to avoid having a buffer and just do whatever you want with the filter function, so it's odd that `make-pipe-process` isn't similar. Creating and destroying a stderr buffer doesn't seem like a big issue for modest amounts of data. I need to buffer my input, and only got slightly better performance using a buffer-less `make-process` and simply buffering data to a string instead of to a process buffer, but that's only evident with large data throughput. If you're not talking about masses of data I would be surprised if it became significant. And you can always delete the data as you process it.

My mental block was less about the amount of data/cpu overhead and more the namespace pollution of useless buffers. But, when I look at my buffer list I see a lot of output buffers from commands I'll never look at, so I'll get over it.

If there's only one process, you can probably create the stderr process pointing at your main buffer and make the filter function always return nil, but store the data in your own data structure. Otherwise I think you have to add to kill buffer hook to kill the process and buffer when your main buffer dies..