IPC

What's your favorite way of doing IPC between programs that you control but wrote in different programming languages? - Pipes, UNIX sockets, or local TCP? - HTTP/1, HTTP/2, ZeroMQ, or nanomsg? - Protocol Buffers, JSON, or msgpack?

UNIX sockets are by far the best way if you're on a UNIX system and can write a half-decent zero-copy decoder. Shared memory is too much of a headache. I don't even know for windows. Custom transport types in the case where extremely low latency encoding/decoding is necessary. Msgpack otherwise.

Why, it is CORBA of course! Servants, Portable Object Adapters, and Interoperable Object References for everyone!

>>33820 What about SysV STREAMS or Solaris Doors or Sun RPC?

>>33820 A bit wierd of a question. It depends on your tasks. Everything is just a tool, you should pick what's most suited for the job.

>>33820 Unix sockets with binary formats. I fuarrrking hate HTTP actual garbage.

>>33820 I've no favourite but, if I had to pick one, I'd choose UDP holding numerical messages. Sending sequences of characters which must be parsed beginning-to-end is wasteful. It's best to send a known data structure which can be examined with the least computation feasible. As for the issues which arise from this, such as needing to change the format at some point, a small amount of foresight works best. An ounce of prevention is worth a pound of cure and all of that.

>>33820 REST with JSON is actually nice. It doesn't have great performance but it helps you to build a clean interface and is easy to work with. Other approaches would make practical sense only under additional constraints.

>>33923 REST with JSON feels a bit like the Unix way, grafted onto HTTP. Plain, inspectable text.

I can't say that I've implement IPC more than let's see, once! Iirc the daemon I wrote had a named socket that my client would then connect to to send instructions over. Something I really like that is specific to glibc is that you can acquire credentials of whoever is connected to your socket with GNU source extensions. Couple that with setting appropriate permissions on the socket file itself and you get pretty good permission management for whatever you instruct your daemon to do.

>>33834 Would you write a custom binary format every time or use something like Protocol Buffers or FlatBuffers?

>>33823 How do you code the custom types?

>>34661 Yes, there is no escape, even when using HTTP, you still have to do it. If you are doing machine learning and have to send huge chunks of binary data (matrix, etc...), is much easier to do it this way than with HTTP, like, 10x less code.

>>36862 Isn't that a bit tedious and error-prone, though? I'd hope that you can use some specification language, binary parser library, or something like that to make it easier. Especially when communicating between different programming languages.

>>36862 >If you are doing machine learning and have to send huge chunks of binary data (matrix, etc...) What kinds of data would be involved? Wouldn't that mostly be multi-dimensional arrays, which are trivial to communicate?

>>33820 I did IPC using a unix socket and a custom text-based protocol turned out pretty alright I guess

UDP for short messages, TCP for data. All with custom binary formats. They're truly cross platform (even on windows) and network transparent, so sutable for most use cases. Pipes for the few cases requiring high performance.

>>33820 I know nothing about IPC or even Unix pipes in general. Is there a simple IPC programming tutorial that uses C or C++? Also, is there quick rundown on dbus? I would like to know more about it. I know that dbus should be avoided because it makes porting your program to other operating systems (like *BSD) harder.

>>33820 >IPC between programs that you control but wrote in different programming languages IPC doesn't have anything to do with different languages, it's system dependent. If you've meant that you can write wrapper, that's again not about IPC. The rest depends on the task. I think unix sockets are great, but I don't think it's always the best.

What the fuarrrk is inter-process communication?

>>33820 shared memory at a fixed base address that way even pointers can be shared (so long as they all point within the shared address space) mmap with MAP_SHARED | MAP_FIXED can achieve this on linux, windows has something with virtualalloc i think >>36961 ipc is different os processes reading and writing to the same memory to communicate with eachother, or sending operating system messages to eachother threads as in multi threading is a form of inter-process communication, as each application thread is technically also a separate process

For an IPC mechanism, I prefer any that supports rendezvous: the idea that a receiver waits until the sender is ready, or the sender waits until the receiver is ready, and both go on their merry way only once the message has either been delivered or delivery has failed. This means both sides know whether the operation succeeded or failed, and different kinds of error can be reported such as timeouts, partner disappearance, and suchlike. More advanced things such as the L4 microkernel family have an additional mode where an IPC syscall can consist of a send phase atomically followed by a receive phase, so that a client can encode an interprocess procedure call in a single operation consisting of two atomic halves, and a server can run in a "reply, then wait for next message" loop. This is highly efficient between processes already because the microkernel can do exactly two full context switches during the entire call, and even nicer between two threads sharing memory because everything the client owns is accessible to the server until reply. Ada encodes all of these semantics in task entries, which is part of its staggering elegance. However this doesn't make a full IPC system. For that you'd need an IDL compiler to take the drudgework out of adapting your language typesystem to the IPC mechanism primitives, and ways to identify peers in a manner suitable to your application. Back in the nineties and early aughts CORBA got much of this right albeit in a bureaucratic and verbose manner, and everything since has been reinventing half of it poorly.

Any good concise guides on UNIX domain sockets? I'm writing a daemon and need a way for a controller to poll it and send commands to it. For threads within I resolved to using pipes, but with the daemon it seems that sockets are the only option.

>>33820 Message passing, textual stateless protocol

Shared mapped memory. Prob the fastest you're gonna get depending on the kind of data structures you need. Basically what >>36962 said, you can do it on linux and windows pretty trivially. On windows, with MSVC you can do this right within a .exe or .dll to have all instances of it share memory. #pragma data_seg("SHARED") // Data #pragma data_seg() #pragma comment(linker, "/section:SHARED,RWS") You can use atomic instructions to implement safe data-structures within this region like a ring-buffer.

Reminder that Windows now supports UNIX domain sockets.

>>37206 >Beginning in Insider Build 17063, you’ll be able to use the unix socket (AF_UNIX) address family on Windows to communicate between Win32 processes. sarah_cooley, (December 19th, 2017). "AF_UNIX comes to Windows". Microsoft Developer Blogs Windows Command https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/

>>36957 >IPC doesn't have anything to do with different languages, it's system dependent. Only if we're talking about the low-level mechanism alone. But I'm also curious about what kind of protocols and programming patterns people use.

>>39782 I personally write up an os-dependant pipes wrapper (or use what's there, if there is something) and blast binary or base-64'd data back and forth over it. Alternatively, if we're talking SQLitre-appropriate situations, I just shart out a SQLite db. For speed, yeah it's pretty much mapped memory or nothing.

>>37204 >have all instances of it share memory Is instances of same exe the limit? I was under the impression security killed sharing memory between different programs so you cant have useful things like "paint program accesses and draws into texture used by a 3d rendering software i.e. blender".

>>40202 On Windows some security software may indeed deny requests (made with documented APIs) to open handles to third-party processes, which is required to do basically anything of note, however, that only applies to protected processes, like as the user-mode component of the EDR/AV, LSASS, SMSS etc. For unprotected programs you can open a handle and simply read or write memory via `ReadProcessMemory`. Alternatively, if you control both programs, it is possible to create a named file mapping that can be accessed and mapped by both processes. Finally, if you have kernel access, which I suppose is more of a /sec/ question, you can mess around with either process' page tables to make certain VAs point to the same physical page.