(untitled)

Why do people at work don't the unix philosophy for write their programs? It makes everything so simple that is ridiculous. For example, let's make an "echo server" with the possibility to connect to multiple clients. We could use sockets and multiple threads to deal with it, but i prefer to use the unix philosophy and write my program thinking to be used with "socat", a program that redirects socket connection to another file descriptor/socket, in my case, the file descriptors will be the stdin and stdout of my program, so my server program will be literally this: C #include <stdio.h> #include <stdlib.h> int main(int argc, char *argv[]) { char buffer[256]; while(fgets(buffer, 1, 256, stdin) != 0) { printf("echo: %s\n", buffer); fflush(stdout); } } And the command to execute this server will be: Bash Script socat TCP-LISTEN:<port>,fork EXEC:<program_location> For every connection, the socat will fork and execute an instance of my program with the standard file descriptors redirecting to socket, so every write to stdout, with fflush to make sure that will be written, will be send to the connected socket and every data the client sends can be read by reading to stdin. With more complex stuff, we could use what Unix provide to us: Message Queues, Semaphores and Shared Memory, so a program doesn't need to "exist" to save some stuff in RAM or have coordination of write process (by using SysV Semaphores). Unix by itself is a excellent tool, and have lot's of minimal programs that do the dirty job for us, but why do we need to reinvent it by using programs like Kafka and Docker (we could implement CPU servers from Plan9, instead of multiple VMs) or even make an monolithic program in some idiotic strange language, like Java or Python? C itself have lot's of really useful functions (https://en.wikipedia.org/wiki/C_standard_library), and is a very clean high level language (yes, it's a high level language, words of Ken Thompson). Why do we need to suffer at work when there are better and clean executable tools for us to use?

>>17028 Im pretty sure verisimilitude guy's computer is just a bunch of lisp programs piping data between each other like a bunch of self-configuring human centipedes.

>Why do people at work don't the unix philosophy for write their programs? Because they often develop proprietary software, which is the opposite of UNIX philosophy. When you develop a proprietary program, you try to push every possible functionality inside, so the program has long list of features, which can be sometimes major argument when a client decides which program to buy. Second problem is obscurity and software maximalism - when you develop a proprietary program, you want to have it so big, no one will waste their time on trying to learn another one. Example: Microsoft Excel - a spreadsheet, a programming language (Visual Basic), a mailing client, a database and I'm afraid to check for more. By making your program bloated, you make it harder to replace. That's why software is often developed against the UNIX philosophy.

>>17028 But this isn't unique to Unix. Within C, you could just made a 'socat' function that takes a port and a function pointer. Yes, Unix lets you glue arbitrary programs together, but then you're either slinging strings or serializing and deserializing the same data over and over again. Piping lots of programs together only works with plenty of massaging from 'awk' and 'cut'. Depending on the format, 'awk' and 'cut' may not be enough.

>>17036 >awk may not be enough awk is turing complete. it is enough.

>>17036 It's not necessarily text, you can pass binary, although you will be violating the "use text streams" rule. The thing about texts is because you know how to process them, and as a bonus you avoid the endianness problem across computers. UTF-8 and ASCII have an specification of endianness, i think. Nowadays, you can't avoid piping binary anyway.

socat is nifty and the UNIX philosophy is great for low-volume tasks, but the UNIX design philosophy is just not very useful when you have to write very high-performance code(Like a web server with nontrivial state). Spawning a process for each user will eventually use up a lot of memory and waste a lot of CPU time under heavy load, whereas using threads or green threads would lower the memory footprint significantly.

>>17028 >Why do people at work don't the unix philosophy for write their programs? Perhaps they have good taste. >It makes everything so simple that is ridiculous. The UNIX Philosophy is what I call brand-name simplicity. There are just a few aspects of it that do result in better software, but they're expressed in a vague way and actually predate computing; the remaining aspects actually result in more complicated software. The separation of functionality on subprogram and library boundaries was known before UNIX; separating functionality on the boundary of an entire program is wasteful and stupid. Text isn't a universal interface and there are many different encodings in common use. >For every connection, the socat will fork and execute an instance of my program with the standard file descriptors redirecting to socket, so every write to stdout, with fflush to make sure that will be written, will be send to the connected socket and every data the client sends can be read by reading to stdin. This is a profound waste of resources. You can use MAKE-ECHO-STREAM in one line of Common Lisp to get this; you can argue it's not fair since the language already provides it and I can argue that the more a language gives the more I can focus on my actual problem. >With more complex stuff, we could use what Unix provide to us: Message Queues, Semaphores and Shared Memory, so a program doesn't need to "exist" to save some stuff in RAM or have coordination of write process (by using SysV Semaphores). UNIX doesn't even provide file locking. Are you calling pipes message queues? A proper message queue can store real messages, not only octets. >Unix by itself is a excellent tool, and have lot's of minimal programs that do the dirty job for us, but why do we need to reinvent it by using programs like Kafka and Docker (we could implement CPU servers from Plan9, instead of multiple VMs) or even make an monolithic program in some idiotic strange language, like Java or Python? That's all crap too. Java and Python are garbage in part because they imitate C. >C itself have lot's of really useful functions (https://en.wikipedia.org/wiki/C_standard_library), and is a very clean high level language (yes, it's a high level language, words of Ken Thompson). The C standard library is anemic. Does this look nice and simple to you: C int pigeons_device(int a, int b, int mode) { int result; /* Isn't C a wonderful language? */ switch (mode) { case 0: if (gloop(a, b)) { case 1: result = arfle(a, b); break; } else { case 2: result = barfle(a, b); break; } } return (result); } >>17032 >Because they often develop proprietary software, which is the opposite of UNIX philosophy. That's asinine. UNIX began as proprietary software and the C language model makes it the default to distribute programs without source code, unlike other languages. >When you develop a proprietary program, you try to push every possible functionality inside, so the program has long list of features, which can be sometimes major argument when a client decides which program to buy. You're not using GNU utilities, with all of their nice extra option flags, are you? I've seen the Suckless people argue about how color is bloat, because they realized columnating output properly with a separate cols program required it to understand the control sequences which produced color; this is easy if understanding color and everything else is a library, and difficult if everything is a completely separate program which must understand everything on its own. >By making your program bloated, you make it harder to replace. That's why software is often developed against the UNIX philosophy. UNIX is certainly hard to replace, due purely to momentum.

>>17038 Sure, but passing binary only works if the receiving program can handle the format. If the receiving program is doing some generic operation, like encrypting a binary stream or forwarding it, then there's no problem. But if the receiving program assumes any structure, the sender has to provide it. Unless the format is generic, the receiver and sender are coupled. It feels like it would be easier to just write C functions and then have a Execute Library Call program that takes a library and function name, loads the library, gets the function symbol, parses extra arguments based on the function signature, and then passes the extra arguments to the dynamically loaded function. execute-loaded-call math cos 3.14

>>17040 >>17039 >>17032 >>17028 You guys have clearly never worked in a large software company before. People do use the unix philosophy extensively. Heard of "microservices"? That's when you write little programs that do one thing and do it well and tie them all together through HTTP or a messaging service of some sort. This architecture is used extensively in massive companies (google, amazon, facebook, twitter, uber, etc.). >brand-name simplicity you don't have the first idea of what you're talking about. it expresses ideas in a concise and precise manner. these ideas work and scale amazingly. >This is a profound waste of resources you don't have the first idea of what you're talking about. processes are cheap as fuarrrk. ever heard of shared memory? the only thing non-shared in a modern operating system is the private data for a given instance. >UNIX doesn't even provide file locking. Are you calling pipes message queues? A proper message queue can store real messages, not only octets. that's because it is completely unnecessary. modern unix provides much more than pure octet streams and file-level primitives. >That's all crap too. Java and Python are garbage in part because they imitate C. Clearly you've never had to deal with substantial code bases. These languages are much more accessible and useful for large projects.

>>17043 >Heard of "microservices"? That's when you write little programs that do one thing and do it well and tie them all together through HTTP or a messaging service of some sort. This architecture is used extensively in massive companies (google, amazon, facebook, twitter, uber, etc.). Yes, I've heard of micro-services. When these big companies use micro-services, they use HTTP, because raw sockets are awful. They use XML and JSON because creating a new poorly specified syntax is a bad idea. Finally, they carefully document the structure of valid messages. Micro-services do not accept arbitrary text over a socket. The micro-services you describe are nothing like Unix pipes.

>>17044 >use HTTP, because raw sockets are awful you're clearly not even a programmer. What do you think HTTP uses under the hood? Pixie dust? >They use XML and JSON These have literally nothing to do with HTTP. You could use raw sockets (which would be more efficient, but given you're not a programmer I'll likely end up having to explain why about this to you too) to accomplish this more efficiently. >Micro-services do not accept arbitrary text over a socket. Wrong. Microservices (no hyphen, genius) accept whatever you program them to accept. This can and does include arbitrary text. >The micro-services you describe are nothing like Unix pipes. They literally are, except they're networked instead of unix-y. You're pretty ignorant to be posting in this thread.(Always argue in good faith and avoid personal attacks.)

>>17046 >to accomplish this more efficiently *as the transport for these same datastructures.

>>17046 Obviously, everything is bytes deep down. It's also obvious there is a difference between a service that performs several layers of message validation based on well specified formats and a raw socket that passes bytes directly to application logic. >You could use raw sockets I *really* hope you don't read all available socket data into a buffer and then try to parse the buffer. That fails pathetically if a message is split into several packets. Protocols like HTTP exist to avoid these issues.

>>17048 >That fails pathetically if a message is split into several packets. Protocols like HTTP exist to avoid these issues. Not a programmer confirmed. I'm done talking to you. I'll be happy to answer any questions anybody else has about the unix philosophy or other fun programming topics.

>>17049 >Not a programmer confirmed. I'm done talking to you. Did I step on a nerve? Did you really write all your software like this? while True: buffer = socket.read() do_stuff(parse(buffer)) How sad!

>>17050 lol, no. But you obviously don't know how sockets work. Also, weren't you rallying against python earlier?

>>17051 Tell me oh great one, how *do* sockets work?

>>17052 >great one flattery will get you nowhere. Sockets can be either streams or datagrams. Generally, datagrams imply UDP which is unreliable. This means if you lose a packet, it will not be recovered without application-layer intervention. This can significantly complicate read processes. Streams are reliable, and fail "loudly". As such, you can expect the data to arrive in the correct order together or fail entirely. You need to read from a socket continuously until you have enough data to either (1) completely parse or (2) know how much more data to read. In the case of a protocol like HTTP, part of the protocol specification indicates you have a pair of newlines to indicate the end of a section. In the case of a body-carrying request/response, part of the first section will include a length so you know how many bytes to read. This can be implemented in many different manners. There's a brief overview of the absolute basics. However, I'm not really interested in proving myself to somebody who can barely articulate the simplest of notions, so that's where the buck stops.

Since >>17049 was too angry to explain himself. I have explained why protocols like HTTP exist. TCP is a stream based protocol. There is no guarantee that the data read in by a socket will include an entire application message. In fact, there's no guarantee a complete message will ever come. If the sender cuts off, then you're stuck with what you have. The sender could cut off because a cable was axed or because they're a terrible client. The only way you can guarantee you'll never have split messages is if your messages are all a single byte long. Usually, reading a socket provides more than one byte, so if your message is small enough you'll usually be able to parse it, but this is not robust. Protocols like HTTP allow servers and clients to reliably get whole messages and they're fairly easy to write a state machine for.

>>17054 >Protocols like HTTP allow servers and clients to reliably get whole messages and they're fairly easy to write a state machine for. Only partially correct. These protocols only exist as a tool to have an abstraction. They do not "allow servers and clients to get whole messages", that is completely incorrect. That's the job of TCP. HTTP gives you an abstraction of "a bunch of text with headers, a noun and a path" versus "a bunch of bytes" that a raw socket delivers. These are very different things.

>>17054 Also, not angry. You should avoid presuming people's mental state given you appear to be retarded when you say such things.

>>17053 >Streams are reliable, and fail "loudly". Only handling loud failures is not robust. >>17056 No matter what I say, you just say "Oh, but there's really no distinction. That's just an abstraction. They're the same underneath!", dismiss me, and announce that you're done talking to me. You did it for microservices and now you're doing it for sockets. If I wanted to be stuck arguing about the difference between being equivalent in theory and being the equivalent in practice I would have started arguing with >>17037 about turing tar pits.

>>17061 Sounds more like you want to argue semantics to me.

>>17061 >Only handling loud failures is not robust. All failures are loud. This is robust. >No matter what I say Maybe you should leave the topic, you clearly don't belong. >If I wanted to be stuck arguing about the difference between being equivalent in theory and being the equivalent in practice I would have started arguing with >>17037 about turing tar pits. Your argument would make sense if you were talking about `sed`.

>>17028 That's a neat case of abstracting code. It reminds me of the y-combinator where recursion is defined separately from the function being recursed. This is taking the unix philosophy a bit further than I think is intended. The gist >do one thing and do it well shouldn't be thought to mean literally "one thing and only one thing ever". That would be absurd. You would have such fragmentation it would be absurd. Look at how unix actually works: >sed >grep >cat >man These commands each do a few things, but they all have a unifying theme. This is really interesting when you look at the HURD kernel, it's broken down into several parts, but not to an absurd level.

>>17066 >Your argument would make sense if you were talking about `sed`. The reason people hate Awk or Unix is not because they incorrectly think certain things are impossible with them. People hate Awk and Unix because using them is painful. Awk and Unix are a pain to use. Massaging, serializing, and deserializing data is awful. That's why nearly every programming language has types, return values, and function signatures. Even Awk has types, return values, and function signatures.

>>17066 >All failures are loud. This is robust. Do you never reuse a connection?

>>17028 >socat TCP-LISTEN:<port>,fork EXEC:<program_location> >>17043 >processes are cheap as fuarrrk Haha! Good joke, old chap(s). Even spawning new threads is considered too expensive by anyone who gives a serious fuarrrk about scalability. A server is usually better implemented using some sort of scheduler to run coroutines (stackful or (better yet) stackless) on already-running threads from a thread pool, all inside a big, evil, MONOLITHIC program.

>>17072 >people hate unix that's why it's the most pervasive server and cell phone os in the world. >is awful it really isn't lol. I use unix for literally everything. I code on it, use it, admin it, etc. It's super simple, and thus why I think it's also super elegant. >every programming language has types, return values, and function signatures. Even Awk has types, return values, and function signatures. it's funny, one moment you're saying these tools don't have these facilities, then you say Awk has types, return values and function signatures. Seems, what's the phrase? Self-contradictory? The fact is, yes (!!), these are useful things that make data transformation easier where they're appropriate (particularly non-text). The thing is most people don't use unix command line tools for things other than text, and most people would laugh at you for using it on anything but. >>17074 patently untrue, but you keep telling yourself that :) >>17073 >>All failures are loud. This is robust. >Do you never reuse a connection? These two ideas are not at all connected in the way you're implying.

>>17074 Yikes, are you really that stupid? Coroutines are used within a tiny, evil, MICROSERVICE to service individual connections. Processes are cheap as fuarrrk, regardless of forking costs which are rarely paid.

>>17074 >A server is usually better implemented using some sort of scheduler to run coroutines (stackful or (better yet) stackless) on already-running threads from a thread pool, all inside a big, evil, MONOLITHIC program. Are you talking about Nginx?

>>17077 They're talking out of their ass :)

>>17075 >that's why it's the most pervasive server and cell phone os in the world. Yup, it's up there with Windows, PHP, and Visual Studio Code. Pervasive! >it's funny, one moment you're saying these tools don't have these facilities, then you say Awk has types, return values and function signatures. Seems, what's the phrase? Self-contradictory? Are you really this dumb? The point is that none of these things are available for Unix programs to communicate with each other. Awk having return values, types, and function signatures doesn't change anything. There are no return values, types, and function signatures between Unix programs. (Well, there is the return value of the main function, but that can't contain anything but a small integer, so only an idiot would bring it up as a counterexample. Which is why I'm mentioning it.) You misunderstanding this reflects very poorly on you. This was already explained. >The thing is most people don't use unix command line tools for things other than text, and most people would laugh at you for using it on anything but. Except this entire thread is supposed to be about how great the Unix philosophy is and starts with an example of building a server with `socat`. Going from "wow, Unix philosophy sure is great, watch me chain these programs together" down to "nobody uses command line tools for things other than text" is a solid change in goal posts. More importantly, many people *do* use command line tools for things other than text. `tar`, `rsync`, and `ssh` are all examples of command line tools that are useful with more than text. You've moved the goal posts so much that now I have to defend Unix just to argue how micro-services are not Unix philosophy. So once again, micro-services are not an example of Unix philosophy.

>>17075 >These two ideas are not at all connected in the way you're implying. But you still didn't answer the question.

>>17079 >Are you really this dumb? Ah yes, show us some authority by getting mad and stomping your feet and attacking the other posters. That really shows how intelligent you are! >You misunderstanding this reflects very poorly on you. This was already explained. You clearly didn't understand what the anon said. Read it a few more times, maybe it'll become clear. >Except this entire thread is supposed to be about how great the Unix philosophy the unix philosophy has literally nothing to do with shell commands. They're simply an implementation of those ideas. You need to go back and actually read the philosophy again. >is a solid change in goal posts. Again, this is your lack of understanding of the unix philosophy. The anon's point is valid and true. >More importantly, many people *do* use command line tools for things other than text. `tar`, `rsync`, and `ssh` are all examples of command line tools that are useful with more than text. This is utter gibberish when compared to what is actually being discussed. >So once again, micro-services are not an example of Unix philosophy. Again, this is your lack of understanding of the unix philosophy. Go. Back. And. Read. It. Again. tl;dr - you don't actually understand the unix philosophy.

>>17075 >patently untrue, but you keep telling yourself that :) no u :))) >>17076 >Coroutines are used within a tiny, evil, MICROSERVICE lol'd. That "micro"service itself is as far removed from UNIX philosophy as it gets. >Processes are cheap as fuarrrk, regardless of forking costs which are rarely paid. As soon as the child process tries to write to its address space, a copy of the appropriate page has to be made, not to mention all the metadata, file descriptors, etc. If you think this is as good as free, I recommend you stay away from high-performance computing. >>17077 Not quite, although it goes a long way by splitting work among existing workers instead of spawning a new "cheap" process (lol) for every fuarrrking connection.

>>17082 >That "micro"service itself is as far removed from UNIX philosophy as it gets. Imagine being this dumb. Thinking coroutines are so heavy as to make a service not micro. lol, wow you people are so fuarrrking dumb. >>17079 >So once again, micro-services are not an example of Unix philosophy. You've clearly never read the unix philosophy: >Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features". Microservices do one thing, and do it well. They are proliferated in number rather than "enhancing" them with additional functionality. >Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input. Microservices are tied together via network or localhost connectivity, or with a message queue of some sort (networked, but with additional perks). Pretty sad you guys are in a conversation about a topic you clearly have zero understanding of.

>>17082 >>17079 Wow, are you guys for real? >https://en.wikipedia.org/wiki/Unix_philosophy Did you ever even read it?

Since it hasn't been posted yet. I'll throw up the Unix Hater's Guide. Note everything in here is relevant, but before you dismiss a complaint, remember that there was an alternative without the problem. https://web.mit.edu/~simsong/www/ugh.pdf >>17081 >Ah yes, show us some authority by getting mad and stomping your feet and attacking the other posters. That really shows how intelligent you are! >tl;dr - you don't actually understand the unix philosophy. >This is utter gibberish when compared to what is actually being discussed. >Again, this is your lack of understanding of the unix philosophy. Go. Back. And. Read. It. Again. I've put in plenty of effort to communicate clearly. If you don't want to justify the Unix philosophy in a thread created to understand why the Unix philosophy isn't more popular, then why post? I get annoyed when people repeatedly miss the same point, but I still try to explain myself. I don't dismiss people out of hand or announce that the argument is over. When I don't want to reply or explain, I'll just stop. >>17084 Yes yes, and if you read The Art of Unix Programming, you'll get lots of good advice by a seasoned programmer, but it doesn't change that the central paradigm of the Unix philosophy is small programs chained through a text interface. >Make each program do one thing well. >Store data in flat text files. >Use shell scripts to increase leverage and portability. >Make every program a filter. >Write programs to handle text streams, because that is a universal interface. >Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.

>>17082 >Not quite, although it goes a long way by splitting work among existing workers instead of spawning a new "cheap" process (lol) for every fuarrrking connection. Even if processes are expensive, you'd still want to have more than one worker, right? What exactly do you have in mind? CUDA, SIMD, ...?

Be civil.

>>17083 >Thinking coroutines are so heavy as to make a service not micro Do you know what a coroutine is? A coroutine is just a function that can be suspended (i.e. while waiting for the result of some async operation). The logic it performs can be arbitrarily complex. And indeed, why incur the cost of passing data to another process when you can do it in the same executable, just in different threads? >>17091 >Even if processes are expensive, you'd still want to have more than one worker, right? Yes, roughly speaking you want as many workers as you have cores. >What exactly do you have in mind? Nothing fancy, just do as mentioned here: >>17074 ; divide work among threads within the same process and have coroutines implement the work logic. Although it's definitely a good thing if you're able to spawn multiple copies of the program on separate physical machines (in that case you would want as little communication between processes as possible to benefit from parallelization; see Amdahl's law).

Yesterday I had the realisation that I can just use entr[0] instead of all these ugly on-save autocmds ib my vimrc. Felt pretty good. It's definitely one of my favourite Unix-school programs. [0] http://eradman.com/entrproject/

>>17093 Let me remind you that in every unix-like or clone, like linux, there's no such a thing like threads, at least in userspace, being only visible to the kernel. Pthreads just emulates the threads in userspace using shared memory, semaphores and message queues. fork() (and clone() in linux) is your only option. :3

>>17100 >fork() (and clone() in linux) is your only option. :3 On Linux, fork is a wrapper around clone. Clone is the primitive, not fork. Linux provides the same amount of support for threads as processes. Linux threads are true threads without the overhead of processes. The memory copy integral to forking, and the overhead it causes, is enabled and disabled with the VM_CLONE flag passed to clone. Processes are expensive enough that Linux provides support for true threads. https://linux.die.net/man/2/clone

>>17054 >TCP is a stream based protocol. There is no guarantee that the data read in by a socket will include an entire application message. In fact, there's no guarantee a complete message will ever come. You might want to review network technology, TCP is there indeed to guarantee information integrity. One can always ditch the HTTP and parse the data however you see fit just above the TCP layer and cut down the overhead of HTTP encapsulating and parsing.

>>17118 >TCP is there indeed to guarantee information integrity. TCP gurantees a very particular type of integrity. If the client provides bad data, the server must recover. >One can always ditch the HTTP and parse the data however you see fit just above the TCP layer and cut down the overhead of HTTP encapsulating and parsing. Yes, you can ditch HTTP, but if you want discrete multibyte messages you must have something like HTTP to convert streams of bytes into discrete multibyte messages. People realized merging the logic for pulling messages off a byte stream and parsing those messages is not a good idea. That's why the structure of HTTP and friends is so simple compared to JSON and XML.

>>17118 You're correct. >>17119 You're moving the goal posts. Nobody cares about "a particular type of integrity", you must presume that what goes in, comes out the same. Making any speculation about the application layer is stupid af. >HTTP and friends is so simple compared to JSON and XML. If you're comparing a protocol (HTTP) to a data structure (JSON/XML), it's obvious you're talking out of your ass.

>>17123 >You're moving the goal posts. The goal posts are exactly where they were. You cannot take in a stream of bytes over TCP without HTTP, FTP, SMTP, a similar protocol, or by rolling a similar ad hoc protocol. Good practice is to separate building a message from a stream and processing it. Good practice is to use an existing protocol rather than making one up. Rolling an ad hoc format and coupling the stream parsing to the message parsing is bad. >Nobody cares about "a particular type of integrity", you must presume that what goes in, comes out the same. When I said "There is no guarantee that the data read in by a socket will include an entire application message." he replied "TCP is there indeed to guarantee information integrity." TCP does *not* guarantee an entire application message will be provided. Protocols like HTTP, FTP, and SMTP translate a stream of bytes into discrete messages that an application can handle. These protocols exist because TCP provides *no* guarantee that an entire application message will make it from client to server or that the application message will be well formed. >If you're comparing a protocol (HTTP) to a data structure (JSON/XML), it's obvious you're talking out of your ass. I'm comparing the structure of HTTP with the structure of JSON and XML. That's why I said "the structure of HTTP and friends is so simple compared to JSON and XML". HTTP has a very simple structure. In a HTTP response, there's the status line, the headers, and an optional body. HTTP requests are similarly simple. In comparison, a JSON or XML support arbitrarily deep nesting. I'm not arguing with you over the fact you called JSON and XML data structures (they're not) because I took the time to try and understand what you wrote and give you a useful reply. Please return the favor.

>>17040 how is it a waste of resources compared to the line of Common Lisp you gave? i know its shorter, but one would think the C version is way lighter on system resources

>>17040 how is it a waste of resources compared to the line of Common Lisp you gave? i know its shorter, but one would think the C version is way lighter on system resources

>>19806 It's unfortunate this thread didn't die. The discussion was largely resolved. >i know its shorter, but one would think the C version is way lighter on system resources That's a misconception, because people pretend C is an efficient programming language. A Common Lisp system uses what's called a single address space model, where all code is run in the same setting; since the language makes safety guarantees, this isn't dangerous to do, as it would be with C. When a process is created under UNIX, it gets tables populated, there's context-switching, and many other wasteful things. To call the function in Common Lisp is a single function call, with no context-switching. Don't be deluded into thinking C is automatically more efficient. Furthermore, even if it were, consider that people tend to write much simpler software in C. It's easy to write a C program which will return wrong answers and leak memory quickly, and it's easy to write a complex Common Lisp program which allows the machine to handle the drudgery. Pain isn't a virtue, and a language which pretends the machine should be hard to program is painful.

>>19815 wasn't trying to restart the debate, just trying to learn more about this stuff. thanks for the info.

>>19978 Perhaps my first sentence was too harsh. I'm glad to have helped.

I don't get it

>>17032 Not that I don't agree with both unix philosophy and free software and I recognize that I'm replying to a 6 month old post, but uhhh, you do realize the entirety of unix was very much proprietary software, right? That's why BSD and GNU even exist.

What about .NET and powershell?

>>20383 I wasn't born yet, so please point out where I'm wrong. Since "Open Letter to Hobbyists" dates to 1976 and I trace Unix to 1969, I say Unix predates the proprietary age of software and comes from the time when software was dominantly de-facto libre, without the label or the recognition. It must have been later when Unix got commercialized. (Indeed the release and development of 1BSD between 1974 and 1978 coincidently frames the publication of "Open Letter to Hobbyists".) I suggest that the fact that legal status was needed to be settled by USL v. BSDi in January 1994, when both BSD and UNIX was more than 15 years old, is indicative that in the years around 1976, the original nature of software distribution has diverged. Does anyone know when and how the corp took Unix from the hands of Thompson and Ritchie and started to prep it as a commercial asset?

>>17028 As with every technology there is a time and a place to use it. the UNIX philosophy is great for certain applications, but its architecture is not the silver bullet its hyped up to be. I don't imagine that having a lot of processes in a lot of pipelines is all that efficient or maintainable when you get to particularly complex programs.

>>20934 It was a commercial asset from the very beginning, and the users paid for site licenses as soon as it was available outside bell labs. These licenses included source as well as an executable program, which was influenced the direction of computing, but that doesn't make it not proprietary. Worth noting as well here to avoid confusion that 1BSD (and 2BSD) required an AT&T license to operate. Both because it used proprietary code (I'm not sure if this was just the early 80s equivalent of kernel headers, or something more), and it was an enhancement, not replacement of the AT&T unix release. I'm not really sure of the details on this though. Was it just

>>22135 Was it just... Was it just the additional "commands" such as ex, or were there deeper changes in 1BSD?

>>22136 IIRC, the BSD operating system was almost a complete rewrite of Unix, modulo 6 files that were not easy to replace without affecting the whole system (or nearly copying them verbatim). Actually, there are two major kinds of Unix variants: System V, which is Bell Labs Unix, and BSD. One major difference is the STREAMS IPC mechanism. I think it was SysV that originally used it, whereas BSD was the one to introduce sockets (and tcp/ip as well). STREAMS still has some place somewhere, but if I'm not mistaken, Linux too dropped it (even though linux is a SystemV clone), making them largely unavailable in the current state of the art. I am not aware of the other differences, I think vnodes were too a CSRG introduction,

>>17042 AFAIK templeOS does that >>17028 >Why do people at work don't the unix philosophy for write their programs? but they do; and you are yet to define "the unix philosophy". let me try doing so from it's most general ideas: >modularity the first and most important idea. most programmers (at least try to) use modular designs in their programs nowadays >ease of write, read, and test should be a consequence of modularity >batch-processing still used, specially in the case of micro-services, web [json/xml/...] apis, and so on. it does not apply to "modern" interactive applications though. but common sense dictates that those programs are not the target of the unix philosophy >fast development, short cycles agile development without the buzzwords. relatively widespread up to this point "the unix philosophy" seems to be mainstream. I think that those 4 points are the core ideas of the unix philosophy. they are not original to the unix philosophy, which was not a new idea, but a compilation of good ideas that have been around since the start of computing (or even before). However, I get the impression from you example that you mean something more with the term "the unix philosophy", be it: >text streams as main interface which might have been a good idea at the time, but it would be an anachronism to try to accommodate them to modern standards and requirements. we are beyond that level of complexity, and it doesn't seem like we are not going back anytime soon

>>17032 This blatantly wrong on its face. Besides the fact that UNIX is itself proprietary, the only true UNIX system that's currently widely-available to the public is macOS, and it uses the UNIX philosophy extensively within the bundled programs in it. Every piece of Apple software does roughly one or at best two or three things, tops, and for everything else it relies on calling upon another piece of software, usually another Apple service. I need not remark how fuarrrking hard this makes some people seethe, first because they feel that the individual programs are underpowered (and thus not worth their time), second because they feel that these individual programs are 'chained' to the Apple ecosystem, and third because Apple often encourages third party devs to follow similar design guidelines. Now consider that the emblematic software of the more fanatical arm of the open source community (FSF and GNU project) is emacs - a text editor so bloated it's become essentially a miniature operating system into itself. And part of why emacs became this bloated is because its developers had such a strict, reductionist view of what constitutes 'free' software that they found themselves forced to expand emacs in order to give it functionality they could otherwise only obtain in software they steadfastly refused to utilize. This is of course the absolute and complete opposite of the UNIX philosophy.

>>24901 >Now consider that the emblematic software of the more fanatical arm of the open source community (FSF and GNU project) is emacs - a text editor so bloated it's become essentially a miniature operating system into itself. If Unix had one language for both scripting and systems work and if that language was as expressive as Emacs Lisp, then no one would bother with Emacs. (These conditions have been met before.) But right now, Emacs makes a better "operating system" than most of its hosts. When init files are elisp, every configuration language is the same. When programs are elisp functions, program chaining is trivial. Function composition has always been easier than piping and more powerful than piping. >And part of why emacs became this bloated is because its developers had such a strict, reductionist view of what constitutes 'free' software that they found themselves forced to expand emacs in order to give it functionality they could otherwise only obtain in software they steadfastly refused to utilize. This is of course the absolute and complete opposite of the UNIX philosophy. What functionality are you thinking of? I've heard the support for different encodings is the ugliest and most bloated part of Emacs, but I don't really know.

>>17028 C functions and unix tools don't scale well. For large data you need indexing and hashing which those tools don't provide ...

>>24922 PHP, Java and node.js have even worse scalability than C and yet they still use them. >For large data you need indexing and hashing which those tools don't provide What about the filesystem itself?

> For every connection, the socat will fork and execute an instance of my program with the standard file descriptors redirecting to socket, so every write to stdout, with fflush to make sure that will be written, will be send to the connected socket and every data the client sends can be read by reading to stdin. Ever heard of the C10k problem and how Apache failed to address it for the longest time due to using that thread per connection model, which simply doesn't scale well past a point? Turns out a core-bound number of workers and a queue are more efficient. https://www.nginx.com/blog/inside-nginx-how-we-designed-for-performance-scale/ For a more home-computing example, chaining program calls in shell or Elisp to do filter and list files based on some predicates rather than doing it directly from CL can be magnitudes slower due to the constant context switching. That's not to say that atomicity is bad, not at all. But there are ways of applying it which work better than others.

>>17044 > The micro-services you describe are nothing like Unix pipes. And that's why Eshell, Rash and Powershell (yes I went there) are vastly more powerful, as you can actually work with structured data.

>>24939 Yes. With the caveat that you're stuck in the (Emacs/.NET/Extended Racket Multiverse) world. Still, if you have to choose your own hell, this is a decent choice.

>>24901 Very based >>24922 Irrelevant. You invoke a tool to build the index and perform operations on the index you built. >>24938 99.9% of people don't have the C10K problem.

>>24938 Parallelism can be done by spawning new instances of a program or forks, forks being more efficient, but still no need for SRC threads or any black magic outside UNIX.

>>27229 >SRC threads or any black magic outside UNIX. What are SRC threads?

>>27230 I think the name means Shared Resources Concurrency. I will be honest with you, it's really difficult to find any information about threading models, but basically, shared states between threads. It's counter-part is CSP, Communicating Sequential Processes, which is mostly used in Golang with the keyword "go" and channels, but you also can find this model in plan9's libthread and libtask. You can also emulate CSP with simple pipes. https://www.hpl.hp.com/techreports/Compaq-DEC/SRC-RR-35.pdf https://swtch.com/~rsc/thread/

>Why do people at work don't the unix philosophy for write their programs? I have been thinking about this recently, but how one would follow this philosophy for making something like Gimp/Photoshop? That is not the kind of software supposed to solve a single problem. Should I have 30 software that do the same thing as Photoshop just so I have "each program do one thing well"? It seems like this philosophy worked well back in Unix time and for any command line program you make, but not if you are doing some big software that uses GUI.

>>28620 Ideally, you'd just make a graphical frontend to smaller existing image processing programs like imagemagick, pipelining them together to achieve complex effects. Same for a video editor, most of the functions you'd expect could probably be implemented using ffmpeg alone.

>>17050 ...isn't this how RFCs say you should parse HTTP?

>>28621 One of the issues with that is that it doesn't scale that well for a number of tasks. Using pipes to chain awk, grep, sed and sort results in less efficient and slower results than simply coding the whole thing in awk. That problem only gets magnitudes worse once you leave the lightweight word of text and get into heavier data formats. Pipes also do not allow for easily sending structured data without serializing it, which means in a lot of cases you end up having to serialize and de-serialize for pipe-transfer several times in a given chain. Or you end-up with blobs like awk that can simply do the whole job themselves better.

>>17053 The primar difference between stream sockets and datagram sockets is that streams are connection-oriented and do not preserve message boundaries. Datagrams do not have connections and preserve mesage boundaries. For SOCK_DGRAM, ordering and reliability very much depend on the underlying protocol/transport. AF_INET / SOCK_DGRAM / IPPROTO_UDP uses UDP which is unreliable and can be re-ordered. But most UNIX implementations of AF_UNIX / SOCK_DGRAM are reliable and do not re-order. So it would be unwise, imo, to conflate the concept of SOCK_DGRAM with the concepts of re-ordering and reliability because it can be misleading in some contexts.