script langs when sh isnt enough

For a while I've been one of those blockheads that just writes everything in python, but recently I've been taking the time to actually learn proper shell scripting (plus awk and sed), and I've found that it's a much better choice for 90% of my general purpose scripting. I'm curious about the remaining 10%. Call it the "hash table threshold" i.e. when whatever you're doing could really stand to benefit from proper data structures. For example: dealing with a complicated web API, lexical file parsing, doing anything with a substantial runtime state. This is of course different for everyone so I'd like to know where your hash table threshold is and what language you tend to choose when you reach it. The obvious answer is "python" but I'm particularly interested in thoughtful alternatives, preferably ones that act more "UNIXy" than python. this is also your chance to inform me on perl

>>17359 I am a Python blockhead. I use Python because the syntax is simpler, the type system is better, argparse makes argument parsing trivial, and argparse auto generates very useful help messages. I use shell scripts when I need a short program to & off other programs and wait on them. foo & FOO_PID=$! bar & BAR_PID=$! wait FOO_ID BAR_ID

>>17359 Don't use Perl, your coworkers will hate your guts and damn you to hell when you leave. Just use Python, it's good enough for nearly everything.

>>17363 Fair point but I'm talking about my own little scripts to make my life easier, not stuff for work. I'm definitely still going to use python for the things it's good for, but at this point I'm competent enough with it that I'd like to broaden my horizons a bit. My main problem with python on my own systems is portability. Sure, you can get python code running anywhere but you're constantly pulling in libraries and creating virtualenvs and such. This is fine on my desktop but on a server or my router I'd rather have something with a bit less overhead. My second problem with it is speed, but that's rarely relevant in the context I'm talking about. >>17360 The type system is by far my favorite thing about python. When I first saw the trick where you override get/setattr in a dict so that you can access entries as if they were member variables it blew my mind. The only gripe I can think of is that polymorphism is kind of weird if you're used to C++. For such a flexible language there doesn't seem to be a great way to do multiple constructors so people resort to class methods like SomeClass() vs. SomeClass.fromstring(s). The syntax is also really nice (although I prefer julia's take on it), but I can't think of many languages where the syntax is really an issue for me.

>>17366 I should also mention that interfacing with various UNIX tools using python is a little more complicated than I'd like. This is certainly not a python-specific problem but the fact that perl is almost uniquely good at this is one of the reasons it caught my attention.

As said, the main problem with I have with *sh is argument parsing, and python is great for that. For personal scripts basic argument parsing could be enough though. The python logging infrastructure is also great. As a shell language I now use fish, which is much "cleaner" than bash (which also means compatibility break, so portability is not great if the script needs to run on random hosts). As for python interface with linux, I think the best way is just to have python deal with the main logic, and to call small bash scripts when needed.

I like racket quite a lot.

>>17369 >I think the best way is just to have python deal with the main logic, and to call small bash scripts when needed. The thing about trying to use python scripts in conjunction with shell scripts is if you're starting the python runtime anyway you might as well shove everything else into the python script too, so it kind of structurally breaks the whole "little programs piped into each other" idea. Why write your python script to handle stdout from a grep command when it's easier and probably just as fast to load in the file and re.match() whatever you need? This isn't even necessarily a bad thing but it's not the most elegant thing in the world either. You're definitely right about the argument parsing though. The python module for that is brilliant to use. >>17371 I've been curious about racket for a while but haven't learned any lisp dialects yet. Do you just use it because you like the syntax and know it really well, or are there more technical things that make it a good scripting language?

>>17359 Unixy isn't always good. You could argue that it's better for your sh alternative not to be Unixy, so it doesn't have the same set of weaknesses. Perl is better than Python for doing awful things to text, but then that's also where sh shines. >>17366 At a small scale it's often easy enough to manage without virtualenvs. Just install packages outside a virtualenv with pip's --user flag, or install them with your language's package manager if they're packaged. It depends on how many libraries you use, of course. But if you just need requests then it's fine.

>>17359 I've started using zig for scripts as it's not too painful to get things going rather quickly when you write in a test block and it produces static executables. Not really a sh replacement yet it's getting there

>>17377 >Unixy isn't always good. True enough. I don't want it to seem like I'm soykafting on python. I think python really shines when used for things like web scrapers where you have to poke an API just right, gobble up a variety of data, and then parse something meaningful out of it. If the response is json just use the json module, if it's xml use the xml module.. Definitely wouldn't try to do that with a shell script. Instead of mentioning python alternatives in the OP i probably should have asked for an awk alternative because I think that's really what I'm looking for: a simple language that I can plug into a command pipeline without introducing a lot of extra dependencies or runtime overhead... that's just a little more powerful than awk (i.e. isn't so line-oriented). Actually the thing that motivated this thread was an ill-advised shell script I had just finished writing for doing template substitution on LaTeX files (essentially: match tags and either prompt the user for substitutions or execute commands based on its attributes). I got it done but it was definitely a bit of a nightmare. Everything was super easy to do with just grep/sed/awk except for the bit where I had to parse the tag itself and work out what to do with it. The logic was easy enough, if embarrassingly reliant on for loops, but coaxing the strings into variables and splitting them up without somehow mangling them was as much of a clusterfuck as it always is with shell scripts. I definitely could have done the whole thing with python but python seemed like an overkill solution for a very simple problem. >>17377 >it's often easy enough to manage without virtualenvs Yeah maybe I'm being a baby about this. I think I may just have some form of ptsd response from working with pytorch/caffe so much.

>>17372 >Do you just use it because you like the syntax and know it really well, or are there more technical things that make it a good scripting language? The libraries and object model are very nice to work with, it's a relatively friendly lisp, and it has the best debugger of any language I've seen.

>>17382 You can actually do nice things to JSON in shell with jq. It's worth learning just for interactive use. I think Perl is often used in AWKish ways, with one-liners embedded directly in shell. It's a natural fit for what you've been describing.

>>17383 I really like developing in languages with a good REPL, but they're sometimes hard to take advantage of in (neo)vim unless you just stick it in a separate terminal. Do you use DrRacket or just your normal text editor? >>17384 >jq Thanks for the tip, I totally forgot this existed.

>>17385 I use drracket for debugging and typically nvim for writing

>>17380 >zig holy soykaf i hadn't heard about this but it looks great. it's like a "C replacement" that actually respects the things that are good about C.

>>17372 >are there more technical things that make it a good scripting language? Unlike Common Lisp, Racket is script based. It's trivial to make a "stand alone" executable with. It comes with batteries included. The standard editor is easy to learn. The module system is simple to understand.

>>17372 > The thing about trying to use python scripts in conjunction with shell scripts is if you're starting the python runtime anyway you might as well shove everything else into the python script too, so it kind of structurally breaks the whole "little programs piped into each other" idea. Why write your python script to handle stdout from a grep command when it's easier and probably just as fast to load in the file and re.match() whatever you need? This isn't even necessarily a bad thing but it's not the most elegant thing in the world either. You're definitely right about the argument parsing though. The python module for that is brilliant to use. Yes for any data manipulation I would stay in python. I was talking about unix utilities for when you need to interact with the system (create users, run another program, this kind of things).

>>17359 Youre making a good decision to not get into shell scripting. A shell is good for interactive use, but for scripts, scripting languages so much more powerful. For a shell Im using zsh because its backwards compatible with bash and adds a lot of nice convenience features and avoids the many problems of bash. I tried fish, I like the syntax a lot and would like to see it get more traction. But the main draw for fish is cleaner syntax for scripting and as Ive already mentioned Ive decided to avoid shell scripting altogether and use scripting languages. For scripting language my main choice is Julia. However I am currently spending my time learning Python for shell scripting. Surprisingly there are not a lot of books out on using Python for systems administration but there are a fuarrrkton of Ruby books that deal with sys admin'ing. The problem with Python is that everything is a ready made library that you just fill in the blanks for, so if you really want to get deep into making your own scripts you will find Python to be a barrier. Here on the docs for using Julia to run shell commands: https://docs.julialang.org/en/v1/manual/running-external-programs/ Bear in mind that many Python libraries are just automating shell commands, so its better to roll your own instead of learning someone elses solutions.

>>17390 Personally I think that the best lisp for scripting is guile scheme. It's fast, it has very good system tools and it's as easy or easier to racket. Don't get me wrong, I love racket, especially for gui development, but for scripting I usually go with guile. The fact that I use guix makes it a no-brainer as well.

What about Ruby?

>>17406 Ruby is actually favored by hackers, pen testers and reversers, its a great language but I feel its getting long in the tooth compared to modern scripting languages, its smalltalk flavored OO tends to cock block it from general data processing tasks (IMO)

>>17405 >>17371 Never tried to make a script in any Lisp, but, hey, Guile is the official GNU scripting language after all. They just got a nice JIT compiler for v3.0 from what I've heard. It's one of the only Scheme dialect that is appealing to me because it could somehow scratch the same kind of immediate itch that Emacs Lisp and SBCL does to me. Otherwise, SBCL has --script mode. But so far Bash and Python have been good to me on that front except the time when the client doesn't want to upgrade its 15-years-old RHEL servers and has a hard time envisioning a move to Java 8 That's when you have to get around bash's lack of fancy data structures, and it's a pain, but I'm still thankful for their array implementation.

>>17407 >Ruby is actually favored by hackers, pen testers and reversers, I know Ruby is used for Metasploit, but I've never met anyone happy about that.

>>17407 Most people write their exploits in python. You're an idiot. Sage

>>17420 This is true, a lot of exploits are in python. I've seen maybe a handful in ruby, and fewer in perl. I've also seen some in PHP (I lawl'd, so should you).

>>17420 >>17422 What are you going on about? Almost every publicly known vulnerability has been exploited with Ruby. That's how they're added to metasploit.

>>17400 >A shell is good for interactive use, but for scripts, Kind of depends what sort of script you're talking about. Check this out. #!/bin/sh idx=`calibredb list -f title,authors,tags -w 10000 --separator=$'\t' | tr -s " " | awk -F $'\t' '/^[1-9]/ {printf "%s %-50.50s %-30.30s %-10s\n",$1,$2,$3,$4 }' | fzy -l 20 | awk '{print $1}'` filepath="`calibredb list -s "id:${idx}" -f formats -w 10000 | grep -oP '(?<=\[).*(?=\])'`" if [ -n "${filepath}" ] then nohup zathura "${filepath}" > /dev/null 2>&1 & fi This is my script for getting a list of pdfs from calibre, providing an interactive selection prompt, and opening one in zathura. I'm very happy with how efficiently I can open books now instead of having to load up calibre's gui and click my way to the file I want. Maybe you had something different in mind when you said "scripting" but it wouldn't make any sense to implement this sort of thing in python. Notice how there's almost zero actual program logic. It's just two assignments, one branch that just checks if a value is nonzero, and a ton of pipes. For stuff like this shell scripts are faster, not only to execute but also to write in the first place. (Btw like I said in the OP I've just started seriously learning sh so if anyone has suggestions for how to improve that script please let me know)

>>17426 fuarrrk i just realized $'\t' is bash syntax... i'll fix it

>>17390 >>17405 glad to hear from some people using lisps for their scripting. i love the way lisp scripts are structured but i wasn't sure about how well it works for this kind of thing (no particular prejudice, just didn't see many people talking about them outside of heavy emacs users). >>17417 What's the deal with SBCL? Isn't it compiled, or does the --script mode do JIT? I just know it's supposed to be the "fastest lisp" or something.

>>17426 >Maybe you had something different in mind when you said "scripting" but it wouldn't make any sense to implement this sort of thing in python. It would make A LOT of sense to write that in Python, that script is a mess >Notice how there's almost zero actual program logic. It's just two assignments, one branch that just checks if a value is nonzero, and a ton of pipes. For stuff like this shell scripts are faster, not only to execute but also to write in the first place. That script is basically sending all the work out to awk and grep, which is not even part of bash but are external shell utilities. In a scripting language that could all be done with regexps internally in the language. Every time you want to refactor you have to rewrite the entire script instead of modularization it inside functions as you would in a scripting language which would allow you to make this a general utility instead of something which can only be used for one specific purpose

>>17428 >Isn't it compiled SBCL compiles to machine code. >does the --script mode do JIT >--script >Causes the system to load the specified file and exit immediately afterwards, instead of entering the readl-eval-print loop. If the file begins with a shebang line, it is ignored. https://linux.die.net/man/1/sbcl

>>17372 Racket is basically what would you'd get if Python had descended from Scheme instead of Modula-3. It's extremely flexible (you can redefine core aspects of the language if you wish) and has a massive set of massive libraries.

I've written very few sh programs, and dislike it every time. Even knowing many of the pitfalls, it's unreasonably difficult to write a sh program which isn't susceptible to runaway execution on malicious inputs. Even if the entire program is a pipeline to avoid the asinine variable transclusion, so many commands have special treatment for certain input characteristics that can slip by. Doing any complex computation, such as to process command flags, is so arduous I don't bother. I do the majority of my computing in Emacs and I also use StumpWM, so I usually have an omnipresent Common Lisp REPL, and I use it for miscellaneous work, including some programmatic file management. I've reflected on how the Common Lisp is much longer than the equivalent sh symbol vomit, but it's so relaxing to know exactly what the program does, that it won't choke on a file just because its name has a space, and also to know if I do need to interrupt the program, I can introspect on its current state, among other pleasant qualities. I can think of a single instance where I've called an external program with Common Lisp, and it's rare at that; for most cases, I'd write what I want from the external command in Common Lisp if I could. My interaction with any sh is mostly just running single commands or chains with semicolons. I've been using eshell lately in more complex ways, since it allows me to easily call external programs while also using the control structures and variable mechanisms I'm familiar with, and it's been nice enough so far, nicer than using the ugly $(()) structure which doesn't do everything I want. If I use sh more in complex ways, it will probably be with eshell for most everything.

>>17359 Just use Haskell or Rust as a scripting language. Do you really want to execute scripts in your computing environment that have tons of unchecked errors, edge cases, and ways to fuarrrk everything up? And are slow, to boot? Type inference + ADTs makes for a better scripting experience than the dynamic languages. More readable documentation, no mystery about what's going on. Functional and expressive aspects make for simpler, nicer code. Compilation means it's dead simple to run without executing some interpreter or VM. The code's safer so there's less of a chance that you can screw up your OS.

>>19007 Because it's not worth the time and effort. If I'm writing a program that exists solely to download youtube videos, I don't really care how good the code is as long as it does its job 'good enough'. I'm not going to waste my time in rust or haskell just so I can fix nonexistent bugs and make my program .001% faster.

>>17372 >The thing about trying to use python scripts in conjunction with shell scripts is if you're starting the python runtime anyway you might as well shove everything else into the python script too eh.... generally python is best for when you need some nontrivial libraries, but shells are best for when you're mostly moving around files or piping to/from other programs(including your python programs)

>>19006 >it's unreasonably difficult to write a sh program which isn't susceptible to runaway execution on malicious inputs this is fair enough, and kind of what motivated this thread. imo the best use case for sh is situations where the script is doing almost no work on its own. ideally a shell script has no conditionals or loops at all, just pipes and something like dmenu or fzf for interactivity. obviously this isn't always feasible, but whenever i find myself using a loop in a shell script that's my personal cue to consider maybe using a more powerful language, and if a need a data structure more complicated than a delimited list i'll definitely use something else. that being said, i've found tons of opportunities to improve my day to day computing just using this "pipes only" sh subset. for instance, setting up a script that launches from a hotkey and lets me open config files from an interactive menu instead of typing out the file path. >>19012 true, i think he was suggesting the opposite though: using python as the "glue" between other programs, which doesn't make sense to me. >>19007 >Do you really want to execute scripts in your computing environment that have tons of unchecked errors, edge cases, and ways to fuarrrk everything up? And are slow, to boot? It really depends. I tend to use shell scripts for the kind of thing you'd use autohotkey or whatever for on windows. Basically they're little more than macros so safety isn't a huge concern since I'm providing the input myself, and speed is perfectly sufficient for this kind of task (especially since I'm generally using dash). The ease of gluing together external programs is the main benefit over a compiled language where you're usually pulling in library deps and have to learn how those libraries work. With a shell script I'm using the same programs I'm already familiar with. I do sometimes use shell scripts for daemon-type tasks but typically this amounts to "download this file from my server every 24 hours" so again it's hard to justify adding additional complexity (and I try to rely on features of the init system so that the script is doing as little work as possible).

>>17433 >It would make A LOT of sense to write that in Python The only thing a python reimplementation would simplify is the awk/grep commands (which are already pretty simple), at the cost of added complexity in interfacing with external utilities. >that script is a mess are you sure you aren't just getting tripped up by the shell's somewhat unintuitive syntax? it's not exceptionally readable if you aren't used to long pipe lines, but it's not logically complicated. i'm admittedly not an expert shell scripter and i'm sure there are places it could be improved but i don't see anything wrong with it (besides the fact that I call calibredb twice, but that decision mostly comes from some annoying design features of the calibredb utility). >That script is basically sending all the work out to awk and grep, which is not even part of bash but are external shell utilities Yes... that's the point. sh makes doing that easy. >which would allow you to make this a general utility instead of something which can only be used for one specific purpose what part of that script's functionality would be useful to abstract? it's three logical lines that operate on very specific data and do a very specific thing. even if i wrote it in python there's nothing in there that would be useful to me in another script. also, you can definitely share code with sh, and its arguably easier to do so than python. for instance, i have a small "interactive choice" script that checks if its running in an interactive shell and executes fzy on stdin if so and dmenu otherwise.

>>17359 The most UNIX way? Typicaly in oldschool UNIX times you would just write a program >Call it the "hash table threshold" i.e. when whatever you're doing could really stand to benefit from proper data structures. For example: dealing with a complicated web API, lexical file parsing, doing anything with a substantial runtime state. This is of course different for everyone so I'd like to know where your hash table threshold is and what language you tend to choose when you reach it. What would be done with yacc and lex typically. Very common case in the past in Bell Labs.

>>17359 >script langs when sh isnt enough >when flexibility, extend-ability, and simplicity isn't enough It's always been enough. I always say I'll learn Python, but never have a need to because I can already do it with a shell script.

>>19011 >If I'm writing a program that exists solely to download youtube videos, I don't really care how good the code is as long as it does its job 'good enough'. I'm not going to waste my time in rust or haskell just so I can fix nonexistent bugs and make my program .001% faster. I'm not talking about a one or two line snippet.

>>19013 >It really depends. I tend to use shell scripts for the kind of thing you'd use autohotkey or whatever for on windows. Basically they're little more than macros so safety isn't a huge concern since I'm providing the input myself, and speed is perfectly sufficient for this kind of task (especially since I'm generally using dash). The ease of gluing together external programs is the main benefit over a compiled language where you're usually pulling in library deps and have to learn how those libraries work. I can totally agree to this. >>19011 >>19029 and let me clarify, if you are scripting anything in Python you have already added as much or more headache and complexity to your script than you would in Rust or Haskell. Yes if you just have a handful of bash commands you keep executing over and over, and you want to make a little shell script, that is easier than writing a script in Rust, Python, or anything else because the code is already in your bash history. But the day that dynamic languages were "easier" is over. We have both functional and procedural languages with ADTs and type inference. And not having to write a bunch of type annotations was one of the only real appeals that the dynamic languages ever had.

Shell scripting is one of the most useful things in existence. The only thing shell scripting is not good at is binary (and maybe performance?)

xonsh is the perfect solution. It is basically a bash+python hybrid so you get the best of both worlds. Here's a sample I used to webscrape a manga while dealing with some malformed urls #!/usr/bin/env xonsh baseurl = 'https://blahblah.com/chapter-' def getchapter(ch): page = baseurl + str(ch) + '/' images = $(curl @(page) | pup '.entry-content a attr{href}') duplicates = {""} for i,x in enumerate(images.split('\n')): img = x[x.rfind('https://'):] if img not in duplicates: duplicates.add(img) wget @(img) -O @("{:03d}-{:03d}.jpg".format(ch, i)) for i in range(1,11): getchapter(i)

>>19426 indentation didn't work (uses python-style indentation) but you get the point. the '@(python stuff)' syntax is for using python expressions in the bash-like parts.

I will have to agree with >>17360 on this, especially for what I do for deployment automation. Fabric2 is a great module, and can be invoked with a command line utility to be bash scripted (fab). Highly recommend if you ever need it, it's well supported, reads from your ssh config for hostnames, and allows for easy concurrent execution across multiple hosts. As for languages I've heard that people like csh, but that might just be hipsterdom. You might be fine with just bash, to be honest. I can't see much reason not to use it these days, when most things you use are going to have it installed by default.

- if the code starts looking longer than about 50 lines or I start wishing for libraries from python then I would move to python. - anything that needs to be optimized for speed and concurrency i.e. fuzzing type of stuff i will move to go go's syntax is very readable, like a mix of python and c. I find it good for writing scripts.

Perl 5/7 seems to be a good mix between Python and Bash

>Python The package management system is a garbage fire, so even if you get it working on one computer, good luck getting your script to run anywhere else. >Ruby It takes at least half a second to start up, which makes it completely unsuitable for any script meant to be run interactively IMO >Scheme* Love it as a language but never tried it for scripting. Seems like a good fit.

>>19476 >It takes at least half a second to start up, It's fast on my machine. :) $ time ruby bash: ruby: command not found real 0m0.001s user 0m0.000s sys 0m0.000s

>>19476 >good luck getting your script to run anywhere else. Oh god a few months ago I had to package python software for distribution and it was an abject nightmare. The fact that there are no good ways to do this is astounding to me. Pip is great if you're a developer but totally useless for users who don't have a full python dev environment set up. >>19479 hahahaha

>>19483 Also, getting python scripts to run across multiple computers isn't that hard as long as you use setuptools. Just put all the dependencies in the setup.py and theyll get pulled in automatically. You can even install from a git repo, so I host my portable scripts in git on my fileserver and install them from there whenever they're needed. Still a lot of overhead compared to shell scripts or C but it makes sense for certain kinds of tasks and I could see how someone could get a pretty efficient python-based scripting workflow going.

>>19426 Xonsh feels like cheating. Not the fastest language in the world but definitely the nicest. I hope it sticks around

>>17359 >script langs when sh isnt enough But it is always enough. #!/bin/bash array_1="a b c d e f" array_2="g h i j k l" array_3="m n o p q r" array_4="s t u v w x" array_5="y z" let nb_array=5 for j in `seq 1 $nb_array` do declare current_data="array_$j" current_data=${!current_data} for letter in $current_data do echo $letter done done

>>19493 What is your example supposed to show? All it does is output each letter on a newline. # You might as well do this. for letter in $(echo {a..z}) do echo $letter done Is it the way you generate variable names with string interpolation? String interpolation is the devil. >But it is always enough. So is m4, but I still don't want to be anywhere near anyone who has it as their programming language of choice.

>>19519 It shows how you can have various variables ending with some kind of index (1, 2, 3, 4, etc.) and use the declare within the first for loop to get the variable name. Then you can browse each variables. The content of those arrays aren't the point here. It's how you can somehow make a variable that points to an array and use a nested for loops to browse multiples variables at once. It might be useless, but it works.

>>19614 >The content of those arrays aren't the point here. It's how you can somehow make a variable that points to an array and use a nested for loops to browse multiples variables at once. I'm not trying to be unduly critical, but this is an issue with UNIX, the WWW, and systems that also follow in these footsteps. It could be summed by easy things are hard and hard things are nearly impossible. Rather than provide the quality actually wanted, instead there's a bizarre but technically complete system that can inefficiently beget the quality, usually. It's like how idiots playing with CSS agonize over how hard it is to get certain layouts, when the system should just support them directly, but it doesn't and it's rarely or never actually updated to. This makes higher features people would want with a good bedrock never even materialize. In the same way, there's really no need for the command shell to be a complete language, unless it's a language intended for most programming. The UNIX sh is just an in-between with utterly bizarre variables, noisy and incomprehensible syntax, horrible inefficiency caused in-part by the parse-then-serialize-then-parse model, and so many other misfeatures that make it utterly unsuitable for all but the absolute simplest of tasks. >It might be useless, but it works. That could be a tagline for UNIX, where previously the last three words weren't present.

>>19618 To expand on this post, since it occured to me, sh doesn't allow even the most basic actions to be written in the most obvious way. I want to list the MP4 files in a directory, and type ls *mp4, but this doesn't work, because the first file starts with a hyphen. Instead, I must type ls ./*mp4 and what's the reason for the magic ./ besides it being how UNIX chooses to refer to the current directory. If I want to list all of the MP4s in the current directory, and all subdirectories, I don't even know how to do that in sh, but that's fine, because even the most basic case imaginable is mired in special cases and exceptions that shouldn't exist in any reasonable design. I know the flaw actually lies in how UNIX programs receive their arguments, but that's not an excuse.

>thread recommending UNIXy scripting languages >Tool Command Language hasn't been mentioned once

>>19479 >It's fast on my machine. :) It's fast until you include gems, then it takes hundreds of milliseconds. Good luck finding a useful Ruby script that doesn't use any gem.

>>19707 tcl has kind of been on my radar for a while but I never dug into it. could you elaborate on why you like it? >>19712 lol you're right of course but read his post again more carefully

>>19707 Maybe you should go to rosetta code and compare some TCL scripts with the implementations in other languages.

>>19633 ls -R | grep "*mp4" I think it should be something like this. list directories recursively, pipe to grep to select lines with mp4 in it. normally commands don't do regular expressions, so ls *mp4 should not work, zsh is fancy and might make it work, but in general, i think not. I am not very familiar with shell scripting so take that with a grain of salt

>>19736 ls does take regex, so "ls *mp4" should work. The situation that >>19633 was talking about was that if the first item in the list has a filename that starts with a "-" then ls takes it as a command line argument. Frankly though, I don't know why >>19633 is talking as if this weird rare edge case is some kind of common problem.

perl is what you're looking for . its fast . its closer to C and Unix style .its just good

>>19736 >>19738 I'm simply pointing out that the common case of sh results in programs which only usually work, which I find unacceptable. It should be easy to have always work, but POSIX makes these things unnecessarily difficult, to the point where sh is almost useless, and would be entirely useless were it not privileged. I use Emacs for most of my computing, including dired for manipulating files, so I rarely even use sh for anything but the most trivial of tasks, but even in this I occasionally taste its many flaws.

>>19738 >as if this weird rare edge case is some kind of common problem. Not worrying about edge cases is bad design and the cause of most vulnerabilities. There is no reason why the easy standard way to list out files with a certain suffix should fail if the prefix is a specific special character. >Much of the New Jersey approach is about getting away with less than is necessary to get the /complete/ job done. E.g., perl, is all about doing as little as possible that can approximate the full solution, sort of the entertainment industry's special effects and make-believe works, which for all practical purposes /is/ the real thing. Regular expressions is a pretty good approximation to actually parsing the implicit language of the input, too, but the rub with all these 90% solutions is that you have /no/ idea when they return the wrong value because the approximation destroys any ability to determine correctness. Most of the time, however, the error is large enough to cause a crash of some sort, but there is no way to do transactions, either, so a crash usually causes a debugging and rescue session to recover the state prior to the crash. This is deemed acceptable in the New Jersery approach. The reason they think this also /should/ be acceptable is that they believe that getting it exactly right is more expensive than fixing things after crashes. Therefore, the whole language must be optimized for getting the first approximations run fast.

>>19633 To find all .mp4 files with no edge cases, use the find command. find -maxdepth 0 -name '*.mp4' > I don't even know how to do that in sh find -name '*.mp4' >there's really no need for the command shell to be a complete language, unless it's a language intended for most programming The shell fills the need for a quick to use and yet flexible language. Do you know of any language that better fills this need? Until more 'complete' languages can compete with its ease of use for quick tasks, saying "there's no need" is kind of vacuous. >>19736 >ls -R | grep "*mp4" 1. grep takes a regex, so it should be: ls -R | grep ".*mp4" 2. This also would mess up if filenames contain newlines, so you should probably use the find command in a script or if you aren't sure if you have wonky file names. >>19738 >ls does take regex ls does not take a regex. Bash is the one who actually replaces the "*.mp4" argument with the list before running ls. You can see this if you run "ls '*.mp4'". The single quotes tell bash not to replace the word with the asterisk in it, so it'll probably say something like: ls: cannot access '*.mp4': No such file or directory This is also why "ls *.mp4" fails when a file starts with "-": ls doesn't know that the file came from an expansion of "*.mp4". It only sees the final list of strings that bash provides as arguments. Also, i should point out that "*.mp4" is not valid regular expression. It's called "glob expansion", similar but much more simple and less powerful than regular expressions. Typical regular expression would be ".*\.mp4".

>>19633 fd -e mp4 although that might be cheating for some for using non-coreutils prog

>>17424 Metasploit has python support and most new exploits are written in python.

Ruby, works surprisingly well in Windows environments too.

I'm going to disappoint you on the unixy side. When things start requiring a proper language, I resort to schemes like Racket or Guile. Both have a fairly decent library ecosystem and powerful features. They also both handle tail recursion properly, which is a massive gain for any algorithmic endeavor. Sometimes when I can't be hassled to write FFI to some library that provides a very specific use-case, I'll resort to python. >>17390 https://www.cliki.net/cl-launch kind of abstracts those issues away.

>>19102 >The only thing shell scripting is not good at is binary (and maybe performance?) You can get around this with small files using od, paste, tr, grep, dd, and printf to turn the binary into text, find the offset you're looking for, and make changes. It's literally a one-liner in bash to find the byte offset of a pattern/file within a binary using the above utilities. You just have to figure out how to turn your binary problem into a text-processing one!

>>21325 >Sometimes when I can't be hassled to write FFI to some library that provides a very specific use-case Lisps are cool, but yeah that's always been the issue that's been keeping me from using them in practice. Not even just FFI in the conventional sense. It's just that the lisp paradigm is so different from everything it's going to interact with as a scripting lang that if you try to approach things in a remotely unixy way you're going to be writing a ton of boilerplate. That boilerplate is less of an issue in bigger programs, but for me once a bit of code is more than one file it crosses the threshold from a "script" to "software" and I'm probably going to just write it in C or C++. Honestly it's a real shame that we ended up with posix shells as they are, rather than something based on a lisp. I recently started using emacs as my primary text editor, and so I'm learning elisp as I go. I was surprised by what a natural fit lisp is for that kind of light "shell scripting" stuff. What do you think of haskell? GHC is super heavy and the way the modules get packaged can be annoying in a python sort of way, but the language itself is really nice to work with.

>>17359 perl scales from throwaway oneliners to large enterprise applications. With -n and -p flags, it can completely replace sed and awk. For example, this will replace all instances of 'foo' in file.txt: perl -pE 's/foo/bar/g' file.txt BTW, if you want to see something cool, look at this: https://metacpan.org/pod/ojo ojo.pm is a version of Mojolicious that can be used in oneliners. Mojolicious is a lightweight (zero dependencies!) framework for web stuff. It contains a sinatra-like web framework, templating engine, HTTP server, HTTP client with jquery-like css selectors, async IO loop and more

>>19719 Not him, but it's basically Lisp if it were designed by UNIX hackers and used strings instead of lists. It's a highly flexible language with great support for metaprogramming, and it plays nice with other languages.

would javascript or golang be a good option?

>>22051 I'm one of the world's biggest Go fanboys, but even I have to admit that it's a poor choice for scripting-type tasks. "go run" is a bit of a hack. Best to stick to some interpreted language with good OS hooks.

>>22047 >Not him, but it's basically Lisp if it were designed by UNIX hackers and used strings instead of lists. It's a highly flexible language with great support for metaprogramming, and it plays nice with other languages. But Lisp written by UNIX hackers already exists - it's called yacc.

Does anyone here use Factor (factorcode.org) for scripting situations? I've been looking at using this language for my hobby/non-work lang. Can anyone suggest what parts might become aggravating about using it? I'm just concerned that since it is dynamically-typed it will piss me off more often than not (I usually prefer rust/ocaml static-typed systems). Other than that it looks wildly flexible to me, with embedded logic programming, macros and quite a lot of libraries - not to mention the fantastic developer tooling.

>>22069 doesn't look like it

>>22284 I guess I'll report back with my own news of how it fairs for scripting! Overall I'm not disappointed which surprised me. First of all in the listener there is a backtick syntax (https://docs.factorcode.org/content/vocab-backticks.html) which basically shunts the contents to a bash env and returns the string back which is cool for when I can't be bothered finding the right vocabulary myself and just want the results. A lot of the script-y stuff is in the io vocab (https://docs.factorcode.org/content/vocab-io.html) like io.files and io.directories and you can "." directory-files ... to get a simple pretty print of the current directory (though it is still an array rather than a bunch of strings) If you've got a 2d array you can use simple-table. to format it nicely in the listener like ".bash_history" utf8 file-lines <enumerated> simple-table. its got a decent charting vocab among other goodies in the ui vocab (https://docs.factorcode.org/content/vocab-ui.gadgets.charts.html) And despite the no strict typing it has some interesting things I didn't expect like a pattern matching vocab (to a degree), monads vocab and lazy evaluation vocab. Haven't looked into how easy the c ffi stuff is with the alien vocab but I made a little interactable second order wave visualiser pretty quickly with the raylib.ffi vocab and it seemed pretty straight forward. The no types did annoy me at first but as I understand this lang a bit more maybe it won't be so annoying. The ctrl+w shortcut in the listener to walk (step) through code is decent enough to verify whats happening during runtime instead of compile time guarantees.

Does anyone use closh? Looks similar to >>19426 except that it combines bash with clojure instead of python. I've been looking for a reason to learn closure and this might be one.

>>17366 >on a server or my router I'd rather have something with a bit less overhead That's the point of a virtualenv. You can replicate exactly what's provided on your router (the distribution) into a virtualenv on your machine for testing purposes. Sure, it does not match reality 100%, especially due to being different architectures, but 99% of the time you're going to get exactly what you need. Then you ship the code on your target and run python setup.py install.

>>17359 Since the Internet interfaces under UNIX are garbage, I've wasted time thinking of ways I could efficiently use (x)inetd. A major issue is large data structures that shouldn't be constantly created and destroyed, as (x)inetd does under its simplest mode of operation, which is simply starting a new program to handle one request. One of these large data structures is a trie, and I realized it was reasonably suited to being shared across programs in a way that doesn't involve some garbage UNIX format; it can be shared by embedding it in the file system. So finding the value of "string" would be a request to open the file "/etc/trie/s/t/r/i/n/g/@" and isn't that neat?

>>30026 >I've wasted time thinking of ways I could efficiently use (x)inetd. >which is simply starting a new program to handle one request. Do you understand that xinetd is something from a different era? It was built to conserve memory by allowing you NOT to have all the daemons running at the same time - to simple start them on demand. However, unlike your complaint, you can choose to leave them running after starting them once or start reduced versions that only handle a single client (in.ftpd, etc.). Xinetd is premised on the idea that you are something like a small business hosting all your daemons on one box - which is security malpractice today - you are advised to host different services on different boxes. It had builtin security features that predate firewalls - tcp wrappers, and could configure policy for daemons like only on these hours , and this many concurrent etc. > major issue is large data structures that shouldn't be constantly created and destroyed Yeah son. This mad science you discovered is serialization. Machine learning developers using python use pickle to prevent this same thing. Also comes with a variety of risks. Xinetd is actually a very UNIX way to handle security features and policy - except , you probably shouldn't be hosting multiple services on a single box like its 99 - So I don't know what do with it. Nonetheless, every linux distribution includes it anyway in the form of systemd-socket-activate which does xinetd style activation for .socket units - many of which are local AF_UNIX services where this makes sense.

>>30033 It still provides a convenient interface for when the server needn't access anything but the TCP stream. >which is security malpractice today It's more reasonable now than ever to use one box for everything, since machines are so powerful. >Yeah son. This mad science you discovered is serialization. No, I referred to creation and destruction even from serialized data. Compiling a data structure into the program works, but doesn't if the program is to be interpreted, which is the case for a sh program.

>>30044 >It's more reasonable now than ever to use one box for everything, since machines are so powerful. xinetd doesn't make sense today. Small Businesses self-hosting in the 90s trying to save resources, specially on intermittently used services, and set security policy like no e-mail on saturdays could. Today its *security* malpractice. Most small business are not self-hosting because cloud solutions are cheaper but even if you are self-hosting, you want isolation and virtual machines are cheap. You don't want your DNS service which is critical to the availability of your website, etc, running on the same host as your httpd, and mail server, because when postfix gets pwned, or bind again, or its vulnerability in your web server, now all of these things are at risk. Further, the no e-mail on Saturdays, and intermittent services are mostly not a thing anymore either. People have dedicated boxes for services that are just left up 24x7 because they are dedicated boxes and the hardware can power it anyway. Its is an absolutely horrible idea to put all your services on one box. >No, I referred to creation and destruction even from serialized data. Compiling a data structure into the program works, but doesn't if the program is to be interpreted, which is the case for a sh program. Are you larping? You think hw is uber powerful now and running all your services on a single box is fine, but de-serializing data from disk , and fork() are too expensive? You want to bake data structures into .data but your writing a shell script? It seems like your worried about a bunch of contradictory things. Also hilariously ironic: >involve some garbage UNIX format; Instead I'll make it a file. >/etc/trie/.../ Those who don't understand UNIX are cursed to reinvent it.

>>30050 >xinetd doesn't make sense today. It's easier to use than POSIX sockets. I've written a little Gopher server in sh with it. >Today its *security* malpractice. People claim the same thing about running a website without TLS fake encryption. I disagree. >Most small business are not self-hosting because cloud solutions are cheaper but even if you are self-hosting, you want isolation and virtual machines are cheap. That just complicates everything for little good reason. >You don't want your DNS service which is critical to the availability of your website, etc, running on the same host as your httpd, and mail server, because when postfix gets pwned, or bind again, or its vulnerability in your web server, now all of these things are at risk. Anyone who is truly concerned about this should probably run OpenBSD using its promise functionality. Some basic customization to avoid the typical server software goes a long way, although I don't usually bother with this. >Its is an absolutely horrible idea to put all your services on one box. I disagree. >You think hw is uber powerful now and running all your services on a single box is fine, but de-serializing data from disk , and fork() are too expensive? Yes and no. I was merely demonstrating how a large data structure could be created once and reused between programs, such as a sh program. >You want to bake data structures into .data but your writing a shell script? It seems like your worried about a bunch of contradictory things. I'm playing around and having fun here. >Those who don't understand UNIX are cursed to reinvent it. UNIX is garbage. Those who don't understand computers are cursed to reinvent UNIX forever, instead of doing something better.

>>17359 Try PowerShell. https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-linux?view=powershell-7.2