Databases

What do you use to store data long term in your programs? I personally just use files and directories, like I'd make a directory called data and in it make directories like usr and in it make a directory for each user which would have files to hold information about that user. Simple to implement and isn't tied to any specific programming language or whatever, everything can read normal text files. Any time I tried to use "real" databases, even something as simple as pic related, it feels so bloated and overcomplicated especially for what I had to do. Also relational model is completely braindead and goes against common sense.

>>40143 Actually more information is needed, what type of programs do you do? What type of information do you keep? Now one point, how you save it goes against all good security practice by doing it in simple plain text files. Regarding the options, SQLite, unlike what you say, is not complicated to integrate (well, it depends on the language but in general not), it is not complicated to use either, in fact the beauty of SQLite is that you do not need to install an additional server like working with MySQL/MariaDB or a similar DBMS, now if it is very difficult for you to plan the information you want to store in tables such as user configuration, then you can create a storage in JSON and encrypt it strongly with some function of the language you are using and create the functions to be able to handle it correctly

>>40143 Databases aren't intrinsically tied to specific programming languages either? You've basically invented your own informal hierarchical database, relying on your kernel for transactions and consistency. To be fair, nothing wrong with that approach. But you're retreading the water that lead to relational calculus and relational database theory in the first place. The relational model to data storage is far more flexible than the hierarchical model as relations (aka "tables") are not reliant on the concept of parents or children, but instead operate more akin to a constrained graph node. Use cases for both, IBM IMS isn't dead after all. And depending on how you've modeled your data, one will likely be a far easier modeling tool to use than another. Just like most modern tools, now that RDBMS has "won" people don't learn anything else, or any of the actual theory behind these tools.

>I personally just use files and directories, Often a perfectly reasonable choice. Most small projects don't have much data complexity and it's easy enough to write in & out plaintext. >Also relational model is completely braindead and goes against common sense. Amateurish take. I understand that some of the terminology can feel unapproachable, but relational data modeling can be as simple as "organize things into tables that make sense & don't' store information redundantly". In general, you'd be a fool not to use one for a project with a complicated data model or high-volume dataset.

>>40161 is right. It depends on the nature of your problem. What information are you storing, and for what purposes? What you're doing is paramount to asking an engineer "what's better? A screwdriver or a nailgun?" And at the end of the day, that's really what programming is: engineering. We like to fancy ourselves as mathematicians, logicians and scientists, in search of universal truths about computation, but in practice, very few of us are actually doing that or even have any interest in doing that. Instead, we're solving specific real-world issues, where the correct way of doing things is informed greatly by said issues. Maybe so-and-so methodology is perfect for problem X, but turns out to be braindead stupid when applied to Y. Focus on the problem; if you're smart, the solution will follow. Fetishization of solutions is for losers.

>>40143 If it's dynamic data, rather than configs that are meant to be edited by hand by the user, then just use SQLite. It has a very small footprint while providing a ton of functionality, it has bindings for every popular programming language, there are a lot of open source tools for viewing and editing it, and everyone has it installed already. Your way may be good for small amateur programs, but the moment your requirements expand a bit - needing efficient lookup on more than one attribute, consistency guarantees, or crash resistance - you're going to start down the path of writing an ad-hoc, informally-specified, and bug-ridden implementation of half of an SQL database. >Also relational model is completely braindead and goes against common sense. This statement is so stupid, I can only imagine that you don't know the relational model and this is an excuse for not learning it (in programming, "common sense" usually means "what I'm used to" and nothing more). The relational model has been successful for a reason. Programming fads come and go - complex inheritance hierarchies, then Clean Code with tiny functions and lots of wrappers, now Rust's semantics that use object-oriented syntax but actually work more like Haskell typeclasses - but business data keeps being stored in SQL databases. Why? Because it's at amazing paradigm for storing real-world data, like users, products, songs, etc. In my experience, the relational model tends to require much less shoving of square pegs into round holes than the object-oriented model does.

>relational model is completely braindead and goes against common sense. this is an opinion that every programming noob has until they get a few years under their belt and come to the realization that the relational model is actually perfect and beautiful. in 500 years from now the relational model will still be where it is because it's impossible to do better. but besides, nothing stops you from using a document database if it triggers you so much. >>40171 >IBM IMS isn't dead after all wow. i had to google for this and was convinced you were bullshitting. actually insane.

>>40161 Even then, I suggest using SQLite as a JSON store if you cannot afford to use PostgreSQL. There are so many OS-level quirks when it comes to dealing with filesystem transactions. A DBMS (any) will abstract away a lot of the work and assuredly do it better than OP's half-assed attempts. SQLite includes a bunch of Jason processing functions.

>>40653 Hello, I would like my project to depend on 200,000 lines of additional C code, and a few dozen additional people I don't know, who are coding and packaging and maintaining said code, to help me avoid learning about my file-system's durability and atomicity guarantees when handling some simple key-value lists. If you really don't want to learn to design for your OS and file-system's limitations, at least use a simple minimal storage engine that protects you from file-system quirks in 4K LOC, like LMDB, instead of bringing in a whole extraneous paradigm with its storage formats and data types and query compilers and planners and extension APIs and all the bloat. They're all tools in the end, and deep knowledge of your problem space and the solutions those tools provide, is required to actually make the most sane decision. When in doubt, learn more, get at the fundamentals, and cut the dependencies out the dependencies that make it "easier". You'll thank yourself in a decade.

I've written enough OS code to know that SQLite is just one include and one link and will save thousands of lines of handling race condition. It's used by big players as much as by the smallest of projects and the development is being led by a single developer — the founder. >simple key-value Ok buddy, you might not need SQLite for your cron task or python script, but if it's already on your system (it is) then there's no cost for using it. The built-in JSON functions might save you a dependency on a dedicated JSON library. You can even invoke it from bash through the Tcl interface. I don't get why you have such a hangup and felt compelled to write this diatribe. >You'll thank yourself in a decade Get off your high horse.

>>40684 > being led by a single developer — the founder. Many projects are lead by one guy or another, doesn't make their code any less bloated. SQLite is admirable in that they only have a handful of trusted developers working on it at most. It is better in that way than most other SQL engines. > SQLite is just one include and one link and > will save thousands of lines of handling race > condition. FUD. In most practical use-cases, an op log file, some sync calls on files and directories, and some atomic rename() function calls and you have everything you need for everything but the most absurd edge-cases (consult your operating system or file system docs). What do you think most SQL databases really do--magic? They throw the operations asap into a write-ahead log (WAL), sync it, then take their sweet time compiling and optimizing and laying data out and compressing and etc. etc. If the database crashes, it reads its WAL up to where it wasn't corrupted, and gets back to a stable state. Heck, SQLite, not using WAL by default, makes writers and readers block each other to "save you from handling race conditions". If that is what you want, rename() a lock file instead, done. > >simple key-value > Ok buddy, you might not need SQLite > for your cron task or python script Ok buddy, you got it backwards. When you outgrow SQLite and PostgreSQL, whether because of throughput, or having weird cases that utterly fail with the relational system, you come back to raw "simple key-value" storage engines, and build your own tailored database engine. If all you have is some inconsequential python script or cron task then by all means, use SQLite, it is easier... Or, plain text files and the file system. "simple key-value" stores, like LMDB (memory mapped B+trees, recommended for beginners), or LevelDB / RockDB (Log-structured merge-trees), or whatever WiredTiger and InnoDB do (used by MongoDB and MariaDB respectively), drop you down the abstraction stack, so you can tailor your solution to yourself, at the expense of more effort and having to learn and think more. Learning and thinking seem to trigger an allergic reaction in today's "developers" for some reason. > save you a dependency on a dedicated JSON library Save yourself; don't use JSON. And if you had to, don't pull a database engine as a dependency because you can't handle some structured strings. Have to deal with a webshit HTTP JSON API in your C program? Write a small wrapper script to curl the API, pipe to jq to format it into a predictable line or field format, and then process it in your application code. Heck, include a JSON library in your application code if you had to. What happened to do one thing and do it well? Since when is bringing an SQL engine to process JSON a good thing? > but if it's already on your system (it is) > then there's no cost for using it The fallacy of free dependence is turning the world into a worse place. There's no "no cost" dependency. You will pay for it, whether you know it or not, now or later, you or someone who inherits your code. Only depend knowing the cost you will pay, and its real bounds (you likely don't). >> When in doubt, learn more, get at the fundamentals, >> and cut the dependencies out the dependencies that >> make it "easier". You'll thank yourself in a decade. > Get off your high horse. Hop on a higher horse, lain, and take a look. You can see farther, it is much more satisfying, peaceful, and even pays better; when every soydev is using the easy route, it really helps to be the "consultant" who knows what happens at the depths, and gets called in to write patches for RDBM systems to stop them from corrupting data, oh the ironies we've seen, lain. You'll truly understand it's all human mistakes, all the way down. -- Disclaimer: I'm not telling OP to not learn the relational model, or never use RDBM systems, or never use SQLite. Learn everything you can get your hands on, use things when it makes sense to use them, or even just if you're curious (as long as it is not a critical system). I'm telling off people who think others too dump to learn, and try to protect them "from themselves" by telling them to use the easy popular defaults. Using the file system works acceptably in most cases, and works very well if you know what you're doing; don't imprison yourself into ideology, especially ones telling you that you can never be as good as them. And once you learn enough, you'll know when to use what, and you'll appreciate the simplicity in simple tools, despite the "hardship" of using them. I don't claim to never use SQLite, heck, I wrote dozens of programs and scripts that call SQLite, I even used SQLite to handle CSV to avoid installing a CSV manipulation library. You'd think me a hypocrite if you don't understand my position. -- tl;dr: The fundamental relational model idea is approximately correct for almost all data needs, you don't need SQL bloat to use it, and you don't need SQL planners and optimizer if you're willing to slightly bend the model, conservatively. Most people use SQL servers to avoid learning more on information and systems theory, and to abstract away hardware and software peculiarities. Don't follow cargo-cult "rules" blindly, but don't break them blindly either. Learn the rules, learn why they're there, experiment with breaking them, and then you will know when to follow them or not. No random lain online can magically advise you on what is right. There's no substitute for personal learning and experience, and a good understanding of the problem you're trying to solve. Good luck lain.

>>40143 I think you're very wrong. A lot of DBs are very easy to use outside of copy pasting some db conn, redis, mongo for instance. If you want to be employed as a dev you should learn something like SQL and postgresql. The only boiler plate you really need is like your db conn, docker-compose/docker-network. Most of the time just using files/directories is fine though, and a lot of stuff doesn't need relational schema, just don't be afraid of databases, or really anything. You could learn how to set this all up in docker and deploy your app with a running DB in 2 days or less. To be fair though, it's pretty rare that something other than a website would need a DB and all it's features like transactions, locks, so on. >>40691 Incorporating bash, curl, jq, calling bash from your code, and then processing it as raw text is way more complicated than using a built in library to marshal and unmarshal json into a struct.

>>40684 >I've written enough OS code to know that SQLite is just one include and one link and will save thousands of lines of handling race condition. It really is super nice to know the chance of file corruption is negligible. I love SQLite.

It's not very hard to get where the OP is coming from, I agree unnecessarily bloated == retarded. There's always some balancing act to do between the UNIX way of "everything is a file" VS some data needs to be secure. to the OP: In principle think about how data is stored on a disk, databases just bundle up all the data more efficiently in a single big chunk than in scattered small files, this means that it's easier for reading. Data is just data, from a programming perspective you could just create a data structure to be fit for the use case, append things to it and save to a disk, essentially reinventing your own database. Again, the answer really depends on context, the nature of what you are trying to store and how often it gets read/written to it, etc.

How can I learn about databases when they're useful for businesses but not really for individuals? I mean I can be interested in compiler theory because I use a compiler, and I can be interested in OS theory because I have an OS, but I've never needed to use SQL before. So I'm asking, am I wrong and can databases be useful to individuals too? And can you recommend an idea for something interesting to make? In the employment thread someone mentioned about copying those websites that let you change filetypes or compress files that you upload, with AWS or some other cloud provider. I think I'll try that.

>>40814 Relational databases can definitely be useful to individuals too. Examples of projects using sqlite internally: TMSU, Calibre, Krita

>>40143 >Also relational model is completely braindead and goes against common sense. Bro hates Excel CSV-type files are not great for querying the data and it's simplier to learn a few SQL expressions than reinventing sorting algorithms or hacking together grep-sed-cut-sort. At some point, it starts taking more than a few seconds.

>>40776 >rare that something other than <x> would need a DB and all its features Databases provide tons of advantages. You can have multiple instances of your app running concurrently - assuming that they are aware of each other and that the workload is not too write-intensive. You can avoid a lot of corruption as most DB engines work on either an append-only principle, or write-ahead logging (WAL). What if the system crashed halfway during a rewrite of your store? Your users will thank you if they have to read from your data store without having to link to your stupid library. sqlite3 your/data/store.db https://www.sqlite.org/testing.html Bet you don't even have 100% branch test coverage.

>>40143 With my personal projects I don't ever need relational databases or anything premade. I store data in plain text files and dotfiles that follow a simple format I invent and then I write a parser for them.