Best database for big data?

Hi lainons what best db for 100M+ rows?

i would imagine that any "serious" DBMS (read: MSSQL, PostgreSQL, Oracle) would be okay for it as long as you set it up well with indexes and stuff, but i'll admit i dont know a whole lot about databases

>>38945 fang dropped sql for their big db's cause sql couldn't keep up with massive update frequency they required. I think op has to think about his access pattern and then find a company with similar usecase to look at what they are using because the big corpo db's are tailored to specific usecases instead of a one fits all solution.

>killed a thread >stupid question >ai generated image glitterboy

I don't think the number of rows is that relevant to the choice of software. Afaik, all the usual solutions can easily handle that many rows. How many tables? How are they related? How much data in each row? How many requests do you need to make (in request/s or row/s)? How often is it updated? Does the database stays online or is it used for storage? Do you need fancy datatypes? Do you need SQL correctness?

>>38943 >randos on a niche imageboard collectively make a massive corporate IT decision Anyway why not try just json?

OP is back to thread >>38945 >>38948 thx >>38950 > How many tables? 1 table with all data > How are they related? not at all > How much data in each row? 12 columns not all of them may be filled in the row up to 27 rows >How many requests do you need to make (in request/s or row/s)? many >How often is it updated? Every day >Does the database stays online or is it used for storage? online >Do you need fancy datatypes? Do you need SQL correctness? No >>38953 >randos on a niche imageboard collectively make a massive corporate IT decision I'm not in corp i don't have a work lol

That's definitely overkill, but just use postgresql. No use thinking too hard about a non-relational, gigabyte-sized db.

>>38969 sqlite mmmmmmmmmmm lawdy trouble so hard mmmmmmmmmmmm lawdy trouble so hard

>>38969 Delta Lake, partition on column which is accessed in query most often

>>38943 >big data >100M rows I have CSV files on my laptop that are larger than that. That's nowhere near 'big data'. What matters at that scale is access patterns and desired redundancy.

>>39107 i have compressed csv files that have together around 1500mln rows. Software i am maintaining is a pipeline that ingest 15MB of data in an hour and takes 2hours to ingest 45GB of data (not only ingestion, most of work is actually spend on validating data). I guess thats what scalable software means >>39107 >What matters at that scale is access patterns and desired redundancy. Explain

>>38943 If you need something with very fast upserts for large data volumes and supporting host clusters, try Scylla. It's a Cassandra rewrite in C++. Still missing some features, like secondary indexes on multiple columns, but overall it's really good. My company uses it since 2019 for billions of operations every day, works like a charm.