How do you manage the leaked info you download?

I have about 500GB of leaked data that I downloaded in the last years from forums like raidforums, breached, xss, etc. in my PC. Until a couple months ago I was just using grep on all text files I have to make search through them, but the performance is terrible. I then created some indexed sqlite3 databases with some of the files and the performance for them is much better, but it still not good (takes too long to make a single query and uses a lot CPU), and I still use grep for the others. Obviously there are better ways to manage this. I was thinking in renting some aws ec2 to handle that, but I don't know much about it, neither about the best database for that. Do you have any tip for that? How do big services like haveibeenpwned, dehashed, pwndb etc. manage to search through billion of lines of credentials in like less than 3 seconds?

>big services you answered your own question

>>21321 Surely it would take only 15 minutes to write a script which broke down a plaintext file into several pieces, grepped each piece in a separate thread, and then combined the results?

>>21322 Of course they have money for that. I'm asking what kind of technology they use and how could I replicate that on a smaller scale for that purpose. >>21323 Yeah, I already did that, it helps a lot but it's not even close to the performance of a indexed SQLite database. But I don't know if there are better ways to do that.

>>21321 >grep You can try using ripgrep, which is a much faster version of grep.

>>21333 I tried it, but for some reason it was ridiculously slower than the native grep. I don't know if it has something to do with my processor or something, but it's totally unusable for me. But I had my friend use it and it was much faster than grep, although it was still slower than querying an indexed database.

Is SQLite really a good solution for performance especially with large datasets?

>>21336 I'm not sure, that's what I'm asking. I guess there are faster ways to do that, but I don't know much about databases.

>>21337 >but I don't know much about databases. Learn, will be faster get it done than the alternatives.

For anything text you should use a search index, for example recoll: https://www.wikiwand.com/en/Recoll

i used elasticsearch years ago for that, indexing (around 400m records at the time) took forever because bad cpu + hdd but querying was fast

use ugrep or ripgrep. Thank me later.

>>21321 This is just my experience, but when I converted a 2gb file from csv to text, it seemed to work better in pandas. I was kind of shocked but then again I was only looking for certain data and filter others, so that may have helped as well, but it was super fast, even faster than loading CSV in pandas. This was also on Linux btw.

>>21446 ripgrep is very nice

>>21378 Seconding this, elastic search sounds like the most appropriate tool for op's endeavours

https://lainchan.org/sec/res/21321.html#21378 >>21321 >>21378 >>2296 IntelTechniques had a podcast episode (Episode 295) on the topic. I believe they mentioned elastisearch. Magnet: magnet:?xt=urn:btih:3c21f5f3a8e4fe2319617c9df6548b02ecab20ac&dn=The%20Privacy%2c%20Security%20and%20OSINT%20Show%20-%20Ep%20001-305&tr=udp%3a%2f%2ftracker.openbittorrent.com%3a80&tr=udp%3a%2 %2ftracker.opentrackr.org%3a1337%2fannounce

>>22964 where do you search to find those podcast episodes? his hosting got nuked

>>22970 Download the torrent via the magnet link in >>22964.

>>22972 but what about the other episodes?