DATA HOARDERS ANONYMOUS

What's your equiptment set up like? How safe is your data? How many petabytes do you have? What are you storing? (Judgement free) Where do you scrape/torrent/download from? Data hoarders unite!

I wanna know the route how to acquire plenty of used hdds. do I just start a recycling center (company)? I really love the idea of recycling and fixing old stuff and maybe preserving relic tech stuffs.

>>62468 If you start buying old laptops from local online listings you will amass a substantial number of 2.5" hdd and ssds. You can repair the computers you like (thinkpads) and strip the rest for parts. Don't buy old junk laptops from repair shops or ebay because they will have already been chopped. Before you know it you will acquire a large number of drives, however they won't come in large capacities, most likely anywhere from 128-512gb.

>What's your equiptment set up like? So far, 2 1TB HDDs, 1 20TB NAS and some smaller used HDDs >How safe is your data? Pretty safe I think, HDDs seem to be in good state, same as NAS and I sync everything to Mega >How many petabytes do you have? I think like 1TB >What are you storing? (Judgement free) I have been focusing a lot on personal stuff (pictures, videos, chats, documents) and so far this is my main focus. But since last month I've started to hoard music, movies, shows, random videos, etc. Mostly everything I like. However I guess it's going to take a lot of time to sort it out. >Where do you scrape/torrent/download from? yt-dlp, 1337x

how come the ladyboy porn torrents are scarce these days?

>>62470 yt-dlp looks useful. thanks. you sync 20tb to mega? is that expensive? >>62472 go to /t/ and you can't miss them

>>62473 >you sync 20tb to mega? is that expensive? No, only my really personal stuff. It's about 200/300GB so for now it's like 50 euros/year

I have 3 cheap(ish) Toshiba HDDs. 1TB for windows (this one is from my old PC), 2TB for Linux and 2TB that has 2 partitions for data (split between Linux and windows). I use the ext4 file system because it just works and I don't trust btrfs (and I don't need ZFS). For windows, I obviously use NTFS. I also have 1 external HDD for backups. I think SSDs are waste of money for most uses (they can be useful if you have an old computer but for 90% of use-cases HDDs are better, imo). If I had a SSD, I would use the F2FS file system on it. The software I use: * SquashFS (Great for backups) * rsync (Great for backups and copying a lot of files, faster than using cp) * gdown * youtube-dl/yt-dlp + ffmpeg * gallery-dl * aria2c * Transmission BitTorrent client * tar + GnuPG with symmetric key Also, Xz format inadequate for long-term archiving: >https://www.nongnu.org/lzip/xz_inadequate.html tl;dr use bzip2, gzip and lzip. instead.

>>62475 >https://www.nongnu.org/lzip/xz_inadequate.html this goes way over my head. can you explain to a sub room temperature IQ person what the difference between bzip2, gzip, and Izip are?

>>62476 Don't worry. He doesn't understand it either.

>>62476 >>62487 >https://www.nongnu.org/lzip/xz_inadequate.html#unprot_len >and §2.10 >https://www.nongnu.org/lzip/xz_inadequate.html#misguided Xz is less tolerant for errors than other formats.

>>62475 You are living in a fantasy land if you think there is any reason to have an OS on a HDD in 2022 other than if all you can afford is a basic computer and don't even have the money to buy a 128 gig drive to replace it with They are more power efficient, much faster, less physically fragile, much more robust than many people give them credit for and increasingly approaching price parity with HDDs HDDs exist as legacy equipment, budget choice and niche use case products now, the future is solid state.

>>62475 Look into rclone if you put anything on S3, B2, etc.

>>62469 I second this, I salvaged allot of my drives out of last-gen or last-last-gen consoles as well. >>62495 >approaching price parity You can practically get HDDs for free nowadays so I have no idea what you mean by "price parity". >more robust SSDs die from being written to too much, for anyone who writes allot they're a non-starter. >less physically fragile Irrelevant, do you really handle your drives that often? >much faster If you're real anal about boot times then maybe, otherwise look at RAM caching. >power efficient Borderline irrelevant, not having your NAS running 24/7 would save allot more power. Unless you're running a random read/write intensive database then SSDs are either unneeded or better served as cache devices.

>>62495 I agree with this. There are too many benefits to running your distro on an SSD. If I could afford to store all my data on SSDs I would. When it comes to data hoarding though, it's hard to beat a 20tb 3.5" hdd. An SSD in that capacity is unaffordable. Even a 2.5" 4TB SSD is around $400 while a 5TB western digital hdd is only around $100. So while the tech may be antiquated, it isn't affordable to abandon HDDs. I've tried quite a few AliExpress solid states and can say they aren't adequately reliable as of yet.

>>62460 Poweredge R720, with some kernel tweaks to harden my debian install 14 x 2,5" 1TB refurb drives I got $25 a pop on amazon, set up in RAID-Z2 = 12TB of usable space. The guy I bought the server from on Ebay threw in about 1,8TB in extra drives I use in the remaining bays as a tertiary backup store for critical files. I also have an old netbook I am setting up with a USB 2 slot bay to put 1.5 TB of HDDs I stripped from my dad's old Cable box, this will be my primary backup. >How safe is your data? Drives are in good condition, no errors so far. Since they are so cheap I figure why not just get refurbs and buy a new one when one inevitably breaks. Since they are refurbs I know they arent from the same lot, and dont have to worry about simultaneous fails. All drives and zpools are Encrypted, all exposed ports are monitored by fail2ban, router running OpenWRT, looking into setting up a data nuke dead-man script. >How many petabytes do you have? Not counting backups, I think about 4.5TB so far >What are you storing? (Judgement free) A million little dockerized services I run for my friends (matrix, jellyfin, mumble, etc.) I use the *arr family and ytdl to download every bit of music, tv, and movies I could possibly need. I also have a nextcloud instance for my friends to share stuff with each other, and for me to archive family photos, important files, rare frogs, books, etc. >Where do you scrape/torrent/download from? z-lib for books, prowlarr to proxy my torrents, and youtube-dl for music I cant find on a torrent.

Hey, hey datahoarders. I've got a question: How do you find something you know you've downloaded? You have an organization system, right?

>>62732 Ugh, what? How do you NOT find something that you know you've downloaded? Bro, your files must be a mess. > You have an organization system, right? Yeah. It's called folders.

>>62460 >What's your equiptment set up like? 2 servers with about 50TB each >How safe is your data? Half of it is backed up in a google drive, the other half is in God's hands >What are you storing? (Judgement free) Movies, vidya, and music mostly >Where do you scrape/torrent/download from? Mostly private trackers and blu rays I get from the library

>What's your equipment set up like? I throw HDDs I remove from external drives into a tower case which I use as a server. I only buy new drives when they are on sale and I think I'll need the space before the next sale. The best time to buy is usually when burgerland has black friday. >How many petabytes do you have? I have a total of about 28TiB usable, ignoring backups and parity etc, using about 17TiB right now. I currently have a lot more than I need because I found a few drives I thought were lost and threw them in as well. >What are you storing? It's a mixture of various shows/anime, games I'll never ever play, ancient backups of machines I'll never need, several terabytes of mostly garbage manga which I only keep because the download link I got them from is dead, audio including podcasts and music, pr0n in Hydrus, and every YouTube channel I'm half interested in. >How safe is your data? I have one off-site backup that is updated weekly and is kept offline when not being updated. I back up my most important data daily into an encrypted folder on the cloud provider that my employer provides via rclone. >Where do you scrape/torrent/download from? Anywhere that has whatever I want. I prefer torrent sites, and am in a few private trackers but so far there's not much I couldn't already find on open trackers. Archive.org has a surprising amount of piracy going on in it, especially for older stuff. While on the topic of scraping, let me say fuarrrk sankaku complex. I just realized that one drive is 99% full, fuarrrk. Now I've gotta troubleshoot why my 7.2TiB drive is full despite only using 4.8TiB. Probably btrfs bullshit.

My ssd of two years fell off the top of my tower, now it's fuarrrked. >can't create new folders >can't copy files >can't delete anything off the drive Any online shops that have decent prices for 2-4TB SSDs? I mostly used this one for hoarding static webpages and video games, but recently it's become my all-purpose drive. Now it's totally fuarrrked, it's gonna be such a pain in the ass to copy all of this stuff onto another drive.

>>62785 I tested moving folders to another drive and it's giving me bullshit source path too long messages left and right. Let me rephrase my previous question. >Any online shops that have decent prices for strong rope?

>>62745 >Bro, your files must be a mess. I have no problems, however: cr@ptop: $ ls ~/ | wc -l 346 but it was more a reference to data hoarder communities that are filled with people who just download and don't categorize them. hence the term 'hoarder' as opposed to 'archivist'.

>>62784 >Archive.org has a surprising amount of piracy going on in it I noticed this more and more. I can often just look up a film name in a search engine and find it on there

>>62460 Maybe you Anons can help me out. So I (accidentally) plugged in a 20v laptop DC into my external 14tb drive and yeah... Seems like the SATA PCB on the drive is fried. Does anybody know where to get datasheets or maybe even a replacement PCB? I have already looked on ali and ebay to no avail. The drive is a WD140EDFZ

>>63121 All my Books, Cartoons and Muzak that I spent weeks, months accumulating and sorting were on there. I lost a part of myself.

>>63121 Send it to a professional data recovery service and hope they can pull it Also, implement a redundant storage strategy for the future

>>63134 >redundant stoarage strategy Okay I'll admit, I had two mirrorred drives and plugged it into both. Must have been drunk or something. Still I curse those expensivs hard drives using those cheapass cases that don't have a buck regulator or some other protection. I mean at least don't use a fuarrrking standard DC plug that is used in almost any 20v laptop adapter. I should've shucked them and will do that for any external drive I buy in the future. >data center That is not an option for me. And what would they do differently from replacing the board? I have soldering skillz, and could do that by myself.

>>63134 Maybe there's only a fuse blown. Hence I'm asking for a datasheet.

Now that I've got a little bit of spare cash, I plan on starting to accumulate an archive of shows and such. Is a Western Digital 1TB external HDD a good place to start? If not, what would be a good HDD/SSD to start with?

>>62460 16 google drive accounts

>>63609 I just buy 4TB externals for like $60 - $80. Wait for "cybermonday", the deals get absurdly good.

>>62786 dd if=/dev/myfuckedupdrive of=/dev/gooddrivewithnothingimportantonit ur welcum

>>63649 Just wait until dd crashes halfway through and takes your hopes of data recovery with it

>>63609 Buy whatever you can but 1tb isn't a lot. I would get something that has at least 4tb or 8tb if you can spend a bit more.

I want to mirror all of library genesis. I have archival-grade DVD's which I shall use for backup, but I also want to run a live mirror for the students in my area (I live in a slightly isolated but populous region). I also intend to 'clean up' the database a little -- I will only host .djvu files, since I would rather not expose users to extremely simple tracking attacks. 1) How should I set up a public-facing libgen mirror? Any important things to keep in mind? I will certainly be running the storage backend from my home PC, although I may try using my university-provided internet connection for the backend. 2) What is the lowest-power solution that I could reasonably find for pretty cheap to host the files? I am but a meager grad student, while I have saved a lot of money I can't afford a huge uptick in our electricity bill. 3) How to suss out and take advantage of bulk sales when you live in the middle of fuarrrkin nowhere and don't have a car? 4) How to get normals to use djview? Should I include a big banner at the top of the search bar page?

>>63751 >I may try using my university-provided internet connection for the backend. I meant frontend in this clause. Although now I think about it this is a terrible idea because it is way more likely to end with me getting kicked out of school

Is there something to automatically sort music folders and scrape metadata?

>>63609 Depends on how many shows you're wanting to hoard, what kind they are, and what quality you're fine with. As for the size of the drive, 8TB was the sweet spot value-wise for a while but it's been getting bigger recently. The last drives I bought were 14TB because they were a better deal than 8TB. Black Friday tends to be the best time to buy, aim for $15/TB or less. Live action shows take up more space than anime for hopefully obvious reasons. Unless you have autism about archiving the best version possible, you can get well encoded shows that are under a GB per episode (especially anime). I've personally compared a 2GB AVC/FLAC and 300MB HEVC/opus encode of the same anime episode and couldn't tell the difference at normal viewing distance with headphones on. Modern codecs are pretty damn efficient if done correctly.

I'm a retard who writes his scrapers in bash. Is there any way to persist wget/curl connections between invocations? Or restructure them to minimize time wasted on establishing connections? An example scraper for an ornery site: bash while read url; do slug=`sed -E 's/.*\/([a-zA-Z0-9\-]+).*/\1/' <<<"$url"` [ ! -e cache/$slug.html ] && wget "$url" -O cache/$slug.html id=`pup -p 'some query' < cache/$slug.html | head -n1` [ ! -e json/$id.json ] && wget \ $basename"/?id=$id&internal=protocol%20query" \ -O json/$id.json \ && ./process.sh cache/$slug.html staging/$id.json done < pages.txt

>>63815 I know that wget can persist connections but I don't remember exactly how to make it happen. Pretty sure it would have to be downloading whatever multiple files in the same invocation, not separately like that snippet does. Using the -i option with multiple urls should do it, but I don't think you can use -O with -i. That may or may not be a problem depending on the site and/or script.

Am I fool for considering switching from mega to wasabi for cloud storage? Or just a fool for having used mega in the first place? I'll miss the convenience of the app but their cli tools are killing me. >>63821 The -O is because I'm downloading index pages so I need to name them after their directory/slug to differentiate them. If it's possible to do this automatically, that could potentially help.

A 500gb SATA with music, ripped from youtube and dvds. As well as some local copies of archives and websites. (rantmedia.ca, gentoomen library, !!lain!!) Some day I plan to install truenas.

>>63828 Be sure to review the minimum monthly storage charge and minimum storage duration policy. You will be charged for a minimum 1TB/mo of active storage regardless of how much you actually use, and you will be charged the cost of keeping each item there for a minimum of 90 days regardless of when you actually delete it. https://wasabi.com/paygo-pricing-faq/ Backblaze is $0.0009 per gig-month cheaper on storage but a cent per gig for egress. Also check out the "Personal backup" product in the top nav if you're just going to mirror the contents of a PC instead of actually using object storage for something, $7/mo unlimited. https://www.backblaze.com/b2/cloud-storage-pricing.html

>>63838 >wasabi Yeah, reading about the caveats has made me change considerations from s3-alikes to using a hetzner box. If it's part of a 3-2-1 scheme with file integrity checks, you don't really need all those 9s and the savings are comfy. I'm cautious about egress fees because I actually do want to pull large amounts of data out every now and again. >Personal backup product My understanding is that this is gimped? You need to use their software, restores are painful, and old data gets silently deleted.

Hello, hijacking this thread to ask a noob question. What would be the best way on linux to span data across multiple drives/combine multiple drives into one volume? I don't care about drive performance, it's primarily an archiving server. I also don't care about drive failures, since I have backups. I have a single 2 TiB drive currently which is almost full, and due to some circumstances I managed to acquire another 2 TiB drive for free. The 3 ways I know of doing this are using RAID 0, which might not be the best solution because of drive failures and the fact I don't really need the performance. I think btrfs, which I already use, can also do it, although I've heard that it's still not as reliable as other tried and true methods. I know of LVM, although I don't know anything about it. What would be the best way to combine these two drives?

>>63847 If it's just for lookup, the simplest solution would be a union mount. A slightly more involved but "dumber" solution would be to track files with git-annex. Always be wary of clever solutions; there are plenty of ways backups can be rendered useless if you're not careful. >btrfs I'm leery of it myself, but the last time I used it was too many years ago. Things change. My understanding is that if you're not using certain features that are under development (usable without warning messages!) it's stable enough. >LVM It's been pretty reliable for me, and it's not so difficult to understand once you get used to it. That said, it's roughly equivalent to software RAID 0 but with better failure modes IIUC. Speaking of btrfs, does lain use https://bcachefs.org/? It looks very promising.

>>63847 Buy another drive and use ZFS

>>63847 I use mergerfs to make multiple drives appear as one, it just werks.

>>63779 Look at the *arr family. I think the music one is Lidarr.

>>62500 >If you're real anal about boot times then maybe, otherwise look at RAM caching. I was using my sisters computer and thought the hard drive was malfunctioning due how slow it was, i bought a new one just to realize thats the expected performance. >SSDs die from being written to too much, for anyone who writes allot they're a non-starter. modern ssd's can take a huge amount of writes, most use cases they die before they can even reach their max. >Borderline irrelevant, not having your NAS running 24/7 would save allot more power. Maybe it doesn't matter for a NAS server that is not used much, but its crucial for extending the battery life of a laptop.

>>64811 >I was using my sisters computer... Right, pretty sure you're already missing the point. If we're talking about a single drive running some run-of-the-mill desktop OS and a web browser then yes; some cheap SSD is probably going to be the easiest option as opposed to making Windows properly cache files or using ZFS or some other FS with sensible caching. It also doesn't help if the workload you're running also uses a lot of ram or your files are too large. With that In mind however, most modern computers have 8-ish GiBs of ram which is, atleast in my experience, enough for running Ubuntu or whatever with ZFS "snappily". >modern ssd's can take a huge amount of writes. Define "huge". getting back to the topic of data hoarding; my current dataset is mainly video, large files, and I don't re-watch them so I delete as much as I save, hot and/or volatile. I ran this dataset on a small 128-ish GiB drive and it wore down to 10% in a little under a year, spanning that data across multiple drives would no doubt soften the load but I'm not running a data centre and so I don't need need to be certain of when I need to replace drives if I can not replace them in the first place for a bit (or in this case a lot) longer. add to all that the fact disks are big and cheap (if not free) and SSDs make very little sense to me and anyone else with big files and hot data (which I'd argue is most people who are hoarding data) >its crucial for extending the battery life of a laptop. And back off topic to laptops... Do you think I think thin-and-lights or gaming laptops should use disks? no, probably not, at least for the workloads that most people use those things for. however if you just treat your laptop as a desktop with a built-in UPS like quite a few people I've met then a loss in battery life isn't all that big a deal. does that mean you should use a disk? maybe, but that depends on what you're running and how you're running it.

>>62732 I have thousands of folders and 2 general sorting systems. I can't find anything ever.

>>62468 Get refurb drives from amazon with 5 year warranty and make sure you take into account the bathtub curves drives have

>>63779 i use beets quite a lot: https://beets.io/ it's slow but it manages my 14,000 song library quite well. it gets its metadata from musicbrainz, occasionally you can use acoustid to find metadata for file that have none automatically

>>64819 >I ran this dataset on a small 128-ish GiB drive and it wore down to 10% in a little under a year On a modern ssd that means you must have written the entire drive full of data every 4 days, and you could do that for 8 years before you going over what is has been rated for, given that manufacturers underestimate how much bytes can be written to their products it could easily give you twice as many writes. That is what you are getting for a ~25 dollar budget ssd drive. Your average budget hdd isn't going to last any longer, you just trade more storage in for less performance. If frequently storing and removing large data files is your use case why even bother with a 128gb drive when you can get a 500gb drive for only twice the price, with that drive you would only have written 2.5% of the TBW in under a year. If you don't write more than 25% of your drives capacity in a single day they last just as long as a hard drive would, removing 25% of your data every day doesn't even fall under the use case of data hoarding by its definition. I am not sure why you are even complaining when that the drive functioned perfectly fine for your use case, would have lasted you 10 years easily, how long would you expect it to last, 20 years? Consumer drives have never been expected to last that long. >add to all that the fact disks are big and cheap (if not free) and SSDs make very little sense to me and anyone else with big files and hot data (which I'd argue is most people who are hoarding data) No one is making the argument that hdd's don't store more data for the same price, i wasn't anyway. Point is they barely beat ssd's when it comes to longevity, and in terms of reliability ssd's usually do better.

10tb drives are only 100 bucks now, got like 50tb of random anime and tv shows ill never watch.

Best Buy started Black Friday early, with sales for external drives up now. >$200 for 14TB ($14.29/TB) >$280 for 18TB ($15.56/TB) >$330 for 20TB ($16.50/TB) They may get slightly cheaper closer to the actual Black Friday but I wouldn't expect more than $10 to $20 cheaper if at all.