Small Scale Data Hording

You see a lot about data hoarding on a large scale: TB of data all stored on a home server for the user to access from any device in their house, but what about small scale data hoarding? I'm talking the manual saving of videos, pictures, pdfs, text files, or anything else. I currently have a little over 8 GB of images, videos, random programming pdfs, etc. When I find something I think is cool I save it and that's that. Does anyone else do this? Are there any tools/programs you would suggest to others to make this job easier? What ways do you find cool data (images, gifs, whatever) to horde? Just thought this might make for a good discussion topic.

>software malware-free premium activators, spyware cleaners, bundles, those self-hosted aren't likely to last. others are software that aren't likely to exist at some point. one example is the symbian OS source code which got open sourced until it got bought by some company. I'm still looking for the source code though. >imageboards some imageboards don't have archives, use MHT and DTA, and screenshots. I usually collect posts that are informative (like lainons speaking from experience, disclosing something, anything screencap-worthy). It seems most anons today don't do this and just repost same decade old soykaf. never use archive.tlds since they're are likely to not last after 3-4 years based from experience, archive.org also gets copyright claims or similar like with ebooks and music claims recently. >imagebooru I usually have a thing for "vent art" and low quality art from new artists. Some I find on /ic/ or sometimes on status:deleted tag. I usually am not a fan of 4chan though as the site itself strips metadata (catering normal people who can't strip exif and end up self-dox) which destroys booru images unique color profiles and original metadata. 4chan also does some kind of image compression which disrespects the spirit of "image board" and you'll notice when the checksum mismatches. Either it's OC or it's not worth it with the destroyed color profile and you must get the direct source. I also take official art from websites, sometimes even anime mascots of altcoins. Usually if you feel the file won't be hosted in that site in a few years, just download it or make archive.org snapshots. >ebooks zines and rare ebooks that are hard to find, or no longer seeded. fonts are also very lightweight to collect. >videos another issue with platform censorship and copyright strikes.

/r/DataHoarders has some potentially useful information. Hydrus might be a useful tool images inb4 imp­lying a soykaf website can't have good content

Small scale to me is like under 10TB, I have probably like 8TB of content hoarded right now. I think the most interesting content I like to hoard is pixiv galleries of random comics/free doujins since a lot of people sleep on that, they usually don't bother reuploading to e-hentai. Also galleries usually get deleted because artist tism. If you like to hoard images gallery-dl and Download All Images for Firefox combined is your best friend. >>35007 Both archive sites are hot soykaf, .tlds IP bans you for archiving too many links and archive.org bans you for uploading anything of actual interest that isn't a random YT video. >>35012 I've always wanted to use Hydrus but I have such an autistic way of setting up folders, I don't think I could tag each and every picture.

>>35027 I'm interested, do you save this content by had, or do you have some sort of automated way of doing it (e.g. a web crawler). I'm new to this whole thing and I can't even begin to imagine how someone accumulates that much data.

Yes but because of my current situation, i just can't afford new offline storage space and i just have 18 Mb left on the HDD where i hoard stuff (i only have this one available, it's a 500gb one; also i have another one but it's already filled with stuff that i can't delete at all, backups mostly) >Software i'm on a ubuntu based OS, so, i slowly started just giving up on hoarding windows installers (i hoard games installers of a lot of indie games i'm interested into, but nothing else, i only use windows for playing games) >ebooks recently started hoarding Lainzines and programming ebooks of languages i'm interested into learning one day, also other λ related stuff like guides for bash >videos here's one of the things i want to hoard the most, animes in high quality, but can't download more because of my lack of free space :( >music i recently started getting interested into getting high quality audio files (FLAC), since i wanted to take listening to music to the next level... i'm quickly filling the 64Gb micro SD card of my phone with this, and having to hook it up to my pc in order to listen to music :( also, some time ago i downloaded a huge collection of old eurobeat from torrent (just one seeder, i downloaded one file of 50Gb from that guy, @ 1/8th a megabit per second, so, thanks kind seeder for the patience) >other this is a mix of both ebooks and video files,wich are actually courses that i downloaded from torrent i also hoard iso images of different SO's, mostly i'm interested into mini images of Ubuntu, and also alternative OSs/distros like tinyCore, tomsrtbt, Ubuntu MATE i'm starting to consider maybe buying dvds to start storing thing, wich is the only realistic way i see of keep downloading stuff (dvds are expensive, like any other storage media right here, but are the most cost effective per Gb with the disadvantage of not being able to replace data)

I manually save good blog posts, forum threads, articles, guides, etc using wget. Plain HTML documents barely take any space and I like having local copies of content I often revisit.

Sometimes I go to /t/ on 4chan and look for good torrents. I've got lots of folders containing ebooks that I got from there. You can find a lot of torrents on there related to topics ranging from engineering to programming.

Bump, I've also recently been interested in this

>>35051 And you never know when it might disappear from the web forever.

>>35051 Is there a good way to parse out the actual content from html documents? You rarely need to actually keep the embedded/dynamic content intact, but if you just save a page in the naive way you often end up a horribly formatted almost unreadable mess. Would be nice to parse it to markdown or something so that basic links and images work but all the other crap gets stripped out. I guess you could probably do this with pandoc and some preprocessing.

I think I'm going to start archiving my favorite youtube channels' backlogs with youtube-dl. I follow a few channels who upload documentary style videos on internet related topics and I'd really hate for those to go down one day.

I'm poor in a poor country, my whole HD is 465gb, so here's what I "hoard": >Music 50gb, all high quality (320kbs+), most lossless. >Videos 23 gb of live shows, interviews, Serial Experiments Lain, two series and Music Videos. >Images 1gb of Alunya pics, Lain pics and anime reactions. >Literature 2.81 gb, mangas, books, comics and some plain text guides. >Chords/Tabs 100kbs, all plain text copy and pasted and a few made by me.

>>34978 I used to hoard basically every picture that I liked (basically just cute anime art, or anything that looked cool or aesthetically pleasing). It became a huge hassle though, and it felt really stressful to have to download and find the correct folder for every picture I would find. It just made the experience of looking at nice art online much less pleasant. Nowadays I never really download any pictures at all, I just save what I can on by liking the picture (eg. on Twitter or Pixiv for are). It means all I have to do is click a single button and then that picture is saved in a list of liked pictures. The problem is of course, that some artists, especially smaller ones or ones that draw more "risky" stuff like loli pictures often end up deleting all of their pictures and their accounts, which can be a real shame. As for movies/TV/anime, I just stream video, and as for music, I'm really satisfied with Spotify, despite the lower quality and the fact that songs sporadically disappear due to licensing problems

I go to my local library and get a bunch of blu-rays and rip them. So far my library is ~20TB, which doesn't include another 40 TB of vidya **I think I have a problem**

>>35064 You can save webpages displayed with firefox's builtin tool : reading mode.

>>35064 https://github.com/eafer/rdrview

>>35084 I tried to do this with rare (region exclusive) 90-00s DVDs but children had scratched them to hell.

>>35202 I don't understand what it is with kids and niggers that make them scratch the soykaf out of discs. Blu-rays are a little more resilient but every once in awhile I get one from the library that's unreadable.

>music I have all my CDs and LPs also in digital form. I dislike streaming services but want to be able to listen to something easily >guitar tabs and other notes Never going to learn all of it, but having it at your fingertip is nice >website articles when I see an interesting article I also often download it, because I fear the website going down and not finding it again >pdfs I read research papers for work and I save them of course, but everything else I find (be it books, the lainzine, or whatever) I also save. I haven't read everything, but when I'm bored I just read some of the stuff >videos When I watch something interesting or see something that I can't watch, I save it for later on my disk >

>>34978 Right now I have over 100Gb of ebooks on my computer. They are all related to CS, EE and Mathematics.

>>36416 nice. I'm still at about 65 gigs, mostly math. I have a bunch of blu ray disks that I'm planning to transfer it to, along with my music collection.

I don't know if its considerened small but I have approximately 1.5TB in data hording. >Music Almost all my music is in FLAC >Movies >TV Shows > Anime >Porn

I have somewhere around 2GB of pdfs. The first site on my browser is libgen. I've been growing a collection of good books that I expect to read. I don't have something like the gentooman library full of soykafty/incomplete scans of just any book that has the syllable 'py' in it. Actually I have very few programming books. I have a few math books too. I also have books on history, botany, several languages including latin (which is a growing collection, recently I got a couple on botanical latin), physics, building, origami, and a few other topics. It's not a huge collection. I only keep books that I believe I am going to want to read and which meet some quality standard. Whenever I think of a topic I want to know more about (say, electronics) I look up books on the topic. Make several queries to try to cover a good range of possible finds, and then start selecting those which look good, download them, and again filter. I then store them in my collection, and start reading one of them. Those books always come with pointers to other books which I then find, or raise my interest on some other related topic. It's not a big collection, but it's one I feel somewhat proud to have and if I end up not having internet for an undefined time I can be sure I'll have plenty to learn not to need internet.

i just go on google images and save every image that comes up under random keywords. Whatever I’m feeling that day.

>>34978 I just use rsync to sync my documents and media from my active computers, a desktop and a laptop, to my measly 500 GB home server. It would probably run out of juice soon and I would need to top it with a terabyte or two for that ZFS mirror magic.

For me personally, it makes sense to put the limit on physical space rather than data (yes I know data is technically physical space not the point). As in, buy a 1U server and fill it with whatever drives you need over time. Out of space somehow? Just upgrade to bigger disks in the 1U. Or cheat and get another 1U JBOD. >tfw day dreaming of having money to build a server

>>35064 wget --page-requisites --convert-links -e robots=off Should download everything you need to display website properly with links fixed to point to local files instead.

>>35042 >i recently started getting interested into getting high quality audio files (FLAC), since i wanted to take listening to music to the next level FLAC is for archiving music. It offers no advantage to a listener over a well made mp3

>>36898 Spending money on extra storage for storing FLAC is more cost-effective than spending it on extra processing power and storage for transcoding mp3s for listening.

AEL here. Started small scale now I have a 4TB drive full of ebooks and other drive for other thigs. So here is my point of view. 1. Start thinking in the order of folders early on. You may start small scale but once you save data it can grow suddendly and is your best interest for it to grow orderly. Trying to order after you have stuff is hard, I can say I should have started making the hierarchical folder structure early on. 2. Get the best deals in drives according to your local buy stores. Sometimes there is a sweet spot for drives of certain amount, and by that I mean you must do basic math to know the best ratio between MONEY PER MEGABYTE. Dont just go for the cheapest or an occasional deal 3. Use the best software, and SAVE A COPY of that software. We know what happened to youtube-dl, but even if is back in github you should be supicious. You should also know and have a backup of other download managers like JDownloader, wayback-machine-downloader, rclone, etc. Also learn to use wget, is your friend. 4. Scared something could disappear from the Internet? You should. I've seen webpages go 404 and getting the hammer in the Wayback Machine (many dont even make it there), videos being taken down from Youtube, censored, or getting locked by region. Do you think there will always be websites that will have the backup? No, I've seen libgen taking down books, Mega, Vk, the hammer never stops. Even Twitch has a big DMCA problem right now, if you are into that kind of content you should have started like a month ago. I regret not saving many threads from the chans. 5. The best backup is peer-to-peer. The All-Embracing Library is a good example. Is good to know I have a backup out there living in the hands of others in a torrent, and I can go back to it in the case of a disaster. Just dont underestimate the situation, dont underestimate the future. The things we lost, they will only live in memory.

>>37072 > Start thinking in the order of folders early on It won't really work. However you try to arrange your files it will always end up in a mess. Folders simply do not scale. What I would recommend is to use some tag system, with which you can run simple queries to find data on specific topics. I use it for taking notes and it works wonderfully. If someone can recommend similar software for files, I would be really thankful! > the best deals in drives In my country there are confiscates — shops in which the government sells other people's goods stolen from them on the border. While you shouldn't support such activities, prices there are really low. > The things we lost Yeah, I would also recommend storing the most important things in the cloud. Sadly there were cases when people lost everything they have because of fire or similar circumstances.

>>37074 >tag system Ah yes, depending on a program that will disappear one day and then you have to remake the tags

>>37090 Well, I started to use TMSU[1] which stores all tags in a single sqlite database, which means that all I need to do is to backup one single file. Being extra cautious I might as well use Nix to ensure that I can always build the program itself. 1: https://tmsu.org/

>>37090 Not necessarily, if the program is well thought-out, reimplementing it could be trivial. The non-FUSE parts of TMSU aren't particularly complicated and don't require unusual knowledge. Of course, the program disappearing doesn't matter if it's FOSS and not broken. If it isn't broken, you can just keep using it as you need.

>>37074. >It won't really work. However you try to arrange your files it will always end up in a mess. Folders simply do not scale. I have terabytes upon terabytes that say otherwise about a well ordered file structure. But to each their own, as long as you START EARLY. That is the point >Yeah, I would also recommend storing the most important things in the cloud. Sadly there were cases when people lost everything they have because of fire or similar circumstances. Careful, some services parse for blacklisted material. Maybe encrypt the data, but even then is a terrible option for private files. And if is not about personal files then you get the problem of blaclisted files. >>37097 The database format is half the work, what you need next is to ensure the program to peek at the files is not prone to disappear. >>37117 This solves the above. A common database format with a program easy to reimplement and FOSS is a good way of not losing your stuff one day. Overall just be mindful of your tools.

>>37137 > I have terabytes upon terabytes that say otherwise about a well ordered file structure But you still can't really run quires on them to find specific files, while all I need is to write.. tmsu files "image and cirno and happy and not horny" ..and then I can combine it with other useful applications.. tmsu files "image and cirno and happy and not horny" | sxiv -t - ..or even.. telegram-desktop -sendpath $(readlink -f -- $(tmsu files "image and cirno and happy and not horny" | sxiv -tio -- -)) ..to select images from a specific query and send them to somebody in telegram.

>>37143 I see your point with pictures. In turn I run searches on ebooks with the find command and has the same effect, but they have adequate filenames and they are not pictures. But anyway, dont you have at least some order of folders as an aide? I figure tags have limitations if you dont have the specific tag in mind or in the tip of the tongue. Having a folder by categories also help to re-discover related pics, at least for me. Might explore the ideas of tags tho, I have in mind this is one way to make use metadata in pictures as a way for a conmon format for tags.

>>37163 No, no structure at all: you can always replicate it with a couple of tags. And TMSU includes user space fs, which will build this structure for you. There are three problems I have with this however: First, it is way easier to just copy a file to a folder, while I need to use the console. So it is somewhat more tedious. Second, I partially use KDE, and its default file browser, which opens when I select or download files, doesn't let me insert a full path to a file. So I'm forced to use FUSE in the end anyway. And third, the explosion of tags. Now I have hundreds of them: for scientific papers, websites, images, videos, books—only music, manga and anime are stored the old way. So searching for tags I have is quite unwieldy now: tmsu files "suiseiseki and angry"

I am thinking of getting a server and hoarding all my soykaf there.

>>37186 Buy one in Russia and download torrents with impunity.

>>34978 This has always been the way that I've interacted with the internet- save things I like, sort them into folders. Certain kinds of things are on different drives- boot drive reserved for OS and a few programs, largest drive is for games, software, music, documents, things I expect to be using one at a time. Another drive for images and videos. Then another drive for backing stuff up. It has gotten fairly unwieldy now that I have a number of years of junk hoarded and am still just using folders, but I'm not sure I care enough to manually tag (at the moment). I am planning on splitting my computer into two builds once the new AMD processors are more available. When I migrate to the new system, I'll set my old one up as a personal server and properly maintain my files then. I am also planning on transitioning to Linux and just remoting into the old computer as needed. If anyone can suggest some software for automatically editing the folder structure, filenames, and metadata of audio files I would appreciate that. I have an autistically specific way that I like to organize my music, and it's getting fairly annoying having a collection that's largely FLAC while wanting to have a smaller archive of music on my smartphone in MP3 with song lyrics in the metadata where applicable.

>>37195 > if anyone can suggest some software for automatically editing the folder structure, filenames, and metadata of audio files I would appreciate that Those things are usually too specific for special software to exists. What you want is to use bash or any other shell and write scripts you are interested in. That's how I do things anyway.

>>37196 Sounds like a reasonable solution for part of what I'd like to accomplish. I'd like to automate the identification of song/album titles and the online retrieval of any data I'd like to include in the tags. Main problem here is cases that would require specific solutions in order to work without human intervention. Ex: there are dumb things like how song titles in German, Icelandic etc have umlauts, accents or runes that often get renamed to only English alphabet characters. People can easily tell, for example, that "Kvadning" is just a convenient way to type "Kvaðning." Should be manageable to have a program deal with that, but I think you can see how the number of possible unique situations like this would make it a bit of a heavy task to ask of a bash script. If some proper software for this existed, I'm guessing a few lainons here might be interested in it. Might be a useful thing to try to develop, atm I'm not much of a progammer though...

>>37183 The problems you mention sound specific to the program. There was this password program that had some guy rewrote on perl to use its database but with tweaks. Maybe you can make a script to solve a few of your problems. For example, I would look on the format of tmsu database and see if I can modify it directly, then i would see if I can make the script interact with a file manager that accepts scripts, like spacefm, and manage files without ruining my tags. The example is if I wanted to sort into folders but may give you an idea to make others things.

>>37195 >>37198 Easytag is for metadata and file renaming. There is a command line metadata tagger called beats but was fairly limited. If I were to work with tags I would use metadata, and when I work with metadata I use exiftool. Everyone knows exiftool, is a fairly universal metadata handler, can do anything, and I have use it in scripts myself.

Got stories about dead, lost or secret data? Happy or melancholy, doesn't matter. >discover band online >started when members were teenagers, included lots of 4chan-edgy tracks, but lessening as members grow up or leave >ton of lore, it was basically a small community that happened to be a band >bandcamp goes down (either due to older edgy tracks or possibly sabotage from salty ex-member) but we archived most of it and reuploaded >fast forward, been taken down again >many band members decide its a good thing; they don't want to be associated with that politically incorrect history as they begin a serious music career There were some legitimately good non-edgy works and lots of effort put in. It's a real shame not to share it with the world, but it would be more of a shame to ruin their careers. I basically just shared the discography privately with three band members and we're probably going to wipe the net of us as best we can. It's really hitting me in my ethics. --- And before someone says this is a non-issue or paranoia for an artist to nuke their history, an ex-member who was kicked out for being an insane, death-threat making edgelord managed to become internet-famous and has now been dropped by promoters for an offensively-titled album they made years ago under an old handle. It's karma (they recently harassed someone and falsely got them and their friends derezzed from a concert, just for liking their the aforementioned band) but it's concerning enough. I'm just grateful I had decent OPSEC when brewing soykaf online.

>>37783 Post the discography

>>37783 >Got stories about dead, lost or secret data? Happy or melancholy, doesn't matter. Remember when that Iranian chick tried to shoot up the youtube HQ a couple years back? I managed to scrape her dailymotion before it got taken down. Lots of funny and bizarre videos I'm glad I was able to save.