What will happen to all the data?

HTML & text are easy to back up--storage techniques are easily outstripping the (still enormous) amount of data humanity is creating online in that format. But what about video & audio data? YouTube is over 15 years old--the oldest content isn't available in HD quality, which reduces overall filesize significantly, but given the sheer amount of stuff humans are filming and throwing online, how realistic is it that the oldest content will remain easily accessible indefinitely? How will it be stored? Where? How will we access it? Pic related: came across this post, started searching really old videos, thought about just the absolute mountains of data that are out there.

I think the sad reality is that old stuff (that isn't widely popular) will die out if nothing is done for them, and nobody will care for most of these videos, until they realize what is lost is precious or important. On a more optimistic view point, if we really want to preserve that much data, we need alot of storage, and the most optimized way to compress them. For keeping the most videos as possible, we probably won't be able to keep them in HD. maybe 720p, but 480p is alot more comfortable storage wise. We'll also need backups and not just one. We can also ask people to hold onto small portions/copies of the archives on their own. Gotta duplicate that data as much as we can. Finally, if we want to access it... we'll need someone to host it, or, preferably, multiple people hosting the same mirror of the archive. I don't really know in what form it would be hosted as. That's my general brainfart for the idea. I like this topic. It's a problem we are hurdling ourselves into, but where no one knows how to solve it.

>>56557 That's an interested idea, to distribute the storage load rather than try to centralize it. We'll probably need to, right? Diving this deep into media from 15+ years ago has got me feeling all sorts of ways, and I agree that if you think about it this can be pretty sad--but like a lot of other things I feel melancholy about these days, I do feel a real sense of optimism. We're never going back to those early days of Web 3.0, but archiving and preserving could give us the perspective we need to actually get a handle on the internet as a whole and come up with something real and new.

>>56564 this is essentially the concept of IPFS, which is becoming more popular, and could be the backbone of the distributed web

>>56557 >I think the sad reality is that old stuff (that isn't widely popular) will die out if nothing is done for them, and nobody will care for most of these videos, until they realize what is lost is precious or important. I share this feel. Sometimes I go into wayback machine and look at sites from 10-15 years and see just how little of the internet has been preserved from then. You'll see only a few captures of old forums that used to be huge at the time and realize that only a drop in the bucket has been preserved. The bytes that formed the backbone of the culture today, almost completely gone and lost to time. >>56567 >this is essentially the concept of IPFS, which is becoming more popular, and could be the backbone of the distributed web Every time i try IPFS the performance is awful. Has your experience been better with connecting IPFS?

>>56556 >>56564 Wasn't there a proposed implementation of blockchain data storage for medical records because they become a permanent record and it creates a secured distributed system? Speaking of data death, is there an archive of the data death thread where lains were talking about storage for long term and the most efficient vs. fastest ways of doing backups and archiving? IIRC magnetic tape is still optimal, and facebook uses ultra efficient magnetic tape to backup basically everything forever.

>>56569 I guess one question that hasn't really been posed yet is SHOULD all this data be preserved? I mean surely some it should be--if it were all to disappear, that would be a huge loss in every way. But considering forums, what more do you need apart from a few captures? What actual knowledge or material culture do forums produce? I care a lot about these sites and places--but maybe the reality is that a huge amount of data on the net is intentionally ephemeral? That what it is, really, is more noise than information? Not all of it, for sure...but if we take the cyberspace and hypertext metaphors seriously (big if), then accessing oldnet will by design be more archaeological than archival. It's a place, really--one big palimpsest. Maybe it's better that way.

>but given the sheer amount of stuff humans are filming and throwing online, how realistic is it that the oldest content will remain easily accessible indefinitely? probably 95% of the data that has ever been on the internet is no longer on the internet and never will be. This is the nature of things... the ravages of time are not kind, not even to supposedly "immortal" digital data.

>>56578 Since so many people are reliant on cloud storage, there will be entire decades and maybe even generations what will not have any kind of written or photographic record if technology were to fail. I think that the new bladerunner describes the idea of very few records being leftover from "the blackout"

>>56576 >I guess one question that hasn't really been posed yet is SHOULD all this data be preserved? That decision is not ephemeral and can be decided in the future. However, as of today you cannot preserve a piece of content from 10 years ago that wasn't preserved since it no longer exists. What has value to people is difficult to determine, which is why you preserve as much as you can now, and 20 years down the line discard the junk and repeats

>>56579 print is still a thing, even after decades of people saying it was dying. Books are still published, especially archival books specifically for libraries are kept very well. Mostly the important stuff that would really be lost if there was a massive solar flare or something would be technical info about computers, and a few online-only newspapers/journals. There would still be plenty of records though.

Eventually we will realize that archiving is futile except for identity data. As computer power raises, whenever people require content it will be generated on demand. Oh, you want to rewatch that old cartoon from back when you were a kid? Well, luckily enough we can estimate with a 0.9999 accuracy what the current state of memories from that age is, so, here, take some AI-generated show made from the scrapes you can remember, some historical recount of the programming from that time and the averaged memories of those that match your demographic. You'll be unable to tell if this is the real stuff or not. We KNOW you'll love it.

>>56586 >mashine learning will be able to make near-perfect recreations of things from extremely lossy unreliable data! this is some bull soykaf. If the data is gone it's gone, that's a fundamental law of the universe.

>>56556 >YouTube is over 15 years old--the oldest content isn't available in HD quality, which reduces overall filesize significantly, but given the sheer amount of stuff humans are filming and throwing online, how realistic is it that the oldest content will remain easily accessible indefinitely? Old obscure content already is basically unaccessible just due to the godawful search on site with very bare filtering functions. If you've ever deep-dived using petittube or incognitube, you know 99% of content uploaded to it is absolute trash. >Text-to-speech house agency ads, word pronounciation or meanings, or wikipedia reading articles or news headlines. >Indian filter spam video messages >0:00 videos (bonus if black screen + shuffling noise) >10/24/100h video loops >Just poorly edited or low-quality (240p or below) trash in general YouTube is not a viable business anymore. Free video/e-mail/gallery/etc hosting is not viable anymore. In the next five or so years we'll be seeing a sudden and rather violent shift towards everything turning paid to both gatekeep bots and spammers away and to try and mitigate operation costs.

>>56556 There's an interesting anecdote to the way things were done in the pre internet age. One of the large collection of the internet archive was built by an activist that feared media would backstep on their framing and recorded every tv program for the next decades on videotape. Back then no one would keep records of the live feed and even if they did for special programs, they could always destroy those copies without anyone batting an eye. They haven't digitized the complete collection yet, because the videotape needs to be replayed in realtime for that. In a similar manner, it will be the odd balls that keep some of the files. Filesharers will have stuff they downloaded on their harddrive and it will only require one torrent to go up to recover it, but you'll be waiting forever on some. Others run homeservers to archive everything they come into contact with online. However, as you can see, the internet as a distributed medium cannot be fully captured by one actor and what becomes archived will not reflect an objective collection, but the biography of the archiver. I believe the Internet Archive will be the central institution of data collection in a decade, even more than it is now.

>>56590 > In the next five or so years we'll be seeing a sudden and rather violent shift towards everything turning paid I would think to disagree. Maybe for Youtube, at least. Maybe a new product will shift towards that solution (I'm thinking of Floatplane, disrupt+, etc...) Youtube is finally figuring out how ads work and is using it full force to get more money. Still a difficult thing to maintain, but it's alot more better financially and it was a couple years ago, I would imagine. Even with the adblockers. I may be missing something here though...

>>56556 You could probably save full text copies and even images of everything you read using computer devices and it won't take much space, storage density improvements would gobble that up. Gwern Branwen says he generates approximately 50GB of web captures each year. For videos that don't bear much aesthetic purpose a subtitle transcript, a few screenshots and comment section (if it's anywhere meaningful, usually if the video has less than 100k views) are enough. Do you really want garbage to be saved permanently? Only what matters to people (you personally first) should be saved. >>56586 What said here makes some sense. Do we need to preserve hundreds of hours of someone playing [video_game] when we could preserve this game's files and just play it ourselves in any foreseeable future, or maybe make a machine play it and attach a small rectangle with animated shark-girl throwing witty comments in Japanese and add overlay machine-generated spam chatroom? Average twitch.tv experience circa 2010s.

>>56590 >Free video/e-mail/gallery/etc hosting is not viable anymore. In the next five or so years we'll be seeing a sudden and rather violent shift towards everything turning paid to both gatekeep bots and spammers away and to try and mitigate operation costs. I don't think that'll work. No one's going to pay for video hosting. They'd sooner go without or use some social media bullshit for it.

To anyone reading this: Start now, archiving offline anything you care about. Even 1 copy with no redundancy is better than no copy at all. I recently went from a 120gb ssd to 2 4TB HDD and i can archive most of the things that i care about (Mostly anime and Vinyl rips, but also some pretty old youtube videos that don't exist anymore) things on the internet will inevitably dissapear and it's up to everyone to save what we care about. Tools like wget and yt-dlp are excellent to archive things without complications (exampĺe, yt-dlp can download whole playlists on the video format/quality one wants, and can also download things from other sites like music from bandcamp) about redundancy: rsync is excellent for this. About optimal storage media: i've read a lot of people saying goog things about M-Disks and Tape drives.

>>56603 > yt-dlp can download whole playlists on the video format/quality one wants, and can also download things from other sites like music from bandcamp You the fuarrrking GOAT. I was looking for something like this way back until i gave up. Great post, lainon

>>56606 It saves youtube comments and subtitles as well, and there is a collection of annotations somewhere at Internet Archive's website. https://wiki.archiveteam.org/index.php/YouTube

>>56594 >a subtitle transcript, a few screenshots and comment section I think this is a really good point--there are ways of "compressing" information that go beyond the files themselves. >Do we need to preserve hundreds of hours of someone playing [video_game] when we could preserve this game's files and just play it ourselves in any foreseeable future No, but I strongly disagree with the sentiment that we should avoid archiving *some* of it just because we *might* be able to recreate that experience in the future. A) I'm skeptical it'll be nearly that easy, B) the accuracy of that experience will, as it turns out, rely on information and indeed video archived elsewhere and C) even if it all this was possible and accurate, there are ethical and aesthetic implications: the copy is not going to bring us any closer to the past, either in terms of an authentic experience or our understanding of it. We need the real thing, even if we can aid our understanding in some ways through simulations. >>56603 These sound like excellent tools, thank you for contributing.

>>56581 Paper books are basically never going away. E-book readers are nice but they don't match the UX/convenience of a physical book. You don't need power to read it, you can write all over it, give it to anybody, it's cheap, just works etc.

>>56630 a) Most of the stuff published today in a video format could have been explained in a small text article with a few quality images and animations (for see-do kind of learning). Each "high production" video has a script in the first place, which they don't even publish as subtitles in most cases, the "content creators" would save themselves and their audience a great deal of man-hours, bandwidth and storage space by just publishing a well-written article. 5000 years of literacy and hypertext were not invented for current generations to stuff our waking hours with 15-second tuk-tuks or whatever that chinese garbage is called ad nauseam. b) Nobody's gonna miss Linus Message From Our Sponsors Tips kind of contemporary video filler content if not for some deranged "archvists" who also miss early cinema era "forever lost" pornographic films. You can archive your favorite comedian soytubers, given that most of the golden web video years were in 480p era which means high storage savings on modern codecs, probably there are hundreds of people who already done this for anyone worth remembering, the rest of video on youtube is ephemeral even to it's creators and does not matter. >>56633 I remember reading an article somewhere that a paper book gives both spacial and "haptic" feedback on it's progress: you see how many pages you have read and how many left, and you also feel weights of both sides, all electronic paper readers I've seen either operate in made-up pages which depend on typeface size currently displayed or percentage, iirc none of modern ones even have option to show a progress bar which is more visually descriptive. Also, not specific to regular computer screen reading because on high pixel density high refresh rate displays anything can be simulated, even page turns, but to e-ink kind of displays, its either a "improved" refresh which could be missed in a blink of an eye and be absolutely frustrating if two pages have similar layout of paragraphs, there are no tactile cues similar to turning a physical page of a physical book, this is why I think in part Amazon and then other companies moved from touchscreen-tablet readers to readers with side grip and tactile page turn buttons; the second page refresh option being full page refresh which I find more useful actually despite it being "outdated function that modern screen technology does not need as often as older e-ink displays did" it gives better feedback confirmation that indeed a page has been successfully turned by the user and it's not the same one I just looked at.