Directories were a mistake

I think there are many pitfalls one can fall into when creating/expanding/managing one's directory (or folder) structure. Unfortunately using directories how most people "naturally" use them pretty much always results in some fuarrrked up soykaf. I don't have any solutions but I'll describe the problems I see and hopefully we can get a discussion going about how to avoid these pitfalls or even some alternative way to manage files on a system. Alright, the first issue is that very often the most useful files are at bottom of the directory structure (or at the leafs of a tree that tends to be very tall). Why is this a problem? Because accessing those useful files or their directory means you'll always have to navigate to the leaf for you most accessed/useful files. It's tedious no matter what tool you use. With graphical file managers it's a game of hunt-and-click: a flatter directory tree means "hunting" is harder but less clicks and a taller directory tree means "hunting" is easier but you'll have to click more. Bash (and I assume other shells too) have a similar issue. Yes, you can autocomplete but it's still very tedious to type those long paths every time you want to access the useful files you have. The same issues apply to terminal file managers and things like dired etc. I think this happens at least partially because we use directories as context which is a very natural way to organize things(e.g. /home/user/documents/papers/tech/...). The more context you add, the longer the paths end up being. I think the typical way of mitigating this issue is to create a bunch of symlinks in your home directory (or where ever they are useful). But this doesn't solve the problem imo, it's duct tape applied to a fundamentally wrong approach. Anecdotally, what ends up happening is that your home directory gets filled with a bunch of symlinks (that you have to manage btw) which makes it cluttered but you still have that tall directory tree so it's kind of "the worst of both worlds". The second issue is that the "standard" directory structures (i.e. directory for user and subdirectories for things like "downloads" "documents" "videos" etc) end up being pretty soykafty ways to organize things. For example, I have a videos directory but actually most of my videos aren't there and instead I have things like ".../lainchan/videos" or ".../programming/lisp/videos". These ARE NOT examples of good ways to organize things but rather my point is that "videos" is not a category I usually want and I'm guessing that's the case for most ppl. Having something like a videos directory that has subdirectories for all your things (programming, lainchan, whatever) and then a directory for, say, programming that symlinks to your .../videos/programming directory also doesn't work well. What just ends up happening (again anecdotally) is that your files are scattered all over the place and there's a whole web of symlinks on top of your directory tree. For similar reasons directories like "programming" and "lainchan" don't work that well so it's not just a matter of removing those typical directories and putting their subdirectories in home. More generally, it's very difficult to find a useful AND meaningful "context" under which you can categorize your files. And again, the standard structure is a pretty "natural" way of doing things when you start out (and most distros and OSes actually come with the standard strucutre out of the box). The standard structure works great at first but as you add more files and more and more context (i.e. directories) for those files, the harder it is to have all files that relate to x close to each other in the tree. Maybe a better example would be this: you have a directory for files relating to some game (excluding the actual game files that are somewhere in the steam subdirectory) e.g. the OST, fan art, your notes. But you also have a music directory where most of your music is stored and a pictures directory where most of your images are. So should you copy these files to music and pictures too? That doesn't scale well. Should you move them to music and pictures respectively and then symlink from this directory? Anecdotally, this also doesn't scale well but it's harder to prove. Well maybe this game directory is a mistake and you should just put the files where they belong (music and pictures) and just deal with it. But I think this kind of a directory often more meaningful than directories based on the type of file (image, video, text etc). The third issue is that these structures often end up being very confusing: just go to .steam and try to navigate to your games. If you've done it a bunch, you'll know the path but it's certainly not intuitive. Similar things will happen to your own directories: at best, you will be the only one who can actually navigate and make sense of them when they get to this point. This is kind of the effect of problems 1 and 2 combined. I'm sure there are other issues that slipped my mind while typing this so feel free to add. Or have I overlooked some obvious solution for all of these problems? Do lainons generally agree that these are issues? Any solutions? I'm currently experimenting with something but it's stupidly complicated, only really works for navigation, doesn't solve the underlying issue and so I have very little faith that it could help with this.

I have thought about this too, and had an idea of a tag based file system. Each file can have multiple tags, and instead if traversing a directory tree you basically do a SELECT query on a set of tags. You still can treat a tag (or maybe even a union of tags) as a "directory" by selecting it as active (Analogous to working directory.) This let's you still have a program specific tag that it can use as it's default place to create and delete files. I still newd to think more about it before I gp ahead and write a simple prototype FS just to teat how it feels.

I have never encountered the problem you describe. But it's a good this you have talked it out of your head.

>>74011 This sounds very interesting. I've seen some "tag" based systems built on top of the "classic model" but what you describe sounds different. Can you elaborate more on this? How would the tags work? Is it incompatible with current filesystems and expected dir structures? If you end up making a prototype it would be nice to hear how it goes and maybe test it too?

>>74005 The problem isn't the directory structure, the problem is that in Unix-likes the true name of a file is its directory path, so there can be only one first-class directory structure (symbolic links etc are dirty hacks that don't count as changing this), and it can't change once one file or program references another without breaking it. If we had an alternative naming scheme you could have many equal directory trees organized by different and simpler schemes, files equally at multiple places in those directories, and the ability to change any of it however you like without altering the functions of your computer. In your example, instead of having a media/videos folder without your lisp videos in it you'd have two places your lisp videos were in, media/videos/programmingvideos/lispvideos and subjects/programming/lisp/videos, so you'd see them when you were browsing either system. This would also enable shortening well-used deep subdirectories by copying them to the top, say /programming/ is six layers deep in your very descriptive tree so you lift it up to be the root of its own tree, increasing its visibility without actually changing how your data is structured. The only problem is that the traditional directory structure is much easier to implement, and any better idea is "nonstandard", so it'll never happen on a non-commercial platform.

https://github.com/oniony/TMSU https://johnnydecimal.com/ (I recommend implementing a "johnny hexadecimal" for extra kool points)

>>74017 sounds like a job for hard- and symlinks.

>>74005 >Yes, you can autocomplete but it's still very tedious to type those long paths every time you want to access the useful files you have Command history removes this problem for me most of the time. You type the entire path once, then you don't really have to do it ever again. >most of my videos aren't there and instead I have things like ".../lainchan/videos" or ".../programming/lisp/videos". I'd structure it like ../videos/lainchan and ../videos/programming >then a directory for, say, programming that symlinks to your .../videos/programming directory I'd never have a programming directory above the directory for a certain type of file. Books about programming go in a sub-directory of "documents". Source code goes in another sub-directory of "documents". I guess I organize things based on how I use them, more than what they're about. It's definitely not perfect though. >you will be the only one who can actually navigate and make sense of them when they get to this point Never been in a situation where somebody else needs to navigate my directories. When I need to navigate some network drive, the location of things tends to be documented, and if not I can search for whatever I need at the highest level I have access to. I've thought about having a tag system on-top of the directory hierarchy, but the problem with that is that on top of being a pain in the ass to tag everything, it's not portable. You upload something to the internet, and nobody else can benefit from the tags you added.

>>74017 > The only problem is that the traditional directory structure is much easier to implement, and any better idea is "nonstandard", so it'll never happen on a n God damn you're smart. Has anyone tried something like this?

>>74021 >Command history removes this problem for me most of the time. You type the entire path once, then you don't really have to do it ever again. It helps a little but it's doesn't cover even nearly all the bases. For example, if you have to refer to a directory that you haven't used before. Also if you have multiple shells open, their histories don't always match. >I'd structure it like ../videos/lainchan and ../videos/programming Yeah, this is the standard way. However, I don't usually want to organize based on the type of media i.e. videos, images etc though sometimes it's useful. Instead I find categories that have all kinds of different media more meaningful. The problem with having a videos superdirectory is that all my <favourite video game> related files (excluding the game files) are not in one place but I instead I have to find them all over the dir tree. You can't flip it around either, it won't work. And you can't "patch it" with symlinks in a way that isn't hacky as fuarrrk.

>>74018 Do you have experience with these?

>>74024 I did a Johnny Hexadecimal and it worked kind of okay. I let it get disorganized and halfway restructured and life got in the way etc. TMSU has been on the list for a while but I've been able to stay afloat with the status quo so I wasn't really compelled to get to it.

>>74027 Yeah, the Johnny (Hexa)decimal idea seems really interesting. I think I'll try it but it seems really high maintenance and it's not exactly clear what kind of areas and categories I should use (though that may be because I didn't read through the whole thing yet). TMSU also seems quite nice but it doesn't really address the second issue except by sidestepping it (which may be good enough, I'll have to try). Both of these are exactly the kinds of things I was hoping to see though so thanks for replying.

>>74023 ok but get a load of this

There are a bunch of tag based and database based filsystems you can use. But I don't think it's really worth the extra effort. I just use globs, `find` and `locate` and have no issues.

Build on top of the mess, user, a file manager is purely a userspace program. If you use linux which has extended attributes and some of Plan 9's magic (https://p9f.org/magic/man2html/1/bind through bindfs/unionfs and user namespaces), I'm sure you can reach a reasonable compromise. https://fuchsia.dev/fuchsia-src/concepts/process/namespaces may be what you're looking for. >>74017 >the problem is that in Unix-likes the true name of a file is its directory path That's dead wrong, a hard link is exactly what you think doesn't exist: a different name pointing to the same inode.

>>74005 >Do lainons generally agree that these are issues? I think they're pretty minor issues but especially important for autistic people. It can get big if you have too much stuff and/or you're really messy with the way you organize files, like I was. No folders, no meaningful file names, just bad vibes. Just having a sensible folder structure has been a godsend, even if it takes time to find files. I think this thread motivated me to make my folder names shorter and keep the things I use often in easy to reach places. I've been thinking about information retrieval lately; before we had Ctrl+F in pdf files, the best way to search for information in a book was going to the index section and checking out which pages relate to your topic. Now, you can use find in shell to get all the files that have a certain word. Or you could go finicky with it and use a tag system. But it would be best to just create the folder tree structure that works best for you because, as you pointed out, it comes naturally for us. Ignore the standard if you have to. And one more thing... something I'm focusing on is really minimizing the amount of "projects" I'm working on, to not overwhelm myself and be on top of everything. I should have a small clear subset of files I need to use often, so I optimize my access specifically to those files. This could mean temporarily creating symlinks, or having an alias command that takes me to that folder in shell. Alias are the coolest soykaf. GUI programs often have a list of recently opened files, but I understand if you don't use those a lot. I don't either.

This article describes non-heirarchical filesystems in depth with comparison to heirarchical filesystems: https://www.nayuki.io/page/designing-better-file-organization-around-tags-not-hierarchies

>>74011 a directory is a file's tag if you think it from another way.

>>74076 Only a file can canonically be only in one directory, while with tags, a file can be tagged with many attributes. Attributes can also, optionaly, have values. Yes, tou can emulate this on top of hierarchical FS with symlinks (or even hardlinks if u want tags to own the file, not just represent a collection.) None of this is a new idea, BeOS had a FS that was more like database, Microsoft had WinFS that it botched by ~being Microsoft~ using managed for everything and feature creeping it into the stratosphere. GMOME once tried to do something similar according to wikipedia (sorry, couldn't find anything else.) There are even older examples that I vaguely remember but cannot give examples. I think no one really wanted to changr the hierarchical view of fa because it breaks many assumptions that programs may have. To that end WinFS, for example, did have a way to access legacy hierarchy. IIRC it was built on top of NTFS as a RDB store. All of that to say that no dir != tag, but you are free to think of it as you want if it helps you navigate the FS. I for one think that many assumed hierarchical things may be emulated in a tag/db FS. For example by providing saved queries/unions of tags.

>>74073 >you can use find in shell to get all the files that have a certain word Yeah and you can use it with fuzzy find with something like this: alias whatever='cd $(find $HOME/files/ -type d -print | fzf)' However, the problem with find (at least for me) is that you have to wait a long time and/or get a lot of irrelevant results or you have to specify a directory for it to search that's lower in the hierarchy. >But it would be best to just create the folder tree structure that works best for you because, as you pointed out, it comes naturally for us Well the problem is that I don't know what works. I think it's a pretty natural initial way to organize things but I do think these issues are more fundamental and it spirals out of control pretty quickly. To keep it under any kind of control I have to agonize over all these organizational things that I would rather not spend my time on. The single hierarchy structure (one tree) works fine if you don't have many files but if you're kind of a data hoarder like me, categorizing files (into dirs) becomes impossible. >And one more thing... something I'm focusing on is really minimizing the amount of "projects" I'm working on, to not overwhelm myself and be on top of everything. Yeah this definitely helps >This could mean temporarily creating symlinks, or having an alias command that takes me to that folder in shell. Alias are the coolest soykaf. I also do this and it worked ok for a long time but aliases are kind of tricky (they can mess with thing in unexpected ways if you're not careful) and managing the symlinks get tedious if your most used files and dirs change somewhat frequently.

>>74098 instead of find, you could use indexed search, like locate, since it's faster.

>>74095 >Only a file can canonically be only in one directory 1) lrn2hardlink 2) xattrs are the best way to tag on most UNIX systems.

>>74011 Same and I think I've got a pretty advanced concept by now but screw me implementing all of that.. I genuinely believe that the classical directory system is plain insufficient for a lot of media storage. Especially for a habitual hoarder of reaction images as myself. It just lands on this pile that I "hunt" items on by a chronological order and it plainly makes it too inefficient to find the one image I know I had saved but is buried in one of many download folder backups. It's a mess. A superior system would be one based on some sort of taxonomy or "sets" and in addition would need to be efficient at categorizing many items at once. Symlinks just ain't doing it. But a storage system that could also handle unusual data too, like individual posts in their entirety from an image board as an example. There's also an archival aspect to it that has me interested and I'm already thinking of a file format that can accommodate that. Ofc the classic dir system is still indispensable for systems and code bases and other working directory structures but if it comes to static, non changeable entries that at best ever only get "remixed", meaning result in a separate copy would need something else for storage and retrieval and even making collections available in a searchable way, like say "boorus", would be certainly an improvement. So yeah, keep me updated on your results. Maybe we should even get in touch some day when I've got more to show for myself on that note. Main problem that I'm facing is how I could make something like that integrateable with existing file browsers. It would need a custom interface but should be interoperable with things like copy/paste and drag&drop. I've got ideas on that already but I'm not sure it would really work out that great. But I guess best way is to just hack something together and see.

my ideal would be a system based on hierarchical tags, that is, a tag isn't just plain string, but a path where every component is a tag itself, and referencing a leaf tag implies all the tags before it in the path for example, where boorus disambiguate tags by doing "<character> (<franchise>)" an equivalent hierarchical tag would be /media/<franchise>/<character> or similar the API would let you supply a list of tag paths to filter by, returning any number of files (instead of the config location mess there currently is on linux, just source files with the tag "/config/<program>"), but if you needed unique reference it would be easy to add a tag inherent to all files like /inode/12345 or some other uuid mechanism (hash and other metadata could be represented as tags too)

>>74095 I know someone touched on this already but it might not be clear to readers. Hardlinks are not a layer, simulation, emulation, or illusion of any kind. They’re two entries in the file database that both point to the first inode of the actual file data just like any other file. Each link has an equally valid claim to be the real and true file. If you make 100 hardlinks to a file, remove the original and 99 other links the one random remaining link is still the file.

this is something i've been looking into for a while, one of those times i was thinking of a post and its here synchronisticly lollers lall been taking a look at the source code for 'non-hierarchical file system' or NHFS. it doesn't compile cause its so old but it is still an implementation to study as pseudocode as its very cleanly written. it uses fuse and relational database to sort tags. the file/folder arbitrary paradigm is a bigger problem in computing than people realize. but it would be sweet to have a public-utility level, viable alternative to ext3/4 that frees up the user to explore different relationships bits of information can have. i imagine it'd be so cool to combine with an OOP system so you could visualize all these clusters of connections and that'd be a whole alternate desktop experience in/of itself. i'm coming at it from a weird angle, trying to do 'hyperthogonal (n. dimensional) cell systems' as a replacement for file systems, moving away from rldb and files all together. but having a base with the tools that exists goes a long way esp with conceptual leaps away from embedded concepts in cybernetics ppl take for granted and never second guess.

>>74005 Get a good mechanical keyboard, train your WPM and you will never complain about taking a lot of time typing to find something ever again.

>>74199 I have model F, I touch type. My WPM isn't anything amazing but it's still better than most ppl. The problem isn't just typing though. Think about how most ppl navigate their directories especially in the beginning. It's cd after ls after cd ad infinitum. Even the best case where you know exactly where your file is, is far from ideal. If I have to go 5 dirs deep, then even with tab completion and a high WPM, the time from you realizing you want to, say, open a file, and the file actually opening is something to be optimized. >>74190 >moving away from rldb and files all together Wdym moving away from files? That seems exceptionally hard.

>>74209 >cd >ls >cd >ls >cd >ls >cd >ls >cd >ls >cd >ls >cd >ls >cd >ls http://www.boomerangsworld.de/cms/worker/index.html https://github.com/jarun/nnn https://midnight-commander.org/ i still use shell often (especially root shell), but a good file manager is a lot of help really.

>>74209 >muh unix philosophy "everything is a file" >based zigzag philosophy "everything is a cell" at the deepest levels it seems like you cant escape files but the idea most people miss is that files/folders is literally a conceptual idea like 'tar files' refers to a literal magnetic tape reel & its all based off of stuffy college campuses in the 1950's how they organized their papers into filing cabinets and their conventions stuck... once you start exploring that as a convetnion 'among other conventions' though, it opens up what you can do with your computer... notoriously difficult to explain without a hands-on example, but, it's like, with a computer screen you get to move away from rectangles and paper yet all of our tools are hopelessly chained to the convention of using all of the computer resources to simulate paper. pdf, word, docs what-have-you. anyway, files, directories, folders, relational database, all comes with implicit hierarchies which restrict and narrow thinking and capabilities for new information / connections

There are database programs to sort files in different views. They map all your files once and then give you fast access to everything. You literally dont need to care about file structure at all for optimizing access.

>>74211 Yeah I use dired myself but sometimes you can't use any of these (like when you ssh into someone else's system etc). >>74216 Seems very interesting though I still don't understand how something like it could replace file systems rather than be an alternative file system, albeit a very unique one. That said, I only looked at zigzag very briefly so I basically don't know anything about it. Is it more than a data structure? What would the cells contain if not, say, text files, image files, executable binaries and such? I can sympathize with the difficulty of explaining these kinds of things though so don't sweat it if you can't explain what's happening in your brain with words. Also the whole unix "everything is a file" is baked into the kernel so even if you have different user level representation, it would be really hard to get rid of file descriptors. I'm not trying to be critical btw, I'm just trying to improve my own understanding. I very much agree with: >with a computer screen you get to move away from rectangles and paper yet all of our tools are hopelessly chained to the convention of using all of the computer resources to simulate paper.

>>74216 Ok, since >>74228 I've done some reading about the history of files and filesystems. I'm starting to see what you mean.

>>74228 >>74230 >I only looked at zigzag very briefly so I basically don't know anything about it. Is it more than a data structure? What would the cells contain if not, say, text files, image files, executable binaries and such? I'll try to put it succ sinctly. The zigzag data structure is 'like' an excel sheet but instead of being strictly limited to the XY grid in 2 dimensions -- any cell can be arbitrarily linked with any other cell in any # of dimensions. that was tl;dr, rest is bat country. -- and (in the ideal implementation that hasn't been written yet) its meant to be visual-first as in the ux is meaningful to how the system runs but not as in wysiwyg and OOP so imagine it'd be incredibly easy to ie change the clockface ticking hands to the pencil used in the paint program because they're both cells and you simply switch the connection. there isn't even a distinct paint program or 'separate applications' you can 'draw/paint' inside a '(word) document' because its not files folders applicatations! realize that practically all software at its base is simply file operations. the cells are universally addressable, so on a network it reduces redundancy because you can have one instance of the same thing instead of making new copies for various contexts. this allows for searching for a book by its 'title' instead of /its/location/on/this/particular/shelf whoops it moved sorry you cannot get the data :) but yeah, the main thrust of xanadu project has been to use zigzag as a database backend for documents so cell contents primarily have to do with text. the design is not meant to be its own OS but a 'CO-operating system' to make connections among your own files, movies, executables etc where you normally just can't because you'd have to copy everything to every other directory or links and end up in a tangled knot. what inspired me was that aspect of questioning why so much cpu/gpu resource has been designed and put to use expressly and unquestioningly rendering cartoon men fighting each other and yet with text rendering we're stuck with like mousepad, proprietary mousepad, libre mousepad, mousepad＋＋ pdf html no one gives a blanq about 'documents' they just keep going fwd and consooming 'gfx'. anyway as much as this kaffinated soydrink beverage is the fu Hwarking future of document systems, rain iwakura sama i appreciate your time but the zigzag cell is but one nhfs in a sea of unrealized alternatives to directory structures that no one will ever know they didn't care about til they read about it on tobaskin throatweaving forumn.

>>74076 skimmed the article but definitely agree with this idea, i've been thinking about a flat file filesystem for a long time that basically has "virtual directories" based on metadata and taggings

I think a half decent file manager that supports shortcuts solves the issue. lf for example allows you to create persistent bookmarks with vim keys and I have them bound to directories I find myself accessing frequently. Naming files well can eliminate the need for tags, just search recursively if you have a deeply nested directory tree. I also think creating subdirectories is something one should take a little more seriously if you've handling so much data that you get lost navigating your own filesystem, but personally I have never really had that problem.

Somewhat on this idea, what are some good standards to follow when naming files, especially media and particularly books and music, on UNIX-like operating systems? I think https://manpager.org/usr/aa/art/001.practice_of_computing.txt has some good suggestions, like restricting your characters to [A-Za-z0-9_.-], but other suggestions like mere brevity seem bad for sorting. Does anyone have any guidelines or general rules they follow? It seems best if things are able to express the relevant info while still being sorted in a sensible way, like how physical books are often sorted last, first. However, these restraints limit the # of delimiters and sometimes it seems generally complicated. Also maybe it actually isn't a good idea to use dots as the main delimiters because that is used for "file extensions"? I would be interested in following a sensible convention for file-naming

>>74005 This is an extremely easy issue to fix lain, you can just alias symlinks and just do ```sh ln -s chosendir /my/long/dir alias myalias="cd $(pwd)/chosendir" ``` for what you want, it's really not that deep. Unix gives you the tools to manage your own folders, if you have problems navigating your own structure, that's on you, not the fault of a dead simple structure

>>74238 t. dingdong who forgot what a hyperlink was https://scribe.rip/hyperlook-nee-hypernews-nee-goodnews-99f411e58ce4? https://xanadu.com/zigzag/ZZdnld/zzRefDef/ >>74372 >i trust the implicit hierarchy in unix unquestioningly good thing the boys at at&t who designed it did all the thinking for me

>>74372 That's exactly what I've been doing for years but it's not a good solution. I also have fuzzy finding and other things to make using them a little easier but ultimately symlinks are just duct tape applied to a more fundamental design problem. They're great for what they're meant for but not as some web for navigation.

>>74444 >That's exactly what I've been doing for years but it's not a good solution Maybe *your* solution isn't good. Y'know what I'm saying. Anyway, what are we complaining about? This discussion seems decent but it is wholly centered on how it's inconvenient for categorizing your extensive hentai collections. You're saying you want the fundamental hierarchical structure gone, okay... there's bound to be issues. >two files with the same name, with multiple tags each, and one tag is shared. This isn't unlikely, consider cover.jpg files all as [Album Cover] but with different [Album] and [Artist]. . Perhaps you shared tag, and have two results. This could be annoying, for instance >Permissions. You'd have to have a sort of permission priority now, or like a principle of least authority in tags. Consider a file tagged as [system], you'd obviously want access to be inaccessible to most users. Likely, tags themselves would impose permissions. If we had per-file permissions, that would probably be an annoyance. I'm guessing the tags of a file altogether would determine the least authority of access. >The sheer number of tags that would exist, would be difficult to manage. You would have to end up 'grouping' the tags eventually... maybe into groupings of groups o_o OK I'm spitballing. At least, in the course of thinking about it I've turned around to the idea. I still think most of this thread is full of wankers who are trying to be bigdicksmart 'I reject this totally fundamental idea in computing which isn't really that bad' because it makes them feel smart Honestly if you just do more careful organization you can do fine. It's mostly specific things like organizing books or documents that this tagging system would be useful, but that could easily be done with software, and if it's just for that benefit, why not? Is there much of a system benefit otherwise? I challenge anyone to answer that particular question. It would be funny if you guys got what you wanted and someone really did do this, but fuarrrked everything up in ways we cannot even imagine. That's always what happens