Filesystems, disks, sync, and backup

Let's have a thread about filesystems! I'd like to talk about how we organize our digital lives. What do you use and why? ZFS, BTRFS, LVM? Good old ext4? Any questions or advice for other users? What backup or synchronization tools do you use? I'll start. I've been using LVM within a single encrypted LUKS partition for a while now on both my desktop and laptop. It works great, but I'd like to start incorporating more disks into my desktop setup and am starting to explore different options. I'm taking suggestions if anyone's got any. I've been using this for using my Yubikey to decrypt my filesystem on boot. It's simple, stable, actively maintained, and I'd recommend it. > https://github.com/agherzan/yubikey-full-disk-encryption For backups I've been using this. I like the simplicity of it, but I'm considering switching over to borgbackup so I can easily host backups on a remote server. > https://github.com/laurent22/rsync-time-backup Right now I'm keeping everything on encrypted disk that I'm keeping offsite at two separate locations, but that's a huge pain and I'd like to replace one of those off-site backups with a server.

>>16390 Same boat for me - mostly LVM on LUKS since most linux distros install that way. Btrfs is really nice if you have the extra space to support the snapshotting, but I have had it fail too many times to be reliable. I use ext3/4 and FAT for external storage since that's what works well. How do you go about managing encrypted external drives? My externals are spinning rust and I worry about performance on those if I encrypt them.

>>16413 Addendum - I encrypt all my portable systems, but my desktop and servers that don't leave my home are not encrypted, purely because they don't leave my home, and if someone has physical access to them there are bigger problems. My backups are also mostly just a copy of my home folder on an external drive. Used to use Deja Dup but it caused a lot of problems. Now I just basically use scp/rsync to an external or network drive.

>>16390 Are there any perceived day to day differences in using another file system from say ext4? Or are these file systems should only be used under particular scenarios and if so, which scenarios are they applicable? I've been really interested in exploring different file systems but still am really new to that side of things.

>>16390 My Thinkpad, VPS and home server all run btrfs on dm-crypt, mainly because I want shiny new things like checksumming, snapshots and compression, but I'm not that excited to muck about with ZFS. I'm sure it's easy to install on Linux, but btrfs is already there and works for me. I've only encountered corruption on one occasion — my machine had bad memory at the time — and btrfsck repaired it just fine without apparent data loss. For backups, I have a homegrown script that takes btrfs snapshots, creates borg¹ archives of them and syncs the encrypted Borg repository to B2². ¹ https://www.borgbackup.org/ ² https://www.backblaze.com/b2/cloud-storage.html >>16417 A file system is a file system; you probably won't notice all that much difference in your standard day-to-day use. Things like transparent compression can be a minor convenience on machines with limited disk space. My Thinkpad runs off a cheap 100G SSD, and btrfs with compress=lzo is currently saving me ~11G of disk space. Snapshots can be a great convenience to test large system changes in a safe environment: take a snapshot, chroot into it and go wild. Copy-on-write can be very useful when you're working with large files that don't need to be changed, but needs to be used explicitly (cp --reflink) and isn't used automatically by any programs I'm aware of, though there's definitely potential for improvement. Oh, and you can pipe subvolumes (this includes snapshots) around with btrfs-send and btrfs-receive. I don't use this currently, though it's definitely an interesting piece for potential backup setups.

>>16413 Encryption does not slow down io if you don't have a CPU from 10 years ago (without hardware AES)

>>16413 For external drives I use pretty much the same LUKs setup, but manage it manually rather than as a hook at boot. That YKFDE tool I mentioned let's me create dm-crypt devices and then open and mount them easily, and for me it's quite fast. >>16414 I feel pretty similar about the home security, but I encrypt it anyway. I'm not trying to protect against 'mossad doing mossad things' necessarily, but I am worried that if my computer gets stolen my drives could end up in the hands of someone who's at best curious and at worst nefarious. I don't worry about using the optimal cipher or about boot partitions or RAM, I just use basic full disk encryption to dissuade some jackass from snooping around the new (stolen) machine they bought on craigslist. >>16418 Being able to automate sending compressed and encrypted backups to a remote server is the thing that finally convinced me to ditch straight up rsync and go in on borg. I haven't made the switch yet though. Any advice for new users?

>>16432 Nothing you won't be able to grasp from reading Borg's manual. Set up a sane retention policy and mirror your Borg repository. Make sure you notice when backups fail. I have my servers email me the backup script output every night. Borg allows you to place backup repositories on remote hosts. If you intend to mount your backup repo/archives via `borg mount`, keep your repository local for faster access times.

I switched to FreeBSD just for ZFS and have been in love with ZFS ever since. It's years ahead of every other FS and still marching forward. Here's a nice video about replication as an alternative to rsync: https://www.youtube.com/watch?v=VolTJ_t4o0M

>>16464 >I switched to FreeBSD just for ZFS Linux is now the primary platform for ZFS development. FreeBSD is just a second class citizen (again). https://lists.freebsd.org/pipermail/freebsd-current/2018-December/072422.html

How do you make threat model diagrams, on paper in mind or digitally?

>>16390 This is effectively a cross-post, but it seems it belongs here. All the relevant partitions on my system are ext4 with the noatime, data=journal, and journal_async_commit mount flags. data=journal solves the issues with potential data loss due to delayed allocation, and jorunal_async_commit is a free performance boost which enables journal_checksum which allows e2fsck to detect corruption. Noatime is just a free performance gain. If I ran a SSD I'd add the discard option to enable TRIM support. My reasoning for not switching to btrfs is due to maturity. I'm conservative with my computing, and I'm only interested in switching to something once its been well tested and confirmed to not have any substantial issues (or at least these substantial issues being resolvable as is the case with ext4's delayed allocation issues). I don't use ZFS because I'm on a old machine with only 2gb of ram, a single disk, and a dual-core x86, also I've heard it's moderately annoying to setup as rootfs and I'd have to install it as it doesn't come with my distribution. For snapshot equivalence I just use emacs's built in backup functionality. At present I don't have a satisfying way of backup to another location, I just rsync /etc/ & /home/ onto a large USB drive, and call it a day.

Why can't I have a properly engineered filesystem with everything besides the bootloader encrypted on linux like I can on OpenBSD. ext{,2,3,4} are not well engineered. There is a good mailing list post about why this is the case and how this relates to linux becoming more popular, I don't have the link on this device nor am I interested in arguing about it. ZFS is mostly well engineered but its overly complex and massively bloated, and really annoying to use on linux (statically without kernel modules, static dev, etc). As for the rest of the topics I do keep everything encrypted I'm just not satisfied with the solutions on linux. I use dm-crypt on top of the raw disk and then ext4 on top of that, I don't want the needless added complexity from LVM. As for backups all of my config files are under vcs so I have copies on all my mcahines. I also keep a backup of the repo as well. I have physical copies of all important media, for other stuff I just keep multiple copies (lossless) on other machines. >>16471 >Linux is now the primary platform for ZFS development. <Can't use zfs on linux without unused(to be removed) kernel symbols and out of tree modules Shouldn't surprise me since these guys came from freebsd.

>>16490 >Why can't I have a properly engineered filesystem with everything besides the >bootloader encrypted on linux like I can on OpenBSD. ext{,2,3,4} are not well >engineered. There is a good mailing list post about why this is the case and >how this relates to linux becoming more popular, I don't have the link on this >device nor am I interested in arguing about it. I suspect this has something to do with soft-updates vs. journalling and/or the delayed allocation issues of ext4, but if you wind up with easy access to that mailing list post I'd be interested in reading it. I think it's important to note for all the impressionable youngsters that OpenBSD's UFS doesn't have TRIM support, or metadata checksums so there might be some situations (such as on a laptop with a SSD) where ext4 would be preferable, even if it was the case that overall OpenBSD's UFS is overall better engineered.

>>16490 >There is a good mailing list post about why this is the case and how this relates to linux becoming more popular, I don't have the link on this device nor am I interested in arguing about it. Opinion discarded then. >Why can't I have a properly engineered filesystem with everything besides the bootloader encrypted on linux like I can on OpenBSD Because OpenBSD is a coherent operating system with a unified kernel and userland, so they can create idiot proof buttons that does everything neatly. Linux is just a kernel and the rest you need to hack together yourself from what's available. If there isn't a distro with what you want and you don't have the skills to do it yourself then Linux is not for you. >Shouldn't surprise me since these guys came from freebsd. ZFS comes from Solaris. FreeBSD does nothing except keep it working on their own platform. >ZFS is overly complex and massively bloated, and really annoying to use on linux ZFS code depends on half of the Solaris kernel so for FreeBSD it's not too bad to use because they're both based on original Unix but Linux is only a Unix lookalike so things are more awkward to port across and hammer into place.

>>16491 >if you wind up with easy access to that mailing list post I'd be interested in reading it Here you go: https://marc.info/?l=tuhs&m=149473630210195&w=2 https://web.archive.org/web/20190403020355/https://marc.info/?l=tuhs&m=149473630210195&w=2 The linked page is also interesting but I will refrain from discussing operating systems in this thread. >>16492 >Opinion discarded then. I should have been more clear, I'm not interested in discussing BSD vs Linux. I will discuss filesystems and encryption since that is the purpose of this thread. >If there isn't a distro with what you want and you don't have the skills to do it yourself then Linux is not for you. Indeed I do it myself, I'm just not happy with it. I have to use a initrd and a seperate unencrypted filesystem containing it and the kernel be this on the same disk or on a seperate thumbdrive. Now that I've done all the work it behaves essentially the same as OpenBSD, prompting me for a password immediately after power on but I don't consider it a good solution.

>>16496 >https://marc.info/?l=tuhs&m=149473630210195&w=2 Thanks lain, that was an interesting read. The main thing that I got from it was that ext[2-4] are poorly designed file-system (not only in not using soft-updates but also in using volume journaling rather than logical journaling which I wasn't familiar with) with extensive heuristic based tooling for recovery. Due to the manner in which it's poorly designed and the extent of the heuristics the poor solution (excluding ext2) tends to work better on poor hardware, like that of consumers, than well designed systems. It's interesting to note that the way the right thing is described is exactly what I've heard about XFS, which uses logical journaling as mentioned in the article, although I've never known why until now. Things along the lines of "This is a exceptional reliable file-system, but it doesn't seem to tolerate bad hardware". Perhaps this could be the file-system you're looking for? I'm currently reading: https://lwn.net/Articles/339337/ out of curiosity but likely won't comment here.

>>16496 >I should have been more clear, I'm not interested in discussing BSD vs Linux. I will discuss filesystems and encryption since that is the purpose of this thread. You're not discussing filesystems though you're just dumping a dumb opinion with a link to somebody else's opinion as if that makes it more legitimate. >Now that I've done all the work it behaves essentially the same as OpenBSD, prompting me for a password immediately after power on but I don't consider it a good solution. Why not? It sounds like you regard OpenBSD as superior simply because you didn't have to open the box and understand how it worked.

>>16500 >The main thing that I got from it >was that ext[2-4] are poorly designed file-system I read it more like Unix tried too hard to be clever and it didn't pay off. Linux did the simplest thing they could and in the real world that was fine. >Due to the manner in which it's poorly designed and the extent of the >heuristics the poor solution (excluding ext2) tends to work better on poor >hardware, like that of consumers, than well designed systems. I think the point there is that Unix is much older than Linux. By the time Linux arrived in the 90s, hard drives started doing their own write scheduling in firmware so the fact that FFS did all this fancy write reordering and ext2 did none at all made no difference anymore.

>>16509 >I read it more like Unix tried too hard to be clever and it didn't pay off. >Linux did the simplest thing they could and in the real world that was fine. It wasn't fine though, it was under-engineered and only ended up fine because of the brute-forcing of heuristics. That's not a sane way to develop software, and can hardly be called design at all. Also the soft-update file-systems did pay off, with something like a 20% performance gain over journaled file-systems it's just that this payoff didn't last long in ways they couldn't have foreseen. >I think the point there is that Unix is much older than Linux. By the time >Linux arrived in the 90s, hard drives started doing their own write scheduling >in firmware so the fact that FFS did all this fancy write reordering and ext2 >did none at all made no difference This is why journaled file-systems started to pop up in the early 90s. It became desirable to take advantage of writeback caching so companies began working on means to do so *without sacrificing safety*. ext2 did not do this, it abandoned the most substantial safe-guards we have access to entirely.

>>16510 >It wasn't fine though, it was under-engineered and only ended up fine because >of the brute-forcing of heuristics. This combination of words has no meaning. >That's not a sane way to develop software This is apex fallacy. You are comparing an OS used in industry and academia for decades vs an OS hacked up by a student in his dorm room. > Also the soft-update file-systems did >pay off, with something like a 20% performance gain over journaled file-systems Where do you get this from. Sounds made up. >it's just that this payoff didn't last long in ways they couldn't have >foreseen. What are these "ways" then? >This is why journaled file-systems started to pop up in the early 90s. It >became desirable to take advantage of writeback caching so companies began >working on means to do so *without sacrificing safety*. Journaling has nothing to do with writeback caching. They were just the two examples of how ext2/3 was different. >ext2 did not do this, >it abandoned the most substantial safe-guards we have access to entirely. And who did do this in the 90s exactly? BSD filesystems didn't get journaling until way after ext3 and ntfs. And again, you're soykafting on Linux for cutting corners during their first few years and not doing things that international corporations with thousands of engineers were doing. sage because there's no content here, just calling this faggt out on his bullsoykaf

>>16490 >>16492 >>16496 >>16507 >>16509 >>16510 >>16523 What in the name of Jesus H. Christ are you animals doing in my thread?! Seriously though, while I find the discussion of the design and history interested I'm not too concerned with which filesystem is best in the abstract sense and want to re-focus on the user experience. For instance, I get that ext3/ext4 have disadvantages that can pose a serious risk of data corruption in certain situations. On the other hand, using a widely supported and relatively mature filesystem feels like a better choice to me--it's familiar, I know what I'm doing, and there's a lot of documentation out there for when I don't. I keep regular, redundant backups anyway so if anything does happen I won't be to fuarrrked. So far I haven't once had to use them because of a corrupted disk, only for accidentally deleting soykaf and other pebkac. That said, I'm interested in other options. Snapshots are appealing to be for the situations I just mentioned, but since I already use LVM I'm considering just doing it at that level. >>16473 I agree that btrfs is appealing but I too feel like it's too green to trust with my daily driver. On the other hand, every time I start reading the documentation for using ZFS with linux my eyes glaze over and I start thinking about everything else I'd rather be doing. Seems like more trouble than it's worth at the moment, but I'd love to be proved wrong!

>>16523 >Where do you get [that soft-updates had a 20% gain of jorunaled filesystems] >from. Sounds made up. That's my bad, I misread the claim from: https://lwn.net/Articles/339337/ that was comparing FFS without soft-update to FFS with soft-update. (This article is against soft-updates for maintainability reasons which is interesting, it's the reason NetBSD switched to journaling with their UFS implementation, but oddly not FreeBSD or OpenBSD) >What are these "ways" then? That write caching would be implemented on hardware and render ordering of writes in software obsolete. >Journaling has nothing to do with writeback caching. They were just the two >examples of how ext2/3 was different. It does though because you can't have both writeback caching and soft-updates. >And who did do [journaling] in the 90s exactly? Microsoft with NTFS in '93, and IBM with JFS in '90 >You are comparing an OS used in industry and academia for decades vs an OS >hacked up by a student in his dorm room. [...] >And again, you're soykafting on Linux for cutting corners during their first >few years and not doing things that international corporations with thousands >of engineers were doing. That's really not what I'm trying to do. I was trying to access the software on its own merits, and came to the conclusion that on consumer hardware (which the vast majority of us are using) it's a better solution than the right thing but that even still it's not the right thing. If you want to add a historical context asterisk to the statement I've got no problem with that. Hopefully that's enough of a compromise for us to return OPs thread. :)

>>16524 >I agree that btrfs is appealing but I too feel like it's too green to trust >with my daily driver. Other than "Move fast and break things." there don't seem to be any substantial users of the file-system in production, which is probably a sound indicator. >For instance, I get that ext3/ext4 have disadvantages that can pose a serious >risk of data corruption in certain situations.On the other hand, using a >widely supported and relatively mature filesystem feels like a better choice >to me--it's familiar, I know what I'm doing, and there's a lot of >documentation out there for when I don't. As far as reliability of a file-system goes you've got to be aware of any implementation/feature-set trade offs you're making. btrfs is the best example of this because even though it's got many feature advantages over ext4 it's not production ready while ext4 is. There aren't many file-systems which are substantially lower risk than ext4 from a feature set point of a view and even fewer from a implementation point of view. >That said, I'm interested in other options. Snapshots are appealing to be for >the situations I just mentioned, but since I already use LVM I'm considering >just doing it at that level. I think other than ZFS, ext4 with my suggestions here: >>16473 in a raid configuration with LVM snapshots is as good as you're going for reliability to get on normal hardware. Maybe XFS, or JFS could do better, but to be honest I'm not familiar enough with these to say, perhaps someone else could.

>>16528 >This article is >against soft-updates for maintainability reasons which is interesting, It's not surprising since the Linux vs BSD situation is reversed. Now it's Linux that has overwhelming industry and academic support and BSDs that need to scrape by on a shoestring budget, they don't have resources to waste on the "doing it right" ego game anymore. >That write caching would be implemented on [firmware] and render ordering of >writes in software obsolete. Exactly what >>16509 said. What else? You said ways. Wayssssssssssss. Implying more than one. >Microsoft with NTFS in '93, and IBM with JFS in '90 Exactly. Massive software companies. Not tiny distributed open source projects. >>16529 > there don't seem to be any substantial >users of the [btrfs] file-system in production, which is probably a sound indicator. SUSE Linux uses it by default for a few years now so it is definitely being used in production. Maybe nobody "substantial" though.