3 Node NUC Cluster

Hello chums. I'm RMS's sentient left foot Today I'm going to build a 3 node Proxmox Intel NUC cluster with i5 & i7 cpus, 10gbe, 96gb of RAM, Optane caching, SSD, and 16tb of spinning disks! My lovely assistant, Waifu will be assisting with assistance throughout the project and I thought you wonderful folks might like to follow along and maybe help if we get stuck.

Expect updates throughout the day until I get so frustrated that I forget all about you. I will definitely try to answer all questions.

>>4494 What are you going to use it for? Or are you just doing it for fun?

>>4496 It will protect all my warez and porno but I will also run plex or something I'll also use it as a lab environment for work and personal projects. Less fleshed out plans that will probably happen are running a wiki, probably a build system at some point, git repository, and a fake windows license server. Things that might happen are I have a lot of leaks and dumps that I'd like to index, maybe some automated stock trading. I may prop up some things that I have ran on amazon and turned off recently to save money, nextcloud for example, maybe even tunnel them out to the internet. I have a similar set up running right now and one of these nucs was part of it except it's much weaker and I took a lot of other shortcuts that caused headaches. Each node in the old cluster has exactly just the hardware it needs for what it does and the lack of uniformity is a headache. Every node is about half as powerful as the next. It's using 1gbps ethernet except the storage node which has two channel bonded. IIRC the entire switch only has 10gbps of switching capacity so this will be quite a boost. Before that I was running a pi grade hardware cluster and it's most powerful node was the only x86 device. Once again the most powerful node becomes the weakest node in the next cluster. That was a total nightmare dealing with arm and x86 mixed, once again differences mean headaches. This will be the first time that the most powerful node is also the storage node. Usually I liked them separate but now the hardware is so overkill for my needs that I'm probably better off trying to put all my best stuff together so I can use it all at once instead of worrying so much about distributing responsibilities.

So I guess I'm done with a node already. The other nodes should have even less work, not sure how these are going to cluster. I have zfs mirroring working already. This is the i7 node and it has an internal 1tb SSD and a 16gb optane nvme, 32 gb of ram and two 8tb spinners mirrored in a usb 3.0 enclosure. Still need to figure out 10gbe over thunderbolt and how to get it to cache the spinning disks to optane. I also have some shares to configure but uhm getting here was easier than installing windows 10 so, wow that's really something. I've installed clustering environments that took me a week to get this far. Totally recommend proxmox. This took me minutes and getting boot drives working in the bios was the hardest thing.

I've never used ZFS at all but I've done LVM with ssd cache before. Should I be using zfs? Is LVM easier, I have no experience with LVM mirroring. My last setup getting the lvm cache working was ok but not exactly fun.

>>4494 post parts list and cost OP

>>4494 Quite possibly the stupidest setup I've ever seen. Why thunderbolt adapters instead of m.2 to PCIe risers with SFP+ cards? Those are going to be a nightmare to pass through, if you can even pass them through at all. The SFP+ to RJ45 adapters literally cost more than the switch they're attached to. This wouldn't have happened with a sensible setup. USB 3.0 drives? For virtualization? Why? I hope you got all this stuff for free, because you could've easily built a single machine which outperforms this kludge of a cluster for a fraction of the price. Hell, you could've built three better machines for cheaper if you wanted to experiment with clustering. And such a setup would actually resemble a real life deployment; I can assure you no one does 10GbE over thunderbolt in such a situation, and much less do they use consumer USB 3.0 drives.

>>4502 >Is LVM easier In my experience, yes, but it also does different things. ZFS is afaik more like window's ntfs in the sense that you can do per-file snapshots and versioning. LVM is just disk virtualization. I use btrfs on top of LVM on my server, so it definitely isn't an either-or thing (this may be kind of dumb because I'm sure btrfs can do the stuff LVM can do, but LVM seems easier for disk management). BTW btrfs is very similar in purpose to ZFS but it's actually FOSS. ZFS is more mature but has a somewhat restrictive oracle license.

>>4510 Believe me you don't buy this soykaf without realizing you could do it cheaper but I wanted to use nucs because they're small and don't consume much power and it seems like fun. I could have hit up craigslist and did the whole thing bigger and better on old servers for a fraction of what this cost. It's hard to say how much this all cost since so much of it is recycled from other projects I took my time making up my mind on stuff. 1 BOXNUC7i7BNHX1 CPU: i7-7567U RAM : 32gb 1tb adata su800 16gb intel optane 1 usb c drive enclosure with 2 7200rpm sata drives Price: I'm seeing about 500 dollars I'd say the drive enclosure and drives are about 400 The memory is maybe 150 The ssd is 100 dollars maybe 2x NUC7i5BNH i5-7260U 1000 total 16gb & 32gb (more memory in the mail) : 300 total 58gb optane nvme 240 total 128gb adata ssd in usb c enclosures 120 total The switch is a 10gbps x 4 mikrotik CRS305-1G-4S+IN and can work as a full router as well. I had to install 4 ethernet sfp+ modules and then after the sheathed cat7 we're talking probably 800 for all of that. >>4512 Oh yeah now that I think of it I might be running btrfs under LVM on my old cluster. That's got snapshots and soykaf doesn't it. Hmm

Oh yeah in that networking I included the cost in my calculation and forgot to list them: SOLO10G-TB3 thunderbolt 3 10gbps ethernet adapters Yeah we're looking around 3000 dollars but I'm sure I got deals on most of this. Some of it like those special parts like the ethernet adapters vary in price very little though staying at right around 160.

>>4510 The spinning drives won't come close to the speed of usb3 if you think about it. The ssd's won't either and they'll have a dedicated channel where they bridge into the bus I believe. The fast optanes sit in the m.2 slots with their however many pcie lanes all to themselves. Those will get used as spinning disk cache in the machine with the enclosure and mostly as nfs cache on the client nodes. I've also decided I'm going to have to boot off the optanes in the 2nd nodes because it doesn't want to boot off the usb ssd's I discovered proxmox is pretty small so it's worth it for the speed boost alone probably. I considered other configurations with risers and external enclosures, off the shelf and 3d printed and went with the SSD over usbc because it was cheap and I'd already blown a bunch on cool soykaf. The ssd drives are really there if I happen to want local storage except for the big machine which will serve it's internal 1tb sata out as a faster storage option. I don't want anything that's like a regular deployment, that's miserable, this is a rpi cluster on steroids.

>>4516 >The switch is a 10gbps x 4 mikrotik CRS305-1G-4S+IN and can work as a full router as well. I had to install 4 ethernet sfp+ modules and then after the sheathed cat7 we're talking probably 800 for all of that. How? That switch should've cost ~$140, and the SFP+ adapters should've been ~$70 each. Even if you did get them at proper retail prices, it would've been cheaper and better to just get an 8 port RJ45 10GbE switch for your RJ45 interfaces. For the $800 you spent you could've bought a 12 port RJ45 + SFP+ 10GbE switch like the CRS312-4C+8XG-RM. >>4518 Yeah we're looking around 3000 dollars but I'm sure I got deals on most of this. You spent $3000 on a kludgy consumer grade setup which will no doubt cause you problems. You could've easily saved a third of that price had you built custom machines (Since you're using consumer hardware anyways) with proper PCIe NICs and DAC cables. >I'm sure I got deals on most of this. You overpaid for already overpriced soykafty consumer hardware. >>4520 >The spinning drives won't come close to the speed of usb3 if you think about it. It's not about speed, it's about the reliability of the interface. USB drives tend to drop out randomly, and are hard (or impossible, depending on the configuration) to pass through to VMs. I'm not the one to tell you how to spend your money, after all I'm just some rando on the Internet. But you quite clearly have gone for the worst setup you could, which will no doubt cause you headaches due to the unusual, poorly matched and seemingly random hardware choices.

>>4526 >How? >That switch should've cost ~$140, and the SFP+ adapters should've been ~$70 each. Even if you did get them at proper retail prices, it would've been cheaper and better to just get an 8 port RJ45 10GbE switch for your RJ45 interfaces. For the $800 you spent you could've bought a 12 port RJ45 + SFP+ 10GbE switch like the CRS312-4C+8XG-RM. I have a much larger brocrade 10gbe switch already actually. I hate it and want to get rid of it, it's too big. I would give it away if anyone wants it, it's probably worth more than this whole setup. The 4 sfp adapters, the switch, the 3 NICs and the cables all added up. I would have had to buy some of that stuff with any switch. >It's not about speed, it's about the reliability of the interface. USB drives tend to drop out randomly, and are hard (or impossible, depending on the configuration) to pass through to VMs. The enclosure works just fine, they won't be passed through anyhow they'll be shared into the vm's with iscsi and nfs. The local disks were intended for use mostly as boot drives , I can say they're flakier so far I was going to mount them into docker containers for scratch too but I'm not sure how it will play out with proxmox's containers I forget what they're using but I've never used it. (Old system was docker containers) In any case the storage is there if I want it. Usb everything drops out in windows but people use USB as a boot disk in linux all the time with no issues. I felt a little weird resorting to usb devices but I don't think it'll actually be a big problem, if it is then I'll figure it out. a project like this is nothing but problems. I can switch the enclosure to esata if all else fails there is a tb3 docking station with 10gbe and esata that I almost went with for the larger node but I decided to try this in the name of cheapness and not trying to support two different tb3 nics when one was going to probably give me headaches anyhow. The 3000 is full price, I payed less than 3000 but couldn't tell you how much without going through my email. Some of it went up in price because of coronavirus while I was buying it but I can tell you there is no way I'm paying 500 for a nuc when people get rid of them all the time. They're popular in digital signs and things. Some of this hardware was purchased as long as two years ago in other projects. If you think this is kludgy you should see what else I've built. I can tell you're confused but I'd rather jump out the window than build a lab like anything at work I hate working and hate the way everything is done there from servers to software dev. It's not wrong or bad it's joyless. I sometimes think of quitting the entire industry because I don't actually enjoy it at all.

>>4512 I think I'm gonna switch to lvm + btrfs the disk is already mirrored and I'm gonna have a 6tb disk to put into my gaming pc that I can use to make another backup of my media library with rsync or something. I hear good things about zfs but I have a feeling about it. Just looking at how I'll have to set up caching feels pretty weird. I'm not likely to ever set u a disk at work again anyhow so those enterprise grade headaches can stay in the enterprise.

>>4512 >ZFS is more mature but has a somewhat restrictive oracle license. what The CDDL is FLOSS, it's just incompatible with the GPL. The original ZFS implementation from Solaris (pre-oracle times) was carried on illumos for a while. OpenZFS, a GPL'd rewrite, is now the canonical ZFS implementation and used by illumos, freebsd, and linux.

>>4540 >OpenZFS, a GPL'd rewrite, Emmmm.... since when it's GPLed? https://github.com/openzfs/zfs/blob/master/LICENSE >Unless otherwise noted, all files in this distribution are released under the Common Development and Distribution License (CDDL). Exceptions are noted within the associated source files. AFAIK it's not GPL-ed it's just run by people who are in denial that GPL and CDDL are incompatible.

>>4563 >AFAIK it's not GPL-ed it's just run by people who are in denial that GPL and CDDL are incompatible. I meant - run by people denying that GNU GPL and CDDL are incompatible.

Wow one asshole derailed the thread and now OP ran away. This is the diy board asshole, not your crappy corporate environment. I use a Thinkpad T60 as a seedbox and run my local git repo. How bout dat fgt! OP pls update, I never used proxmox before. Does proxmox set up clusters for you, or do you still have to fuarrrk around with ssh, dhcp, setting up heartbeat/whatever, etc?

>>4596 It's cool I'm still here. Guy didn't really bother me too much I can see why he'd feel that way. I hate when people waste money on bad computer purchases and it's hard to understand why I'd make these choices. I've been informed that my office will be closed and there was sketchy exec crap involved and that along with the corona and the state of the world has sort of dampened my desire to work on this stuff and also it's a pain to set up and tear down which will have to happen every session until it's ready to replace the old system and I've been busy taking classes to prepare for finding a new job so I just haven't felt like it at all. But based on the comments I saw while doing the research for this build other people are definitely trying to do the same thing so I'm not totally crazy for making this. To the best of my knowledge this will be the first place anywhere where someone has detailed building one though which I thought would be fun. The good news is that I'll be able to spend all next week working on this so there'll be some updates soon and maybe I'll even get the whole thing finished.

Ok so I have proxmox set up on all 3 nodes and I have them clustered. I need to get the disks handled and get thunderbolt auto-authorizing by installing something called "bolt". By far the hardest part was working around almost-bugs in the installer. For whatever reason it's very picky about what device I use. I had a USB that was given to my wife for free and it didn't work. The ssd in external enclosures didn't work. But a microsd card and reader worked. fuarrrking hell. >>4596 >OP pls update, I never used proxmox before. Does proxmox set up clusters for you, or do you still have to fuarrrk around with ssh, dhcp, setting up heartbeat/whatever, etc? Ok so once I had proxmox on all nodes getting the clustering working was as easy as using the webui to click create cluster. Then getting a "cluster information" blob to paste into a field on the other node's web UIs and also the root password. Worked like a charm. It does everything else for you. The webui has some ways of setting up disks but they're very basic. I'll handle it tomorrow because I so don't feel like dealing with it today. Looks like for my needs (software RAID 1 plus caching to optane) but there are good guides for ZFS and LVM.

Ok so I got the disks set up. I ended up using btrfs on LVM. They're raid 1 with an optane cache and the performance is out of this world. /dev/storage/mirrored: Timing cached reads: 25912 MB in 1.99 seconds = 13021.78 MB/sec 13GBPS!!! Oh yeah!!! Now to finish setting up ssd on the two smaller nodes. Right now they're booting totally off from the optane drives and will be configured to use the remaining space as NFS cache so it'd be nice if they had at least a little local storage in the form of 128gb ssd's. I plan on sharing these out with NFS as well.

>>4673 That's really cool. I've always wanted to try LVM, but the thing is that I don't know _when_ to install it in the installation process, that's not something you can do with fdisk, no?

>>4685 don't know what is the installation process meant here, but usually there are two options to setup LVM - user friendly distros will ask you if you want it (or infer from other answers); and advanced ones, where installation medium gets you a shell. In the later case, you first format your disks with gdisk/fdisk/dd/etc, setting some partition as LVM container. Next, using LVM tools (lv*) you create all the logical volumes/groups/etc. After that and any extra needed steps (e.g. cryptsetup) you should end up with a block device you'd use to install the OS onto as if there was no LVM involved.

>>4685 So most linux distros will let you pick LVM. I forget what it looks like but it'll be talking about physical volumes and logical volumes. To get the mirroring and caching set up you'll need to use google. For me this did indeed involve fdisk but also the LVM tools like pvcreate. I had to use one set of directions for mirroring and another for caching. You'll need a bit of intuition to figure out what to skip and what you can change to get caching set up. IIRC it involved finding the name of my mirrored logical volume and then applying cache to it. Where my instructions wanted me to apply cache to a single device. I was half expecting to get an error or find that the caching behavior wasn't evident but it has worked out for me every time.