Core infrastructure

Hypervisor

Professionally, I’ve worked with VMware ESXi, Xen, oVirt, and touched upon Hyper-V. When evaluating options for home use, I first google esxi xen ovirt hyper-v to see what other people are saying about the gamut of hypervisors these days. Some of the results are comparisons like Storage Review’s Selecting the Best Hypervisor, others are polls like /r/homelab’s Which Hypervisor are you using in your Homelab?. Find what’s missing from your initial query (in this case Proxmox) and search again. We’re not yet looking for “the best” as everyone will have different opinions; We just want to know what’s out there and to start getting a feel for community opinion.

I’ve never dealt with Proxmox before and most talk about it is positive in the context of a homelab environment. I download the ISO, spin it up in a desktop instance of VirtualBox environment, and get comfortable with the interface. It seems to hit all the bullet points. Given this is all in the spirit of exploration, let’s give it a go. If we discover limitations in the process of using it, (A) we’ve broadened our knowledge of what’s out there and (B) we can make use of our backed up data to replace Proxmox with something else and likely be up and running within a few hours.

Hardware-wise, I’ll likely be happy with an Intel NUC. In an apartment, size comes at a premium. I make sure to load it up with enough memory to run 3+ VMs, knowing that I’ll end up running GitLab which demands 8 GiB (though we’ll later see that can be pruned down to about 2.5 GiB). If I need more capacity, I can always buy another.

Network-attached storage

For network storage, I’ve only worked with NetApp professionally, so I’m a bit at a loss of what’s out there for a self-hosted environment. /r/datahoarder, /r/backup, and /r/homelab all provide insight. Browsing hardware store categories (e.g. NewEgg’s Desktop NAS is also instructive of what consumer options are out there.

Knowing that I don’t want to spend an inordinate amount of money on this, but also knowing that I want to have sufficient redundancy, I end up going with FreeNAS, 5 drives running RAID-Z2. While this might end up being a bit more expensive, the argument for Z2 goes as follows:

  1. Imagine one drive dies
  2. You insert your cold spare
  3. Your NAS begins to build parity on your new disk
  4. Your other drives experience significantly more read requests than usual
  5. You increase your risk of another drive dying during the process
  6. Imagine a second drive now dies. You’re now in an unrecoverable state.

I ultimately decided against consumer desktop NAS like Synology or QNAP as I’m more comfortable using something standard like ZFS than whatever proprietary approach they’re using. Unfortunately, this means the NAS is bound to be larger than the consumer options (it needs a motherboard that supports 5+ SATA connections and a chassis with bays for 5+ drives), but I’ll find a way to live with it.

Tools I found instructive during the decision-making process were ZFS RAID Calculator, PCPartPicker, and Backblaze Hard Drive Stats Q2 2020.

Note I ended up purchasing drives from multiple manufacturers despite Backblaze’s report. I’d like to avoid the unlikely scenario where a batch of manufactured drives are defective and I lose all my data. So I’ll spread my risk across multiple vendors. I also end up purchasing a sixth drive as a cold spare. The recovery process in mind is

  1. Imagine one drive dies
  2. Cold spare is inserted
  3. Replacement cold spare is ordered

Purchasing the parts

I end up purchasing the NUC and NAS from NewEgg if only because I’ve had good results with them in the past. The NUC runs me $1270, the 6 drives $1320, the remainder of the NAS hardware $1170. No doubt I could spend more time being more conservative with the purchase and save maybe 25% of the cost, but time is at a premium when there’s a baby at home.

The UDM was ordered directly from Unifi.