Homelab Standardization Project

So, after many months of annoyances and generally poor encoding performance when compared to the main desktop it is time to upgrade the homelab! The current configuration is fine for most things, but when I ramp up the video encoding pipeline, the current setup leaves a lot to be desired.

The goal for this adventure is simple: to standardize all of the nodes CPU and memory while also upgrading the CPUs with respect to power efficiency and performance per thread. I do want to have the same number of threads available across the cluster and remain relatively close to the amount of power I’m using in the cluster.

Current Configuration¹

The current cluster has two Proxmox servers. The oldest is izumi and has some beefy hardware, but the server is definitely showing its age. The other server is urushi and is generally a faster server for almost all workloads. I don’t have issues with memory bandwidth constraints, but I do crave better number-crunching performance.

Proxmox Node 1 (izumi)

• Motherboard: ASUS ROG Zenith Extreme (X399)
• CPU: AMD Ryzen Threadripper 2990WX (32C/64T)
• GPU: Generic Low-profile GPU (Radeon RX 540)
• Memory: 64 GiB
• Networking:
  • Mellanox ConnectX-4 (2x 10GbE)
  • Mellanox ConnectX-4 (2x 40GbE)
• Local Storage: Generic NVMe drive

Proxmox Node 2 (urushi)

• Motherboard: ASUS ProArt B550-CREATOR
• CPU: AMD Ryzen 9 3950X (16C/32T)
• GPU: Generic Low-profile GPU (Radeon HD 5400)
• Memory: 32 GiB
• Networking:
  • Mellanox ConnectX-4 (2x 10GbE)
  • Mellanox ConnectX-4 (2x 40GbE)
• Local Storage: Generic NVMe drive

The Plan

So, what is the plan?

The upgrade path for urushi should be fairly straightforward. After a quick BIOS update, I’ll slot in an AMD Ryzen 9 5950X and swap out the 32GiB of RAM with a 64GiB kit which is fast and fairly inexpensive. I have absolutely no idea where the RAM came from, but this server should be simple to get going.

The upgrade plan for izumi is a straight replace. I’m going to replace the internals with an ASUS Pro WS X570-ACE motherboard, another AMD Ryzen 9 5950X CPU, and a 64GiB RAM kit and then transfer all the internals from izumi into the new server.

The unmentioned third part is to build a third server with yet another ASUS Pro WS X570-ACE motherboard, another AMD Ryzen 9 5950X, and 64 GiB of RAM. I don’t have anymore Mellanox Connect-X 4 cards, but I do have a Connect-X 3 40GbE card and an older Intel 10GbE card which will work perfectly well.

Why So Much Network? (An aside)²

Each Proxmox server has 2x 10GbE connections and 2x 40GbE connections. The 10GbE connections are for VM networking and management. The 2x 40GbE connections are for iSCSI and VM migrations. All of the VM storage is hosted on the TrueNAS server via an iSCSI LUN shared out to all the nodes.

This Seems Like a Sidegrade³

In a sense it is. I’m not gaining any CPU cores/threads, and there is increased overhead associated with running two computers versus only one. However, the CPUs are far more efficient than the 2990WX and it is far easier and cheaper to replace AM4 CPUs and motherboards than the older Threadripper stuff.

Also, if you’re running different generation CPUs across the cluster, your VMs will run using the lowest common set of instructions/capabilities across all the CPUs. This means if you’ve got a Ryzen 9 3950X and a Threadripper 2990WX, you’re limited to the features of the 2990WX instead of the other CPU.

Footnotes

1. The three servers on the shelf from left to right: phoenix which is the NAS, urushi, and then izumi. ↩

2. The iSCSI share is configured on all of the Proxmox servers to use a round robin configuration which means there’s close to 80Gb of bandwidth to each server coming through the Mellanox SX6012 from the NAS. The LUN is carved from a raidz1 array of 4 NVMe drives which provides ample bandwidth and performance. ↩

3. The AMD Ryzen Threadripper 2990WX is a Zen 1+ CPU. The AMD Ryzen 9 3950X is Zen 2 and adds a few nice instruction sets, but very similar with respect to core configuration. The AMD Ryzen 9 5950X is Zen 3 which has some absolutely major changes considering it was the first “ground up redesign of the Zen CPU core”. ↩