My Homelab Efficiency Upgrades
My homelab uses quite a bit of power – somewhere in the ballpark of 700w, but that includes general networking hardware as well. Combine that with a Threadripper workstation and a separate work PC setup, and the power meter is not too happy. So what can I do?
Fortunately that only translates to about $65 a month due to relatively low-priced electricity. However, it also produces a lot of heat, which translates to more A/C usage during the summer. It also means lower UPS runtime, and risk of popping breakers if other heavy loads are on the circuit.
For reference, here’s a detailed breakdown of the current setup:
Proxmox Node 1
- Supermicro X11SDV-16C-TP8F (Soldered Xeon D-2183IT 16C32T)
- 4x32GB DDR4 2400 RDIMM
- No add-on NICs, only using the two SFP+ onboard NICs
- SAS3008 HBA + 82885T expander
- 9 HDDs, 4 SAS/SATA SSDs, 2 U.2 NVMe (SN100 3.82TB, SN100 1.6TB)
- Both of the U.2 drives support ASPM.
Averages 225W, with about 50-60W from the CPU. However, it tends to have the most VMs running on it at any given time due to the high core count.
Proxmox Node 2
- Supermicro X11SRM-VF
- Xeon W-2150B (10C20T, 120W TDP)
- 4x DDR4 2666 RDIMM
- ConnectX-4 Lx 25GbE (Running at 10Gb)
- SAS3808 HBA
- 2x SAS SSDs, 3x U.2 NVMe (SN100 3.82TB, SN100 1.6TB, SN200 800GB)
- The SN100s support ASPM, while the SN200 does not.
- Will eventually have spinners as well
Averages 150W, with 50-75W from the CPU.
Proxmox Node 3
- Supermicro X11SRM-VF
- Xeon W-2145 (8C16T, 140W TDP)
- 4x DDR4 2666 RDIMM
- ConnectX-3 40GbE
- SAS3008 HBA
- Expander on BPN BPN-SAS3-826EL1-N4
- 2x SATA SSDs, 5x HDDs, 4x U.2 NVMe (2x Kioxia CM5 1.6TB, 2x Hynix 1.92TB branded as HPE VO001920KXAVP)
- None of the U.2 drives in this system support ASPM.
Averages 185W. However, I don’t always have this node powered on – especially in the summer, where the added heat is a problem.
Virtual Router Host
- Supermicro X10SDV-8C (Soldered Xeon D-1541 8C16T)
- 4x16GB DDR4 2133 RDIMMs
- Onboard NICs + X550-T2 (branded as AOC-STGS-i2T)
- One M.2 NVMe
Unknown total power draw as it does not have PMBus, but the CPU is only about 10w.
Networking Hardware
- MikroTik CRS328-24P-4S+ for ethernet ports around the house + PoE for WAPS
- MikroTik CRS326-24S+2Q+ for core switching
- X-Fi XB8 cable modem
- Mini PC for home automation
Initial Analysis
The first problem is that many of the NVMe drives are small. A single 4TB drive will use significantly less power than 4x 1TB drives of a similar model, for example. Thus one possible path would be to consolidate some of the smaller drives into larger ones. Some of them are also not very efficient. The SN100 datasheet indicates a 25W max power consumption. Same for the SN200. Thus, the smaller HGST/WD drives are a good candidate for replacement. It is less important to replace the more efficient drives like the Hynix and Kioxia drives.
Another issue is the lack of package deep sleep states. Certain devices, especially SAS3008s and most Mellanox NICs, do not support ASPM, so they do not allow the CPU to fall into deeper sleep states like C6 and above. The router box, however, does seem to allow this according to powertop.
I am somewhat confused, as I tested with a spare X10SDV and found that the CPU could in fact go into deeper sleep states despite having peripherals attached which do not support ASPM. It is possible that the Xeon D-15xx series works a bit differently, or that some of the lanes are treated closer to “chipset lanes” would be on a traditional motherboard.
Thus, my first set of purchases are:
- Two Intel DC P4510 4TB NVMe drives to replace some of the smaller and less efficient drives. I found a good deal – about $310 for both before tax – so it’s possible that I can recoup most of the cost by selling the less efficient drives that they will replace.
- One AOC-S25G-B2S (Broadcom BCM57414) – the datasheet explicitly states ASPM support. If I put this in pve2, it should allow deeper sleep, as the SAS3808 supports ASPM as well. However, my fear is that the new U.2 drives might also prevent those sleep states.
New U.2 Drives: Intel DC P4510 4TB
Fortunately, the two 4TB P4510 drives support ASPM. However, until I get the new NIC as well, the CPU still can’t enter package-level C-states. Nevertheless, the drives are still more efficient than the SN100s, and let me drop from 3 drives to 2. After installing the new drives, the average baseline power of pve3 dropped from 150w to 125-130w.
I do not plan on re-using the 800GB SN200, as its small capacity and lack of ASPM makes for a very poor effective power consumption to capacity ratio. I moved the 3.8TB SN100 to pve1, replacing the 1.6TB that was previously there. This maybe increased pve1’s baseline by about 5w. Thus, the overall power savings from this is about 20w, and the overall capacity of the cluster increased – from 5.4/6.2/7.0TB per node to 7.6/8.0/7.0. I can also recoup some cost by selling the old drives.
I may also grab another 3.8-4TB drive with known working ASPM, and install it in pve3. I would replace the existing drives with the new drive, and the two SN100s, so that I could have working ASPM.
New Nic: AOC-S25G-B2S (BCM57414)
The datasheet and product page explicitly mention ASPM support. Not to mention, the base power usage is theoretically lower than the Mellanox ConnectX-4 Lx card. So I just have to plug it in, and….
66:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller (rev 01)
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
66:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller (rev 01)
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
Are you *** kidding me? You had one job. Updating the firmware didn’t help. It still lowers power draw slightly due to being a more efficient card, but this is a bummer.
Let’s take another look at the Mellanox cards. There is a newer firmware from Supermicro. One card absolutely refused to burn the new firmware. Another one took it, and it seems to have enabled ASPM!
04:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
04:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
Still No ASPM, and Yet…
The P4510s support ASPM, but it is currently disabled. Why?
b4:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] (prog-if 02 [NVM Express])
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s, Exit Latency L0s <64ns
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
b5:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] (prog-if 02 [NVM Express])
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s, Exit Latency L0s <64ns
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
I used the script from z8.re available here to force ASPM on. However, I ran into I/O errors and an eventual lockup and reboot. Pretty bad ones, too – I had to power off and on, not just reset, to get the system back up.
But wait… even without using that script, when it clearly says “ASPM Disabled”, the CPU idles at about 25w, and powertop reports working C6! The lesson here is to not actually trust the ASPM Enabled/Disabled status in lspci. Under baseline load, the CPU hovers between 40w-50w, a reasonable improvement from the previous amount of 50-75w. The overall system power draw is 120-125w, down from 150w. Not bad considering it barely has any time to actually enter package C-states. The worst offender is k8s nodes.
One thing to note is that the motherboard (X11SRM-VF) defaulted to L1 only for ASPM on all of its PCIe slots (including the JNVME ports). I changed it to “Auto”. For whatever it’s worth, enabling or disabling VMD did not seem to change anything. I leave it off, because it seems to make the green/amber “ejecting”/”safe to eject” LED states not work on the backplane.
Next Steps
As mentioned, the main culprit is k8s nodes, specifically the control plane. I don’t want to shove them all onto a single system for obvious reasons, so I’ll probably look at how I can safely reduce the CPU usage of the apiserver, calico, and other “busy” components.
pve3 has four U.2 drives and an HBA which do not support ASPM. pve1 has an HBA that does not support ASPM, but drives that do. Unfortunately, it’s also by far the busiest node, so it doesn’t make sense to try to go for ASPM. If anything, once I get a U.2 backplane for it, I may swap pve1 and pve3’s drives around, and put a newer HBA in pve3. This should allow pve3 to achieve ASPM. I am not as worried about ASPM on pve1, as it has the most VMs assigned to it. I figure that since it has a very efficient Xeon-D, it should take the load for anything not performance-sensitive. Thus, it would very rarely be eligible to enter a deep package-level C-state in the first place.
Another option I have is to use one or two very low-power systems, like my SYS-E50-9AP-N5s, to hold the k8s control plane nodes.
One oddity I noticed during testing is that my spare X10SDV seems to be able to enter C-states regardless of ASPM support of connected devices. Despite its age, it may still be a good choice for a lower-power system without sacrificing enterprise-y features like ECC, onboard 10GbE, IPMI, and still has 20 usable PCIe lanes.
Other Thoughts
z8.re’s The Curse of ASPM and Part 2 were very helpful. However, I will disagree with one thing – the choice of a 9600 HBA. While the 9600 (SAS40xx) does support ASPM, its typical power usage is 17W for the 16i and 20W for 24i according to the datasheet, while the 9500 (SAS38xx) uses a miniscule 8.5W for 16i, and only 6W for 8i. The 9500 also supports ASPM. I suspect that the increase in power draw of the 9600 is due to it supporting SAS4 (24gb/s/lane) whereas the 9500 only supports SAS3 (12gb/s/lane). Now, if only Broadcom would make non-tri-mode controllers once again, they could probably make their controllers even more efficient.