Switch-Based NVMe Hotplug – a Few Attempts, and one Success

Let’s say you’ve just bought a chassis with an NVMe backplane, or retrofit one into your chassis. Now, it’s time to see if we can get hotplugging and backplane management working.

First of all, PCIe hotplugging is hard. It’s nothing new – after all, PCI hotplug has been around in the form of PCMCIA cards for decades, and PCIe got the same treatment with the later ExpressCard standard. But the reality is that whether it’s a laptop with a card slot, a system with Thunderbolt, or a server with an NVMe backplane, it’s one of those things that you can only expect to work seamlessly if you buy a full OEM system validated for that purpose. If you cobble together a machine from parts, it’s much more difficult to get any sort of PCIe hotplugging working.

I went through this recently after adding a U.2 backplane. Here’s a few things I tried, some of which worked better than others.

For reference, I’m doing this on an X9DRD board, with a BPN-SAS3-826EL1-N4 backplane, and two KCM51VUG1T60 drives. Unlike newer Supermicro boards where you can use a simple redriver or retimer card to get NVMe hotplug, a switch/bridge is likely the only way to get hotplugging working on these.

First Attempt: Oracle NVMe Switch