OpenWRT on SR-IOV: Good Idea?
Wednesday, May 11th, 2022If you’re not familiar, SR-IOV is a technology that allows you to expose a single physical network device (or more accurately, “physical function”) as multiple virtual functions (VFs). Each VF is a separate logical PCIe device, so a board equipped with an IOMMU should be able to pass them into VMs individually. This allows for basically the same level of performance as a passthrough of an entire PCIe device, but still allows it to be shared amongst many VMs. At the hardware level, it acts as essentially a network switch, instead of doing that in software like with typical virtual NICs.
Now, in my previous post, I talked about some of the new hardware going into my new router. The star of the show is the X10SDV-8C-TLN4F. So why not run OpenWRT baremetal on here? A few reasons:
- Sometimes, you have to reboot. Rebooting a board like this can take several minutes. Not good for uptime. Meanwhile, a VM takes about 20 seconds to reboot fully. A custom kernel plus bypassing the bootloader (either via efistub or KVM direct kernel boot) could probably get it down to 10 seconds.
- Lockouts – if you screw up your network configuration, it can be annoying to get back in. Not impossible, since it still has a VGA port and USB ports, but it’s definitely nicer to be able to still access the host and fix the VM from there.
- OpenWRT isn’t necessarily going to have support for every bit of hardware on a board like this. It may lack drivers, or userland applications necessary to control said hardware. e.g. IPMI tools are lacking.
- I can easily benchmark routing performance using virtual networks.
- Since I can run ZFS on the host, I get ZFS snapshots and all that.
But….was it difficult? Was it worth it?
Yes and yes.
(more…)