I wanted to give a heads up of a couple specific issues from running OpenWRT in such a configuration from the previous post.

First, this script is incredibly useful. OpenWRT has neither udev rules nor systemd’s device naming scheme. Thus, it is likely you will break things as you play around with virtual interfaces, since interfaces may get different ethX names as they get rearranged. The mac-static-interfaces script gives you an easy way around this, on top of letting you give descriptive names to each device (for example, I named mine gb-top, gb-bottom, tengb-top and tengb-bottom).

Secondly, the default OpenWRT x86 does not have journaling enabled on its filesystem. This is bad, because it also won’t try to fsck a bad FS – it will just mount it read-only. This can be a problem for a variety of reasons, but the best fix is to simply enable journaling with the command tune2fs -j /dev/vda2 (or whatever your device is). This will cause a simple journal recovery to be performed rather than a read-only mount.

Something I’d like to try is using a KVM direct kernel boot, bypassing the bootloader entirely. This would allow for easy kernel and kmod updates.

If you’re not familiar, SR-IOV is a technology that allows you to expose a single physical network device (or more accurately, “physical function”) as multiple virtual functions (VFs). Each VF is a separate logical PCIe device, so a board equipped with an IOMMU should be able to pass them into VMs individually. This allows for basically the same level of performance as a passthrough of an entire PCIe device, but still allows it to be shared amongst many VMs. At the hardware level, it acts as essentially a network switch, instead of doing that in software like with typical virtual NICs.

Now, in my previous post, I talked about some of the new hardware going into my new router. The star of the show is the X10SDV-8C-TLN4F. So why not run OpenWRT baremetal on here? A few reasons:

1. Sometimes, you have to reboot. Rebooting a board like this can take several minutes. Not good for uptime. Meanwhile, a VM takes about 20 seconds to reboot fully. A custom kernel plus bypassing the bootloader (either via efistub or KVM direct kernel boot) could probably get it down to 10 seconds.
2. Lockouts – if you screw up your network configuration, it can be annoying to get back in. Not impossible, since it still has a VGA port and USB ports, but it’s definitely nicer to be able to still access the host and fix the VM from there.
3. OpenWRT isn’t necessarily going to have support for every bit of hardware on a board like this. It may lack drivers, or userland applications necessary to control said hardware. e.g. IPMI tools are lacking.
4. I can easily benchmark routing performance using virtual networks.
5. Since I can run ZFS on the host, I get ZFS snapshots and all that.

But….was it difficult? Was it worth it?

Yes and yes.