Archive for the ‘Uncategorized’ Category

Switch-Based NVMe Hotplug – a Few Attempts, and one Success

Monday, April 10th, 2023

Let’s say you’ve just bought a chassis with an NVMe backplane, or retrofit one into your chassis. Now, it’s time to see if we can get hotplugging and backplane management working.

First of all, PCIe hotplugging is hard. It’s nothing new – after all, PCI hotplug has been around in the form of PCMCIA cards for decades, and PCIe got the same treatment with the later ExpressCard standard. But the reality is that whether it’s a laptop with a card slot, a system with Thunderbolt, or a server with an NVMe backplane, it’s one of those things that you can only expect to work seamlessly if you buy a full OEM system validated for that purpose. If you cobble together a machine from parts, it’s much more difficult to get any sort of PCIe hotplugging working.

I went through this recently after adding a U.2 backplane. Here’s a few things I tried, some of which worked better than others.


Upgrading an SC847 with a rear 2×2.5″ Drive Cage

Saturday, April 8th, 2023

I read this post about upgrading an older SC826 to support the rear drive cage option, and wondered if I could do the same with an SC847. The newer ‘B’ models support this natively, but there are still tons of cheap non-B models out there.

The first question is “why”? To which there are several answers:

  • More drives! Free up a couple 3.5″ bays for 3.5″ drives, rather than using an entire 3.5″ bay for a 2.5″ drive.
  • Cheapest NVMe option. $60-65 for the upgraded motherboard tray, and $80-90 for the drive cage, compared to $200 or so for the 4x U.2 rear backplane.
  • More NVMe. You could get both the 4x U.2 rear backplane (BPN-SAS3-826EL1-N4) and the rear 2x NVMe cage for 6 NVMe bays. (There is also an 8x U.2 front backplane, but it’s hard to find). for a little over $200. The 8x U.2 front backplane is much more difficult to find. Maybe 4 NVMe bays isn’t quite enough for you.
  • Dual-expander (EL2) backplanes: There are no backplane options with both dual SAS expanders and NVMe support.

Enough intro, I was able to get this conversion working, and it was much easier than the 826 conversion (though took a lot longer). Here’s how.


Restoring eBay’s Sale History Link

Friday, April 7th, 2023

It’s very useful to be able to see the sale history for an item that isn’t yours. You might want to see how quickly it sells, or whether offers are likely to be accepted or rejected. Unfortunately, eBay seems to have recently removed this link. The good news is that the page still exists and can be accessed via the same URL as before. Here’s a GreaseMonkey script to turn the “x sold” text into a clickable link like it previously was:

// ==UserScript== 
// @name     Restore eBay sold items link 
// @version  1
// @grant    none
// @match *://**
// ==/UserScript==

element = document.querySelector("div.d-quantity__availability span:last-child")
text = element.textContent
re = /(.*)\/itm\/([0-9]+).*/
url = document.location.href.replace(re, '$1/bin/purchaseHistory?item=$2')
element.innerHTML = '<a href="' + url + '">' + text + '</a>'

I have not tested it on other userscript plugins. The end result looks like this:

Clicking the link takes you to the sale history page:

Broadcom 9400 – Should You Buy One for a Homelab?

Monday, April 3rd, 2023

The 9400 series is LSI/Avago/Broadcom’s first “Tri-Mode” HBA, capable of supporting SAS, SATA, and NVMe all in one adapter. There’s a few catches, but despite that, it might still be worth the buy depending on your circumstances.


Quick Bash Tip: Alt-Shift-3 (Alt-#)

Sunday, December 11th, 2022

Alt-Shift-3 in Bash inserts a ‘#’ at the beginning of the line, and then runs it. # is the comment character, so running it does nothing. So why would you want to do this?

The simplest use case is when you have a command typed out, but you realize there’s another command you need to run first. By pressing alt-#, you push the command into your history, so you can quickly recall it after running whatever other commands first.


# Set some dataset properties...but wait, what was the name of the dataset again?
# I have the whole line except for the dataset ready, so press Alt-#
$ #zfs set primarycache=none secondarycache=metadata tank3/
# It's in my bash history now, so I can run whatever other commands first
$ zfs list
# Okay, figured it out, so now I press up-arrow/^P (or search with ^R) to recall the command, then home/^A to go to the start of the line, then del/^D to remove the #
$ zfs set primarycache=none secondarycache=metadata tank3/path/to/my/data/set

Quick Fix: AppArmor+Libvirt Errors in Debian

Thursday, September 15th, 2022

A few months ago, I was getting some permissions errors when trying to create new VMs with virt-install. My main symptom was that I would get this error:

Could not open '/var/lib/libvirt/qemu/nvram/openwisp_VARS.fd': Permission denied
I recently tracked it down to an AppArmor profile issue. Fixing it was simple. I appended the following to /etc/apparmor.d/abstractions/libvirt:

  /var/lib/libvirt/qemu/nvram/** rwk,  
  /usr/share/OVMF/** rk,

This will also fix some other errors, such as the ‘Failed to lock byte 100’ and ‘cannot load apparmor profile’ errors.

2.5GbE Direct Attach SFP on the Omnia

Thursday, June 30th, 2022

I was able to get my Omnia to talk to a MikroTik CRS305 at 2.5Gbps via an SFP DAC, in this forum post.

Short version: You need to force 2.5Gbps on both sides (disable auto-negotiation). Doing so requires the other side to have a switch/NIC ASIC that can talk to the SFP+ module at 2.5Gbps (this is not always supported – even if you were to run 2.5GBASE-T off a transceiver, the ASIC and transceiver may still be communicating at a higher rate). In other words, this isn’t going to work on all hardware. For the long version, read on.


OpenWRT on SR-IOV: Follow-Up

Tuesday, May 31st, 2022

I wanted to give a heads up of a couple specific issues from running OpenWRT in such a configuration from the previous post.

First, this script is incredibly useful. OpenWRT has neither udev rules nor systemd’s device naming scheme. Thus, it is likely you will break things as you play around with virtual interfaces, since interfaces may get different ethX names as they get rearranged. The mac-static-interfaces script gives you an easy way around this, on top of letting you give descriptive names to each device (for example, I named mine gb-top, gb-bottom, tengb-top and tengb-bottom).

Secondly, the default OpenWRT x86 does not have journaling enabled on its filesystem. This is bad, because it also won’t try to fsck a bad FS – it will just mount it read-only. This can be a problem for a variety of reasons, but the best fix is to simply enable journaling with the command tune2fs -j /dev/vda2 (or whatever your device is). This will cause a simple journal recovery to be performed rather than a read-only mount.

Something I’d like to try is using a KVM direct kernel boot, bypassing the bootloader entirely. This would allow for easy kernel and kmod updates.

OpenWRT on SR-IOV: Good Idea?

Wednesday, May 11th, 2022

If you’re not familiar, SR-IOV is a technology that allows you to expose a single physical network device (or more accurately, “physical function”) as multiple virtual functions (VFs). Each VF is a separate logical PCIe device, so a board equipped with an IOMMU should be able to pass them into VMs individually. This allows for basically the same level of performance as a passthrough of an entire PCIe device, but still allows it to be shared amongst many VMs. At the hardware level, it acts as essentially a network switch, instead of doing that in software like with typical virtual NICs.

Now, in my previous post, I talked about some of the new hardware going into my new router. The star of the show is the X10SDV-8C-TLN4F. So why not run OpenWRT baremetal on here? A few reasons:

  1. Sometimes, you have to reboot. Rebooting a board like this can take several minutes. Not good for uptime. Meanwhile, a VM takes about 20 seconds to reboot fully. A custom kernel plus bypassing the bootloader (either via efistub or KVM direct kernel boot) could probably get it down to 10 seconds.
  2. Lockouts – if you screw up your network configuration, it can be annoying to get back in. Not impossible, since it still has a VGA port and USB ports, but it’s definitely nicer to be able to still access the host and fix the VM from there.
  3. OpenWRT isn’t necessarily going to have support for every bit of hardware on a board like this. It may lack drivers, or userland applications necessary to control said hardware. e.g. IPMI tools are lacking.
  4. I can easily benchmark routing performance using virtual networks.
  5. Since I can run ZFS on the host, I get ZFS snapshots and all that.

But….was it difficult? Was it worth it?

Yes and yes.


Mini-Review: SolarFlare SFN7501

Thursday, April 14th, 2022

Yet another SFP+ dual port 10GbE NIC.


  • Works fine. Much less disastrous than the last review.
  • Supports manipulating link state for VFs (e.g. so you can have VMs talk to each other even if the external link is down)
  • Supports multiple PFs per physical port
  • Lots of settings governing the internal switching and such
  • If you don’t need SR-IOV, it has an “ultra low latency” mode


  • Requires standalone program to do configuration. alien worked fine, but be aware that there is no official version for Debian-based systems. However, once you apply a configuration you want to the card, it will persist. Once you get it close enough, you can use sysfs and ip to do the rest (e.g. configuring number of VFs and VF properties).
  • Some setting changes require hard reboots.
  • Doesn’t appear to support trusted VFs. This unfortunately would make it inappropriate for certain tasks, like the router build.
  • Poor support for modern Windows versions. The driver and utilities package installed, but would BSOD consistently.


  • Not a terrible choice if you just need a good 10GbE NIC for Linux
  • Don’t buy for Windows or if you need special SR-IOV features