A few months ago, I was getting some permissions errors when trying to create new VMs with virt-install. My main symptom was that I would get this error:

Could not open '/var/lib/libvirt/qemu/nvram/openwisp_VARS.fd': Permission denied

I recently tracked it down to an AppArmor profile issue. Fixing it was simple. I appended the following to /etc/apparmor.d/abstractions/libvirt:

  /var/lib/libvirt/qemu/nvram/** rwk,  
  /usr/share/OVMF/** rk,

This will also fix some other errors, such as the ‘Failed to lock byte 100’ and ‘cannot load apparmor profile’ errors.

I was able to get my Omnia to talk to a MikroTik CRS305 at 2.5Gbps via an SFP DAC, in this forum post.

Short version: You need to force 2.5Gbps on both sides (disable auto-negotiation). Doing so requires the other side to have a switch/NIC ASIC that can talk to the SFP+ module at 2.5Gbps (this is not always supported – even if you were to run 2.5GBASE-T off a transceiver, the ASIC and transceiver may still be communicating at a higher rate). In other words, this isn’t going to work on all hardware. For the long version, read on.

Read the rest of this entry »

I wanted to give a heads up of a couple specific issues from running OpenWRT in such a configuration from the previous post.

First, this script is incredibly useful. OpenWRT has neither udev rules nor systemd’s device naming scheme. Thus, it is likely you will break things as you play around with virtual interfaces, since interfaces may get different ethX names as they get rearranged. The mac-static-interfaces script gives you an easy way around this, on top of letting you give descriptive names to each device (for example, I named mine gb-top, gb-bottom, tengb-top and tengb-bottom).

Secondly, the default OpenWRT x86 does not have journaling enabled on its filesystem. This is bad, because it also won’t try to fsck a bad FS – it will just mount it read-only. This can be a problem for a variety of reasons, but the best fix is to simply enable journaling with the command tune2fs -j /dev/vda2 (or whatever your device is). This will cause a simple journal recovery to be performed rather than a read-only mount.

Something I’d like to try is using a KVM direct kernel boot, bypassing the bootloader entirely. This would allow for easy kernel and kmod updates.

If you’re not familiar, SR-IOV is a technology that allows you to expose a single physical network device (or more accurately, “physical function”) as multiple virtual functions (VFs). Each VF is a separate logical PCIe device, so a board equipped with an IOMMU should be able to pass them into VMs individually. This allows for basically the same level of performance as a passthrough of an entire PCIe device, but still allows it to be shared amongst many VMs. At the hardware level, it acts as essentially a network switch, instead of doing that in software like with typical virtual NICs.

Now, in my previous post, I talked about some of the new hardware going into my new router. The star of the show is the X10SDV-8C-TLN4F. So why not run OpenWRT baremetal on here? A few reasons:

1. Sometimes, you have to reboot. Rebooting a board like this can take several minutes. Not good for uptime. Meanwhile, a VM takes about 20 seconds to reboot fully. A custom kernel plus bypassing the bootloader (either via efistub or KVM direct kernel boot) could probably get it down to 10 seconds.
2. Lockouts – if you screw up your network configuration, it can be annoying to get back in. Not impossible, since it still has a VGA port and USB ports, but it’s definitely nicer to be able to still access the host and fix the VM from there.
3. OpenWRT isn’t necessarily going to have support for every bit of hardware on a board like this. It may lack drivers, or userland applications necessary to control said hardware. e.g. IPMI tools are lacking.
4. I can easily benchmark routing performance using virtual networks.
5. Since I can run ZFS on the host, I get ZFS snapshots and all that.

But….was it difficult? Was it worth it?

Yes and yes.

Read the rest of this entry »

Yet another SFP+ dual port 10GbE NIC.

Good:

• Works fine. Much less disastrous than the last review.
• Supports manipulating link state for VFs (e.g. so you can have VMs talk to each other even if the external link is down)
• Supports multiple PFs per physical port
• Lots of settings governing the internal switching and such
• If you don’t need SR-IOV, it has an “ultra low latency” mode

Bad:

• Requires standalone program to do configuration. alien worked fine, but be aware that there is no official version for Debian-based systems. However, once you apply a configuration you want to the card, it will persist. Once you get it close enough, you can use sysfs and ip to do the rest (e.g. configuring number of VFs and VF properties).
• Some setting changes require hard reboots.
• Doesn’t appear to support trusted VFs. This unfortunately would make it inappropriate for certain tasks, like the router build.
• Poor support for modern Windows versions. The driver and utilities package installed, but would BSOD consistently.

Overall:

• Not a terrible choice if you just need a good 10GbE NIC for Linux
• Don’t buy for Windows or if you need special SR-IOV features

Not that this is an unpopular take or anything, but this story reinforces reinforces my opinion that cable modems are the single worst piece of network equipment out there.

Several years ago, I had Xfinity’s 40Mbit downstream package. This just wasn’t cutting it, so I upgraded to the 400m down plan – the fastest they had at the time. I even got a promo rate on the upgrade. One minor problem – a rep told me that my current modem, a Cisco, would be sufficient, but this turned out to not be true – it could only do 300-some down. So I did some research, and bought the Arris SB8200. It was on Xfinity’s list of supported modems, and could do 2Gb/s down – plenty of headroom for future upgrades. Of course, since it only has gigabit Ethernet ports, you’d need to use link aggregation to actually get the full 2Gbps. OpenWRT didn’t have great support for bonding back then (it was doable, but not through UCI), but I figured that by the time it became an issue, that would change anyway.

Fast-forward a bit. The fastest speed they offer is now a gigabit. And they’re offering a promotional rate on it! On top of that, my current setup is good for 1g down, so no need to change out any hardware. Seems like a win-win!

Then, a bit later, my speed gets bumped up to 1.2Gb/s down, for free! Unfortunately, while it might have been doable, this meant it was probably time for the Omnia to be demoted to AP duty, and a new primary router would take its place (TODO put link). It’s easy – just set up the new router (which would be a breeze now that UCI supports bonding), enable link aggregation on the modem, and bam, 1200Mb/s down.

And it worked!

Since the router, desktop, and home server all have 10GbE interfaces, I was, for the first time, able to get >1Gbps downstream speeds.

For about 24 hours, that is. Stick around for the clownshow!

Read the rest of this entry »

I found a pair of these dual-port 10GbE NICs for $25, and had no idea what to expect.

The good:

• Excellent secondhand value
• Supports SR-IOV, VM switching, and the other usual stuff
• No SFP lock
• Can be cross-flashed
• Appears to use the same driver (be2net) for both PFs and VFs, unlike Intel cards which have a separate VF driver

The bad:

• There’s a known issue with these where you have to do trickery to bring the ports back up after a power cycle. It doesn’t seem to be OS/driver-specific, because even the BIOS shows “Link Down”. This is the deal breaker. On Linux, you can do it with ethtool -t but on Windows it requires a proprietary program.
• SR-IOV support is toggled on and off in the BIOS. However, the IBM firmware doesn’t seem to expose this. I had to use an HPE or Lenovo image instead.
• Would it kill Broadcom to just let you download the firmware from them, rather than having to peruse every 3rd party vendor site to figure out who has the latest firmware?
• The card is x8 but seems to have some extra pins on the end of the PCIe connector. It seems to work fine in a x16 slot, but does not physically fit in a closed-ended x8 slot.

In short: they’re cheap for a reason. Don’t buy them.

Notes:

You may have to flash twice if the card is running older firmware. There’s also this oddity where the interface has to be up (ip link set up, not physically plugged in) in order to flash – you’d think it would be the other way around.

Back with another niche case. This time, the Goodisory SR01, which I think is almost a perfect case for a high powered router. The use case I’m covering looks something like this:

• Mini-ITX board
• A small amount of active cooling needed (e.g. 25-50W SoC range), but the CPU socket doesn’t support mounting a fan.
• Effectively silent (i.e. shouldn’t be able to hear it at 6ft+)
• External PSU (PicoPSU or motherboard with support for DC-in)
• PCIe slot (ideally 2+ for motherboards with bifurcation support, but 1 is fine)
• No drive bay needed (it will use M.2 or other onboard storage)

• 

• 

• 

Does such a case exist? Read on and find out. Short version since I hate cliffhanger clickbait: it almost does.

Read the rest of this entry »

Overall Rating

8/10, pretty good. Great price and no extreme flaws.

Info

This case is the ??? manufactured by ??? in…..probably China? There’s no branding other than “NAS-8” on the backplane PCB.

Here is the eBay listing that I bought it from: https://www.ebay.com/itm/164928683806

It also appears to be available on AliExpress, along with the 2- and 4-bay versions: https://www.aliexpress.com/item/1005001520370713.html. This link has more pictures available.

I actually plan to use it as a DAS rather than a NAS via an external SAS connection, but it is easier to just put a motherboard in there anyway for power + fan control. There are JBOD “motherboards” available that tell the power supply to turn on and control the fans, but it would cost more money to get one of those than to just throw in a mini-ITX Atom motherboard that I already own.

Read the rest of this entry »

I have a Supermicro Chassis that has the I2C/SMBus connector for the power supply. Unfortunately, the motherboard I am using doesn’t have LOM, so I can’t grab PSU data using the normal methods, and the PMBus support seems to not be working either. Fortunately, the PSU exposes an I2C device which exposes temperature and fan speed. These values might not be correct for every PSU – I had to use a combination of spec sheets for similar PSUs and some reverse engineering to get these values.

#!/bin/bash

address=$1

function getValue () {
        local rawVal=$(i2cget -y 0 $address $1 | cut -c3- | tr a-f A-F)
        local decVal=$(echo "obase=10; ibase=16; $rawVal" | bc -l)
        echo $decVal
}

TEMPERATURE=$(getValue 0x09)
FAN1RAW=$(getValue 0x0a)
FAN2RAW=$(getValue 0x0b)
DCSTATUS=$(getValue 0x0c)
ALARMTEMP=$(getValue 0x0d)
FAN1MIN=$(getValue 0x0e)
FAN2MIN=$(getValue 0x0f)

FAN1=$(echo $FAN1RAW '* 60 / 2 / .262' | bc)
FAN2=$(echo $FAN2RAW '* 60 / 2 / .262' | bc)

echo Temperature: $TEMPERATURE
echo Temp Max: $ALARMTEMP
echo Fan1 RPM: $FAN1
echo Fan2 RPM: $FAN2
echo Fan1 Raw: $FAN1RAW
echo Fan2 Raw: $FAN2RAW
echo Fan1 Min: $FAN1MIN
echo Fan2 Min: $FAN2MIN
echo DC Status Raw: $DCSTATUS

Example:

# ./psu.sh 0x38
Temperature: 33
Temp Max: 75
Fan1 RPM: 4122
Fan2 RPM: 5038
Fan1 Raw: 36
Fan2 Raw: 44
Fan1 Min: 0
Fan2 Min: 35
DC Status Raw: 1

The single argument is the i2c address of the PSU you want to read from. They seem to usually be in the 0x30-0x3f or 0x70-0x7e range. Systems with redundant power supplies will have two separate addresses. In addition, you may need to edit it if your system has multiple i2c buses (change i2cget -y 0 to i2cget -y 1 for example).

Disclaimer: reading/writing random i2c devices can hang your system or cause other issues.