Matt Ventura's blog https://mattventura.net Matt Ventura's blog about various stuff. Mon, 21 Aug 2023 16:50:36 +0000 en-US hourly 1 https://wordpress.org/?v=5.9.9 Quick Fix: grub-probe ZFS error https://mattventura.net/2023/08/21/quick-fix-grub-probe-zfs-error/?utm_source=rss&utm_medium=rss&utm_campaign=quick-fix-grub-probe-zfs-error https://mattventura.net/2023/08/21/quick-fix-grub-probe-zfs-error/#respond Mon, 21 Aug 2023 16:50:36 +0000 https://mattventura.net/?p=677 I ran into an issue with being unable to apt-update because of grub-probe failing to determine my root filesystem type. This wasn’t a ZFS-root system, it had a separate /boot partition. The error was “grub-probe: error: failed to get canonical path of `tank/root’.” Most solutions suggest the use of the ZPOOL_VDEV_NAME_PATH environment variable, but this didn’t fix the problem for me.

I found out that the actual problem was that in my single-drive zpool, a networking hiccup caused it to be marked as degraded. Despite the fact that the pool still worked, the fact that there was no fully-online vdev caused some confusion for grub-probe. Upon clearing the errors with zpool clear, it worked fine.

]]>
https://mattventura.net/2023/08/21/quick-fix-grub-probe-zfs-error/feed/ 0
Repairing a CRS328-24p-4s+ https://mattventura.net/2023/06/21/repairing-a-crs328-24s-4s/?utm_source=rss&utm_medium=rss&utm_campaign=repairing-a-crs328-24s-4s https://mattventura.net/2023/06/21/repairing-a-crs328-24s-4s/#respond Thu, 22 Jun 2023 05:43:43 +0000 https://mattventura.net/?p=660 I scored a used CRS328 for $135 which is dirt cheap for a switch that would normally be $400+. Problem is, ports 17-24 had broken PoE – so broken that RouterOS believed they were non-PoE ports. No pictures since the switch is in use.

Fortunately, the switch has three separate PoE daughterboards, one for each group of eight ports. These are not easy to remove, due to the design of the front panel of the switch. There is a cable underneath each card as well as a pin header, so be careful to not yank cables or damage any connectors. In my case, swapping the second and third caused ports 9-16 to break and 17-24 to work, so it was easy to confirm it was a bad card. If you wish to quickly install a card for testing, you don’t need to plug in the power input nor the power output – only the pin header connector to the mainboard.

Upon inspecting the failed board, I noticed that the component “FB3” seemed to have been knocked off the board. The “FB” designation would typically refer to a “ferrite bead”, but I didn’t have any surface mount ferrite beads on hand, so I replaced it with a jumper wire. Not a good long-term plan, but it was enough to get the card working, which would indicate that the missing ferrite bead was indeed the culprit. The long-term fix would be to find an appropriate replacement bead, or replace the board entirely.

]]>
https://mattventura.net/2023/06/21/repairing-a-crs328-24s-4s/feed/ 0
Quick Fix: AppArmor+Libvirt Errors in Debian, Round 2 https://mattventura.net/2023/06/21/quick-fix-apparmorlibvirt-errors-in-debian-round-2/?utm_source=rss&utm_medium=rss&utm_campaign=quick-fix-apparmorlibvirt-errors-in-debian-round-2 https://mattventura.net/2023/06/21/quick-fix-apparmorlibvirt-errors-in-debian-round-2/#comments Thu, 22 Jun 2023 05:04:07 +0000 https://mattventura.net/?p=664 After smooth sailing for a while after the last post on the subject, I ran into another one. Once again, I was getting errors when trying to start guests.

I was getting error messages such as these in syslog:

2023-06-20T14:14:41.858010-07:00 store libvirtd[8623]: internal error: Process exited prior to exec: libvirt:  error : Cannot delete directory '/run/libvirt/qemu/4-autoserver.shm': Device or resource busy
2023-06-20T14:14:42.060935-07:00 store libvirtd[8623]: internal error: Failed to autostart VM 'autoserver': internal error: Process exited prior to exec: libvirt:  error : Cannot delete directory '/run/libvirt/qemu/4-autoserver.shm': Device or resource busy

/var/log/audit/audit.log showed errors such as these:

type=AVC msg=audit(1687295681.852:196): apparmor="DENIED" operation="umount" class="mount" profile="libvirtd" name="/run/libvirt/qemu/4-autoserver.dev/" pid=9441 comm="daemon-init"

The fix is to add this to /etc/apparmor.d/abstractions/libvirt:

umount /run/libvirt/qemu/**,

Reload apparmor (systemctl reload apparmor) and try starting a guest.

]]>
https://mattventura.net/2023/06/21/quick-fix-apparmorlibvirt-errors-in-debian-round-2/feed/ 1
Switch-Based NVMe Hotplug – a Few Attempts, and one Success https://mattventura.net/2023/04/10/switch-based-nvme-hotplug-a-few-attempts-and-one-success/?utm_source=rss&utm_medium=rss&utm_campaign=switch-based-nvme-hotplug-a-few-attempts-and-one-success https://mattventura.net/2023/04/10/switch-based-nvme-hotplug-a-few-attempts-and-one-success/#comments Tue, 11 Apr 2023 02:48:40 +0000 https://mattventura.net/?p=637 Let’s say you’ve just bought a chassis with an NVMe backplane, or retrofit one into your chassis. Now, it’s time to see if we can get hotplugging and backplane management working.

First of all, PCIe hotplugging is hard. It’s nothing new – after all, PCI hotplug has been around in the form of PCMCIA cards for decades, and PCIe got the same treatment with the later ExpressCard standard. But the reality is that whether it’s a laptop with a card slot, a system with Thunderbolt, or a server with an NVMe backplane, it’s one of those things that you can only expect to work seamlessly if you buy a full OEM system validated for that purpose. If you cobble together a machine from parts, it’s much more difficult to get any sort of PCIe hotplugging working.

I went through this recently after adding a U.2 backplane. Here’s a few things I tried, some of which worked better than others.

For reference, I’m doing this on an X9DRD board, with a BPN-SAS3-826EL1-N4 backplane, and two KCM51VUG1T60 drives. Unlike newer Supermicro boards where you can use a simple redriver or retimer card to get NVMe hotplug, a switch/bridge is likely the only way to get hotplugging working on these.

First Attempt: Oracle NVMe Switch

This is the Oracle NVMe switch card. It goes by a few different model numbers, including 706434 and NVMSW8.

This one’s pretty short: I bought it because someone was selling them for $35 on eBay. It sounded too good to be true, and it was. It didn’t work at all for me, but for that price, I had to try. The card and its downstream ports showed up, but refused to recognize any drives. Supposedly, the card uses some kind of “lane sharing” to squeeze out more performance, but that is only supported on Oracle Linux and RHEL. On other OSes, it’s supposed to still work as a plain old switch, but I could never get it to work. I found some references to an oprom that might have helped, but my card seemed to have neither a UEFI nor BIOS oprom, nor did it have an HII option in the BIOS. Tested it on multiple hosts, same story every time.

Attempt Two: Ceacent ANU24PE08

Bought it here. This one actually worked, but hotplugging was….weird. I couldn’t get it to work right at first. Then I remembered this very informative article about getting Thunderbolt to work in an unsupported system. I saw some lines in dmesg about bus numbering, so I decided to try it in the highest-numbered slot. Unfortunately, on my X9DRD, the BMC VGA controller always took the highest “slot” number on CPU1. The good news is that CPU2 was wide open due to having none of the onboard peripherals connected to it – it’s only connected to raw PCIe slots. This actually got it working!

However, it’s not perfect. For some reason, /sys/bus/pci/slots only shows one or two slots, rather than the four you’d expect. It also had issues with surprise hotplugging and removal. It would sometimes kind of work on one port, but I wouldn’t count on it. You can still “safely eject” a drive by using echo 1 > /sys/bus/pci/devices/x/remove, and can trigger a rescan with echo 1 > /sys/bus/pci/rescan to make it recognize new devices.

Attempt Three: Supermicro AOC-SLG3-2E4

To absolutely nobody’s surprise, using an SMC PCIe switch on an SMC motherboard with an SMC backplane worked the best. I bought one for $41 (and then with my luck, the seller dropped the price to $32 shortly thereafter). I get slot power control, hotplugging works fine in any PCIe slot on either CPU, surprise insertion works, and surprise removal seems to work despite not being officially supported. The only real downside is that it only has two ports rather than four.

The Hard Part: Backplane Management

SMC U.2 backplanes are supposed to support locate/fail as well as telling you when a drive is safe to remove. The AOC-SLG3-2E4 even has a program from SMC made specifically for this card. Let’s give it a shot!

As you can probably guess, it didn’t work. The SMC utilities are built with the PLX SDK, so you have to use a script from the SDK which insmods the PlxSvc.ko and creates the device nodes. Problem is, the bundled version is built against what seems to be a 2015-era kernel, so it wouldn’t load on a 6.x kernel. I downloaded a new version of the PLX SDK, built the driver, and loaded it up with the script. Despite all that, the SMC programs claim the card doesn’t exist.

A bit of Ghidra and GDB later, and I found that yep, it really thinks the card isn’t there. I tried some of the test programs included with the SDK, and they worked fine. Perhaps you need to match the driver version and SDK version? Since the utilities statically link the PLX library, I can’t just run it against a newer version. I changed gears, and instead decided to just pick through the decompiled AOCSLG32E4EnclosureCli code. I found the PLX API calls it was doing, and created a Python script to do the same:

#!/usr/bin/python2

import sys
import os
import sys
import getopt
import time
import datetime
import random
sys.path.insert(0,"../code")

from PlxSdk import *
from const import *
from misc import *
from bar import *

def main():
        SdkLib = LoadLibrary()
        if SdkLib == None:
                print "Library Not loaded"
                sys.exit(0)
        pkey = PLX_DEVICE_KEY()
        pkey.set_bus(84)
        pkey.set_slot(0)
        pkey.set_function(0)
        rc, dp = DeviceOpen(SdkLib,pkey)
        print "rc,dp", rc, dp
        if rc != PLX_STATUS_OK:
                print "Open Failed: %s" % rc
                sys.exit(0)
        rc, data = PlxMappedRegisterRead(SdkLib, dp, 0x2b8)
        print "%s, %X" % PlxMappedRegisterRead(SdkLib, dp, 0x2c0)
        print "rc %s, data %X" % (rc,data)
        # This check specifically doesn't work because 'rc' returns some kind of wrapped integer rather than a normal python type
        if rc != PLX_STATUS_OK:
                pass
                #print "Read Failed: %s" % rc
                #sys.exit(0)
        print "Data old: %X" % data
        #data = (data & 0xff00) | 0x51040000
        data = (data & 0xff0000) | 0x51000100
        print "Data new: %X" % data
        print PlxMappedRegisterWrite(SdkLib, dp, 0x2b8, data)
        print "Wrote part 1"
        # Idk what this is, maybe some kind of 'commit' command?
        print PlxMappedRegisterWrite(SdkLib, dp, 0x2c0, 0x86033)
        print "Wrote part 2"

        DeviceClose(SdkLib, dp)

if __name__ == "__main__":
    main()

I got successful responses from all the API calls, yet no LED. I probably won’t mess around with it more, since there’s not really much left I could try here.

However, I’d argue that you don’t really need these to work to be able to locate drives. If you disable the device (echo 1 > /sys/bus/pci/devices/x/remove), the activity LED turns off (it is normally on and blinks off to signal activity).

But Would It Work on Another System?

No idea. It’s very difficult to say and depends on a large number of factors.

Things to Test

If you’re looking at doing similar testing in another system, here is a non-exhaustive checklist of things you should check:

  1. Boot the system with all the drives attached to make sure that at least basic functionality is working.
  2. Boot the system with no drives attached, then make sure they hotplug correctly. If #1 works but #2 doesn’t, it’s likely that there’s an issue with reserving bus numbers or some other PCI topology issue.
  3. Check that each switch port is correctly exposed in /sys/bus/pci/slots. If this is your only hotplug-supporting bridge, you should see slots ‘0’ and ‘0-1’ for a two-port switch. A four port switch would also have ‘0-2’ and ‘0-3’. You can additionally confirm by reading /sys/bus/pci/slots/x/address.
  4. Coordinated hotplug – check that echo 1 > /sys/bus/pci/devices/x/remove and echo 0 > /sys/bus/pci/slots/x/power work to remove the device from the PCIe topology, and that echo 1 > /sys/bus/pci/slots/x/power and that echo 1 > /sys/bus/pci/rescan will re-add it.
  5. If surprise insertion is desired – with slot power on (echo 1 > /sys/bus/pci/slots/x/power), do you see messages in dmesg and does the drive function after inserting the drive?
  6. Surprise removal – remove the drive unexpectedly – does it remove itself from the topology, and work once again if you re-insert it? You should wait for a while after removing the drive, as rapid removal and insertion of a device is known to confuse some setups.

Other Devices

There are a few other candidates for NVMe PCIe switches:

  1. Linkreal switches (I believe DiLinKer/DiLiVing switches are also rebrands of these): I found a test report that seems to indicate that at least one of their switch models (LRNV9349-8I) supports coordinated insertion. However, it just doesn’t quite have enough information in the test report to make me take a gamble on a card that costs more than double what a similarly-specced Ceacent switch would cost. Plus, the model for which I found the test report is a x16 card, and I need a x8 at most.
  2. Supermicro AOC-SLG3-4E2P: It would probably work fine. The problem is that it’s $200 – as long as I have spare PCIe slots, it’s cheaper to just buy more AOC-SLG3-2E4 cards. Even without getting a great deal, the usual going rate for the AOC-SLG3-2E4 is about $70, and I ended up buying three of them for $100 total.
  3. Just get a newer system. You can get a newer motherboard with NVMe ports on the motherboard, or at least one that supports hotplugging on a redriver/retimer card, which will typically be significantly cheaper. If you’re running an ancient system, you’ll potentially run into bottlenecks anyway when using NVMe. Even a single-socket first gen Epyc would be a huge upgrade compared to a system like this, and doesn’t cost all that much.
]]>
https://mattventura.net/2023/04/10/switch-based-nvme-hotplug-a-few-attempts-and-one-success/feed/ 2
Upgrading an SC847 with a rear 2×2.5″ Drive Cage https://mattventura.net/2023/04/08/upgrading-an-sc847-with-a-rear-2x2-5-drive-cage/?utm_source=rss&utm_medium=rss&utm_campaign=upgrading-an-sc847-with-a-rear-2x2-5-drive-cage https://mattventura.net/2023/04/08/upgrading-an-sc847-with-a-rear-2x2-5-drive-cage/#respond Sat, 08 Apr 2023 18:53:17 +0000 https://mattventura.net/?p=614 I read this post about upgrading an older SC826 to support the rear drive cage option, and wondered if I could do the same with an SC847. The newer ‘B’ models support this natively, but there are still tons of cheap non-B models out there.

The first question is “why”? To which there are several answers:

  • More drives! Free up a couple 3.5″ bays for 3.5″ drives, rather than using an entire 3.5″ bay for a 2.5″ drive.
  • Cheapest NVMe option. $60-65 for the upgraded motherboard tray, and $80-90 for the drive cage, compared to $200 or so for the 4x U.2 rear backplane.
  • More NVMe. You could get both the 4x U.2 rear backplane (BPN-SAS3-826EL1-N4) and the rear 2x NVMe cage for 6 NVMe bays. (There is also an 8x U.2 front backplane, but it’s hard to find). for a little over $200. The 8x U.2 front backplane is much more difficult to find. Maybe 4 NVMe bays isn’t quite enough for you.
  • Dual-expander (EL2) backplanes: There are no backplane options with both dual SAS expanders and NVMe support.

Enough intro, I was able to get this conversion working, and it was much easier than the 826 conversion (though took a lot longer). Here’s how.

There are three main things you will need to do this conversion:

  • MCP-220-84708-ASM: This is the motherboard tray. In the 847, the motherboard sits on a tray which can slide out of the back of the chassis (you’ll need to pull the blue knob again once it hits the end of its travel, and then it can slide all the way out). This includes the rear window.
  • MCP-220-82616-0N (SAS/SATA only) or MCP-220-82619-0N (NVMe ONLY – not hybrid!): The actual drive cage.

There is another item worth mentioning: MCP-240-82608-0N. This is the rear window – the less-wide version which will have room for the rear cage. However, this is included in the MCP-220-84708-ASM – I only mention this because the tray takes so long to acquire that you may wish to just go for the quick and dirty route and do something more akin to the post linked in the first paragraph.

There are also some things you may wish to buy or modify on your own:

  • Air Shroud: The older air shroud that you’d get with one of these won’t have space for the rear bays. You’ll need to either dremel it, buy a newer one, or not use an air shroud.

The motherboard tray is the problem here. It only costs about $60-65, but this is a niche part, so it’s very rare to find a seller that keeps these in stock. Most vendors order it from the manufacturer as needed, so expect it to take months. I ordered mine in January and received it in April. Furthermore, I couldn’t find a single picture of the part anywhere, so I had to deduce that it was the correct part by comparing parts lists from different SC847 models. I also ended up with a spare rear window since I had ordered that separately, not knowing that it was included.

Finally, the cages themselves range from $60 – $100. Since you’ll be patiently waiting for the motherboard tray, take your time and wait for a good deal.

With any luck, you’ll be in and out for less than $150.

The procedure is fairly simple. First, completely remove the old motherboard tray. Disconnect the cables from the motherboard, pull the blue knob, and slide the tray out as far as it will go. It may require some force. When it gets stuck further down, you need to pull the blue knob again to allow the tray to completely exit the case.

Next, before doing anything else, put the new tray in as a fit check. This is a good idea any time you’re dealing with rails, since shipping damage can cause rails to become bent enough to not slide freely.

At this point, you can leave it in, or take it back out to make it easier to install everything else.

First, check the locations of standoff nuts. Unlike typical cases, where standoffs screw into the tray, the tray comes with standoff nuts which screw onto screws built into the tray (i.e. instead of a female side on top, and a male side on the bottom, they’re female on both ends). Make sure you only have them installed in places where your motherboard actually has a hole, otherwise, you might short something. You’ll also need one where the drive cage will go, shown here:

Remove the three screws holding the placeholder window to the tray, and install the drive cage (it only uses the two screws on the top, not the one on the bottom).

After that, it’s pretty standard stuff. Remember to install the I/O shield first, and then install the motherboard and expansion cards. Finally, re-connect cables (including the new cables for your drive cage).

One thing I was a little disappointed about was that the BPN-SAS3-826EL1-N4’s SAS expander does not support SGPIO sideband on cascade ports, so while drives will work if you do that, you won’t be able to control their LEDs. To get full support for this, you’ll need to connect them to a SAS or SATA controller that supports such sideband signals for direct-attached drives.

So, in conclusion, wait times aside, this is a very cost effective way of upgrading to a ‘B’ model chassis. Normally, you’d be paying much more than a $60-65 premium to get a B model.

]]>
https://mattventura.net/2023/04/08/upgrading-an-sc847-with-a-rear-2x2-5-drive-cage/feed/ 0
Restoring eBay’s Sale History Link https://mattventura.net/2023/04/07/restoring-ebays-sale-history-link/?utm_source=rss&utm_medium=rss&utm_campaign=restoring-ebays-sale-history-link https://mattventura.net/2023/04/07/restoring-ebays-sale-history-link/#respond Fri, 07 Apr 2023 16:06:00 +0000 https://mattventura.net/?p=631 It’s very useful to be able to see the sale history for an item that isn’t yours. You might want to see how quickly it sells, or whether offers are likely to be accepted or rejected. Unfortunately, eBay seems to have recently removed this link. The good news is that the page still exists and can be accessed via the same URL as before. Here’s a GreaseMonkey script to turn the “x sold” text into a clickable link like it previously was:

// ==UserScript== 
// @name     Restore eBay sold items link 
// @version  1
// @grant    none
// @match *://*.ebay.com/itm/*
// ==/UserScript==

element = document.querySelector("div.d-quantity__availability span:last-child")
text = element.textContent
re = /(.*)\/itm\/([0-9]+).*/
url = document.location.href.replace(re, '$1/bin/purchaseHistory?item=$2')
element.innerHTML = '<a href="' + url + '">' + text + '</a>'

I have not tested it on other userscript plugins. The end result looks like this:

Clicking the link takes you to the sale history page:

]]>
https://mattventura.net/2023/04/07/restoring-ebays-sale-history-link/feed/ 0
Broadcom 9400 – Should You Buy One for a Homelab? https://mattventura.net/2023/04/03/broadcom-9400-should-you-buy-one-for-a-homelab/?utm_source=rss&utm_medium=rss&utm_campaign=broadcom-9400-should-you-buy-one-for-a-homelab https://mattventura.net/2023/04/03/broadcom-9400-should-you-buy-one-for-a-homelab/#respond Mon, 03 Apr 2023 17:40:27 +0000 https://mattventura.net/?p=618 The 9400 series is LSI/Avago/Broadcom’s first “Tri-Mode” HBA, capable of supporting SAS, SATA, and NVMe all in one adapter. There’s a few catches, but despite that, it might still be worth the buy depending on your circumstances.

U.2 Tri-Mode – Not What it Seems

First, what does “Tri-Mode” really mean in this context? It may not mean quite what you think it does.

The 9400 supports U.2 NVMe drives. The first annoyance is that you have to buy their expensive cables, although DiLinKer makes a few cheaper cables for certain connector types. This is because the pinouts of SAS/SATA and PCIe are not directly interchangeable. However, the real issue is that U.2 tri-mode really just doesn’t make much sense.

In U.2, completely disjoint sets of pins are used for SAS/SATA and NVMe. This is why, on a U.2 NVMe/SAS/SATA hybrid backplane, there will be a separate set of connectors for SAS and NVMe. For example, if we have an 8-slot backplane, where 2 of the slots also support U.2, we’ll end up with two 4-lane SAS connectors, and two 4-lane PCIe connectors, all of which need to be plugged back into an HBA. So we use the four ports on our 9400-16i – two for SAS, two for NVMe, problem solved, right?

Well, yes. But in terms of homelab secondhand prices, it would have been significantly cheaper to buy an 8-lane SAS HBA and 8-lane PCIe redriver/retimer/switch/whatever – $70+70 (or even less depending on what you buy) versus $280. This is the problem with “U.2 Tri-Mode” – in our example, two of the cables will only ever carry PCIe, and two of the cables will only ever carry SAS. So what good did the tri-mode HBA do, when there’s no single line that requires switching modes? I’ve seen it jokingly called “try-mode”, and that pretty much sums it up.

To be clear, it’s not U.2 at fault here – there is a perfectly valid reason to keep the SAS/SATA and PCIe lanes completely separate: it, quite ironically, eliminates the need for a tri-mode controller because the SAS lanes can be connected to a pure SAS controller while the PCIe lanes can be connected to the PCIe bus/HBA, and neither one of them needs to understand the other. But this also means that a tri-mode controller is in an odd position.

U.3 fixes this – it uses the same sets of pins for SAS, SATA, and NVMe (yes, theoretically, you could have a quad-lane SAS drive – whether anyone will make one remains to be seen). If you have an eight-port non-expander backplane, you’d need eight connections back to an HBA, and each of them would actually be capable of carrying SAS or NVMe at any given time, depending on what drive is inserted. Thus, a tri-mode HBA is actually warranted (assuming the backplane isn’t responsible for splitting out the connections) with U.3. But not U.2.

So Is the Card a Bad Buy?

Despite the tri-mode rant, the card itself isn’t necessarily a bad value. For one, the card is still perfectly fine as a plain SAS HBA. In fact, Broadcom doesn’t seem to be interested in making plain SAS/SATA HBAs anymore – their current HBA lineups only have tri-mode and pure NVMe solutions. I had some issues making it talk to SGPIO direct-attach backplanes, but if you’re using an expander or no backplane whatsoever, it works. It’s more expensive than the 9300-8i on a per-port basis, but it has better port density. As for 16i cards, the 9300-16i is bulky (no low-profile option), runs hot, and uses a lot of power – so much power that it requires a 6-pin auxiliary power input! The 9305-16i fixes these problems – no extra power connector, and has 16i and even 24i low-profile versions. But in terms of secondhand prices, the 9305-16i is not any cheaper than the 9400-16i – they’re both $270 or so, and the 9400 has some AliExpress generic versions for even less. The 9400 is also good if you’re tight on PCIe slots, and need your NVMe and SAS combined into one slot.

Furthermore, because the card exposes NVMe drives as SAS devices, you can get NVMe boot support on systems that don’t natively support such. It also means that you can have full NVMe hotplug support even if neither your hardware nor OS would otherwise support PCIe hotplugging. Due to the drives being isolated from the system’s PCIe topology, you’re also less likely to bork something with a surprise removal. However, all of these advantages come with a downside: potentially lower performance compared to the other NVMe options, and inability to manage drives using NVMe-specific tools.

There are other niche advantages as well. For example, this makes passing multiple NVMe drives into a VM easier, as you can simply pass the entire card in. Thus, even if you’re adding or removing drives on the fly, you won’t need to update your configuration to pass in each individual drive.

Overall, you’ll have to evaluate based on your own needs. I wouldn’t buy one if I only needed an 8i SAS controller, or wanted to use NVMe on a more modern system with good hotplug support. But if you legitimately need that many SAS lanes on a single card, or you need the excellent hotplugging support offered by the 9400, then go for it.

Tips

If you are planning on buying one and making use of NVMe, I recommend buying the DiLinKer direct attach cable. Even if you’re planning on using a backplane, having a direct cable can assist with troubleshooting, since you can cut both the backplane as well as your other cables out of the picture entirely.

]]>
https://mattventura.net/2023/04/03/broadcom-9400-should-you-buy-one-for-a-homelab/feed/ 0
Quick Bash Tip: Alt-Shift-3 (Alt-#) https://mattventura.net/2022/12/11/quick-bash-tip-alt-shift-3-alt/?utm_source=rss&utm_medium=rss&utm_campaign=quick-bash-tip-alt-shift-3-alt https://mattventura.net/2022/12/11/quick-bash-tip-alt-shift-3-alt/#respond Sun, 11 Dec 2022 19:07:36 +0000 https://mattventura.net/?p=602 Alt-Shift-3 in Bash inserts a ‘#’ at the beginning of the line, and then runs it. # is the comment character, so running it does nothing. So why would you want to do this?

The simplest use case is when you have a command typed out, but you realize there’s another command you need to run first. By pressing alt-#, you push the command into your history, so you can quickly recall it after running whatever other commands first.

Example:

# Set some dataset properties...but wait, what was the name of the dataset again?
# I have the whole line except for the dataset ready, so press Alt-#
$ #zfs set primarycache=none secondarycache=metadata tank3/
# It's in my bash history now, so I can run whatever other commands first
$ zfs list
...
# Okay, figured it out, so now I press up-arrow/^P (or search with ^R) to recall the command, then home/^A to go to the start of the line, then del/^D to remove the #
$ zfs set primarycache=none secondarycache=metadata tank3/path/to/my/data/set

]]>
https://mattventura.net/2022/12/11/quick-bash-tip-alt-shift-3-alt/feed/ 0
Quick Fix: AppArmor+Libvirt Errors in Debian https://mattventura.net/2022/09/15/quick-fix-apparmorlibvirt-errors-in-debian/?utm_source=rss&utm_medium=rss&utm_campaign=quick-fix-apparmorlibvirt-errors-in-debian https://mattventura.net/2022/09/15/quick-fix-apparmorlibvirt-errors-in-debian/#respond Fri, 16 Sep 2022 01:45:48 +0000 https://mattventura.net/?p=598 A few months ago, I was getting some permissions errors when trying to create new VMs with virt-install. My main symptom was that I would get this error:
Could not open '/var/lib/libvirt/qemu/nvram/openwisp_VARS.fd': Permission denied
I recently tracked it down to an AppArmor profile issue. Fixing it was simple. I appended the following to /etc/apparmor.d/abstractions/libvirt:

  /var/lib/libvirt/qemu/nvram/** rwk,  
  /usr/share/OVMF/** rk,

This will also fix some other errors, such as the ‘Failed to lock byte 100’ and ‘cannot load apparmor profile’ errors.

]]>
https://mattventura.net/2022/09/15/quick-fix-apparmorlibvirt-errors-in-debian/feed/ 0
2.5GbE Direct Attach SFP on the Omnia https://mattventura.net/2022/06/30/2-5gbe-direct-attach-sfp-on-the-omnia/?utm_source=rss&utm_medium=rss&utm_campaign=2-5gbe-direct-attach-sfp-on-the-omnia https://mattventura.net/2022/06/30/2-5gbe-direct-attach-sfp-on-the-omnia/#respond Thu, 30 Jun 2022 21:54:52 +0000 https://mattventura.net/?p=595 I was able to get my Omnia to talk to a MikroTik CRS305 at 2.5Gbps via an SFP DAC, in this forum post.

Short version: You need to force 2.5Gbps on both sides (disable auto-negotiation). Doing so requires the other side to have a switch/NIC ASIC that can talk to the SFP+ module at 2.5Gbps (this is not always supported – even if you were to run 2.5GBASE-T off a transceiver, the ASIC and transceiver may still be communicating at a higher rate). In other words, this isn’t going to work on all hardware. For the long version, read on.

Omnia Side

First, you’ll need to switch your Omnia to the SFP boot image. In the /boot directory, you’ll notice two images (*.dtb files), and a symlink to one or the other:

root@omnia:/boot# ls -ladrwxr-xr-x    1 root     root           158 May 18 08:33 .drwxr-xr-x    1 root     root           134 May 18 08:33 ..-rw-r--r--    1 root     root         19290 May 16 01:15 armada-385-turris-omnia-phy.dtb-rw-r--r--    1 root     root         19286 May 16 01:15 armada-385-turris-omnia-sfp.dtb-rw-r--r--    1 root     root          1199 Apr 29 08:12 boot.scrlrwxrwxrwx    1 root     root            31 May 13 11:41 dtb -> armada-385-turris-omnia-sfp.dtb-rwxr-xr-x    1 root     root       3881560 May 18 01:28 zImage

If the dtb symlink is pointing to the phy image, remove the symlink, and re-create it to point to the sfp image.

You’ll also need to add a line to your /etc/rc.local (before the ‘exit 0’ line):

ethtool -s eth2 speed 2500 autoneg off

Reboot after making these changes.

MikroTik Side

First off, I couldn’t figure out a way to get this to work on SwOS, which is a shame since I prefer it over RouterOS for simple switch configuration, where I don’t need any of the advanced functionality of ROS. You’ll need at least RouterOS 7.3.1, and of course a switch ASIC that supports 2.5Gbps.

In Interfaces, open up whichever interface you’d like to use. In my case, sfp-sfpplus3. On the ‘Ethernet’ tab, turn off Auto Negotiation, and select 2.5Gbps. Note that just because you see an option does not mean your hardware supports it – the CRS305 certainly doesn’t do 25/40/100Gb.

Once that’s done, everything should be working!

# iperf3 -c 192.168.1.1
Connecting to host 192.168.1.1, port 5201
[  5] local 192.168.1.4 port 43936 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   282 MBytes  2.36 Gbits/sec    0    814 KBytes
[  5]   1.00-2.00   sec   279 MBytes  2.33 Gbits/sec    0    947 KBytes
[  5]   2.00-3.00   sec   279 MBytes  2.35 Gbits/sec    0    947 KBytes
[  5]   3.00-4.00   sec   278 MBytes  2.33 Gbits/sec    0   1.06 MBytes
[  5]   4.00-5.00   sec   278 MBytes  2.33 Gbits/sec    0   1.11 MBytes
[  5]   5.00-6.00   sec   280 MBytes  2.35 Gbits/sec    0   1.11 MBytes
[  5]   6.00-7.00   sec   275 MBytes  2.31 Gbits/sec    0   1.22 MBytes
[  5]   7.00-8.00   sec   276 MBytes  2.32 Gbits/sec    0   1.30 MBytes
[  5]   8.00-9.00   sec   280 MBytes  2.35 Gbits/sec    0   1.30 MBytes
[  5]   9.00-10.00  sec   275 MBytes  2.31 Gbits/sec    0   1.30 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.72 GBytes  2.33 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  2.71 GBytes  2.33 Gbits/sec                  receiver

iperf Done.
]]>
https://mattventura.net/2022/06/30/2-5gbe-direct-attach-sfp-on-the-omnia/feed/ 0