All About Backplanes: Part 1: Types of SATA/SAS Backplane Management
Server backplanes are a thing I care a little too much about. Despite them being a mainstay of most servers, there isn’t really a comprehensive guide about them. Now, there is.
I would categorize backplanes into three main types of logic levels:
- Passive
- Sideband-managed
- Expander-based
Passive Backplanes
Passive backplanes have no logic directly relating to drives. They may have logic such as fan control, but it is performed independently of the drives. The most “communication” you get is that the backplane may have a per-drive presence and/or activity LED. The activity LED is driven by the drive itself, via a pin on the SATA power connector.
The primary advantage of a passive backplane is that it is the lowest-cost option. Most “mobile racks” – devices that occupy one or more 5.25″ bays and provide hot-swap bays in return – are of this variety. In addition, because they do not have any electronics in the data path between the controllers and disks, you may be able to use newer generations than what the backplane was designed for. For example, a backplane designed for SATA3/SAS2 (6gb/s) may very well be able to accommodate a SAS3 (12gb/s) or even SAS4 (22.5gb/s) drive.
The downside is that they have no management functionality whatsoever. For example, if a disk fails, the backplane will not be of any help in identifying which slot contains the failed disk. The onus is on you to figure that out – you could try interpreting port numbers (but this has its own pitfalls), or proactively print labels for every drive bay with the drive’s serial number. There are also certain workarounds, like issuing ‘dd’ commands to drives to force the activity LED to turn on.
Passive backplanes may use individual SATA data connectors, or use one of the many types of multi-lane SAS connectors. The SAS connectors are still usable in SATA-only setups, but you may need to purchase a “reverse breakout” cable (as opposed to a normal/forward breakout) in order to plug individual SATA connectors on a motherboard or add-in card into your backplane.
Sideband-Managed Backplanes
Backplanes with “sideband management” have a management chip that allows for, among other possible features, control over individual disk status LEDs. You can generally manually “locate” a drive (i.e. cause an LED, usually red in color, to blink on the drive bay of the requested drive), and RAID software or hardware can automatically illuminate the LED of failed drives. This is the primary advantage. Intel’s ‘ledmon’ package contains both ‘ledmon’, which provides automatic monitoring of software RAID and zpools, as well as the ‘ledctl’ utility which allows manual control of LEDs so that you can locate drives on-demand.
Similar to passive backplanes, the management functionality is not in the “hot path”, but rather runs parallel to it – hence the name “sideband” – so it will not be a limiting factor in speed or forwards-compatibility. The advantage is that you get the ability to identify drives. The primary downside is cost. In addition, it is difficult to find “mobile racks” with management functionality. The few that do exist may have compatibility issues with standard PC cases. They generally lack the “notch” in between each 5.25″ bay. You may need to modify your case, or 3D print some custom parts.
Another downside is controller compatibility. First, you need a controller that actually has sideband capability in the first place. Consumer-grade motherboards will generally not have any sort of sideband header to use with their onboard SATA controller. Server-grade boards might give you access to the sideband pins of the integrated SATA (look for a header labeled “SGPIO”). SAS HBAs are generally the only device that is guaranteed to support it. When using a multi-lane connector, such as SlimSAS, iPass, or MCIO, the sideband signals are part of the cable already. When using a breakout cable, you need one that specifically has the extra sideband connection – a 4-lane cable will break out to 4 individual SATA connectors, and one sideband connector (8+2 for an 8-lane cable).
However, you’re not out of the water yet. There are different types of sideband protocols (i2c, SGPIO, UBM, and more), and there are still potential incompatibilities within a single controller. For example, I have found that SAS30xx generation and below controllers work well with Supermicro’s SAS3 and below backplanes, but not the newer backplanes (such as the BPN-NVMe5-LA26-S12). Meanwhile, many of the newer controllers, like the SAS38xx are compatible with the newer backplanes, but not the older ones. I had a SAS34xx that didn’t want to manage any sideband backplane I plugged it into!
One possible workaround is to find a standalone SAS expander that is compatible with your backplane’s management, and pipe through that. For example, I find that the sub-$25 Adaptec 82885T works great with my SAS3 and below backplanes.
As mentioned briefly, sideband backplanes can use individual SATA connectors plus a separate sideband connection, or a multi-lane connector. However, I strongly recommend going for a multilane connector on both the HBA side and the backplane side. Proper drive slot mapping requires the cables to all be connected to the proper ports, otherwise the wrong LEDs will be activated. Since the sideband management chip is not in the hot path, it does not know what drive is in what slot. It simply receives a command like “activate failure LED on slot 2” – if you plugged slot 2 on the backplane into slot 1 on the controller, then you’re about to pull the wrong drive. Not to mention, consolidated multi-lane cables are just a lot cleaner to work with, as they replace 5 or 10 connectors (including sideband) with a single cable.
Expander Backplanes
Finally, we have the heavyweight category – expander backplanes. Expander backplanes are exactly what they sound like – a backplane with an integrated SAS expander. Sometimes two, since SAS drives can be dual-ported, providing secondary data path for redundancy. All communication between the HBA and the drives goes through the expander. The expander is in fact a SAS device itself – think of it like a network switch. The management is performed by the expander, and is usually a bit more expansive. For example, backplanes with fan headers and temperature sensors may allow you to access those readouts through the expander.
The expander knows its own ports, so you can directly query which drive is in which slot. You can also tell it to activate LEDs, and there is no possibility of getting the wrong slot like with backplanes that use individual SATA ports.
The advantage of expander backplanes is that they tend to have the highest degree of compatibility. The expander is just another SAS device – if the HBA and OS recognize it, that’s generally all you need.
Another advantage is that this is the only option which reduces the amount of other equipment needed. You only need a single HBA, which you can connect to the backplane with as little as a single cable, though you may prefer more if you are running SSDs and don’t want to bottleneck them. Expander backplanes may also have “cascade ports” – extra ports which you can use to connect to another backplane. This further saves on HBAs. You can even make use of these extra ports plus adapters for external cabling to cascade to other chassis, in a JBOD manner.
There are three main disadvantages of an expander backplane, all of them stemming from the fact that the expander is an active component which sits in the data path. In other words, it is “in-band” management rather than “sideband”.
The first disadvantage is that this means you cannot use a SATA controller with them. Even if you plan to only use SATA drives, the controller needs to speak SAS to the expander in order to access any downstream drives. If you try to plug in a SATA controller, neither the drives nor the expander will be recognized.
The second disadvantage is that you are locked to whatever SAS/SATA generation the expander supports. If you get an older BP with a SAS2 expander, there is no possibility of getting SAS3 or SAS4 speeds. Fortunately, many server chassis allow you to upgrade backplanes, so a chassis with an old backplane is not necessary a dealbreaker. Furthermore, the industry has generally moved away from SAS SSDs and towards NVMe SSDs for fast storage. SAS is still great for spinners, and spinners don’t require much speed.
The third disadvantage is power. The expander is an active component with power draw and cooling needs. The good news is that it might still be more efficient than the alternatives, when you have a lot of drives. For example, if you have a 36-bay dual-backplane chassis, like the Supermicro 847 or clone thereof, you would need a total of nine 4-lane HBA ports. If you’re using 8-lane SAS HBAs, you would need five of them. Compare that to using expander backplanes, where you could use a single HBA and 2-4 cables thanks to the cascade ports on the backplanes. For that kind of setup, trying to save money and power budget by not using expander backplanes will likely cost you more in the end. However, if you only need eight drives, then the expander might do more harm than good.
Conclusion
I hope that clears things up a bit. In short:
- Unmanaged: no non-trivial logic whatsoever. Maybe activity lights. Can use single SATA connectors or multilane connectors.
- Sideband: supports management, but card compatibility may be a problem. Can use single SATA connectors or multilane connectors, but multilane is preferred as it it eliminates a possiblemistake.
- Expander: the heavyweight solution. Overkill for small setups, but simplifies larger (>8 disk) setups. Best compatibility with SAS HBAs, but not compatible with SATA controllers.