I was able to get my Omnia to talk to a MikroTik CRS305 at 2.5Gbps via an SFP DAC, in this forum post.

Short version: You need to force 2.5Gbps on both sides (disable auto-negotiation). Doing so requires the other side to have a switch/NIC ASIC that can talk to the SFP+ module at 2.5Gbps (this is not always supported – even if you were to run 2.5GBASE-T off a transceiver, the ASIC and transceiver may still be communicating at a higher rate). In other words, this isn’t going to work on all hardware. For the long version, read on.

Omnia Side

First, you’ll need to switch your Omnia to the SFP boot image. In the /boot directory, you’ll notice two images (*.dtb files), and a symlink to one or the other:

root@omnia:/boot# ls -ladrwxr-xr-x    1 root     root           158 May 18 08:33 .drwxr-xr-x    1 root     root           134 May 18 08:33 ..-rw-r--r--    1 root     root         19290 May 16 01:15 armada-385-turris-omnia-phy.dtb-rw-r--r--    1 root     root         19286 May 16 01:15 armada-385-turris-omnia-sfp.dtb-rw-r--r--    1 root     root          1199 Apr 29 08:12 boot.scrlrwxrwxrwx    1 root     root            31 May 13 11:41 dtb -> armada-385-turris-omnia-sfp.dtb-rwxr-xr-x    1 root     root       3881560 May 18 01:28 zImage

If the dtb symlink is pointing to the phy image, remove the symlink, and re-create it to point to the sfp image.

You’ll also need to add a line to your /etc/rc.local (before the ‘exit 0’ line):

ethtool -s eth2 speed 2500 autoneg off

Reboot after making these changes.

MikroTik Side

First off, I couldn’t figure out a way to get this to work on SwOS, which is a shame since I prefer it over RouterOS for simple switch configuration, where I don’t need any of the advanced functionality of ROS. You’ll need at least RouterOS 7.3.1, and of course a switch ASIC that supports 2.5Gbps.

In Interfaces, open up whichever interface you’d like to use. In my case, sfp-sfpplus3. On the ‘Ethernet’ tab, turn off Auto Negotiation, and select 2.5Gbps. Note that just because you see an option does not mean your hardware supports it – the CRS305 certainly doesn’t do 25/40/100Gb.

Once that’s done, everything should be working!

# iperf3 -c 192.168.1.1
Connecting to host 192.168.1.1, port 5201
[  5] local 192.168.1.4 port 43936 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   282 MBytes  2.36 Gbits/sec    0    814 KBytes
[  5]   1.00-2.00   sec   279 MBytes  2.33 Gbits/sec    0    947 KBytes
[  5]   2.00-3.00   sec   279 MBytes  2.35 Gbits/sec    0    947 KBytes
[  5]   3.00-4.00   sec   278 MBytes  2.33 Gbits/sec    0   1.06 MBytes
[  5]   4.00-5.00   sec   278 MBytes  2.33 Gbits/sec    0   1.11 MBytes
[  5]   5.00-6.00   sec   280 MBytes  2.35 Gbits/sec    0   1.11 MBytes
[  5]   6.00-7.00   sec   275 MBytes  2.31 Gbits/sec    0   1.22 MBytes
[  5]   7.00-8.00   sec   276 MBytes  2.32 Gbits/sec    0   1.30 MBytes
[  5]   8.00-9.00   sec   280 MBytes  2.35 Gbits/sec    0   1.30 MBytes
[  5]   9.00-10.00  sec   275 MBytes  2.31 Gbits/sec    0   1.30 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.72 GBytes  2.33 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  2.71 GBytes  2.33 Gbits/sec                  receiver

iperf Done.

This entry was posted on Thursday, June 30th, 2022 at 13:54 and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.