SDLMUX Revisited

The SDLMUX software combines two network adapters into one IP interface, providing a fail over capability if the link on the active adapter goes down or the adapter fails. Once set up it requires virtually no administration but there are some things you should be aware of.

First, SDLMUX does not provide any load balancing; all traffic is transmitted out of and received on the active adapter.

Second, on the ftServer VSeries hardware the active adapter has a MAC address of the form 00:00:A8:4v:wx:yz. The 00:00:A8 is the Stratus organizationally unique identifier (OUI), assigned by the IEEE. The “v” is an index based on the order that the SDLMUX devices where initialized, starting at 0. The value “wx:yz” is based on the system serial number. The standby address differs from the active address only in the upper nibble of the fourth byte, instead of a 4 it is a 6. Also the difference between the MAC addresses of two active (or standby) adapters on the same module will only be in the lower nibble of the fourth byte. The easiest way to get a list of all the MAC addresses is with the analyze_system request “dump_sdlmux” matching on “MAC”, see figure 1.

Just the MAC addresses aren’t very helpful; by matching on “MAC” or “#” you get the MAC addresses and the SDLMUX and network adapter device names. Note that this list will not contain any network adapters that are not part of an SDLMUX partnership.

Third, SDLMUX network adapters send Ethernet 802.2 LLC frames to their partners to insure that the network path is functioning correctly. Five sets of these test frames go out at three second intervals, followed by a 33 second interval. Trace 1 shows three cycles of this pattern. Trace 1 also shows the actual frame from the active adapter (frame 1) and the standby adapter (frame 2). I’ve highlighted the frame following the 33 second gap just to make the cycles easier to read. These are not Ethernet type II frames or IP packets and the switches connected to the network adapters and any switches along the path between the two adapters must have a configuration that does not block these 802.2 LLC frames.

Note, that you cannot use packet_monitor to see these test frames. The frames are sent below the point where packet_monitor taps into the stack and SDLMUX removes the frames from the stack before packet_monitor can read them.

Fourth, if each adapter is connected to a different switch and the link between those switches fails or for some other reason the test frames do not get though you will see something that looks like figure 3 in the syserr_log. Starting in release 16.2.1ak and 17.0.0ah if there is a test frame failure SDLMUX triggers an ARP request to the last host that was successfully ARP’ed over the suspect interface. If it gets an answer it knows that the active adapter is working so it resets the standby adapter to try to get it to work. If there is a network issue blocking the test frames resetting the adapter doesn’t help and the adapter eventually goes MTBF, see time stamp 08:08:10 in figure 3. The dlmux_admin command will report the broken adapter as DOWN, figure 4, and a trace will show no test frames since at this point there is nothing to test. If an ARP reply is not received or the problem is on a release that does not send the ARP request SDLMUX will fail over the adapters and break the new standby adapter. If during the next cycle the test frames again fail and no ARP reply is received the fail over and break will be done again. Eventually one adapter will go MTBF.

To bring the adapter back into service you need to add it back with the board_admin command. Of course if you do that without correcting the underlying problem the adapter will just MTBF again.

Fifth, since the active adapter always has the same MAC address the effect of a fail over is to move the MAC address from one switch port to another. Any security settings on the switch ports must allow this change. Also switch ports can be configured to talk to other switch ports, negotiating various settings. These can be triggered when the switch notices a change in topology, like a new MAC address or a link being restored. Until these settings are negotiated a switch may not pass regular frames. The switch ports connected to the SDLMUX adapters should be configured not to perform these negotiations. In addition the switch ports connected to the SDLMUX adapters should be configured not to run the spanning tree protocol or to skip the learning and listening steps (this is called portfast by Cisco). During the learning and listening steps the switch will not pass regular data frames. In extreme cases the delay caused by these settings can be so long that SDLMUX triggers another fail over.

Finally, the link status of the adapters needs to be monitored to be sure that both links are up. The failure of a link will not cause the system to call home and because its fault tolerant there is no loss of system connectivity I covered this in a previous blog post, see Monitoring network adapter status.