Switching: MultiLayer Switching


This article continues the series on LAN switching and Cisco Catalyst switches.

MultiLayer Switching (MLS) is a key component of high performance at Layer 3 with the Catalyst 5000 switches. It also is a marvelous opportunity to review what happens when packets pass through a router. Understanding this is very useful in the understanding router and switch performance optimization(s), also in troubleshooting network design and configuration issues.

Prior articles on switching or related topics:


What is MultiLayer Switching?


MultiLayer Switching (MLS) is where a Layer 2 switchblade does Layer 3 (or pseudo-Layer 3) forwarding of packets between subnets. You can find out about this in the recent documentation by looking for Layer 3 Switching Guides under the 5000 or 6000 models.

The particular form of Layer 3 switching (hardware-assisted routing) we’ll be looking at is implemented with the Netflow Feature Card (NFFC) functionality in Catalyst 5000 series switches. This requires that you have either the Supervisor III engine with NFFC  daughter-board or the Supervisor IIG or IIIG card (no daughter-board required). It also requires that you have either a suitable external router or the internal Route-Switch Module (RSM) blade (or Route-Switch Feature Card — RSFC — with the Sup IIG or IIIG blade). Suitable external routers are those implementing the router component of MultiLayer Switching, which is now all the models of larger size / more capability, equipped with an IOS image that is MLS-capable. The online documentation specifies that this is the Cisco 7500, 7200, 4700, 4500, or 3600 series router with an IOS image that supports IP MLS. (IPX MLS is also available but of less general interest. Ditto, IP multicast MLS).

The MLS functionality is also supported (with greater simplicity, and fewer configuration options) in the 6000 series!

To understand what’s going on, let’s first talk through what happens when a packet is forwarded by a router. There are lots that go on in addition to what follows, but let’s not muddle things with unrelated detail. These steps are the ones we’re interested in:

  • router interface hardware receives the bits and checks the checksum
  • router strips off the Layer 2 header (de-encapsulates the packet)
  • the router determines outbound interface and next hop IP address from the routing table
  • the router builds new Layer 2 header
  • new Layer 2 header and payload are assembled and queued for the outbound interface
  • the outbound interface sends out the packet, tacking on FCS checksum at the end

If you’re really into switching performance, you understand that the routers cache the outbound interface and new Layer 2 header information. And the routing table lookup is preceded by cache lookup(s), possibly hardware-assisted and distributed to the interface cards, to get the packet out of the box as fast as possible, with as little central CPU involvement as possible. Let’s not go into the details of this, since that’s probably a whole article in itself.

The key operation that is needed to understand MLS is the de-encapsulation / re-encapsulation. The router strips off the old Layer 2 header, slaps on the new one and queues the packet for transmission. The Layer 2 headers are per-hop (from router to router across a subnet). The MAC addresses in those Layer 2 headers are per-hop, local to the Layer 2 segment (broadcast domain; VLAN). IP addresses in the Layer 3 IP header are end to end: the ultimate sender and recipient.

In doing MLS, an NFFC switchblade (or Supervisor IIG / IIIG with built-in comparable functionality) is doing the job of emulating the router, without actually sending the packets to the router. In order to simulate a router transparently, the NFFC has to de-encapsulate and re-encapsulate the packet, and also re-calculate the Layer 2 checksum (FCS).

A requirement for the MLS functionality to work is that you enable it in both the participating router and switch components. (With the 6000, MLS on the switch is enabled by default, and is only disabled when you disable MLS on the switch router component, the MSM or MSFC). Furthermore, MLS only applies to packets where the switch is in position to observe the packet before and after it passes through the router. See the figure below.

My analogy for MLS is a manager with a smart assistant. The manager and assistant agree that certain responsibilities are delegated to the assistant. When the assistant observes how the manager handles certain tasks, the assistant will step in and imitate the manager’s previous decision(s). The manager agrees to this and will let the assistant know when the policy changes.

In the case of MLS, the router is the manager, with packet switching at Layer 3 being delegated to the switch. When a packet passes through the switch, the switch snoops (in effect) to see what the router does with the packet. As it comes back through the switch (post-router, after de-encapsulation and re-encapsulation), the switch learns the new Layer 2 header. When subsequent packets reach the switch bound for the same IP destination, the switch can then just imitate the router, by replacing the Layer 2 header with the cached (stored) Layer 2 header, just as the router previously did. If routing changes, the router does need to notify the switch, which flushes the cached Layer 2 header, since a new routing policy is now in effect.

I hope you’re wondering, “but switches make forwarding decisions based on MAC addresses”. True! When does a router get involved in forwarding packets? When they are headed for a different subnet. Within the same subnet, the MAC address is all that is relevant. That’s where switches forward — within the same VLAN. But when a PC or host computer is sending to another subnet, it does something different. The destination MAC address is no longer that of the destination computer. It is instead the MAC address of the default gateway router interface.

This is another function of enabling MLS in the router. It can in effect tell the switch (or we can configure into the switch) the MAC address of the router. The switch needs to know that the router’s MAC address is special. The frames sent to that MAC address are the frames where the switch needs to subsequently snoop to “observe” the router’s imposition of a new Layer 2 header!

Let’s re-visit this in more detail now.


Details of MLS Activity in a Switch


When MLS is configured (enabled) in a router (the MLS Route Processor, or MLS-RP), the MLS-RP sends out a multicast announcement that it is doing MLS. The multicast address is the CGMP well-known address, 01-00-0c-dd-dd-dd. This multicast should pass transparently through non-Cisco switches, and tells MLS-capable switches to learn the router’s MAC address. The MAC address is internally associated with a one-byte XTAG, an internal identifier for fast lookup.

Now suppose a host or PC attached to the MLS switch sends a packet destined for a different subnet. Assuming that the MLS-RP is also the PC’s default gateway, the switch observes that the router’s MAC address is the destination of the Layer 2 frame. Since the router is an MLS-RP, the switch checks its cache, to see if it already knows what to do for packets from that flow (we’ll talk about flows later). If there is a cache entry, the switch replaces the Layer 2 header, then uses normal switching procedures to deliver the frame (Layer 2 destination MAC lookup and forwarding).

If there is no cache entry, the switch creates a temporary cache entry, a place holder for when the post-router frame is seen. (A candidate cache entry). The switch forwards the frame to the router. The router does its thing. The frame comes back to the switch (different VLAN, same switch). The switch sees the candidate entry, records the Layer 2 header in the cache. And forwards the frame.

Subsequently, packets from the same flow have Layer 2 headers rewritten (and checksums recalculated and rewritten) by the switch hardware and are forwarded without actually having gone through the router.

Should a router get disconnected from the switch, the XTAG provides a fast way for the switch to invalidate entries learned from a particular router.


Configuring Routing Between VLAN’s


External Router

If you’re routing between VLAN’s on a switch with an external router, you are either trunking (see earlier article) or you are not. If you’re not trunking, each router interface is a separate subnet,  and the router is configured just as usual for IP subnets, on the main interfaces. If you are trunking, then you need to configure the router accordingly, using subinterface. This will look something like the following:

Note that we’ve made the subinterface (101 or 105) match the VLAN number in the encapsulation isl (or dot1q) statement. This convention means not having to track which subinterface goes with which VLAN. A best practice!

The “encapsulation isl” and the subinterfaces are your clues that this router thinks the FastEthernet is a trunk and that it is doing ISL VLAN’s.


Internal Router (RSM/RSFC/MSM/MSFC)


Internal routers in Catalyst switches think their interfaces have names like “interface VLAN 1”. So the above example becomes merely:

Referring to the above figure, the trunk in the picture is now instead of the internal backplane of the switch. Both the router and switch components are inside the chassis.


Configuring MultiLayer Switching


Configuring MLS has two components: configuring the router (external or RSM/RSFC blade), and configuring the switch (Supervisor blade). We’ll talk about each of these in turn. The following are for a Catalyst 5000.


External Router


You need to globally enable MLS with:

If the switch has a VTP domain, the MLS physical (primary) interface must agree:

You then enable MLS per interface:

You also need one interface which is connected to the switch set up as the MLS management interface:

If the external router has only non-trunk links to the switch, you need to configure an associated VLAN id with each:

(With trunk interfaces, the ISL or 802.1q tag provides VLAN info, needed in the MLS-RP hello).


Internal Router (RSM/RSFC/MSM/MSFC)


With internal routers, basically, just the interface names change (and you can omit the VLAN-id since it is implicit with each interface).




MLS is enabled by default on MLS-capable switches. If disabled, re-enable it on a Catalyst 5000 with the command:

With an external router, you need to tell the switch the router’s IP address:

(where ip-address is the address shown on the router with the “show mls rp” command).

Catalyst 6000 MSFC


The only important thing you can configure with Catalyst 6000's is MLS on an interface. It defaults to “on” on the MLS-RP, and cannot be globally disabled (except perhaps by disabling it on all interfaces?). The MLS-SP defaults to “on”, and is apparently only disabled if MLS is off on all MSFC interfaces. Sample of how to turn MLS off on an interface:


But What Is a Flow?


A flow might be all packets for a certain IP address. Or it might be packets between an IP source and destination. Or it might be packets between a certain IP source and destination on a single port. The meaning varies here because of the switch caches based on flows, and it only caches what it has to. With normal routing, only destination matters. Throw in a standard IP access list on the MLS-RP, and suddenly source address matters. So in such situations, the MLS-SP (switch) caches based on source-destination IP pairs. Throw in an extended IP access list and transport protocol and destination port make a difference. So then the switch caches based on source-destination-transport protocol-port information. The fancy name for this is “flow masks”, which are respectively destination-ip, source-destination-ip, and ip-flow in the Cisco documentation.

If you want, you can configure the switch to force it to cache more specifically. This may impact performance and would be a fairly odd thing to configure. For instance, with no access lists on the router component, you could configure the switch component to force at least source-destination-ip mask:

The other keyword options for minimal flow mask are a destination and full, corresponding to destination-ip and ip-flow above.