Arista MLAG (Multi Chassis Link Aggregation)

MLAG (Multi Chassis Link Aggregation)

Topology:

        

1. On both switches, ensure that the control plane ACL configuration is compatible with MLAG. These two rules exist in the default-control-plane-acl configuration. You can verify with the command: show ip access-lists default-control-plane-acl

        

2. Create port-channel for the peer-link

1. They can be different number as shown in above picture
2. switch1# config t
   switch1(conf)#interface eth1-2
   switch1(config-if-Et1-2)# channel-group 101 mode active
   switch1(config)# interface port-channel 101
   switch1(config-if-Po101)# switchport mode trunk
3. Similarly, configure on Switch 2 as well with the desired port-channel number
4. Note: It is recommended, for redundancy reasons to use a port-channel. The peer link is recommended to be at least a two port port-channel to avoid having a single point of failure.

3. On both switches, create a VLAN with an unused vlan-id for the MLAG peers to communicate.

1. switch1(conf)#vlan 4094
   switch1(config-vlan-4094)# trunk group mlagpeer
   switch1(config-vlan-4094)# interface port-channel 101
   switch1(config-if-Po101)# switchport trunk group mlagpeer
   switch1(config-if-Po101)# exit
   switch1(conf)#no spanning-tree vlan 4094
2. Use exactly same config on Switch 2
3. Note: The trunk group names for the peer VLAN (mlagpeer in the above example) should be configured to be the same on both switches. In order to successfully establish an MLAG association, the configuration for vlans and vlan trunk groups must be identical
4. Assigning VLAN4094 and Port-Channel10 to trunk group ‘mlagpeer’ prevents VLAN4094 from being carried on any trunk other than Po10. This allows you to safely disable Spanning-Tree on VLAN4094
5. Another option other than using trunk groups is to prune Vlan 4094 from all other Vlans which is cumbersome

4. Configure the SVI for peer-to-peer communication:

1. On Switch 1:

switch1(conf)#int vlan 4094
        switch1(config-if-Vl4094)# ip address 10.0.0.1/30

switch1(config-if-Vl4094)#no autostate

2. On Switch 2:

        switch2(conf)#int vlan 4094
        switch2(config-if-Vl4094)# ip address 10.0.0.2/30

        switch2(config-if-Vl4094)#no autostate

3. Check for connectivity by pinging each other
4. Note: The The local and peer addresses must be located on the same IP address subnet. Autostate should be disabled on the SVI configured as the local interface.

5. Configure the MLAG peering on both the switches:

1. On Switch 1:

switch1(config)#mlag
switch1(config-mlag)#local-interface vlan 4094
switch1(config-mlag)#peer-address 10.0.0.2
switch1(config-mlag)#peer-link port-channel 101
switch1(config-mlag)#domain-id mlag1

2. On Switch 2:

switch2(config)#mlag
switch2(config-mlag)#local-interface vlan 4094
switch2(config-mlag)#peer-address 10.0.0.1
switch2(config-mlag)#peer-link port-channel 201
switch2(config-mlag)#domain-id mlag1

6. Verify MLAG operation:

1. Check if the MLAG is up by running (config)#show mlag and checking if the MLAG STATUS is shown as ACTIVE

---

Troubleshooting: MLAG Status not becoming ACTIVE:

1. Check if the configuration is similar on both the peers: domain-id, vlan, ip address in same subnet, trunk group name.
2. Verify that Spanning tree disabled on Vlan
3. Check if lower layer layers are up and not errDisabled. If yes, then, shut and unshut to bring them up

Troubleshooting: MLAG Status ACTIVE but INCONSISTENT:

1. Use this command to see the inconsistencies: (config)#show mlag config-sanity
2. To check for inconsistencies in MLAG (even though MLAG is active):

1. Check if different Vlans are configured on the peers that allow the MLAG port-channel. (eg: On switch 1, po10 is allowed on Vlan 4094, default, Vlan 2 and Vlan3; whereas on Switch 2 , po10 is allowed on Vlan 4094, default)
2. VLANs must be created on each MLAG peer. The primary MLAG peer does not communicate VLAN information to the secondary. So, Take care to configure VLANs and port settings (Port-specific bridging configuration comes from the switch where the port physically lives. This includes switchport access vlan, switchport mode, trunk allowed vlans, trunk native vlan, and switchport trunk groups) identically on both MLAG peers

3. (config)#show vlan   —> check if other vlans have po10 and also if u can see all peer interfaces as pE
4. Verify if same EOS versions on both the peers

---

7) Configure MLAG Services:

1. Note: The mlag identification number does not have to match the port-channel number
2. Note: The port-channel numbers grouped in an MLAG must match, they cannot be two different values.
3. Note: A port-channel in an MLAG can have multiple members.
4. In short: Port channels configured as an MLAG must have identical port channel numbers. Although the MLAG ID is a distinct parameter from the port channel number, best practices recommend assigning the MLAG ID to match the port channel number. The following example does not follow this convention to emphasize the parameters that are distinct (see that po20 has been used but mlag id is 12...though not recommended).
5. These Switch1 commands bundle Ethernet interfaces 3 and 4 in port channel 20, then associate that port channel with MLAG 12.

        switch1(config)#interface ethernet 3-4

switch1(config-if-et3-4)#channel-group 20 mode active switch1(config-if-et3-4)#interface port-channel 20 switch1(config-if-po20)#mlag 12

switch1(config-if-po20)#exit

switch1(config)#

6. These Switch2 commands bundle Ethernet interfaces 9 and 10 in port channel 15, then associate that port channel with MLAG 12.

1. Note that same mlag id and same port-channel number (for downstream device) [here: mlag 12 and po20] should be used on both the peers

switch2(config)#interface ethernet 9-10 switch2(config-if-et9-10)#channel-group 15 mode active switch2(config-if-et9-10)#interface port-channel 20 switch2(config-if-po20)#mlag 12

switch2(config-if-po20)#exit

switch2(config)#

7. These commands configure the port channels that attach to the MLAG on network attached device:

1. Note that on the device, there is no MLAG specific configuration. It is configured as a regular port channel

NAD(config)#interface ethernet 1-4

NAD(config-if-Et1-4)#channel-group 1 mode active

NAD(config-if-Et1-4)#exit

NAD(config)#

---

• FOR ADVANCED TOPOLOGY CONFIGURATION, see the EOS Config Manual (there is an example in that with full config)
• FOR more details on MLAG, see EOS Config Manual
• To view any syslog messages, you will need to change MLAG level to debugging: Switch(config)# logging level mlag 7
• Troubleshooting and Debugging Mlag- Useful commands:

• Show mlag detail
• Show mlag interface detail
• Show mlag tunnel counter detail
• Show lacp nei
• Show lldp nei
• Trace commands
• Cd /var/log/messages
• Cd /var/log/agents

Link Aggregation(LAG) and LACP Protocol:

LAG (Link Aggregation):

• It is a virtual concept

Thus, the LAG makes two links appear as one in STP

• The basic concept of LAG is that multiple physical links are combined into one logical bundle. This provides two major benefits, depending on the LAG configuration:
  • Increased capacity – traffic may be balanced across the member links to provide aggregated throughput
  • Redundancy – the LAG bundle can survive the loss of one or more member link

• LAG makes two connections appear as one to STP, so the second connection won’t be blocked.
• Load sharing among the LAG links: Traffic from each flow of packets goes through a single link. So, even if we get a flow of 20G, it will go via a single link only. So, the throughput is limited to the bandwidth of the single link only (here: 10G).

Note: The physical type (SFP,...) and speeds(10G,...) of all links in a port-channel should be the same.

#int et 17-18

#channel-group mode on //static configuration of link aggregation

Where channelgroupnumber = 1

#show span //we can see the port channel 1 in show spanning-tree

#show int port-Channel 1 (OR) #show port-channel

• Now, if we make the port-channel of et17 and 18 in vlan 51. And now, if we make individual et17 and et 18 in van 30. Then, even then the port-channel only will take effect as it will act as a single interface. Thus, ALWAYS port-channel configuration overrides any interface specific configuration.
• Even after link aggregation, show lldp nei will show the individual interfaces only, since lldp considers only the physical topology.
• If we tcpdump on int et 17, we can only see the lldp protocols. If we do tcpdump on port-channel 1, we can see all other protocols like STP, etc

#tcpdump int po1

• If we use link aggregation on one side (MT701) and stop link aggregation on other side (MT703), then:In EOS,

• On MT701, the STP will not even consider the individual ports. It only considers the port-channel as an interface and it only has the port channel’s state.
• On MT703, the interfaces are considered different. So, if any ARP request broadcast comes from the port-channel on MT701, then, MT703 receives it on both the interfaces and floods it again via the other link.

• RTAG 7:

• • RTAG7 is a hashing algorithm that load balances the traffic.
  • Hash engine comes out with a number for each flow.
  • Using the number, the flow is sent through that port.
  • This ensures that the flows are distributed among all the links
  • #show port-channel load-balance trident fields //we can see which all fields of a packet participate in the hashing.
  • #port-channel load-balance trident fields ? //we can configure which all fields we can disable or enable to influence the hashing.

• Why is the spanning tree cost of port channel comes down? (et 18, 19 have cost of 2000 each but if we use port-channel the cost of port-channel becomes 1999)

• • Because of bandwidth since cost depends on bandwidth
  • 10G links have cost of 2000 by default. Higher the bandwidth, lower is the cost.
  • But, if another 10G link is there, cost will be 2000, so STP may consider that link. So, we use cost of 1999 on the port-channel link.

• • If any speed mismatch is there among the links in a port channel, then, only the higher bandwidth link is active.

DYNAMIC LINK AGGREGATION:

• LACP (Link Aggregation Control Protocol)
• #int et 17-18

#channel-group mode active //dynamic lacp active

(OR)

#channel-group mode passive //dynamic lacp passive

• If both the sides are configured as active, then, both sides can start LACP transmission. If one side is configured as passive, then, it can only receive lacp, not send. (both sides cannot be passive, since both sides will only be waiting for lacp)

• Passive LACP: the port prefers not transmitting LACPDUs. The port will only transmit LACPDUs when its counterpart uses active LACP (preference not to speak unless spoken to).
• Active LACP: the port prefers to transmit LACPDUs and thereby to speak the protocol, regardless of whether its counterpart uses passive LACP or not (preference to speak regardless).

• In L2 header, the type will be a Slow Protocol and the subtype in slow protocols will be LACP. It is a slow protocol since if one side LACP is configured, then, it will keep sending LACP PDUs every 30 seconds.
• The LACP rate fast feature is used to set the rate (once every second) at which the LACP control packets are sent to an LACP-supported interface. The normal rate at which LACP packets are sent is 30 seconds.

#lacp rate fast //1 seconds

(OR)

#lacp rate normal //30 seconds

• The above timeout is used to configure the timeout for the partner. If we use rate fast, then, within 3 seconds if it doesn’t get a reply, it knows that the link is not active. It tears down the port-channel.

(OR)

In the rate fast timeout configuration, an LACPDU is sent every second. If no response comes from its partner after 3 LACPDUs are sent, a timeout event occurs and the port channel is removed.

• If switch A is set to fast timeout and switch B is set to normal timeout. Then, switch A sends LACPDU every 30 seconds to B and the switch B sends LACPDU every 1 second to A. This information is shared in the timeout flag of LACPDU flags.
• The LACP sender is called actor and the receiver is called partner. (For each switch, it will be the actor and the other side will be the partner)
• Fields in LACP:        http://www.juniper.net/documentation/en_US/junos/topics/reference/command-summary/show-lacp-interfaces.html

• • Key: Tells the port-channel number (there are two separate fields for both the actor and partner)
  • Port: tells the port number .eg. et 17 (there are two separate fields for both the actor and partner)
  • Actor State:

• • • LACP Activity: tells whether the actor is in active (1) or passive state (0)
    • LACP Timeout: tells whether the timeout is normal (1) or fast (0)
    • Aggregation: 1 tells whether the port is a part of port channel. 0 tells that it is operating as a single link.
    • Synchronization: the flag is set when we get back a LACPDU from the partner.

(1 Means it has been allocated to the correct link aggregation group, the group has been associated with a compatible aggregator, and the identity of the link aggregation group is consistent with the system ID and operational key information transmitted. If the value is 0, the link is not synchronized .ie. it is currently not in the right aggregation.)

• • • Collecting: after sync, if we receiver data from partner, accept it.
    • Distributing: after sync, even send data.
    • Default: 1 indicates that the actor’s receive machine is using the default operational partner information, administratively configured for the partner. 0 indicates the operational partner information in use has been received in an LACP PDU.
    • Expired: 1 indicates the actor or partner is in an expired state. 0 indicates the actor or partner is not in an expired state .ie. no lacpdu received within timeout

• Now, a case when we aggregate many links into a port channel but the links are connected to different switches. Then, when the actor sends out a LACPDU, whichever partner sends the first reply, that link will become active. The other links connected to different partners will get inactive state.
• #show lacp internal detailed //shows the current switch’s details. Also we can see all the flags which are set and not set. We can also see the port-priorities, so that we choose which link should come up in case of mismatched-aggregate
• #show lacp neighbor detailed  //shows the system id, port number and key of the neighbor
• #show port-channel //in this itself we can see the reason for the port channel not active
• #show lacp neighbor all-port //shows details about the neighbour for each of the physical port. We can then use the oper-key value to know if the port-channel link is connected to same port-channel on other side also.

• • If oper-key is different and admin-key is same, then, this port channel is connected to 2 different port-channels on the other side. But, they are in same switch
  • If oper-key and admin-key are both different, then, this port-channel is connected to 2 different switches.

• #show lacp interface all-port //shows details for lacp not working...If the partner’s sys-id is same and oper-key is different, then, it means same port channel on our side is connected to two port-channels to same partner on other side. (mismatched-aggregate)
• NOTE: Oper-key is the number assigned to port-channel. For eg, if I give my channel-group 100, then, port channel number is 100 and oper-key will be shown as 64 (since: 100 in decimal is 64 in hex)
• #show lacp interface detailed all-ports and #show etherchannel detailed all-ports are two ultimate troubleshooting commands if we can’t find out issue