Large Scale Data-Center Networks- Part 2

2. Edge Services and Border Leafs

For two-tier and three-tier data center topologies, the role of the border leaf in the network is to provide external connectivity to the data center site. In addition, since all traffic enters and exits the data center through the border leaf switches, they present the ideal location in the network to connect network services like firewalls, load balancers, and edge VPN routers. The border leaf switches connect to the WAN edge devices in the network to provide external connectivity to the data center site. As a design principle, two border leaf switches are recommended for redundancy. The WAN edge devices provide the interfaces to the Internet and DCI solutions. For DCI, these devices function as the Provide Edge (PE) routers, enabling connections to other data center sites through WAN technologies like Multiprotocol Label Switching (MPLS) VPN and Virtual Private LAN Services (VPLS). The Brocade validated design for DCI solutions is discussed in a separate validated design document.

There are several ways that the border leafs connect to the data center site. In three-tier (super-spine) architectures, the border leafs are typically connected to the super-spines as depicted in Figure 2 (3tier) below. In two-tier topologies, the border leafs are connected to the spines as depicted in Figure 1 (2tier) above. Certain topologies may use the spine as border leafs (known as a border spine), overloading two functions into one. This topology adds additional forwarding requirements to spines—they need to be aware of the tenants, VNIs, and VXLAN tunnel encapsulation and de-encapsulation functions.

3. Optimized 5-Stage Layer 3 Clos Topology (Three-Tier)

Multiple PoDs based on leaf-spine topologies can be connected for higher scale in an optimized 5-stage folded Clos (three-tier) topology. This topology adds a new tier to the network, known as a super-spine. This architecture is recommended for interconnecting several EVPN VXLAN PoDs. Super-spines function similar to spines: BGP control-plane and IP forwarding based on the outer IP header in the underlay network. No endpoints are connected to the super-spine. Figure 2 shows four super-spine switches connecting the spine switches across multiple data center PoDs.

The connection between the spines and the super-spines follows the Clos principles:

• Each spine connects to all super-spines in the network.
• Neither spines nor super-spines are interconnected with each other.

FIGURE 2 Optimized 5-Stage L3 Clos Topology

---

• FOR MORE INFORMATION ON Underlay Routing, see:

• http://www.brocade.com/content/html/en/brocade-validated-design/brocade-ip-fabric-bvd/GUID-2B84EB71-9AF7-40B0-9E17-5A460F53AB1A.html
• http://www.cisco.com/c/en/us/products/collateral/switches/nexus-7000-series-switches/white-paper-c11-737022.html

• For more information on Network Virtualization with BGP EVPN, see:

• http://www.brocade.com/content/html/en/brocade-validated-design/brocade-ip-fabric-bvd/GUID-1B9BB4EF-D77D-40BB-BECD-9AF926B1ABC3.html

• For information on validated Designs:

• http://www.brocade.com/content/html/en/brocade-validated-design/brocade-ip-fabric-bvd/GUID-4E59B6A1-985B-4D01-BED8-8C2CFB55B525.html

---

Large Scale Data-Center Networks- Part 1

1. Leaf-Spine Layer 3 Clos Topology (Two-Tier):

• The leaf-spine topology has become the de facto standard for networking topologies when building medium- to large-scale data center infrastructures. The leaf-spine topology is adapted from Clos telecommunications networks.
• The IP fabric within a PoD resembles a two-tier or 3-stage folded Clos fabric.
• The two-tier leaf-spine topology is shown in Figure 1.
• The bottom layer of the IP fabric has the leaf devices (top-of-rack switches), and the top layer has spines.
• The role of the leaf is to provide connectivity to the endpoints in the data center network. These endpoints include compute servers and storage devices as well as other networking devices like routers, switches, load balancers, firewalls, and any other physical or virtual networking endpoints. Because all endpoints connect only to the leaf, policy enforcement, including security, traffic-path selection, QoS marking, traffic policing, and shaping, is implemented at the leaf. More importantly, the leafs act as the anycast gateways for the server segments to facilitate mobility with the VXLAN overlay.
• The role of the spine is to provide connectivity between leafs. The major role of the spine is to participate in the control-plane and data-plane operations for traffic forwarding between leafs.
• The spine devices serve two purposes: BGP control plane (route reflectors for leaf or eBGP peering with leaf) and IP forwarding based on the outer IP header in the underlay network. Since there are no network endpoints connected to the spine, tenant VRFs or VXLAN segments are not created on spines. Their routing table size requirements are also very light to accommodate just the underlay reachability. Note that all spine devices need not act as BGP route reflectors; only selected spines in the spine layer can act as BGP route reflectors in the overlay design.

1.1. As a design principle, the following requirements apply to the leaf-spine topology:

• Each leaf connects to all spines in the network through 40-GbE links.
• Spines are not interconnected with each other.
• Leafs are not interconnected with each other for data-plane purposes. (The leafs may be interconnected for control-plane operations such as forming a server-facing vLAG.)
• The network endpoints do not connect to the spines.

This type of topology has the predictable latency and also provides the ECMP forwarding in the underlay network. The number of hops between two leaf devices is always two within the fabric. This topology also enables easier scale out in the horizontal direction as the data center expands and is limited by the port density and bandwidth supported by the spine devices.

This validated design recommends the same hardware in the spine layer. Mixing different hardware is not recommended.

1.2. IP Fabric Infrastructure Links:

All fabric nodes—leafs, spines, and super-spines—are interconnected with Layer 3 interfaces. In the validated design,

• 40-GbE links are used between the fabric nodes.
• All these links are configured as Layer 3 interfaces with /31 IPv4 address. For a simple 3-stage fabric, IP unnumbered interfaces can be used. We do not recommend a mix of unnumbered and numbered interfaces within a fabric. Also, for a 5-stage IP fabric, numbered interfaces are highly recommended.
• The MTU for these links is set to jumbo MTU. This is a requirement to handle the VXLAN encapsulation of Ethernet frames.

1.3.1 Server-Facing Links (Individual Leaf/ToR):

The server-facing or access links are on the leaf nodes. In the validated design,

• 10-GbE links are used for server-facing VLANs.
• These links are configured as Layer 2 trunks with associated VLANs.
• The MTU for these links is set to the default: 1500 bytes.
• Spanning tree is disabled.
• 

1.3.2 Server-Facing Links (vLAG Pair/ToR):

vLAG configuration involves three steps:

• Node ID configuration on the pair of devices.
• Inter-switch links or ISL configuration on both devices.
• Configuring the server-facing port channels and adding the required VLANs on them.

See below for more info on configurations: http://www.brocade.com/content/html/en/brocade-validated-design/brocade-ip-fabric-bvd/GUID-F54693FB-7866-43B5-8610-71F2B2B85ECA.html

1.4 Leaf-Spine L3 Clos Topology:

Spanning tree must be enabled if there are Layer 2 switches/bridges between a leaf and servers.

Arista MLAG (Multi Chassis Link Aggregation)

MLAG (Multi Chassis Link Aggregation)

Topology:

        

1. On both switches, ensure that the control plane ACL configuration is compatible with MLAG. These two rules exist in the default-control-plane-acl configuration. You can verify with the command: show ip access-lists default-control-plane-acl

        

2. Create port-channel for the peer-link

1. They can be different number as shown in above picture
2. switch1# config t
   switch1(conf)#interface eth1-2
   switch1(config-if-Et1-2)# channel-group 101 mode active
   switch1(config)# interface port-channel 101
   switch1(config-if-Po101)# switchport mode trunk
3. Similarly, configure on Switch 2 as well with the desired port-channel number
4. Note: It is recommended, for redundancy reasons to use a port-channel. The peer link is recommended to be at least a two port port-channel to avoid having a single point of failure.

3. On both switches, create a VLAN with an unused vlan-id for the MLAG peers to communicate.

1. switch1(conf)#vlan 4094
   switch1(config-vlan-4094)# trunk group mlagpeer
   switch1(config-vlan-4094)# interface port-channel 101
   switch1(config-if-Po101)# switchport trunk group mlagpeer
   switch1(config-if-Po101)# exit
   switch1(conf)#no spanning-tree vlan 4094
2. Use exactly same config on Switch 2
3. Note: The trunk group names for the peer VLAN (mlagpeer in the above example) should be configured to be the same on both switches. In order to successfully establish an MLAG association, the configuration for vlans and vlan trunk groups must be identical
4. Assigning VLAN4094 and Port-Channel10 to trunk group ‘mlagpeer’ prevents VLAN4094 from being carried on any trunk other than Po10. This allows you to safely disable Spanning-Tree on VLAN4094
5. Another option other than using trunk groups is to prune Vlan 4094 from all other Vlans which is cumbersome

4. Configure the SVI for peer-to-peer communication:

1. On Switch 1:

switch1(conf)#int vlan 4094
        switch1(config-if-Vl4094)# ip address 10.0.0.1/30

switch1(config-if-Vl4094)#no autostate

2. On Switch 2:

        switch2(conf)#int vlan 4094
        switch2(config-if-Vl4094)# ip address 10.0.0.2/30

        switch2(config-if-Vl4094)#no autostate

3. Check for connectivity by pinging each other
4. Note: The The local and peer addresses must be located on the same IP address subnet. Autostate should be disabled on the SVI configured as the local interface.

5. Configure the MLAG peering on both the switches:

1. On Switch 1:

switch1(config)#mlag
switch1(config-mlag)#local-interface vlan 4094
switch1(config-mlag)#peer-address 10.0.0.2
switch1(config-mlag)#peer-link port-channel 101
switch1(config-mlag)#domain-id mlag1

2. On Switch 2:

switch2(config)#mlag
switch2(config-mlag)#local-interface vlan 4094
switch2(config-mlag)#peer-address 10.0.0.1
switch2(config-mlag)#peer-link port-channel 201
switch2(config-mlag)#domain-id mlag1

6. Verify MLAG operation:

1. Check if the MLAG is up by running (config)#show mlag and checking if the MLAG STATUS is shown as ACTIVE

---

Troubleshooting: MLAG Status not becoming ACTIVE:

1. Check if the configuration is similar on both the peers: domain-id, vlan, ip address in same subnet, trunk group name.
2. Verify that Spanning tree disabled on Vlan
3. Check if lower layer layers are up and not errDisabled. If yes, then, shut and unshut to bring them up

Troubleshooting: MLAG Status ACTIVE but INCONSISTENT:

1. Use this command to see the inconsistencies: (config)#show mlag config-sanity
2. To check for inconsistencies in MLAG (even though MLAG is active):

1. Check if different Vlans are configured on the peers that allow the MLAG port-channel. (eg: On switch 1, po10 is allowed on Vlan 4094, default, Vlan 2 and Vlan3; whereas on Switch 2 , po10 is allowed on Vlan 4094, default)
2. VLANs must be created on each MLAG peer. The primary MLAG peer does not communicate VLAN information to the secondary. So, Take care to configure VLANs and port settings (Port-specific bridging configuration comes from the switch where the port physically lives. This includes switchport access vlan, switchport mode, trunk allowed vlans, trunk native vlan, and switchport trunk groups) identically on both MLAG peers

3. (config)#show vlan   —> check if other vlans have po10 and also if u can see all peer interfaces as pE
4. Verify if same EOS versions on both the peers

---

7) Configure MLAG Services:

1. Note: The mlag identification number does not have to match the port-channel number
2. Note: The port-channel numbers grouped in an MLAG must match, they cannot be two different values.
3. Note: A port-channel in an MLAG can have multiple members.
4. In short: Port channels configured as an MLAG must have identical port channel numbers. Although the MLAG ID is a distinct parameter from the port channel number, best practices recommend assigning the MLAG ID to match the port channel number. The following example does not follow this convention to emphasize the parameters that are distinct (see that po20 has been used but mlag id is 12...though not recommended).
5. These Switch1 commands bundle Ethernet interfaces 3 and 4 in port channel 20, then associate that port channel with MLAG 12.

        switch1(config)#interface ethernet 3-4

switch1(config-if-et3-4)#channel-group 20 mode active switch1(config-if-et3-4)#interface port-channel 20 switch1(config-if-po20)#mlag 12

switch1(config-if-po20)#exit

switch1(config)#

6. These Switch2 commands bundle Ethernet interfaces 9 and 10 in port channel 15, then associate that port channel with MLAG 12.

1. Note that same mlag id and same port-channel number (for downstream device) [here: mlag 12 and po20] should be used on both the peers

switch2(config)#interface ethernet 9-10 switch2(config-if-et9-10)#channel-group 15 mode active switch2(config-if-et9-10)#interface port-channel 20 switch2(config-if-po20)#mlag 12

switch2(config-if-po20)#exit

switch2(config)#

7. These commands configure the port channels that attach to the MLAG on network attached device:

1. Note that on the device, there is no MLAG specific configuration. It is configured as a regular port channel

NAD(config)#interface ethernet 1-4

NAD(config-if-Et1-4)#channel-group 1 mode active

NAD(config-if-Et1-4)#exit

NAD(config)#

---

• FOR ADVANCED TOPOLOGY CONFIGURATION, see the EOS Config Manual (there is an example in that with full config)
• FOR more details on MLAG, see EOS Config Manual
• To view any syslog messages, you will need to change MLAG level to debugging: Switch(config)# logging level mlag 7
• Troubleshooting and Debugging Mlag- Useful commands:

• Show mlag detail
• Show mlag interface detail
• Show mlag tunnel counter detail
• Show lacp nei
• Show lldp nei
• Trace commands
• Cd /var/log/messages
• Cd /var/log/agents

Arista Command CAPI/ eAPI

        Arista Command eAPI (CAPI) 

• The Arista Command API is a simple and complete API that allows you to configure and monitor your Arista switches.
• Once the API is enabled, the switch accepts commands using the industry standard CLI syntax, and responds with machine readable output and errors serialized in JSON, served over HTTP.

CONFIGURING CAPI:

• It is very easy to configure eAPI
• Although disabled by default, it is very simple to get the Command API server running on your switch.
• bash$ ssh username@myswitch
    Password: <passw0rd>
    myswitch> enable
    myswitch# configure terminal
    myswitch(config)# management api http-commands
    myswitch(config-mgmt-api-http-cmds)# no shutdown
    myswitch(config-mgmt-api-http-cmds)# show management api http-commands
    Enabled:            Yes
    HTTPS server:       running, set to use port 443
    HTTP server:        shutdown, set to use port 80
    Local HTTP server:  shutdown, no authentication, set to use port 8080
    Unix Socket server: shutdown, no authentication
    VRF:                default
    …
• This enables eAPI only for HTTPS. For using HTTP, switch to it as shown below:

• From configure mode, enter management api http-commands mode. In this submode, you can turn on or off the server by typing [no] shutdown, switch between accepting HTTP or HTTPS traffic via [no] protocol http[s], and adjust the ports the server should listen on using protocol http[s] port <portNumber>.
•   myswitch> enable
   myswitch# configure terminal
   myswitch(config)# management api http-commands
   myswitch(config-mgmt-api-http-cmds)# [no] shutdown
   myswitch(config-mgmt-api-http-cmds)# [no] protocol https [port <portNumber>]
   myswitch(config-mgmt-api-http-cmds)# [no] protocol http [port <portNumber>]
   myswitch(config-mgmt-api-http-cmds)# [no] protocol http localhost [port <portNumber>]
   myswitch(config-mgmt-api-http-cmds)# [no] protocol unix-socket

• On-box usage of CAPI: It is often useful to run scripts that use Command API directly on the switch itself. The first is an HTTP server bound to localhost (on port 8080 by default), which only accepts connections arriving from the same machine.  The other solution is a Unix domain socket. Both can be used simultaneously also.

• Once Command API is enabled then you access via the local domain socket unix:/var/run/command-api.sock

 switch = Server( "unix:/var/run/command-api.sock" )

• If configured to use HTTP over localhost, your script can access the API as follows:

 switch = Server( "http://localhost:8080/command-api" )

Configuring a Certificate:

• Because clients use HTTP basic authentication to send usernames and passwords to the switch, we recommend using HTTPS so no passwords are sent in the clear over the network.
• By default a self-signed certificate will be used.
• You can view the current certificate using show management api http-commands https certificate

Exploring the Command API:

• To explore the API, point your web browser to http[s]://<switch-name>/, after enabling Command API.
• This web-app lets you interactively explore the protocol, return values, and model documentation.

Using Command API with Python:

• Install jsonrpclib library for installing Python JSON-RPC:

 admin:~ admin$ sudo pip install jsonrpclib

• from jsonrpclib import Server
• switch = Server( "https://username:passw0rd@myswitch/command-api" ) #Note that both username and password are compulsory. If no password, give the username itself in password field also
• response = switch.runCmds( 1, ["show version"] ) #instead of 1, we can also use “latest” to take latest version
• print "The switch's system MAC addess is", response[0]["systemMacAddress"]

• Another easy way is to import CAPIClient library and use as per its documentation on: https://github.com/arista-eosext/CAPIClient  

How it Works:

• The client starts by sending a JSON-RPC request via an HTTP POST request to http://<yourswitch>/command-api, which encapsulates a list of CLI commands it wishes to run, and the switch replies with a JSON-RPC response containing the result of each CLI command that was executed.
• If any of the commands emit an error, no further commands from that request are executed, and the response from the switch will contain an error object containing the details of the error that occurred.

Command Specification:

• In most cases, the client will use a simple string to specify the CLI command in the cmds parameter in the request. In certain cases, however, clients may wish to specify additional parameters during the command's execution.
• To use complex commands, pass a JSON object in lieu of a string, with the following attributes:

• cmd (mandatory): specify the CLI command to run.
• input (optional): specify a string to be provided as standard input while running the cmd
• revision (optional): in the case of 'show' commands that have been modified over the course of different EOS releases, this parameter allows clients to request an old model format. At this time, all models are at revision 1, and this attribute will be ignored.

• For example, to set the message of the day to Hello World!, the client should set cmds to

[ "enable", "configure", { "cmd": "banner motd", "input": "Hello World!\nEOF" } ]

• Similarly, if the switch requires an enable password, the following cmds value would let you enter exec mode and clear interface counters

[ { "cmd": "enable", "input": "hunter2" },  "clear counters" ]

Error Codes:

• The responses generated by the client library usually follow language conventions.
• For example, in Python, an error response results in an Exception being thrown, while Javascript expects an error handler callback.

Unsupported Commands:

• Certain commands are not permitted and will always return an error.
• The largest class of such commands are interactive commands .ie. those that need a response back from user or shows output continuously to user:

• watch
• reload #can be overcome by using the non-interactive ‘reload now’ command

• The bash command is only allowed with the timeout <timeout> argument, ie. bash timeout <timeout> <ARG>.
• Commands that attempt to use CLI pipes are also not allowed.

• (e.g. show interfaces | grep Ethernet1 )

• Also, no abbreviations are allowed in commands. This is necessary because future versions of EOS may add more commands, rendering previous abbreviations ambiguous.

Unconverted Commands:

• Although you can access almost any CLI command via the Command API, not all show commands have been converted to return formatted data, and trying to run the command with the format parameter set to json will result in an error.
• However, you can still get the CLI ASCII output for the unconverted command by setting the format parameter to text.

SEE: Command documentation for the respective command: http://<Switch name>/documentation.html