Elastic Stack for Network and Security Engineers

In recent years network engineers turned from CLI jockeys into a hybrid between an application developer and a networking expert… what acronym-loving people call NetDevOps. In practice, a NetDevOps engineer should be able to:

• manage and troubleshoot networks;
• helps to troubleshoot any information system issue (including “slow” applications);
• automate networking tasks;
• monitor network and application performance;
• continue auditing the infrastructure, eventually using (partial) automation to make it less time-consuming;

and do everything else not covered by other IT teams.

To get there, NetDevOps started to learn Linux, Python, and automation frameworks. In this article, we’ll add log management to the mix.

Log management is the ability to collect any event from information systems and get them automatically analyzed to help NetDevOps react faster to information system issues.

There are many commercial or open-source log management platforms; I would mention:

• Graylog
• Elastic Stack
• InfluxDB

Each one of these has a different focus: Graylog was born as a log management solution, Elastic Stack is a Big Data solution, and InfluxDB is a time-series database.

We won’t go into discussing the pros and cons of these products. There are already plenty of blog posts doing that.

I chose to work with Elastic Stack because of its:flexibility: I’m able to inject logs from almost any device I encountered;
scalability: I can manage hundreds or thousands of log messages per seconds without any issue;
integration: I’m able to build a very robust solution that includes other open-source components.
vision: I honestly like where Elastic company is going, and I agree with its vision.

Configuring ZTP on Arista EOS

ZTP

• CLIs:
• zerotouch cancel - cancel this time
• zerotouch disable - distable ztp permanently
• bash vi /mnt/flash/zerotouch-config
• DHCPd config:
• see below
• Supported version:
• Fixed - v3.7
• Chassis - v4.10
• If no startup-config the switch will default to ZTP, if not disable ZTP, the switch will not function properly.  
• automatic configuration based on DHCP
• configure all eth and management ports with "no switchport" to allow DHCP
• can use dhcpd on the Linux
• below bootfile can be either config or script

DHCPd config:
option subnet-mask 255.255.255.0
option broadcast-address 192.168.1.255

option routers 192.168.1.1
option domain-name-server 192.168.1.200, 192.168.1.205
option domain-name "gad.net"

subnet 192.168.1.0 netmask 255.255.255.0 {
range 192.168.1.160 192.168.1.167;
}

host Arista1 {
 option dhcp-client-identifier 00:1c:73:08:91:33;
 fixed-address 192.168.1.170
 option bootfile-name "http://www2.gad.net/config/Arista1-ZTP"
}

ZTP Script:
#!/usr/bin/Cli -p2
enable
copy http://<url>/arista1-startup flash:startup-config
copy http://<url>/EOS-4.9.3.swi flash:
config
boot system flash:EOS-4.9.3.swi

Arista Command CAPI/ eAPI

        Arista Command eAPI (CAPI) 

• The Arista Command API is a simple and complete API that allows you to configure and monitor your Arista switches.
• Once the API is enabled, the switch accepts commands using the industry standard CLI syntax, and responds with machine readable output and errors serialized in JSON, served over HTTP.

CONFIGURING CAPI:

• It is very easy to configure eAPI
• Although disabled by default, it is very simple to get the Command API server running on your switch.
• bash$ ssh username@myswitch
    Password: <passw0rd>
    myswitch> enable
    myswitch# configure terminal
    myswitch(config)# management api http-commands
    myswitch(config-mgmt-api-http-cmds)# no shutdown
    myswitch(config-mgmt-api-http-cmds)# show management api http-commands
    Enabled:            Yes
    HTTPS server:       running, set to use port 443
    HTTP server:        shutdown, set to use port 80
    Local HTTP server:  shutdown, no authentication, set to use port 8080
    Unix Socket server: shutdown, no authentication
    VRF:                default
    …
• This enables eAPI only for HTTPS. For using HTTP, switch to it as shown below:

• From configure mode, enter management api http-commands mode. In this submode, you can turn on or off the server by typing [no] shutdown, switch between accepting HTTP or HTTPS traffic via [no] protocol http[s], and adjust the ports the server should listen on using protocol http[s] port <portNumber>.
•   myswitch> enable
   myswitch# configure terminal
   myswitch(config)# management api http-commands
   myswitch(config-mgmt-api-http-cmds)# [no] shutdown
   myswitch(config-mgmt-api-http-cmds)# [no] protocol https [port <portNumber>]
   myswitch(config-mgmt-api-http-cmds)# [no] protocol http [port <portNumber>]
   myswitch(config-mgmt-api-http-cmds)# [no] protocol http localhost [port <portNumber>]
   myswitch(config-mgmt-api-http-cmds)# [no] protocol unix-socket

• On-box usage of CAPI: It is often useful to run scripts that use Command API directly on the switch itself. The first is an HTTP server bound to localhost (on port 8080 by default), which only accepts connections arriving from the same machine.  The other solution is a Unix domain socket. Both can be used simultaneously also.

• Once Command API is enabled then you access via the local domain socket unix:/var/run/command-api.sock

 switch = Server( "unix:/var/run/command-api.sock" )

• If configured to use HTTP over localhost, your script can access the API as follows:

 switch = Server( "http://localhost:8080/command-api" )

Configuring a Certificate:

• Because clients use HTTP basic authentication to send usernames and passwords to the switch, we recommend using HTTPS so no passwords are sent in the clear over the network.
• By default a self-signed certificate will be used.
• You can view the current certificate using show management api http-commands https certificate

Exploring the Command API:

• To explore the API, point your web browser to http[s]://<switch-name>/, after enabling Command API.
• This web-app lets you interactively explore the protocol, return values, and model documentation.

Using Command API with Python:

• Install jsonrpclib library for installing Python JSON-RPC:

 admin:~ admin$ sudo pip install jsonrpclib

• from jsonrpclib import Server
• switch = Server( "https://username:passw0rd@myswitch/command-api" ) #Note that both username and password are compulsory. If no password, give the username itself in password field also
• response = switch.runCmds( 1, ["show version"] ) #instead of 1, we can also use “latest” to take latest version
• print "The switch's system MAC addess is", response[0]["systemMacAddress"]

• Another easy way is to import CAPIClient library and use as per its documentation on: https://github.com/arista-eosext/CAPIClient  

How it Works:

• The client starts by sending a JSON-RPC request via an HTTP POST request to http://<yourswitch>/command-api, which encapsulates a list of CLI commands it wishes to run, and the switch replies with a JSON-RPC response containing the result of each CLI command that was executed.
• If any of the commands emit an error, no further commands from that request are executed, and the response from the switch will contain an error object containing the details of the error that occurred.

Command Specification:

• In most cases, the client will use a simple string to specify the CLI command in the cmds parameter in the request. In certain cases, however, clients may wish to specify additional parameters during the command's execution.
• To use complex commands, pass a JSON object in lieu of a string, with the following attributes:

• cmd (mandatory): specify the CLI command to run.
• input (optional): specify a string to be provided as standard input while running the cmd
• revision (optional): in the case of 'show' commands that have been modified over the course of different EOS releases, this parameter allows clients to request an old model format. At this time, all models are at revision 1, and this attribute will be ignored.

• For example, to set the message of the day to Hello World!, the client should set cmds to

[ "enable", "configure", { "cmd": "banner motd", "input": "Hello World!\nEOF" } ]

• Similarly, if the switch requires an enable password, the following cmds value would let you enter exec mode and clear interface counters

[ { "cmd": "enable", "input": "hunter2" },  "clear counters" ]

Error Codes:

• The responses generated by the client library usually follow language conventions.
• For example, in Python, an error response results in an Exception being thrown, while Javascript expects an error handler callback.

Unsupported Commands:

• Certain commands are not permitted and will always return an error.
• The largest class of such commands are interactive commands .ie. those that need a response back from user or shows output continuously to user:

• watch
• reload #can be overcome by using the non-interactive ‘reload now’ command

• The bash command is only allowed with the timeout <timeout> argument, ie. bash timeout <timeout> <ARG>.
• Commands that attempt to use CLI pipes are also not allowed.

• (e.g. show interfaces | grep Ethernet1 )

• Also, no abbreviations are allowed in commands. This is necessary because future versions of EOS may add more commands, rendering previous abbreviations ambiguous.

Unconverted Commands:

• Although you can access almost any CLI command via the Command API, not all show commands have been converted to return formatted data, and trying to run the command with the format parameter set to json will result in an error.
• However, you can still get the CLI ASCII output for the unconverted command by setting the format parameter to text.

SEE: Command documentation for the respective command: http://<Switch name>/documentation.html 

Advanced Event Management on Arista EOS

Advanced Event Management

• Proactive tools include Event Manager and the Scheduler; which focus on automation.  Both tools enable scripted actions to take place in response to a pre-defined trigger.  
• When leveraged alongside SYSDB and the wealth of Linux tools that can be run on an the EOS platform, the user is offered the capability to trigger actions on virtually any aspect of system state, all without the requirement for real time user input !

Event Manager:

• Event Manager provides a platform to enable automation of actions in response to pre-defined event triggers.  
• It allows the creation of an event, the definition of under which circumstances the event should trigger and what action should occur in such a situation.
• See the below link for complete details and configuration guide: https://eos.arista.com/introduction-to-managing-eos-devices-automation-and-extensibility/
• Once an event has been triggered the configured action will be executed, this action will be initiated natively from the Linux bash shell, which means the action is not limited by the EOS CLI syntax, but rather any function or action achievable natively in the Linux bash shell.  Typical examples of actions would be to call a file, call a script, execute a native bash command or even EOS CLI commands.

• Call a bash script – action bash /mnt/flash/EmailOnLinkDown
• Call a python script to run as a daemon – action bash daemonize /mnt/flash/IntfMonitor
• Execute a single CLI command, which sends an IM to all Network admins – action bash FastCli -p15 -c ‘xmpp send NetworkAdmins command Interface Ethernet1 is down’
• Execute a series of CLI commands, which bring down a particular interface – action bash FastCli -p15 -c $’conf\n interface ethernet2\n shut’

Email Client Configuration in EOS:

• See given link: https://eos.arista.com/email-client-configuration-on-arista-switches/ 
• Once Email Client has been configured, emails can be used along with event-handler to send emails (either via CLI or bash)

Scheduler:

• While the Advance Event Manager enables actions based on complex triggers, the scheduler provides a similar functionality to repetitive time based triggers.
• The major addition to the Scheduler is that it captures the standard output of an action to a gzipped file in flash, and enables the user to configure how many of these files they wish to keep at any one time.
• To create a scheduled job, a user simply defines how often a task should run, how many log files it should store, and what the job should be.
• Unlike Event-manager, this command is executed natively in EOS, however by prepending the ‘bash’ argument   we can execute bash commands and call scripts, for example ‘command bash /mnt/flash/ConfigBackup’.
• By default EOS has a scheduled task configured to collect a show tech every 60 minutes and store up to 100 instances of the show tech, ensuring that following an issue we have both the pre and post issue data that we need to assist with analysis.

switch#show run all | grep schedule
schedule tech-support interval 60 max-log-files 100 command show tech-support

---