Whoop Dee Doo for my SuzieQ!

Posted on by Claudia de Luna

or

How I Gained Fabric Like Visibility on a Campus Network Without Any Upgrades

Last Updated: 2023-10-29

The network automation landscape moves fast and there is always something you come across which sparks your interest, but sometimes all you can do is it make note of it and hope you have time later on to take a look.

In October of last year, that is exactly what happened to me. I received my Packet Coders newsletter and noticed it was about SuzieQ. I had come across mention of SuzieQ before but mostly had made note of it because Dinesh Dutt was involved. Mr. Dutt, being one of those automation heavyweights you follow, made it noteworthy even before I knew what it was all about.

sq_email

In his newsletter Rick Donato explained the power of SuzieQ and firmly placed it on my “to learn” list for very specific reasons.

At the time, I was already involved in a network refresh project for a global Fortune 100 company and in 2022 it looked like I would be involved in a security project.

Both of those initiatives could make considerable use of the SuzieQ capabilities.

Campus Network Refresh

Part of our verification and QA process for every switch or stack we replaced involved taking a PRE snapshot and a POST snapshot (in this case a snapshot == the configuration and ~30 show commands) and doing lots of comparisons and being able to query the PRE and POST data sets for devices.

As part of verification and QA:

  • Are all the user MACs that were on the network before, there now?
  • Did a MAC move to another port (at which point we have to either move it back or move the port configuration)?
  • Where was this MAC before the switches were replaced?
  • Where was this hostname or IP before the switches were replace?
  • Are all the interfaces that were UP before Up after?
  • Are all the CDP/LLDP neighbors back?
  • Has the routing table changed?
  • Did we preserve the access vlan configuration across the switch interfaces?

This verification process worked well. It significantly reduced our testing time and more importantly reduced user post-migration issues by ~80% but I’m sure you can see the obvious issues with it.

The PRE snapshot was just that, a snapshot of state at a point in time before we started the work. If a device was not on the network when we took that snapshot we had no data for it readily available. In fact, on my “to do” list was to update our scripts to handle multiple snapshots.

SuzieQ solved this issue by giving us network state across time! So two days later, when a MAC shows up that wasn’t in our PRE or POST snapshots, and we have to figure out its interface configuration, no problem!

Security Zones and Segmentation

At the time I saw the Packet Coders email about SuzieQ, I was not sure if I would be working on a security initiative to place IoT and OT devices behind firewalls but I knew that a tool to help find and track user endpoints on the network over time would be invaluable.

In this project, a set of devices performing a specific function or supporting a specific business objective, are identified and moved over to a new subnet/vlan.

The previous sentence does not do justice to the level of effort that is actually involved in doing that but at the end of the day I get handed a list of switches, interfaces, MACs and IPs and I need to make sure those MACs are on those switches and interfaces so that I can change the port configuration and move each device over to a subnet behind the firewall.

The first list I was given was about 70% wrong. If I had taken that list without validating it and reconfigured those interfaces I would have disconnected more user endpoints than I secured.

Now we just ask for the MAC, IP, and FQDN (if the device has one). Everyone knows we have the best source of data for where that MAC was and is. In addition, we can determine when it was on the network last, where its been on the campus, etc.

All thanks to SuzieQ.

Wrapping up the “Why”

Before we get to the good stuff let me just say that those were the initial use cases I had for SuzieQ. Those uses cases do not come close to scratching the surface of what you can do with SuzieQ.

If you have had a chance to work with a controller based (“SDN”) data center solution like Cisco’s ACI where you have full visibility into the fabric and the endpoints you can start to appreciate the level of visibility you can gain into your existing “classical ethernet” network, be it data center or campus, with SuzieQ. This is what I mean by “Fabric-like” visibility.

  • Now that I have the SuzieQ “database” at my disposal, I can skip the logging in, extracting show commands, and parsing step and focus my automation efforts on customizations for each client.
  • I use the “what changed between this time and that time” output as validation to close out Change Tickets
  • For the security initiative we often have to find all the switches which have a particular vlan configured. That information is now readily available with a single command.

I could keep going but you will come up with your own use cases so let’s get started.

What IS SuzieQ

As a network engineer, I would describe SuzieQ as a Python module and a set of applications which allow you to build a “big data” repository or “database” of network state over time across your supported (by SuzieQ) devices.

Currently:

  • Arista EOS
  • Cisco’s IOS, IOS-XE, and IOS-XR platforms
  • Cisco’s NXOS (N7K with versions 8.4.4 or higher, and N9K with versions 9.3.1 or higher)
  • Cumulus Linux
  • Juniper’s Junos(QFX, EX, MX and SRX platforms)
  • Palo Alto’s Panos (version 8.0 or higher)
  • SoNIC devices
  • Linux servers

Please check the documentation for the latest list. The supported list of platforms is continually growing.

SuzieQ is open source and also offers an Enterprise version for support and additional capabilities you will want to look at if you support the network of a large organization.

If you love Pandas like I do, you are really going to love SuzieQ.

Python Module

If you have dabbled in Python and network automation this should look very familiar. We are creating a Python virtual environment, activating it, and then installing the SuzieQ module.

claudia@ubuntu20sq:~/sq$ # Making a SuzieQ directory sq under my home directory
claudia@ubuntu20sq:~/sq$ mkdir sq
claudia@ubuntu20sq:~/sq$ # Change directory (cd) into this new sq directory where I will build the SuzieQ environment
claudia@ubuntu20sq:~/sq$ cd sq
claudia@ubuntu20sq:~/sq$ # Creating a virtual environment called venv_sq
claudia@ubuntu20sq:~/sq$ python3 -m venv venv_sq
claudia@ubuntu20sq:~/sq$ # Activating the new virtual environment called venv_sq
claudia@ubuntu20sq:~/sq$ source venv_sq/bin/activate
(venv_sq) claudia@ubuntu20sq:~/sq$ # Note that the prompt now includes (venv_sq) 
(venv_sq) claudia@ubuntu20sq:~/sq$ # Make sure you have pip
(venv_sq) claudia@ubuntu20sq:~/sq$ pip -V
(venv_sq) claudia@ubuntu20sq:~/sq$ # Always a good idea to upgrade pip to the latest version
(venv_sq) claudia@ubuntu20sq:~/sq$ pip install pip --upgrade
(venv_sq) claudia@ubuntu20sq:~/sq$ # Check to see if there are any python modules installed (there should not be any)
(venv_sq) claudia@ubuntu20sq:~/sq$ pip freeze
(venv_sq) claudia@ubuntu20sq:~/sq$
(venv_sq) claudia@ubuntu20sq:~/sq$ # Install the latest version of SuzieQ
(venv_sq) claudia@ubuntu20sq:~/sq$  pip install suzieq

Applications

I split the SuzieQ applications into two basic functions, data collection and data analysis:

  • Data Collection – The SuzieQ Poller
    • The application which gathers, parses, timestamps, and stores all the network state data from your devices
    • sq-poller
    • Fun Fact: Obviously SuzieQ will log in to your network devices and gather the configuration of each device along with a series of show commands to capture the state at that time but you can also point the poller to a directory of static show commands. This comes in really handy when, for example, all you have are show command output files and no direct access to the devices on the network.
  • Data Analysis – via CLI, Web GUI, or REST
    • The various ways to view, query, and analyze your network data
    • CLI
    • suzieq-cli
    • Web GUI
    • suzieq-gui
    • REST API
    • sq-rest-server

Talk about “batteries included”!

Think of doing EDA (Exploratory Data Analysis) on your network state!

Organization

One brief note on how to organize your SuzieQ polling.

SuzieQ uses the concept of a “Namespace” for top level organization. These “Namespaces” are configured in your YAML inventory file and define groupings of devices you will poll given a source (host list, Ansible inventory file, Netbox), a platform type (Arista, Cisco, Juniper, etc.), and an authentication mechanism. For our example here I’m going to use a namespace to represent a location.

This comes in handy for example when I want to see where vlan 100 exists at the campus level. So I can filter on the namespace (representing the location or campus) and then query which switches at that location have vlan 100 configured.

As you delve deeper into SuzieQ you may find other ways to organize your network (and I’d love to hear about them). Right now namespace = location and device type is working well for my use cases but lets not overthink this. We will have alot of flexibility in querying our network state data.

The inventory file you feed to the poller (which we will create in just a bit) is highly flexible and you can define many namespaces and only poll some of them or all of them so that is another thing to consider as you organize your inventory and polling. Right now, for the security project, I have a hand-full of sources in my inventory file and I enable and disable polling depending on what is going on in the upcoming weeks.

Installation

There is quite alot of good material out there showing you how to use the SuzieQ Docker Container so I’m going to focus on the native Python module installation.

Requirements

  • Linux or Mac OSX
  • Python >3.8.1 and <3.10

If you are a Windows user, don’t worry! In this example we will bring up an Ubuntu 20 Virtual Machine on Virtual Box on a Windows system.

Here is the lab set up I will be using for this introduction.

SuzieQ_Lab

Install (as needed) and Launch Virtual Box

If you don’t already have Virtual Box installed, download:

  • the latest version (for Windows and other platforms) of the application and
  • the Extension Pack (which I strongly recommend).

Download Virtual Box and Oracle VM VirtualBox Extension Pack

Build Your Linux Virtual Machine

We are going to use an Ubuntu 20 image for our Linux machine as it already has Python 3.8 on it. If you decide to go with the latest Ubuntu (or another image you prefer) just be mindful of the Python version. You may run into compatibility issues particularly if your distribution uses Python 3.10.

Download the Ubuntu 20 Server ISO image

https://ubuntu.com/download/server

Recommended Settings

Name: Ubuntu20_SQ (Entirely up to you)

Type: Linux

Version Ubuntu (64-bit)

  • 6GB Memory
  • 40GB Disk

Once you have the initial configuration complete for your VM, click on Settings:

  • Network: Adapter 1: Bridged Network Adapter (Default is NAT)
    • This will effectively put your SuzieQ Linux server on your network so you can SSH to it as though it was just another host on your network
  • Storage: Controller IDE: Highlight “Empty” and Mount the ISO image you downloaded from Ubuntu and check Live CD
    • ubuntu-20.04.4-live-server-amd64.iso

Of course you can reduce both the storage and memory requirements based on your system capabilities. These settings worked well for me in my lab but were likely overkill.

Install Python Developer Tools

There are usually a few additional things you need to get your environment ready to go. Here is a great resource from the Digital Ocean team that I use.

How To Install Python 3 and Set Up a Programming Environment on Ubuntu 20.04 [Quickstart]

claudia@ubuntu20sq:~$ # Refresh or update your apt repository
claudia@ubuntu20sq:~$ sudo apt update
claudia@ubuntu20sq:~$
claudia@ubuntu20sq:~$ # Upgrade your system (-y will automatically answer "Yes" to almost every "Do you want to proceed?" question)
claudia@ubuntu20sq:~$ sudo apt -y upgrade
claudia@ubuntu20sq:~$
claudia@ubuntu20sq:~$ # Check your version of python - 3.7 and 3.8 work best with SuzieQ
claudia@ubuntu20sq:~$ python3 -V
Python 3.8.10
claudia@ubuntu20sq:~$
claudia@ubuntu20sq:~$ # Install PIP
claudia@ubuntu20sq:~$ sudo apt install -y python3-pip
claudia@ubuntu20sq:~$
claudia@ubuntu20sq:~$ # Install the venv module
claudia@ubuntu20sq:~$ sudo apt install -y python3-venv
claudia@ubuntu20sq:~$
claudia@ubuntu20sq:~$ # Install some handy extras
claudia@ubuntu20sq:~$ sudo apt install build-essential libssl-dev libffi-dev python3-dev
claudia@ubuntu20sq:~$

Command Summary – Complete your Python3 environment on your Ubuntu Server

sudo apt update
sudo apt -y upgrade
python3 -V
sudo apt install -y python3-pip
sudo apt install -y python3-venv
sudo apt install build-essential libssl-dev libffi-dev python3-dev
python3 -V

Install SSH

Working in the Virtual Box console is necessary initially, but once you have your system up and running its much better to SSH into it so you can use your preferred SSH client and copy and paste.

To that end, lets install SSH on your new Ubuntu system.

Here are some great directions on how to Enable SSH on Ubuntu 20.04.

These are the commands I typically use:

claudia@ubuntu20sq:~$ # Install the OpenSSH Server. Client is often part of the distribution but you should confirm.  It can be handy to be able to ssh to devices from your SuzieQ system)
claudia@ubuntu20sq:~$ sudo apt install -y openssh-server openssh-client
claudia@ubuntu20sq:~$ # Update the Ubuntu firewall to allow ssh if needed
claudia@ubuntu20sq:~$ sudo ufw allow ssh
claudia@ubuntu20sq:~$ 
claudia@ubuntu20sq:~$ # Verify SSH is runnning - Look for Loaded: loaded and Active: active (running)
claudia@ubuntu20sq:~$ sudo systemctl status ssh
[sudo] password for claudia: 
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-04-30 14:51:23 UTC; 1 day 4h ago
       Docs: man:sshd(8)
             man:sshd_config(5)
   Main PID: 1435 (sshd)
      Tasks: 1 (limit: 7024)
     Memory: 9.8M
     CGroup: /system.slice/ssh.service
             └─1435 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

Apr 30 14:57:04 ubuntu20sq sshd[20548]: Accepted password for claudia from 10.1.10.32 port 64273 ssh2
Apr 30 14:57:04 ubuntu20sq sshd[20548]: pam_unix(sshd:session): session opened for user claudia by (uid=0)
Apr 30 14:57:04 ubuntu20sq sshd[20559]: Accepted password for claudia from 10.1.10.32 port 64274 ssh2
Apr 30 14:57:04 ubuntu20sq sshd[20559]: pam_unix(sshd:session): session opened for user claudia by (uid=0)
Apr 30 16:25:53 ubuntu20sq sshd[21675]: Accepted password for claudia from 10.1.10.32 port 52413 ssh2
Apr 30 16:25:53 ubuntu20sq sshd[21675]: pam_unix(sshd:session): session opened for user claudia by (uid=0)
Apr 30 16:25:53 ubuntu20sq sshd[21677]: Accepted password for claudia from 10.1.10.32 port 52414 ssh2
Apr 30 16:25:53 ubuntu20sq sshd[21677]: pam_unix(sshd:session): session opened for user claudia by (uid=0)
Apr 30 18:14:33 ubuntu20sq sshd[22051]: Accepted password for claudia from 10.1.10.17 port 59254 ssh2
Apr 30 18:14:33 ubuntu20sq sshd[22051]: pam_unix(sshd:session): session opened for user claudia by (uid=0)
claudia@ubuntu20sq:~$

Once you have completed these steps, make sure you can SSH into your new Ubuntu server.

Command Summary – Install SSH on your Ubuntu Server

sudo apt install -y openssh-server openssh-client
sudo ufw allow ssh
sudo systemctl status ssh

Installing the SuzieQ module

Now that our server is ready to go, all that is left is to install SuzieQ and configure it.

You’ve already seen the commands to install the module. In this case we are going to install a specific version (the latest version as of this writting).

claudia@ubuntu20sq:~$ # Making a SuzieQ directory sq under my home directory
claudia@ubuntu20sq:~$ mkdir sq
claudia@ubuntu20sq:~$ # Change directory (cd) into this new sq directory where I will build the SuzieQ environment
claudia@ubuntu20sq:~$ cd sq
claudia@ubuntu20sq:~/sq$ # Creating a virtual environment called venv_sq
claudia@ubuntu20sq:~/sq$ python3 -m venv venv_sq
claudia@ubuntu20sq:~/sq$ # Activating the new virtual environment called venv_sq
claudia@ubuntu20sq:~/sq$ source venv_sq/bin/activate
(venv_sq) claudia@ubuntu20sq:~/sq$ # Note that the prompt now includes (venv_sq) 
(venv_sq) claudia@ubuntu20sq:~/sq$ # Make sure you have pip
(venv_sq) claudia@ubuntu20sq:~/sq$ pip -V
(venv_sq) claudia@ubuntu20sq:~/sq$ # Always a good idea to upgrade pip to the latest version
(venv_sq) claudia@ubuntu20sq:~/sq$ pip install pip --upgrade
(venv_sq) claudia@ubuntu20sq:~/sq$ # Check to see if there are any python modules installed (there should not be any)
(venv_sq) claudia@ubuntu20sq:~/sq$ pip freeze
(venv_sq) claudia@ubuntu20sq:~/sq$
(venv_sq) claudia@ubuntu20sq:~/sq$ # Install the latest version of SuzieQ or simply use pip install suzieq
(venv_sq) claudia@ubuntu20sq:~/sq$ pip install suzieq==0.21
(venv_sq) claudia@ubuntu20sq:~/sq$ # When you run this next command you will see many modules in your venv
(venv_sq) claudia@ubuntu20sq:~/sq$ pip freeze
(venv_sq) claudia@ubuntu20sq:~/sq$ # Lots of modules listed but removed for brevity
(venv_sq) claudia@ubuntu20sq:~/sq$ # You can now check the version of each of our SuzieQ applications
(venv_sq) claudia@ubuntu20sq:~/sq$ sq-poller -V
0.17.2
(venv_sq) claudia@ubuntu20sq:~/sq$ suzieq-cli -V
0.17.2
(venv_sq) claudia@ubuntu20sq:~/sq$ suzieq-gui -V
0.17.2
(venv_sq) claudia@ubuntu20sq:~/sq$ sq-rest-server -V
0.17.2
(venv_sq) claudia@ubuntu20sq:~/sq$

If you can successfully execute the version (-V or –version) check for each of the applications, you are ready to start configuring SuzieQ.

Command Summary

mkdir sq
cd sq
python3 -m venv venv_sq
pip -V
pip install pip --upgrade pip freeze pip install suzieq==0.21 pip freeze sq-poller -V suzieq-gui -V sq-rest-server -V

Configuring the Poller

So at this point we have:

  1. An Ubuntu Server (Virtual Machine) with a working Python3 environment and one can SSH into (and ultimately HTTPS into so we can take a peek at the GUI)
  2. A directory and a Virtual Environment for SuzieQ which we have activated
  3. SuzieQ installed in our Virtual Environment

We have two remaining steps before we have data we can work with:

  1. Configure how you want SuzieQ to “poll” or gather state from your devices <<– We are here!
  2. Configure which devices you want to poll

Configuring the poller has two parts. First, the basic behavior can be defined in the SuzieQ configuration file, suzieq-cfg.yml.

If you have worked with other automation frameworks (Ansible, Nornir, PyATS) this YAML configuration file will be familiar. Also familiar will be the “where do you put the configuration file” question.

SuzieQ will look in two places for its configuration file (unless you override this behavior on the command line):

Order Location
1 In the directory where you execute the SuzieQ commands.
In our example, ~/sq/suzieq-cfg.yml or /home/claudia/sq/suzieq-cfg.yml
2 In a hidden directory under your home directory.
~/.suzieq/suzieq-cfg.yml
For me, ~ is equivalent to /home/claudia/ and the configuration file would be under
/home/claudia/.suzieq/suzieq-cfg.yml

I prefer to have my configuration file local or within my working directory (~/sq/). I know right where it is if I need to check something and it gives me the option of setting different parameters (like polling intervals) for different use cases.

Of course, you can use both so you can have an additional level of customization.

Say by default you want to poll every 15 minutes (900 seconds). You can put a configuration file in the ~/.suzieq/ directory with that setting. Every time you launch your poller and there is no local configuration file those are the settings which will be used. When you have a specific need to poll say every 1 minute, you can put a configuration file locally with those settings. So you have the best of both worlds!

For this example, we will keep it simple. We have our sq working directory. In it we have our virtual environment. This is the directory we will use to execute the SuzieQ application commands, the first being the poller (sq-poller).

We will put our configuration file in this same directory so that when we launch the poller (or any of the other application commands), it is that local configuration file that will be used.

image-20220501081139811

Sample Configuration File

One of the things you installed when you installed the SuzieQ module is a directory with some handy samples including a sample configuration file. If you followed along with the installation documented here, you have an “sq” directory in your home directory (~) and under that a venv_sq directory for your virtual environment. From your virtual environment directory follow the path below to the lib> python3.x > site-packages > suzieq > config> etc folder and you will find the sample suzieq-cfg.yml file.

~/sq/venv_sq/lib/python3.8/site-packages/suzieq/config/etc

Once you are in the directory above you can use the Linux cat command to display the contents of the sample configuration file. I’ve also included it at the end of this article.

(venv_sq) claudia@ubuntu20sq:~/sq/venv_sq/lib/python3.8/site-packages/suzieq/config/etc$ cat suzieq-cfg.yml

Study it for a bit so you can see what controls you have within the configuration file and for which applications. As with other automation frameworks, many of these can be overwritten on the command line. If you want to delve into each available parameter here is a description of each parameter.

For the purposes of our introduction to SuzieQ, below is the configuration file I suggest you start with. I’ve added a few comments of my own. Mostly because I want to note where I changed the default setting and I always forget the time increments for the poller items.

How is this file different from the sample file?:

Section Parameter Comment
REST All Default – No Changes
POLLER logging-level Changed from WARNING to DEBUG so you can see what the poller is doing and in case you run into any issues
POLLER period Changed from poll every minute (60 seconds) to poll every 5 minutes (300 seconds)
POLLER connect-timeout Changed from 15 seconds to 60 seconds. In a lab environment the default will serve but I wanted you to know where to change this. In a production environment you will likely need to increase this value.
POLLER workers In this file you can essentially enable multi-threading by launching multiple workers to do your polling. We are taking the default in this example. If this is something of interest to you check out the sample suzieq-cfg.yml.
COALESCER All Default – No Changes
ANALYZER timezone Changed from default (use system) to explicit timezone (Los Angeles)

You will need to create a file in your sq directory called suzieq-cfg.yml with the contents below. You can use a local editor on the Linux server (I used vi in the video) and paste the file in or create the file locally and use a file transfer application like WinSCP to move the file over to your Linux server.

---
# SuzieQ Configuration File suzieq-cfg.yml
# The data-directory is your "big data" "database". By default it will be in your SuzieQ directory
data-directory: ./parquet
temp-directory: /tmp/

rest:
  # Uncomment these lines if you're using your own files for the REST server
  # The certificates listed below are provided purely to get started, In any
  # secure deployment, these must be generated specifically for the site and
  # these lines uncommented and containing the location of the site-specific file.
  # rest-certfile: /secrets/cert.pem
  # rest-keyfile: /secrets/key.pem
  #
  API_KEY: 496157e6e869ef7f3d6ecb24a6f6d847b224ee4f
  logging-level: WARNING
  address: 127.0.0.1
  port: 8000
  # no-https: True
  # logfile: /tmp/sq-rest-server.log
  # logsize is in bytes
  # logsize: 10000000
  # log-stdout: True
  # Accept all default REST settings

poller:   
# logging level default is WARNING but setting to DEBUG initially  
logging-level: DEBUG   
# period in seconds Default is 60 so poll every minute 
period: 300   
# device connection timeout in seconds Default is 15 
connect-timeout: 60 
# logfile: /tmp/sq-poller.log 
# logsize is in bytes 
# logsize: 10000000 log-stdout: True 
# inventory-file: "suzieq/conf/etc/inventory/inventory.yaml" 
# Inventory file to use if it is not specified # inventory-timeout: 10 
# The maximum time in seconds for a source to 
# return its node list to the poller 
# update-period: 3600 
# The update period of the inventory in seconds 
# The manager is the component of the poller producing the final inventory 
# and launching the workers (the components in charge of polling the nodes) 
# Uncomment these lines to customize its behaviour 
# manager: workers: 1 
# Number of workers polling the nodes 
# The chunker is the component of the poller in charge of splitting the 
# final inventory into a number of chunks equal to the number of workers 
# Uncomment these lines to customize its behaviour 
# # chunker: 
# policy: sequential

  # We will initially take the defaults:
  # One worker polling the devices or nodes in the inventory file
  # One sequential chunker for the single worker

coalescer:
  # The coalescer has the role to group the single parquet files into a bigger
  # one which represent a snapshot of the entire network, which is performed at
  # every coalescing period. This value can be expressed using the following
  # notation <value><m,h,d,w>, where:
  # m: minutes (i.e. 30m )
  # h: hours (i.e. 12h )
  # d: days (i.e. 1d )
  # w: weeks (i.e. 3w )
  period: 1h
  # Comment the next line if you don't want the archive directory. This dir is
  # purely to save the uncoalesced data in raw format to avoid data loss in case
  # of a bug in the coalescer.
  archive-directory:
  logging-level: WARNING
  # logfile: /tmp/sq-coalescer.log
  # logsize is specified in bytes
  # logsize: 10000000
  # log-stdout: True
  # Accepting all coalescer defaults

analyzer:
  # By default, the timezone is set to the local timezone. Uncomment
  # this line if you want the analyzer (CLI/GUI/REST) to display the time
  # in a different timezone than the local time. The timestamp stored in
  # the database is always in UTC.
  # Check all the supported values at the "TZ database name" columns of the
  # table at this link: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
  timezone: America/Los_Angeles
  # I will explicitly set the timezone. Default is to Use local system timezone

Your Network Inventory

We had two remaining steps, the configuration file which should now be ready and on your Linux system in your ~/sq/ directory (if you are following along), and the inventory file.

  1. Configure how you want SuzieQ to “poll” or gather state from your devices <<–Complete!
  2. Configure which devices you want to poll <<– We are here!

As with every network automation framework, you won’t get far without an inventory of your network devices.

Luckily SuzieQ supports:

  • Its own Host List format (which we will use in this introduction)
  • An Ansible Inventory file
  • Netbox

Your SuzieQ Host List inventory file will be a YAML file with 4 sections.

Section (Key) Function Example
Sources A list of sources you can use for your inventory image-20220501150508521
Devices A list of device types and connectivity parameters image-20220501150614330
Auths A list of types of authentication mechanisms image-20220501150725365
Namespaces It all comes together in this section.
This is a list of namespaces and each namespace describes its source of inventory, the device type, and the authentication mechanism to use.
You could easily add a second namespace such as:
– name: testing2
source: dc-02-suzieq-native
device: devices-using-rest
auth: suzieq-user-01
And so when starting the poller with this inventory file you would be polling two sets of “sources”, the net box-instance-123 and the dc-02-suzieq-native which has two devices.		 image-20220501150757489

SuzieQ Inventory Documentation (Source of the examples above)

Your Inventory File

Lets look at the inventory file we will be using.

I’ve named it sq_inv.yml but obviously you can use any name you like. Make sure its a valid YAML file. In fact the poller has a handy syntax checker to make our lives a little easier and which we will look at later.

Now, obviously you need some devices!

For this introduction I will be using two physical Cisco switches in my lab and a virtual Arista vEOS switch running on my Mac (also using Virtual Box). This will give us two namespaces to work with.

Sources

In our file we will use the SuzieQ native “Host List” format. I will have two sources. A “Campus” source with Cisco IOS devices and a “Data Center” source with a single Arista vEOS device.

Devices

The devices section defines two types of devices, Cisco IOS and Arista EOS. For each, you can sets the transport (ssh), the port (22, the default) and ignores the SSH keys (handy in a lab environment).

Auths

The auths section defines different sets of credentials to use for different devices. In addition, you have some nice flexibility in terms of how to deal with your passwords:

Password Parameter Use
password: plain: Use the plain notation if you want to put your
password directly in the inventory file.
This is never recommended outside
of a lab setting.
password: env: Use the env notation if you want to store your password in a system environment variable.
This is what we will do for the Cisco IOS devices.
password: ask Interactively ask for the account password 
(venv_sq) claudia@ubuntu20sq:~/sq$ cat sq_inv.yml 
---

sources:
- name: UWACO_campus_inventory_source
  hosts:
  - url: ssh://10.1.10.61
  - url: ssh://10.1.10.62
- name: UWACO_DC_inventory_source
  hosts:
  - url: ssh://10.1.10.40

devices:
- name: cisco_ios_devices
  transport: ssh
  ignore-known-hosts: true
  slow-host: true
  port: 22
- name: arista_veos_devices
    transport: ssh
    ignore-known-hosts: true
  port: 22

auths:
- name: cisco
  username: cisco
  password: plain:cisco
- name: cisco_env
  username: cisco
  password: env:NET_PWD
- name: veos
  username: admin
  password: plain:veos

namespaces:
- name: UWACO_Campus
  source: UWACO_campus_inventory_source
  device: cisco_ios_devices
  auth: cisco_env
- name: UWACO_Datacenter
  source: UWACO_DC_inventory_source
  device: arista_veos_devices
  auth: veos

Since our inventory file uses an environment variable, we had better set it.

Note: Even if you don’t use the cisco_env auths element, you do need to have that environment variable defined or you will get an error.

Setting Your Device Password in an Environment Variable

Just a quick note on environment variables. By convention, environment variables are all UPPER CASE.

If you are new to Linux, please keep in mind that Unix and Linux based operating systems are case sensitive. Coming from a Windows only environment that may be a bit of a surprise, so always be mindful of case.

Notice that in the Inventory file we said password: env:NET_PWD so SuzieQ is going to look or an environment variable named NET_PWD.

First lets make sure there is no existing NET_PWD environment variable. Note that to access the environment variable you put a $ in front of the name so the operating system recognizes that you are looking for the variable NET_PWD which holds a particular variable if defined.

(venv_sq) claudia@ubuntu20sq:~/sq$ echo $NET_PWD

(venv_sq) claudia@ubuntu20sq:~/sq$

Nothing was returned (note the blank line) so we are safe to set our password as an environment variable.

(venv_sq) claudia@ubuntu20sq:~/sq$ export NET_PWD='cisco'
(venv_sq) claudia@ubuntu20sq:~/sq$ echo $NET_PWD
cisco
(venv_sq) claudia@ubuntu20sq:~/sq$

Please note the single quotes used (NET_PWD=’cisco’ not NET_PWD=”cisco”).

Further reading on how to set Environment variables in Linux.

Starting the SuzieQ Poller

Lets take inventory of what we have so far.

  1. An Ubuntu Server (Virtual Machine) with a working Python3 environment and one can SSH into (and ultimately HTTPS into so we can take a peek at the GUI)
  2. A directory and a Virtual Environment for SuzieQ which we have activated
  3. SuzieQ installed in our Virtual Environment
  4. A SuzieQ Configuration file which defines how you want SuzieQ to “poll” or gather state from your devices
  5. A SuzieQ Inventory file using the “Host List” syntax to define which devices you want to poll

Let’s make sure our YAML inventory file is syntactically correct.

(venv_sq) claudia@ubuntu20sq:~/sq$ sq-poller -I sq_inv.yml --syntax-check
2022-05-02 01:41:14,911 - suzieq.poller.controller - WARNING - log level DEBUG
Inventory syntax check passed
(venv_sq) claudia@ubuntu20sq:~/sq$

For comparison, I created an inventory file with a TAB and bad spacing so we could see what the --syntax-check returns.

It’s not happy about the TAB character (\t) in front of the transport statement in line 19.

(venv_sq) **claudia@ubuntu20sq**:**~/sq**$ sq-poller -I sq_inv_tab.yml --syntax-check

2022-05-02 01:43:43,099 - suzieq.poller.controller - WARNING - log level DEBUG

2022-05-02 01:43:43,100 - suzieq.poller.controller - ERROR - Invalid Suzieq inventory file: while scanning for the next token

found character '\t' that cannot start any token

 in "<unicode string>", line 19, column 1:

    transport: ssh

  ^

(venv_sq) **claudia@ubuntu20sq**:**~/sq**$

Non space character in inventory file:

image-20220501184541709

Lets start gathering state from our devices.

To start the poller we use the sq-poller SuzieQ script. We use the -I option and provide the name of our YAML inventory file. In the example below I am also using the & control operator which tells the shell to run this command in the background. This returns my shell promt so I can do other things (although with our settings your shell will be tied up with all the DEBUG messages).

(venv_sq) claudia@ubuntu20sq:~/sq$ sq-poller -I sq_inv.yml &

Remember that you have to have your SuzieQ Python virtual environment activated. If you get an error like below (command not found) verify that you have indeed activated your virtual environment.

(venv_sq) claudia@ubuntu20sq:~/sq$ deactivate
claudia@ubuntu20sq:~/sq$ 
claudia@ubuntu20sq:~/sq$ 
claudia@ubuntu20sq:~/sq$ 
claudia@ubuntu20sq:~/sq$ sq-poller -h
sq-poller: command not found
claudia@ubuntu20sq:~/sq$ 
claudia@ubuntu20sq:~/sq$ source venv_sq/bin/activate
(venv_sq) claudia@ubuntu20sq:~/sq$ sq-poller
2022-05-02 02:29:08,169 - suzieq.poller.controller - WARNING - log level DEBUG
2022-05-02 02:29:08,255 - suzieq.poller.controller - ERROR - Inventory file not found in the default location:suzieq/config/etc/inventory.yaml, use -I argument to provide it, use -i instead to provide an input directory with pre-captured output and simulate an input.
(venv_sq) claudia@ubuntu20sq:~/sq$

Like all well behaved scripts you can use the -h option to look at all the options available to sq-poller. You will note that we’ve already used one of the options, --syntax-check.

(venv_sq) claudia@ubuntu20sq:~/sq$ sq-poller -h
usage: sq-poller [-h] [-I INVENTORY | -i INPUT_DIR] [-c CONFIG] [--debug] [-x EXCLUDE_SERVICES] [--no-coalescer]
                 [-o {parquet} [{parquet} ...]] [--run-once {gather,process,update}] [-s SERVICE_ONLY]
                 [--ssh-config-file SSH_CONFIG_FILE] [-p UPDATE_PERIOD] [-w WORKERS] [-V] [--syntax-check]

optional arguments:
  -h, --help            show this help message and exit
  -I INVENTORY, --inventory INVENTORY
                        Input inventory file
  -i INPUT_DIR, --input-dir INPUT_DIR
                        Directory where run-once=gather data is. Process the data in directory as they were retrieved by the hosts
  -c CONFIG, --config CONFIG
                        Controller configuration file
  --debug               Build the node list and exit without polling the nodes
  -x EXCLUDE_SERVICES, --exclude-services EXCLUDE_SERVICES
                        Exclude running this space separated list of services
  --no-coalescer        Do not start the coalescer
  -o {parquet} [{parquet} ...], --outputs {parquet} [{parquet} ...]
                        Output formats to write to: parquet. Use this option multiple times for more than one output
  --run-once {gather,process,update}
                        The poller do not run forever, three modes are available: (1) gather: store the output as it has been
                        collected, (2) process: performs some processing on the data. Both cases store the results in a plain output
                        file, one for each service, and exit. (3) update: poll the nodes only once, write the result and stop
  -s SERVICE_ONLY, --service-only SERVICE_ONLY
                        Only run this space separated list of services
  --ssh-config-file SSH_CONFIG_FILE
                        Path to ssh config file, that you want to use
  -p UPDATE_PERIOD, --update-period UPDATE_PERIOD
                        How frequently the inventory updates [DEFAULT=3600]
  -w WORKERS, --workers WORKERS
                        The number of workers polling the nodes
  -V, --version         Print suzieq version
  --syntax-check        Check inventory file syntax and return
(venv_sq) claudia@ubuntu20sq:~/sq$

Check if the Poller is Running

The poller will kick off at least three processes:

  • The poller itself
  • The worker
  • The coalescer

If you ever need to check to see if the poller processes are running you can use the Linux process status or ps command to look for these processes. Use the options -e for all the processes and f to get a full listing of each process. “Pipe” (the vertical bar symbol |) the output of that command to grep and look for the process names (poller, worker, coalescer)

ps -ef | grep poller
ps -ef | grep worker
ps -ef | grep coalescer

image-20220501191233944

Stop the Poller

Occasionally you will need to kill the poller. For this you can use the Linux kill command and provide the PID or process ID you get from the ps -ef command for the poller above.

In the example below the PID of the poller was 21925. Killing the poller should stop the worker and coalesced but its a good idea to verify using the ps -ef | grep <process name> commands above.

kill 21925

Examples of expected output before (when the poller was running) and after (after executing the kill command) are below.

image-20220501185709477

Launching the SuzieQ CLI

If you have hung in with me this far, I can’t let you go without a taste of the capability you now have at your fingertips!

It takes a single command to launch the SuzieQ CLI suzieq-cli and exit to exit from the CLI interface.

(venv_sq) claudia@ubuntu20sq:~/sq$ suzieq-cli
claudia> exit
(venv_sq) claudia@ubuntu20sq:~/sq$

Let the poller run for some time. In this example, I’ve had the poller running in my lab for over a day and I’ve made some changes periodically.

  • Added a namespace
  • Added a route to the data center core switch
  • Removed a laptop

Exploring Your Network

The CLI syntax may take a bit to get used to. In my next article I’ll write about the GUI and the REST API and delve a bit more into the commands themselves.

The cadence is typically going to be <table> <verb or action> and <filters>

Tables and Devices

Within the CLI, table show will give you a list of all the tables with parsed data you have available to query against.

The command device show will give you a listing of your namespaces and the devices you are polling within each namespace.

Quite a handy little summary.

image-20220501194413092

Where is Vlan 100 configured across my campus?

I need to know which switches have vlan 100 configured on them on the Campus devices.Based on the table show output above, I’m betting I need to query the vlan table, so I start with the command vlan. Notice that once I hit the spacebar I get some interactive help as to where to go next. Well describe looks interesting. What columns of data do I have available in the vlan table. Lets check.image-20220501195131510

OK..now I know what columns of data I can query. I want SuzieQ to show me which switches have vlan 100 configured so I’m going to go with show next. I only want to look at the Campus namespace so I will filter the namespaces accordingly. Next I want to see what has vlan 100.claudia> vlan show namespace='UWACO_Campus' vlan=100So within my UWACO_Campus namespace, only the reds-as01 switch has vlan 100 configured.Let see if that has changed at all since we started polling. By default SuzieQ will give you the data from the last poll but you can use view=all to look at every record over the time its been polling.claudia> vlan show namespace='UWACO_Campus' vlan=100 view=allIn this case, no change since I stared polling.I’m not happy with the default columns in the output. Since I know what columns are available, I’ll customize the output to be shorter and include the Timestamp.claudia> vlan show namespace='UWACO_Campus' vlan=100 view=latest columns='hostname vlan vlanName state timestamp'

image-20220501195438533

Have my Routes Changed?

I started polling my lab devices yesterday. Today I added a route to my core Data Center device (2.2.2.1/32). Will SuzieQ show me that?image-20220501201336130

Indeed it does!!

Where is my Laptop?

My windows laptop has a MAC address of 00:e0:4c:36:01:9e. I’d like to find it on the network.claudia> mac show macaddr='00:e0:4c:36:01:9e'Notice that my first query returns an Empty DataFrame! I know it was on the network at some point.Remembering that a query like that defaults to view=latest, this is true. In the latest poll, my Laptop was not on the network.Lets try view=all.claudia> mac show macaddr='00:e0:4c:36:01:9e' view=allTwo records were returned. The first one, number 0, has a + which tells me that’s when it was first seen on the network and added to the “datbase”.The second, number 1, has a – which tells me thats when the laptop fell off the network.OK..Id like to see the respective timestamps. I know if I run mac describe I will get a listing of all the columns available. Let see if timestamp data is available. It is!So in my last query I’ll specify the columns I want to view and include the timestamp column.claudia> mac show macaddr='00:e0:4c:36:01:9e' view=all columns='active namespace hostname vlan macaddr oif moveCount timestamp'Now I can see when the Laptop fell off the network.image-20220501203507803A day later, if I run the same queries you can see that my Laptop is back on the network but on a different switch and interface and so moveCount is now 1! On the left you can see the common “Diff” notation of + and – clearly showing when the device went away and came back.image-20220502074316586

Videos

Part 1 – Setting up your Linux Virtual Machine

Part 2 – Installing SuzieQ

Tips

Credentials and Accounts

  • The SuzieQ Team recommends a read only account be used for polling but keep in mind that some of the show commands required do need elevated privilege.

Speed

  • SuzieQ is FAST! In some cases, too fast for the environment. In larger and global enterprises where latency can become a factor for connectivity and authentication services like TACACS SuzieQ may require some “fine tuning”

Enabling and Disabling the Poller and Changing your data file

  • Your poller data file (what I keep calling a “database” but is actually a Big Data file using Apache parquet ) will be in your working directory. If you have followed along it will be under your home directory in your sq directory. Recall that its location is defined in your suzieq-cfg.yml file. You can change that! Say you want to test something but you want to keep your original file. Just update your configuration file to point to a new directory. It will create the directory for you.
  • data-directory: ./parquet is the default but you can stop the poller and temporarily change that to say data-directory: ./parquet_test. Restart the poller and now your data file will be parquet_test. Once you are done testing, you can revert back to the original parquet file.

image-20220502073103103

  • Tap into a Pandas query
    • Remember I mentioned that SuzieQ makes extensive use of Pandas (at least in the open source version). You can actually craft your own query strings for the data frame!

image-20220502094239289

No one ever gets my references…

I fear almost no one gets the “whoop de doo” reference, apparently myself included. That little phrase always stuck in my head. Its cute and catchy to be sure and my first car was a Subaru. This is the point where my father would groan and chuckle simultaneously and explain to you that my little Subaru came at the cost of his new car. My response was always..”I would have happily driven your new car!” That thought always gave him pause.In any case, I was convinced it was from a Subaru commercial. I wanted to find it for this article and much to my surprise this phrase is actually from a Midas commercial from the 70s!midas_comercial_imagehttps://youtu.be/8KWw7zh05Lw?t=24

Resources

Documentation

SuzieQ Documentation

SuzieQ has great starter documentation but does focus on using the Docker container so I’m hoping this will be helpful for those who can’t use the Docker container or prefer not to.

Training

Rick Donato and his team at Packet Coders offer SuzieQ training (in addition to other excellent Automation training)!

Community Support

There is an active development community and as I mentioned new features and platforms are in the queue. In the Documentation link you an find information on how to join the community Slack channel.

Adding Platforms

Dinesh Dutt and the SuzieQ team are incredibly helpful if you run into any issues. If you work with a platform that is currently not supported log a feature enhancement on GitHub.

Sample SuzieQ Configuration File (suzieq-cfg.yml)

(venv_sq) claudia@ubuntu20sq:~/sq/venv_sq/lib/python3.8/site-packages/suzieq/config/etc$ cat suzieq-cfg .yml
data-directory: ./parquet
temp-directory: /tmp/
rest:
# Uncomment these lines if you're using your own files for the REST server
# The certificates listed below are provided purely to get started, In any
# secure deployment, these must be generated specifically for the site and
# these lines uncommented and containing the location of the site-specific file.
# rest-certfile: /secrets/cert.pem
# rest-keyfile: /secrets/key.pem
#
API_KEY: 496157e6e869ef7f3d6ecb24a6f6d847b224ee4f
logging-level: WARNING
address: 127.0.0.1
port: 8000
# no-https: True
# logfile: /tmp/sq-rest-server.log
# logsize is in bytes
# logsize: 10000000
# log-stdout: True
poller:
logging-level: WARNING
period: 60
connect-timeout: 15
# logfile: /tmp/sq-poller.log
# logsize is in bytes
# logsize: 10000000
log-stdout: True
# inventory-file: "suzieq/conf/etc/inventory/inventory.yaml" # Inventory file to use if it is not sp ecified
# inventory-timeout: 10 # The maximum time in seconds for a sou rce to
# return its node list to the poller
# update-period: 3600 # The update period of the inventory in seconds
# The manager is the component of the poller producing the final inventory
# and launching the workers (the components in charge of polling the nodes)
# Uncomment these lines to customize its behaviour
#
# manager:
# workers: 1 # Number of workers polling the nodes
# The chunker is the component of the poller in charge of splitting the
# final inventory into a number of chunks equal to the number of workers
# Uncomment these lines to customize its behaviour
#
# chunker:
# policy: sequential
coalescer:
# The coalescer has the role to group the single parquet files into a bigger
# one which represent a snapshot of the entire network, which is performed at
# every coalescing period. This value can be expressed using the following
# notation <value><m,h,d,w>, where:
# m: minutes (i.e. 30m )
# h: hours (i.e. 12h )
# d: days (i.e. 1d )
# w: weeks (i.e. 3w )
period: 1h
# Comment the next line if you don't want the archive directory. This dir is
# purely to save the uncoalesced data in raw format to avoid data loss in case
# of a bug in the coalescer.
archive-directory:
logging-level: WARNING
# logfile: /tmp/sq-coalescer.log
# logsize is specified in bytes
# logsize: 10000000
# log-stdout: True
analyzer:
# By default, the timezone is set to the local timezone. Uncomment
# this line if you want the analyzer (CLI/GUI/REST) to display the time
# in a different timezone than the local time. The timestamp stored in
# the database is always in UTC.
# Check all the supported values at the "TZ database name" columns of the
# table at this link: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# timezone: America/Los_Angeles