If you have seen any of the TextFSM posts on this site you know how useful the Network To Code TextFSM Template repository can be. Rarely do I not find what I need there!
I recently had to parse route summary information from JUNOS Looking Glass routers. I always check the very rich set of templates in the NTC Template index repository but in this case I was out of luck. I was going to have to build my own... and you get to watch.
Two fantastic resources you can use when you are in the same boat are here:
Its good to begin by familiarizing yourself with the output you need to parse. Here is a snippet of the show command output.
>show route summary
Autonomous system number: 2495
Router ID: 164.113.193.221
inet.0: 762484 destinations, 1079411 routes (762477 active, 0 holddown, 12 hidden)
Direct: 1 routes, 1 active
Local: 1 routes, 1 active
BGP: 1079404 routes, 762470 active
Static: 5 routes, 5 active
inet.2: 3073 destinations, 3073 routes (3073 active, 0 holddown, 0 hidden)
BGP: 3073 routes, 3073 active
Start with something simple like ASN and RouterID
A basic TextFSM Template
I wanted to start slowly with something I knew I could get to work. Looking at the data, it should be simple to extract the first two values I need:
- ASN
- Router ID
I started with those values as they are by far the simpler to extract from the 'show route summary' command. I will try not to cover material that is covered by the two Google links above. However I do want to point out the concept of TextFSM (as I understand it or explain it to myself) which is to provide context for your regular expressions. That is, not only can you define the specific pattern to search for but you can also define its "environment". As you can see below the "Value" keyword lets me define a variable I want to pluck out of the unstructured text (the show command output). LIne 4 defines the "action" section to start processing and the first thing to look for is a line that starts with "Autonomous system number:" one or more space noted by the \s+ and then our ASN variable which we defined above as being a pattern of one or more digits \d+. So you have the power of the regular expression that defines the value you want and the power of regular expressions to help you define the context where your value will be found.
For this exercise we will use my textfsm3 GitHub repository and the "test_textfsm.py" script for our testing rather than the Python command interpreter. Simply clone the repository to get started.
Note that the repository has the completed version of the template. Look at the history of the template file on GitHub to see its "evolution".
(txtfsm3) Claudias-iMac:textfsm3 claudia$ python test_textfsm.py -h
usage: test_textfsm.py [-h] [-v] template_file output_file
This script applys a textfsm template to a text file of unstructured data (often show commands). The resulting structured data is saved as text (output.txt) and CSV (output.csv).
positional arguments:
template_file TextFSM Template File
output_file Device data (show command) output
optional arguments:
-h, --help show this help message and exit
-v, --verbose Enable all of the extra print statements used to investigate the results
In the first iteration of the template file, we obtain the output below.
(txtfsm3) Claudias-iMac:textfsm3 claudia$ python test_textfsm.py junos_show_route_summary
.template junos_show_route_summary.txt
TextFSM Results Header:
['ASN', 'RTRID']
================================
['2495', '164.113.193.221']
================================
Extract more details
So we have successfully built a template that will extract ASN and RouterID from the Junos show route summary command. Now it will get interesting because we also want this next set of values.
- Interface
- Destinations
- Routes
- Active
- Holddown
- Hidden
The first challenge here was to pick up the totals line. Here, one of my favorite tools comes into play, RegEx101. Regular expressions don't come easy to me and this site makes it so easy! I saved the working session for trying to match the first part of that long totals line. As you can see, you can't just match "inet", or "inet" plus a digit, you also have to account for the "small." Using RegEx101 and trial and error I came up with the following regular expression.
Value INT (([a-z]+.)?[a-z]+(\d)?.\d+)
inet.0: 762484 destinations, 1079411 routes (762477 active, 0 holddown, 12 hidden)
inet6.0: 66912 destinations, 103194 routes (66897 active, 0 holddown, 30 hidden)
Direct: 3 routes, 3 active
small.inet6.0: 31162 destinations, 31162 routes (31162 active, 0 holddown, 0 hidden)
BGP: 31162 routes, 31162 active
Let's break it down...
The diagram below breaks the regex down into the key sections and numbers them. At the bottom you can see the actual text we are trying to parse and the numbers above indicate which section of the regex picked up the text we were interested in.
The regex for INT (inet.x) was by far the most complicated. See 3 and 4 above. The rest of the line is far simpler and you just need to make sure you have it exactly as it appears in the raw text. Note that the parenthesis, which are part of the raw text show command, must also be 'escaped' just like the period.
Here is the TextFSM Template so far:
Value Filldown ASN (\d+)
Value Filldown RTRID (\S+)
Value INT (([a-z]+.)?[a-z]+(\d)?.\d+)
Value DEST (\d+)
Value Required ROUTES (\d+)
Value ACTIVE (\d+)
Value HOLDDOWN (\d+)
Value HIDDEN (\d+)
Start
^Autonomous system number:\s+${ASN}
^Router ID:\s+${RTRID}
^${INT}:\s+${DEST}\s+destinations,\s+${ROUTES}\s+routes\s+\(${ACTIVE}\s+active,\s+${HOLDDOWN}\s+holddown,\s+${HIDDEN}\s+hidden\) -> Record
...and the resulting structured data:
(txtfsm3) Claudias-iMac:textfsm3 claudia$ python test_textfsm.py junos_show_route_summary.template junos_show_route_summary.txt
TextFSM Results Header:
['ASN', 'RTRID', 'INT', 'DEST', 'ROUTES', 'ACTIVE', 'HOLDDOWN', 'HIDDEN']
['2495', '164.113.193.221', 'inet.0', '762484', '1079411', '762477', '0', '12']
['2495', '164.113.193.221', 'inet.2', '3073', '3073', '3073', '0', '0']
['2495', '164.113.193.221', 'small.inet.0', '116371', '116377', '116371', '0', '0']
['2495', '164.113.193.221', 'inet6.0', '66912', '103194', '66897', '0', '30']
['2495', '164.113.193.221', 'small.inet6.0', '31162', '31162', '31162', '0', '0']
A few things to highlight, I used the 'Filldown' keyword for ASN and RTRID so that each "record" would have that information. The 'Filldown' keyword will take a value that appears once and duplicate it in subsequent records. If nothing else, it IDs the router from which the entry came but it also serves to simplify some things you might want to do down the line as each "record" has all the data. I also used the 'Required' keyword for routes to get rid of the empty last row that is generated when you used 'Filldown'.
Almost there! We just need to pick up the source routes under each totals line.
Value SOURCE (\w+)
Value SRC_ROUTES (\d+)
Value SRC_ACTIVE (\d+)
Here is what the final (for now anyway) template looks like:
Value Filldown ASN (\d+)
Value Filldown RTRID (\S+)
Value Filldown INT (([a-z]+.)?[a-z]+(\d)?.\d+)
Value DEST (\d+)
Value ROUTES (\d+)
Value ACTIVE (\d+)
Value HOLDDOWN (\d+)
Value HIDDEN (\d+)
Value SOURCE (\w+)
Value SRC_ROUTES (\d+)
Value SRC_ACTIVE (\d+)
Start
^Autonomous system number:\s+${ASN}
^Router ID:\s+${RTRID}
^${INT}:\s+${DEST}\s+destinations,\s+${ROUTES}\s+routes\s+(${ACTIVE}\s+active,\s+${HOLDDOWN}\s+holddown,\s+${HIDDEN}\s+hidden) -> Record
^\s+${SOURCE}:\s+${SRC_ROUTES}\s+routes,\s+${SRC_ACTIVE}\s+active -> Record
A few highlights. Because I wanted to store the source routes in a different value (SRC_ROUTES) I had to remove required from Routes in order to pick up the rows. I now have an extra row at the end but I can live with that for now. I also added Filldown to INT so that its clear where the source information came from.
(txtfsm3) Claudias-iMac:textfsm3 claudia$ python test_textfsm.py junos_show_route_summary.template junos_show_route_summary.txt
TextFSM Results Header:
['ASN', 'RTRID', 'INT', 'DEST', 'ROUTES', 'ACTIVE', 'HOLDDOWN', 'HIDDEN', 'SOURCE', 'SRC_ROUTES', 'SRC_ACT
IVE']
['2495', '164.113.193.221', 'inet.0', '762484', '1079411', '762477', '0', '12', '', '', '']
['2495', '164.113.193.221', 'inet.0', '', '', '', '', '', 'Direct', '1', '1']
['2495', '164.113.193.221', 'inet.0', '', '', '', '', '', 'Local', '1', '1']
['2495', '164.113.193.221', 'inet.0', '', '', '', '', '', 'BGP', '1079404', '762470']
['2495', '164.113.193.221', 'inet.0', '', '', '', '', '', 'Static', '5', '5']
['2495', '164.113.193.221', 'inet.2', '3073', '3073', '3073', '0', '0', '', '', '']
['2495', '164.113.193.221', 'inet.2', '', '', '', '', '', 'BGP', '3073', '3073']
['2495', '164.113.193.221', 'small.inet.0', '116371', '116377', '116371', '0', '0', '', '', '']
['2495', '164.113.193.221', 'small.inet.0', '', '', '', '', '', 'BGP', '116377', '116371']
['2495', '164.113.193.221', 'inet6.0', '66912', '103194', '66897', '0', '30', '', '', '']
['2495', '164.113.193.221', 'inet6.0', '', '', '', '', '', 'Direct', '3', '3']
['2495', '164.113.193.221', 'inet6.0', '', '', '', '', '', 'Local', '2', '2']
['2495', '164.113.193.221', 'inet6.0', '', '', '', '', '', 'BGP', '103185', '66888']
['2495', '164.113.193.221', 'inet6.0', '', '', '', '', '', 'Static', '4', '4']
['2495', '164.113.193.221', 'small.inet6.0', '31162', '31162', '31162', '0', '0', '', '', '']
['2495', '164.113.193.221', 'small.inet6.0', '', '', '', '', '', 'BGP', '31162', '31162']
['2495', '164.113.193.221', 'small.inet6.0', '', '', '', '', '', '', '', '']
The test_textfsm.py file will save your output into a text file as well as into a CSV file.
I did try using ROUTES for both sections and making it Required again. This got rid of the extra empty row but really impacts readability. I would have to keep track of how I used ROUTES as I would have lost the SRC_ROUTES distinction. That is a far greater sin in my opinion than an empty row at the end which is clearly just an empty row.