Building a Custom TextFSM Template

If you have seen any of the TextFSM posts on this site you know how useful the Network To Code TextFSM Template repository can be. Rarely do I not find what I need there!

I recently had to parse route summary information from JUNOS Looking Glass routers. I always check the very rich set of templates in the NTC Template index repository but in this case I was out of luck. I was going to have to build my own… and you get to watch.

Two fantastic resources you can use when you are in the same boat are here:

Its good to begin by familiarizing yourself with the output you need to parse. Here is a snippet of the show command output.

>show route summary
Autonomous system number: 2495
Router ID: 164.113.193.221
inet.0: 762484 destinations, 1079411 routes (762477 active, 0 holddown, 12 hidden)
Direct: 1 routes, 1 active
Local: 1 routes, 1 active
BGP: 1079404 routes, 762470 active
Static: 5 routes, 5 active
inet.2: 3073 destinations, 3073 routes (3073 active, 0 holddown, 0 hidden)
BGP: 3073 routes, 3073 active

Start with something simple like ASN and RouterID

A basic TextFSM Template

I wanted to start slowly with something I knew I could get to work. Looking at the data, it should be simple to extract the first two values I need:
– ASN
– Router ID

I started with those values as they are by far the simpler to extract from the ‘show route summary’ command. I will try not to cover material that is covered by the two Google links above. However I do want to point out the concept of TextFSM (as I understand it or explain it to myself) which is to provide context for your regular expressions. That is, not only can you define the specific pattern to search for but you can also define its “environment”. As you can see below the “Value” keyword lets me define a variable I want to pluck out of the unstructured text (the show command output). LIne 4 defines the “action” section to start processing and the first thing to look for is a line that starts with “Autonomous system number:” one or more space noted by the \s+ and then our ASN variable which we defined above as being a pattern of one or more digits \d+. So you have the power of the regular expression that defines the value you want and the power of regular expressions to help you define the context where your value will be found.

Junos ‘show route summary’ TextFSM Template – Version 1

For this exercise we will use my textfsm3 GitHub repository and the “test_textfsm.py” script for our testing rather than the Python command interpreter. Simply clone the repository to get started.
Note that the repository has the completed version of the template. Look at the history of the template file on GitHub to see its “evolution”.

(txtfsm3) Claudias-iMac:textfsm3 claudia$ python test_textfsm.py -h
usage: test_textfsm.py [-h] [-v] template_file output_file
This script applys a textfsm template to a text file of unstructured data (often show commands). The resulting structured data is saved as text (output.txt) and CSV (output.csv).
positional arguments:
template_file TextFSM Template File
output_file Device data (show command) output
optional arguments:
-h, --help show this help message and exit
-v, --verbose Enable all of the extra print statements used to investigate the results

In the first iteration of the template file, we obtain the output below.

(txtfsm3) Claudias-iMac:textfsm3 claudia$ python test_textfsm.py junos_show_route_summary
.template junos_show_route_summary.txt

TextFSM Results Header:
['ASN', 'RTRID']
================================
['2495', '164.113.193.221']
================================

Extract more details

So we have successfully built a template that will extract ASN and RouterID from the Junos show route summary command. Now it will get interesting because we also want this next set of values.

  • Interface
  • Destinations
  • Routes
  • Active
  • Holddown
  • Hidden

The first challenge here was to pick up the totals line. Here, one of my favorite tools comes into play, RegEx101. Regular expressions don’t come easy to me and this site makes it so easy! I saved the working session for trying to match the first part of that long totals line. As you can see, you can’t just match “inet”, or “inet” plus a digit, you also have to account for the “small.” Using RegEx101 and trial and error I came up with the following regular expression.

Value INT (([a-z]+.)?[a-z]+(\d)?.\d+)

inet.0: 762484 destinations, 1079411 routes (762477 active, 0 holddown, 12 hidden)

inet6.0: 66912 destinations, 103194 routes (66897 active, 0 holddown, 30 hidden)
Direct: 3 routes, 3 active

small.inet6.0: 31162 destinations, 31162 routes (31162 active, 0 holddown, 0 hidden)
BGP: 31162 routes, 31162 active

Let’s break it down…

The diagram below breaks the regex down into the key sections and numbers them. At the bottom you can see the actual text we are trying to parse and the numbers above indicate which section of the regex picked up the text we were interested in.

Breaking down the regular expression to extract the interface identifier (inet.x) for your TextFSM Template

The regex for INT (inet.x) was by far the most complicated. See 3 and 4 above. The rest of the line is far simpler and you just need to make sure you have it exactly as it appears in the raw text. Note that the parenthesis, which are part of the raw text show command, must also be ‘escaped’ just like the period.

Here is the TextFSM Template so far:

 Value Filldown ASN (\d+)
Value Filldown RTRID (\S+)
Value INT (([a-z]+.)?[a-z]+(\d)?.\d+)
Value DEST (\d+)
Value Required ROUTES (\d+)
Value ACTIVE (\d+)
Value HOLDDOWN (\d+)
Value HIDDEN (\d+)
Start
^Autonomous system number:\s+${ASN}
^Router ID:\s+${RTRID}
^${INT}:\s+${DEST}\s+destinations,\s+${ROUTES}\s+routes\s+\(${ACTIVE}\s+active,\s+${HOLDDOWN}\s+holddown,\s+${HIDDEN}\s+hidden\) -> Record

…and the resulting structured data:

(txtfsm3) Claudias-iMac:textfsm3 claudia$ python test_textfsm.py junos_show_route_summary.template junos_show_route_summary.txt
TextFSM Results Header:
['ASN', 'RTRID', 'INT', 'DEST', 'ROUTES', 'ACTIVE', 'HOLDDOWN', 'HIDDEN']
['2495', '164.113.193.221', 'inet.0', '762484', '1079411', '762477', '0', '12']
['2495', '164.113.193.221', 'inet.2', '3073', '3073', '3073', '0', '0']
['2495', '164.113.193.221', 'small.inet.0', '116371', '116377', '116371', '0', '0']
['2495', '164.113.193.221', 'inet6.0', '66912', '103194', '66897', '0', '30']
['2495', '164.113.193.221', 'small.inet6.0', '31162', '31162', '31162', '0', '0']

A few things to highlight, I used the ‘Filldown’ keyword for ASN and RTRID so that each “record” would have that information. The ‘Filldown’ keyword will take a value that appears once and duplicate it in subsequent records. If nothing else, it IDs the router from which the entry came but it also serves to simplify some things you might want to do down the line as each “record” has all the data. I also used the ‘Required’ keyword for routes to get rid of the empty last row that is generated when you used ‘Filldown’.

Almost there! We just need to pick up the source routes under each totals line.

Value SOURCE (\w+)
Value SRC_ROUTES (\d+)
Value SRC_ACTIVE (\d+)

Here is what the final (for now anyway) template looks like:

 Value Filldown ASN (\d+)
Value Filldown RTRID (\S+)
Value Filldown INT (([a-z]+.)?[a-z]+(\d)?.\d+)
Value DEST (\d+)
Value ROUTES (\d+)
Value ACTIVE (\d+)
Value HOLDDOWN (\d+)
Value HIDDEN (\d+)
Value SOURCE (\w+)
Value SRC_ROUTES (\d+)
Value SRC_ACTIVE (\d+)

Start
^Autonomous system number:\s+${ASN}
^Router ID:\s+${RTRID}
^${INT}:\s+${DEST}\s+destinations,\s+${ROUTES}\s+routes\s+(${ACTIVE}\s+active,\s+${HOLDDOWN}\s+holddown,\s+${HIDDEN}\s+hidden) -> Record
^\s+${SOURCE}:\s+${SRC_ROUTES}\s+routes,\s+${SRC_ACTIVE}\s+active -> Record

A few highlights. Because I wanted to store the source routes in a different value (SRC_ROUTES) I had to remove required from Routes in order to pick up the rows. I now have an extra row at the end but I can live with that for now. I also added Filldown to INT so that its clear where the source information came from.

(txtfsm3) Claudias-iMac:textfsm3 claudia$ python test_textfsm.py junos_show_route_summary.template junos_show_route_summary.txt

TextFSM Results Header:
['ASN', 'RTRID', 'INT', 'DEST', 'ROUTES', 'ACTIVE', 'HOLDDOWN', 'HIDDEN', 'SOURCE', 'SRC_ROUTES', 'SRC_ACT
IVE']
['2495', '164.113.193.221', 'inet.0', '762484', '1079411', '762477', '0', '12', '', '', '']
['2495', '164.113.193.221', 'inet.0', '', '', '', '', '', 'Direct', '1', '1']
['2495', '164.113.193.221', 'inet.0', '', '', '', '', '', 'Local', '1', '1']
['2495', '164.113.193.221', 'inet.0', '', '', '', '', '', 'BGP', '1079404', '762470']
['2495', '164.113.193.221', 'inet.0', '', '', '', '', '', 'Static', '5', '5']
['2495', '164.113.193.221', 'inet.2', '3073', '3073', '3073', '0', '0', '', '', '']
['2495', '164.113.193.221', 'inet.2', '', '', '', '', '', 'BGP', '3073', '3073']
['2495', '164.113.193.221', 'small.inet.0', '116371', '116377', '116371', '0', '0', '', '', '']
['2495', '164.113.193.221', 'small.inet.0', '', '', '', '', '', 'BGP', '116377', '116371']
['2495', '164.113.193.221', 'inet6.0', '66912', '103194', '66897', '0', '30', '', '', '']
['2495', '164.113.193.221', 'inet6.0', '', '', '', '', '', 'Direct', '3', '3']
['2495', '164.113.193.221', 'inet6.0', '', '', '', '', '', 'Local', '2', '2']
['2495', '164.113.193.221', 'inet6.0', '', '', '', '', '', 'BGP', '103185', '66888']
['2495', '164.113.193.221', 'inet6.0', '', '', '', '', '', 'Static', '4', '4']
['2495', '164.113.193.221', 'small.inet6.0', '31162', '31162', '31162', '0', '0', '', '', '']
['2495', '164.113.193.221', 'small.inet6.0', '', '', '', '', '', 'BGP', '31162', '31162']
['2495', '164.113.193.221', 'small.inet6.0', '', '', '', '', '', '', '', '']

The test_textfsm.py file will save your output into a text file as well as into a CSV file.
I did try using ROUTES for both sections and making it Required again. This got rid of the extra empty row but really impacts readability. I would have to keep track of how I used ROUTES as I would have lost the SRC_ROUTES distinction. That is a far greater sin in my opinion than an empty row at the end which is clearly just an empty row.

A quick example of using TextFSM to parse data from Cisco show commands

TextFSM is a powerful parsing tool (python module) developed by Google.    There are some great examples out there to get you started. Here are two I urge you to read if this topic is of interest to you:

I can never get enough of examples so here is a very simple one to get you started or keep you practicing.  I find that a quick example where I can see results of my own making really energizes my learning process.

For this example, you need python 2.7 and the textfsm module installed.

TextFSM Getting Started is an excellent resource which includes the installation process for textfsm (textfsm is a pip installable module).

In addition to the environment, you will need 2 things to get started.

  • A TextFMS template
  • Content to parse with the TextFMS template

The template may be the tricky part.  This template defines the structure of the data you are trying to gather or parse out of your content.  We are very fortunate that the Network to Code (NTC) team has given us a large library of templates from which to choose and so we don’t have to worry too much about the template itself. We just need to know what Cisco IOS show command has the information we want.

Once you have your template, you need content to parse.  You can get this in a variety of ways.  You can query your devices real time via Ansible or via a python scirpt or you can act on file based data that you already have.

In this example we will keep it simple and assume we have a text file of show commands that we need to parse to get the device hardware information and software version.

To get hardware and software information, the “show version” output should have what we want.  So looking at the existing templates in the NTC library, it looks like the cisco_ios_show_version.template has what we need, If we look at the template we can see that it has two variables, VERSION and HARDWARE (which will return a list).  That looks just about right for the information we want to extract and luckily the file of show commands includes the output of the “show version” command.

So here are the two files we will work with in this example:

– the textFSM template file (downloadable from GitHub)
ntc-templates/templates/cisco_ios_show_version.template

– the content file with show commands including the output of the show version command (downloadable here)
lab-swtich-show-cmds

For simplicity I’ve put them both in a temp directory and I will launch the python interpreter from there so we can work real time. I also list the modules that I have in the virtual environment. The only one you need for this example is textfsm.

Working directory and files
(textfsm) Eugenias-PB:textfsm-example eugenia$ pwd
/Users/eugenia/Downloads/textfsm-example
(textfsm) Eugenias-PB:textfsm-example eugenia$ ls
cisco_ios_show_version.template lab-swtich-show-cmds.txt
(textfsm) Eugenias-PB:textfsm-example eugenia$ ls -al
total 24
drwxr-xr-x 4 eugenia staff 136 May 11 05:06 .
drwx------+ 118 eugenia staff 4012 May 11 05:04 ..
-rw-r--r--@ 1 eugenia staff 680 May 11 05:06 cisco_ios_show_version.template
-rw-r--r--@ 1 eugenia staff 7425 May 11 05:05 lab-swtich-show-cmds.txt

The environment
(textfsm) Eugenias-PB:~ eugenia$ python --version
Python 2.7.15


(textfsm) Eugenias-PB:textfsm-example eugenia$ pip freeze
et-xmlfile==1.0.1
graphviz==0.8.2
jdcal==1.3
netaddr==0.7.19
openpyxl==2.5.1
**textfsm==0.3.2**
xlrd==1.1.0
(textfsm) Eugenias-PB:textfsm-example eugenia$
Lets get started…
Launch the interpreter and import the textfsm module.  The ">>>" tells you that you are in the python command interpreter. 


(textfsm) Eugenias-PB:textfsm-example eugenia$ python
Python 2.7.10 (default, Feb 7 2017, 00:08:15)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import textfsm
Open the template file into a file handle I've called template and pass that as the argument to the textfsm.TextFSM method
creating a "parsing object" based on that template (the show version template in our case).


>>> template = open('cisco_ios_show_version.template')
>>> results_template = textfsm.TextFSM(template)
Look at some of the methods available in the results_template object by using dir() on the results_template object you just created.  Make note of the 'ParseText' one 
as that is the one we will use shortly to parse our content.


>>> dir(results_template)
['GetValuesByAttrib', 'MAX_NAME_LEN', 'ParseText', 'Reset', '_AppendRecord', '_AssignVar', '_CheckLine', '_CheckRule', '_ClearAllRecord', '_ClearRecord', '_DEFAULT_OPTIONS', '_GetHeader', '_GetValue', '_Operations', '_Parse', '_ParseFSMState', '_ParseFSMVariables', '_ValidateFSM', '_ValidateOptions', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_cur_state', '_cur_state_name', '_line_num', '_options_cls', '_result', 'comment_regex', 'header', 'state_list', 'state_name_re', 'states', 'value_map', 'values']

>>> results_template.value_map
{'UPTIME': '(?P<UPTIME>.+)', 'HOSTNAME': '(?P<HOSTNAME>\\S+)', 'RUNNING_IMAGE': '(?P<RUNNING_IMAGE>\\S+)', 'CONFIG_REGISTER': '(?P<CONFIG_REGISTER>\\S+)', 'HARDWARE': '(?P<HARDWARE>\\S+\\d\\S+)', 'VERSION': '(?P<VERSION>.+?)', 'SERIAL': '(?P<SERIAL>\\S+)', 'ROMMON': '(?P<ROMMON>\\S+)'}

>>> results_template.values
[<textfsm.TextFSMValue object at 0x107c0ac90>, <textfsm.TextFSMValue object at 0x107c181d0>, <textfsm.TextFSMValue object at 0x107c18450>, <textfsm.TextFSMValue object at 0x107cf1750>, <textfsm.TextFSMValue object at 0x107cf1790>, <textfsm.TextFSMValue object at 0x107cf17d0>, <textfsm.TextFSMValue object at 0x107cf18d0>, <textfsm.TextFSMValue object at 0x107cf1950>]


You don’t need the last three commands. I just wanted to show you how to investigate the object a little which is quite easy when working in the interpreter.

This is the heart of what we are trying to do now that we’ve selected our parsing template and created an object that will parse our content and pull out the information we want.

First we open our text file of show commands. Below I create a file handle (basically a variable) to the 
show command file called content2parse and read the contents of that file into the variable content. 
You have to open the file first before you can read its contents.

>>> content2parse = open('lab-swtich-show-cmds.txt')
>>> content = content2parse.read()



Now we parse out the data we want using our results_template object (and its ParseText) method against 
our content and store the results in the parsed_results variable. As you can see, this is a list of lists. 
If you ran this against show command files from 5 different devices you can begin to see the possibilities. 
You would have a list of 5 lists with the show version information for each device.

>>> parsed_results = results_template.ParseText(content)
>>> parsed_results
[['15.2(2)E3', 'Bootstrap', 'artic-sw01', '4 days, 14 hours, 2 minutes', 'c2960s-universalk9-mz.152-2.E3.bin', ['WS-C2960S-24TS-S'], ['FOC1709W1DT'], '0xF']]



Your first inclination might be to iterate over the results but remember that its a list of lists so iterating over the results
would iterate over the only item in the "top level" list [0]. What you want is to iterate over each element of the results list
which is also a list. I know that my top level list has 1 element, the [0] element, and that list has 8 elements.  We only parsed one file which is why our list only has one element.  If I iterate 
over the [0] element of the list I get each individual bit of information.

Get the length of the top level list (should only have one element since we only parsed 1 file)
>>> len(parsed_results)
1
Get the length of that first element
>>> len(parsed_results[0])
8
>>> for item in parsed_results:
... print(item)
...
['15.2(2)E3', 'Bootstrap', 'artic-sw01', '4 days, 14 hours, 2 minutes', 'c2960s-universalk9-mz.152-2.E3.bin', ['WS-C2960S-24TS-S'], ['FOC1709W1DT'], '0xF']

>>> index = 0
>>> for item in parsed_results[0]:
... print("Element # {}: {}".format(index,item))
... index = index + 1
...
Element # 0: 15.2(2)E3
Element # 1: Bootstrap
Element # 2: artic-sw01
Element # 3: 4 days, 14 hours, 2 minutes
Element # 4: c2960s-universalk9-mz.152-2.E3.bin
Element # 5: ['WS-C2960S-24TS-S']
Element # 6: ['FOC1709W1DT']
Element # 7: 0xF
>>>


I know I have 8 elements (0-7) because i checked the length and if you do a length on the value_map you can see 
that the template is designed to get 8 pieces of information so that makes sense.

Get the length of the first list
>>> len(parsed_results[0])
8
Get the length of the value map
>>> len(results_template.value_map)
8



Finally, after all of this I wanted to get the version and hardware information out of this file and here it is:


>>> parsed_results[0][0]
'15.2(2)E3'
>>> parsed_results[0][5]
['WS-C2960S-24TS-S']
>>>

I hope these basics help you get started or keep practicing.

Here are all the main commands in sequence so you can see the flow without all of the interruptions. Also easier to copy and paste!

Python 2.7.10 (default, Feb 7 2017, 00:08:15)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import textfsm
>>> template = open('cisco_ios_show_version.template')
>>> results_template = textfsm.TextFSM(template)
>>> content2parse = open('lab-swtich-show-cmds.txt')
>>> content = content2parse.read()
>>> parsed_results = results_template.ParseText(content)
>>> parsed_results
[['15.2(2)E3', 'Bootstrap', 'artic-sw01', '4 days, 14 hours, 2 minutes', 'c2960s-universalk9-mz.152-2.E3.bin', ['WS-C2960S-24TS-S'], ['FOC1709W1DT'], '0xF']]