The Hunt for a Cisco ACI Lab

As an independent consultant one of the things I have to provide for myself are labs.

It’s a wonderful time for labs! Virtual capabilities and offerings make testing and modeling a clients network easier than ever.

Cisco DevNet offers “Always On” Sandbox devices that are always there if you need to test a “unit of automation”. Network to Code labs are also available on demand (at a small cost) and a huge time saver.

But Cisco’s Application Centric Infrastructure (ACI) is a different animal altogether.

Cisco offers a simulator (hardware and VM) (check out the DevNet Always On APIC to see it in action) which is terrific for automation testing and configuration training but there is no data plane so testing routing, vpcs, trunks, and contracts is a non starter. For that you need the hardware…and so began my search for an ACI Lab for rent.

The compilation below is still a work in progress but I thought I would share my findings, so far, over the last 6 months.

First let’s set the context.

I was looking to rent a physical ACI lab (not a simulator) with the following requirements:

  • Gen2 or better hardware (to test ACL logging among other things) and ACI 4.0 or later
  • Accessible (Configurable) Layer 3 device or devices (to test L3 Outs)
  • Accessible (Configurable) Layer 2 device or devices (to test L2)
  • VMM integration (vCenter and ESXi Compute)
  • Pre configured test VMs
  • My own custom test VMs
  • A means of running Ansible and/or Python within the lab environment and cloning repositories

Going in, I didn’t have a timeframe in mind. I would take a 4 hour block of time or a week and that is still the case. I also realized that it was unlikely anything for rent would meet all of my requirements but that was the baseline I would use to assess the various offerings.

A lower cost option for just a few hours is handy for quick tests.  Having a lab for a week is very nice and gives you the latitude to test without the time pressure.

Out of 13 possible options, I’ve tried 4. Two were very good and I would use them again and 2 were the opposite.

Recommendations:

While the INE lab is a bit difficult to schedule (you have to plan ahead which isn’t compatible with my “immediate gratification” approach to things) it’s a terrific lab with configuration access to the L2/L3 devices which gives you good flexibility.  

NterOne also offers a terrific lab and I was able to schedule it within a week of when I called. The lab is superbly documented and designed so as to make it very easy to understand. I got two pods/tenants which gave me some good flexibility in terms of exporting and testing contracts etc. The L2 and L3 devices are read only and pre-configured so those are a little limiting.

Some observations:

  • So far, no one is running Gen2+ equipment.
  • Almost all of the designs I have seen have single link L2/L3s so its difficult to test vpcs (and you generally need access to the other device unless its been preconfigured for you).
  • All the labs were running ACI 4.x

Global Knowledge has some interesting offerings and I was very excited initially but even getting the simplest answer was impossible. Like many larger companies, if you try to deviate from the menu it does not go well. I moved on.

INE, NterOne, and Firefly all spent the time understanding my requirements and offering solutions. Sadly Firefly was way out of my price range.

On a final note, I would avoid My CCIE Rack and CCIE Rack Rentals which may actually use the same lab. Documentation is terrible and I’ve tried 3 or 4 times and have gotten in about 50% of the time. The first time, I didn’t realize I needed to rent both of their DC labs (one has ACI and the other gives you access to the L2/L3 devices). The last time I rented a lab (both labs) they just simply cancelled my labs and never responded to emails either before or after.  If you have money you would like to dispose of, send it here (Coral Restoration Foundation Curaçao) or some other worthy cause. A much better use of those funds I’d have to say.

If anyone has has good experience with an ACI Lab rental that I’ve not included here I would love to hear about it!

Kudos to INE and NterOne for great customer service and flexibility!  

Summary

No alt text provided for this image

1. The INE Staff was open to allowing me to put some of my own repos and tools into the environment but when I scheduled the lab that became problematic.    INE was very honorable and let me have the lab in the non-customized way for the week without charge since they were not able to honor my customization request at that time!!

2. The Student Jump box can be customized which was very nice (I had access to my GitHub repos) and Python was available although it was Python 2.7.

3. Cost is not unreasonable but there is a minimum of

4. students so unless you have 3 like minded friends it becomes very expensive. 4 I’ve always been a big fan of Global Knowledge but my interactions with them were not positive.  I could not get even the most basic question answered (for example, did they have a money back guarantee or 30 day return policy since I was never able to get my more specific questions answered?  I figured if I had 30 days to see if the lab met my requirements then I could test it out and find out for myself.)

5.  Great customer service but the pricing was a non starter. $$$+ per day and it would have bene limited to business hours

6. When I first reached out with questions about their ACI lab they said it would not be available until late October (I assumed this year).  When I reached out in November, they didn’t even answer the question so clearly this is still al work in progress.

7. Worthy of further investigation


Details and Links

Cost Legend:

  • $ Less than $200
  • $$ Hundreds
  • $$$ Thousands

INE $$

CCIE Data Center – 850 Tokens/Week (Weekly rentals only) (1$ = 1 Token)

Excellent lab but very busy (because it’s very good) and so can be difficult to schedule.

NterOne $$

Excellent lab with good functionality at a reasonable price point.

Fast Lane $$$

Minimum of 4 Students @ $439/Student

Global Knowledge $$$

On Demand (12 Months)

Very poor customer support (my experience)

CloudMyLab

Lab not available yet. No timeframe given.

Octa Networks

More course focused but awaiting response.

Labs 4 Rent

INDIA: +91-9538 476 467  |  UAE: +971-589 703 499 | Email: info@labs4rent.com

No response to emails

FireFly $$$+ /day

Too expensive (for me)!

Rack Professionals

Needs further investigation

NH Networkers Home

+91-8088617460 / +91-8088617460

Needs further investigation

Micronics Training

They do not rent out their racks.

My CCIE Rack $

support@myccierack.com. Whatsapp number- 7840018186

Very poor experience

CCIE Rack Rentals $

support@ccierack.rentals WhatsApp : +918976927692

Very poor experience

Building a Custom TextFSM Template

If you have seen any of the TextFSM posts on this site you know how useful the Network To Code TextFSM Template repository can be. Rarely do I not find what I need there!

I recently had to parse route summary information from JUNOS Looking Glass routers. I always check the very rich set of templates in the NTC Template index repository but in this case I was out of luck. I was going to have to build my own… and you get to watch.

Two fantastic resources you can use when you are in the same boat are here:

Its good to begin by familiarizing yourself with the output you need to parse. Here is a snippet of the show command output.

>show route summary
Autonomous system number: 2495
Router ID: 164.113.193.221
inet.0: 762484 destinations, 1079411 routes (762477 active, 0 holddown, 12 hidden)
Direct: 1 routes, 1 active
Local: 1 routes, 1 active
BGP: 1079404 routes, 762470 active
Static: 5 routes, 5 active
inet.2: 3073 destinations, 3073 routes (3073 active, 0 holddown, 0 hidden)
BGP: 3073 routes, 3073 active

Start with something simple like ASN and RouterID

A basic TextFSM Template

I wanted to start slowly with something I knew I could get to work. Looking at the data, it should be simple to extract the first two values I need:
– ASN
– Router ID

I started with those values as they are by far the simpler to extract from the ‘show route summary’ command. I will try not to cover material that is covered by the two Google links above. However I do want to point out the concept of TextFSM (as I understand it or explain it to myself) which is to provide context for your regular expressions. That is, not only can you define the specific pattern to search for but you can also define its “environment”. As you can see below the “Value” keyword lets me define a variable I want to pluck out of the unstructured text (the show command output). LIne 4 defines the “action” section to start processing and the first thing to look for is a line that starts with “Autonomous system number:” one or more space noted by the \s+ and then our ASN variable which we defined above as being a pattern of one or more digits \d+. So you have the power of the regular expression that defines the value you want and the power of regular expressions to help you define the context where your value will be found.

Junos ‘show route summary’ TextFSM Template – Version 1

For this exercise we will use my textfsm3 GitHub repository and the “test_textfsm.py” script for our testing rather than the Python command interpreter. Simply clone the repository to get started.
Note that the repository has the completed version of the template. Look at the history of the template file on GitHub to see its “evolution”.

(txtfsm3) Claudias-iMac:textfsm3 claudia$ python test_textfsm.py -h
usage: test_textfsm.py [-h] [-v] template_file output_file
This script applys a textfsm template to a text file of unstructured data (often show commands). The resulting structured data is saved as text (output.txt) and CSV (output.csv).
positional arguments:
template_file TextFSM Template File
output_file Device data (show command) output
optional arguments:
-h, --help show this help message and exit
-v, --verbose Enable all of the extra print statements used to investigate the results

In the first iteration of the template file, we obtain the output below.

(txtfsm3) Claudias-iMac:textfsm3 claudia$ python test_textfsm.py junos_show_route_summary
.template junos_show_route_summary.txt

TextFSM Results Header:
['ASN', 'RTRID']
================================
['2495', '164.113.193.221']
================================

Extract more details

So we have successfully built a template that will extract ASN and RouterID from the Junos show route summary command. Now it will get interesting because we also want this next set of values.

  • Interface
  • Destinations
  • Routes
  • Active
  • Holddown
  • Hidden

The first challenge here was to pick up the totals line. Here, one of my favorite tools comes into play, RegEx101. Regular expressions don’t come easy to me and this site makes it so easy! I saved the working session for trying to match the first part of that long totals line. As you can see, you can’t just match “inet”, or “inet” plus a digit, you also have to account for the “small.” Using RegEx101 and trial and error I came up with the following regular expression.

Value INT (([a-z]+.)?[a-z]+(\d)?.\d+)

inet.0: 762484 destinations, 1079411 routes (762477 active, 0 holddown, 12 hidden)

inet6.0: 66912 destinations, 103194 routes (66897 active, 0 holddown, 30 hidden)
Direct: 3 routes, 3 active

small.inet6.0: 31162 destinations, 31162 routes (31162 active, 0 holddown, 0 hidden)
BGP: 31162 routes, 31162 active

Let’s break it down…

The diagram below breaks the regex down into the key sections and numbers them. At the bottom you can see the actual text we are trying to parse and the numbers above indicate which section of the regex picked up the text we were interested in.

Breaking down the regular expression to extract the interface identifier (inet.x) for your TextFSM Template

The regex for INT (inet.x) was by far the most complicated. See 3 and 4 above. The rest of the line is far simpler and you just need to make sure you have it exactly as it appears in the raw text. Note that the parenthesis, which are part of the raw text show command, must also be ‘escaped’ just like the period.

Here is the TextFSM Template so far:

 Value Filldown ASN (\d+)
Value Filldown RTRID (\S+)
Value INT (([a-z]+.)?[a-z]+(\d)?.\d+)
Value DEST (\d+)
Value Required ROUTES (\d+)
Value ACTIVE (\d+)
Value HOLDDOWN (\d+)
Value HIDDEN (\d+)
Start
^Autonomous system number:\s+${ASN}
^Router ID:\s+${RTRID}
^${INT}:\s+${DEST}\s+destinations,\s+${ROUTES}\s+routes\s+\(${ACTIVE}\s+active,\s+${HOLDDOWN}\s+holddown,\s+${HIDDEN}\s+hidden\) -> Record

…and the resulting structured data:

(txtfsm3) Claudias-iMac:textfsm3 claudia$ python test_textfsm.py junos_show_route_summary.template junos_show_route_summary.txt
TextFSM Results Header:
['ASN', 'RTRID', 'INT', 'DEST', 'ROUTES', 'ACTIVE', 'HOLDDOWN', 'HIDDEN']
['2495', '164.113.193.221', 'inet.0', '762484', '1079411', '762477', '0', '12']
['2495', '164.113.193.221', 'inet.2', '3073', '3073', '3073', '0', '0']
['2495', '164.113.193.221', 'small.inet.0', '116371', '116377', '116371', '0', '0']
['2495', '164.113.193.221', 'inet6.0', '66912', '103194', '66897', '0', '30']
['2495', '164.113.193.221', 'small.inet6.0', '31162', '31162', '31162', '0', '0']

A few things to highlight, I used the ‘Filldown’ keyword for ASN and RTRID so that each “record” would have that information. The ‘Filldown’ keyword will take a value that appears once and duplicate it in subsequent records. If nothing else, it IDs the router from which the entry came but it also serves to simplify some things you might want to do down the line as each “record” has all the data. I also used the ‘Required’ keyword for routes to get rid of the empty last row that is generated when you used ‘Filldown’.

Almost there! We just need to pick up the source routes under each totals line.

Value SOURCE (\w+)
Value SRC_ROUTES (\d+)
Value SRC_ACTIVE (\d+)

Here is what the final (for now anyway) template looks like:

 Value Filldown ASN (\d+)
Value Filldown RTRID (\S+)
Value Filldown INT (([a-z]+.)?[a-z]+(\d)?.\d+)
Value DEST (\d+)
Value ROUTES (\d+)
Value ACTIVE (\d+)
Value HOLDDOWN (\d+)
Value HIDDEN (\d+)
Value SOURCE (\w+)
Value SRC_ROUTES (\d+)
Value SRC_ACTIVE (\d+)

Start
^Autonomous system number:\s+${ASN}
^Router ID:\s+${RTRID}
^${INT}:\s+${DEST}\s+destinations,\s+${ROUTES}\s+routes\s+(${ACTIVE}\s+active,\s+${HOLDDOWN}\s+holddown,\s+${HIDDEN}\s+hidden) -> Record
^\s+${SOURCE}:\s+${SRC_ROUTES}\s+routes,\s+${SRC_ACTIVE}\s+active -> Record

A few highlights. Because I wanted to store the source routes in a different value (SRC_ROUTES) I had to remove required from Routes in order to pick up the rows. I now have an extra row at the end but I can live with that for now. I also added Filldown to INT so that its clear where the source information came from.

(txtfsm3) Claudias-iMac:textfsm3 claudia$ python test_textfsm.py junos_show_route_summary.template junos_show_route_summary.txt

TextFSM Results Header:
['ASN', 'RTRID', 'INT', 'DEST', 'ROUTES', 'ACTIVE', 'HOLDDOWN', 'HIDDEN', 'SOURCE', 'SRC_ROUTES', 'SRC_ACT
IVE']
['2495', '164.113.193.221', 'inet.0', '762484', '1079411', '762477', '0', '12', '', '', '']
['2495', '164.113.193.221', 'inet.0', '', '', '', '', '', 'Direct', '1', '1']
['2495', '164.113.193.221', 'inet.0', '', '', '', '', '', 'Local', '1', '1']
['2495', '164.113.193.221', 'inet.0', '', '', '', '', '', 'BGP', '1079404', '762470']
['2495', '164.113.193.221', 'inet.0', '', '', '', '', '', 'Static', '5', '5']
['2495', '164.113.193.221', 'inet.2', '3073', '3073', '3073', '0', '0', '', '', '']
['2495', '164.113.193.221', 'inet.2', '', '', '', '', '', 'BGP', '3073', '3073']
['2495', '164.113.193.221', 'small.inet.0', '116371', '116377', '116371', '0', '0', '', '', '']
['2495', '164.113.193.221', 'small.inet.0', '', '', '', '', '', 'BGP', '116377', '116371']
['2495', '164.113.193.221', 'inet6.0', '66912', '103194', '66897', '0', '30', '', '', '']
['2495', '164.113.193.221', 'inet6.0', '', '', '', '', '', 'Direct', '3', '3']
['2495', '164.113.193.221', 'inet6.0', '', '', '', '', '', 'Local', '2', '2']
['2495', '164.113.193.221', 'inet6.0', '', '', '', '', '', 'BGP', '103185', '66888']
['2495', '164.113.193.221', 'inet6.0', '', '', '', '', '', 'Static', '4', '4']
['2495', '164.113.193.221', 'small.inet6.0', '31162', '31162', '31162', '0', '0', '', '', '']
['2495', '164.113.193.221', 'small.inet6.0', '', '', '', '', '', 'BGP', '31162', '31162']
['2495', '164.113.193.221', 'small.inet6.0', '', '', '', '', '', '', '', '']

The test_textfsm.py file will save your output into a text file as well as into a CSV file.
I did try using ROUTES for both sections and making it Required again. This got rid of the extra empty row but really impacts readability. I would have to keep track of how I used ROUTES as I would have lost the SRC_ROUTES distinction. That is a far greater sin in my opinion than an empty row at the end which is clearly just an empty row.