Introduction
- Is your interrupt-driven day no longer supportable?
- Is there a particular Project Manager that asks you the same question every morning?
- Do you often have to take some technical data and simplify it for semi- or non-technical consumption?
- Would you like to pull out relevant sections of technical data for a sibling team? Today you send them all the data and hope they can pull out what they need, or find what they need because you just don't have the time to put together something that is customized for them containing only what they need?
I suspect many of us have these situations far too often.
I spend quite a bit of time on Webex Teams (referred to as Spark from here on out in protest of the terrible re-branding of Spark to WebEx Teams which not only completely lacks imagination but also confuses anyone who also has to deal with Microsoft Teams…. a rebranding so awful it might actually unseat the rebranding of brigade). It is currently the messaging application of choice for my two biggest projects and a principal culprit in my interrupt-driven day. Maybe this can help turn it into an advantage... a mini-you if you will.
Not long ago, I lost a young and incredibly capable engineer to another project and that loss was pretty impactful. Many of the day to day activities fell back to me and I found lots of little (and not so little) things falling through the cracks. How was I going to work smarter not harder?
The Back Story
I got to attend Interop this year and was very inspired by every session I attended but a few in particular really got me thinking about how I might automate my way out of my current predicament.
Jeremy Schulman- - Self-service Network Automation Using Slack
Nick Russo - Automation for Bureaucracies
Hank Preston - A Practical Look at NetDevOps - While Hank's session inspired other ideas it is his work with DevNet and a SparkBot module that is most relevant for this exercise.
The Resulting Scene
The effort documented here is a real world experience and solution. To protect the innocent I've created a "demo" version of this solution that will highlight the functionality and the possibilities. Over the course of several posts, I'll document the requirements, use cases, solutions, and issues.
I like to start with the problem statement:
- I need to be able to provide fine grained project status on demand
- I need to be able to print a pretty Summary report for management on demand
- I need to be able to extract the relevant portion of some technical information for a parallel project on demand
- I'd like to incorporate something fun to get the team comfortable and familiar with the Bot (on demand of course)
I also like to document the initial state and the characteristics of the environment:
- Spark was the required platform for messaging. Enough of the team utilized Spark so as to make it an extremely viable delivery platform.
- A SAAS platform was used for project status.
- All the collateral and technical information resided in a document repository that everyone constantly works on and syncs to.
The solution concept: Develop a SparkBot attendant available to the Spark Team under which all of our spaces are created which can access an external API, a document repository, and respond to simple commands starting with:
- site_status
- site_summary
- site_networks
- comic_relief
OK..I started Googling and found lots of material but there were two issues:
- Almost everything leveraged a dummy test environment often using ngrok, a fabulous little tool, but I needed this to be at least in the neighborhood of production ready. I did not want it to run on my home system.
- Assembly was required in that there was good material on specific steps but not much on putting it all together. I suspect the thought there was that some of that should have been obvious but it wasn't to me.
When I knew enough to be dangerous I figured I needed the following components:
| Component | Description | Platform Used |
|---|---|---|
| Messaging Client & Account | The most basic requirement was a messaging client that was in use by enough of the team so as to be a viable delivery platform. This one was easy. Spark was already the messaging application of choice for the project. Using the messaging client solved all kinds of issues. I already had a client for a wide range of platforms, web, desktop, and mobile and I had a known User Interface to which the team was already accustomed. | Cisco WebEx Teams (Spark!!) Web Client, Desktop App, Mobile App |
| Messaging Platform | Of course the back end messaging platform had to support the required functionality for a Bot type application and research showed that it did. Spark has all the hooks for provisioning a Webhook and a Bot application. | Cisco WebEx Teams (Spark!!) |
| Document Repository | The Document repository the Team used had to have a client that I could install on the Web Server so that the data store would be available to bot functions and would always be synchronized (the latest). | Google Drive |
| Project Status API | The Project Status tool had to support an API and I knew that the project SAAS did. (More on this later) | SAAS Project Management Tool |
| Web Server (Front End) | Here is where it started getting tricky. I am by no means a System Administrator but I had to find a suitable web server and web server technology to use. The Web Server had to support the back end environment for the functionality and as I mentioned, also had to have a client so that I could present and sync the document repository. | Nginx |
| Web Server Backend | For the back end technology I always gravitate to Django but it was not a good fit in this case. 1. I did not foresee a need for a database 2. Most of the examples I found utilized Flask and I needed to get this working quickly before the village came after me with pitchforks and torches. This was my first use of Flask and it is a testament to what you always here about it. It is simple and lightweight with quite a lot of functionality. More than enough for my purposes. With the excellent examples out there I had no issues getting it to work. | Flask & Python |
| Hosting Platform | This was another challenge. I mentioned I'm not a SysAdmin and initially I felt that bringing up a Linux server that I could "harden" sufficiently was probably not something I was going to do well so I started with a MacInCloud instance. It was a good Proof of Concept but I wasn't happy with the performance or the cost so I abandoned it as the platform for the Bot Server. There was nothing for it. I needed to go down the path of a Linux host. I think I knew that already but I've been wanting to check out MacInCloud for a while and I thought that route would also save me time in researching how to secure a Linux server. I always gravitate to Digital Ocean for this. They have superb documentation and How-Tos, simple and intuitive interface, excellent price points, and I'm a diver. The first iteration of the Bot was actually on Digital Ocean and working quite well but I ran into a showstopper. The document sync client for Linux did not give you the option of changing the mount point. I could not grow the home volume on my Droplet sufficiently (the project data store is at about ~500GB) to host the data. I could add a volume but the sync client would try to sync to the home directory. I found a hack but it talked about sync instability and if I could not rely on the data being in sync on the Web Server then the rest was going to be pointless. Much of the functionality I needed involved accessing the latest data and documents on the repository. Well, the other hosting platform I've been wanting to try is the Google Compute Platform. I must say I'm very impressed. Everything is pretty intuitive. I already had the domain I wanted to use on Google so that made DNS a breeze! The production Bot is currently on GCP. The public Demo system that is part of this post is on Digital Ocean. The Demo does not have the constraints of the production system and without the Digital Ocean documentation its doubtful I would have a working solution. | Google Compute Platform (GCP) Note: The Demo is built on a Digital Ocean Droplet |
| Messaging Platform SDK or Module | I knew Spark had an SDK so worst case I had tools for the Bot. Further research resulted in quite a few options for the Bot. Here is where Hank Preston really saved me a ton of time. Hank's webexteamsbot module provides a very nice framework for your own bot functions. It comes with a few that you will want to keep (/help is one) and provides some very good examples. As your bot gets more capable (i.e. understands new functions or commands) you simply add those functions to your main script. The biggest issue I had was understanding what the module abstracted out. I needed to tap into a lot of the information...for example, the Spark space or room name had information that I needed to parse out so that if you ran a command within a space it would customize the output. | hpreston/webexteamsbot |
| Scripts | Each "function" or Bot command would need its own script or set of scripts and functions. | Python |
| HTTPS | I wanted to include as much security as my feeble SysAdmin abilities could muster. At a minimum SSL and some basic firewall functions. | HTTPS certbot |
| Domain | I wanted to use a FQDN. | cdl-automation.net |
This is the first of a 5 Part Series on creating a production-ish ready WebEx Teams Chat Bot.
- Introduction
- The Basic Web Server
- The WebEx Teams (Spark) Backend Webhook
- The Bot Application on the Web Server
- The Bot