clmc-development-guide.md

FLAME CLMC Development Guide

Authors

Authors	Organisation
Nikolay Stanchev	University of Southampton, IT Innovation Centre

Description

This document describes the internal implementations of the CLMC and provides guidelines on how to develop, extend and maintain the project.

Top-level view

CLMC consists of the following software components:

• InfluxDB - open-source external implementation - https://github.com/influxdata/influxdb - used for storing time-series data
• Chronograf - open-source external implementation - https://github.com/influxdata/chronograf - used for visualising time-series data
• Kapacitor - open-source external implementation - https://github.com/influxdata/kapacitor - used for managing alerts and trigger notifications
• Neo4j - open-source external implementation - https://github.com/neo4j/neo4j - used for performing graph-based monitoring
• Nginx - open-source external implementation - https://github.com/nginx/nginx - used as a reverse proxy in front of the other CLMC APIs and services
• clmc-service - internal implementation - provides API endpoints for different monitoring/alerting-related features

Managing and installing software components

• InfluxDB, Kapacitor, Chronograf - service version number and an installation script are located in scripts/clmc-service/install-tick-stack.sh
• Neo4j - service version number and an installation script are located in scripts/clmc-service/install-neo4j.sh
• clmc-service - service installation script located in scripts/clmc-service/install-clmc-service.sh
• nginx - service version number and an installation script located in scripts/clmc-service/install-nginx.sh

In general, the preferred way to install CLMC will be to use the scripts/clmc-service/install.sh script which calls the install scripts mentioned above in the correct order and checks if the correct environment variables have been provided. The required environment variables are:

• SDN_CONTROLLER_IP - the IP address for CLMC to use when collecting network data from an SDN controller.
• NETWORK_DEPENDENCY - the systemd dependency after which CLMC should start, e.g. network.target
• SFEMC_FQDN - the FQDN for CLMC to use when communicating with the FLAME SFEMC component

After running the install script, there must be 5 systemd services - kapacitor, influxdb, chronograf, neo4j and flameclmc. These could be managed with systemctl:

• systemctl status flameclmc
• systemctl status influxdb
• systemctl status kapacitor
• systemctl status chronograf
• systemctl status neo4j
• systemctl status nginx

Configuration of clmc-service

The clmc-service software component is managed through a systemd unit file which is generated by the installation script and located in /lib/systemd/system/flameclmc.service. Thus, the clmc-service can be started/restarted/stopped by running:

• systemctl start flameclmc
• systemctl restart flameclmc
• systemctl stop flameclmc

In addition, the installation script generates a start script (this is what the systemd service executes to start the CLMC service). This is located in /opt/flame/clmc/start.sh. Through this script, the service could be further reconfigured with the following environment variables:

• SFEMC_FQDN - defaults to what was used during installation.
• SDN_CONTROLLER_IP - defaults to what was used during installation.
• SFEMC_PORT - the port number the CLMC service uses when communicating with the FLAME SFEMC component, defaults to 8080.
• SDN_CONTROLLER_PORT - the port number the CLMC service uses when communicating with the SDN controller, defaults to 8080.

To reconfigure any of the environment variables above, add/edit an export line in the start script, for example:

export SFEMC_PORT=5000

The internal configuration of clmc-service (e.g. how to connect to Kapacitor) is managed through {production/development}.ini files. For example, the following configuration variables control how the connection with Kapacitor is established:

• kapacitor_host
• kapacitor_port

These files can be found at src/service/production.ini and src/service/development.ini. To change configuration at run-time, edit the src/service/production.ini and restart the clmc-service with:

systemctl restart flameclmc

CLMC services default configuration

Currently, everything is deployed on the same server, therefore, clmc-service uses localhost to connect to any of its prerequisites.

• InfluxDB - listens on its default port 8086
• Chronograf - listens on its default port 8888
• Kapacitor - listens on its default port 9092
• Neo4j (uses protocols HTTP and Bolt) - HTTP API listens on default port 7474, Bolt API listens on default port 7687
• clmc-service - listens on port 9080

To alleviate connections to CLMC, a reverse proxy (nginx) is installed during deployment with the following configuration:

• proxy listening on port 80
• URLs starting with /kapacitor are forwarded to Kapacitor without altering the request
• URLs starting with /influxdb are forwarded to InfluxDB by stripping the /influxdb part of the request URL
• URLs starting with /chronograf are forwarded to Chronograf without altering the request (Chronograf is configured to use base path /chronograf)
• URLs starting with /neo4j are forwarded to Neo4j by stripping the /neo4j part of the request URL
• URLs starting with /clmc-service are forwarded to the CLMC service by stripping the /clmc-service part of the request URL

Each service manages its own logging software:

InfluxDB - journalctl -fu influxdb

Chronograf - journalctl -fu chronograf

Kapacitor - tail -f /var/log/kapacitor/kapacitor.log

Neo4j - tail -f /var/log/neo4j/debug.log

Nginx

• tail -f /var/log/nginx/access.log - requests log file
• tail -f /var/log/nginx/error.log - errors log file

clmc-service

• tail -f /var/log/flame/clmc/service.log - general service logging while handling API requests
• tail -f /var/log/flame/clmc/service-exceptions.log - logging of unexpected errors while handling API requests
• journalctl -fu flameclmc - logging of all the above plus logging from the graph-based monitoring script

The logging configuration of the CLMC service can be found at src/service/production.ini and src/service/development.ini. Defaults to rotating file-based logging with 5 backup files and each file configured to store 40MB at most.

CLMC project structure

docs/ - any documentation related to CLMC can be found in this folder, some important files:

• docs/AlertsSpecification.md - the documentation for the TOSCA alerts configuration
• docs/clmc-service.md - the documentation of the CLMC service API endpoints
• docs/graph-monitoring-user-guide.md - the user guide for using graph-based monitoring of CLMC
• docs/total-service-request-delay.md - calculation model of the CLMC graph API for estimating round-trip time of a service
• docs/clmc-information-model.md - documentation on the CLMC information model

scripts/clmc-agent - scripts and configuration files for installing Telegraf on service functions.

scripts/clmc-service - installation scripts for CLMC including a reverse proxy configuration - nginx.conf, and the graph monitoring script - graph-pipeline.sh

scripts/test/fixture.sh - a script for setting up and tearing down the CLMC test environment

src/service - the source code implementation of the CLMC service including tests for the API features

src/test - integration tests which require the full CLMC test environment

CLMC service implementation details

The CLMC service is implemented in Python using the web framework called Pyramid. All source code, tests and configuration files of the service can be found in the src/service folder.

In general, the structure follows a typical python package structure, with the root package being called clmcservice and a setup.py provided on the same folder level. Python dependencies (external python packages) are declared in two places:

• setup.py file located at src/service/setup.py - defines the explicit Python dependencies, that is the external packages that are used within the CLMC service source code
• requirements.txt file located at src/service/requirements.txt - defines the full list of Python dependencies (both explicit and implicit) - these are all the packages that are used either within the CLMC source code or are used by external packages that the CLMC service depends on

To manage these dependencies, we use a Python virtual environment (through virtualenv and virtualenvwrapper). The virtual environment is created by the installation script but in case it needs to be manually created:

export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh
mkvirtualenv CLMC

If the virtual environment has been already created, run the following to get into it:

export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh
workon CLMC

Then to replicate the Python environment for the CLMC service, use the requirements.txt file which has all the external packages (explicit and implicit) with a fixed version number:

cd src/service
pip3 install -r requirements.txt

Finally, to install the CLMC service python package, run the following:

cd src/service
pip3 install .

Once the clmcservice package has been installed, the Pyramid application can be started with the waitress application server by using either of the production.ini or development.ini files.

pserve src/service/production.ini

All of the instructions above are executed by the clmc-service install script.

The source code is organised in various python subpackages. Each subpackage is the implementation of a given API (including unit and local integration tests) with the exception of the models package which is responsible for things like object relational mappings.

• src/service/clmcservice/alertsapi - the source code of the CLMC alerts API used for managing alerts and trigger notifications

  • src/service/clmcservice/alertsapi/alerts_specification_schema.py - defines the validation schema for alerts specification documents
  • src/service/clmcservice/alertsapi/views.py - defines the API functions of the alerts API, API endpoints are named with identifiers which are then mapped to URLs
  • src/service/clmcservice/alertsapi/utilities.py - utility functions used to fill Kapacitor templates (with data extracted from the alerts specification document)

• src/service/clmcservice/graphapi - the source code of the CLMC graph API used for calculating round-trip time and performing graph-based measurements

  • src/service/clmcservice/graphapi/views.py - defines the API functions of the graph API, API endpoints are named with identifiers which are then mapped to URLs
  • src/service/clmcservice/graphapi/utilities.py - utility functions used for interacting with the Neo4j graph

• src/service/clmcservice/managementapi - the source code of the CLMC data management API used for managing (e.g. deleting) the data of a service function chain

  • src/service/clmcservice/managementapi/views.py - defines the API functions of the graph API, API endpoints are named with identifiers which are then mapped to URLs

• src/service/clmcservice/models - package for any persistency related code (e.g. object relational mappings)

• src/service/clmcservice/static - static files that are required by the CLMC service, e.g. the TOSCA alerts definitions file

• src/service/clmcservice/tweens.py - tweens a.k.a. middlewares, code that is executed for each request before it is processed by its respective API function

• src/service/clmcservice/__init__.py - the entry point of the CLMC service, this is where tweens (middlewares) and API routes are registered

Additional files that are not part of the Python package implementation but are used as resources by the CLMC services:

• src/service/resources/GraphAPI - static mappings between the IP address of a service router to the name of an emulated UE or a cluster; these files are used as a temporary solution until we have a solid approach of retrieving this information from the SFR (Service Function Routing) component of FLAME

• src/service/resources/TICKscript - TICK script implementations for Kapacitor task templates that are used by CLMC to configure alerts and trigger notifications

• src/service/resources/tosca - TOSCA test data, including valid/invalid alerts configuration and resource specification files

Releasing a new version of CLMC service

Ultimately the process of releasing a new CLMC version is achieved by creating a merge request to the integration branch of the flame-clmc project. The description of the merge request should act as a CHANGELOG for the new version and the title should specify the new version number.

In addition, there are a few files which must be updated with the new CLMC version number before submitting the merge request:

• src/service/VERSION (line 1) - ensures the clmcservice python package is installed with the correct CLMC version number

• src/test/VERSION (line 1) - ensures the clmctest python package is installed with the correct CLMC version number

• gitlab-ci.yml (line 38, 39, 53 and 54) - ensures the gitlab CI runs with the correct version numbers

Updating the list of CLMC dependencies

• Service prerequisites - if a new service is used in the CLMC implementation, the IPRREGISTRY.md must be updated by adding the service and its LICENSE (license type and link) to the list of Unmodified Service Prerequisites

• Python dependencies - if a new python package is used in the CLMC service implementation, the src/service/setup.py, src/service/requirements.txt and IPRREGISTRY.md files must be updated

  • setup.py - add the package name and package version in the list of required packages to install

  • IPREGISTRY.md - add the package name, license type and license link to the list of Unmodified Libraries

  • requirements.txt - generate a new requirements.txt file which must include the new package as well as its package dependencies with fixed version numbers

How to generate a requirements.txt file for replicating the Python virtual environment

To generate the full list of packages in the Python virtual environment, simply run:

export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh
workon CLMC
pip3 freeze > requirements.txt

Then, by running the command below, pip will install all packages going line by line through the requirements.txt file.

pip3 install -r requirements.txt

However, the output list from the command above is not sorted in a way that packages which depend on other packages are placed (and later installed) after their dependencies. This could lead to incorrect package versions being installed (not every python package fixes the versions of its dependencies) which might later turn to be a dependency conflict between different packages. To avoid this issue, we fix the versions of every package in the Python virtual environment (explicit and implicit) and generate an ordered list so that a package is only installed once all its dependencies are installed. To achieve this, we use two utilities:

• pipdeptree https://pypi.org/project/pipdeptree/ - used to generate a tree-like list showing package dependencies (explicit and implicit)

• custom utility script located at scripts/clmc-service/dependencies_util.py - used to sort the output of pipdeptree so that packages are only listed after all of their dependencies have been listed

The procedure to generate a properly sorted requirements.txt file is the following:

# go into the CLMC container
lxc exec clmc-service -- bash
# switch to the CLMC Python virtual environment
export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh
workon CLMC
# install pipdeptree if not installed already
pip3 install pipdeptree
# check output of pipdeptree (--reverse flag is used, so that the tree shows leaf packages first and the packages that require them underneath)
pipdeptree --reverse
# run the utility script with the output of pipdeptree as input
cd /opt/clmc/scripts/clmc-service
python3 dependencies_util.py "$(pipdeptree --reverse)"

CLMC test environment

The CLMC test environment is built using:

• vagrant and virtualbox - to get a Unix-based environment on a Windows OS
• lxd - to manage containerized deployments of a number of simple service functions as well as a CLMC deployment

With the provided Vagrantfile in the flame-clmc project, simply run the following to start and get access to the virtual machine:

vagrant up
vagrant ssh

Note: Virtualbox and vagrant must be installed. In addition, the vagrant disk size plugin is also required.

vagrant plugin install vagrant-disksize

Vagrant is configured to synchronise the VM's /vagrant folder with the content of the flame-clmc project folder.

The LXC containers are controlled with the fixture script located at scripts/test/fixture.sh.

Usage: fixture.sh create|start|stop|destroy [-f config_file] [-r repo_root] [-c container_name|all]

The test environment is defined in the configuration file located at src/test/clmctest/rspec.json

An example on how to create the clmc-service container:

/vagrant/scripts/test/fixture.sh create -f /vagrant/src/test/clmctest/rspec.json -c clmc-service

The installation of several of the services depend on accessing the Nexus binary repository (for the custom Telegraf agent). To do this, a username and password for the repository must be specified in a reporc file in the user's home directory, e.g.

REPO_USER=itinnov.flame.integration
REPO_PASS=xxxxxxx

An example on how to create the full test environment with all containers:

sudo su
/vagrant/scripts/test/fixture.sh create -f /vagrant/src/test/clmctest/rspec.json -c all

The fixture script will fail if any individual service installation fails to install (or fails its tests).

To get access to any of the containers run:

lxc exec <container name> -- bash

For example:

lxc exec clmc-service -- bash

lxc exec test-runner -- bash

All tests are implemented using Python and Pytest. The integration tests (located at src/test) are installed in the test-runner container. To execute these, run:

lxc exec test-runner -- pytest -s --tb=short -rfp --pyargs /opt/clmc/src/test/clmctest

Implementation details of the Alerts API

The Alerts API is implemented on top of the Kapacitor HTTP API and includes the following main parts:

• Wrapper interfaces for the Kapacitor HTTP API
• Parsing and validation of a TOSCA-compliant CLMC-specific alerts configuration document
• Kapacitor task templates

When creating/updating alerts, the following steps are performed (in the specified order):

1. TOSCA alerts configuration document is parsed as a yaml file
2. yaml content is parsed as a TOSCA document
3. the TOSCA document is validated against the CLMC alerts specification schema
4. if a TOSCA resource specification document was passed too, parse it as a TOSCA document and compare both documents for inconsistencies
5. go through each policy and trigger in the alerts configuration document and extract the data
6. convert the extracted data from each trigger to a Kapacitor task based on the chosen event type (determines which Kapacitor task template to use)
7. the Kapacitor task identifier is generated based on the sfc, sfc instance, policy and trigger identifiers and represents a hash of the concatenation of these identifiers
8. the Kapacitor topic is named after the Kapacitor task (they share the same identifier)
9. register all URLs as Kapacitor HTTP handlers for the created Kapacitor topic
10. the Kapacitor handler identifier is generated as a hash of the concatenated sfc, sfc instance, policy and trigger identifiers with the handler URL
11. fill in the Kapacitor templates and forward to the Kapacitor HTTP API
12. return a response which contains the errors (if any) that were encountered while interacting with Kapacitor (e.g. if an alert already exist)

Something that is not mentioned in the steps above is the generation of the SFEMC handler URL, that is every time the alerts configuration document references flame_sfemc as an alert handler. SFEMC expects a POST request for a given alert to a URL in the following format:

http://<sfemc fqdn>:<sfemc port number>/<sfc identifier>/<policy identifier>/<trigger identifier>

However, policy and trigger identifiers are the identifiers included in the TOSCA resource specification document since this is the only deployment information that the SFEMC knows (SFEMC doesn't know anything about the CLMC alert management). This is why we have the restriction that policy identifiers that reference flame_sfemc from the alerts configuration document must match with StateChange policy identifiers from the resource specification document. Thus, we ensure that the policy identifier in the generated URL is what SFEMC expects. For the trigger identifiers, we ensure that a trigger name that references flame_sfemc in the alerts configuration must match with an identifier from a clmc::<trigger name> constraint from the resource specification document. Thus, the SFEMC URL is consistently generated with what SFEMC expects as a request URL. The SFC identifier is extracted from the metadata section of the alerts configuration.