clmc-service.md

<!--
// © University of Southampton IT Innovation Centre, 2018
//
// Copyright in this software belongs to University of Southampton
// IT Innovation Centre of Gamma House, Enterprise Road,
// Chilworth Science Park, Southampton, SO16 7NS, UK.
//
// This software may not be used, sold, licensed, transferred, copied
// or reproduced in whole or in part in any manner or form or in or
// on any media by any person other than in accordance with the terms
// of the Licence Agreement supplied with the software, or otherwise
// without the prior written consent of the copyright owners.
//
// This software is distributed WITHOUT ANY WARRANTY, without even the
// implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
// PURPOSE, except where stated in the Licence Agreement supplied with
// the software.
//
//      Created By :            Nikolay Stanchev
//      Created Date :          02-05-2018
//      Created for Project :   FLAME
-->

# **Flame CLMC Service Documentation**

#### **Authors**

|Authors|Organisation|                    
|:---:|:---:|  
|[Nikolay Stanchev](mailto:ns17@it-innovation.soton.ac.uk)|[University of Southampton, IT Innovation Centre](http://www.it-innovation.soton.ac.uk)|

#### Description

This document describes the CLMC service and its API endpoints. The CLMC service is implemented using the *Python* web framework called **Pyramid**.
It offers different API endpoints such as GraphAPI for calculating round trip time, CRUD API for service function endpoints 
configuration data and Alerts API for creating and subscribing to alerts in Kapacitor. All source code, tests and 
configuration files of the service can be found in the **src/service** folder.

#### Notes
* Interacting with *Chronograf* - use `http://<clmc-host>/chronograf`. You will be asked to enter connection details.
The only input that you need to edit is the *Connection String* - set it to `http://<clmc-host>:8086` and click the
**Add Source** button.

* Interacting with *Kapacitor* - the Kapacitor HTTP API documentation can be found here: https://docs.influxdata.com/kapacitor/v1.4/working/api/
Notice that all of the URL paths provided by Kapacitor are already namespaced using base path ***/kapacitor/v1***. Therefore, no other prefix is required
when interacting with the Kapacitor application running on the clmc container, e.g.  
`http://<clmc-host>/kapacitor/v1/tasks` 
as described in the Kapacitor API reference.

* Interacting with *InfluxDB* - the InfluxDB HTTP API documentation can be found here: https://docs.influxdata.com/influxdb/v1.5/tools/api/
In order to interact with the InfluxDB application running on the clmc container, prefix all URL paths in the documentation 
with **/influxdb**, e.g.  
`http://<clmc-host>/influxdb/query`

* Interacting with *neo4j* - use `http://<clmc-host>/neo4j/browser/`. This will open the neo4j browser, which lets you
interact with the graph using Cypher queries (if necessary).

* Interacting with *clmc-serivce* - the API endpoints listed in the following sections relate to direct interactions with the clmc-service 
application server (listening on port 9080). If interacting with the clmc container, all of the listed below URIs must be prefixed 
with **/clmc-service** so that the nginx reverse proxy server (listening on port 80) can forward to requests to the correct application, e.g.  
`http://<clmc-host>/clmc-service/alerts?sfc={service function chain id}&sfci={service function chain instance id}&policy={policy id}&trigger={trigger id}`.

## Alerts API Endpoints

* **GET** ***/alerts?sfc={service function chain id}&sfci={service function chain instance id}&policy={policy id}&trigger={trigger id}***

    This API method can be used to retrieve the generated alert task and alert topic identifiers during the processing of an alerts specification document.
    These identifiers can then be used to interact with the Kapacitor HTTP API for further configiuration or modification of alerts - https://docs.influxdata.com/kapacitor/v1.4/working/api/.

    * Request:
    
        Expects a URL query string with the request parameters - **sfc**, **sfci**, **policy** and **trigger**. The given parameters must match the values used in the alerts specification
        document. Otherwise, a wrong ID will be returned.
        
    * Request URL Examples:
    
        **/alerts?sfc=MSDemo&sfci=MSDemo-premium&policy=requests_diff&trigger=low_requests**
        
        **/alerts?sfc=SimpleMediaService&sfci=SimpleMediaService-1&policy=rtt_deviation&trigger=increase_in_rtt**
     
    * Response
    
        The response of this request is a JSON-formatted content, which contains the task and topic identifiers, along with the Kapacitor
        API endpoints to use for configuring the given task, topic and the respective handlers.
        
        Returns a 400 Bad Request if the URL query string parameters are invalid or otherwise incorrect.
        
    * Response Body Example:
    
        ```json
        {
            "task_identifier": "094f23d6e948c78e9fa215528973fb3aeefa5525898626c9ea049dc8e87a7388",
            "topic_identifier": "094f23d6e948c78e9fa215528973fb3aeefa5525898626c9ea049dc8e87a7388",
            "task_api_endpoint": "/kapacitor/v1/tasks/094f23d6e948c78e9fa215528973fb3aeefa5525898626c9ea049dc8e87a7388",
            "topic_api_endpoint": "/kapacitor/v1/alerts/topics/094f23d6e948c78e9fa215528973fb3aeefa5525898626c9ea049dc8e87a7388",
            "topic_handlers_api_endpoint": "/kapacitor/v1/alerts/topics/094f23d6e948c78e9fa215528973fb3aeefa5525898626c9ea049dc8e87a7388/handlers"
        }
        ```

* **POST** ***/alerts***

    This API method can be used to send an alert specification document, which is then used by the CLMC service to create
    alert tasks and subscribe alert handlers to those tasks in Kapacitor. For further information on the alert specification
    document, please check the [CLMC Alert Specification Documentation](AlertsSpecification.md).

    * Request:
        
        Expects two YAML-formatted files in the request - one referenced with ID ***alert-spec*** representing the TOSCA alert specification 
        document and one referenced with ID ***resource-spec*** representing the TOSCA resource specification document. 
        The alert specification document is then parsed with the openstack TOSCA parser (https://github.com/openstack/tosca-parser/tree/master/toscaparser)
        and validated against the CLMC alerts specification schema (again check [documentation](AlertsSpecification.md) for more info on this). The TOSCA resource
        specification document is used only for consistency verification between the two documents - ensuring that they refer
        to the same service function chain, as well as making sure that there is at least one
        trigger alert in the alerts specification that relates to a state change policy in the resources specification.
        
    * Example for sending a request with curl:
    
        `curl -F "alert-spec=@alert-specification.yaml" -F "resource-spec=@resource-specification.yaml" http://localhost:9080/alerts`
        
        where **alert-specification.yaml** is the path to the alerts specification file and **resource-specification.yaml** is the
        path to the resource specification file.
    
    * Response:
        
        The response of this request is a JSON-formatted content, which contains the SFC and SFC instance identifiers
        from the alert specification along with any errors encountered when interacting with Kapacitor.
        
        Returns a 400 Bad Request if the request does not contain a yaml file referenced with ID **resource-spec**.
        
        Returns a 400 Bad Request if the resource specification file is not a TOSCA-compliant valid YAML file.
        
        Returns a 400 Bad Request if the request does not contain a yaml file referenced with ID **alert-spec**.
        
        Returns a 400 Bad Request if the alert specification file is not a valid YAML file.
        
        Returns a 400 Bad Request if the alert specification file cannot be parsed with the TOSCA parser.
        
        Returns a 400 Bad Request if the alert specification file fails validation against the CLMC alerts specification schema.
        
        Returns a 400 Bad Request if there is inconsistencies between the alert specification and resource specification files -
        e.g. referring to different service function chain and service function chain instance identifier or if there is no
        alert in the alerts specification related to a given state change policy in the resources specification.
        
    * Response Body Example:
    
        ```json
        {
          "msg": "Alerts specification has been successfully validated and forwarded to Kapacitor", 
          "service_function_chain_id": "<sfc_id>",
          "service_function_chain_instance_id": "<sfc_instance_id>"
        }
        ```
        
        If the CLMC service encounters any errors when creating alerts and handlers in Kapacior, they will
        be reported in the response as two lists of error objects. The **triggers_specification_errors** list contains
        any errors encountered while trying to create the alert tasks; the **triggers_action_errors** list contains 
        any errors encountered while trying to subscribe the HTTP handlers to the created tasks.
     
        ```json
        {
          "msg": "Alerts specification has been successfully validated and forwarded to Kapacitor", 
          "service_function_chain_id": "<sfc_id>",
          "service_function_chain_instance_id": "<sfc_instance_id>",
          "triggers_action_errors": [
                  {
                      "trigger": "<trigger ID the error is related to>",
                      "handler": "<handler URL the error is related to>",
                      "policy": "<policy ID the error is related to>",
                      "error": "<error message returned from Kapacitor>"
                  }
          ],
          "triggers_specification_errors": [
                {
                      "trigger": "<trigger ID the error is related to>",
                      "policy": "<policy ID the error is related to>",
                      "error": "<error message returned from Kapacitor>"
                }
          ]
        }
        ```

## Graph API Endpoints

* **Assumptions**
    * For each service function, there is a field/fields from which the service function response time (service delay) can be derived.
    * For each service function, there is a field/fields from which an average estimation of the size of a **request** to this service function can be derived.
    * For each service function, there is a field/fields from which an average estimation of the size of a **response** from this service function can be derived.
    * All the aforementioned fields reside in a single measurement.

* **POST** ***/graph/temporal***

    This API method sends a request to the CLMC service to build a graph snapshot in the time range between the *from* and *to* timestamps.

   * Request:

        Expects a JSON-formatted request body which declares the service function chain and service function chain instance for which the graph is built.
        The request body must define the service functions that will be included in the graph along with the measurement name, response time field, request size field and
        response size field for each service function. The declared fields could be influx functions across multiple fields. 
        The time range is also declared in the body by the *from* and *to* parameters. 

   * Request Body Example:

        ```json
        {
          "from": 1528385420,
          "to": 1528385860,
          "service_function_chain": "MSDemo",
          "service_function_chain_instance": "MSDemo_1",
          "service_functions": {
            "nginx": {
              "response_time_field": "mean(response_time)",
              "request_size_field": "mean(request_size)",
              "response_size_field": "mean(response_size)",
              "measurement_name": "nginx"
            },
            "minio": {
              "response_time_field": "mean(sum)/mean(count)",
              "request_size_field": "mean(request_size)/mean(count)",
              "response_size_field": "mean(response_size)/mean(count)",
              "measurement_name": "minio_http_requests_duration_seconds"
            }
          }
        }
        ```

        These parameters are then filled in the following influx query template:

        ```
        SELECT {0} AS mean_response_time, {1} AS mean_request_size, {2} AS mean_response_size FROM "{3}"."{4}".{5} WHERE "flame_sfc"=\'{6}\' and "flame_sfci"=\'{7}\' and "flame_sfp"=\'{8}\' and time>={9} and time<{10} GROUP BY "flame_sfe", "flame_location", "flame_sf"
        ```

        E.g. for the minio service function (i.e. flame_sfp=minio), the following query will be used to retrieve the data from InfluxDB:

        ```
        SELECT mean(sum)/mean(count) AS mean_response_time, mean(request_size)/mean(count) AS mean_request_size, mean(response_size)/mean(count) AS mean_response_size FROM "MSDemo"."autogen".minio_http_requests_duration_seconds WHERE "flame_sfc"='MSDemo' and "flame_sfci"='MSDemo_1' and "flame_sfp"='minio' and time>=1528385420000000000 and time<1528385860000000000 GROUP BY "flame_sfe", "flame_location", "flame_sf"
        ```
        
        N.B. database name is assumed to be the SFC identifier
        N.B. timestamps are converted to nano seconds.

   * Response:

        The response of this request is a JSON content, which contains all request parameters used to build the graph, along with a request UUID. 
        This request ID can then be used to manage the temporal subgraph that was created in response to this request.

        Returns a 400 Bad Request error if the request body is invalid.

   * Response Body Example:

        ```json
        {
          "database": "MSDemo",
          "retention_policy": "autogen",
          "service_function_chain": "MSDemo",
          "service_function_chain_instance": "MSDemo_1",
          "service_functions": {
            "nginx": {
              "response_time_field": "mean(response_time)",
              "request_size_field": "mean(request_size)",
              "response_size_field": "mean(response_size)",
              "measurement_name": "nginx"
            },
            "minio": {
              "response_time_field": "mean(sum)/mean(count)",
              "request_size_field": "mean(request_size)/mean(count)",
              "response_size_field": "mean(response_size)/mean(count)",
              "measurement_name": "minio_http_requests_duration_seconds"
            }
          },
          "graph": {
             "uuid": "75df6f8d-3829-4fd8-a3e6-b3e917010141",
             "time_range": {
               "from": 1528385420,
               "to": 1528385860
             }
          }
        }
        ```

* **DELETE** ***/graph/temporal/{graph_id}***

    This API method sends a request to delete the temporal graph associated with a given request UUID (retrieved from the response of a build-graph request).
    The request UUID must be given in the request URL, e.g. request sent to */graph/temporal/75df6f8d-3829-4fd8-a3e6-b3e917010141*

    * Response:

        The response of this request is a JSON content, which contains the request UUID and the number of deleted nodes.

        Returns a 404 Not Found error if the request UUID is not associated with any nodes in the graph.

   * Response Body Example:

        ```json
        {
           "uuid": "75df6f8d-3829-4fd8-a3e6-b3e917010141",
           "deleted": 5
        }
        ```

* **GET** ***/graph/temporal/{graph_id}/round-trip-time?startpoint={startpoint_id}&endpoint={endpoint_id}***

    This API method sends a request to run the Cypher Round-Trip-Time query over a temporal graph associated with a request UUID (retrieved from the response of a build-graph request).
    The request UUID must be given in the request URL, e.g. request sent to */graph/temporal/75df6f8d-3829-4fd8-a3e6-b3e917010141/round-trip-time?startpoint=DC2&endpoint=minio_1_ep1*,
    which will return the RTT breakdown for the path starting at Cluster DC2 to SF endpoint minio_1_ep1. The startpoint can be a Cluster, Switch, or UserEquipment node.

    * Response:

        The response to this request is a JSON content, which contains the result from the Cypher query including forward latencies, reverse latencies and service function response time along with the
        calculated round trip time and tag values for a potential measurement.

        Returns a 400 Bad Request error if the URL parameters are invalid

        Returns a 404 Not Found error if the request UUID and the endpoint ID are not associated with an endpoint node in the graph.

        Returns a 404 Not Found error if the startpoint ID is not associated with a Cluster or Switch node in the graph.

   * Response Body Example:

        ```json
        {
            "request_size": 2048,
            "response_size": 104857,
            "bandwidth": 104857600,
            "forward_latencies": [
               22, 11
            ],
            "total_forward_latency": 33,
            "reverse_latencies": [
               11, 22
            ],
            "total_reverse_latency": 33,
            "response_time": 15.75,
            "round_trip_time": 81.75,
            "global_tags": {
                "flame_sfe": "minio_1_ep1",
                "flame_sfc": "MSDemo",
                "flame_sfci": "MSDemo_1",
                "flame_sfp": "minio",
                "flame_sf": "minio_1",
                "flame_location": "DC1",
                "flame_server": "DC1"
            },
            "local_tags": {
               "traffic_source": "DC2"
            }
        }
        ```

        Here, the *forward_latencies* and *reverse_latencies* lists represent the reported latency at each hop between switches/clusters/ues. For example, if the path was DC2-DC3-DC4 and the SF endpoint was hosted
        on DC4, the response data shows that latency(DC2-DC3) = 22, latency(DC3-DC4) = 11, latency(DC4-DC3) = 11, latency(DC3-DC2) = 22, response_time(minio_1_ep1) = 15.75

        N.B. if the endpoint is hosted on the cluster identified in the URL parameter, then there will be no network hops between clusters/switches, so the latency lists would be empty, example:

        ```json
        {
            "request_size": 2048,
            "response_size": 104857,
            "bandwidth": 104857600,
            "forward_latencies": [],
            "total_forward_latency": 0,
            "reverse_latencies": [],
            "total_reverse_latency": 0,
            "response_time": 3,
            "round_trip_time": 3,
            "global_tags": {
                "flame_sfe": "minio_1_ep1",
                "flame_sfc": "MSDemo",
                "flame_sfci": "MSDemo_1",
                "flame_sfp": "minio",
                "flame_sf": "minio_1",
                "flame_location": "DC1",
                "flame_server": "DC1"
            },
            "local_tags": {
               "traffic_source": "DC2"
            }
        }
        ```

* **POST** ***/graph/network***

    This API method instructs CLMC to build the network topology in its graph database by querying the SDN controller and retrieving the switch-to-switch latency measurements -
    currently only Floodlight is supported as SDN controller. 

    * Response:

        The response to this request is a JSON content which shows how many Switch, Cluster and UserEquipment nodes were created in the graph.

        Returns a 503 Service Unavailable error if the SDN controller cannot be reached.

        Returns a 501 Not Implemented error if the SDN controller is reachable but cannot respond with a valid JSON content on the API endpoint for querying the network topology.

   * Response Body Example:

        ```json
        {
           "new_switches_count": 12,
           "new_clusters_count": 5,
           "new_ues_count": 3
        }
        ```

* **DELETE** ***/graph/network***

    This API method instructs CLMC to delete the network topology from its graph database.

    * Response:

        The response to this request is a JSON content which shows how many Switch, Cluster and UserEquipment nodes were deleted from the graph.

   * Response Body Example:

        ```json
        {
           "deleted_switches_count": 12,
           "deleted_clusters_count": 5,
           "deleted_ues_count": 3
        }
        ```

## CRUD API for service function endpoint configurations

**Note: this API is experimental and is not intended to be used at this stage**

* **GET** ***/whoami/endpoints***

    This API method retrieves all service function endpoint configurations in a JSON format.

    * Response:

        Returns a JSON-formatted response - a list of JSON objects, each object representing a service function endpoint configuration.

    * Response Body Example:

        - No service function endpoint configurations found.
        ```json
        []
        ```

        - Multiple service function endpoint configurations found.
        ```json
        [
          {
           "location": "location_1",
           "server": "location_1",
           "sfc": "sfc_1",
           "sfc_instance": "sfc_i_1",
           "sf_package": "sf_1",
           "sf": "sf_i_1",
           "sf_endpoint": "sf_endpoint_1"
           },
          {
           "location": "location_2",
           "server": "location_2",
           "sfc": "sfc_2",
           "sfc_instance": "sfc_i_2",
           "sf_package": "sf_2",
           "sf": "sf_i_2",
           "sf_endpoint": "sf_endpoint_2"
           }
        ]
        ```

* **GET** ***/whoami/endpoints/instance?sf_endpoint={sf_endpoint_id}***

    This API method retrieves the uniquely defined service function endpoint configuration associated with the given URL parameter - sf_endpoint.

    * Response:

        Returns a JSON-formatted response - a JSON object representing the service function endpoint configuration if it exists.

        Returns a 404 Not Found error if there is no service function endpoint configuration associated with the given URL parameter.

        Returns a 400 Bad Request error if the url parameter is invalid or missing.

    * Response Body Example:

        - Request made to /whoami/endpoints/instance?sf_endpoint=sf_endpoint_1
        ```json
          {
           "location": "location_1",
           "server": "location_1",
           "sfc": "sfc_1",
           "sfc_instance": "sfc_i_1",
           "sf_package": "sf_1",
           "sf": "sf_i_1",
           "sf_endpoint": "sf_endpoint_1"
          }
        ```

* **POST** ***/whoami/endpoints***

    This API method creates a new service function endpoint configuration.

    * Request:

        Expects a JSON-formatted request body with the new service function endpoint configuration.

    * Request Body Example:

        ```json
          {
           "location": "location_1",
           "server": "location_1",
           "sfc": "sfc_1",
           "sfc_instance": "sfc_i_1",
           "sf_package": "sf_1",
           "sf": "sf_i_1",
           "sf_endpoint": "sf_endpoint_1"
          }
        ```

    * Response

        Returns a JSON-formatted response - a JSON object representing the service function endpoint configuration that was created.

        Returns a 400 Bad Request error if the request body is invalid.

        Returns a 409 Conflict error if there exists another service function endpoint configuration with the same 'sf_endpoint' ID.

    * Response Body Example:

        ```json
          {
           "location": "location_1",
           "server": "location_1",
           "sfc": "sfc_1",
           "sfc_instance": "sfc_i_1",
           "sf_package": "sf_1",
           "sf": "sf_i_1",
           "sf_endpoint": "sf_endpoint_1"
          }
        ```

* **PUT** ***/whoami/endpoints/instance?sf_endpoint={sf_endpoint_id}***

    This API method replaces the uniquely defined service function endpoint configuration associated with the given URL parameter - sf_endpoint, with a new service
    function endpoint configuration given in the request body (JSON format). It can also be used for updating.

    * Request:

        Expects a JSON-formatted request body with the new service function endpoint configuration.

    * Request Body Example:

        ```json
          {
           "location": "location_2",
           "server": "location_2",
           "sfc": "sfc_1",
           "sfc_instance": "sfc_i_1",
           "sf_package": "sf_1",
           "sf": "sf_i_1",
           "sf_endpoint": "sf_endpoint_1"
          }
        ```

    * Response

        Returns a JSON-formatted response - a JSON object representing the new service function endpoint configuration that was created (updated).

        Returns a 400 Bad Request error if the request body is invalid.

        Returns a 400 Bad Request error if the url parameter is invalid.

        Returns an 404 Not Found error if there is no service function endpoint configuration associated with the given URL parameter.

        Returns a 409 Conflict error if there exists another service function endpoint configuration with the same 'sf_endpoint' ID as the ones in the request body.

    * Response Body Example:

        - Request made to /whoami/endpoints/instance?sf_endpoint=sf_endpoint_1

        ```json
          {
           "location": "location_2",
           "server": "location_2",
           "sfc": "sfc_1",
           "sfc_instance": "sfc_i_1",
           "sf_package": "sf_1",
           "sf": "sf_i_1",
           "sf_endpoint": "sf_endpoint_1"
          }
        ```

* **DELETE** ***/whoami/endpoints/instance?sf_endpoint={sf_endpoint_id}***

    This API method deletes the uniquely defined service function endpoint configuration associated with the given URL parameter - sf_endpoint.

    * Response:

        Returns the JSON representation of the deleted object.

        Returns an 404 Not Found error if there is no service function endpoint configuration associated with the given URL parameter.

        Returns a 400 Bad Request error if the url parameter is invalid.

    * Response Body Example:

        - Request made to /whoami/endpoints/instance?sf_endpoint=sf_endpoint_1

        ```json
          {
           "location": "location_1",
           "server": "location_1",
           "sfc": "sfc_1",
           "sfc_instance": "sfc_i_1",
           "sf_package": "sf_1",
           "sf": "sf_i_1",
           "sf_endpoint": "sf_endpoint_1"
          }
        ```