## FLAME Configuration and Monitoring Specification This document describe the low-level configuration and monitoring specification for cross-layer management and control within the FLAME platform. ### Principles #### Configuration Data Briefly describe * the characteristics of configuration data as ways to describe the structure of the system over time. * how configuration data provides context for measurements of system behaviour * the lifecycle of configuration data within the platform and how it is used. * the type of configuration data Configuration includes the following aspects * Topology (nodes and links) * Capacity (servers and networks) * Allocation (Media Service Instance, Service Function Instance, Surrogate Instance) #### Monitoring Data Briefly descirbe: * the characteristics of monitoring data as ways to measure the behaviour of the system overtime including usage and performance * how measurements relate to resources within the configured system * the lifecycle of monitoring data within the platform and how it is used * the type of monitoring data Monitoring includes the following aspects * network resource usage * host resource usage * service usage #### Measurements Principles and Model ##### General The measurement model is based on a time-series model using the TICK stack from influxdata. The data model is based on the line protocol which has the format `<measurement>[,<tag-key>=<tag-value>...] <field-key>=<field-value>[,<field2-key>=<field2-value>...] [unix-nano-timestamp]` Each series has * a name "measurement" * 0 or more tags for configuration context * 1 or more fields for the measurement values * a timestamp. The model will be used to report configuration and monitoring data. In general, tags are used to provide configuration context for measurement values stored in fields. The tags are structured to provide queries by dimensions defined in the FLAME architecture. Tags are automatically indexed by InfluxDB. Global tags are automatically inserted by contexualised agents collecting data from monitoring processes. The global tags used across different measurements are a key part of the database design. Although, InfluxDB is schemaless database allowing arbirtary measurement fields to be stored (e.g. allowing for a media component to have a set of specific metrics), using common global tags allows the aggregation of measurements across time with common context. Although similar to SQL influx is not a relational database and the primary key for all measuremetns is time. Further schema design recommendations can be found here: https://docs.influxdata.com/influxdb/v1.4/concepts/schema_and_data_layout/ ##### Temporal Measurements Monitoring data must have time-stamp values that are consistent and sycnrhonised across the platform. This means that all VMs hosting SFs should have a synchronised system clock, or at least (and more likely) a means by which an millisecond offset from the local time can be retrieved so that a 'platform-correct' time value can be calculated. Integrating temporal measurements ##### Spatial Measurements Location can be provided in two forms: labelled (tag) and numeric (longitude and latitude as digitial degrees). Note that the location label is likely to be a _global tag_. Tag location | location | loc_long | loc_lat | | --- | --- | --- | | DATACENTRE_1 | 0 | 0 | A SF media transcoder is placed in a lamp-post. It has no means to obtain GPS coordinates but has a _loc_label_ provided to it as a VM environment variable. It provides zeros in the longitude and latitude. In subsequent data analysis we can search for this SF by location label. GPS coordination location | loc_label | loc_long | loc_lat | | --- | --- | --- | | LAMP_1 | 50.842715 | -0.778276 | A SF that is a proxy to a user attached to a NAP running in street lamp post LAMP_1. Here we have knowledge both of the logical location of the service and also the fine-grained, dynamic position of the service user. Lots of interesting possibilities with both of these bits of information! Note that tags are always strings and cannot be floats, therefore log and lat will always be stored as a measurement field. Integrating and analysing location measurements If tags are used then measurements of GPS coordinates will need to be translated into tag based approximation. For example, if a user device is tracking location information then for that to be combined with a server location the GPS coordinate needs to be translated. Matching on tags is limited to matching and potentially spatial hierarchies (e.g. country.city.street). Using a coordiante system allows for mathatical functions to be developed (e.g. proximity functions) ##### Measurement Context Monitoring data is collected to support design, management and control decisions. The link between decisions and data is through queries applied to contextual information stored with measurement values.  Every measurement has a measurement context. The context allows for time-based series to be created according to a set of query criteria which are then be processed to calculate statistical data over the desired time-period for the series. For example, in the following query the measurement is avg_response_time, the context is “service A” and the series are all of the data points from now minus 10 minutes. `find avg response time for service A between over the last 10 minutes` To support this query the following measurement would be created: `service_response,service_id=(string) response_time=(float) timestamp` Designing the context for measurements is an important step in the schema design. This is especially important when measurements from multiple monitoring sources need to be integrated and processed to provided data for queries and decision. The key design principles adopted include: * identify common context across different measurements * use the same identifiers and naming conventions for context across different measurements * organise the context into hierarchies that are automatically added to measurements during the collection process  The following figure shows the general structure approach for two measurements A and B. Data points in each series have a set of tags that shares a common context and have a specific context related to the measurement values. Now let’s look at the FLAME platform context for measurements within the FLAME platform. In the diagram below core of the model is the VM/Container instance as the primary measurement point as this is the realisation of computational processes running on the platform. A VM/Container is aone or more process running on a physical or virtual host with ports connecting to other VM/Container instances over network links. The VM/Container has measurement processes running to capture different views on the VM/Container include the network, host, and service. The acquisition of these different views of the VM/Container together are a key element of the cross-layer information required for management and control. The measurements about a VM/Container are captured by different processes running on the VM or container but are brought together by comon context allowing the information to be integrated, correlated and analysed. We consider three views on the VM/Container instance including (in orange) * service: specific metrics associated within the SF (either media component or platform component) * network: data usage TX/RX, latency, jitter, etc. * host: cpu, storage, memory, storage I/O, etc  All of the measurements on a specific VM/Container instance share a common context (green) that includes * sfc – an orchestration template * sfc_instance – an instance of the orchestration template * sf_package – a SF type * sf_instance – an instance of the SF type * vm_instance – an authoritive copy of the SF instance * server – a physical or virtual server for hosting VM instances * location – the location of the server By including this context with service, network and host measurements it is possible to support a wide range of queries associated with SFC’s whether they are Media Services or the Platform components themselves. By adopting the same convention for identifiers it is possible to combine measurements across service, network and host to create new series that allows exploration of diffeent aspects of the VM instance. Give a worked example across service and network measurements * Decide on the KPI of interest and how it's calculated from a series of measurements * Decide on time window for the series and sample rate * Decide on interpolation approach for data points in the series Discuss specific tags  ### Architecture The monitoring model using an agent based approach. The general architecture is shown in the diagram below.  An agent is deployed on each of the container/VM implementing a SF. The agent is deployed by the orchestrator when the SF is provisioned. The agent is configured with a set of input plugins that collect measurements from three aspects of the SF including network, host and SF usage/perf. The agent is configured with a set of global tags that are inserted for all measurements made by the agent on the host. Telegraf agent-based monitoring * Telegraf AMQP: https://github.com/influxdata/telegraf/tree/release-1.5/plugins/inputs/amqp_consumer * Telegrapf http json: https://github.com/influxdata/telegraf/tree/release-1.5/plugins/inputs/httpjson * Telegraf http listener: https://github.com/influxdata/telegraf/tree/release-1.5/plugins/inputs/http_listener * Telegraf Bespoke Plugin: https://www.influxdata.com/blog/how-to-write-telegraf-plugin-beginners/ * Telegraf Existing Plugins for common services, relevant plugins include * Network Response https://github.com/influxdata/telegraf/tree/release-1.5/plugins/inputs/net_response: could be used to performance basic network monitoring * nstat https://github.com/influxdata/telegraf/tree/release-1.5/plugins/inputs/nstat : could be used to monitor the network * webhooks https://github.com/influxdata/telegraf/tree/release-1.5/plugins/inputs/webhooks: could be used to monitor end devices * prostat https://github.com/influxdata/telegraf/tree/release-1.5/plugins/inputs/procstat: could be used to monitor containers * SNMP https://github.com/influxdata/telegraf/tree/release-1.5/plugins/inputs/snmp: could be used to monitor flows * systat https://github.com/influxdata/telegraf/tree/release-1.5/plugins/inputs/sysstat: could be used to monitor hosts Agents: * deployed at monitoring points (e.g surrogates and other network elements) * insert contextual metadata as tags into measurements * How does this relate to the Mona agents Hierarchical monitoring and scalability considerations * AMQP can be used to buffer monitoring info * InfluxDB can be used to provide aggregation points when used with Telegraf input and output plugin * How does this relate to the pub/sub and mySQL aggregator in FLIPS? Using FLIPS monitoring FLIPS offers a hightly scalable pub/sub system. We'll most likely need to use this in place of RabbitMQ for the infrastructure monitoring. The monitoring specification is here: https://drive.google.com/file/d/0B0ig-Rw0sniLMDN2bmhkaGIydzA/view  **Trust in measurements** If the agent is deployed in a VM/container that a tenant has root access then a tenant could change the configuration to fake measuremnents associated with network and host in an attempt gain benefit. This is a security risk. Some ideas include * Deploy additional agents on hosts rather than agents to measure network and VM performance. Could be hard to differentiate between the different SFs deployed on a host * Generate a hash from the agent configuration file that's checked within the monitoring message. Probably too costly and not part of the telegraf protocol * Use unix permissions (e.g. surrogates are deployed within root access to them) A couple of comments * CPU_UTILISATION_M: will be replaced by other metrics provided directly by Telegraf plugins * END_TO_END_LATENCY_M (not clear what this measurement means) ## Measurements #### Infrastructure Slice Capacity Measurements Capacity measurements measure the size of the infrastructure slice available to the platform that can be allocated on demand to tenants. Common tags * slice_id – an idenfication id for the infrastructure slice **host_resource** The *host_resource* measurement measures the wholesale host resources available to the platform that can be allocated to media services. `host_resource,slice_id="",server_id="",location="" cpus=(integer),memory=(integer),storage=(integer) timestamp` **network_resource** network_resource measures the overall capacity of the network available to the platform for allocation to tenants. There are currently no metrics defined for this in the FLIPS monitoring specification, although we can envisage usage metrics such as bandwidth being part of this measurement. `network_resource,slice_id="",network_id="", bandwidth=(integer),X=(integer),Y=(integer),Z=(integer) timestamp` #### Platform Measurements The following fields need further analysis as they seem to relate to core ICN and buffering. These do not seem that relevant * BUFFER_SIZES_M * FILE_DESCRIPTORS_TYPE_M * MATCHES_NAMESPACE_M * PUBLISHERS_NAMESPACE_M * SUBSCRIBERS_NAMESPACE_M **topology_manager** PATH_CALCULATIONS_NAMESPACE_M **nap** nap measurements are the platforms view on IP endpoints such as user equipment and services. A NAP is therefore the boundary of the platform. NAP also measures aspects of co-incidental multicast performance Questions * What is the group id for CHANNEL_AQUISITION_TIME_M and how can this be related to FQDN of the content? * what is the predefined time interval for CMC_GROUP_SIZE_M? * How does NETWORK_LATENCY_FQDN_M relate to END_TO_END_LATENCY? * How are multicast groups identified? i.e. "a request for FQDN within a time period", what's the content granularity here? * HTTP_REQUESTS_FQDN_M says from an endpoint yet the measurement does not have a node id, it could be just the total number of requests for a FQDN, it which case it is very much like service request stats of a media service * RX _BYTES_IP_MULTICAST_M * TX_BYTES_IP_MULTICAST_M * RX_PACKETS_HTTP_M * TX_PACKETS_HTTP_M * TX_BYTES_HTTP_M * RX_PACKETS_IP_MULTICAST_M * TX_PACKETS_IP_MULTICAST_M * RX_BYTES_IP_M * TX_BYTES_IP_M `nap_node,<global_tags>,nodeId="", CHANNEL_AQUISITION_TIME_M, timestamp` * Can we ignore the specific nodes and look at aggregate measurements associated with a multicast group? * Here the assumption is that nodes are grouped around requests to access content idenified by hashes of FQDN. `nap_multicast,<global_tags>,groupId="",fqdn="" CHANNEL_AQUISITION_TIME_M, CMC_GROUP_SIZE_M, NETWORK_FQDN_LATENCY timestamp` * CHANNEL_AQUISITION_TIME_M: avg time for all nodes in this group over sample period * CMC_GROUP_SIZE_M: avg multicastgroup size over sample period * NETWORK_FQDN_LATENCY: avg network latency over sample period **orch_media_service** **orch_sfc_instance** **orch_sf_instance** **clmc** #### Media Service Measurements Common tags * sfc – an orchestration template * sfc_instance – an instance of the orchestration template * sf_package – a SF type * sf_instance – an instance of the SF type * vm_instance – an authoritive copy of the SF instance * server – a physical or virtual server for hosting VM instances * location – the location of the server ##### Network Measurements Network Measurements measure aspects of network performance in relation to VMs/containers. **node_network_perf** node_network_perf provides the network measurement view for network elements. Network elements can be in the role of gateway, forwarding node, network attachment point, rendezvous, service, topology manager or user equipment as defined by the FLIPS monitoring specification. The measurements are made by the Mona monitoring agent. Questions * Can a single value of jitter (e.g. avg jitter) be calculated from the set of measurements in PACKET_JITTER_CID_M message? What is the time period for the list of jitter measurements? * What does CID actually mean? `node_network_perf,<global_tags>,node_role="",node_name="" timestamp` * PACKET_JITTER_CID_M * RX_BYTES_CID_M * TX_BYTES_CID_M * RX_PACKETS_IP_M (ipversion) * TX_PACKETS_IP_M (ipversion) Specific Tags: * node_role * name * state **node_port_perf** The netnode_port series provides network measurements on host ports as defined by the FLIPS monitoring specification. `port_network_perf,<global_tags>,node_id="",port_id="",port_name="" timestamp` Fields * PACKET_DROP_RATE_M * PACKET_ERROR_RATE_M * RX_PACKETS_M * RX_BYTES_PORT_M * TX_BYTES_PORT_M * TX_PACKETS_PORT_M **link** The link series provides measurements about network links. Currently the FLIPS monitoring specification defines only topological configuration information and does not provide any measurements related to links. All performance information is included as part of the nodes. Further investigation is needed to understand if derived measurements related to links are needed or whether this is just useful for monitoring the temporal evolution of the topology. Fields * ?? * ?? Tags * link_name * link_id * source_node_id * destination_node_id * link_type ##### Host Measurements SF Host Resource Measurements measures the host resources allocated to a service function deployed by the platform. All measurements have the following global tags to allow the data to be sliced and diced according to dimensions. **node_host_resource** The resources allocated to a VM/Container `node_host_resource,<global-tags> cpu,memory,storage timestamp` **node_cpu_usage** [[inputs.cpu]] **node_disk_usage** [[inputs.disk]] **node_disk_IO** [[inputs.diskio]] **node_kernel_stats** [[inputs.kernel]] **node_memory_usage** [[inputs.mem]] **node_process_status** [[inputs.processes]] **node_swap_memory_usage** [[inputs.swap]] **node_system_load_uptime** [[inputs.system]] ##### Service Measurements Each SF developed will offer service specific usage and performance measurements. `service_request,<global_tags>,cont_nav="",cont_rep="",user_id="" <request-params> timestamp` `service_response,<global_tags>,cont_nav="",cont_rep="",user_id="" response_time timestamp` Specific Tags * cont_nav: the content requested * cont_rep: the content representation requested * user_id: the pseudonym of the user