Merge branch 'integration' into aggregation

.gitattributes

@@ -7,4 +7,3 @@ scripts/*  text eol=lf
 *.png binary
 *.jpg binary
 *.build-config/
-*.pytest_cache/
\ No newline at end of file

.gitignore

@@ -4,4 +4,6 @@
 *__init__.pyc
 *egg-info*
 *git-commit-ref*
+*_version.py*
 ubuntu-xenial-16.04-cloudimg-console.log
+**/.pytest_cache/
\ No newline at end of file

.gitlab-ci.yml

@@ -26,6 +26,8 @@ stages:
 
 build:tests:
   stage: build
+  only:
+    - schedules    
   script: 
     - python setup.py sdist --dist-dir=build
   artifacts:
@@ -35,6 +37,8 @@ build:tests:
 
 test:all:
   stage: test
+  only:
+    - schedules    
   dependencies: 
     - build:tests
   script: 
@@ -49,6 +53,8 @@ test:all:
   
 clean:
   stage: clean
+  only:
+    - schedules    
   script: 
-    - scripts/build/deleteallvms.sh   
+    - python scripts/build/deleteallvms.py   
   when: always       
\ No newline at end of file

MANIFEST.in

 include MANIFEST.in
-recursive-include clmctest *.yml *.sh *.json *.conf
+recursive-include clmctest _version.py *.yml *.sh *.json *.conf
\ No newline at end of file

README.md

@@ -22,9 +22,10 @@
 -->
 
 # FLAME Cross Layer Management and Control
-#### Version: X.X
+#### Version: 1.0.0
 
 ### About this document
+
 #### Authors
 
 |Authors|Organisation|                    
@@ -34,31 +35,21 @@
 
 ### Contents
 
-##### Information Model
-
-The informational model describes the structure and format of configuration and monitoring information collected by the CLMC and how the information is used to support service management decision amking 
-
-https://gitlab.it-innovation.soton.ac.uk/mjb/flame-clmc/blob/integration/docs/monitoring.md
-
-##### Adaptive Streaming Use Case Scenario
+#### Documentation
 
-The use case scenario provides an example usage of the information model for an mpeg-dash adaptive streaming service 
-
-https://gitlab.it-innovation.soton.ac.uk/mjb/flame-clmc/blob/integration/docs/adaptive-streaming-usecase-scenario.md
+Implementation documentation and discussion can be found in the [FLAME CLMC Information Model Specification](docs/clmc-information-model.md)
 
 #### Development Environment
 
-tbd
-
-#### Testing
+The development environment is currently Vagrant/Virtualbox providing the infrastructure for dev and testing.
 
-Testing is implemented using pytest. 
+Vagrant-disksize plugin must be installed. 
 
-The installation script is here:
+`vagrant plugin install vagrant-disksize`
 
-`sudo clmctest/services/pytest/install.sh`
+#### Testing
 
-using the following convention:
+Testing is implemented using pytest using the following convention:
 
 * Tests are written in python using pytest
 * Related tests are stored in a python module `clmctest/<testmodule>` to create a suite of tests. All tests are stored in files test_*.py, there can be many tests per file, and many files per module
@@ -69,11 +60,11 @@ using the following convention:
 
 #### Creating a deployment for a test
 
-To set up a simulation of the adaptive streaming use case scenario first install the vagrant-disksize plugin (if not already installed)
+Each test has a fixture defining a set of VMs required for the test. The fixture is described in an rspec.yml file that is stored within the clmctest/<testmoduel> directory. For example, the monitoring test fixture is stored in the following file.
 
-`vagrant plugin install vagrant-disksize`
+`clmctest\monitoring\rspec.yml`
 
-and then execute the following command
+The fixture is created by running the vagrant up command with the test module name as a parameter
 
 `vagrant --fixture=monitoring -- up`
 
@@ -85,38 +76,32 @@ The **clmc-service** vm includes influx, Kapacitor and Chronograf. The following
 * Chronograf: 8888
 * Kapacitor: 9092
 
-#### Running the streaming-sim test
+Typically, tests are run from a test-runner VM that has pytest installed. The script used to install pytest is 
 
-SSH into the CLMC server
+`clmctest/services/pytest/install.sh`
 
-`vagrant --fixture=monitoring -- ssh clmc-service`
+To run a test, SSH into the test runner machine
+
+`vagrant --fixture=monitoring -- ssh test-runner`
 
 Then go to the 'vagrant' directory.
 
 `cd /vagrant`
 
-The next step is to generate the test data, which could be done in two ways.
-
-First option is to run a python script to generate the test data sets
-
-`python3 clmctest/monitoring/StreamingSim.py`
-
-This script could also be used to clear the generated data by using the '-c' option
-
-`python3 clmctest/monitoring/StreamingSim.py -c`
+Run the pytest.
 
-#### Running the monitoring tests
+`pytest -s clmctest/monitoring/`
 
-The second option is to directly run the testing module, which will detect if the data was generated, and if not, will automatically generate the data before executing the tests. Keep in mind that if the test data is being generated using this way, a 10 seconds timeout is given after the generation is finished so that the data could properly be inserted into the database. If the data was already generated using the first option, only the tests would be executed.
+#### Tests run from the host
 
-The command for running the testing module is:
+Tests can be run from the host by sshing commands to the test running. This is described in the .gitlab-ci.yml file. The file shows how a python package is created
 
-`pytest -s clmctest/monitoring/test_simresults.py`
+`python setup.py sdist --dist-dir=build`
 
-The `-s` option in the command is used to output prints used in the test code and is, therefore, optional.
+Then the package is installed 
 
-If pytest is not installed, an easy solution is to use the Python Package Index (PyPI)
+`vagrant --fixture=scripts -- ssh test-runner -- -tt "pip3 install /vagrant/build/clmctest-SNAPSHOT.tar.gz"`
 
-`sudo apt-get install python3-pip` 
+Then the tests are run
 
-`pip3 install pytest`
+`vagrant --fixture=scripts -- ssh test-runner -- -tt "pytest -s --pyargs clmctest.scripts"`
\ No newline at end of file

clmctest/e2e_response_time/rspec.yml

+## (c) University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          02-02-2018
+##      Created for Project :   FLAME
+
+hosts:
+  - name: clmc-service
+    cpus: 1
+    memory: 2048
+    disk: "10GB"
+    forward_ports:
+      - guest: 8086
+        host: 8086
+      - guest: 8888
+        host: 8888
+      - guest: 9092
+        host: 9092
+    ip_address: "172.40.231.51"
+  - name: minio
+    service_name: "minio"
+    cpus: 1
+    memory: 2048
+    disk: "10GB"
+    forward_ports:
+      - guest: 9000
+        host: 9000
+    ip_address: "172.40.231.155"
+    location: "DC1"
+    sfc_id: "MS_Template_1"
+    sfc_id_instance: "MS_I1"
+    sf_id: "adaptive_streaming"
+    sf_id_instance: "adaptive_streaming_I1"
+    ipendpoint_id: "adaptive_streaming_I1_minio"
+    influxdb_url: "http://172.40.231.51:8086"
+    database_name: "CLMCMetrics"
+  - name: test-runner
+    cpus: 1
+    memory: 2048
+    disk: "10GB"
+    ip_address: "172.40.231.200"

clmctest/inputs/rspec.yml

-## © University of Southampton IT Innovation Centre, 2018
+## (c) University of Southampton IT Innovation Centre, 2018
 ##
 ## Copyright in this software belongs to University of Southampton
 ## IT Innovation Centre of Gamma House, Enterprise Road, 
@@ -31,7 +31,7 @@ hosts:
         host: 8888
       - guest: 9092
         host: 9092
-    ip_address: "203.0.113.100"
+    ip_address: "172.40.231.51"
   - name: apache
     cpus: 1
     memory: 2048
@@ -40,14 +40,14 @@ hosts:
     forward_ports:
       - guest: 80
         host: 8881
-    ip_address: "203.0.113.101"
+    ip_address: "172.40.231.150"
     location: "DC1"
     sfc_id: "MS_Template_1"
     sfc_id_instance: "MS_I1"
     sf_id: "adaptive_streaming"
     sf_id_instance: "adaptive_streaming_I1"
     ipendpoint_id: "adaptive_streaming_I1_apache1"
-    influxdb_url: "http://203.0.113.100:8086"
+    influxdb_url: "http://172.40.231.51:8086"
     database_name: "CLMCMetrics"
   - name: nginx
     cpus: 1
@@ -57,14 +57,14 @@ hosts:
     forward_ports:
       - guest: 80
         host: 8882
-    ip_address: "203.0.113.102"
+    ip_address: "172.40.231.151"
     location: "DC1"
     sfc_id: "MS_Template_1"
     sfc_id_instance: "MS_I1"
     sf_id: "adaptive_streaming"
     sf_id_instance: "adaptive_streaming_nginx_I1"
     ipendpoint_id: "adaptive_streaming_nginx_I1_apache1"
-    influxdb_url: "http://203.0.113.100:8086"
+    influxdb_url: "http://172.40.231.51:8086"
     database_name: "CLMCMetrics"
   - name: mongo
     cpus: 1
@@ -74,14 +74,14 @@ hosts:
     forward_ports:
       - guest: 80
         host: 8883
-    ip_address: "203.0.113.103"
+    ip_address: "172.40.231.152"
     location: "DC1"
     sfc_id: "MS_Template_1"
     sfc_id_instance: "MS_I1"
     sf_id: "metadata_database"
     sf_id_instance: "metadata_database_I1"
     ipendpoint_id: "metadata_database_I1_apache1"
-    influxdb_url: "http://203.0.113.100:8086"
+    influxdb_url: "http://172.40.231.51:8086"
     database_name: "CLMCMetrics" 
   - name: ffmpeg
     cpus: 1
@@ -91,14 +91,14 @@ hosts:
     forward_ports:
       - guest: 80
         host: 8884
-    ip_address: "203.0.113.104"
+    ip_address: "172.40.231.153"
     location: "DC1"
     sfc_id: "MS_Template_1"
     sfc_id_instance: "MS_I1"
     sf_id: "metadata_database"
     sf_id_instance: "metadata_database_I1"
     ipendpoint_id: "metadata_database_I1_apache1"
-    influxdb_url: "http://203.0.113.100:8086"
+    influxdb_url: "http://172.40.231.51:8086"
     database_name: "CLMCMetrics" 
   - name: host
     cpus: 1
@@ -108,17 +108,34 @@ hosts:
     forward_ports:
       - guest: 80
         host: 8885
-    ip_address: "203.0.113.105"
+    ip_address: "172.40.231.154"
     location: "DC1"
     sfc_id: "MS_Template_1"
     sfc_id_instance: "MS_I1"
     sf_id: "adaptive_streaming"
     sf_id_instance: "adaptive_streaming_I1"
     ipendpoint_id: "adaptive_streaming_I1_apache1"
-    influxdb_url: "http://203.0.113.100:8086"
+    influxdb_url: "http://172.40.231.51:8086"
     database_name: "CLMCMetrics"
   - name: test-runner
     cpus: 1
     memory: 2048
     disk: "10GB"
-    ip_address: "203.0.113.150"    
+    ip_address: "172.40.231.200"
\ No newline at end of file
+  - name: minio
+    service_name: "minio"
+    cpus: 1
+    memory: 2048
+    disk: "10GB"
+    forward_ports:
+      - guest: 9000
+        host: 9000
+    ip_address: "172.40.231.155"
+    location: "DC1"
+    sfc_id: "MS_Template_1"
+    sfc_id_instance: "MS_I1"
+    sf_id: "adaptive_streaming"
+    sf_id_instance: "adaptive_streaming_I1"
+    ipendpoint_id: "adaptive_streaming_I1_minio"
+    influxdb_url: "http://172.40.231.51:8086"
+    database_name: "CLMCMetrics"
\ No newline at end of file

clmctest/inputs/test_config_collector.py

+#!/usr/bin/python3
+"""
+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          29-04-2018
+##      Created for Project :   FLAME
+"""
+
+import pytest
+import time
+import random
+import logging
+import sys
+
+from config_collector import ConfigCollector
+
+STATE_INDEX = 0
+TIME_INDEX = 1
+ 
+samples = [[['active', 0], ['active', 2]], 
+        [['active', 0], ['active', 2], ['active', 4]],
+        [['active', 0], ['failed', 2]],
+        [['active', 0], ['active', 2], ['inactive', 4], ['active', 6], ['failed', 8], ['inactive', 10]],
+        [['active', 0], ['inactive', 2], ['failed', 4], ['active', 6], ['inactive', 8], ['failed', 10]]]
+
+def get_sample_test():
+    global sample_set
+    global current_index
+ 
+    sample = (samples[sample_set][current_index][STATE_INDEX], time.time())    
+    sample_count = len(samples[sample_set])
+    if current_index < sample_count-1:
+        current_index +=1
+    else:
+        current_index = 0
+    return sample
+
+def write_output(measurement):
+    print("Writing measurement output {0}".format(measurement))
+
+sample_set = 0
+current_index = 0
+
+def test_agg():
+    t = ConfigCollector(get_sample_test, write_output, "resource")    
+    measurement = t.create_measurement(samples[0], 10, 12)
+    assert measurement[0]['fields']['current_state'] == 'active'
+    assert measurement[0]['fields']['current_state_time'] == 12
+    assert measurement[0]['fields']['active_sum'] == 12
+    assert measurement[0]['fields']['active_count'] == 1
+    assert measurement[0]['time'] == 12000000000    
+
+    t = ConfigCollector(get_sample_test, write_output, "resource")
+    measurement = t.create_measurement(samples[1], 10, 14)
+    assert measurement[0]['fields']['current_state'] == 'active'
+    assert measurement[0]['fields']['current_state_time'] == 14
+    assert measurement[0]['fields']['active_sum'] == 14
+    assert measurement[0]['fields']['active_count'] == 1
+    assert measurement[0]['time'] == 14000000000    
+
+    t = ConfigCollector(get_sample_test, write_output, "resource")
+    measurement = t.create_measurement(samples[2], 8, 10)
+    assert measurement[0]['fields']['current_state'] == 'failed'
+    assert measurement[0]['fields']['current_state_time'] == 0
+    assert measurement[0]['fields']['active_sum'] == 2
+    assert measurement[0]['fields']['active_count'] == 1
+    assert measurement[0]['fields']['failed_sum'] == 0
+    assert measurement[0]['fields']['failed_count'] == 1
+    assert measurement[0]['time'] == 10000000000    
+
+    t = ConfigCollector(get_sample_test, write_output, "resource")
+    measurement = t.create_measurement(samples[3], 2, 12)
+    assert measurement[0]['fields']['current_state'] == 'inactive'
+    assert measurement[0]['fields']['current_state_time'] == 0
+    assert measurement[0]['fields']['active_sum'] == 6
+    assert measurement[0]['fields']['active_count'] == 2
+    assert measurement[0]['fields']['inactive_sum'] == 2
+    assert measurement[0]['fields']['inactive_count'] == 2
+    assert measurement[0]['fields']['failed_sum'] == 2
+    assert measurement[0]['fields']['failed_count'] == 1
+    assert measurement[0]['time'] == 12000000000       
+    
+    t = ConfigCollector(get_sample_test, write_output, "resource")    
+    measurement = t.create_measurement(samples[4], 4, 14)
+    assert measurement[0]['fields']['current_state'] == 'failed'
+    assert measurement[0]['fields']['current_state_time'] == 0
+    assert measurement[0]['fields']['active_sum'] == 4
+    assert measurement[0]['fields']['active_count'] == 2
+    assert measurement[0]['fields']['inactive_sum'] == 4
+    assert measurement[0]['fields']['inactive_count'] == 2
+    assert measurement[0]['fields']['failed_sum'] == 2
+    assert measurement[0]['fields']['failed_count'] == 2
+    assert measurement[0]['time'] == 14000000000   
+
+def test_one_period_collection():
+    global sample_set
+    global current_index
+
+    # one measurementing period
+    sample_set = 1
+    current_index = 0
+    t = ConfigCollector(get_sample_test, write_output, "resource", 2, 6)
+    t.start()
+    time.sleep(8)
+    t.stop()
+    print("Current measurement: {0}".format(str(t.current_measurement)))
+    assert t.current_measurement[0]['fields']['current_state'] == 'active'
+    assert int(round(t.current_measurement[0]['fields']['current_state_time'])) == 6
+    assert int(round(t.current_measurement[0]['fields']['active_sum'])) == 6
+    assert int(round(t.current_measurement[0]['fields']['active_count'])) == 1
+
+def test_multi_period_single_state_collection():
+    global sample_set
+    global current_index
+    # two measurementing periods
+    sample_set = 1
+    current_index = 0
+    t = ConfigCollector(get_sample_test, write_output, "resource", 1, 3)
+    t.start()
+    time.sleep(7)
+    t.stop()
+    print("Current measurement: {0}".format(str(t.current_measurement)))
+    assert t.current_measurement[0]['fields']['current_state'] == 'active'
+    assert int(round(t.current_measurement[0]['fields']['current_state_time'])) == 6
+    assert int(round(t.current_measurement[0]['fields']['active_sum'])) == 6
+    assert int(round(t.current_measurement[0]['fields']['active_count'])) == 1
+
+# [['active', 0], ['inactive', 2], ['failed', 4], ['active', 6], ['inactive', 8], ['failed', 10]]
+def test_multi_period_multi_state_collection():
+    global sample_set
+    global current_index
+    # 6 samples and 2 measurementing periods
+    sample_set = 4
+    current_index = 0
+    t = ConfigCollector(get_sample_test, write_output, "resource", 2, 10)
+    t.start()
+    time.sleep(13)
+    t.stop()
+    print("Current measurement: {0}".format(str(t.current_measurement)))
+    assert t.current_measurement[0]['fields']['current_state'] == 'failed'
+    assert int(round(t.current_measurement[0]['fields']['current_state_time'])) == 0
+    assert int(round(t.current_measurement[0]['fields']['active_sum'])) == 4
+    assert int(round(t.current_measurement[0]['fields']['active_count'])) == 2
+    assert int(round(t.current_measurement[0]['fields']['inactive_sum'])) == 4
+    assert int(round(t.current_measurement[0]['fields']['inactive_count'])) == 2
+    assert int(round(t.current_measurement[0]['fields']['failed_sum'])) == 2
+    assert int(round(t.current_measurement[0]['fields']['failed_count'])) == 2
\ No newline at end of file

clmctest/inputs/test_systemctl_mon.py

+#!/usr/bin/python3
+"""
+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          29-04-2018
+##      Created for Project :   FLAME
+"""
+
+import pytest
+import time
+import random
+import logging
+import sys
+
+from systemctl_monitor import SystemctlMonitor
+
+URL = "localhost"
+PORT = "8186"
+DATABASE = "CLMCMetrics"
+
+@pytest.mark.parametrize("service_name", [('nginx')])
+def test_create_measurement(telegraf_agent_config, service_name):
+
+    service = 'unknown'
+    for s in telegraf_agent_config['hosts']:
+        if s['name'] == service_name:
+            service = s
+            continue
+    assert service != 'unknown', "{0} not in list of hosts".format(service_name)   
+
+    mon = SystemctlMonitor(service_name, 2, 10, service['ip_address'], 8186, service['database_name'])
+    report = {'time': 1526042434.1773288, 'fields': {'loaded.active.running_sum': 231.85903143882751, 'current_state_time': 231.85903143882751, 'current_state': 'loaded.active.running', 'loaded.active.running_count': 1}}
+    measurement = mon.create_measurement(report)
+    assert measurement[0]['tags']['resource_name'] == service_name
+    assert measurement[0]['fields']['current_state'] == report['fields']['current_state']
+ 
+def test_get_systemctl_status(telegraf_agent_config):
+    mon = SystemctlMonitor('nginx', 2, 10, URL, PORT, DATABASE)
+    state = mon.get_systemctl_status('nginx')
+    assert state == 'loaded.active.running'
+
+def test_monitor(telegraf_agent_config):
+    mon = SystemctlMonitor('nginx', 2, 10, URL, PORT, DATABASE)
+    mon.start()
+    time.sleep(21)
+    mon.stop()
+    measurement = mon.get_current_measurement()
+    print("Current measurement: {0}".format(str(measurement)))
\ No newline at end of file

clmctest/inputs/test_telegraf_agents.py

@@ -34,6 +34,7 @@ from influxdb import InfluxDBClient
     ('mongo'),
     ('ffmpeg'),
     ('host'),
+    ('minio')
     ])
 def test_service_name(telegraf_agent_config, service_name):
     assert any(s['name'] == service_name for s in telegraf_agent_config['hosts']), "{0} not in list of hosts".format(service_name)
@@ -63,7 +64,11 @@ def test_ping(telegraf_agent_config):
     ('net', 'SELECT mean("bytes_sent") AS "mean" FROM "CLMCMetrics"."autogen"."net"', 0),
     ('disk', 'SELECT mean("free") AS "mean" FROM "CLMCMetrics"."autogen"."disk"', 0),
     ('diskio', 'SELECT mean("write_bytes") AS "mean" FROM "CLMCMetrics"."autogen"."diskio"', 0),
-    ('mem', 'SELECT mean("free") AS "mean" FROM "CLMCMetrics"."autogen"."mem"', 0)              
+    ('mem', 'SELECT mean("free") AS "mean" FROM "CLMCMetrics"."autogen"."mem"', 0),
+    # Report MINIO's HTTP request response time (as a rolling difference of the sum total)
+    ('minio_http_requests_duration_seconds', 'SELECT difference(max("sum")) AS "mean" FROM "CLMCMetrics"."autogen"."minio_http_requests_duration_seconds" WHERE time > now() - 1h GROUP BY time(10s)',0),              
+    # Report the average change in difference of MINIO's HTTP response time (the inner query determines a rolling difference between sampling periods [respTimeDiff])
+    ('minio_http_requests_duration_seconds', 'SELECT mean("respTimeDiff") AS "mean" FROM (SELECT difference(max("sum")) AS "respTimeDiff" FROM "CLMCMetrics"."autogen"."minio_http_requests_duration_seconds" WHERE time > now() - 1h GROUP BY time(10s))',0)              
     ])
 def test_all_inputs(influxdb, measurement, query, expected_result):
     """

clmctest/monitoring/rspec.yml

-## © University of Southampton IT Innovation Centre, 2018
+## (c) University of Southampton IT Innovation Centre, 2018
 ##
 ## Copyright in this software belongs to University of Southampton
 ## IT Innovation Centre of Gamma House, Enterprise Road, 
@@ -31,7 +31,7 @@ hosts:
         host: 8888
       - guest: 9092
         host: 9092
-    ip_address: "203.0.113.100"
+    ip_address: "172.40.231.51"
   - name: ipendpoint1
     cpus: 1
     memory: 2048
@@ -40,14 +40,14 @@ hosts:
     forward_ports:
       - guest: 80
         host: 8081
-    ip_address: "203.0.113.140"
-    location: "DC1"
+    ip_address: "172.40.231.170"
+    location: "nova"
     sfc_id: "media_service_A"
-    sfc_id_instance: "MS_I1"
+    sfc_id_instance: "StackID"
     sf_id: "test-sf-clmc-agent-build"
     sf_id_instance: "ms-A.ict-flame.eu"
     ipendpoint_id: "endpoint1.ms-A.ict-flame.eu"
-    influxdb_url: "http://203.0.113.100:8086"
+    influxdb_url: "http://172.40.231.51:8086"
     database_name: "CLMCMetrics"
   - name: ipendpoint2
     cpus: 1
@@ -57,17 +57,17 @@ hosts:
     forward_ports:
       - guest: 80
         host: 8082
-    ip_address: "203.0.113.141"
-    location: "DC2"
+    ip_address: "172.40.231.171"
+    location: "nova"
     sfc_id: "media_service_A"
-    sfc_id_instance: "MS_I1"
+    sfc_id_instance: "StackID"
     sf_id: "test-sf-clmc-agent-build"
     sf_id_instance: "ms-A.ict-flame.eu"
     ipendpoint_id: "endpoint2.ms-A.ict-flame.eu"
-    influxdb_url: "http://203.0.113.100:8086"
+    influxdb_url: "http://172.40.231.51:8086"
     database_name: "CLMCMetrics"      
   - name: test-runner
     cpus: 1
     memory: 2048
     disk: "10GB"
-    ip_address: "203.0.113.150"       
+    ip_address: "172.40.231.200"       
\ No newline at end of file

clmctest/scripts/.pytest_cache/v/cache/lastfailed

-{
-  "test_config_telegraf.py::test_write_telegraf_conf": true
-}
\ No newline at end of file

clmctest/scripts/rspec.yml

-## © University of Southampton IT Innovation Centre, 2018
+## (c) University of Southampton IT Innovation Centre, 2018
 ##
 ## Copyright in this software belongs to University of Southampton
 ## IT Innovation Centre of Gamma House, Enterprise Road, 
@@ -24,5 +24,5 @@ hosts:
     cpus: 1
     memory: 2048
     disk: "10GB"
-    ip_address: "203.0.113.150"
+    ip_address: "172.40.231.200"
     
\ No newline at end of file

clmctest/services/minio/install.sh

+#!/bin/bash
+#/////////////////////////////////////////////////////////////////////////
+#//
+#// (c) University of Southampton IT Innovation Centre, 2018
+#//
+#// Copyright in this software belongs to University of Southampton
+#// IT Innovation Centre of Gamma House, Enterprise Road,
+#// Chilworth Science Park, Southampton, SO16 7NS, UK.
+#//
+#// This software may not be used, sold, licensed, transferred, copied
+#// or reproduced in whole or in part in any manner or form or in or
+#// on any media by any person other than in accordance with the terms
+#// of the Licence Agreement supplied with the software, or otherwise
+#// without the prior written consent of the copyright owners.
+#//
+#// This software is distributed WITHOUT ANY WARRANTY, without even the
+#// implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+#// PURPOSE, except where stated in the Licence Agreement supplied with
+#// the software.
+#//
+#//      Created By :            Simon Crowle
+#//      Created Date :          18/04/2018
+#//      Created for Project :   FLAME
+#//
+#/////////////////////////////////////////////////////////////////////////
+
+# Prerequisites
+sudo apt-get update
+sudo apt-get install -y wget git
+
+# Environment variables for this process
+GOROOT=/usr/local/go
+PATH=$PATH:$GOROOT/bin
+CGO_ENABLED=0
+ENV MINIO_UPDATE=off
+
+# And also for separate MINIO process
+echo "export GOROOT=/usr/local/go" >> ~/.profile
+echo "export PATH=$PATH:$GOROOT/bin" >> ~/.profile
+echo "export CGO_ENABLED=0" >> ~/.profile
+echo "export ENV MINIO_UPDATE=off" >> ~/.profile
+
+# Install GO
+# -----------------------------------------------------------------------
+cd /tmp
+wget https://dl.google.com/go/go1.10.1.linux-amd64.tar.gz
+sudo tar -C /usr/local -xzf go1.10.1.linux-amd64.tar.gz
+
+# Install MINIO
+# -----------------------------------------------------------------------
+sudo mkdir -p /usr/local/go/src/github.com/minio
+cd /usr/local/go/src/github.com/minio
+git clone https://github.com/minio/minio
+
+cd minio
+go install -v -ldflags "$(go run buildscripts/gen-ldflags.go)"
+
+# check minio configuration available
+MINIO_CONF_SOURCE=$REPO_ROOT"/clmctest/services/minio/minio.conf"
+if [ ! -f "$MINIO_CONF_SOURCE" ]; then
+    echo "Error: MINIO conf file does not exist on in the repo. "$MINIO_CONF_SOURCE
+    exit 1
+fi
+
+# configure minio & check
+sudo mkdir -p /etc/minio
+MINIO_CONF_TARGET="/etc/minio/config.json"
+sudo cp -rf $MINIO_CONF_SOURCE $MINIO_CONF_TARGET
+
+if [ ! -f "$MINIO_CONF_TARGET" ]; then
+    echo "Error: MINIO conf copy failed to target machine. "$MINIO_CONF_TARGET
+    exit 1
+fi
+
+mkdir -p /vagrant/minio_data
+
+# Start MINIO
+# -----------------------------------------------------------------------
+nohup minio server --config-dir /etc/minio /vagrant/minio_data &>/dev/null &
+echo Started MINIO
+
+
+
+

clmctest/services/minio/minio.conf

+{
+    "version": "23",
+    "credential": {
+        "accessKey": "F2H4IJ5SITH4C85LAVZL",
+        "secretKey": "RiI7f6b9KMKwo+rSuOkzUa13+2dFA7oMyaTqgF/q"
+    },
+    "region": "",
+    "browser": "on",
+    "domain": "",
+    "storageclass": {
+        "standard": "",
+        "rrs": ""
+    },
+    "cache": {
+        "drives": [],
+        "expiry": 90,
+        "exclude": []
+    },
+    "notify": {
+       
+        }
+}
\ No newline at end of file

clmctest/services/minio/telegraf_minio.conf

+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Simon  Crowle
+##      Created Date :          18-04-2018
+##      Created for Project :   FLAME
+
+[[inputs.prometheus]]
+    urls = ["http://localhost:9000/minio/prometheus/metrics"]
\ No newline at end of file

clmctest/services/nginx/install.sh

@@ -50,3 +50,32 @@ fi
 
 nginx -s reload
 systemctl start nginx
+
+## install a configuration monitoring service, this needs to be in a venv with the rest of the CLMC
+sudo apt-get install python3 python3-pip -y
+sudo pip3 install pyaml influxdb
+
+svc="nginxmon"
+
+echo "install systemctl monitoring service"
+svc_file="${svc}.service"
+echo "[Unit]" > $svc_file
+echo "Description=nginxmon" >> $svc_file
+echo "After=network-online.target" >> $svc_file
+echo "" >> $svc_file
+echo "[Service]" >> $svc_file
+echo "WorkingDirectory=${inst}/${dir}" >> $svc_file
+echo "ExecStart=/usr/bin/python3 ${REPO_ROOT}/src/monitoring/systemctl_monitor.py -service nginx -rate 2 -agg 10 -host localhost -port 8186 -db CLMCMetrics" >> $svc_file
+echo "ExecStop=/usr/bin/bash ${REPO_ROOT}/src/monitoring/stop_systemctl_monitor.sh" >> $svc_file
+echo "" >> $svc_file
+echo "[Install]" >> $svc_file
+echo "WantedBy=network-online.target" >> $svc_file
+sudo cp $svc_file /lib/systemd/system
+rm $svc_file
+
+echo "enable"
+sudo systemctl daemon-reload
+sudo systemctl enable ${svc}
+
+echo "start"
+sudo systemctl start ${svc}
\ No newline at end of file

clmctest/services/nginx/telegraf_nginx.conf

@@ -26,3 +26,8 @@
 
   ## HTTP response timeout (default: 5s)
 #  response_timeout = "5s"
+
+# # Influx HTTP write listener
+[[inputs.http_listener]]
+  ## Address and port to host HTTP listener on
+  service_address = ":8186"
\ No newline at end of file

clmctest/streaming/rspec.yml

-## © University of Southampton IT Innovation Centre, 2018
+## (c) University of Southampton IT Innovation Centre, 2018
 ##
 ## Copyright in this software belongs to University of Southampton
 ## IT Innovation Centre of Gamma House, Enterprise Road, 

docs/CLMC monitoring specification for a basic scenario.md

-<!--
-// © University of Southampton IT Innovation Centre, 2018
-//
-// Copyright in this software belongs to University of Southampton
-// IT Innovation Centre of Gamma House, Enterprise Road, 
-// Chilworth Science Park, Southampton, SO16 7NS, UK.
-//
-// This software may not be used, sold, licensed, transferred, copied
-// or reproduced in whole or in part in any manner or form or in or
-// on any media by any person other than in accordance with the terms
-// of the Licence Agreement supplied with the software, or otherwise
-// without the prior written consent of the copyright owners.
-//
-// This software is distributed WITHOUT ANY WARRANTY, without even the
-// implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
-// PURPOSE, except where stated in the Licence Agreement supplied with
-// the software.
-//
-//      Created By :            Simon Crowle
-//      Created Date :          10-01-2018
-//      Created for Project :   FLAME
--->
-
-# Adaptive Streaming Use Case Scenario
-
-## Infrastructure Slice
-
-### *compute_node_config*
-
-| compute_node_config | slice | location | comp_node | cpu | memory | storage | timestamp |
-| --- | --- | --- | --- | --- | --- |--- | --- |
-| compute_node_config | SLICE1 | locA | dc1 | 4 | 8 | 16 | 1515583926868000000 |
-| compute_node_config | SLICE1 | locB | dc2 | 8 | 16 | 64 | 1515583926868000000 |
-| compute_node_config | SLICE1 | locC | dc3 | 48 | 128 | 4000 | 1515583926868000000 |
-
-### *network_config*
-
-| network_config | slice | network | bandwidth | timestamp |
-| --- | --- | --- | --- | --- | --- |--- | 
-| network_config | SLICE1 | data1 | 100 | 1515583926868000000 |
-
-__How do we describe network configuration ?__
-__What is a format of an infrastructure slices ?__
-__What is the relevant information ?__
-
-### *network_interface_config*
-
-| network_interface_config | slice | comp_node | port | network | rx_constraint | tx_constraint | timestamp |
-| --- | --- | --- | --- | --- | --- |--- |--- | 
-| network_config | SLICE1 | dc1 | enps03 | data1 | 1000 | 1000 | 1515583926868000000 |
-| network_config | SLICE1 | dc2 | enps03 | data1 | 1000 | 1000 |  1515583926868000000 |
-| network_config | SLICE1 | dc3 | enps03 | data1 | 1000 | 1000 |  1515583926868000000 |
-
-## NAP
-
-### ipendpoint_route
-
-| ipendpoint_route | location | ipendpoint_id | cont_nav | avg_http_requests_fqdn_rate | avg_network_fqdn_latency | time |
-| --- | --- | --- | --- | --- | --- | --- |
-| ipendpoint_route | \<common tags> | DC1 | ipendpoint1 | http://netflix.com/scream | 386, | 50 | 1515583926868000000 |
-
-## Media Service 
-
-There are various aggregated metrics we can calculate but in the use case scenario we postpone that till later.
-
-### sfc_instance_config
-
-`sfc_i_config,<common_tags>,state <fields> timestamp`
-
-### sf_i_config
-
-`sf_i_config,<common_tags>,state <fields> timestamp`
-
-## IPEndpoint
-
-All IPEndpoint measurements have the following global tags injected by a configured Telegraf agent
-
-* location
-* compute_node
-* sfc
-* sfc_i
-* sf
-* sfc_i
-* ipendpoint
-
-Also NOTE: the metrics provided in the measurements below are effectively a 'snapshot' of usage over a relatively small period of time. The length of this snapshot may vary, depending on the underlying implementation of the instrumentation, so we might have to assume this snapshot is essentially an average of a period of 1 second. Measuring 'usage' is dependent on the units, for example as a proportion of a resource or as a proportion of time.
-
-### ipendpoint_config
-
-| ipendpoint_config | location | sfc | sfc_i | sf | sf_i | ipendpoint | state | cpu| memory | storage |timestamp |
-| --- | --- | --- | --- | --- | --- |--- | --- | --- |  --- |  --- |  --- | 
-| ipendpoint_config | dc1 | MediaServiceTemplate | MediaServiceA | AdaptiveStreamingComp | AdaptiveStreamingComp1 | ipendpoint1 | placed | 2 | 4 | 16 | 1515583926868000000 |
-| ipendpoint_config | dc2 | MediaServiceTemplate | MediaServiceA | AdaptiveStreamingComp | AdaptiveStreamingComp1 | ipendpoint2 | placed | 8 | 16 | 64 | 1515583926868000000 |
-| ipendpoint_config | dc3 | MediaServiceTemplate | MediaServiceA | AdaptiveStreamingComp | AdaptiveStreamingComp1 | ipendpoint3 | placed | 48 | 128 | 4000 | 1515583926868000000 |
-| ipendpoint_config | dc1 | MediaServiceTemplate | MediaServiceA | AdaptiveStreamingComp | AdaptiveStreamingComp1 | ipendpoint1 | booted | 2 | 4 | 16 | 1515583926868000000 |
-| ipendpoint_config | dc2 | MediaServiceTemplate | MediaServiceA | AdaptiveStreamingComp | AdaptiveStreamingComp1 | ipendpoint2 | booted | 8 | 16 | 64 | 1515583926868000000 |
-| ipendpoint_config | dc3 | MediaServiceTemplate | MediaServiceA | AdaptiveStreamingComp | AdaptiveStreamingComp1 | ipendpoint3 | booted | 48 | 128 | 4000 | 1515583926868000000 |
-
-### cpu_usage
-
-| cpu_usage | \<common tags> | cpu | avg_cpu_time_user | avg_cpu_time_idle | timestamp |
-| --- | --- | --- | --- | --- |--- |
-| cpu | \<common tags> | 1 | 40 | 5 | 1515583926868000000 |
-
-### net_port_io
-
-| net_port_io | \<common tags> | avg_packet_drop_rate | avg_packet_error_rate | rx_bytes_port_m | rx_packets_m | tx_bytes_port_m | tx_packets_port_m | timestamp |
-| --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| net_port_io | \<common tags> | 0.3 | 0.1 | 13567 | 768 | 8102 | 356 | 1515583926868000000 |
-
-### mpegdash_service
-
-| mpegdash_service_mon | \<common tags> | cont_nav | cont_rep | user_profile |avg_req_rate | avg_resp_time | peak_resp_time | avg_error_rate | avg_throughput | avg_quality_delivered | avg_startup_delay | avg_dropped_segments |  timestamp |
-| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |--- |
-| mpegdash_service_mon | \<common tags> | http://netflix.com/scream | h264 | profileA | 10 | 40 | 230 | 0.2 | 200 | | 5 | 1200 | 2 | 1515583926868000000 |
-
-