Merge branch 'mediaComponentConfig' into integration

.gitlab-ci.yml

+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          21-03-2018
+##      Created for Project :   FLAME
+
 stages:  
   - build
   - test

README.md

@@ -56,14 +56,14 @@ Testing is implemented using pytest.
 
 The installation script is here:
 
-`test/services/pytest/install.sh`
+`sudo clmctest/services/pytest/install.sh`
 
 using the following convention:
 
 * Tests are written in python using pytest
-* Related tests are stored in a python module `test/<testmodule>` to create a suite of tests. All tests are stored in files test_*.py, there can be many tests per file, and many files per module
+* Related tests are stored in a python module `clmctest/<testmodule>` to create a suite of tests. All tests are stored in files test_*.py, there can be many tests per file, and many files per module
 * Each test module has a rspec.yml that provides the baseline "fixture" for the tests in the module
-* Tests are executed against fixtures. Fixtures are modular "setups" created for a test, that are inserted into the python code using dependancy injection. This offers more flexibility than the *unit style testing. The baseline deployment is created using `vagrant up` with an appropriate rspec, and the pytest fixture reads the rspec.yml and makes the configuration available to the test.
+* Tests are executed against fixtures. Fixtures are modular "setups" created for a test, that are inserted into the python code using dependancy injection. This offers more flexibility than the unit style testing. The baseline deployment is created using `vagrant up` with an appropriate rspec, and the pytest fixture reads the rspec.yml and makes the configuration available to the test.
 * Tests are executed from a guest VM (not the host) in the repo root using the command `pytest test/<testmodule>`
 * Pytest will scan the directory for all tests including in files test_*.py and run them
 
@@ -75,7 +75,7 @@ To set up a simulation of the adaptive streaming use case scenario first install
 
 and then execute the following command
 
-`vagrant --fixture=streaming-sim -- up`
+`vagrant --fixture=monitoring -- up`
 
 This will provision the following VMs clmc-service, ipendpoint1, ipendpoint2
 
@@ -89,7 +89,7 @@ The **clmc-service** vm includes influx, Kapacitor and Chronograf. The following
 
 SSH into the CLMC server
 
-`vagrant --fixture=streaming-sim -- ssh clmc-service`
+`vagrant --fixture=monitoring -- ssh clmc-service`
 
 Then go to the 'vagrant' directory.
 
@@ -99,20 +99,19 @@ The next step is to generate the test data, which could be done in two ways.
 
 First option is to run a python script to generate the test data sets
 
-`python3 test/streaming-sim/StreamingSim.py`
+`python3 clmctest/monitoring/StreamingSim.py`
 
 This script could also be used to clear the generated data by using the '-c' option
 
-`python3 test/streaming-sim/StreamingSim.py -c`
+`python3 clmctest/monitoring/StreamingSim.py -c`
 
-The second option is to directly run the testing module, which will detect if the data was generated, and if not, will automatically
-generate the data before executing the tests. Keep in mind that if the test data is being generated using this way, a 10 seconds timeout 
-is given after the generation is finished so that the data could properly be inserted into the database. If the data was already generated 
-using the first option, only the tests would be executed.
+#### Running the monitoring tests
 
-The command for running the testing module is
+The second option is to directly run the testing module, which will detect if the data was generated, and if not, will automatically generate the data before executing the tests. Keep in mind that if the test data is being generated using this way, a 10 seconds timeout is given after the generation is finished so that the data could properly be inserted into the database. If the data was already generated using the first option, only the tests would be executed.
 
-`pytest -s test/streaming-sim/test_simresults.py`
+The command for running the testing module is:
+
+`pytest -s clmctest/monitoring/test_simresults.py`
 
 The `-s` option in the command is used to output prints used in the test code and is, therefore, optional.
 
@@ -120,4 +119,4 @@ If pytest is not installed, an easy solution is to use the Python Package Index
 
 `sudo apt-get install python3-pip` 
   
-`pip3 install pytest`
+`pip3 install pytest`
\ No newline at end of file

clmctest/inputs/conftest.py

 #!/usr/bin/python3
+"""
+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          02-03-2018
+##      Created for Project :   FLAME
+"""
 
 import pytest
 import time

clmctest/inputs/rspec.yml

+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          02-02-2018
+##      Created for Project :   FLAME
+
 hosts:
   - name: clmc-service
     cpus: 1

clmctest/inputs/test_rspec.py

 #!/usr/bin/python3
+"""
+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          25-02-2018
+##      Created for Project :   FLAME
+"""
 
 from subprocess import run
 from platform import system

clmctest/inputs/test_telegraf_agents.py

 #!/usr/bin/python3
+"""
+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          02-03-2018
+##      Created for Project :   FLAME
+"""
 
 import pytest
 from subprocess import run

clmctest/monitoring/LineProtocolGenerator.py

 #!/usr/bin/python3
+"""
+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          02-02-2018
+##      Created for Project :   FLAME
+"""
 
 # line protocol
 
@@ -23,23 +45,6 @@ def generate_network_report(recieved_bytes, sent_bytes, time):
     return result
 
 
-# Formats VM config
-def generate_vm_config(state, cpu, mem, storage, time):
-    result = [{"measurement": "vm_res_alloc",
-               "tags": {
-                   "vm_state": state
-               },
-               "fields": {
-                   "cpu": cpu,
-                   "memory": mem,
-                   "storage": storage
-               },
-               "time": _getNSTime(time)
-               }]
-
-    return result
-
-
 # Reports cpu usage, scaling on requests
 def generate_cpu_report(cpu_usage, cpu_active_time, cpu_idle_time, time):
     result = [{"measurement": "cpu_usage",
@@ -87,60 +92,83 @@ def generate_ipendpoint_route(resource, requests, latency, time):
     return result
 
 
-# InfluxDB likes to have time-stamps in nanoseconds
-def _getNSTime(time):
-    # Convert to nano-seconds
-    timestamp = int(1000000000*time)
+def generate_endpoint_config(time, cpu, mem, storage, current_state, current_state_time, **kwargs):
+    """
+    generates a measurement for a VM configuration states
 
-    return timestamp
+    :param cpu: the number of CPUs of VM endpoint
+    :param mem: memory of VM endpoint
+    :param storage: storage capacity of VM endpoint
+    :param current_state: the current state the endpoint is in (TAG)
+    :param current_state_time: the part of the sampling period the endpoint was in the current state
+    :param time: time of measurement
+    :param kwargs: 'python-style' keyword arguments used to store the state as a key and it's respective state period (in seconds) as value
 
-def generate_mc_service_config( mcMeasurement, stateTimeStats, time ):
+    :return: dictionary object representing the data to post on influx
+    """
 
-    validStats = validate_state_time_stats( stateTimeStats )
+    # lambda function to validate whether a state is given as a key in the keyword arguments dictionary
+    validate = lambda key: kwargs.get(key) if key in kwargs else 0.0
 
-    result = [{ "measurement" : mcMeasurement,
-                "fields" :
-                { "stopped"      : validStats['stopped'],
-                  "avg_stopped"  : validStats['avg_stopped'],
-                  "starting"     : validStats['starting'],
-                  "avg_starting" : validStats['avg_starting'],
-                  "running"      : validStats['running'],
-                  "avg_running"  : validStats['avg_running'],
-                  "stopping"     : validStats['stopping'],
-                  "avg_stopping" : validStats['avg_stopping']
-                },
-                "time" : _getNSTime(time) }]
+    # generate and validate the state values
+    fields = {"cpus": cpu, "memory": mem, "storage": storage, "current_state_time": current_state_time}   # NOTE: Do we need the cpus, memory and storage fields ?
+    for state in ("unplaced", "placing", "placed", "booting", "booted", "connecting", "connected"):
+        fields["{0}_sum".format(state)] = validate(state)
+        fields[("{0}_mst".format(state))] = validate("{0}_mst".format(state))
+
+    result = [{"measurement": "endpoint_config",
+               "tags": {
+                   "current_state": current_state
+               },
+               "fields": fields,
+               "time": _getNSTime(time)}]
 
     return result
 
-def validate_state_time_stats( stateTimeStats ):
+def generate_mc_service_config( time, mcMeasurement, current_state, current_state_time, config_state_values ):
+    """
+    generates a measurement line for a media component configuration state
 
-    if ( not 'stopped' in stateTimeStats ):
-        stateTimeStats['stopped'] = 0.0
+    : time                - timestamp for the measurement
+    : mcMeasurement       - measurement label
+    : current_state       - the current state of the service configuration
+    : current_state_time  - the current length of time in the current state
+    : config_state_values - dictionary of media component configuration states (summed time and mean average over the sampling period)
+    :                     - stopped, starting, running, stopping [use '_sum' and '_mst' for sum and average respectively]
+    """
 
-    if ( not 'avg_stopped' in stateTimeStats ):
-        stateTimeStats['avg_stopped'] = 0.0
-    
-    if ( not 'starting' in stateTimeStats ):
-        stateTimeStats['starting'] = 0.0
+    # define state value validation function (inserting key/0 where key is supplied)
+    validate_f = lambda key: config_state_values.get(key) if key in config_state_values else 0.0
 
-    if ( not 'avg_starting' in stateTimeStats ):
-        stateTimeStats['avg_starting'] = 0.0
+    # define expected keys
+    state_keys = [ "stopped_sum",  "stopped_mst", 
+                   "starting_sum", "starting_mst",
+                   "running_sum",  "running_mst",
+                   "stopping_sum", "stopping_mst" ]
 
-    if ( not 'running' in stateTimeStats ):
-        stateTimeStats['running'] = 0.0
+    # define current state time first
+    fields = {}
+    fields["current_state_time"] = current_state_time
 
-    if ( not 'avg_running' in stateTimeStats ):
-        stateTimeStats['avg_running'] = 0.0
+    # then add in validated state values
+    for key in state_keys :
+        fields[key] = validate_f(key)
 
-    if ( not 'stopping' in stateTimeStats ):
-        stateTimeStats['stopping'] = 0.0
+    # compose result
+    result = [{ "measurement" : mcMeasurement,
+                "tags"        : { "current_state" : current_state },
+                "fields"      : fields,
+                "time"        : _getNSTime(time)
+             }]
 
-    if ( not 'avg_stopping' in stateTimeStats ):
-        stateTimeStats['avg_stopping'] = 0.0
+    return result
 
-    return stateTimeStats
+# InfluxDB likes to have time-stamps in nanoseconds
+def _getNSTime(time):
+    # Convert to nano-seconds
+    timestamp = int(1000000000*time)
 
+    return timestamp
 
 
 # DEPRECATED
@@ -153,6 +181,23 @@ def quote_wrap(string):
     return "\"" + string + "\""
 
 
+# Formats VM config
+def generate_vm_config(state, cpu, mem, storage, time):
+    result = [{"measurement": "vm_res_alloc",
+               "tags": {
+                   "vm_state": state
+               },
+               "fields": {
+                   "cpu": cpu,
+                   "memory": mem,
+                   "storage": storage
+               },
+               "time": _getNSTime(time)
+               }]
+
+    return result
+
+
 def _generateClientRequest(cReq, id, time):
     # Tags first
     result = 'sid="' + str(id) + '",' + cReq

clmctest/monitoring/conftest.py

 #!/usr/bin/python3
+"""
+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          25-02-2018
+##      Created for Project :   FLAME
+"""
 
 import pytest
 import yaml
@@ -12,16 +34,16 @@ def streaming_sim_config():
     """
     Reads the service configuration deployed for the streaming simulation test.
 
-    :param request: access the parameters of the fixture
     :return: the python object representing the read YAML file
     """
     rspec = pkg_resources.resource_filename('clmctest.monitoring', 'rspec.yml')
-    print("rspec file: {0}".format(rspec))
+    print("\nrspec file: {0}".format(rspec))
 
     with open(rspec, 'r') as stream:
         data_loaded = yaml.load(stream)
     return data_loaded
 
+
 @pytest.fixture(params=[{'database': 'CLMCMetrics'}], scope='module')
 def influx_db(streaming_sim_config, request):
     """
@@ -34,6 +56,7 @@ def influx_db(streaming_sim_config, request):
 
     return InfluxDBClient(host=streaming_sim_config['hosts'][0]['ip_address'], port='8086', database=request.param['database'], timeout=10)
 
+
 @pytest.fixture(scope="module")
 def simulator(streaming_sim_config):
 
@@ -42,10 +65,7 @@ def simulator(streaming_sim_config):
     agent1_url     = "http://" + streaming_sim_config['hosts'][1]['ip_address'] + ":8186"
     agent2_url     = "http://" + streaming_sim_config['hosts'][2]['ip_address'] + ":8186"
 
-    simulator = Sim( influx_url, influx_db_name, agent1_url, agent2_url )
-
-    dbs = simulator.db_client.get_list_database()
-    dbs = [db.get("name") for db in dbs]
+    simulator = Sim(influx_url, influx_db_name, agent1_url, agent2_url)
 
     simulator.reset()
 

clmctest/monitoring/rspec.yml

+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          02-02-2018
+##      Created for Project :   FLAME
+
 hosts:
   - name: clmc-service
     cpus: 1

clmctest/monitoring/test_rspec.py

 #!/usr/bin/python3
+"""
+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          25-02-2018
+##      Created for Project :   FLAME
+"""
 
 from subprocess import run
 from platform import system

clmctest/monitoring/test_simresults.py

 #!/usr/bin/python3
+"""
+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Nikolay Stanchev
+##      Created Date :          06-03-2018
+##      Created for Project :   FLAME
+"""
 
 import pytest
 import time
+import random
 
 
 class TestSimulation(object):
     """
     A testing class used to group all the tests related to the simulation data
     """
-    @pytest.fixture(scope='class')
-    def run_simulator( self, simulator ):
-        simulator.run( 3600 )
-      
-        print( "Waiting for INFLUX to finish receiving simulation data..." )
-        time.sleep( 10 ) # wait for data to finish arriving at the INFLUX database
+
+    @pytest.fixture(scope='class', autouse=True)
+    def run_simulator(self, simulator):
+        random.seed(0) # Seed random function so we can reliably test for average queries
+
+        print("Running simulation, please wait...")
+        simulator.run(3600)
+
+        print("Waiting for INFLUX to finish receiving simulation data...")
+        time.sleep(10)  # wait for data to finish arriving at the INFLUX database
+        print( "... simulation data fixture finished" )
 
     @pytest.mark.parametrize("query, expected_result", [
         ('SELECT count(*) FROM "CLMCMetrics"."autogen"."cpu_usage"',
@@ -24,22 +52,61 @@ class TestSimulation(object):
          {"time": "1970-01-01T00:00:00Z", "count_avg_response_time": 7200, "count_peak_response_time": 7200, "count_requests": 7200}),
         ('SELECT count(*) FROM "CLMCMetrics"."autogen"."net_port_io"',
          {"time": "1970-01-01T00:00:00Z", "count_RX_BYTES_PORT_M": 7200, "count_TX_BYTES_PORT_M": 7200}),
-        ('SELECT count(*) FROM "CLMCMetrics"."autogen"."vm_res_alloc"',
-         {"time": "1970-01-01T00:00:00Z", "count_cpu": 12, "count_memory": 12, "count_storage": 12}),
-         
-        ('SELECT count(*) FROM "CLMCMetrics"."autogen"."mpegdash_service_config" WHERE ipendpoint=\'endpoint1.ms-A.ict-flame.eu\'',
-         {"time" : "1970-01-01T00:00:00Z", "count_avg_running" : 3604, "count_avg_starting" : 3604, "count_avg_stopped" : 3604, "count_avg_stopping" : 3604, "count_running" : 3604, "count_starting" : 3604, "count_stopped" : 3604, "count_stopping" : 3604}),
-        ('SELECT count(*) FROM "CLMCMetrics"."autogen"."mpegdash_service_config" WHERE ipendpoint=\'endpoint2.ms-A.ict-flame.eu\'',
-         {"time" : "1970-01-01T00:00:00Z", "count_avg_running" : 3604, "count_avg_starting" : 3604, "count_avg_stopped" : 3604, "count_avg_stopping" : 3604, "count_running" : 3604, "count_starting" : 3604, "count_stopped" : 3604, "count_stopping" : 3604}),
+
+        ('SELECT count(*) FROM "CLMCMetrics"."autogen"."endpoint_config" WHERE ipendpoint=\'endpoint1.ms-A.ict-flame.eu\'',
+         {"time": "1970-01-01T00:00:00Z", "count_current_state_time": 3639, "count_unplaced_sum": 3639, "count_unplaced_mst": 3639, "count_placing_sum": 3639, "count_placing_mst": 3639, "count_placed_sum": 3639, "count_placed_mst": 3639, "count_booting_sum": 3639, "count_booting_mst": 3639, "count_booted_sum": 3639,
+          "count_booted_mst": 3639, "count_connecting_sum": 3639, "count_connecting_mst": 3639, "count_connected_sum": 3639, "count_connected_mst": 3639, "count_cpus": 3639, "count_memory": 3639, "count_storage": 3639}),
+        ('SELECT count(*) FROM "CLMCMetrics"."autogen"."endpoint_config" WHERE ipendpoint=\'endpoint2.ms-A.ict-flame.eu\'',
+         {"time": "1970-01-01T00:00:00Z", "count_current_state_time": 3639, "count_unplaced_sum": 3639, "count_unplaced_mst": 3639, "count_placing_sum": 3639, "count_placing_mst": 3639, "count_placed_sum": 3639, "count_placed_mst": 3639, "count_booting_sum": 3639, "count_booting_mst": 3639, "count_booted_sum": 3639,
+          "count_booted_mst": 3639, "count_connecting_sum": 3639, "count_connecting_mst": 3639, "count_connected_sum": 3639, "count_connected_mst": 3639, "count_cpus": 3639, "count_memory": 3639, "count_storage": 3639}),
+
+        ('SELECT count(*) FROM "CLMCMetrics"."autogen"."mpegdash_mc_config" WHERE ipendpoint=\'endpoint1.ms-A.ict-flame.eu\'',
+         {"time": "1970-01-01T00:00:00Z", "count_current_state_time": 3607, "count_running_mst": 3607, "count_running_sum": 3607, "count_starting_mst": 3607, "count_starting_sum": 3607, "count_stopped_mst": 3607, "count_stopped_sum": 3607, "count_stopping_mst": 3607, "count_stopping_sum": 3607}),
+        ('SELECT count(*) FROM "CLMCMetrics"."autogen"."mpegdash_mc_config" WHERE ipendpoint=\'endpoint2.ms-A.ict-flame.eu\'',
+         {"time": "1970-01-01T00:00:00Z", "count_current_state_time": 3607, "count_running_mst": 3607, "count_running_sum": 3607, "count_starting_mst": 3607, "count_starting_sum": 3607, "count_stopped_mst": 3607, "count_stopped_sum": 3607, "count_stopping_mst": 3607, "count_stopping_sum": 3607}),
+
+        ('SELECT mean(placing_mst) as "placing_mst" FROM "CLMCMetrics"."autogen"."endpoint_config" WHERE placing_mst <> 0 and ipendpoint=\'endpoint1.ms-A.ict-flame.eu\'',
+         {"time": "1970-01-01T00:00:00Z", "placing_mst": 9.4}),
+        ('SELECT mean(booting_mst) as "booting_mst" FROM "CLMCMetrics"."autogen"."endpoint_config" WHERE booting_mst <> 0 and ipendpoint=\'endpoint1.ms-A.ict-flame.eu\'',
+         {"time": "1970-01-01T00:00:00Z", "booting_mst": 9.6}),
+        ('SELECT mean(connecting_mst) as "connecting_mst" FROM "CLMCMetrics"."autogen"."endpoint_config" WHERE connecting_mst <> 0 and ipendpoint=\'endpoint1.ms-A.ict-flame.eu\'',
+         {"time": "1970-01-01T00:00:00Z", "connecting_mst":  10.2}),
+        ('SELECT mean(connected_mst) as "connected_mst" FROM "CLMCMetrics"."autogen"."endpoint_config" WHERE connected_mst <> 0 and ipendpoint=\'endpoint1.ms-A.ict-flame.eu\'',
+         {"time": "1970-01-01T00:00:00Z", "connected_mst": 3605.0}),
+        ('SELECT mean(placing_mst) as "placing_mst" FROM "CLMCMetrics"."autogen"."endpoint_config" WHERE placing_mst <> 0 and ipendpoint=\'endpoint2.ms-A.ict-flame.eu\'',
+         {"time": "1970-01-01T00:00:00Z", "placing_mst": 9.4}),
+        ('SELECT mean(booting_mst) as "booting_mst" FROM "CLMCMetrics"."autogen"."endpoint_config" WHERE booting_mst <> 0 and ipendpoint=\'endpoint2.ms-A.ict-flame.eu\'',
+         {"time": "1970-01-01T00:00:00Z", "booting_mst": 9.6}),
+        ('SELECT mean(connecting_mst) as "connecting_mst" FROM "CLMCMetrics"."autogen"."endpoint_config" WHERE connecting_mst <> 0 and ipendpoint=\'endpoint2.ms-A.ict-flame.eu\'',
+         {"time": "1970-01-01T00:00:00Z", "connecting_mst": 10.2}),
+        ('SELECT mean(connected_mst) as "connected_mst" FROM "CLMCMetrics"."autogen"."endpoint_config" WHERE connected_mst <> 0 and ipendpoint=\'endpoint2.ms-A.ict-flame.eu\'',
+         {"time": "1970-01-01T00:00:00Z", "connected_mst": 3605.0}),
+
+        ('SELECT mean(stopped_sum) as "stopped_sum" FROM "CLMCMetrics"."autogen"."mpegdash_mc_config" WHERE stopped_sum <> 0',
+         {"time": "1970-01-01T00:00:00Z", "stopped_sum": 0.2}),
+        ('SELECT mean(stopped_mst) as "stopped_mst" FROM "CLMCMetrics"."autogen"."mpegdash_mc_config" WHERE stopped_mst <> 0',
+         {"time": "1970-01-01T00:00:00Z", "stopped_mst": 0.2}),
+        ('SELECT mean(starting_sum) as "starting_sum" FROM "CLMCMetrics"."autogen"."mpegdash_mc_config" WHERE starting_sum <> 0',
+         {"time": "1970-01-01T00:00:00Z", "starting_sum": 5.5}),
+        ('SELECT mean(starting_mst) as "starting_mst" FROM "CLMCMetrics"."autogen"."mpegdash_mc_config" WHERE starting_mst <> 0',
+         {"time": "1970-01-01T00:00:00Z", "starting_mst": 5.5}),
+        ('SELECT mean(running_sum) as "running_sum" FROM "CLMCMetrics"."autogen"."mpegdash_mc_config" WHERE running_sum <> 0',
+         {"time": "1970-01-01T00:00:00Z", "running_sum": 3602.1000000000004}),
+        ('SELECT mean(running_mst) as "running_mst" FROM "CLMCMetrics"."autogen"."mpegdash_mc_config" WHERE running_mst <> 0',
+         {"time": "1970-01-01T00:00:00Z", "running_mst": 3602.1000000000004}),
+        ('SELECT mean(stopping_sum) as "stopping_sum" FROM "CLMCMetrics"."autogen"."mpegdash_mc_config" WHERE stopping_sum <> 0',
+         {"time": "1970-01-01T00:00:00Z", "stopping_sum": 1.1}),
+        ('SELECT mean(stopping_mst) as "stopping_mst" FROM "CLMCMetrics"."autogen"."mpegdash_mc_config" WHERE stopping_mst <> 0',
+         {"time": "1970-01-01T00:00:00Z", "stopping_mst": 1.1}),
     ])
-    def test_simulation( self, run_simulator, influx_db, query, expected_result ):
+    
+    def test_simulation(self, influx_db, query, expected_result):
         """
         This is the entry point of the test. This method will be found and executed when the module is ran using pytest
 
         :param query: the query to execute (value obtained from the pytest parameter decorator)
         :param expected_result: the result expected from executing the query (value obtained from the pytest parameter decorator)
         :param influx_db the import db client fixture - imported from contest.py
-        :param run_simulation_fixture: the imported fixture to use to generate the testing data - the return value of the fixture is not needed in this case
         """
 
         # pytest automatically goes through all queries under test, declared in the parameters decorator
@@ -51,7 +118,7 @@ class TestSimulation(object):
         # test the error attribute of the result is None, that is no error is returned from executing the DB query
         assert query_result.error is None, "An error was encountered while executing query {0}.".format(query)
 
-        # get the dictionary of result points; the next() function just gets the first element of the query results iterator (we only expect one item in the iterator)
+        # get the dictionary of result points; the next() function just gets the first element of the query results generator (we only expect one item in the generator)
         actual_result = next(query_result.get_points())
 
         assert expected_result == actual_result, "Simulation test failure"

clmctest/scripts/rspec.yml

+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          20-03-2018
+##      Created for Project :   FLAME
+
 hosts:
   - name: test-runner
     cpus: 1

clmctest/scripts/test_config_telegraf.py

 #!/usr/bin/python3
+"""
+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          20-03-2018
+##      Created for Project :   FLAME
+"""
 
 import pytest
 import subprocess

clmctest/services/apache/telegraf_apache.conf

+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Rowan Powell
+##      Created Date :          27-02-2018
+##      Created for Project :   FLAME
+
 [[inputs.apache]]
   ## An array of URLs to gather from, must be directed at the machine
   ## readable version of the mod_status page including the auto query string.

clmctest/services/ffmpeg/install.sh

 #!/bin/bash
+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          09-02-2018
+##      Created for Project :   FLAME
 
 apt-get update
 apt-get -y install zip python

clmctest/services/ffmpeg/telegraf_ffmpeg.conf

+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          08-03-2018
+##      Created for Project :   FLAME
+
 # # Influx HTTP write listener
 [[inputs.http_listener]]
   ## Address and port to host HTTP listener on

clmctest/services/ffmpeg/transcode.sh

 #!/bin/bash
+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          09-02-2018
+##      Created for Project :   FLAME
+
 #
 # Bento4 must be manually obtained from https://www.bento4.com/downloads/ ffmpeg can be installed as follows 
 # (Debian - http://www.deb-multimedia.org/) ~$ 

clmctest/services/host/telegraf_host.conf

+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          20-03-2018
+##      Created for Project :   FLAME
 
 ###############################################################################
 #                                  INPUTS                                     #

clmctest/services/ipendpoint/telegraf_ipendpoint.conf

+## © University of Southampton IT Innovation Centre, 2018
+##
+## Copyright in this software belongs to University of Southampton
+## IT Innovation Centre of Gamma House, Enterprise Road, 
+## Chilworth Science Park, Southampton, SO16 7NS, UK.
+##
+## This software may not be used, sold, licensed, transferred, copied
+## or reproduced in whole or in part in any manner or form or in or
+## on any media by any person other than in accordance with the terms
+## of the Licence Agreement supplied with the software, or otherwise
+## without the prior written consent of the copyright owners.
+##
+## This software is distributed WITHOUT ANY WARRANTY, without even the
+## implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+## PURPOSE, except where stated in the Licence Agreement supplied with
+## the software.
+##
+##      Created By :            Michael Boniface
+##      Created Date :          08-03-2018
+##      Created for Project :   FLAME
+
 # # Influx HTTP write listener
 [[inputs.http_listener]]
   ## Address and port to host HTTP listener on