Merge branch 'clmc-service' into 'integration'

Clmc service - updated alerts API

See merge request FLAME/consortium/3rdparties/flame-clmc!65

docs/clmc-service.md

@@ -62,41 +62,6 @@ with **/clmc-service** so that the nginx reverse proxy server (listening on port
 
 ## Alerts API Endpoints
 
-* **GET** ***/alerts?sfc={service function chain id}&sfci={service function chain instance id}&policy={policy id}&trigger={trigger id}***
-
-    This API method can be used to retrieve the generated alert task and alert topic identifiers during the processing of an alerts specification document.
-    These identifiers can then be used to interact with the Kapacitor HTTP API for further configiuration or modification of alerts - https://docs.influxdata.com/kapacitor/v1.4/working/api/.
-
-    * Request:
-    
-        Expects a URL query string with the request parameters - **sfc**, **sfci**, **policy** and **trigger**. The given parameters must match the values used in the alerts specification
-        document. Otherwise, a wrong ID will be returned.
-        
-    * Request URL Examples:
-    
-        **/alerts?sfc=MSDemo&sfci=MSDemo-premium&policy=requests_diff&trigger=low_requests**
-        
-        **/alerts?sfc=SimpleMediaService&sfci=SimpleMediaService-1&policy=rtt_deviation&trigger=increase_in_rtt**
-     
-    * Response
-    
-        The response of this request is a JSON-formatted content, which contains the task and topic identifiers, along with the Kapacitor
-        API endpoints to use for configuring the given task, topic and the respective handlers.
-        
-        Returns a 400 Bad Request if the URL query string parameters are invalid or otherwise incorrect.
-        
-    * Response Body Example:
-    
-        ```json
-        {
-            "task_identifier": "094f23d6e948c78e9fa215528973fb3aeefa5525898626c9ea049dc8e87a7388",
-            "topic_identifier": "094f23d6e948c78e9fa215528973fb3aeefa5525898626c9ea049dc8e87a7388",
-            "task_api_endpoint": "/kapacitor/v1/tasks/094f23d6e948c78e9fa215528973fb3aeefa5525898626c9ea049dc8e87a7388",
-            "topic_api_endpoint": "/kapacitor/v1/alerts/topics/094f23d6e948c78e9fa215528973fb3aeefa5525898626c9ea049dc8e87a7388",
-            "topic_handlers_api_endpoint": "/kapacitor/v1/alerts/topics/094f23d6e948c78e9fa215528973fb3aeefa5525898626c9ea049dc8e87a7388/handlers"
-        }
-        ```
-
 * **POST** ***/alerts***
 
     This API method can be used to send an alert specification document, which is then used by the CLMC service to create
@@ -179,6 +144,139 @@ with **/clmc-service** so that the nginx reverse proxy server (listening on port
         }
         ```
 
+* **PUT** ***/alerts***
+
+    This API method can be used to send an alert specification document, which is then used by the CLMC service to create or update
+    alert tasks and subscribe alert handlers to those tasks in Kapacitor. For further information on the alert specification
+    document, please check the [CLMC Alert Specification Documentation](AlertsSpecification.md).
+    
+    The request/response format of this method is the same as the **POST /alerts** API endpoint with the only difference being that existing alert tasks
+    or handlers will be re-created rather than returning a duplication error.
+
+* **GET** ***/alerts/{sfc_id}/{sfc_instance_id}***
+
+    This API method can be used to fetch all alerts that are registered for a specific service function chain instance identified
+    by the ***sfc_id*** and ***sfc_instance_id*** url parameters.
+    
+    * Request:
+    
+        Expects a URL query with the following parameters - **sfc_id**, **sfc_instance_id**. The given parameters should uniquely
+        identify a service function chain instance for which the registered alerts information will be returned.
+        
+    * Request URL Examples:
+    
+        **/alerts/MSDemo/MSDemo_1**
+        
+        **/alerts/SimpleMediaService/SimpleMediaService_1**
+    
+    * Response:
+    
+        The response of this request is a JSON-formatted content, which contains a list of alert objects each of which represents
+        a registered alert and contains the alert policy ID, trigger ID and Kapacitor resources including handlers.
+    
+    * Response Example:
+        
+        ```json
+        [
+            {
+                "policy": "<TOSCA policy ID>",
+                "trigger": "<TOSCA trigger ID>",
+                "handlers": ["<list of URLs that receive alerts through POST requests>"],
+                "task_identifier": "<Kapacitor task identifier>",
+                "topic_identifier": "<Kapacitor topic identifier>",
+                "task_api_endpoint": "/kapacitor/v1/tasks/<Kapacitor task identifier>",
+                "topic_api_endpoint": "/kapacitor/v1/alerts/topics/<Kapacitor topic identifier>",
+                "topic_handlers_api_endpoint": "/kapacitor/v1/alerts/topics/<Kapacitor topic identifier>/handlers"
+            }
+        ]
+        ```
+
+* **DELETE** ***/alerts***
+
+    This API method can be used to send an alert specification document, which is then used by the CLMC service to delete
+    alert tasks and deregister alert handlers in Kapacitor. Essentially, it is a clean-up endpoint for the alerts API. 
+    For further information on the alert specification document, please check the [CLMC Alert Specification Documentation](AlertsSpecification.md).
+
+    * Request:
+    
+        Expects a YAML-formatted file in the request referenced with ID ***alert-spec*** representing the TOSCA alert specification 
+        document. The alert specification document is then parsed with the openstack TOSCA parser (https://github.com/openstack/tosca-parser/tree/master/toscaparser)
+        and validated against the CLMC alerts specification schema (again check [documentation](AlertsSpecification.md) for more info on this).
+
+    * Example for sending a request with curl:
+    
+        `curl -X DELETE -F "alert-spec=@alert-specification.yaml" http://localhost:9080/alerts`
+        
+        where **alert-specification.yaml** is the path to the alerts specification file.
+    
+    * Response:
+    
+        The response of this request is a JSON-formatted content, which contains two lists - one for the alerts that were found in the TOSCA specification
+        and then deleted from Kapacitor, and one for the respective alert handlers that were deleted from Kapacitor.
+                        
+        Returns a 400 Bad Request if the request does not contain a yaml file referenced with ID **alert-spec**.
+        
+        Returns a 400 Bad Request if the alert specification file is not a valid YAML file.
+        
+        Returns a 400 Bad Request if the alert specification file cannot be parsed with the TOSCA parser.
+        
+        Returns a 400 Bad Request if the alert specification file fails validation against the CLMC alerts specification schema.
+    
+    * Response Body Example:
+    
+        ```json
+        {
+          "deleted_alerts": [{"policy": "scale_nginx_policy", "trigger": "high_requests"}, 
+                             {"policy": "scale_nginx_policy", "trigger": "increase_in_active_requests"},
+                             {"policy": "scale_nginx_policy", "trigger": "increase_in_running_processes"}, 
+                             {"policy": "deadman_policy", "trigger": "no_measurements"}], 
+          "deleted_handlers": [{"policy": "scale_nginx_policy", "trigger": "increase_in_active_requests", "handler": "flame_sfemc"},
+                               {"policy": "scale_nginx_policy", "trigger": "increase_in_running_processes", "handler": "flame_sfemc"},
+                               {"policy": "deadman_policy", "trigger": "no_measurements", "handler": "flame_sfemc"},
+                               {"policy": "scale_nginx_policy", "trigger": "high_requests", "handler": "http://172.40.231.200:9999/"},
+                               {"policy": "scale_nginx_policy", "trigger": "increase_in_active_requests", "handler": "http://172.40.231.200:9999/"},
+                               {"policy": "scale_nginx_policy", "trigger": "increase_in_running_processes", "handler": "http://172.40.231.200:9999/"},
+                               {"policy": "deadman_policy", "trigger": "no_measurements", "handler": "http://172.40.231.200:9999/"}]
+        }
+        ```
+
+* **GET** ***/alerts?sfc={service function chain id}&sfci={service function chain instance id}&policy={policy id}&trigger={trigger id}***
+    
+    (Deprecated - please use *GET /alerts/{sfc_id}/{sfc_instance_id}* instead)
+    
+    This API method can be used to retrieve the generated alert task and alert topic identifiers during the processing of an alerts specification document.
+    These identifiers can then be used to interact with the Kapacitor HTTP API for further configiuration or modification of alerts - https://docs.influxdata.com/kapacitor/v1.4/working/api/.
+
+    * Request:
+    
+        Expects a URL query string with the request parameters - **sfc**, **sfci**, **policy** and **trigger**. The given parameters must match the values used in the alerts specification
+        document. Otherwise, a wrong ID will be returned.
+        
+    * Request URL Examples:
+    
+        **/alerts?sfc=MSDemo&sfci=MSDemo-premium&policy=requests_diff&trigger=low_requests**
+        
+        **/alerts?sfc=SimpleMediaService&sfci=SimpleMediaService-1&policy=rtt_deviation&trigger=increase_in_rtt**
+     
+    * Response
+    
+        The response of this request is a JSON-formatted content, which contains the task and topic identifiers, along with the Kapacitor
+        API endpoints to use for configuring the given task, topic and the respective handlers.
+        
+        Returns a 400 Bad Request if the URL query string parameters are invalid or otherwise incorrect.
+        
+    * Response Body Example:
+    
+        ```json
+        {
+            "task_identifier": "094f23d6e948c78e9fa215528973fb3aeefa5525898626c9ea049dc8e87a7388",
+            "topic_identifier": "094f23d6e948c78e9fa215528973fb3aeefa5525898626c9ea049dc8e87a7388",
+            "task_api_endpoint": "/kapacitor/v1/tasks/094f23d6e948c78e9fa215528973fb3aeefa5525898626c9ea049dc8e87a7388",
+            "topic_api_endpoint": "/kapacitor/v1/alerts/topics/094f23d6e948c78e9fa215528973fb3aeefa5525898626c9ea049dc8e87a7388",
+            "topic_handlers_api_endpoint": "/kapacitor/v1/alerts/topics/094f23d6e948c78e9fa215528973fb3aeefa5525898626c9ea049dc8e87a7388/handlers"
+        }
+        ```
+
 ## Graph API Endpoints
 
 * **Assumptions**

src/service/clmcservice/__init__.py

@@ -76,6 +76,7 @@ def main(global_config, **settings):
 
     # add routes of the Alerts Configuration API
     config.add_route('alerts_configuration', '/alerts')
+    config.add_route('alerts_configuration_instance', '/alerts/{sfc_id}/{sfc_instance_id}')
 
-    config.scan()  # This method scans the packages and finds any views related to the routes added in the app configuration
+    config.scan()  # this method scans the packages and finds any views related to the routes added in the app configuration
     return config.make_wsgi_app()

src/test/clmctest/alerts/conftest.py

@@ -31,13 +31,9 @@ from shutil import rmtree
 from signal import SIGKILL
 from json import load
 from pkg_resources import resource_filename
-from requests import delete, get
 from clmctest.alerts.alert_handler_server import LOG_TEST_FOLDER_PATH
 
 
-KAPACITOR_PORT = 9092
-
-
 @fixture(scope="module")
 def rspec_config():
     """
@@ -55,23 +51,11 @@ def rspec_config():
 
 
 @fixture(scope="module")
-def set_up_tear_down_fixture(rspec_config):
+def set_up_tear_down_fixture():
     """
     Set up/tear down fixture for the alerts integration test.
     """
 
-    global KAPACITOR_PORT
-
-    kapacitor_host = None
-    for host in rspec_config:
-        if host["name"] == "clmc-service":
-            kapacitor_host = host["ip_address"]
-            break
-
-    assert kapacitor_host is not None
-
-    kapacitor_url = "http://{0}:{1}".format(kapacitor_host, KAPACITOR_PORT)
-
     if exists(LOG_TEST_FOLDER_PATH):
         rmtree(LOG_TEST_FOLDER_PATH)  # clean out the log directory
     makedirs(LOG_TEST_FOLDER_PATH)  # create the log directory
@@ -89,19 +73,3 @@ def set_up_tear_down_fixture(rspec_config):
     kill(process_id, SIGKILL)
     if exists(LOG_TEST_FOLDER_PATH):
         rmtree(LOG_TEST_FOLDER_PATH)
-
-    print("Deleting Kapacitor tasks, topics and handlers that were created for this test...")
-    # get all tasks from kapacitor (that were created in this test) and delete them
-    kapacitor_tasks = get("{0}/kapacitor/v1/tasks".format(kapacitor_url)).json()["tasks"]
-    kapacitor_task_links = [task["link"]["href"] for task in kapacitor_tasks]
-    for task_link in kapacitor_task_links:
-        delete("{0}{1}".format(kapacitor_url, task_link))
-
-    # get all topics and handlers from kapacitor (that were created in this test) and delete them
-    kapacitor_topics = get("{0}/kapacitor/v1/alerts/topics".format(kapacitor_url)).json()["topics"]
-    for topic in kapacitor_topics:
-        topic_handlers = get("{0}{1}".format(kapacitor_url, topic["handlers-link"]["href"])).json()["handlers"]
-        for handler in topic_handlers:
-            delete("{0}{1}".format(kapacitor_url, handler["link"]["href"]))
-
-        delete("{0}{1}".format(kapacitor_url, topic["link"]["href"]))

src/test/clmctest/alerts/test_alerts.py

@@ -21,9 +21,9 @@
 ##      Created Date :          22-08-2018
 ##      Created for Project :   FLAME
 """
-
+import datetime
 from time import sleep, strptime
-from requests import post, get
+from requests import post, get, delete, put
 from os import listdir
 from os.path import join, dirname
 from json import load
@@ -84,11 +84,14 @@ class TestAlerts(object):
     def test_alert_triggers(self, rspec_config, set_up_tear_down_fixture):
         """
         Test is implemented using the following steps:
-            * Send clmc service a TOSCA alert spec. file
-            * Wait 15 seconds for Kapacitor to configure and start executing the defined tasks
+            * Send to clmc service a POST request with TOSCA alert spec. and resource spec. files
+            * Check that the registered alerts can be fetched with a GET request
+            * Wait 10 seconds for Kapacitor to configure and start executing the defined tasks
             * Send some test requests to nginx to increase the load
-            * Wait 20 seconds for alerts to be triggered
+            * Wait 15 seconds for alerts to be triggered
             * Check that 4 log files have been created - one for each alert defined in the alert spec.
+            * Send to clmc service a DELETE request with TOSCA alert spec. file
+            * Check that the returned lists of deleted handlers and alerts are correct
 
         :param rspec_config: fixture from conftest.py
         """
@@ -105,6 +108,7 @@ class TestAlerts(object):
             if clmc_service_host is not None and nginx_host is not None:
                 break
 
+        # create the alerts with a POST request
         print("Sending alerts specification to clmc service...")
         alerts_spec = join(dirname(__file__), "alerts_test_config.yaml")
         resources_spec = join(dirname(__file__), "resources_test_config.yaml")
@@ -119,8 +123,50 @@ class TestAlerts(object):
         assert "triggers_specification_errors" not in clmc_service_response, "Unexpected error was returned for triggers specification"
         assert "triggers_action_errors" not in clmc_service_response, "Unexpected error was returned for handlers specification"
 
+        sfc, sfc_instance = "MS_Template_1", "MS_Template_1_1"
+        assert (sfc, sfc_instance) == (clmc_service_response["service_function_chain_id"], clmc_service_response["service_function_chain_instance_id"])
         print("Alert spec sent successfully")
 
+        # check that the alerts can be fetched with a GET request
+        print("Validate that the alerts were registered and can be fetched with a GET request.")
+        response = get("http://{0}/clmc-service/alerts/{1}/{2}".format(clmc_service_host, sfc, sfc_instance))
+        assert response.status_code == 200
+        clmc_service_response = response.json()
+        clmc_service_response = sorted(clmc_service_response, key=lambda x: x["trigger"])  # sort by trigger so that the response can be compared to what's expected
+
+        # sort the handlers of returned alerts to ensure comparison order is correct
+        for alert in clmc_service_response:
+            alert["handlers"] = sorted(alert["handlers"])
+
+        # compare actual response with expected response
+        assert clmc_service_response == [
+            {"policy": "scale_nginx_policy", "trigger": "high_requests", "task_identifier": "46fb8800c8a5eeeb04b090d838d475df574a2e6d854b5d678fc981c096eb6c1b",
+             "handlers": ["http://172.40.231.200:9999/"],
+             "topic_identifier": "46fb8800c8a5eeeb04b090d838d475df574a2e6d854b5d678fc981c096eb6c1b",
+             "task_api_endpoint": "/kapacitor/v1/tasks/46fb8800c8a5eeeb04b090d838d475df574a2e6d854b5d678fc981c096eb6c1b",
+             "topic_api_endpoint": "/kapacitor/v1/alerts/topics/46fb8800c8a5eeeb04b090d838d475df574a2e6d854b5d678fc981c096eb6c1b",
+             "topic_handlers_api_endpoint": "/kapacitor/v1/alerts/topics/46fb8800c8a5eeeb04b090d838d475df574a2e6d854b5d678fc981c096eb6c1b/handlers"},
+            {"policy": "scale_nginx_policy", "trigger": "increase_in_active_requests", "task_identifier": "7a9867f9270dba6635ac3760a3b70bc929f5bd0f3bf582e45d27fbd437f528ca",
+             "handlers": ["flame_sfemc", "http://172.40.231.200:9999/"],
+             "topic_identifier": "7a9867f9270dba6635ac3760a3b70bc929f5bd0f3bf582e45d27fbd437f528ca",
+             "task_api_endpoint": "/kapacitor/v1/tasks/7a9867f9270dba6635ac3760a3b70bc929f5bd0f3bf582e45d27fbd437f528ca",
+             "topic_api_endpoint": "/kapacitor/v1/alerts/topics/7a9867f9270dba6635ac3760a3b70bc929f5bd0f3bf582e45d27fbd437f528ca",
+             "topic_handlers_api_endpoint": "/kapacitor/v1/alerts/topics/7a9867f9270dba6635ac3760a3b70bc929f5bd0f3bf582e45d27fbd437f528ca/handlers"},
+            {"policy": "scale_nginx_policy", "trigger": "increase_in_running_processes", "task_identifier": "f5edaeb27fb847116be749c3815d240cbf0d7ba79aee1959daf0b3445a70f2c8",
+             "handlers": ["flame_sfemc", "http://172.40.231.200:9999/"],
+             "topic_identifier": "f5edaeb27fb847116be749c3815d240cbf0d7ba79aee1959daf0b3445a70f2c8",
+             "task_api_endpoint": "/kapacitor/v1/tasks/f5edaeb27fb847116be749c3815d240cbf0d7ba79aee1959daf0b3445a70f2c8",
+             "topic_api_endpoint": "/kapacitor/v1/alerts/topics/f5edaeb27fb847116be749c3815d240cbf0d7ba79aee1959daf0b3445a70f2c8",
+             "topic_handlers_api_endpoint": "/kapacitor/v1/alerts/topics/f5edaeb27fb847116be749c3815d240cbf0d7ba79aee1959daf0b3445a70f2c8/handlers"},
+            {"policy": "deadman_policy", "trigger": "no_measurements", "task_identifier": "f7dab6fd53001c812d44533d3bbb6ef45f0d1d39b9441bc3c60402ebda85d320",
+             "handlers": ["flame_sfemc", "http://172.40.231.200:9999/"],
+             "topic_identifier": "f7dab6fd53001c812d44533d3bbb6ef45f0d1d39b9441bc3c60402ebda85d320",
+             "task_api_endpoint": "/kapacitor/v1/tasks/f7dab6fd53001c812d44533d3bbb6ef45f0d1d39b9441bc3c60402ebda85d320",
+             "topic_api_endpoint": "/kapacitor/v1/alerts/topics/f7dab6fd53001c812d44533d3bbb6ef45f0d1d39b9441bc3c60402ebda85d320",
+             "topic_handlers_api_endpoint": "/kapacitor/v1/alerts/topics/f7dab6fd53001c812d44533d3bbb6ef45f0d1d39b9441bc3c60402ebda85d320/handlers"}
+        ], "Incorrect response for GET alerts request"
+        print("Alert spec validated successfully")
+
         print("Wait 10 seconds for Kapacitor stream/batch tasks to start working...")
         sleep(10)
 
@@ -136,6 +182,7 @@ class TestAlerts(object):
         alert_logs = listdir(LOG_TEST_FOLDER_PATH)
         assert len(alert_logs) == 4, "4 log files must have been created - one for each alert defined in the specification."
 
+        # check the content of each log file
         for alert_log in alert_logs:
             alert_log_path = join(LOG_TEST_FOLDER_PATH, alert_log)
 
@@ -149,3 +196,116 @@ class TestAlerts(object):
                 valid = False
 
             assert valid, "Alert log content is invalid - {0}".format(alert_log_path)
+
+        # delete the alerts with a DELETE request
+        with open(alerts_spec, 'rb') as alerts:
+            files = {'alert-spec': alerts}
+            response = delete("http://{0}/clmc-service/alerts".format(clmc_service_host), files=files)
+
+        assert response.status_code == 200, "Incorrect status code returned after deleting the alert specification"
+
+        json_response = response.json()
+        # sort by trigger to ensure comparison order is correct
+        assert sorted(json_response["deleted_alerts"], key=lambda x: x['trigger']) == [{"policy": "scale_nginx_policy", "trigger": "high_requests"}, {"policy": "scale_nginx_policy", "trigger": "increase_in_active_requests"},
+                                                                                       {"policy": "scale_nginx_policy", "trigger": "increase_in_running_processes"}, {"policy": "deadman_policy", "trigger": "no_measurements"}], \
+            "Incorrect list of deleted alerts"
+        # sort by handler and trigger to ensure comparison order is correct
+        assert sorted(json_response["deleted_handlers"], key=lambda x: (x['handler'], x['trigger'])) == [{"policy": "scale_nginx_policy", "trigger": "increase_in_active_requests", "handler": "flame_sfemc"},
+                                                                                                         {"policy": "scale_nginx_policy", "trigger": "increase_in_running_processes", "handler": "flame_sfemc"},
+                                                                                                         {"policy": "deadman_policy", "trigger": "no_measurements", "handler": "flame_sfemc"},
+                                                                                                         {"policy": "scale_nginx_policy", "trigger": "high_requests", "handler": "http://172.40.231.200:9999/"},
+                                                                                                         {"policy": "scale_nginx_policy", "trigger": "increase_in_active_requests", "handler": "http://172.40.231.200:9999/"},
+                                                                                                         {"policy": "scale_nginx_policy", "trigger": "increase_in_running_processes", "handler": "http://172.40.231.200:9999/"},
+                                                                                                         {"policy": "deadman_policy", "trigger": "no_measurements", "handler": "http://172.40.231.200:9999/"}], \
+            "Incorrect list of deleted handlers"
+
+    def test_alerts_update_request(self, rspec_config):
+        """
+        Test is implemented using the following steps:
+            * Send to clmc service a POST request with TOSCA alert spec. and resource spec. files
+            * Send to clmc service a PUT request with TOSCA alert spec. and resource spec. files
+            * Check that the alerts have a "created" timestamp that is later than the timestamp of the alerts during the POST request,
+                implying that the alerts were re-created during the PUT request
+
+        :param rspec_config: fixture from conftest.py
+        """
+
+        clmc_service_host = None
+        for host in rspec_config:
+            if host["name"] == "clmc-service":
+                clmc_service_host = host["ip_address"]
+                break
+
+        # create the alerts with a POST request
+        print("Sending alerts specification to clmc service...")
+        alerts_spec = join(dirname(__file__), "alerts_test_config.yaml")
+        resources_spec = join(dirname(__file__), "resources_test_config.yaml")
+
+        with open(alerts_spec, 'rb') as alerts:
+            with open(resources_spec, 'rb') as resources:
+                files = {'alert-spec': alerts, 'resource-spec': resources}
+                response = post("http://{0}/clmc-service/alerts".format(clmc_service_host), files=files)
+        assert response.status_code == 200
+        clmc_service_response = response.json()
+        assert "triggers_specification_errors" not in clmc_service_response, "Unexpected error was returned for triggers specification"
+        assert "triggers_action_errors" not in clmc_service_response, "Unexpected error was returned for handlers specification"
+        sfc, sfc_instance = "MS_Template_1", "MS_Template_1_1"
+        assert (sfc, sfc_instance) == (clmc_service_response["service_function_chain_id"], clmc_service_response["service_function_chain_instance_id"])
+        print("Alert spec sent successfully")
+
+        # find the latest timestamp of the registered alerts
+        max_post_timestamp = 0
+        tasks = get("http://{0}/kapacitor/v1/tasks".format(clmc_service_host)).json()["tasks"]
+        for timestamp in tasks_timestamps(tasks, sfc, sfc_instance):
+            max_post_timestamp = max(max_post_timestamp, timestamp)
+
+        delay = 2  # seconds
+        print("Sleeping {0} seconds to ensure a difference between the timestamps when creating the alerts and when updating them...".format(delay))
+        sleep(delay)
+
+        # update the alerts with a PUT request and check that the "created" metadata is updated implying that the alerts were recreated
+        print("Sending alerts specification to clmc service for updating...")
+        with open(alerts_spec, 'rb') as alerts:
+            with open(resources_spec, 'rb') as resources:
+                files = {'alert-spec': alerts, 'resource-spec': resources}
+                response = put("http://{0}/clmc-service/alerts".format(clmc_service_host), files=files)
+        assert response.status_code == 200
+        clmc_service_response = response.json()
+        assert "triggers_specification_errors" not in clmc_service_response, "Unexpected error was returned for triggers specification"
+        assert "triggers_action_errors" not in clmc_service_response, "Unexpected error was returned for handlers specification"
+        sfc, sfc_instance = "MS_Template_1", "MS_Template_1_1"
+        assert (sfc, sfc_instance) == (clmc_service_response["service_function_chain_id"], clmc_service_response["service_function_chain_instance_id"])
+        print("Alert spec updated successfully")
+
+        # find the earliest timestamp of the updated alerts
+        min_put_timestamp = float("inf")
+        tasks = get("http://{0}/kapacitor/v1/tasks".format(clmc_service_host)).json()["tasks"]
+        for timestamp in tasks_timestamps(tasks, sfc, sfc_instance):
+            min_put_timestamp = min(min_put_timestamp, timestamp)
+
+        print("Latest timestamp during the POST request", max_post_timestamp, "Earliest timestamp during the PUT request", min_put_timestamp)
+        assert min_put_timestamp - max_post_timestamp >= delay, "There is an alert that wasn't updated properly with a PUT request"
+
+        # delete the alerts with a DELETE request
+        with open(alerts_spec, 'rb') as alerts:
+            files = {'alert-spec': alerts}
+            delete("http://{0}/clmc-service/alerts".format(clmc_service_host), files=files)
+
+
+def tasks_timestamps(all_tasks, sfc_id, sfc_instance_id):
+    """
+    Generates the timestamps for the tasks related to the given SFC and SFC instance.
+
+    :param all_tasks: the full list of tasks from kapacitor
+    :param sfc_id: SFC identifier
+    :param sfc_instance_id: SFC instance identifier
+    """
+
+    for task in all_tasks:
+        # get the configured variables of this alert
+        task_config = task["vars"]
+        # if configured for this SFC instance
+        if task_config["sfc"]["value"] == sfc_id and task_config["sfci"]["value"] == sfc_instance_id:
+            created_datestr = task["created"][:26]  # ignore the timezone and only take the first 6 digits of the microseconds
+            task_created_timestamp = datetime.datetime.strptime(created_datestr, "%Y-%m-%dT%H:%M:%S.%f")
+            yield task_created_timestamp.timestamp()