Merge branch 'master' into dev

README.md

 # PEDASI v0.1.1
 
+[![DOI](https://zenodo.org/badge/144545470.svg)](https://zenodo.org/badge/latestdoi/144545470)
+[![Documentation Status](https://readthedocs.org/projects/pedasi/badge/?version=stable)](https://pedasi.readthedocs.io/en/stable/?badge=stable)
+
 Developed as a platform and a service to explore research challenges in data security, privacy, and ethics, PEDASI enables providers of data - particularly [Internet of Things](https://en.wikipedia.org/wiki/Internet_of_things) data - to share their data securely within a common catalogue for use by application developers and researchers. Data can either be hosted and made accessible directly within PEDASI as an internal data source, or hosted elsewhere and accessible as an external data source through PEDASI.
 
 An initial deployment of the platform is available at https://dev.iotobservatory.io.

api/views/datasources.py

@@ -14,6 +14,7 @@ from requests.exceptions import HTTPError
 
 from .. import permissions
 from datasources import models, serializers
+from datasources.connectors.base import DatasetNotFoundError
 from provenance import models as prov_models
 
 
@@ -104,7 +105,8 @@ class DataSourceApiViewset(viewsets.ReadOnlyModelViewSet):
         """
         Attempt to pass a response from the data connector using the function `map_response`.
 
-        If the data connectors raises an error (AttributeError or NotImplementedError) then return an error response.
+        If the data connectors raises an error (AttributeError, DatasetNotFoundError or NotImplementedError)
+        then return an error response.
 
         :param map_response: Function to get response from data connector - must return HttpResponse
         :param error_message: Error message in case data connector raises an error
@@ -120,8 +122,16 @@ class DataSourceApiViewset(viewsets.ReadOnlyModelViewSet):
                 params = None
 
             if dataset is not None:
+                try:
                     data_connector = data_connector[dataset]
 
+                except DatasetNotFoundError:
+                    data = {
+                        'status': 'error',
+                        'message': 'Dataset does not exist within this data source'
+                    }
+                    return response.Response(data, status=404)
+
             # Record this action in PROV
             if not instance.prov_exempt:
                 self._create_prov_entry(instance)

datasources/connectors/base.py

@@ -16,6 +16,13 @@ import requests.auth
 from core import plugin
 
 
+class DatasetNotFoundError(Exception):
+    """
+    Exception raised when a requested dataset cannot be found within a data source.
+    """
+    pass
+
+
 @enum.unique
 class AuthMethod(enum.IntEnum):
     """
@@ -232,6 +239,13 @@ class DataCatalogueConnector(BaseDataConnector, collections_abc.Mapping):
 
     @abc.abstractmethod
     def __getitem__(self, item: str) -> BaseDataConnector:
+        """
+        Get a data connector for a single dataset within this catalogue.
+
+        :param item: Dataset id
+        :return: Data connector for dataset
+        :raises DatasetNotFoundError: Requested dataset cannot be found
+        """
         raise NotImplementedError
 
     def __iter__(self):

datasources/connectors/hypercat.py

@@ -4,7 +4,7 @@ This module contains data connector classes for retrieving data from HyperCat ca
 
 import typing
 
-from .base import BaseDataConnector, DataCatalogueConnector, DataSetConnector
+from .base import BaseDataConnector, DataCatalogueConnector, DataSetConnector, DatasetNotFoundError
 
 
 class HyperCat(DataCatalogueConnector):
@@ -27,6 +27,7 @@ class HyperCat(DataCatalogueConnector):
 
         response = self._get_response(params)
 
+        try:
             dataset_item = self._get_item_by_key_value(
                 response['items'],
                 'href',
@@ -34,6 +35,11 @@ class HyperCat(DataCatalogueConnector):
             )
             metadata = dataset_item['item-metadata']
 
+        except KeyError as e:
+            raise DatasetNotFoundError(
+                'Dataset {0} could not be found'.format(item)
+            ) from e
+
         try:
             try:
                 content_type = self._get_item_by_key_value(

docs/source/guide_administrator.rst

@@ -80,6 +80,7 @@ It it necessary to provide some configuration before deploying PEDASI.
       :caption: deploy/.env
 
       SECRET_KEY=<random string>
+      ALLOWED_HOSTS=hostname.domain
 
       SOCIAL_AUTH_GOOGLE_OAUTH2_KEY=<Google OAuth2 key>
       SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET=<Google OAuth2 secret>
@@ -88,17 +89,14 @@ It it necessary to provide some configuration before deploying PEDASI.
 Deployment
 ^^^^^^^^^^
 
-You may now deploy PEDASI using the Ansible provisioning script. If you have set up your Ubuntu instance to use SSH passwordless access, do the following:
+You may now deploy PEDASI using the Ansible provisioning script. If you have set up your Ubuntu instance to use SSH passwordless access, with passwordless sudo, do the following:
 
 .. code-block:: console
 
    $ ansible-playbook -v -i inventory.yml playbook.yml -u <your username on the remote host>
 
-Otherwise, you will need Ansible to prompt for passwords for the remote user and superuser accounts:
-
-.. code-block:: console
-
-   $ ansible-playbook -v -i inventory.yml playbook.yml -u <your username on the remote host> -k -K
+If you need to provide a password to log in via SSH, add a `-k` flag.
+If you need to provide a password to sudo, add a `-K` flag.
 
 
 Create Administrator Account

docs/source/guide_provider.rst

@@ -22,7 +22,7 @@ Data Providers: Managing Data Sources
 
 In order for users to begin using PEDASI, you should provide access to a range of data sources. The following sections will walk you through adding and managing your first data source. We will use the IoTUK Nation Database API (see https://iotuk.org.uk/iotuk-nation-database-api/) as a basic example.
 
-If you are not a Central Administrator or don't have Data Provider privileges associated with your account, you'll need to obtain these first. Contact the Central Administrator to grant these privileges for your account.
+If you are not a Central Administrator or don't have Data Provider privileges associated with your account, you'll need to obtain these first. Contact the PEDASI Central Administrator to grant these privileges for your account.
 
 
 Adding a Data Source
@@ -116,7 +116,7 @@ Application Providers: Managing Applications
 
 In order for a developer to access PEDASI's capabilities within their application, their application needs to be first registered within PEDASI in order to obtain an API key they can use to authenticate with PEDASI. The following sections will walk you through adding and managing your first application. We will use the IoTUK Nation Map Demo application (see https://github.com/Southampton-RSG/app-iotorgs-map) as a basic example.
 
-If you are not a Central Administrator or don't have Application Provider privileges associated with your account, you'll need to obtain these first. Contact the Central Administrator to grant these privileges for your account.
+If you are not a Central Administrator or don't have Application Provider privileges associated with your account, you'll need to obtain these first. Contact the Central Administrator to request these privileges for your account.
 
 
 Adding an Application
@@ -130,7 +130,7 @@ To add a new application:
     - *Name*: add a full name for the application, e.g. IoTUK Nation Map Demo
     - *Description*: add a brief description of the application, including what it aims to achieve using PEDASI
     - *Url*: specify a public URL for the deployed application itself if it's web-based, or alternatively a source code repository URL if one exists, e.g. https://github.com/Southampton-RSG/app-iotorgs-map
-    - *Access control*: TODO: add in text here, e.g. leave unselected
+    - *Access control*: specify whether the application details are publicly viewable, e.g. leave unselected
 
  3. Select *Create* to register the new application within PEDASI, and you'll be presented with an overview page for that application, with a new API key
 

docs/source/index.rst

@@ -37,14 +37,14 @@ Resources
 
 Documentation is available for the following stakeholders:
 
- - :doc:`Researchers and other end-users<guide_user>`: for users wishing to discover, explore, and make use of datasets using the web interface
+ - :doc:`Researchers and other End-users<guide_user>`: for users wishing to discover, explore, and make use of datasets using the web interface
  - :doc:`Data or Application Providers<guide_provider>`: for those aiming to provide data or use applications through PEDASI
  - :doc:`System Administrators<guide_administrator>`: for those aiming to deploy and manage PEDASI either for development or production
- - :doc:`Application developers<guide_developer>`: for developers wanting to create applications that access data available through PEDASI
+ - :doc:`Application Developers<guide_developer>`: for developers wanting to create applications that access data available through PEDASI
 
 This documentation is also available on `Read the Docs`_.
 
-.. _`Read the Docs`: https://pedasi.readthedocs.io/en/dev
+.. _`Read the Docs`: https://pedasi.readthedocs.io/en/master
 
 
 Licence

docs/source/ref_applications_api.rst

@@ -8,7 +8,7 @@ Applications API Reference
    :caption: Contents:
 
 
-.. note:: Please read the :doc:`Developer Guide<guide_developer>` first before reading this reference.
+.. note:: Please read the :doc:`Developer Guide<guide_developer>` first before reading this reference; it contains examples on how to use the API.
 
 
 Overview
@@ -17,80 +17,609 @@ Overview
 This document provides a schema reference to the PEDASI Applications API which is used by third-party applications to request data, metadata, or provenance records from a PEDASI instance and its data sources.
 
 
-Using the API
--------------
-
-
-API Endpoints
-^^^^^^^^^^^^^
+API Endpoints - General
+-----------------------
 
+--------
 
 GET /api/datasources/
 ^^^^^^^^^^^^^^^^^^^^^
 
-Params:
- TODO
+Implementation notes:
   Retrieves the list of all data sources known to PEDASI, that the authenticated user has the ability to see. This will include some sources which they are unable to use, but have not had their details hidden.
 
+Parameters:
+  None
+
+Response class (Status 200): application/json
+  .. code-block:: json
+
+     [
+       {
+         "id": 0,
+         "name": "string",
+         "description": "string",
+         "url": "string",
+         "plugin_name": "string",
+         "licence": {
+           "name": "string",
+           "short_name": "string",
+           "version": "string",
+           "url": "string"
+         },
+         "is_encrypted": true,
+         "encrypted_docs_url": "string",
+         "metadata_items": [
+           {
+             "field": {
+               "name": "string",
+               "short_name": "string"
+             },
+             "value": "string"
+           }
+         ]
+       }
+     ]
+
+Responses messages:
+  .. list-table::
+     :widths: 16 80 16 16
+     :header-rows: 1
+
+     * - HTTP Status Code
+       - Response
+       - Reason
+       - Response Type
+
+     * - 200
+       - List of data sources
+       - Successful
+       - application/json
 
-GET /api/datasources/<int>/
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
+--------
 
+GET /api/datasources/{datasource_id}/
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Implementation notes:
   Retrieves the PEDASI metadata for a given data source, if the authenticated user has the ability to see it.
 
+Parameters:
+  .. list-table::
+     :widths: 16 80 16
+     :header-rows: 1
+
+     * - Parameter
+       - Description
+       - Type
+
+     * - datasource_id
+       - The numeric id of the data source
+       - integer
+
+Response class (Status 200): application/json
+  .. code-block:: json
+
+     {
+       "id": 0,
+       "name": "string",
+       "description": "string",
+       "url": "string",
+       "plugin_name": "string",
+       "licence": {
+         "name": "string",
+         "short_name": "string",
+         "version": "string",
+         "url": "string"
+       },
+       "is_encrypted": true,
+       "encrypted_docs_url": "string",
+       "metadata_items": [
+         {
+           "field": {
+             "name": "string",
+             "short_name": "string"
+           },
+           "value": "string"
+         }
+       ]
+     }
+
+Responses messages:
+  .. list-table::
+     :widths: 16 80 16 16
+     :header-rows: 1
+
+     * - HTTP Status Code
+       - Response
+       - Reason
+       - Response Type
+
+     * - 200
+       - Single data source
+       - Successful
+       - application/json
+
+     * - 404
+       - .. code-block:: json
+
+            {
+                "detail": "Not found."
+            }
+
+       - Parameter datasource_id was not valid
+       - application/json
 
-GET /api/datasources/<int>/metadata/
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+--------
 
-Params:
-Any supported by data source API
-Retrieves metadata via an API query to a data source. The authenticated user must have permission to use the given data source.
+GET /api/datasources/{datasource_id}/metadata/
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-E.g. A HyperCat catalogue
+Implementation notes:
+  Retrieves metadata for a given data source which may include an API query to the data source, if supported by the data source. The authenticated user must have META permissions or above to access the data source. In the general case, this is implementation-specific for the data connector, in the case of a HyperCat catalogue, it presents the catalogue record for that dataset.
 
+Parameters:
+  .. list-table::
+     :widths: 16 80 16
+     :header-rows: 1
 
-GET /api/datasources/<int>/metadata/<int>/
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-GET /api/datasources/<int>/metadata/<URI>/		(maybe)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+     * - Parameter
+       - Description
+       - Type
+
+     * - datasource_id
+       - The numeric id of the data source
+       - integer
+
+Response class (Status 200): application/json
+  Dependent on data source
+
+Responses messages:
+  .. list-table::
+     :widths: 16 80 16 16
+     :header-rows: 1
+
+     * - HTTP Status Code
+       - Response
+       - Reason
+       - Response Type
+
+     * - 200
+       - Data source metadata
+       - Successful
+       - application/json
+
+     * - 400
+       - .. code-block:: json
+
+            {
+                "status": "error",
+                "message": "Data source does not provide metadata"
+            }
+
+       - Data source/connector does not support metadata queries
+       - application/json
+
+     * - 404
+       - .. code-block:: json
+
+            {
+                "detail": "Not found."
+            }
+
+       - Parameter datasource_id was not valid
+       - application/json
+
+     * - any other
+       - Error response from external data source
+       - An error occured within the external data source
+       - determined by data source
+
+--------
+
+GET /api/datasources/{datasource_id}/data/?{query_string}
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Implementation notes:
+  In the case where a data source represents a single dataset, retrieve the dataset via an API query to the data source using the given query string. The authenticated user must have permission to use the given data source.
+
+Parameters:
+  .. list-table::
+     :widths: 16 80 16
+     :header-rows: 1
+
+     * - Parameter
+       - Description
+       - Type
 
-Retrieves metadata for a single dataset contained within the source via an API query to the data source. The authenticated user must have permission to use the given data source.
+     * - datasource_id
+       - The numeric id of the data source
+       - integer
 
-E.g. An entry within a HyperCat catalogue
+     * - query_string
+       - An ampersand-separated set of key=value pairs
+       - string
 
+Response class (Status 200): type-specific to datasource
+  Dependent on data source
 
-GET /api/datasources/<int>/data/
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Responses messages:
+  .. list-table::
+     :widths: 16 80 16 16
+     :header-rows: 1
 
-Params:
-Any supported by data source API
-In the case where a data source represents a single dataset, retrieve the dataset via an API query to the data source. The authenticated user must have permission to use the given data source.
+     * - HTTP Status Code
+       - Response
+       - Reason
+       - Response Type
 
-If the data source does not represent a single dataset, error.
+     * - 200
+       - Data source data
+       - Successful
+       - determined by data source
 
+     * - 400
+       - .. code-block:: json
 
-GET /api/datasources/<int>/data/<int>/
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-GET /api/datasources/<int>/data/<URI>/			(maybe)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+            {
+                "status": "error",
+                "message": "Data source does not provide data"
+            }
 
-Params:
-In the case where a data source represents multiple datasets, retrieve a single dataset via an API query to the data source. The authenticated user must have permission to use the given data source.
+       - Data source/connector does not support data queries
+       - application/json
 
-If the data source represents a single dataset, error.
+     * - 404
+       - .. code-block:: json
 
+            {
+                "detail": "Not found."
+            }
 
-GET /api/datasources/<int>/prov/
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Params:
-TODO
+       - Parameter datasource_id was not valid
+       - application/json
+
+     * - any other
+       - Error response from external data source
+       - An error occured within the external data source
+       - determined by data source
+
+--------
+
+GET /api/datasources/{datasource_id}/prov/
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Implementation notes:
   Retrieve all PROV records related to a single data source.
 
+Parameters:
+  .. list-table::
+     :widths: 16 80 16
+     :header-rows: 1
+
+     * - Parameter
+       - Description
+       - Type
+
+     * - datasource_id
+       - The numeric id of the data source
+       - integer
+
+Response class (Status 200): application/json
+  .. code-block:: json
+
+     {
+       "prov": [
+         {
+           "_id": {
+             "$oid": "string"
+           },
+           "prefix": {
+             "piot": "string",
+             "foaf": "string"
+           },
+           "entity": {
+             "string": {
+               "prov:type": "string",
+               "xsd:anyURI": "string"
+             }
+           },
+           "activity": {
+             "string": {
+               "prov:startTime": "string",
+               "prov:type": "string"
+             }
+           },
+           "agent": {
+             "string": {
+               "prov:type": "string"
+             },
+             "piot:app-pedasi": {
+               "prov:type": "string",
+               "xsd:anyURI": "string"
+             }
+           },
+           "actedOnBehalfOf": {
+             "_:id1": {
+               "prov:delegate": "string",
+               "prov:responsible": "string",
+               "prov:activity": "string",
+               "prov:type": "string"
+             }
+           }
+         }
+       ]
+     }
+
+Responses messages:
+  .. list-table::
+     :widths: 16 80 16 16
+     :header-rows: 1
+
+     * - HTTP Status Code
+       - Response
+       - Reason
+       - Response Type
+
+     * - 200
+       - Data source PROV records
+       - Successful
+       - application/json
+
+     * - 404
+       - .. code-block:: json
+
+            {
+                "detail": "Not found."
+            }
+
+       - Parameter datasource_id was not valid
+       - application/json
+
+--------
+
+API Endpoints - Catalogues
+--------------------------
+
+Data catalogues are a subset of data sources which contain a number of distinct datasets.
+
+--------
+
+GET /api/datasources/{datasource_id}/datasets/
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Implementation notes:
+  Retrieves the list of dataset IDs within a catalogue, which may be via an API query to the data source, if supported by the data source. The authenticated user must have META permissions or above to access the data source. In the general case, this is implementation-specific for the data connector.
+
+Parameters:
+  .. list-table::
+     :widths: 16 80 16
+     :header-rows: 1
+
+     * - Parameter
+       - Description
+       - Type
+
+     * - datasource_id
+       - The numeric id of the data source
+       - integer
+
+Response class (Status 200): application/json
+  .. code-block:: json
+
+     {
+         "status": "success",
+         "data": [
+             "string"
+         ]
+
+Responses messages:
+  .. list-table::
+     :widths: 16 80 16 16
+     :header-rows: 1
+
+     * - HTTP Status Code
+       - Response
+       - Reason
+       - Response Type
+
+     * - 200
+       - Data source list of datasets
+       - Successful
+       - application/json
+
+     * - 400
+       - .. code-block:: json
+
+            {
+                "status": "error",
+                "message": "Data source does not contain datasets"
+            }
+
+       - Data source/connector is not a catalogue
+       - application/json
 
-GET /api/datasources/<int>/prov/<int>/
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+     * - 404
+       - .. code-block:: json
 
-Retrieve a single PROV record related to a the data source.
+            {
+                "detail": "Not found."
+            }
+
+       - Parameter datasource_id was not valid
+       - application/json
+
+     * - any other
+       - Error response from external data source
+       - An error occured within the external data source
+       - determined by data source
+
+--------
+
+GET /api/datasources/{datasource_id}/datasets/{dataset_id}/metadata/
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Implementation notes:
+  Specific to data sources which purport to act as catalogues of data, it retrieves metadata for a single dataset contained within the given data source in most cases via an API query to the data source (e.g. HyperCat), although is dependent on data connector implementation. The authenticated user must have permission to use the given data source.
+
+Parameters:
+  .. list-table::
+     :widths: 16 80 16
+     :header-rows: 1
+
+     * - Parameter
+       - Description
+       - Type
+
+     * - datasource_id
+       - The numeric id of the data source
+       - integer
+
+     * - dataset_id
+       - The id of the dataset within the specific data source, which may be a URI
+       - string
+
+Response class (Status 200): application/json
+  Dependent on data source
+
+Responses messages:
+  .. list-table::
+     :widths: 16 80 16 16
+     :header-rows: 1
+
+     * - HTTP Status Code
+       - Response
+       - Reason
+       - Response Type
+
+     * - 200
+       - Dataset metadata
+       - Successful
+       - application/json
+
+     * - 400
+       - .. code-block:: json
+
+            {
+                "status": "error",
+                "message": "Data source does not provide metadata"
+            }
+
+       - Data source/connector does not support metadata queries
+       - application/json
+
+     * - 404
+       - .. code-block:: json
+
+            {
+                "detail": "Not found."
+            }
+
+       - Parameter datasource_id was not valid
+       - application/json
+
+     * - 404
+       - .. code-block:: json
+
+            {
+                "status": "error",
+                "message": "Dataset does not exist within this data source"
+            }
+
+       - Parameter dataset_id did not refer to a valid dataset within this data source
+       - application/json
+
+     * - any other
+       - Error response from external data source
+       - An error occured within the external data source
+       - determined by data source
+
+--------
+
+
+GET /api/datasources/{datasource_id}/datasets/{dataset_id}/data/?{query_string}
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Implementation notes:
+  In the case where a data source represents a catalogue of datasets, retrieve a single dataset via an API query to the data source. The authenticated user must have permission to use the given data source.
+
+Parameters:
+  .. list-table::
+     :widths: 16 80 16
+     :header-rows: 1
+
+     * - Parameter
+       - Description
+       - Type
+
+     * - datasource_id
+       - The numeric id of the data source
+       - integer
+
+     * - dataset_id
+       - The id of the dataset within the specific data source, which may be a URI
+       - string
+
+     * - query_string
+       - An ampersand-separated set of key=value pairs
+       - string
+
+Response class (Status 200): type-specific to datasource
+  Dependent on data source
+
+Responses messages:
+  .. list-table::
+     :widths: 16 80 16 16
+     :header-rows: 1
+
+     * - HTTP Status Code
+       - Response
+       - Reason
+       - Response Type
+
+     * - 200
+       - Dataset data
+       - Successful
+       - application/json
+
+     * - 400
+       - .. code-block:: json
+
+            {
+                "status": "error",
+                "message": "Data source does not provide data"
+            }
+
+       - Data source/connector does not support data queries
+       - application/json
+
+     * - 404
+       - .. code-block:: json
+
+            {
+                "detail": "Not found."
+            }
+
+       - Parameter datasource_id was not valid
+       - application/json
+
+     * - 404
+       - .. code-block:: json
+
+            {
+                "status": "error",
+                "message": "Dataset does not exist within this data source"
+            }
+
+       - Parameter dataset_id did not refer to a valid dataset within this data source
+       - application/json
+
+     * - any other
+       - Error response from external data source
+       - An error occured within the external data source
+       - determined by data source
+
+--------
 
 
 References