<sup>1</sup> Digital Health and Biomedical Engineering, School of Electronics and Computer Science, Faculty of Engineering and Physical Sciences, University of Southampton<br>
<sup>1</sup> Digital Health and Biomedical Engineering, School of Electronics and Computer Science, Faculty of Engineering and Physical Sciences, University of Southampton<br>
<sup>2</sup> School of Primary Care Population Sciences and Medical Education, University of Southampton <br>
<sup>2</sup> School of Primary Care Population Sciences and Medical Education, University of Southampton <br>
<sup>3</sup> Population Data Science, Swansea University Medical School, Faculty of Medicine, Health & Life Science, Swansea University <br>
<sup>3</sup> Population Data Science, Swansea University Medical School, Faculty of Medicine, Health & Life Science, Swansea University <br>
<br>
<sup>*</sup>Correspondence to: Jakub J. Dylag, Digital Health and Biomedical Engineering, School of Electronics and Computer Science, Faculty of Engineering and Physical Sciences, University of Southampton, J.J.Dylag@soton.ac.uk
### 🖋 How to cite this work
*Correspondence to: Jakub J. Dylag, Digital Health and Biomedical Engineering, School of Electronics and Computer Science, Faculty of Engineering and Physical Sciences, University of Southampton, J.J.Dylag@soton.ac.uk*
### Citation
> Dylag JJ, Chiovoloni R, Akbari A, Fraser SD, Boniface MJ. A Tool for Automating the Curation of Medical Concepts derived from Coding Lists. GitLab [Internet]. May 2024. Available from: https://git.soton.ac.uk/meldb/concepts-processing
> Dylag JJ, Chiovoloni R, Akbari A, Fraser SD, Boniface MJ. A Tool for Automating the Curation of Medical Concepts derived from Coding Lists. GitLab [Internet]. May 2024. Available from: https://git.soton.ac.uk/meldb/concepts-processing
## 🙌 Introduction
This project generate the medical coding lists that defines cohort phenotypes used for inclusion criteria in MELD-B. The goal is to automatically prepare a code list from an approved clinical specification of inclusion criteria.
The output code list is then used by data providers to select MELD-B cohorts.
## Introduction
This tool automates the verification, translation and organisation of medical coding lists defining cohort phenotypes for inclusion criteria. By processing externally sourced clinical inclusion criteria into actionable code lists, this tool ensures consistent and efficient curation of cohort definitions. These code lists can be subsequently used by data providers (e.g. SAIL) to construct study cohorts.
## 📃 Method
## Methods
### Process
### Workflow Overview
1. Approved MELB-B concepts are defined in a CSV spreadsheet (currently PHEN_summary_working.csv).
1. Approved MELD-B concepts are outlined in a CSV spreadsheet (e.g., `PHEN_summary_working.csv`).
2. Imported Code Lists in `/src`are verified against all NHS TRUDregistered codes
2. Imported code lists in the `/src`directory are validated against NHS TRUD-registered codes.
3. Mappings from Imported Code Lists to Outputted MELD-B Concept's Code list are defined in JSON format within`PHEN_assign_v3.json`.
3. Mappings from imported code lists to outputted MELD-B concepts are defined in the`PHEN_assign_v3.json`file.
- See "JSON Phenotype Mapping" section for more details
- See "JSON Phenotype Mapping" section for more details
4. Process is executed from command line either manually or from bash script `run.sh`
4. The process is executed via command-line. Refer to the "Usage" section for execution instructions.
- See "Usage" section for more details
5. Outputted concept code lists are saved to the `/concepts` Git repository, with all changes tracked.
5. Output Concept Code Lists are saved to the `/concepts` git repository and any changes are tracked.
6. The code lists can be exported to SAIL or any other Data Bank.
6. Output Concept Code Lists can be exported into SAIL or any other Data Bank
### Medical Coding Standards Supported
### Supported Medical Coding Standards
| Code Type | Verification | Maps to |
The tool supports verification and mapping across various diagnostic coding formats:
@@ -43,53 +44,51 @@ The output code list is then used by data providers to select MELD-B cohorts.
...
@@ -43,53 +44,51 @@ The output code list is then used by data providers to select MELD-B cohorts.
| OPCS4 | NHS TRUD | |
| OPCS4 | NHS TRUD | |
| ATC | None | |
| ATC | None | |
MELD-B refers to various diagnostic code formats included in target datasets.
#### Notes on Code Systems:
* Read V2
-**Read V2:** Replaced by SNOMED-CT in 2018, but still supported by SAIL (restricted to five-character codes).
* Read codes were used widely in primary care but were replaced by SNOMED-CT from around 2018 https://isd.digital.nhs.uk/trud/user/guest/group/0/pack/9
-**SNOMED-CT:** Adopted widely by the NHS in 2018; mappings to Read codes are partially provided by CPRD and NHS TRUD.
* SAIL only supports five character read codes V2
-**ICD-10:** Widely used in hospital settings and critical for HES-linked datasets.
* SNOMED-CT was adopted by the NHS around 2018
-**ATC Codes:** Maintained by WHO and used internationally for medication classification.
* CPRD AURUM uses SNOWMED codes and include mapping to read codes but no other database (CPRD Gold, SAIL) does.
* Mappings exist from SNOWMED to Read codes, some provided by CPRD and others NHS Trud
## Installation
* ICD-10 are codes used in hospital settings and are importnat for the HES linked datasets.
* ATC codes are interntionally accepted for the classification of medicinces and maintained by the WHO.
1.**Sign Up:** Register at [NHS TRUD](https://isd.digital.nhs.uk/trud/user/guest/group/0/account/form) and accept the following licenses:
-[OPCS-4.10 Data Files](https://isd.digital.nhs.uk/trud/users/guest/filters/0/categories/10/items/119/releases)
1. Due to the licencing of NHS TRUD resources, you <mark>MUST first [Sign Up](https://isd.digital.nhs.uk/trud/user/guest/filters/0/account/form) to NHS TRUD and accept the following licences</mark>:
2. Once all licences are accepted, get your [API Key](https://isd.digital.nhs.uk/trud/users/authenticated/filters/0/account/manage) for NHS TRUD.
2.**Obtain API Key:** Retrieve your API key from [NHS TRUD Account Management](https://isd.digital.nhs.uk/trud/users/authenticated/filters/0/account/manage).
3. Finally, run the automated extraction script, inputting your API Key to granty temporary access to the resources above. Use the command `python trud_api.py --key <INSERT KEY>` (replacing your key in the marked area).
3.**Install TRUD:** Download and Install NHS TRUD medical code resources.
- The convertion Tables will be saved as `.parquet` tables in the folder `maps/processed/`.
Executing the script using the command: `python trud_api.py --key <API_KEY>`.
- NHS TRUD defines one-way mappings and does <b>NOT ADVISE</b> reversing the mappings. If you still wish to reverse these into two-way mappings, duplicate the given `.parquet` table and reverse the filename (e.g. `read2_code_to_snomed_code.parquet` to `snomed_code_to_read2_code.parquet`)
Processed tables will be saved as `.parquet` files in the `maps/processed/` directory.
-*Note: NHS TRUD defines one-way mappings and does <b>NOT ADVISE</b> reversing the mappings. If you still wish to reverse these into two-way mappings, duplicate the given `.parquet` table and reverse the filename (e.g. `read2_code_to_snomed_code.parquet` to `snomed_code_to_read2_code.parquet`)*
4. Populate the SQLite3 database with OMOP Vocabularies. These can be download from https://athena.ohdsi.org/vocabulary/list.
- Install the following vocabularies by ticking the box:
4.***Optional: Install OMOP Database:** Download and install OMOP vocabularies from [Athena OHDSI](https://athena.ohdsi.org/vocabulary/list).
- 1-SNOMED
- Required vocabularies include:
- 2-ICD9CM
- 1) SNOMED
- 17-Readv2
- 2) ICD9CM
- 21-ATC
- 17) Readv2
- 55-OPCS4
- 21) ATC
- 57-HES Specialty
- 55) OPCS4
- 70-ICD10CM
- 57) HES Specialty
- 75-dm+d
- 70) ICD10CM
- 144-UK Biobank
- 75) dm+d
- 154-NHS Ethnic Category
- 144) UK Biobank
- 155-NHS Place of Service
- 154) NHS Ethnic Category
- Use the command `python omop_api.py --install <INSERT PATH>` to load vocabularies into database (insert your own path to unzipped download folder).
- 155) NHS Place of Service
- Un-zip the downloaded folder and copy it's path.
-In certain cases where you wish to sub-divde a code list table or a column features multiple code types additional processing is required. Add a `action` object inside of the `file` object.
In certain cases where you wish to sub-divde a code list table or a column features multiple code types additional processing is required. Add a `action` object inside of the `file` object.
- Table with a sub-categorical column:
#### Table with a sub-categorical column:
- In order to sub-divide a table by a categorical column use the "divide_col" action
In order to sub-divide a table by a categorical column use the "divide_col" action
- e.g. ``` "actions":{"divide_col": "MMCode"}```
```json
"actions":{
- Table with multiple code types in single column:
"divide_col":"MMCode"
- Need to split column into multiple columns, so only one code type per column.
}
- The "split_col" attribute is the categorical column indicating the code type in that row. The <b>category names should replace column</b> names in the `columns` properties.
```
- The "codes_col" attribute is the code column with mulitple code types in a single column
- e.g.
```
"actions":{
"split_col":"coding_system",
"codes_col":"code"
},
"columns":{
"read2_code":"Read codes v2",
"med_code":"Med codes",
"icd10_code":"ICD10 codes",
"metadata":["description"]
},
```
#### Table with multiple code types in single column:
Need to split column into multiple columns, so only one code type per column.
- The "split_col" attribute is the categorical column indicating the code type in that row. The <b>category names should replace column</b> names in the `columns` properties.
- The "codes_col" attribute is the code column with mulitple code types in a single column
```json
"actions":{
"split_col":"coding_system",
"codes_col":"code"
},
"columns":{
"read2_code":"Read codes v2",
"med_code":"Med codes",
"icd10_code":"ICD10 codes",
"metadata":["description"]
},
```
*<b>Large Code lists</b> with numerous phenotypes (e.g. Ho et al), require lots of JSON to be generated. See the "Ho generate JSON" section in process_codes_WP.ipynb for example code to generate
**<b>Large Code lists</b> with numerous phenotypes (e.g. Ho et al), require lots of JSON to be generated. See the "Ho generate JSON" section in process_codes_WP.ipynb for example code to generate*
## ⚡ Usage
## Usage
Script preprocess code lists and to map to given concept/phenotype
Script preprocess code lists and to map to given concept/phenotype
@@ -180,12 +180,11 @@ git tag -a v1.0.0 -m "added features ..."
...
@@ -180,12 +180,11 @@ git tag -a v1.0.0 -m "added features ..."
git push
git push
```
```
## 🏦 Funding
## Acknowledgements
This project has received funding from the National Institute of Health Research under grant agreement NIHR203988.
This project was developed in the context of the [MELD-B](https://www.southampton.ac.uk/publicpolicy/support-for-policymakers/policy-projects/Current%20projects/meld-b.page) project, which is funded by the UK [National Institute of Health Research](https://www.nihr.ac.uk/) under grant agreement NIHR203988.
