[fix] Allow multiple files per concept set

Originates from Concepts #1

Enabling convertion of MELDB SAIL Phenotypes to acmc 0.1.12

• Update Validation Function + Type Checker
• Run Validation Command
• Update Map Function
• Run Map Command
• Add metadata (static dict per concept set)
• Add metadata flag to exclude metadata from output
• Run actions divide_col
• Run actions split_col
• Update Yaml Examples
• Update Diff Function
• Run Diff Command
• Add Non-translate Flag
• Fix PyTest
• Update Docs (index.md)

assigned to @jjd1c23

made the issue visible to everyone

created branch 57-fix-convertion-of-meldb-sail-phenotypes-to-acmc-0-1-12 to address this issue

changed the description

mentioned in commit df2271e2

created branch 57-fix-convertion-of-meldb-sail-phenotypes-to-acmc-0-1-12-2 to address this issue

@mjbonifa Allowing multiple files per concept - causes issue with diff command in phen.py. Deepdiff uses dictionaries to compare old and new phenotypes. Python does not allow duplicate keys in dictionaries. In order to store concept set name to file path relation, must use list of tuples instead.

@jjd1c23 I have updated branch 57-fix-convertion-of-meldb-sail-phenotypes-to-acmc-0-1-12 with the following

• Merged dev into 57 so that it had some of the latest changes with the tests
• Updated phen.py def extract_concepts... so that it works with a list of paths
• Ran all tests and they pass as previously defined (although noting that the testing is limited and many things are not checked)

You should pull the branch before you start again. This is good practice anyway as you'll get the latest changes.

We need to talk about the metadata support as you've commented out a load of stuff during the implementation of the path list. It will be easier for chat for 15 to understand what we have both done to ensure we're on the same page.

Thanks @mjbonifa. I have deleted the duplicate 57-fix-convertion-of-meldb-sail-phenotypes-to-acmc-0-1-12-2, as all commits have been merged.

The commented out code metadata code didn't seem to do much, the result of the for loop were overwritten and only the last element was actually used because of the missing indentation. However by this commit df2271e2, I understand metadata shouldn't be included in the example yamls?

From by understanding the "metadata" was a static property for every concept set. e.g. the tracking columns from the excel sheet. I have created a simple add_metadata() function (commit: 62c4b649), which duplicates these for all rows in the concept set, just after translation.

There may also be a need for a "metacolumn", where values for each row are not constant and copied from the source concept tables. e.g. "WILK_READCODE" column for the ALL_MEDICATIONS concept for the workaround WP3 wanted.

Happy to discuss further in Fridays meeting if need be.

@mjbonifa WP2 meeting decisions:

• "metadata" may clutter the output -> add flag to exclude metadata from output csv
• "metacolumn" requires reimplementation. Difficulty in preserving consistent metacolumn names in output (e.g. "Description" and "descriptions" metacolumns). Would require merging with database? -> deemed outside of scope, and remove implementation.

PyTest seems also to be broken when run locally on Windows 11.

Error Output

``` ------------------------------------------------ Captured stdout setup ------------------------------------------------ TEMP DIR C:\Users\jjd1c23\AppData\Local\Temp\tmpmf0_ws27 ------------------------------------------------ Captured stdout call ------------------------------------------------- Temporary directory: C:\Users\jjd1c23\AppData\Local\Temp\tmpmf0_ws27 -------------------------------------------------- Captured log call -------------------------------------------------- INFO acmc_logger:phen.py:234 Initialising Phenotype in directory: C:\Users\jjd1c23\AppData\Local\Temp\tmpmf0_ws27\phen =============================================== short test summary info =============================================== FAILED tests/test_acmc.py::test_phen_init_local_specified - FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:\\Users\\jjd1c23\\AppData\\L... FAILED tests/test_acmc.py::test_phen_workflow[config1.yml] - FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:\\Users\\jjd1c23\\AppData\\L... FAILED tests/test_acmc.py::test_phen_workflow[config2.yml] - FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:\\Users\\jjd1c23\\AppData\\L... FAILED tests/test_acmc.py::test_phen_workflow[config3.yml] - FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:\\Users\\jjd1c23\\AppData\\L... FAILED tests/test_acmc.py::test_diff - FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:\\Users\\jjd1c23\\AppData\\L... ================================================== 5 failed in 2.43s ================================================== ```

created issue: #59 (closed)

changed title from [fix] Convertion of MELDB SAIL Phenotypes to acmc 0.1.12 to [fix] Allow multiple files per concept set

changed the description

mentioned in issue concepts#1

changed the description

marked the checklist item Run Map Command as completed

mentioned in commit 4eae3f81

mentioned in issue #59 (closed)

changed the description