Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor DCC configuration #122

Merged
merged 78 commits into from
Nov 2, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
e96532e
Move DCC config folders in dcc folder
afwillia Oct 26, 2023
a55507b
Add dcc tenant config file
afwillia Oct 26, 2023
5f53c3d
Move test fileview configs to test directory
afwillia Oct 26, 2023
96574bd
Add github link to each dcc config file
afwillia Oct 26, 2023
4302edf
Update links in dca demo config
afwillia Oct 26, 2023
68edd2b
Remove dcc_config.csv in favour of tenants.json
afwillia Oct 26, 2023
93bff11
Update paths to template config files and logos for each DCC
afwillia Oct 26, 2023
62009b2
Fix name of EL dca config file
afwillia Oct 26, 2023
814bd4c
Remove brackets from demo upsert config
afwillia Oct 26, 2023
f46d35d
Remove out array from json config menu
afwillia Oct 26, 2023
128e4d1
Update links to template drop down for tests
afwillia Oct 26, 2023
e8afa6c
Use relative filepaths instead of githubusercontent link to config fi…
afwillia Oct 27, 2023
ddf6f9a
Use relative filepath to dca demo config in tenants.json
afwillia Oct 27, 2023
68a5435
Use relative filepath instead of url to reference config files.
afwillia Oct 27, 2023
2944916
Update configs to use relative filepath instead of githubusercontent url
afwillia Oct 27, 2023
3a47c1c
Add schemas for tenants.json and dca_config.json
afwillia Oct 27, 2023
a31c9ed
Change tenants array to an oject
afwillia Oct 30, 2023
9aa8655
change tenants from array to object and add tenant object
afwillia Oct 30, 2023
71d8732
Update dca config schema
afwillia Oct 30, 2023
f1a9b78
correct manifest submit type for GF
afwillia Oct 30, 2023
b80ee85
Add uri-refernce format to the URL properties of dca config schema
afwillia Oct 30, 2023
e895dd5
Change tenant schema to an array of objects. Define object specificat…
afwillia Oct 31, 2023
ac8a50d
Update tenants.json to an object of arrays
afwillia Oct 31, 2023
f836c11
Add github workflow to validate tenants.json
afwillia Oct 31, 2023
15e118d
Fix whitespace
afwillia Oct 31, 2023
71c2fc9
remove - from workflow name
afwillia Oct 31, 2023
297f6ee
remove path trigger for testing
afwillia Oct 31, 2023
c3ab00f
use docker form of json validate
afwillia Oct 31, 2023
2fc6ddb
switch json inputs to env vars
afwillia Oct 31, 2023
5f53faf
fix variable name
afwillia Oct 31, 2023
96ab333
add repo to json filepath
afwillia Oct 31, 2023
e528808
tenants.schema not tenant.schema
afwillia Oct 31, 2023
bab0acc
maybe we don't need the repo dir
afwillia Oct 31, 2023
db8f49f
remove filler ID field and try using http for schema
afwillia Oct 31, 2023
41695dd
test removing the schema
afwillia Oct 31, 2023
da00d16
commit an invalid tenants.json to test validation
afwillia Oct 31, 2023
4e45521
Fix tenants.json
afwillia Oct 31, 2023
267bb30
Add tenant json paths to trigger workflow
afwillia Oct 31, 2023
a60c528
Update workflow description
afwillia Oct 31, 2023
f7ae9b4
test validate dcc config workflow
afwillia Oct 31, 2023
1f689b2
fix indentation
afwillia Oct 31, 2023
8a3fb9f
Modify smoke test config for test
afwillia Oct 31, 2023
98f50bc
try filtering modified files
afwillia Oct 31, 2023
16b1015
fix smoke test config
afwillia Oct 31, 2023
0c46ad8
test smoke test change
afwillia Oct 31, 2023
a52c757
Change file to trigger dca config validation workflow
afwillia Oct 31, 2023
68bef96
remove extra echo
afwillia Oct 31, 2023
fcb1259
test edit to config
afwillia Oct 31, 2023
1bfd74d
remove extra parenthesis from echo
afwillia Oct 31, 2023
0e2402a
test edit to smoke test json
afwillia Oct 31, 2023
162bf97
print config var
afwillia Oct 31, 2023
eea8bb9
change smoke test
afwillia Oct 31, 2023
c729f60
remove single quotes in grep
afwillia Oct 31, 2023
b19544b
test formatting output with xargs
afwillia Oct 31, 2023
93b145b
remove schema from dca config schema for compatibility
afwillia Oct 31, 2023
ac68636
set output to json
afwillia Oct 31, 2023
002213e
try splitting file changed list with tr
afwillia Nov 1, 2023
3062033
use space-delimited string for changed file output
afwillia Nov 1, 2023
e0d6f3f
quote echo
afwillia Nov 1, 2023
a2f2521
echo the changed files variable
afwillia Nov 1, 2023
b41183b
create filter_file step id to reference varible
afwillia Nov 1, 2023
c3649f5
fix smoke test config
afwillia Nov 1, 2023
1f6846a
remove printing config variable for debugging
afwillia Nov 1, 2023
5c14d21
Update template path for mrt test config
afwillia Nov 1, 2023
e2a8a35
change files changed string to csv instead of space-delimited to hand…
afwillia Nov 1, 2023
7aee822
pipe changed files output to tr again to remove new liens
afwillia Nov 1, 2023
6a4d69d
make json files list comma-separated
afwillia Nov 1, 2023
aa89ff4
print env variables to debug
afwillia Nov 1, 2023
d12f4f1
print env variables to debug 2
afwillia Nov 1, 2023
7974ad5
print env variables to debug 3
afwillia Nov 1, 2023
165c5d0
use sed to remove last comma
afwillia Nov 1, 2023
670dce1
Remove printing of env var for debugging
afwillia Nov 1, 2023
a5ed142
test adding template config workflow in R
afwillia Nov 1, 2023
587f767
remove template config generation workflow. This is in the example da…
afwillia Nov 2, 2023
590bb91
Add json schema for template dropdown config
afwillia Nov 2, 2023
a2b6ff1
Add workflow to validate template config files
afwillia Nov 2, 2023
b75e649
filter files that trigger validation workflows on PR
afwillia Nov 2, 2023
10687ce
only run validation if template json files have changed
afwillia Nov 2, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions .github/workflows/validate-dcc-config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# ------------------------------------------------------------------------------
# Validate a dcc config file against the dca_config schema
# ------------------------------------------------------------------------------
name: validate dcc config
on:
push:
paths:
- 'dcc/*dca_config.json'
- 'test/*dca_config.json'
- 'schemas/dca_config.schema.json'
pull_request:
branches:
- 'main'
- 'staging'
- 'dev'
paths:
- 'dcc/*dca_config.json'
- 'test/*dca_config.json'
- 'schemas/dca_config.schema.json'

jobs:
validate-dcc-config:
runs-on: ubuntu-latest
steps:
- id: files
uses: masesgroup/retrieve-changed-files@v3
with:
format: 'csv'
- id: filter_files
name: Filter files to dca_config.json files
run: |
echo "configs=$(echo ${{ steps.files.outputs.added_modified }} | tr ',' '\n' | grep dca_config\\.json | tr '\n' ',' | sed 's/\(.*\),/\1 /')" >> "$GITHUB_OUTPUT"
- uses: actions/checkout@v4
- name: Validate JSON
uses: docker://orrosenblatt/validate-json-action:latest
env:
INPUT_SCHEMA: schemas/dca_config.schema.json
INPUT_JSONS: ${{ steps.filter_files.outputs.configs }}

39 changes: 39 additions & 0 deletions .github/workflows/validate-template-config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# ------------------------------------------------------------------------------
# Validate a dca template config file against the dca template config schema
# ------------------------------------------------------------------------------
name: Validate Template Config
on:
push:
paths:
- 'dcc/*dca-template-config.json'
- 'test/*dca-template-config.json'
- 'schemas/dca_template_config.schema.json'
pull_request:
branches:
- 'main'
- 'staging'
- 'dev'
paths:
- 'dcc/*dca-template-config.json'
- 'test/*dca-template-config.json'
- 'schemas/dca_template_config.schema.json'

jobs:
validate-template-config:
runs-on: ubuntu-latest
steps:
- id: files
uses: masesgroup/retrieve-changed-files@v3
with:
format: 'csv'
- id: filter_files
name: Filter files to dca-template-config.json files
run: |
echo "configs=$(echo ${{ steps.files.outputs.added_modified }} | tr ',' '\n' | grep dca-template-config\\.json | tr '\n' ',' | sed 's/\(.*\),/\1 /')" >> "$GITHUB_OUTPUT"
- uses: actions/checkout@v4
- name: Validate JSON
if: ${{ steps.filter_files.outputs.configs != '' }}
uses: docker://orrosenblatt/validate-json-action:latest
env:
INPUT_SCHEMA: schemas/dca_template_config.schema.json
INPUT_JSONS: ${{ steps.filter_files.outputs.configs }}
28 changes: 28 additions & 0 deletions .github/workflows/validate-tenants-config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# ------------------------------------------------------------------------------
# Validate the tenants.json file against the tenants.schema.json schema
# ------------------------------------------------------------------------------
name: validate tenants config
on:
push:
paths:
- 'tenants.json'
- 'schemas/tenants.schema.json'
pull_request:
branches:
- 'main'
- 'staging'
- 'dev'
paths:
- 'tenants.json'
- 'schemas/tenants.schema.json'

jobs:
validate-tenants-config:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Validate JSON
uses: docker://orrosenblatt/validate-json-action:latest
env:
INPUT_SCHEMA: schemas/tenants.schema.json
INPUT_JSONS: tenants.json
File renamed without changes
19 changes: 19 additions & 0 deletions dcc/ADKP/adkp_dca_config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"project_name":"AD Knowledge Portal",
"synapse_asset_view":"syn51324810",
"data_model_url":"https://raw.githubusercontent.com/adknowledgeportal/data-models/main/divco.data.model.v1.jsonld",
"template_menu_config_file":"dcc/ADKP/dca-template-config.json",
"logo_location":"dcc/ADKP/ADKnowledgePortal.png",
"logo_link":"https://adknowledgeportal.synapse.org",
"manifest_output_format":"excel",
"manifest_use_annotations":false,
"validate_restrict_rules":false,
"submit_use_schema_labels":true,
"submit_table_manipulation":"replace",
"submit_manifest_record_type":"file_only",
"submit_hide_blanks":false,
"use_compliance_dashboard":false,
"primary_col":"#2a668d",
"secondary_col":"#184e71",
"sidebar_col":"#191919"
}
File renamed without changes.
17 changes: 17 additions & 0 deletions dcc/BTC/btc_dca_config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"project_name":"BTC DCC",
"synapse_asset_view":"syn51407795",
"data_model_url":"https://raw.githubusercontent.com/Sage-Bionetworks/btc-data-models/main/btc.model.jsonld",
"template_menu_config_file":"dcc/BTC/dca-template-config.json",
"manifest_output_format":"excel",
"manifest_use_annotations":false,
"validate_restrict_rules":false,
"submit_use_schema_labels":true,
"submit_table_manipulation":"replace",
"submit_manifest_record_type":"table_and_file",
"submit_hide_blanks":false,
"use_compliance_dashboard":false,
"primary_col":"#2a668d",
"secondary_col":"#184e72",
"sidebar_col":"#191920"
}
File renamed without changes.
File renamed without changes.
17 changes: 17 additions & 0 deletions dcc/EL/el_dca_config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"project_name":"EL",
"synapse_asset_view":"syn51753858",
"data_model_url":"https://raw.githubusercontent.com/Sage-Bionetworks/ELITE-data-models/main/example.model.jsonld",
"template_menu_config_file":"dcc/EL/dca-template-config.json",
"manifest_output_format":"excel",
"manifest_use_annotations":false,
"validate_restrict_rules":false,
"submit_use_schema_labels":true,
"submit_table_manipulation":"replace",
"submit_manifest_record_type":"table_and_file",
"submit_hide_blanks":false,
"use_compliance_dashboard":false,
"primary_col":"#2a668d",
"secondary_col":"#184e71",
"sidebar_col":"#191919"
}
File renamed without changes.
17 changes: 17 additions & 0 deletions dcc/GF/gf_dca_config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"project_name":"Gray Foundation",
"synapse_asset_view":"syn28142805",
"data_model_url":"https://raw.githubusercontent.com/gf-dcc/data-model/v1.1.0-rc/GF.jsonld",
"template_menu_config_file":"dcc/GF/dca-template-config.json",
"manifest_output_format":"google_sheet",
"manifest_use_annotations":false,
"validate_restrict_rules":true,
"submit_use_schema_labels":true,
"submit_table_manipulation":"replace",
"submit_manifest_record_type":"table_and_file",
"submit_hide_blanks":false,
"use_compliance_dashboard":false,
"primary_col":"#2a668d",
"secondary_col":"#184e71",
"sidebar_col":"#191919"
}
File renamed without changes
File renamed without changes.
19 changes: 19 additions & 0 deletions dcc/HTAN/htan_dca_config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"project_name":"HTAN All Projects",
"synapse_asset_view":"syn20446927",
"data_model_url":"https://raw.githubusercontent.com/ncihtan/data-models/main/HTAN.model.jsonld",
"template_menu_config_file":"dcc/HTAN/dca-template-config.json",
"logo_location":"dcc/HTAN/HTAN_text_logo.png",
"logo_link":"https://humantumoratlas.org",
"manifest_output_format":"google_sheet",
"manifest_use_annotations":false,
"validate_restrict_rules":false,
"submit_use_schema_labels":true,
"submit_table_manipulation":"replace",
"submit_manifest_record_type":"file_only",
"submit_hide_blanks":false,
"use_compliance_dashboard":false,
"primary_col":"#605ca8",
"secondary_col":"#5F008C",
"sidebar_col":"#191919"
}
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Binary file added dcc/INCLUDE/INCLUDE DCC Logo-01.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
File renamed without changes
File renamed without changes.
19 changes: 19 additions & 0 deletions dcc/MC2/mc2_dca_config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"project_name":"Cancer Complexity Knowledge Portal - Database",
"synapse_asset_view":"syn27210848",
"data_model_url":"https://raw.githubusercontent.com/mc2-center/data-models/main/mc2.model.jsonld",
"template_menu_config_file":"dcc/MC2/dca-template-config.json",
"logo_location":"dcc/MC2/cckp_logo.png",
"logo_link":"https://cancercomplexity.synapse.org",
"manifest_output_format":"excel",
"manifest_use_annotations":false,
"validate_restrict_rules":false,
"submit_use_schema_labels":false,
"submit_table_manipulation":"upsert",
"submit_manifest_record_type":"table_and_file",
"submit_hide_blanks":false,
"use_compliance_dashboard":false,
"primary_col":"#407BA0",
"secondary_col":"#5BB0B5",
"sidebar_col":"#191919"
}
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
17 changes: 17 additions & 0 deletions dcc/NF-OSI/nf-osi_dca_config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"project_name":"NF-OSI",
"synapse_asset_view":"syn16858331",
"data_model_url":"https://raw.githubusercontent.com/nf-osi/nf-metadata-dictionary/v8.0.1/NF.jsonld",
"template_menu_config_file":"dcc/NF-OSI/dca-template-config.json",
"manifest_output_format":"google_sheet",
"manifest_use_annotations":true,
"validate_restrict_rules":false,
"submit_use_schema_labels":true,
"submit_table_manipulation":"replace",
"submit_manifest_record_type":"table_and_file",
"submit_hide_blanks":true,
"use_compliance_dashboard":false,
"primary_col":"#2a668d",
"secondary_col":"#184e71",
"sidebar_col":"#191919"
}
File renamed without changes
File renamed without changes.
19 changes: 19 additions & 0 deletions dcc/VEOIBD/veoibd_dca_config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"project_name":"VEOIBD",
"synapse_asset_view":"syn51397378",
"data_model_url":"https://raw.githubusercontent.com/VEOIBD/data_models/main/veoibd.data.model.jsonld",
"template_menu_config_file":"dcc/VEOIBD/dca-template-config.json",
"logo_location":"dcc/VEOIBD/VEOIBD Logo.png",
"logo_link":"https://veoibd.org",
"manifest_output_format":"excel",
"manifest_use_annotations":false,
"validate_restrict_rules":true,
"submit_use_schema_labels":true,
"submit_table_manipulation":"upsert",
"submit_manifest_record_type":"file_only",
"submit_hide_blanks":false,
"use_compliance_dashboard":false,
"primary_col":"#2a668d",
"secondary_col":"#184e72",
"sidebar_col":"#191920"
}
File renamed without changes
File renamed without changes.
21 changes: 21 additions & 0 deletions dcc/demo/dca_demo_dca_config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"project_name":"DCA Demo",
"synapse_asset_view":"syn33715412",
"data_model_url":"https://raw.githubusercontent.com/Sage-Bionetworks/data-models/main/example.model.jsonld",
"data_model_info": "",
"template_menu_config_file":"dcc/demo/dca-template-config.json",
"logo_location":"dcc/demo/Logo_Sage_Logomark.png",
"logo_link":"https://synapse.org",
"dcc_help_link": "",
"manifest_output_format":"google_sheet",
"manifest_use_annotations":false,
"validate_restrict_rules":false,
"submit_use_schema_labels":true,
"submit_table_manipulation":"replace",
"submit_manifest_record_type":"table_and_file",
"submit_hide_blanks":false,
"use_compliance_dashboard":false,
"primary_col":"#2a668d",
"secondary_col":"#184e71",
"sidebar_col":"#191919"
}
File renamed without changes.
File renamed without changes.
17 changes: 17 additions & 0 deletions dcc/demo_upsert/demo_upsert_dca_config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"project_name":"DCA Demo Upsert",
"synapse_asset_view":"syn51489635",
"data_model_url":"https://raw.githubusercontent.com/Sage-Bionetworks/schematic/develop/tests/data/example.model.jsonld",
"template_menu_config_file":"dcc/demo_upsert/dca-template-config.json",
"manifest_output_format":"excel",
"manifest_use_annotations":false,
"validate_restrict_rules":false,
"submit_use_schema_labels":false,
"submit_table_manipulation":"upsert",
"submit_manifest_record_type":"table_and_file",
"submit_hide_blanks":false,
"use_compliance_dashboard":false,
"primary_col":"#2a668d",
"secondary_col":"#184e71",
"sidebar_col":"#191919"
}
21 changes: 0 additions & 21 deletions dcc_config.csv

This file was deleted.

Loading
Loading