-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add dvc pipelines #14
base: master
Are you sure you want to change the base?
Add dvc pipelines #14
Conversation
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
…ifeZone Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
@@ -0,0 +1,6 @@ | |||
[core] | |||
remote = processed_output | |||
['remote "gcbm_belize_logs"'] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we add a comment alerting users to configure this to their own storage?
1900,Total Biomass,Tropical Dry,0.0014951294179772342 | ||
1900,Total Biomass,Tropical Moist,6.140465519536419 | ||
1900,Total Biomass,Tropical Premontane Wet,6.079037757144233 | ||
1900,Deadwood,Tropical Dry, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@radistoubalidis - do you know why these tables have changed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't know I'll look it up.
Postprocessing/metrics/Deadwood.json
Outdated
@@ -0,0 +1 @@ | |||
{"Deadwood, Tropical Dry": {"pool_tc_sum_MEAN": "15624317.95", "area_sum_MEAN": "1142790.38", "pool_tc_per_ha_MEAN": "13.67"}, "Deadwood, Tropical Moist": {"pool_tc_sum_MEAN": "15802289.91", "area_sum_MEAN": "608498.41", "pool_tc_per_ha_MEAN": "25.97"}, "Deadwood, Tropical Premontane Wet": {"pool_tc_sum_MEAN": "13011268.76", "area_sum_MEAN": "417245.77", "pool_tc_per_ha_MEAN": "31.18"}} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is cool. Does it refer to the mean over the lifetime of the simulation? Alternative/additional summary statistics might be start, mid- and endpoints. (e.g 1900, 1950, 2000) or decadal timestamps.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wrote a simple script that calculates the mean value of for each indicator for each LifeZone so we can use it for dvc metrics. But I didn't use it yet cause there were some inconcistencies.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is cool. Does it refer to the mean over the lifetime of the simulation? Alternative/additional summary statistics might be start, mid- and endpoints. (e.g 1900, 1950, 2000) or decadal timestamps.
Update : in b1fa0de I modified analyze.py to create a json metric file for start,mid and end endpoints .
|
||
REM Set Python path - change this to your Python installation directory. | ||
set GCBM_PYTHON=C:\Python37 | ||
set GCBM_PYTHON=C:\Develop\Python\Python37 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you use an environmental variable for this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I refactored all the python paths in bat files in 5296555
dvc.yaml
Outdated
@@ -0,0 +1,116 @@ | |||
stages: | |||
tiler: | |||
cmd: C:\Develop\Python\Python37\python.exe ..\..\tools\Tiler\tiler.py |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you add python to PATH this could be cmd: python ..\..\tools\Tiler\tiler.py
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For some reason, even though I added python37 to my PATH
this stage fails when I try to run it with python
instead of the whole path. But I 'll note it on the pipeline readme
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update :
In 237d2a5 I added a vars list in dvc.yml where the local python path is included so it can be used as a var in the stages
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks awesome, well done @radistoubalidis! The only thing missing is a README update to describe the new pipeline and steps to reproduce.
I'm not quite sure why the sensitivity figures have changed. It suggests that you're getting different results somehow. Or, alternatively, the problem could be that the post-processing pipeline is missing a grouping variable - this can sometimes cause the sawtooth pattern you see in the new figures. I'll double check the R code and get back to you.
I noticed it too about the figures, they're not the same with HEAD , I'll try to modify the pipeline so the outputs come out the same as in HEAD UPDATE : |
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
…t to post_processing metrics in dvc Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
@aornugent the changes after your review (updated):
|
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
Signed-off-by: radis toubalidis <[email protected]>
a7aa8e3
to
7775c63
Compare
Output of
|
…e order Signed-off-by: radis toubalidis <[email protected]>
In fd9b236 This is needed because dvc as a default does not define an order in the pipeline stages , it does it only if for each |
Signed-off-by: radis toubalidis <[email protected]>
@radistoubalidis or @aldeav - can we please update this branch to develop? @aldeav has fixed a bug in the postprocessing code to remedy the saw-tooth lines in the output figures. |
Signed-off-by: radis toubalidis [email protected]
Description
The goal of this PR is :
Standalone_GCBM\readme.txt
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Additional Context (Please include any Screenshots/gifs if relevant)
Current stages:
DVC creates these files:
.dvc
dvc.lock
--> it captures hashes of the dependencies (usually md5s)dvc.yml
--> which contains the stages of the pipeline