diff --git a/README.md b/README.md index 2b5a057..e369e0d 100644 --- a/README.md +++ b/README.md @@ -17,8 +17,8 @@ Depending on the context, there are two different ways the programs can be used: - To remove quantile biases between an historical model simulation and observations from model data in order to produce bias corrected model data. This has been referred to as - *equidistant CDF matching* (EDCDFm; in the case of additive bias correction; [Li et al, 2010](https://doi.org/10.1029/2009JD012882)) or - *equiratio CDF matching* (EQCDFm; in the case of multiplicative bias correction; [Wang and Chen, 2013](https://doi.org/10.1002/asl2.454)). + *equidistant CDF matching* (EDCDFm) in the case of additive bias correction or + *equiratio CDF matching* (EQCDFm) in the case of multiplicative bias correction. See [docs/method_ecdfm.md](docs/method_ecdfm.md) and [docs/method_qdc.md](docs/method_qdc.md) for a detailed description of these methods and how they are implemented in the qqscale software. @@ -55,7 +55,7 @@ pre-installed you'll need to create your own. For example: ``` -$ conda install -c conda-forge netCDF4 xclim xesmf cmdline_provenance gitpython +$ conda install -c conda-forge netCDF4 xclim=0.36.0 pint=0.19.2 xesmf cmdline_provenance gitpython ``` You can then clone this GitHub repository and run the help option @@ -80,6 +80,9 @@ At the command line, QDC and/or ECDFm can be achieved by running the following s See the files named `docs/example_*.md` for detailed worked examples using these two command line programs. +Various command line workflows that use the qqscale software can be found at: +https://github.com/AusClimateService/qq-workflows + ### Jupyter notebook Starting with historical (`ds_hist`), reference (`ds_ref`) and target (`ds_target`) xarray Datasets @@ -114,20 +117,6 @@ ds_qq = adjust.adjust( ) ``` -### Performance - -The adjustment step (`adjust.py`) is the most time and memory intensive. -Here's some examples of time and memory requirements for different applications: - -EDCDFm with 30 years of daily CORDEX and AGCD data (on AUS-05i 691 x 886 horizontal grid): -- Training step: Compute (1 core) requires 250GB and 1hr 30min. Produces 1.3GB adjustment factor file. -- Adjustment step: Compute (1 core) requires 185GB and 1hr 50min. Produces output files of approx 200MB per year. -- Regridding the target data prior to performing the adjustment makes very little difference to processing time. - -EDCDFm with 20 years of daily CORDEX and AGCD data (on AUS-20i 171 x 211 horizontal grid): -- Training step: Compute (1 core) requires 7.3GB and 3min. -- Adjustment step: Compute (1 core) requires 7.9GB and 4min. - ## Questions Questions or comments are welcome at the GitHub repostory diff --git a/docs/example_cih.md b/docs/example_cih.md deleted file mode 100644 index e047c29..0000000 --- a/docs/example_cih.md +++ /dev/null @@ -1,157 +0,0 @@ -# Worked example: CIH - -The CSIRO Climate Innovation Hub (CIH) intends produce -application ready climate projections data -by applying relative changes simulated by CMIP6 global climate models -to observational data. - -The following example describes how to produce projections data -for the 2056-2085 period using the quantile delta change method -(see [method_qdc.md](method_qdc.md) for details) -for a given variable (pr), -global climate model (ACCESS-ESM1-5), -model experiment (ssp370), -model run (r1i1p1f1), -observation reference period (1990-2019 = "2005") -and historical model baseline (1995-2014 = "2005"). - -> **Shortcut** -> -> You'll see in steps 1-3 below that there are lots of -> positional arguments and program options/flags to remember, -> which means it can be easy to forget something or make an error. -> -> If you don't already have a process for managing complicated workflows like this, -> using a build tool like Make can be helpful. -> The Makefile and CIH configuation file at https://github.com/climate-innovation-hub/qq-workflows -> have been built to coordinate the bias correction workflow described below. -> -> To run steps 1-3 below, -> you could simply clone the qq-workflows repo and run the following at the command line: -> ``` -> git clone git@github.com:climate-innovation-hub/qq-workflows.git -> cd qq-workflows -> make validation CONFIG=cih/config_cih.mk VAR=pr MODEL=ACCESS-ESM1-5 EXPERIMENT=ssp370 RUN=r1i1p1f1 REF_START=2056 REF_END=2085 -> ``` - -## Step 1: Training - -The `train.py` command line program can be used to compare future and historical model data -in order to produce the "adjustment factors" that need to be applied to observations in the adjustment step. -You can run `python train.py -h` at the command line to view the positional arguments -and options/flags associated with the program. -To give an example, -the options/flags for the CIH projections described above would be as follows... - -The `--hist_files` flag is used to specify the historical model data files -(in this case for the 1995-2014 reference period): -``` ---hist_files -/g/data/fs38/publications/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_historical_r1i1p1f1_gn_19500101-19991231.nc -/g/data/fs38/publications/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_historical_r1i1p1f1_gn_20000101-20141231.nc -``` - -The `--ref_files` flag is used to specify the reference ssp370 data files: - -``` ---ref_files -/g/data/fs38/publications/CMIP6/ScenarioMIP/CSIRO/ACCESS-ESM1-5/ssp370/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_ssp370_r1i1p1f1_gn_20150101-20641231.nc -/g/data/fs38/publications/CMIP6/ScenarioMIP/CSIRO/ACCESS-ESM1-5/ssp370/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_ssp370_r1i1p1f1_gn_20650101-21001231.nc -``` - -There are a number of other flags that are also required to complete this task: -- `--scaling multiplicative`: Multiplicative scaling is typically used for rainfall (and `additive` for temperature). -- `--nquantiles 100`: We like to use 100 quantiles. -- `--input_hist_units "kg m-2 s-1"`, `--input_ref_units "kg m-2 s-1"`, `--output_units "mm day-1"`: Specify input and desired output units and the code will convert units if required. -- `--hist_time_bounds 1995-01-01 2014-12-31`: Time bounds for historical period. -- `--ref_time_bounds 1990-01-01 2019-12-31`: Time bounds for reference period. -- `--ssr`: Apply Singularity Stochasitc Removal (for precipitation data). -- `--verbose`: As the program runs it will print progess updates to the screen. - -We would use the `--time_grouping monthly` for temperature data but no time grouping for precipitation -(see [method_qdc.md](method_qdc.md) for an explanation). - -Putting these options together with the positional arguments (the historical variable, reference variable and output adjustment factor file name) looks as follows: - -``` -python train.py pr pr /g/data/wp00/data/QQ-CMIP6/ACCESS-ESM1-5/historical/r1i1p1f1/day/pr/v20191115/pr-qdc-multiplicative-monthly-q100-adjustment-factors_ACCESS-ESM1-5_ssp370_r1i1p1f1_gn_20560101-20851231_wrt_19950101-20141231.nc --hist_files /g/data/fs38/publications/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_historical_r1i1p1f1_gn_19500101-19991231.nc /g/data/fs38/publications/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_historical_r1i1p1f1_gn_20000101-20141231.nc --ref_files /g/data/fs38/publications/CMIP6/ScenarioMIP/CSIRO/ACCESS-ESM1-5/ssp370/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_ssp370_r1i1p1f1_gn_20150101-20641231.nc /g/data/fs38/publications/CMIP6/ScenarioMIP/CSIRO/ACCESS-ESM1-5/ssp370/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_ssp370_r1i1p1f1_gn_20650101-21001231.nc --hist_time_bounds 1995-01-01 2014-12-31 --ref_time_bounds 2056-01-01 2085-12-31 --scaling multiplicative --nquantiles 100 --input_hist_units "kg m-2 s-1" --input_ref_units "kg m-2 s-1" --output_units "mm day-1" --verbose --ssr -``` - -See the [software environment instructions](https://github.com/climate-innovation-hub/qqscale/tree/master#software-environment) for details on the python environment. - -See the [performance notes](https://github.com/climate-innovation-hub/qqscale/tree/master#performance) -for details on the expected time and memory requirements for this training step. - -## Step 2: Adjustment - -The `adjust.py` command line program can be used to apply -the adjustment factors to observations. -You can run `python adjust.py -h` at the command line to view the positional arguments -and options/flags associated with the program. - -The positional arguments for this program are the target model files -(i.e. the data that needs to be bias corrected): -``` -/g/data/xv83/agcd-csiro/precip/daily/precip-total_AGCD-CSIRO_r005_19000101-20220405_daily_space-chunked.zarr -``` -followed by the variable: -``` -precip -``` -adjustment factor file: -``` -/g/data/wp00/data/QQ-CMIP6/ACCESS-ESM1-5/historical/r1i1p1f1/day/pr/v20191115/pr-qdc-multiplicative-monthly-q100-adjustment-factors_ACCESS-ESM1-5_ssp370_r1i1p1f1_gn_20560101-20851231_wrt_19950101-20141231.nc -``` -and name of the output file -(in this case following the CIH -[file naming convenions](https://github.com/climate-innovation-hub/.github/blob/main/drs-qq-cmip6.md)): -``` -/g/data/wp00/data/QQ-CMIP6/ACCESS-ESM1-5/ssp370/r1i1p1f1/day/pr/v20191115/pr_day_ACCESS-ESM1-5_ssp370_r1i1p1f1_AUS-r005_20560101-20851231_qdc-multiplicative-monthly-q100-nearest_AGCD-19900101-20191231_historical-19950101-20141231.nc -``` - -There are a number of options/flags that are also required to complete this task: -- `--spatial_grid input`: The flag specifies whether to output data on the adjustment factor (`af`) or input data (`input`) grid. We want data on the input AGCD grid. -- `--interp nearest`: The method for interpolating between adjustment factors (see [method_qdc.md](method_qdc.md) for an explanation of why we prefer nearest neighbour). -- `--input_units "mm day-1"`, `--output_units "mm day-1"`: Specify input and desired output units and the code will convert units if required. -- `--adjustment_tbounds 1990-01-01 2019-12-31`: Time bounds for the data to be adjusted. -- `--ssr`: Apply Singularity Stochasitc Removal (for precipitation data only; see [method_qdc.md](method_qdc.md) for details). -- `--ref_time`: Apply the time axis from the reference (future) data to the output data. -- `--verbose`: As the program runs it will print progess updates to the screen. - -Putting all these positional and options/flags together: -``` -python adjust.py /g/data/xv83/agcd-csiro/precip/daily/precip-total_AGCD-CSIRO_r005_19000101-20220405_daily_space-chunked.zarr precip /g/data/wp00/data/QQ-CMIP6/ACCESS-ESM1-5/historical/r1i1p1f1/day/pr/v20191115/pr-qdc-multiplicative-monthly-q100-adjustment-factors_ACCESS-ESM1-5_ssp370_r1i1p1f1_gn_20560101-20851231_wrt_19950101-20141231.nc /g/data/wp00/data/QQ-CMIP6/ACCESS-ESM1-5/ssp370/r1i1p1f1/day/pr/v20191115/pr_day_ACCESS-ESM1-5_ssp370_r1i1p1f1_AUS-r005_20560101-20851231_qdc-multiplicative-monthly-q100-nearest_AGCD-19900101-20191231_historical-19950101-20141231.nc --adjustment_tbounds 1990-01-01 2019-12-31 --input_units "mm day-1" --output_units "mm day-1" --spatial_grid input --interp nearest --verbose --ssr --ref_time -``` -See the [software environment instructions](https://github.com/climate-innovation-hub/qqscale/tree/master#software-environment) for details on the python environment. - -See the [performance notes](https://github.com/climate-innovation-hub/qqscale/tree/master#performance) -for details on the expected time and memory requirements for this adjustment step. - -## Step 3: Visualise (Optional) - -It can be useful to visualise the quantile delta mapping process to understand what -modifications have been made to the original data. -A template visualisation notebook is available at: -https://github.com/climate-innovation-hub/qq-workflows/blob/main/validation.ipynb - -The [papermill](https://papermill.readthedocs.io) tool can be used to insert -the various parameters used in the training and adjustment steps into the template notebook -and run it. -For instance, -the example workflow described about can be visualised by running the following at the -command line: - -``` -papermill -p adjustment_file /g/data/wp00/data/QQ-CMIP6/ACCESS-ESM1-5/historical/r1i1p1f1/day/pr/v20191115/pr-qdc-multiplicative-monthly-q100-adjustment-factors_ACCESS-ESM1-5_ssp370_r1i1p1f1_gn_20560101-20851231_wrt_19950101-20141231.nc -p qq_file /g/data/wp00/data/QQ-CMIP6/ACCESS-ESM1-5/ssp370/r1i1p1f1/day/pr/v20191115/pr_day_ACCESS-ESM1-5_ssp370_r1i1p1f1_AUS-r005_20560101-20851231_qdc-multiplicative-monthly-q100-nearest_AGCD-19900101-20191231_historical-19950101-20141231.nc -r hist_files "/g/data/fs38/publications/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_historical_r1i1p1f1_gn_19500101-19991231.nc /g/data/fs38/publications/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_historical_r1i1p1f1_gn_20000101-20141231.nc" -r ref_files "/g/data/fs38/publications/CMIP6/ScenarioMIP/CSIRO/ACCESS-ESM1-5/ssp370/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_ssp370_r1i1p1f1_gn_20150101-20641231.nc /g/data/fs38/publications/CMIP6/ScenarioMIP/CSIRO/ACCESS-ESM1-5/ssp370/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_ssp370_r1i1p1f1_gn_20650101-21001231.nc" -r target_files "/g/data/xv83/agcd-csiro/precip/daily/precip-total_AGCD-CSIRO_r005_19000101-20220405_daily_space-chunked.zarr" -r hist_time_bounds "1995-01-01 2014-12-31" -r ref_time_bounds "2056-01-01 2085-12-31" -r target_time_bounds "1990-01-01 2019-12-31" -p hist_units "kg m-2 s-1" -p ref_units "kg m-2 s-1" -p target_units "mm day-1" -p output_units "mm day-1" -p hist_var pr -p ref_var pr -p target_var precip -p scaling multiplicative validation.ipynb /g/data/wp00/data/QQ-CMIP6/ACCESS-ESM1-5/ssp370/r1i1p1f1/day/pr/v20191115/pr_day_ACCESS-ESM1-5_ssp370_r1i1p1f1_AUS-r005_20560101-20851231_qdc-multiplicative-monthly-q100-nearest_AGCD-19900101-20191231_historical-19950101-20141231.ipynb -``` - -The result is a new notebook called: -``` -/g/data/wp00/data/QQ-CMIP6/ACCESS-ESM1-5/ssp370/r1i1p1f1/day/pr/v20191115/pr_day_ACCESS-ESM1-5_ssp370_r1i1p1f1_AUS-r005_20560101-20851231_qdc-multiplicative-monthly-q100-nearest_AGCD-19900101-20191231_historical-19950101-20141231.ipynb -``` - -You can view an example of what such a notebook looks like -[here](https://github.com/climate-innovation-hub/qq-workflows/tree/main/cih). - -Feel free to open an issue at the qq-workflows repository -if you'd like help creating a configuration file to use make for your workflow. diff --git a/docs/example_acs.md b/docs/example_ecdfm.md similarity index 51% rename from docs/example_acs.md rename to docs/example_ecdfm.md index 64d781e..5251d45 100644 --- a/docs/example_acs.md +++ b/docs/example_ecdfm.md @@ -15,13 +15,6 @@ the central decacde (2060-2069) and then move on and process the 2060-2089 window (keeping 2070-2079). -The following example describes how to process the 2050-2079 window using the -equidistant CDF matching (EDCDFm) bias correction method -(see [method_ecdfm.md](method_ecdfm.md) for details) -for a given variable (tasmin), -regional climate model (BOM-BARPA-R), and -parent global climate model (CMCC-CMCC-ESM2). - > **Shortcut** > > You'll see in steps 1-2 below that there are lots of @@ -30,28 +23,16 @@ parent global climate model (CMCC-CMCC-ESM2). > > If you don't already have a process for managing complicated workflows like this, > using a build tool like Make can be helpful. -> The Makefile and ACS configuation file at https://github.com/climate-innovation-hub/qq-workflows -> have been built to coordinate the bias correction workflow described below. +> The Make-based workflow at https://github.com/AusClimateService/qq-workflows/tree/main/acs +> have been built to coordinate the bias correction workflow described above. > -> To run steps 1-2 below, -> you could simply clone the qq-workflows repo and run the following at the command line: -> ``` -> git clone git@github.com:climate-innovation-hub/qq-workflows.git -> cd qq-workflows -> make adjust CONFIG=acs/config_acs.mk VAR=tasmin RCM_NAME=BOM-BARPA-R GCM_NAME=CMCC-CMCC-ESM2 TARGET_START=2050 TARGET_END=2079 OUTPUT_START=2060 OUTPUT_END=2069 -> ``` - -> **Full timeseries** -> -> The `bias_correct_timeseries.sh` shell script in the qq-workflows repo -> runs the `make` command for sequential 30-year windows in 10-year increments -> from 1960 to 2100. -> -> For example, in this case we would run the following to process -> the entire 1960-2100 timeseries: -> ``` -> bash bias_correct_timeseries.sh tasmin BOM-BARPA-R CMCC-CMCC-ESM2 -> ``` + +The following example describes how to process the 2050-2079 window using the +equidistant CDF matching (EDCDFm) bias correction method +(see [method_ecdfm.md](method_ecdfm.md) for details) +for a given variable (tasmin), +regional climate model (BOM-BARPA-R), and +parent global climate model (CMCC-CMCC-ESM2). ## Step 1: Training @@ -67,9 +48,9 @@ The `--hist_files` flag is used to specify the historical model data files (in this case for the 1985-2014 reference period): ``` --hist_files -/g/data/ia39/australian-climate-service/release/CORDEX-CMIP6/output/AUS-15/BOM/CMCC-CMCC-ESM2/historical/r1i1p1f1/BOM-BARPA-R/v1/day/tasmin/tasmin_AUS-15_CMCC-CMCC-ESM2_historical_r1i1p1f1_BOM-BARPA-R_v1_day_198[5,6,7,8,9]*.nc -/g/data/ia39/australian-climate-service/release/CORDEX-CMIP6/output/AUS-15/BOM/CMCC-CMCC-ESM2/historical/r1i1p1f1/BOM-BARPA-R/v1/day/tasmin/tasmin_AUS-15_CMCC-CMCC-ESM2_historical_r1i1p1f1_BOM-BARPA-R_v1_day_199*.nc -/g/data/ia39/australian-climate-service/release/CORDEX-CMIP6/output/AUS-15/BOM/CMCC-CMCC-ESM2/historical/r1i1p1f1/BOM-BARPA-R/v1/day/tasmin/tasmin_AUS-15_CMCC-CMCC-ESM2_historical_r1i1p1f1_BOM-BARPA-R_v1_day_2*.nc +/g/data/py18/BARPA/output/CMIP6/DD/AUS-15/BOM/CMCC-ESM2/historical/r1i1p1f1/BARPA-R/v1-r1/day/tasmin/v20231001/tasmin_AUS-15_CMCC-ESM2_historical_r1i1p1f1_BOM_BARPA-R_v1-r1_day_198[5,6,7,8,9]*.nc +/g/data/py18/BARPA/output/CMIP6/DD/AUS-15/BOM/CMCC-ESM2/historical/r1i1p1f1/BARPA-R/v1-r1/day/tasmin/v20231001/tasmin_AUS-15_CMCC-ESM2_historical_r1i1p1f1_BOM_BARPA-R_v1-r1_day_199*.nc +/g/data/py18/BARPA/output/CMIP6/DD/AUS-15/BOM/CMCC-ESM2/historical/r1i1p1f1/BARPA-R/v1-r1/day/tasmin/v20231001/tasmin_AUS-15_CMCC-ESM2_historical_r1i1p1f1_BOM_BARPA-R_v1-r1_day_2*.nc ``` The `--ref_files` flag is used to specify the reference observational data files: @@ -97,13 +78,11 @@ There are also some flags that aren't necessarily needed in this case but can be Putting these options together with the positional arguments (the historical variable, reference variable and output adjustment factor file name) looks as follows: ``` -python train.py tasmin tmin /g/data/ia39/australian-climate-service/test-data/CORDEX-CMIP6-ECDFm/output/AUS-05i/BOM/CMCC-CMCC-ESM2/ssp370/r1i1p1f1/BOM-BARPA-R/v1/day/tasmin/tasmin-ecdfm-additive-monthly-q100-adjustment-factors_AGCD_AUS-05i_CMCC-CMCC-ESM2_historical_r1i1p1f1_BOM-BARPA-R_v1_day_19850101-20141231.nc --hist_files /g/data/ia39/australian-climate-service/release/CORDEX-CMIP6/output/AUS-15/BOM/CMCC-CMCC-ESM2/historical/r1i1p1f1/BOM-BARPA-R/v1/day/tasmin/tasmin_AUS-15_CMCC-CMCC-ESM2_historical_r1i1p1f1_BOM-BARPA-R_v1_day_198[5,6,7,8,9]*.nc /g/data/ia39/australian-climate-service/release/CORDEX-CMIP6/output/AUS-15/BOM/CMCC-CMCC-ESM2/historical/r1i1p1f1/BOM-BARPA-R/v1/day/tasmin/tasmin_AUS-15_CMCC-CMCC-ESM2_historical_r1i1p1f1_BOM-BARPA-R_v1_day_199*.nc /g/data/ia39/australian-climate-service/release/CORDEX-CMIP6/output/AUS-15/BOM/CMCC-CMCC-ESM2/historical/r1i1p1f1/BOM-BARPA-R/v1/day/tasmin/tasmin_AUS-15_CMCC-CMCC-ESM2_historical_r1i1p1f1_BOM-BARPA-R_v1_day_2*.nc --ref_files /g/data/xv83/agcd-csiro/tmin/daily/tmin_AGCD-CSIRO_r005_198[5,6,7,8,9]*.nc /g/data/xv83/agcd-csiro/tmin/daily/tmin_AGCD-CSIRO_r005_199*.nc /g/data/xv83/agcd-csiro/tmin/daily/tmin_AGCD-CSIRO_r005_200*.nc /g/data/xv83/agcd-csiro/tmin/daily/tmin_AGCD-CSIRO_r005_201[0,1,2,3,4]*.nc --hist_time_bounds 1985-01-01 2014-12-31 --ref_time_bounds 1985-01-01 2014-12-31 --scaling additive --nquantiles 100 --time_grouping monthly --input_hist_units K --input_ref_units C --output_units C --verbose +python train.py tasmin tmin tasmin-ecdfm-additive-monthly-q100-adjustment-factors_AGCD_AUS-05i_CMCC-ESM2_historical_r1i1p1f1_BARPA-R_v1_day_19850101-20141231.nc --hist_files /g/data/py18/BARPA/output/CMIP6/DD/AUS-15/BOM/CMCC-ESM2/historical/r1i1p1f1/BARPA-R/v1-r1/day/tasmin/v20231001/tasmin_AUS-15_CMCC-ESM2_historical_r1i1p1f1_BOM_BARPA-R_v1-r1_day_198[5,6,7,8,9]*.nc /g/data/py18/BARPA/output/CMIP6/DD/AUS-15/BOM/CMCC-ESM2/historical/r1i1p1f1/BARPA-R/v1-r1/day/tasmin/v20231001/tasmin_AUS-15_CMCC-ESM2_historical_r1i1p1f1_BOM_BARPA-R_v1-r1_day_199*.nc /g/data/py18/BARPA/output/CMIP6/DD/AUS-15/BOM/CMCC-ESM2/historical/r1i1p1f1/BARPA-R/v1-r1/day/tasmin/v20231001/tasmin_AUS-15_CMCC-ESM2_historical_r1i1p1f1_BOM_BARPA-R_v1-r1_day_2*.nc --ref_files /g/data/xv83/agcd-csiro/tmin/daily/tmin_AGCD-CSIRO_r005_198[5,6,7,8,9]*.nc /g/data/xv83/agcd-csiro/tmin/daily/tmin_AGCD-CSIRO_r005_199*.nc /g/data/xv83/agcd-csiro/tmin/daily/tmin_AGCD-CSIRO_r005_200*.nc /g/data/xv83/agcd-csiro/tmin/daily/tmin_AGCD-CSIRO_r005_201[0,1,2,3,4]*.nc --hist_time_bounds 1985-01-01 2014-12-31 --ref_time_bounds 1985-01-01 2014-12-31 --scaling additive --nquantiles 100 --time_grouping monthly --input_hist_units K --input_ref_units C --output_units C --verbose ``` -See the [software environment instructions](https://github.com/climate-innovation-hub/qqscale/tree/master#software-environment) for details on the python environment. +See the [software environment instructions](https://github.com/AusClimateService/qqscale/tree/master#software-environment) for details on the python environment. -See the [performance notes](https://github.com/climate-innovation-hub/qqscale/tree/master#performance) -for details on the expected time and memory requirements for this training step. ## Step 2: Adjustment @@ -115,7 +94,7 @@ and options/flags associated with the program. The positional arguments for this program are the target model files (i.e. the data that needs to be bias corrected): ``` -/g/data/ia39/australian-climate-service/release/CORDEX-CMIP6/output/AUS-15/BOM/CMCC-CMCC-ESM2/ssp370/r1i1p1f1/BOM-BARPA-R/v1/day/tasmin/tasmin_AUS-15_CMCC-CMCC-ESM2_ssp370_r1i1p1f1_BOM-BARPA-R_v1_day_20[5,6,7]*.nc +/g/data/py18/BARPA/output/CMIP6/DD/AUS-15/BOM/CMCC-ESM2/ssp370/r1i1p1f1/BARPA-R/v1-r1/day/tasmin/v20231001/tasmin_AUS-15_CMCC-ESM2_ssp370_r1i1p1f1_BOM_BARPA-R_v1-r1_day_20[5,6,7]*.nc ``` followed by the variable: ``` @@ -123,13 +102,13 @@ tasmin ``` adjustment factor file: ``` -/g/data/ia39/australian-climate-service/test-data/CORDEX-CMIP6-ECDFm/output/AUS-05i/BOM/CMCC-CMCC-ESM2/ssp370/r1i1p1f1/BOM-BARPA-R/v1/day/tasmin/tasmin-ecdfm-additive-monthly-q100-adjustment-factors_AGCD_AUS-05i_CMCC-CMCC-ESM2_historical_r1i1p1f1_BOM-BARPA-R_v1_day_19850101-20141231.nc +tasmin-ecdfm-additive-monthly-q100-adjustment-factors_AGCD_AUS-05i_CMCC-ESM2_historical_r1i1p1f1_BARPA-R_v1_day_19850101-20141231.nc ``` and name of the output bias corrected data file (in this case following the ACS Data and Code Group [file naming convenions](https://github.com/AusClimateService/data-code-group/blob/main/data_standards.md#cordex-cmip6)): ``` -/g/data/ia39/australian-climate-service/test-data/CORDEX-CMIP6-ECDFm/output/AUS-05i/BOM/CMCC-CMCC-ESM2/ssp370/r1i1p1f1/BOM-BARPA-R/v1/day/tasmin/tasmin_AUS-05i_CMCC-CMCC-ESM2_ssp370_r1i1p1f1_BOM-BARPA-R_v1_day_20600101-20691231-from-20500101-20791231_ecdfm-additive-monthly-q100-nearest-AGCD-19850101-20141231.nc +tasmin_AUS-05i_CMCC-ESM2_ssp370_r1i1p1f1_BARPA-R_v1_day_20600101-20691231-from-20500101-20791231_ecdfm-additive-monthly-q100-nearest-AGCD-19850101-20141231.nc ``` There are a number of options/flags that are also required to complete this task: @@ -144,12 +123,9 @@ As for the training step, if we were processing precipitation data we would also Putting all these positional and options/flags together: ``` -python adjust.py /g/data/ia39/australian-climate-service/release/CORDEX-CMIP6/output/AUS-15/BOM/CMCC-CMCC-ESM2/ssp370/r1i1p1f1/BOM-BARPA-R/v1/day/tasmin/tasmin_AUS-15_CMCC-CMCC-ESM2_ssp370_r1i1p1f1_BOM-BARPA-R_v1_day_20[5,6,7]*.nc tasmin /g/data/ia39/australian-climate-service/test-data/CORDEX-CMIP6-ECDFm/output/AUS-05i/BOM/CMCC-CMCC-ESM2/ssp370/r1i1p1f1/BOM-BARPA-R/v1/day/tasmin/tasmin-ecdfm-additive-monthly-q100-adjustment-factors_AGCD_AUS-05i_CMCC-CMCC-ESM2_historical_r1i1p1f1_BOM-BARPA-R_v1_day_19850101-20141231.nc /g/data/ia39/australian-climate-service/test-data/CORDEX-CMIP6-ECDFm/output/AUS-05i/BOM/CMCC-CMCC-ESM2/ssp370/r1i1p1f1/BOM-BARPA-R/v1/day/tasmin/tasmin_AUS-05i_CMCC-CMCC-ESM2_ssp370_r1i1p1f1_BOM-BARPA-R_v1_day_20500101-20791231-from-20500101-20791231_ecdfm-additive-monthly-q100-nearest-AGCD-19850101-20141231.nc --adjustment_tbounds 2050-01-01 2079-12-31 --output_tslice 2060-01-01 2069-12-310. --input_units K --output_units C --spatial_grid af --interp nearest --verbose +python adjust.py /g/data/py18/BARPA/output/CMIP6/DD/AUS-15/BOM/CMCC-ESM2/ssp370/r1i1p1f1/BARPA-R/v1-r1/day/tasmin/v20231001/tasmin_AUS-15_CMCC-ESM2_ssp370_r1i1p1f1_BOM_BARPA-R_v1-r1_day_20[5,6,7]*.nc tasmin tasmin-ecdfm-additive-monthly-q100-adjustment-factors_AGCD_AUS-05i_CMCC-CMCC-ESM2_historical_r1i1p1f1_BOM-BARPA-R_v1_day_19850101-20141231.nc tasmin_AUS-05i_CMCC-ESM2_ssp370_r1i1p1f1_BARPA-R_v1_day_20500101-20791231-from-20500101-20791231_ecdfm-additive-monthly-q100-nearest-AGCD-19850101-20141231.nc --adjustment_tbounds 2050-01-01 2079-12-31 --output_tslice 2060-01-01 2069-12-31 --input_units K --output_units C --spatial_grid af --interp nearest --verbose ``` -See the [software environment instructions](https://github.com/climate-innovation-hub/qqscale/tree/master#software-environment) for details on the python environment. - -See the [performance notes](https://github.com/climate-innovation-hub/qqscale/tree/master#performance) -for details on the expected time and memory requirements for this adjustment step. +See the [software environment instructions](https://github.com/AusClimateService/qqscale/tree/master#software-environment) for details on the python environment. -Feel free to open an issue at the qq-workflows repository +Feel free to open an issue at the [qq-workflows repository](https://github.com/AusClimateService/qq-workflows) if you'd like help creating a configuration file to use make for your workflow. diff --git a/docs/example_npcp.md b/docs/example_npcp.md deleted file mode 100644 index 6003bcf..0000000 --- a/docs/example_npcp.md +++ /dev/null @@ -1,158 +0,0 @@ -# Worked example: NPCP - -The [NPCP bias correction intercomparison](https://github.com/AusClimateService/npcp) -includes a cross validation task that involves producing bias corrected data -for even years from 1980-2019 (i.e. every second year), -using odd years from 1980-2019 as training data. - -The following example describes how to complete this task using the -equidistant CDF matching (EDCDFm) bias correction method -(see [method_ecdfm.md](method_ecdfm.md) for details) -for a given variable (tasmin), -regional climate model (UQ-DES-CCAM-2105), and -parent global climate model (CSIRO-ACCESS-ESM1-5). - -> **Shortcut** -> -> You'll see in steps 1-3 below that there are lots of -> positional arguments and program options/flags to remember, -> which means it can be easy to forget something or make an error. -> -> If you don't already have a process for managing complicated workflows like this, -> using a build tool like Make can be helpful. -> The Makefile and NPCP configuation file at https://github.com/climate-innovation-hub/qq-workflows -> have been built to coordinate the bias correction workflow described below. -> -> To run steps 1-3 below, -> you could simply clone the qq-workflows repo and run the following at the command line: -> ``` -> git clone git@github.com:climate-innovation-hub/qq-workflows.git -> cd qq-workflows -> make validation CONFIG=npcp/config_npcp.mk VAR=tasmin TASK=xvalidation RCM_NAME=UQ-DES-CCAM-2105 GCM_NAME=CSIRO-ACCESS-ESM1-5 -> ``` - -## Step 1: Training - -The `train.py` command line program can be used to compare historical model data with reference observational data -in order to produce the bias correction "adjustment factors" needed for the adjustment step. -You can run `python train.py -h` at the command line to view the positional arguments -and options/flags associated with the program. -To give an example, -the options/flags for the NPCP cross validation task would be as follows... - -The `--hist_files` flag is used to specify the historical model data files -(in this case for odd years from 1980-2019): -``` ---hist_files -/g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19?[1,3,5,7,9]*.nc -/g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_200[1,3,5,7,9]*.nc -/g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_*_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_201[1,3,5,7,9]*.nc -``` - -The `--ref_files` flag is used to specify the reference observational data files -(also odd years from 1980-2019): - -``` ---ref_files -/g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_19?[1,3,5,7,9]*.nc -/g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_200[1,3,5,7,9]*.nc -/g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_201[1,3,5,7,9]*.nc -``` - -There are a number of other flags that are also required to complete this task: -- `--scaling additive`: Additive scaling is typically used for temperature (and `multiplicative` for rainfall). -- `--time_grouping monthly`: We like to use monthly time grouping (see [method_ecdfm.md](method_ecdfm.md) for an explanation). -- `--nquantiles 100`: We like to use 100 quantiles. - -There are also some flags that aren't necessarily needed in this case but can be useful: -- `--hist_time_bounds 1980-01-01 2019-12-31`: Time bounds for historical period (i.e. for if your input files span a longer time period than required). -- `--ref_time_bounds 1980-01-01 2019-12-31`: Time bounds for reference period (i.e. for if your input files span a longer time period than required). -- `--input_hist_units C`, `--input_ref_units C`, `--output_units C`: Specify input and desired output units and the code will convert units if required. -- `--ssr`: If we were processing precipitation data we would use this flag to apply Singularity Stochasitc Removal (see [method_ecdfm.md](method_ecdfm.md) for details). -- `--verbose`: As the program runs it will print progess updates to the screen. - -Putting these options together with the positional arguments (the historical variable, reference variable and output adjustment factor file name) looks as follows: - -``` -python train.py tasmin tasmin /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/ecdfm/task-xvalidation/tasmin-ecdfm-additive-monthly-q100-adjustment-factors_AGCD_NPCP-20i_CSIRO-ACCESS-ESM1-5_ssp370_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19810101-20191231-odd-years.nc --hist_files /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19?[1,3,5,7,9]*.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_200[1,3,5,7,9]*.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_*_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_201[1,3,5,7,9]*.nc --ref_files /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_19?[1,3,5,7,9]*.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_200[1,3,5,7,9]*.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_201[1,3,5,7,9]*.nc --hist_time_bounds 1980-01-01 2019-12-31 --ref_time_bounds 1980-01-01 2019-12-31 --scaling additive --nquantiles 100 --time_grouping monthly --input_hist_units C --input_ref_units C --output_units C --verbose -``` -See the [software environment instructions](https://github.com/climate-innovation-hub/qqscale/tree/master#software-environment) for details on the python environment. - -See the [performance notes](https://github.com/climate-innovation-hub/qqscale/tree/master#performance) -for details on the expected time and memory requirements for this training step. - -## Step 2: Adjustment - -The `adjust.py` command line program can be used to bias correct model data using -the adjustment factors calculated in the previous step. -You can run `python adjust.py -h` at the command line to view the positional arguments -and options/flags associated with the program. - -The positional arguments for this program are the target model files -(i.e. the data that needs to be bias corrected): -``` -/g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19?[0,2,4,6,8]*.nc -/g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_200[0,2,4,6,8]*.nc -/g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_201[0,2,4,6,8]*.nc -``` -followed by the variable: -``` -tasmin -``` -adjustment factor file: -``` -/g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/ecdfm/task-xvalidation/tasmin-ecdfm-additive-monthly-q100-adjustment-factors_AGCD_NPCP-20i_CSIRO-ACCESS-ESM1-5_ssp370_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19810101-20191231-odd-years.nc -``` -and name of the output bias corrected data file -(in this case following the file naming convenions of the NPCP project): -``` -/g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/ecdfm/task-xvalidation/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_ssp370_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19800101-20181231-even-years_ecdfm-additive-monthly-q100-nearest-AGCD-19810101-20191231-odd-years.nc -``` - -There are a number of options/flags that are also required to complete this task: -- `--spatial_grid af`: The flag specifies whether to output data on the adjustment factor (`af`) or input data (`input`) grid. In this case both grids are the same, but in general we want data on the adjustment grid because that's the observational (as opposed to model) grid. -- `--interp nearest`: The method for interpolating between adjustment factors (see [method_ecdfm.md](method_ecdfm.md) for an explanation of why we prefer nearest neighbour). - -There are also some flags that aren't necessarily needed in this case but can be useful: -- `--adjustment_tbounds 1980-01-01 2019-12-31`: Time bounds to perform the bias correction over (i.e. for if your input files span a longer time period than required). -- `--input_units C`, `--output_units C`: Specify input and desired output units and the code will convert units if required. -- `--ssr`: As for the training step, if we were processing precipitation data we would use this flag to apply Singularity Stochasitc Removal (see [method_ecdfm.md](method_ecdfm.md) for details). -- `--verbose`: As the program runs it will print progess updates to the screen. - -Putting all these positional and options/flags together: -``` -python adjust.py /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19?[0,2,4,6,8]*.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_200[0,2,4,6,8]*.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_201[0,2,4,6,8]*.nc tasmin /g/data/xv83/dbi599/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/ecdfm/task-xvalidation/tasmin-ecdfm-additive-monthly-q100-adjustment-factors_AGCD_NPCP-20i_CSIRO-ACCESS-ESM1-5_ssp370_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19810101-20191231-odd-years.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/ecdfm/task-xvalidation/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_ssp370_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19800101-20181231-even-years_ecdfm-additive-monthly-q100-nearest-AGCD-19810101-20191231-odd-years.nc --adjustment_tbounds 1980-01-01 2018-12-31 --input_units C --output_units C --spatial_grid af --interp nearest --verbose -``` -See the [software environment instructions](https://github.com/climate-innovation-hub/qqscale/tree/master#software-environment) for details on the python environment. - -See the [performance notes](https://github.com/climate-innovation-hub/qqscale/tree/master#performance) -for details on the expected time and memory requirements for this adjustment step. - -## Step 3: Visualise (Optional) - -It can be useful to visualise the bias correction process to understand what -modifications have been made to the original data. -A template visualisation notebook is available at: -https://github.com/climate-innovation-hub/qq-workflows/blob/main/validation.ipynb - -The [papermill](https://papermill.readthedocs.io) tool can be used to insert -the various parameters used in the training and adjustment steps into the template notebook -and run it. -For instance, -the example workflow described about can be visualised by running the following at the -command line: - -``` -papermill -p adjustment_file /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/ecdfm/task-xvalidation/tasmin-ecdfm-additive-monthly-q100-adjustment-factors_AGCD_NPCP-20i_CSIRO-ACCESS-ESM1-5_ssp370_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19810101-20191231-odd-years.nc -p qq_file /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/ecdfm/task-xvalidation/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_ssp370_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19800101-20181231-even-years_ecdfm-additive-monthly-q100-nearest-AGCD-19810101-20191231-odd-years.nc -r hist_files "/g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19810101-19811231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19830101-19831231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19850101-19851231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19870101-19871231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19890101-19891231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19910101-19911231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19930101-19931231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19950101-19951231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19970101-19971231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19990101-19991231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20010101-20011231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20030101-20031231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20050101-20051231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20070101-20071231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20090101-20091231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20110101-20111231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20130101-20131231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_ssp370_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20150101-20151231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_ssp370_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20170101-20171231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_ssp370_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20190101-20191231.nc" -r ref_files "/g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_19810101-19811231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_19830101-19831231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_19850101-19851231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_19870101-19871231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_19890101-19891231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_19910101-19911231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_19930101-19931231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_19950101-19951231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_19970101-19971231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_19990101-19991231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_20010101-20011231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_20030101-20031231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_20050101-20051231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_20070101-20071231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_20090101-20091231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_20110101-20111231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_20130101-20131231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_20150101-20151231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_20170101-20171231.nc /g/data/ia39/npcp/data/tasmin/observations/AGCD/raw/task-reference/tasmin_NPCP-20i_AGCD_v1-0-1_day_20190101-20191231.nc" -r target_files "/g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19800101-19801231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19820101-19821231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19840101-19841231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19860101-19861231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19880101-19881231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19900101-19901231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19920101-19921231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19940101-19941231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19960101-19961231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19980101-19981231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20000101-20001231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20020101-20021231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20040101-20041231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20060101-20061231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20080101-20081231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20100101-20101231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20120101-20121231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_historical_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20140101-20141231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_ssp370_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20160101-20161231.nc /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/raw/task-reference/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_ssp370_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_20180101-20181231.nc" -r hist_time_bounds "1981-01-01 2019-12-31" -r ref_time_bounds "1981-01-01 2019-12-31" -r target_time_bounds "1980-01-01 2018-12-31" -p hist_units C -p ref_units C -p target_units C -p output_units C -p hist_var tasmin -p ref_var tasmin -p target_var tasmin -p scaling additive -p method ecdfm validation.ipynb /g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/ecdfm/task-xvalidation/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_ssp370_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19800101-20181231-even-years_ecdfm-additive-monthly-q100-nearest-AGCD-19810101-20191231-odd-years.ipynb -``` - -The result is a new notebook called: -``` -/g/data/ia39/npcp/data/tasmin/CSIRO-ACCESS-ESM1-5/UQ-DES-CCAM-2105/ecdfm/task-xvalidation/tasmin_NPCP-20i_CSIRO-ACCESS-ESM1-5_ssp370_r6i1p1f1_UQ-DES-CCAM-2105_v1_day_19800101-20181231-even-years_ecdfm-additive-monthly-q100-nearest-AGCD-19810101-20191231-odd-years.ipynb` -``` - -You can view an example of what such a notebook looks like -[here](https://github.com/climate-innovation-hub/qq-workflows/tree/main/npcp). - -Feel free to open an issue at the qq-workflows repository -if you'd like help creating a configuration file to use make for your workflow. diff --git a/docs/example_qdc.mk b/docs/example_qdc.mk new file mode 100644 index 0000000..37b571e --- /dev/null +++ b/docs/example_qdc.mk @@ -0,0 +1,131 @@ +# Worked example: QDC-CMIP6 + +The CSIRO Climate Innovation Hub intends produce +application ready climate projections data +by applying relative changes simulated by CMIP6 global climate models +to observational data. + +The following example describes how to produce projections data +for the 2035-2064 period using the Quantile Delta Change (QDC) method +(see [method_qdc.md](method_qdc.md) for details) +for a given variable (pr), +global climate model (ACCESS-ESM1-5), +model experiment (ssp370), +model run (r1i1p1f1), +observation reference period (1985-2014) +and historical model baseline (1985-2014). + +> **Shortcut** +> +> You'll see in steps 1-3 below that there are lots of +> positional arguments and program options/flags to remember, +> which means it can be easy to forget something or make an error. +> +> If you don't already have a process for managing complicated workflows like this, +> using a build tool like Make can be helpful. +> The Make-based workflow at https://github.com/AusClimateService/qq-workflows/tree/main/qdc-cmip6 +> has been built to coordinate the workflow described above. +> + +## Step 1: Training + +The `train.py` command line program can be used to compare future and historical model data +in order to produce the "adjustment factors" that need to be applied to observations in the adjustment step. +You can run `python train.py -h` at the command line to view the positional arguments +and options/flags associated with the program. +To give an example, +the options/flags for the QDC-CMIP6 projections described above would be as follows... + +The `--hist_files` flag is used to specify the historical model data files +(in this case for the 1985-2014 reference period): +``` +--hist_files +/g/data/fs38/publications/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_historical_r1i1p1f1_gn_19500101-19991231.nc +/g/data/fs38/publications/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_historical_r1i1p1f1_gn_20000101-20141231.nc +``` + +The `--ref_files` flag is used to specify the reference ssp370 data files: + +``` +--ref_files +/g/data/fs38/publications/CMIP6/ScenarioMIP/CSIRO/ACCESS-ESM1-5/ssp370/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_ssp370_r1i1p1f1_gn_20150101-20641231.nc +``` + +There are a number of other flags that are also required to complete this task: +- `--scaling multiplicative`: Multiplicative scaling is typically used for rainfall (and `additive` for temperature). +- `--nquantiles 100`: We like to use 100 quantiles. +- `--input_hist_units "kg m-2 s-1"`, `--input_ref_units "kg m-2 s-1"`, `--output_units "mm day-1"`: Specify input and desired output units and the code will convert units if required. +- `--hist_time_bounds 1985-01-01 2014-12-31`: Time bounds for historical period. +- `--ref_time_bounds 2035-01-01 2064-12-31`: Time bounds for reference period. +- `--ssr`: Apply Singularity Stochasitc Removal (for precipitation data). +- `--verbose`: As the program runs it will print progess updates to the screen. + +We would use the `--time_grouping monthly` for temperature data but no time grouping for precipitation +(see [method_qdc.md](method_qdc.md) for an explanation). + +Putting these options together with the positional arguments (the historical variable, reference variable and output adjustment factor file name) looks as follows: + +``` +python train.py pr pr pr-qdc-multiplicative-monthly-q100-adjustment-factors_ACCESS-ESM1-5_ssp370_r1i1p1f1_gn_20350101-20641231_wrt_19850101-20141231.nc --hist_files /g/data/fs38/publications/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_historical_r1i1p1f1_gn_19500101-19991231.nc /g/data/fs38/publications/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_historical_r1i1p1f1_gn_20000101-20141231.nc --ref_files /g/data/fs38/publications/CMIP6/ScenarioMIP/CSIRO/ACCESS-ESM1-5/ssp370/r1i1p1f1/day/pr/gn/v20191115/pr_day_ACCESS-ESM1-5_ssp370_r1i1p1f1_gn_20150101-20641231.nc --hist_time_bounds 1985-01-01 2014-12-31 --ref_time_bounds 2035-01-01 2064-12-31 --scaling multiplicative --nquantiles 100 --input_hist_units "kg m-2 s-1" --input_ref_units "kg m-2 s-1" --output_units "mm day-1" --verbose --ssr +``` + +See the [software environment instructions](https://github.com/AusClimateService/qqscale/tree/master#software-environment) for details on the python environment. + + +## Step 2: Adjustment + +The `adjust.py` command line program can be used to apply +the adjustment factors to observations. +You can run `python adjust.py -h` at the command line to view the positional arguments +and options/flags associated with the program. + +The positional arguments for this program are the target model files +(i.e. the data that needs to be bias corrected): +``` +/g/data/xv83/agcd-csiro/precip/daily/precip-total_AGCD-CSIRO_r005_19000101-20220405_daily_space-chunked.zarr +``` +followed by the variable: +``` +precip +``` +adjustment factor file: +``` +pr-qdc-multiplicative-monthly-q100-adjustment-factors_ACCESS-ESM1-5_ssp370_r1i1p1f1_gn_20350101-20641231_wrt_19850101-20141231.nc +``` +and name of the output file +(in this case following the CIH +[file naming convenions](https://github.com/climate-innovation-hub/.github/blob/main/drs-qq-cmip6.md)): +``` +pr_day_ACCESS-ESM1-5_ssp370_r1i1p1f1_AUS-r005_20350101-20641231_qdc-multiplicative-monthly-q100-nearest_AGCD-19850101-20141231_historical-19850101-20141231.nc +``` + +There are a number of options/flags that are also required to complete this task: +- `--spatial_grid input`: The flag specifies whether to output data on the adjustment factor (`af`) or input data (`input`) grid. We want data on the input AGCD grid. +- `--interp nearest`: The method for interpolating between adjustment factors (see [method_qdc.md](method_qdc.md) for an explanation of why we prefer nearest neighbour). +- `--input_units "mm day-1"`, `--output_units "mm day-1"`: Specify input and desired output units and the code will convert units if required. +- `--adjustment_tbounds 1985-01-01 2014-12-31`: Time bounds for the data to be adjusted. +- `--ssr`: Apply Singularity Stochasitc Removal (for precipitation data only; see [method_qdc.md](method_qdc.md) for details). +- `--ref_time`: Apply the time axis from the reference (future) data to the output data. +- `--verbose`: As the program runs it will print progess updates to the screen. + +Putting all these positional and options/flags together: +``` +python adjust.py /g/data/xv83/agcd-csiro/precip/daily/precip-total_AGCD-CSIRO_r005_19000101-20220405_daily_space-chunked.zarr precip pr-qdc-multiplicative-monthly-q100-adjustment-factors_ACCESS-ESM1-5_ssp370_r1i1p1f1_gn_20350101-20641231_wrt_19850101-20141231.nc pr_day_ACCESS-ESM1-5_ssp370_r1i1p1f1_AUS-r005_20350101-20641231_qdc-multiplicative-monthly-q100-nearest_AGCD-19850101-20141231_historical-19850101-20141231.nc --adjustment_tbounds 1985-01-01 2014-12-31 --input_units "mm day-1" --output_units "mm day-1" --spatial_grid input --interp nearest --verbose --ssr --ref_time +``` +See the [software environment instructions](https://github.com/AusClimateService/qqscale/tree/master#software-environment) for details on the python environment. + + +## Step 3: Visualise (Optional) + +It can be useful to visualise the quantile delta mapping process to understand what +modifications have been made to the original data. +A template visualisation notebook is available at: +https://github.com/AusClimateService/qq-workflows/blob/main/validation.ipynb + +The [papermill](https://papermill.readthedocs.io) tool can be used to insert +the various parameters used in the training and adjustment steps into the template notebook +and run it. + + +Feel free to open an issue at the [qq-workflows repository](https://github.com/AusClimateService/qq-workflows) +if you'd like help creating a configuration file to use make for your workflow. diff --git a/docs/method_qdc.md b/docs/method_qdc.md index e4365bc..03f6402 100644 --- a/docs/method_qdc.md +++ b/docs/method_qdc.md @@ -68,7 +68,8 @@ There are a number of choices to make when implementing QDC: We've found that something like a 30-day running window is far more computationally expensive and produces similar results to monthly grouping. When processing precipitation data (a multiplicative application of QDC) - we've found ([see rough notebook](https://github.com/climate-innovation-hub/qq-workflows/blob/main/qdm-vs-ecdfm/seasonal_cycle.ipynb)) + we've found (see [simple](https://github.com/AusClimateService/qq-workflows/blob/main/qdm-vs-ecdfm/seasonal_cycle_simple.ipynb) + and [more complex](https://github.com/climate-innovation-hub/qq-workflows/blob/main/qdm-vs-ecdfm/seasonal_cycle.ipynb) notebooks) that in many locations the model bias in the timing of the seasonal cycle means that monthly time grouping dramatically modifies the climate trend in the data (i.e. the mean change between the future data produced by QDC and the observations diff --git a/match_mean_change.py b/match_mean_change.py deleted file mode 100644 index c206a18..0000000 --- a/match_mean_change.py +++ /dev/null @@ -1,202 +0,0 @@ -"""Command line program for matching up GCM and QQ-scaled mean change.""" - -import logging -import argparse - -import dask.diagnostics -import xesmf as xe - -import utils - - -def match_mean_change(ds_qq, qq_var, da_hist, da_ref, da_target, scaling, timescale): - """Match the model and quantile delta mean change. - - Parameters - ---------- - ds_qq : xarray Dataset - Quantile delta changed dataset - qq_var : str - Variable (in ds_qq) - da_hist : xarray DataArray - Historical model data - da_ref : xarray DataArray - Reference model data - da_target : xarray DataArray - Data that the quantile delta changes were applied to - scaling : {'additive', 'multiplicative'} - Scaling method - timescale : {'annual', 'monthly'} - Timescale for mean matching - - Returns - ------- - ds_qq_adjusted : xarray Dataset - Quantile delta change dataset adjusted so it matches model mean change - """ - - if timescale == 'monthly': - hist_clim = da_hist.groupby('time.month').mean('time', keep_attrs=True) - ref_clim = da_ref.groupby('time.month').mean('time', keep_attrs=True) - target_clim = da_target.groupby('time.month').mean('time', keep_attrs=True) - qq_clim = ds_qq[qq_var].groupby('time.month').mean('time', keep_attrs=True) - elif timescale == 'annual': - hist_clim = da_hist.mean('time', keep_attrs=True) - ref_clim = da_ref.mean('time', keep_attrs=True) - target_clim = da_target.mean('time', keep_attrs=True) - qq_clim = ds_qq[qq_var].mean('time', keep_attrs=True) - else: - raise ValueError(f'Invalid mean match timescale: {timescale}') - - dims = ds_qq[qq_var].dims - spatial_grid = ('lat' in dims) and ('lon' in dims) - if spatial_grid: - qq_clim['lat'] = target_clim['lat'] - qq_clim['lon'] = target_clim['lon'] - - if scaling == 'multiplicative': - ref_hist_clim_ratio = ref_clim / hist_clim - if spatial_grid: - if len(ref_hist_clim_ratio['lat']) != len(qq_clim['lat']): - regridder = xe.Regridder(ref_hist_clim_ratio, qq_clim, 'bilinear') - ref_hist_clim_ratio = regridder(ref_hist_clim_ratio) - adjustment_factor = (ref_hist_clim_ratio * target_clim) / qq_clim - if timescale == 'monthly': - da_qq_adjusted = ds_qq[qq_var].groupby('time.month') * adjustment_factor - elif timescale == 'annual': - da_qq_adjusted = ds_qq[qq_var] * adjustment_factor - elif scaling == 'additive': - ref_hist_clim_diff = ref_clim - hist_clim - if spatial_grid: - if len(ref_hist_clim_diff['lat']) != len(qq_clim['lat']): - regridder = xe.Regridder(ref_hist_clim_diff, qq_clim, 'bilinear') - ref_hist_clim_diff = regridder(ref_hist_clim_diff) - adjustment_factor = ref_hist_clim_diff - (qq_clim - target_clim) - if timescale == 'monthly': - da_qq_adjusted = ds_qq[qq_var].groupby('time.month') + adjustment_factor - elif timescale == 'annual': - da_qq_adjusted = ds_qq[qq_var] + adjustment_factor - else: - raise ValueError(f'Invalid scaling method: {scaling}') - - if timescale == 'monthly': - del da_qq_adjusted['month'] - da_qq_adjusted.attrs = ds_qq[qq_var].attrs - ds_qq_adjusted = da_qq_adjusted.to_dataset(name=qq_var) - ds_qq_adjusted.attrs = ds_qq.attrs - - return ds_qq_adjusted - - -def main(args): - """Run the program.""" - - dask.diagnostics.ProgressBar().register() - ds_hist = utils.read_data( - args.hist_files, - args.hist_var, - time_bounds=args.hist_time_bounds, - input_units=args.input_hist_units, - output_units=args.output_units, - ) - ds_ref = utils.read_data( - args.ref_files, - args.ref_var, - time_bounds=args.ref_time_bounds, - input_units=args.input_ref_units, - output_units=args.output_units, - ) - ds_target = utils.read_data( - args.target_files, - args.qq_var, - time_bounds=args.target_time_bounds, - input_units=args.input_target_units, - output_units=args.output_units, - ) - ds_qq = utils.read_data( - args.qq_file, - args.qq_var, - ) - ds_qq_adjusted = match_mean_change( - ds_qq, - args.qq_var, - ds_hist[args.hist_var], - ds_ref[args.ref_var], - ds_target[args.qq_var], - args.scaling, - args.timescale - ) - infile_logs = {args.qq_file: ds_qq.attrs['history']} - ds_qq_adjusted.attrs['history'] = utils.get_new_log(infile_logs=infile_logs) - ds_qq_adjusted.to_netcdf(args.outfile) - - -if __name__ == '__main__': - parser = argparse.ArgumentParser( - description=__doc__, - argument_default=argparse.SUPPRESS, - formatter_class=argparse.RawDescriptionHelpFormatter - ) - parser.add_argument("qq_file", type=str, help="input qq-scaled data (to be adjusted)") - parser.add_argument("qq_var", type=str, help="variable to process") - parser.add_argument("outfile", type=str, help="output file") - - parser.add_argument("--output_units", type=str, default=None, help="output data units") - - parser.add_argument("--hist_files", type=str, nargs='*', required=True, help="historical data files") - parser.add_argument("--hist_var", type=str, required=True, help="historical variable") - parser.add_argument("--input_hist_units", type=str, default=None, help="input historical data units") - parser.add_argument( - "--hist_time_bounds", - type=str, - nargs=2, - metavar=('START_DATE', 'END_DATE'), - default=None, - help="time bounds for historical period (in YYYY-MM-DD format)" - ) - parser.add_argument("--ref_files", type=str, nargs='*', required=True, help="reference data files") - parser.add_argument("--ref_var", type=str, required=True, help="reference variable") - parser.add_argument("--input_ref_units", type=str, default=None, help="input reference data units") - parser.add_argument( - "--ref_time_bounds", - type=str, - nargs=2, - metavar=('START_DATE', 'END_DATE'), - default=None, - help="time bounds for the reference/future period (in YYYY-MM-DD format)" - ) - parser.add_argument("--target_files", type=str, nargs='*', required=True, help="target data files") - parser.add_argument("--input_target_units", type=str, default=None, help="input target data units") - parser.add_argument( - "--target_time_bounds", - type=str, - nargs=2, - metavar=('START_DATE', 'END_DATE'), - default=None, - help="time bounds for the target data (in YYYY-MM-DD format)" - ) - parser.add_argument( - "--scaling", - type=str, - choices=('additive', 'multiplicative'), - default='additive', - help="scaling method", - ) - parser.add_argument( - "--timescale", - type=str, - choices=('annual', 'monthly'), - default='annual', - help="timescale for mean matching", - ) - parser.add_argument( - "--verbose", - action="store_true", - default=False, - help='Set logging level to INFO', - ) - args = parser.parse_args() - log_level = logging.INFO if args.verbose else logging.WARNING - logging.basicConfig(level=log_level) - main(args) -