Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[develop] Port SRW to Gaea C6 #1163

Open
wants to merge 11 commits into
base: develop
Choose a base branch
from

Conversation

RatkoVasic-NOAA
Copy link
Collaborator

@RatkoVasic-NOAA RatkoVasic-NOAA commented Dec 4, 2024

DESCRIPTION OF CHANGES:

Enable SRW for new machine Gaea-C6

NOTE1: needs to update SRW to new UFS-WM version (after PR2448)
NOTE2: on Gaea C6 for tests use account bil-fire8, not epic, i.e. ./run_WE2E_tests.py -t fundamental -m gaea-c6 -a bil-fire8

UFS_SRW_data copied to /gpfs/f6/bil-fire8/world-shared/UFS_SRW_data location.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update (updated in this PR)

TESTS CONDUCTED:

  • derecho.intel
  • gaea-c5.intel
  • gaea-c6.intel - comprehensive
  • hera.gnu
  • hera.intel
  • hercules.intel
  • jet.intel
  • orion.intel
  • wcoss2.intel
  • NOAA Cloud (indicate which platform)
  • Jenkins
  • fundamental test suite
  • comprehensive tests (specify which if a subset was used)

DEPENDENCIES:

  • ufs-community/ufs-weather-model/pull/2448 Issue: #2407

DOCUMENTATION:

rst files updated.

ISSUE:

Closes issue #1120

CHECKLIST

  • My code follows the style guidelines in the Contributor's Guide
  • I have performed a self-review of my own code using the Code Reviewer's Guide
  • I have commented my code, particularly in hard-to-understand areas
  • My changes need updates to the documentation. I have made corresponding changes to the documentation
  • My changes do not require updates to the documentation (explain).
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

  • Work In Progress
  • bug
  • enhancement
  • documentation
  • release
  • high priority
  • run_ci
  • run_we2e_fundamental_tests
  • run_we2e_comprehensive_tests
  • Needs Cheyenne test
  • Needs Jet test
  • Needs Hera test
  • Needs Orion test
  • help wanted

@MichaelLueken MichaelLueken linked an issue Dec 4, 2024 that may be closed by this pull request
@MichaelLueken MichaelLueken added enhancement New feature or request Work in Progress labels Dec 4, 2024
@RatkoVasic-NOAA
Copy link
Collaborator Author

Passed comprehensive tests:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
2020_CAD_20241204002937                                            COMPLETE              41.87
2020_CAPE_20241204003011                                           COMPLETE              43.16
2019_hurricane_barry_20241204003011                                COMPLETE              41.60
2019_halloween_storm_20241204003012                                COMPLETE              53.42
2019_hurricane_lorenzo_20241204003012                              COMPLETE              48.37
2019_memorial_day_heat_wave_20241204003013                         COMPLETE              50.84
2020_denver_radiation_inversion_20241204003014                     COMPLETE              41.79
2020_easter_storm_20241204003014                                   COMPLETE              42.76
2020_jan_cold_blast_20241204003015                                 COMPLETE              44.63
community_20241204003015                                           COMPLETE              24.79
custom_ESGgrid_20241204003016                                      COMPLETE              30.05
custom_ESGgrid_Central_Asia_3km_20241204003016                     COMPLETE              33.77
custom_ESGgrid_IndianOcean_6km_20241204003017                      COMPLETE              23.92
custom_ESGgrid_NewZealand_3km_20241204003017                       COMPLETE              88.16
custom_ESGgrid_Peru_12km_20241204003056                            COMPLETE              27.01
custom_ESGgrid_SF_1p1km_20241204003057                             COMPLETE             262.59
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE_202  COMPLETE              10.53
custom_GFDLgrid_20241204003132                                     COMPLETE               9.44
deactivate_tasks_20241204003132                                    COMPLETE               1.13
get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2me  COMPLETE             974.46
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS_20241204003133              COMPLETE              13.86
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_202  COMPLETE               9.39
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_  COMPLETE             377.41
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20241  COMPLETE             138.34
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20241204003210  COMPLETE             267.88
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20241204003  COMPLETE              45.10
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              50.35
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024120400  COMPLETE              55.12
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_202  COMPLETE              74.02
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20241  COMPLETE               6.27
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              22.12
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              18.96
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024120400  COMPLETE              18.61
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20241204003  COMPLETE              54.79
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_202  COMPLETE              23.36
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP_20241204003217  COMPLETE              13.82
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2_20241  COMPLETE               9.88
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024120400321  COMPLETE              42.45
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta_202412040  COMPLETE              19.51
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_202412  COMPLETE             319.71
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             485.13
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_20241204  COMPLETE             497.48
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20241204003  COMPLETE             511.55
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_2024  COMPLETE             510.44
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_  COMPLETE              41.38
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR_20241204  COMPLETE              34.99
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              39.68
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_  COMPLETE              11.70
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024120  COMPLETE              32.31
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              11.03
grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_v1beta_  COMPLETE              14.20
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_2  COMPLETE             394.59
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR_202412040  COMPLETE             417.01
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_20  COMPLETE             428.75
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20241204003229  COMPLETE              93.25
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0_202412  COMPLETE              31.25
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024120400323  COMPLETE              23.93
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20241204003  COMPLETE              18.41
grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16_202412040032  COMPLETE              24.20
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot_202  COMPLETE              11.92
MET_ensemble_verification_only_vx_20241204003234                   COMPLETE               0.88
MET_ensemble_verification_winter_wx_20241204003236                 COMPLETE             138.80
MET_verification_only_vx_20241204003237                            COMPLETE               0.26
pregen_grid_orog_sfc_climo_20241204003239                          COMPLETE               9.91
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS_20241204003241               COMPLETE               8.82
specify_template_filenames_20241204003242                          COMPLETE               9.97
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            7277.08

Detailed summary written to /gpfs/f6/bil-fire8/world-shared/Ratko.Vasic/SRW/expt_dirs/WE2E_summary_20241204031723.txt

For runs used UFS-WM branch: https://github.com/BrianCurtis-NOAA/ufs-weather-model.git -b gaeac6 PR#2448

@MichaelLueken MichaelLueken changed the title Port SRW to Gaea C6 [develop] Port SRW to Gaea C6 Dec 4, 2024
@MichaelLueken
Copy link
Collaborator

@RatkoVasic-NOAA I have discussed the Jenkins labels for Gaea-C5 and Gaea-C6 with Kris Booker. The newly added gaea-c6 label will work in Jenkins, but there are no plans for changing the current gaea label to gaea-c5. We will need to change the gaea-c5 entries back to gaea, otherwise the automated tests will fail.

I'm working in a copy of your branch to remove -c5 from gaea-c5 throughout your gaea-c6 branch. I'll let you know and open a PR into your branch once the automated build and run scripts work on Gaea using gaea rather than gaea-c5.

@RatkoVasic-NOAA
Copy link
Collaborator Author

@MichaelLueken that's OK. I'm currently running tests on: Hera, Hercules, Jet, Orion, Gaea-C5, Gaea-C6, and Derecho. Some are already done. I'll retest gaea-c5 after your changes.

@MichaelLueken
Copy link
Collaborator

@RatkoVasic-NOAA The majority of the modifications have been made and the both the automated build script (srw_build.sh) and the wrapper test script (wrapper_srw_ftest.sh) have been successfully tested on Gaea-C5, following the changes from gaea-c5 back to gaea. The final srw_test.sh script is now running. I should be able to open a PR into your gaea-c6 branch in the morning.

@RatkoVasic-NOAA
Copy link
Collaborator Author

Hera:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              13.62
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20241  COMPLETE               9.53
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              26.56
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024121  COMPLETE              44.96
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20241219180  COMPLETE              25.86
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024121918043  COMPLETE              48.59
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             169.12

Jet:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              12.26
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20241  COMPLETE               9.97
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              28.55
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024121  COMPLETE              49.54
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20241219210  COMPLETE              31.96
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024121921091  COMPLETE              48.82
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             181.10

Gaea-c6:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              10.32
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20241  COMPLETE               8.07
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              19.36
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024121  COMPLETE              30.18
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20241219150  COMPLETE              15.71
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024121915080  COMPLETE              39.11
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             122.75

Orion:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              17.78
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20241  COMPLETE              12.36
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              41.80
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024121  COMPLETE              48.84
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20241219145  COMPLETE              32.64
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024121914553  COMPLETE              79.76
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             233.18

Hercules:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              10.50
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20241  COMPLETE               8.09
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              22.77
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024121  COMPLETE              41.00
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20241219120  COMPLETE              27.51
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024121912043  COMPLETE              39.71
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             149.58

Derecho:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              18.77
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20241  COMPLETE              15.59
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              36.19
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024121  COMPLETE              56.55
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20241219135  COMPLETE              27.86
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024121913560  COMPLETE              64.30
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             219.26

@MichaelLueken
Copy link
Collaborator

@RatkoVasic-NOAA PR #15 was created in your fork. Once merged, this PR will be ready for review.

Copy link
Collaborator

@MichaelLueken MichaelLueken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RatkoVasic-NOAA -

I'm building the SRW App on Gaea-C6 using the Jenkins scripts. Once built, I will run the fundamental WE2E tests using the Jenkins scripts as well. I'll run the Jenkins task test script after the fundamental tests successfully finish. I should be able to approve the changes once those tests are complete.

I noted one minor issue with the doc/UsersGuide/CustomizingTheWorkflow/ConfigWorkflow.rst file. There is still mention of GAEA-C5 and GAEA-C6, rather than GAEA and GAEA-C6. Please remove the -C5 from GAEA-C5 on line 33 of the rst file.

doc/UsersGuide/CustomizingTheWorkflow/ConfigWorkflow.rst Outdated Show resolved Hide resolved
@MichaelLueken
Copy link
Collaborator

The Jenkins srw_build.sh script successfully built the SRW on Gaea-C6. The srw_test.sh script was modified to run the fundamental tests on Gaea-C6 and all tests successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              10.22
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20250  COMPLETE               9.28
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              26.49
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2025010  COMPLETE             164.98
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20250109161  COMPLETE              20.41
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2025010916164  COMPLETE              48.78
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             280.16

Submitting the wrapper_srw_ftest.sh script now.

@MichaelLueken
Copy link
Collaborator

The Functional Tasks Test has successfully passed on Gaea-C6 using the wrapper_srw_ftest.sh script:

# Try gaea-c6 with the first few simple SRW tasks ...
run_make_grid: COMPLETE
run_get_ics: COMPLETE
run_get_lbcs: COMPLETE
run_make_orog: COMPLETE
run_make_sfc_climo: COMPLETE
run_make_ics: COMPLETE
run_make_lbcs: COMPLETE
run_fcst: COMPLETE
run_post: COMPLETE

Copy link
Collaborator

@MichaelLueken MichaelLueken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the change, @RatkoVasic-NOAA! Everything looks great and the Jenkins scripts are working correctly on Gaea-C6. I will now approve this PR.

@MichaelLueken
Copy link
Collaborator

I was also able to successfully run the UFS-Fire WE2E tests on Gaea-C6:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
UFS_FIRE_multifire_one-way-coupled_20250110125635                  COMPLETE              17.47
UFS_FIRE_one-way-coupled_20250110125635                            COMPLETE              17.40
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE              34.87

Copy link

@rickgrubin-tomorrow rickgrubin-tomorrow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@EdwardSnyder-NOAA
Copy link
Collaborator

Ran fundamental test on Gaea c6 and it passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              10.40
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20250  COMPLETE               7.78
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              20.05
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2025011  COMPLETE              30.20
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20250110132  COMPLETE              17.97
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2025011013251  COMPLETE              45.22
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             131.62

Experienced strange behaviors were the devbuild script didn't build the srw_app conda env and the make_ics task failed for an experiment. Both issues were resolved by re-running their corresponding scripts (devbuild with the conda-only option for the conda env and rocotorewind for the make_ics task.

@MichaelLueken MichaelLueken added the run_we2e_jenkins_coverage_tests SRW App automated CI testing with modified Jenkinsfile label Jan 10, 2025
@MichaelLueken
Copy link
Collaborator

Jenkins is not yet running on Gaea-C6, so I will manually run the coverage tests on that machine. Additionally, Hera epic space has exceeded disk quota, so will manually run in stmp space.

@MichaelLueken
Copy link
Collaborator

The coverage tests have successfully passed on Gaea-C6:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
community_20250110151055                                           COMPLETE              23.24
custom_ESGgrid_NewZealand_3km_20250110151059                       COMPLETE              78.88
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              41.56
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20250110151  COMPLETE              59.20
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2025011015  COMPLETE              57.33
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             480.58
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2025011  COMPLETE              37.01
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_20  COMPLETE             432.59
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot_202  COMPLETE              17.09
2020_CAPE_20250110151134                                           COMPLETE              42.11
2020_easter_storm_20250110151138                                   COMPLETE              40.97
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            1310.56

@MichaelLueken
Copy link
Collaborator

The Hera Intel coverage WE2E tests have successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
2019_memorial_day_heat_wave_20250110201104                         COMPLETE              64.11
custom_ESGgrid_Peru_12km_20250110201105                            COMPLETE              42.05
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200_2025011  COMPLETE              10.93
get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_netcdf_2022040400_ensemble_2  COMPLETE            1759.62
get_from_HPSS_ics_HRRR_lbcs_RAP_20250110201107                     COMPLETE              21.46
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              27.80
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP_20250110201109  COMPLETE              15.75
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2_20250  COMPLETE              11.24
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_202501  COMPLETE             550.31
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_20250110  COMPLETE             726.12
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR_202501102  COMPLETE             733.17
pregen_grid_orog_sfc_climo_20250110201114                          COMPLETE              12.04
vx-det_long-fcst_custom-vx-config_aiml-graphcast_20250110201115    COMPLETE               0.75
vx-det_multicyc_long-fcst-overlap_nssl-mpas_20250110201117         COMPLETE              12.58
vx-det_multicyc_long-fcst-no-overlap_nssl-mpas_20250110201119      COMPLETE              13.89
vx-det_multicyc_first-obs-00z_ncep-hrrr_20250110201121             COMPLETE               0.95
vx-det_multicyc_no-00z-obs_nssl-mpas_20250110201122                COMPLETE               0.92
vx-det_multicyc_no-fcst-overlap_ncep-hrrr_20250110201124           COMPLETE               2.11
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            4005.80

@chan-hoo
Copy link
Collaborator

chan-hoo commented Jan 11, 2025

@RatkoVasic-NOAA @MichaelLueken , I am trying to run the sample configuration config.community.yaml on Gaea-c6, but the make_grid task does not run even though it looks submitted as follows:

201906151800               make_grid                   207521833              QUEUED                   -         0           0.0

The get_extrn_ics/lbcs tasks are working well though:

-rw-r--r-- 1 Chan-hoo.Jeon ncep  466 Jan 11 07:49 FV3LAM_wflow.log
-rw-r--r-- 1 Chan-hoo.Jeon ncep 6632 Jan 11 07:52 get_extrn_ics_2019061518.log
-rw-r--r-- 1 Chan-hoo.Jeon ncep 7502 Jan 11 07:52 get_extrn_lbcs_2019061518.log

Do you have any idea what is wrong? I am using bil-fire8 for this run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request run_we2e_jenkins_coverage_tests SRW App automated CI testing with modified Jenkinsfile Work in Progress
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enable SRW on Gaea-C6
5 participants