October ESMValTool Workshop

[mo-tgeddes]

Hi,

The dates have been chosen for the workshop: October 18th - 20th (Tuesday - Thursday). You can attend either remotely or in person at the Met Office in Exeter, UK. In addition to the main workshop, two optional additional days for those who would like to work with the Iris developers are available on Monday 17th and Friday 21st.

Registration for in-person attendance is now closed, but remains open for online attendance: https://forms.office.com/r/8L2C3nvWwx.

Thanks,
Theo & Alistair

UPD:
Programme for the workshop: https://docs.google.com/document/d/1iIBZnH0lV04XIqJC1XAviv5Fm5hO3MJkF44kR-nUMMw/edit?usp=sharing

[mo-tgeddes]

Hi @ESMValGroup/esmvaltool-developmentteam,
Could you please put any discussion points and session items you want to have at the workshop here please?
Thanks,
Theo.

[alistairsellar]

Hi Ag, thanks for the offer! Agenda still under development so we can probably work around your availability. Mornings would be best - you have any availability in the mornings of 18, 19, 20 Oct?

[agstephens]

Thanks @alistairsellar, here is my availability:
18th: 0900-1100
19th: 0900-1100
20th: 0900-1200

[alistairsellar]

Thanks @agstephens. Can we put you down for 0900-1000 on the 19th then?

[agstephens]

Hi @alistairsellar @zklaus : yes, that is fine. How long do you want me to talk for?

[bouweandela]

There should be no need to talk for an hour if feel that you can tell the story in 15 minutes, but it's nice if we have time for questions and discussion, so I think it's good to book the full hour and then finish early if we're done early.

[alistairsellar]

Hi @ESMValGroup/esmvaltool-developmentteam here is some travel and accommodation information for those travelling to Exeter for the workshop...

Accommodation
There are two chain hotels within a 5-minute walk of the Met Office:

• Premier Inn Exeter (M5 J29): https://www.tripadvisor.co.uk/Hotel_Review-g186254-d5555181-Reviews-Premier_Inn_Exeter_M5_J29_hotel-Exeter_Devon_England.html
• Holiday Inn Express Exeter M5 JCT. 29: https://www.tripadvisor.co.uk/Hotel_Review-g186254-d189356-Reviews-Holiday_Inn_Express_Exeter_M5_JCT_29_an_IHG_Hotel-Exeter_Devon_England.html

There are several evening food options near to these, including a couple of pubs, a Dominoes Pizza and Subway. However, the Met Office and these hotels are 2.5 miles from the city centre and this edge-of-town area has nothing "exciting" in the way of nightlife, so those who prefer to be near the nightlife would be better to choose one of the many hotels in the city centre. We don't have any particular recommendations for these.

Travel
For those arriving by Eurostar, the simplest connection is to take the tube to London Paddington and on to Exeter St Davids. The train from Paddington to Exeter takes approx 2hr15 through rolling countryside (once you get past Reading). See this page for bus connections across Exeter to the Met Office: https://www.metoffice.gov.uk/about-us/contact/in-exeter (the number 56 goes direct from St Davids station to the Met Office).

For those arriving by air,

• the nearest international airport is Bristol, and there are coaches direct from Bristol Airport to Exeter: https://www.stagecoachbus.com/promos-and-offers/south-west/south-west-falcon
• there are also direct coaches from London Heathrow to Exeter, operated by National Express: https://www.nationalexpress.com/en
• the Bristol and Heathrow coaches both stop at Honiton Road Park & Ride close to the Met Office and a 2-minute walk from the above hotels
• both airports also have connections to the rail network, for trains to Exeter St Davids: the Bristol Flyer bus takes approx 30 mins to Bristol Temple Meads; the Heathrow Express train takes 15-20 mins from terminals 2, 3, 5 to London Paddington.

See https://www.metoffice.gov.uk/about-us/contact/in-exeter for additional travel information, and do ask us if you have any questions.

[alistairsellar]

ESMValTool Workshop registration

Hi all, the header of this discussion thread has been updated to link to a registration form for the ESMValTool workshop on 18-20 October. Please see #45 for details. The registration deadline is 26th September.

Please note that even if you filled in the availability poll in July, we still need you to register to confirm your attendance and provide some details to support the workshop organisation.

Thanks,
Alistair

[schlunma]

Could you maybe add some details about this UK Security Clearance? Is this something you need to apply for before entering the UK? Googling this was not really helpful 😬

[alistairsellar]

No, is absolutely not needed!

But some of the UK attendees may already have it, and it helps our reception people if they know who does. Sorry for any confusion - I'll add a clarification to the form now.

[alistairsellar]

Thanks to everyone who has registered. We have 26 attending in person (split 50-50 between Met Office and others), and 16 registered for online attendance. Very exciting!

Registration for online attendance remains open: https://forms.office.com/r/8L2C3nvWwx.

Hoping to share the draft programme next week...

[malininae]

Is there a rough program available? I wouldn't mind joining remotely, but being on the North American West Coast, I'd need to understand if I'd be able/ if it makes sense to join at all.

[alistairsellar]

Hi Liza, thanks for you interest and sorry for the delay. The programme is still under development, but these example topics give a rough flavour:

• Reports from teams: User Engagement Team, Technical Lead Developer Team, Scientific Lead Developer Team
• Parallel(?) team meetings: user, tech, science
• Backward incompatibility
• Governance update and next steps
• Station based observations
• ESGF search interface based on STAC
• First experiences with iris-esmf-regrid

I hope that helps?

[malininae]

@alistairsellar Thanks! I have registered, but just FYI, I won't be participating very actively (time zones :( )!

[alistairsellar]

Great! Looking forward to meeting you. Appreciate that it doesn't work well for your time zone!

[alistairsellar]

I'm excited to share a draft programme for the workshop: https://docs.google.com/document/d/1iIBZnH0lV04XIqJC1XAviv5Fm5hO3MJkF44kR-nUMMw/edit?usp=sharing

If your name is in here, please can you email me to confirm your willingness (or not) to take the role proposed?

Please also reply here to raise any questions about the programme.

[catherinehardacre]

Hi Alistair, In a role reversal, I've deputised Jane to lead the station based observations discussion. Given the time difference, I'm unlikely to be able to join from NZ, unfortunately. Catherine

[alistairsellar]

Thanks Catherine and Jane!

[rswamina]

@alistairsellar , @ehogan and @remi-kazeroni - If it's not too late, I'd like to include a topic for discussion - the distribution or availability of observation data sets on JASMIN/Levante. Some questions for discussion :

• Should we have the full set made available on JASMIN and Levante (right now, I am told JASMIN only has a subset)?
• What is the best way to store, in a distributed yet accessible way CMO-ized obs data sets?
• Potential issues with access restriction etc.
  We talked a bit about this at the June ESMValTool meeting but perhaps we can revisit the topic summarized under Observations in the summary from our June workshop. I couldn't find the document online so didn't link it here.

[rswamina]

@alistairsellar - Yes, that sounds like a good plan. We can initiate a discussion next week and have a working group to formulate policy for obs.
@remi-kazeroni - Would you be interested in summarizing the current state and any plans for the future with regards to obs? I am happy to help.

[remi-kazeroni]

Hi @rswamina, @alistairsellar, I'd be happy to contribute to a session on obs data at the upcoming workshop if you think that is needed. During the last workshop, various aspects of the obs data sharing problem were discussed: legal issues, technicalities, Tier classification, ... see summary of previous workshops on the website. I'm not sure the situation has changed much since then but we can discuss the particular case of JASMIN if that can ease development of ESMValTool there.
As you may recall, the limitation to Tier2 CMORiezd data on Jasmin is not only due to legal concerns but also lack of storage, @valeriupredoi could tell us more on that. FYI, the storage requirements are:

• Tier2: 500 GB
• Tier3 (except ERA5): 1.5 TB
• RAWOBS/Tier3/ERA5/v1/mon: 300 GB
• RAWOBS/Tier3/ERA5/v1/*hr: 4 TB.
  But we may not have to sync ERA5 data to Jasmin, I think these are already available there (tbc by actual Jasmin users).

[alistairsellar]

Great, thanks @rswamina & @remi-kazeroni! I've added the session to the programme and include both your names as leads.

[bouweandela]

What is the best way to store, in a distributed yet accessible way CMOR-ized obs data sets?

The ESGF. There is the obs4MIPs project for observations and the CREATE-IP project for reanalyses. It would be good to try and collaborate more. Recently @glpotter from the CREATE-IP project joined us, so this is a really great start. If we can demonstrate that it works well for the CREATE-IP project, the obs4MIPs project might also be interested in collaborating with us.

[hb326]

How fortunate that I am sitting now on the obs4MIPS panel. ;)
Let's brainstorm at the workshop how to make use of my new "position".

[malininae]

For those who are far away but would like to know updates. Would it be possible to record at least some pre-lunch discussions? I'll try to join on Tuesday and maybe Thursday, but I'm not guaranteeing that I will stay awake :-)

[bouweandela]

Maybe we could consider sharing the recordings only with regostered workshop participants? That will hopefully take away any reluctance to speak out.

[alistairsellar]

Perhaps it would, it's hard to know. But it wouldn't solve the issue of Met Office needing everyone to sign consent forms. I don't know why the Met Office is more restrictive than other organisations on this, but at this late stage there's nothing I can do to challenge our process.

I'm hoping that a transcript provides almost as much value as a recording?

[malininae]

@alistairsellar @bouweandela Ok, got you! I work for the Canadian Government (same crown, lol), and we also have to have special security assessments and other stuff to record the meetings, so I know the drill. In that sense, if others are not so interested in the recordings, then maybe event transcripts are not needed. I'll brew a big pot of coffee :-) And maybe as you say, will just reach out to presenters in case of questions.

[axel-lauer]

Back then, we only recorded the presentations (with the consent of the presenter). To me, that's a different story than recording discussions. I think automatic transcripts would be fine (but possibly also not extremely useful to everyone). In the past, we always provided workshop summaries with the most important points discussed. Maybe that would be enough?

[alistairsellar]

Hi @bouweandela, I realised after I switched off the laptop yesteday that my tone was a bit dismissive in my replies to you. Sorry about that. I think I was a bit nervous about opening up discussion on something I wouldn't know how to fix. But that's not the right spirit. So... all, I too would be interested in hearing thoughts on the merits or otherwise of recording presentations.

Thanks @malininae & @axel-lauer for your replies too. @malininae, the transcripts are quite straightforward to generate, so unless anyone has concerns I'm happy to switch them on even if they are only a little useful to you.

[alistairsellar]

📢 In-person attendees 📢

I've just emailed some detailed information to in-person attendees. Please shout if you believe that you have registered to attend in-person but didn't get an email from me!

[alistairsellar]

Thanks everyone who participated in last week's ESMValTool workshop - it was a blast!

We'll email registered participants soon with links to transcripts and presentations. In the meantime, we thought it'd be nice if participants could post here some things that were achieved, or progressed, or some next steps that they plan to follow up, e.g. sparked by conversations or plans that were made.

[bouweandela]

Thanks for organizing! Here are my personal highlights:

• We had some good team-building sessions with the @ESMValGroup/technical-lead-development-team, thanks everyone!
• First steps towards lazy saving with iris in Basic functional lazy saving. SciTools/iris#5031 together with @pp-mo @bjlittle and @zklaus. This will hopefully lead to much lower memory consumption and improved parallelization. To try it out, have a look at Use dask to run tasks ESMValCore#1714. Currently, this saves via xarray and loses all kinds of metadata in the process so it's only useful for performance testing, but once Basic functional lazy saving. SciTools/iris#5031 is merged and released it can be used for real work.
• Support lazy vertical interpolation with Python stratify Support interpolation on dask arrays SciTools/python-stratify#52. Now we're waiting on @bjlittle to make a new release and then we can merge Make extract_levels preprocessor function lazy ESMValCore#1761 and have lazy vertical interpolation in ESMValCore!

[axel-lauer]

Other achievements accomplished during the workshop include:

• Finalized and merged our new broken recipe policy (Extending documentation: broken recipe policy ESMValTool#2720)
• Finalized and merged a documentation extension on the role of a recipe maintainer (Extending documentation: recipe maintainer + broken recipe policy ESMValTool#2719)

[nhsavage]

working with Iris developers two fixes important for regridding CORDEX data were completed:

• Prioritise dim coord in _get_lon_lat_coords() Prioritise dim coord in _get_lon_lat_coords() SciTools/iris#5029
• Allow regridding of cubes with multiple coordinates per axis Allow regridding of cubes with multiple coordinates per axis SciTools/iris-esmf-regrid#217

[schlunma]

Thanks for organizing!!

• Resolved MeshCoord constructor always uses metadata of node coordinate regardless of location SciTools/iris#4860 with Meshcoord metadata follows face/edge coords where present. SciTools/iris#5020
• had very good discussion regarding UGRID with Iris developers

[remi-kazeroni]

Thanks for organising the workshop!

• Discussion on the distribution and availability of observational data. Consensus to establish a working-group to formulate the policy for obs data.
• Discussion with @bjlittle and @bsolino on support for Zarr in Iris for usage within ESMValTool
• Discussion with @egalytska and @bsolino on possible ways to connect a Causal Discovery Python package to ESMValTool.