Moving from CDAT to xcdat

[jypeter]

Hi @jasonb5 and everybody

I have just learned about xcdat by reading CDAT/cdms#440 (comment)

I think it would be nice to advertise the ongoing move from the CDAT ecosystem to xcdat, to the existing CDAT community, unless the project is too young yet to be used by somebody outside PCMDI, or you (incorrectly) think only LLNL people use CDAT related tools. I have just checked the CDAT site and CDAT @ github and I have not found any mention of xcdat. Intestingly enough, a Google search of xcdat python does not yield any useful result! I was lucky to find the github and readthedoc pages by just looking for xcdat

I think the tools developed at PCMDI have always been awesome for the climate community and I have used them for a long loooong time, since when Dean and Bob (and even Karl) were still young, when python did not exist, and when we used shell scripts and a very nice tool+gui called vcs on our X terminals and IBM RS6000 servers. I have just checked my archives and found a reprint of Dean's Visualization for climate modeling 1993 paper.

What the PCMDI tools have unfortunately always missed is a committed and growing user community. I have always attributed this to a lack of communication and online documentation (that other similar but not as powerful tools/libraries have). And it also took some time and expertise to compile python+CDAT before the nice conda-based distribution. But the switch to conda took place quite some time ago, with not much extra advertisement/support to new users

So I hope that the new version of the tools I have been using for years will take off quickly (with appropriate advertisement/documentation and user support), and be as much backward compatible as possible with the old tools. At least I need to have access to the features of cdms2+cdtime+cdutil/genutil I have always relied on, hopefully with a similar enough syntax. I confess I also loved vcs, but had to gradually stop using it when the backend moved from gks to VTK and started to be difficult to use on remote servers

For netCDF input AND ouput, I have only used cdms2, with no need for netcdf4 or xarray, but I guess I can envision trying xcdat+xarray. Somebody can also deal with CDAT/cdat.github.io#206 at last

I have quickly checked the advertised Planned features of xcdat and they seem nice enough but:

• they don't mention easy generation of CF compliant netCDF files (done by cdms2), i.e netCDF output
• they don't mention easily handling time coordinates with exotic calendars (like no_leap or 360_days) and time origins
• Ability to operate on both (0 to 360) and (-180 to 180) longitudinal axes orientations is nice but I'm expecting someting like reordering longitudes on-demand by taking the 360 degrees periodicity into account. e.g. if the input data is stored on [0, 360[, I want to be able to read [-180, 180[ or [-30, 50[ or even [-180, 360[ if I want

[pochedls]

Thank you @jypeter - we appreciate some of the history you've outlined and you raise some excellent points!

At the moment, xcdat is not ready for widespread use (we're still developing some barebones functionality and addressing existing bugs in v0.1.0). Our initial goal is to develop a subset of the key features of CDAT and make those available. In general, we would like to extend xarray to reproduce the key features in CDAT.

A realistic timeline is an announcement of xcdat in late 2021 / early 2022 with version 0.2.0, which will include basic features such as spatial averaging, temporal averaging (climatologies, departures, and seasonal averages), I/O (via xarray, including multi-file datasets), handling of time axes, etc. Subsequently, we plan to steadily (over the course of months and years) reproduce the most useful and highest value CDAT functionality that is not already available from native xarray. Ideally, we will be able to entrain some of the broader community in this development.

In many ways, we'd like xcdat to pick up where CDAT left off. You highlight some of the important areas we hope to improve / continue:

• extensive documentation: The project is off to a good start in this regard (see the readthedocs page), which will set the standard for future contributions.
• easy-to-install: Continuing the trend from the latest versions of CDAT, by enabling easy installation via conda
• alignment with community trends: Much of the community is moving toward xarray and xarray based tools (e.g., pangeo). Our hope is that we can better leverage existing tools (e.g., xESMF) to reproduce the CDAT functionality that many of us have come to rely on.
• extensive unit testing: Another aspect of CDAT that we would like to improve on is extensive unit testing (which is baked into the development lifecycle).

We hope that these principles, paired with improved communication, will help to produce a committed and growing user community. Even though the project is in early days, we welcome your continued feedback on how to best engage the broader community (e.g., with the early release of v0.2.0).

To address the immediate issue here, we will ensure that CDAT respositories note that CDAT tools are in bare minimum maintenance mode and that we will be migrating to xcdat.

Some other notes:

• backwards-compatibility: It will be tough/impossible to retain identical syntax, but the goal is to reimplement the same functionality in a more modern software framework. The downside is that analysis code will need to be updated to reflect xarray-like syntax, but the potential upside is that we will be able to continue use CDAT functionality with a more modern, maintainable, and performant software infrastructure (that may attract broader community support and engagement). Note also that we are carefully testing CDAT and xcdat to ensure that xcdat reproduces CDAT results.
• generation of cf-compliant netCDF files: This is in xcdat via xarray's to_netcdf() method
• exotic calendars: "Exotic" calendar support should be included in xcdat via cftime (e.g., here, which is used by xarray). We also already include support for some non-CF compliant time dimensions (e.g., Support for time units of "months since..." #77).
• periodicity support: This has been flagged as a powerful CDAT feature. It is already included in spatial averaging and there is (recent) discussion about how to include this more generally. Perhaps you could weigh in on this here.

[Response from @pochedls, @chengzhuzhang, @lee1043, @tomvothecoder, and @jasonb5]

[jypeter]

Sounds good and reasonable! Thanks!

I have uploaded some old but useful documentation and detailed it in #165

[tomvothecoder]

I will be moving this discussion to the repo's Discussions forum and closing this issue.

[jypeter]

Perfect, thanks!

[jypeter]

For your information @jservonnat and @stefraynaud

You may want to look at this and the other discussions, if you have not heard yet about the porting of PCMDI tools to something xarray based

I still only used only cdms2 that fits my needs perfectly, but I'll have a look at xarray once xcdat is stable/mature