Make mepo status serial by default

This release changes mepo status to be a serial command by default rather than using multiprocessing. It was found at NAS that mepo status was acting fork-bomb-like spawning many, many processes. This isn't really needed anyway for status in almost any situation.

If users want the old behavior, they can add the --parallel option.

---

From CHANGELOG.md

Changed

• Moved mepo status and mepo restore-state to default to their serial variants rather than parallel by default. At the same time, we remove the --serial option from these commands and add a --parallel option if users want to run them in parallel.

---

What's Changed

• v2: Default status and restore-state to serial by @mathomp4 in #338
• Updated version to 2.3.0 by @pchakraborty in #345

Full Changelog: v2.2.1...v2.3.0

v1: Make mepo status serial

This is a release of mepo v1. It changes mepo status to be a serial command rather than using multiprocessing. It was found at NAS that mepo status was acting fork-bomb-like spawning many, many processes. This isn't really needed anyway for status so we make it serial based on code in mepo v2

---

From CHANGELOG.md

Changed

• Made mepo status serial by default due to odd fork-bomb-like behavior on some systems

---

What's Changed

• v1: Make mepo status serial by default by @mathomp4 in #336

Full Changelog: v1.52.0...v1.53.0

Fixes for less used options of clone

This version of mepo has bug fixes for lesser-used options of mepo clone, allrepos and registry

---

From CHANGELOG.md

Fixed

• Fixed bugs in the lesser used options of mepo clone, allrepos and registry

Added

• Added tests for mepo clone --allrepos and mepo clone --registry

---

What's Changed

• Fixes for less used options of "mepo clone" by @pchakraborty in #335

Full Changelog: v2.2.0...v2.2.1

Branches no longer detached and display stashes

This version of mepo has the following changes:

• Branches cloned are no longer put in detached head state
• Added ability to print number of stashes in mepo status

along with other backend changes.

---

From CHANGELOG.md

Added

• Hidden option --location for mepo that returns the path to the mepo directory
• Added ability to print number of stashes in mepo status
• Added new tests for mepo clone

Changed

• Removed legacy bin directory
• Checked out branches are no longer in 'detached head' state

---

What's Changed

• Merge develop into main by @pchakraborty in #333

Full Changelog: v2.1.0...v2.2.0

Store full remote URL in state file and other updates

This version of mepo has some backend changes. The most important is that the full path of remote url is now stored in the state file. This allows for reducing the number of git calls needed.

There is also some refactoring of code as well as added newly added tests and updates to documentation.

---

From CHANGELOG.md

Fixed

• Fixed mepo completion

Added

• Added ability to print version info via mepo --version
• Added install instructions for spack and brew tap
• Added tests for Registry and MepoComponent classes

Changed

• Full path of remote url is now stored in the state file
• Some refactor of component.py
• Removed MepoState dependency in git.py

---

What's Changed

• Added ability to print version info by @pchakraborty in #309
• Add spack install instructions by @mathomp4 in #310
• Update README by @pchakraborty in #315
• Store complete remote url in state file by @pchakraborty in #316
• Merge main into develop by @pchakraborty in #317
• Merge develop into main by @pchakraborty in #318
• Do a merge commit from main into develop by @mathomp4 in #319
• Updated CHANGELOG.md by @pchakraborty in #320
• Merge develop into main by @pchakraborty in #321

Full Changelog: v2.0.0...v2.1.0

Initial release of mepo 2

This is a major release of mepo that includes a number of changes to the codebase and the way mepo is used.

The most significant change is the conversion of mepo to a Python project. This change was made to improve the maintainability of the codebase and to make it easier to add new features in the future. The conversion to a Python project also allows for better integration with other Python tools and libraries. See this commit message for more information.

Internal State Change

This major version also introduces internal changes to mepo that end users might encounter. The main change is moving from a pickle format to a json format for the internal state of mepo. This change was made to improve the readability and extensibility of the state file.

For example, if a user has an existing clone of a mepo repository that was created with mepo version 1, when they run with mepo version 2, they will see a message like:

> mepo status
Checking status...
Detected mepo1 style state
Run <mepo update-state> to permanently convert to mepo2 style
GEOSfvdycore           | (b) main
env                    | (t) v4.29.0 (DH)
cmake                  | (t) v3.45.1 (DH)
...

As the message says, the user can run mepo update-state to convert the state to mepo2 style (i.e., from pickle to json). Note that this is a one-way change!

macOS multiprocessing issue

If you have a mepo1 style state and you are on macOS, you may encounter an issue when running mepo status or mepo restore-state. For example:

> mepo status
Checking status...
Detected mepo1 style state
Run <mepo update-state> to permanently convert to mepo2 style
Process SpawnPoolWorker-2:
Traceback (most recent call last):
Process SpawnPoolWorker-5:
Process SpawnPoolWorker-1:
Traceback (most recent call last):
Process SpawnPoolWorker-3:
Traceback (most recent call last):
  File "/Users/mathomp4/.homebrew/brew/Cellar/[email protected]/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
...

This seems to be a macOS issue with Python multiprocessing. The solution is to run mepo update-state to convert the state to mepo2 style. Then all works as expected:

> mepo update-state
Detected mepo1 style state
Run <mepo update-state> to permanently convert to mepo2 style

Converted mepo1 state to mepo2

> mepo status
Checking status...
GEOSfvdycore           | (b) main
env                    | (t) v4.29.0 (DH)
cmake                  | (t) v3.45.1 (DH)
ecbuild                | (t) geos/v1.3.0 (DH)
GMAO_Shared            | (t) v1.9.7 (DH)
GEOS_Util              | (t) v2.0.8 (DH)
...

Missing features

Note that at the moment the mepo completion scripts and mepo-cd scripts are not installed by default. This will be addressed in a future release.

---

From CHANGELOG.md

• Added pyproject.toml to aid with pip installation.

• Engineering
  -- Formatting with Black
  -- Linting with Pylint
  -- Dependency management and packaging with Rye

• Added tests to cover more mepo commands

• Add new command update-state to permanently convert mepo1 style state to mepo2

Changed

• Converted mepo to a Python project via the following renaming
  -- Added src/mepo/__init__.py
  -- Renamed mepo.d -> src/mepo
  -- Renamed mepo.d/utest -> tests
  -- Renamed doc --> docs
  -- A mepo config file is now called a mepo registry
  -- More code reorganization

• Helper script mepo, used for development, moved to the bin directory.

• Added README for docs/make_md_docs.py script

• State: pickle format (mepo1 style) to json format (mepo2 style)
  -- If mepo1 style state is detected, print warning and suggest running mepo update-state

---

What's Changed

• Make mepo a pip-installable Python project by @pchakraborty in #269
• Fix small typo by @mathomp4 in #271
• Patch for reading mepo1 state by @pchakraborty in #279
• Python package with expanded testing by @pchakraborty in #278
• Make doc generation work again by @pchakraborty in #286
• Switch from Poetry to Rye for dependency management and packaging by @pchakraborty in #290
• Some engineering by @pchakraborty in #292
• Fixed a bug in mepo state creation during 'mepo update-state' by @pchakraborty in #295
• Make mdutils a dev-dependency instead of a dependency by @pchakraborty in #294
• A bug fix and '--serial' option for status and restore-state commands by @pchakraborty in #296
• Version 2.0.0rc4 by @pchakraborty in #297
• Add docs README, update main README by @mathomp4 in #272
• Ready to release 2.0.0 by @pchakraborty in #298
• Merge develop into main, for release (2.0.0) by @pchakraborty in #300
• Add brew tap installation instructions by @mathomp4 in #302

Full Changelog: v1.52.0...v2.0.0

Support for partial clones

This release adds support for partial clones with mepo clone. A new option --partial has three settings:

• off: performs a "normal" git clone
• blobless: performs a blobless clone via git clone --filter=blob:none
• treeless: performs a treeless clone via git clone --filter=tree:0

The default is tecnically none of these, but that is equivalent to off. The off option is mainly added here to allow overriding of any .mepoconfig setting (see below).

The motivation for this is that it has been noticed that clones of GEOS (mainly MAPL) are often extremely slow. For example:

❯ time git clone [email protected]:GEOS-ESM/MAPL.git MAPL-normal
Cloning into 'MAPL-normal'...
remote: Enumerating objects: 704212, done.
remote: Counting objects: 100% (271489/271489), done.
remote: Compressing objects: 100% (6882/6882), done.
remote: Total 704212 (delta 269709), reused 266119 (delta 264584), pack-reused 432723
Receiving objects: 100% (704212/704212), 997.08 MiB | 2.59 MiB/s, done.
Resolving deltas: 100% (694344/694344), done.
noglob git clone [email protected]:GEOS-ESM/MAPL.git MAPL-normal  95.46s user 13.88s system 24% cpu 7:17.37 total

This took over 7 minutes to clone!

However, git supports partial clones as detailed in this GitHub Blog post. Now, of the two, blobless clones are fairly safe and give you faster initial clone speed at the cost of slower operations after that. As a test:

❯ time git clone --filter=blob:none [email protected]:GEOS-ESM/MAPL.git MAPL-blobless-from-git
Cloning into 'MAPL-blobless-from-git'...
remote: Enumerating objects: 21299, done.
remote: Counting objects: 100% (3171/3171), done.
remote: Compressing objects: 100% (1324/1324), done.
remote: Total 21299 (delta 1972), reused 2989 (delta 1829), pack-reused 18128
Receiving objects: 100% (21299/21299), 20.80 MiB | 2.99 MiB/s, done.
Resolving deltas: 100% (13343/13343), done.
remote: Enumerating objects: 942, done.
remote: Counting objects: 100% (552/552), done.
remote: Compressing objects: 100% (500/500), done.
remote: Total 942 (delta 112), reused 126 (delta 50), pack-reused 390
Receiving objects: 100% (942/942), 1.85 MiB | 136.00 KiB/s, done.
Resolving deltas: 100% (249/249), done.
Updating files: 100% (1064/1064), done.
noglob git clone --filter=blob:none [email protected]:GEOS-ESM/MAPL.git   1.38s user 0.54s system 6% cpu 27.609 total

28 seconds!

Treeless clones are usually faster than blobless as you aren't just filtering out blobs but whole trees. But per the blog:

We strongly recommend that developers do not use treeless clones for their daily work. Treeless clones are really only helpful for automated builds when you want to quickly clone, compile a project, then throw away the repository. In environments like GitHub Actions using public runners, you want to minimize your clone time so you can spend your machine time actually building your software! Treeless clones might be an excellent option for those environments.

As there are possible scenarios with CI that this could be useful for, the option is added. As for speed:

❯ time git clone --filter=tree:0 [email protected]:GEOS-ESM/MAPL.git MAPL-treeless
Cloning into 'MAPL-treeless'...
remote: Enumerating objects: 6875, done.
remote: Counting objects: 100% (843/843), done.
remote: Compressing objects: 100% (730/730), done.
remote: Total 6875 (delta 124), reused 805 (delta 113), pack-reused 6032
Receiving objects: 100% (6875/6875), 2.24 MiB | 277.00 KiB/s, done.
Resolving deltas: 100% (757/757), done.
remote: Enumerating objects: 106, done.
remote: Counting objects: 100% (70/70), done.
remote: Compressing objects: 100% (66/66), done.
remote: Total 106 (delta 1), reused 19 (delta 0), pack-reused 36
Receiving objects: 100% (106/106), 37.34 KiB | 538.00 KiB/s, done.
Resolving deltas: 100% (3/3), done.
remote: Enumerating objects: 942, done.
remote: Counting objects: 100% (552/552), done.
remote: Compressing objects: 100% (500/500), done.
remote: Total 942 (delta 112), reused 126 (delta 50), pack-reused 390
Receiving objects: 100% (942/942), 1.85 MiB | 2.09 MiB/s, done.
Resolving deltas: 100% (249/249), done.
Updating files: 100% (1064/1064), done.
noglob git clone --filter=tree:0 [email protected]:GEOS-ESM/MAPL.git   0.45s user 0.25s system 4% cpu 15.950 total

16 seconds!

Along with this option, we also add a new .mepoconfig setting where one can add:

[clone]
partial = blobless

and blobless clones will be the default.

---

From CHANGELOG.md

Added

• Added new --partial option to mepo clone with two settings: off, blobless, and treeless. If you set, --partial=blobless then
  the clone will not download blobs by using --filter=blob:none. If you set --partial=treeless then the clone will not download
  trees by using --filter=tree:0. The blobless option is useful for large repos that have a lot of binary files that you don't
  need. The treeless option is even more aggressive and SHOULD NOT be used unless you know what you are doing. The
  --partial=off option allows a user to override the default behavior of --partial in .mepoconfig and turn it off for a
  run of mepo clone.
• Add a new section for .mepoconfig to allow users to set --partial as a default for mepo clone.

---

What's Changed

• GitFlow: Merge main into develop for hotfix by @mathomp4 in #262
• Add support for partial clones by @mathomp4 in #264
• Protect submodule repos from treeless, add off option by @mathomp4 in #266
• GitFlow: Merge develop into main by @mathomp4 in #265

Full Changelog: v1.51.1...v1.52.0

Fix for ignore_submodules in older mepo clones

This patch release of mepo fixes an issue running mepo commands on older mepo clones made before v1.51.0 when ignore_submodules was added. Mea culpa.

---

From CHANGELOG.md

Fixed

• Fixes to allow mepo to work on older mepo clones that don't have ignore_submodules in their state

---

What's Changed

• Fixes for older mepo clones in re ignore_submodules by @mathomp4 in #261

Full Changelog: v1.51.0...v1.51.1

Add ability to ignore submodules in diff and status

TL;DR This release adds a new field to components.yaml, ignore_submodules which when true will ignore submodules in the mepo status and mepo diff calls.

---

A mepo oddity occurs in GEOSgcm because of how CICE and icepack are handled.

Technically, icepack is a submodule of CICE in the repo (see https://github.com/GEOS-ESM/CICE) and git sees it as that. However, currently GEOS handles icepack as a separate repo (most likely for ease of development) but git doesn't really know this. So when you go to CICE in GEOS in a fresh clone you see:

$ git status
HEAD detached at geos/v0.0.2
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   icepack (new commits)

Submodules changed but not updated:

* icepack 81a7628...8d80ff5 (1):
  > Merge pull request #1 from GEOS-ESM/bugfix/zhaobin74/recompute-enthalpy

no changes added to commit (use "git add" and/or "git commit -a")

and since mepo status just sort of "echoes" status output with mepo status we see:

cice6                  | (t) geos/v0.0.2 (DH)
   | icepack: modified, not staged

So git says "something changed" but we want to ignore that in this case.

Thus, this PR adds a new field to components.yaml, ignore_submodules:

cice6:
  local: ./src/Components/@GEOSgcm_GridComp/GEOSogcm_GridComp/GEOSseaice_GridComp/CICE_GEOSPlug/@cice6
  remote: ../CICE.git
  tag: geos/v0.0.2
  develop: geos/develop
  ignore_submodules: true

If this is true, then we add --ignore-submodules=all to underlying git calls when we run mepo diff and mepo status

---

From CHANGELOG.md

Added

• Added new ignore_submodules field in components.yaml to allow ignoring submodules in a repo. Currently used for status and
  diff commands.

---

What's Changed

• Add ability to ignore submodules by @mathomp4 in #258
• GitFlow: Merge Develop into main by @mathomp4 in #259

Full Changelog: v1.50.0...v1.51.0

Add hashes option to status command

This release of mepo adds a new --hashes option for mepo status that displays the hashes of each repo:

❯ mepo status
Checking status...
MAPL       | (b) main
ESMA_env   | (t) v4.19.0 (DH)
ESMA_cmake | (t) v3.31.0 (DH)
ecbuild    | (t) geos/v1.3.0 (DH)
❯ mepo status --hashes
Checking status...
MAPL       | (b) main (1afeb43d6af485c6fbedd7ce72ccc1eec20a5c5a)
ESMA_env   | (t) v4.19.0 (DH) (7a8e3b158967bf3144d396d836489f46888b7f56)
ESMA_cmake | (t) v3.31.0 (DH) (7dc4c819dc5169e6b887374aa0fa0a8f71846832)
ecbuild    | (t) geos/v1.3.0 (DH) (4a8c91a579e112a6e2a5224068e95c3a8935204b)

---

From CHANGELOG.md

Added

• Command status has now a --hashes option that list current HEAD hash for each component.

---

What's Changed

• Update zenodo.json by @mathomp4 in #252
• Add --hashes reading HEAD to get the current hash by @FlorianDeconinck in #253
• Update changelog for 1.50 release by @mathomp4 in #254
• GitFlow: Merge Develop into main for release by @mathomp4 in #255

New Contributors

• @FlorianDeconinck made their first contribution in #253

Full Changelog: v1.49.0...v1.50.0