Cache Shell Script of spack load and spack env activate

[RikkiButler20]

Fixes #47603
Fixes #21413

This will fix the problem of spack load and spack env activate taking too long and currently iterating over all the package's dependency's python package files. This will also prevents the drifting of the installed spec from the current, updated version on Spack

• Create a new hook that will save shell script at install time
• Execute shell script when spack load is run
• Check if there is shell script is cached and if any of the specs have changed when activated environment is installed
• Add tests
• Update documentation

[adamjstewart]

Very excited for this and happy to test on fish!

[aweits]

What if we cached the environment changes at install-time instead? That would also solve issues related to allowing older installs to still function/load... (willing to help with this, would allow me to jettison some local code that is getting harder and harder to maintain)

[tgamblin]

What if we cached the environment changes at install-time instead?

This is the plan. There are some details in the slack discussion linked from #47603, and the bit about “package drift” there.

i think @RikkiButler20 is doing this in steps (see checklist) — probably need one or two more to move the logic to package install time and env install time.

@aweits I think if you want to help, iterating on reviews would be great so we can get this in for 1.0!

[RikkiButler20]

@spackbot Fix style

[spackbot-app]

Let me see if I can fix that for you!

[spackbot-app]

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: import, isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/cmd/load.py
  lib/spack/spack/test/cmd/load.py
==> Running import checks
import check requires Python 3.9 or later
  import checks were clean
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/cmd/load.py
reformatted lib/spack/spack/test/cmd/load.py
All done! ✨ 🍰 ✨
2 files reformatted.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
lib/spack/spack/version/version_types.py:145: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:452: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:481: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/variant.py:131: error: Unsupported right operand type for in ("Union[Collection[Any], Callable[..., Any]]")  [operator]
Found 4 errors in 2 files (checked 624 source files)
  mypy found errors

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

[RikkiButler20]

@spackbot Fix style

[spackbot-app]

Let me see if I can fix that for you!

[spackbot-app]

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: import, isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/cmd/load.py
  lib/spack/spack/test/cmd/load.py
==> Running import checks
import check requires Python 3.9 or later
  import checks were clean
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/test/cmd/load.py
All done! ✨ 🍰 ✨
1 file reformatted, 1 file left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
lib/spack/spack/version/version_types.py:145: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:452: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:481: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/variant.py:131: error: Unsupported right operand type for in ("Union[Collection[Any], Callable[..., Any]]")  [operator]
Found 4 errors in 2 files (checked 624 source files)
  mypy found errors

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

[adamjstewart]

Going to unsubscribe from this PR while it's still a WIP (too many notifications) but please ping me to review fish support once it is complete.

[RikkiButler20]

@spackbot Fix style

[spackbot-app]

Let me see if I can fix that for you!

[spackbot-app]

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: import, isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/cmd/load.py
  lib/spack/spack/hooks/__init__.py
  lib/spack/spack/hooks/cache_shell_script.py
  lib/spack/spack/test/environment_modifications.py
  lib/spack/spack/test/hooks/cache_shell_script.py
  lib/spack/spack/test/util/environment.py
  lib/spack/spack/util/environment.py
==> Running import checks
import check requires Python 3.9 or later
  import checks were clean
==> Running isort checks
Fixing /tmp/tmpu87evsa3/spack/lib/spack/spack/test/hooks/cache_shell_script.py
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/test/hooks/cache_shell_script.py
reformatted lib/spack/spack/util/environment.py
All done! ✨ 🍰 ✨
2 files reformatted, 5 files left unchanged.
  black checks were clean
==> Running flake8 checks
lib/spack/spack/cmd/load.py:12: [F401] 'spack.user_environment as uenv' imported but unused
lib/spack/spack/cmd/load.py:13: [F401] 'spack.util.environment' imported but unused
  flake8 found errors
==> Running mypy checks
lib/spack/spack/util/environment.py:662: error: Item "NameModifier" of "Union[NameModifier, NameValueModifier]" has no attribute "_cache_str"  [union-attr]
lib/spack/spack/util/environment.py:662: error: Item "NameValueModifier" of "Union[NameModifier, NameValueModifier]" has no attribute "_cache_str"  [union-attr]
lib/spack/spack/version/version_types.py:135: error: Incompatible types in assignment (expression has type "Tuple[Any, ...]", variable has type "Tuple[str]")  [assignment]
lib/spack/spack/variant.py:130: error: Unsupported right operand type for in ("Union[Collection[Any], Callable[..., Any]]")  [operator]
lib/spack/spack/build_environment.py:171: error: Overloaded function signatures 1 and 2 overlap with incompatible return types  [misc]
lib/spack/spack/build_environment.py:171: error: Overloaded function signatures 1 and 3 overlap with incompatible return types  [misc]
Found 6 errors in 4 files (checked 636 source files)
  mypy found errors

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.