conda env update --prune does not install require dependencies

[laughingrice]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

When updating packages in the environment with

conda env update -n <name> -f <yaml file> --prune

Dependencies are not installed resulting with a broken environment.
As a simple test case, define the environment.yml as

channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.12
  - numpy=2.1.2

Then

conda create -n demo python=3.12
conda env upgrade --name=demo --file=environment.yml --prune
conda activate demo
python -c "import numpy"

import fails because numpy does not have required dependencies. If instead this is run without the --prune option it works

conda create -n demo python=3.12
conda env upgrade --name=demo --file=environment.yml
conda activate demo
python -c "import numpy"

Conda Info

(demo2) PS C:\Users\Ido\Documents\micha\apeiron\137> conda info

     active environment : demo2
    active env location : C:\Users\Ido\miniconda3\envs\demo2
            shell level : 2
       user config file : C:\Users\Ido\.condarc
 populated config files : C:\Users\Ido\.condarc
          conda version : 24.11.2
    conda-build version : not installed
         python version : 3.12.8.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=broadwell
                          __conda=24.11.2=0
                          __win=0=0
       base environment : C:\Users\Ido\miniconda3  (writable)
      conda av data dir : C:\Users\Ido\miniconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/pytorch/win-64
                          https://conda.anaconda.org/pytorch/noarch
                          https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\Ido\miniconda3\pkgs
                          C:\Users\Ido\.conda\pkgs
                          C:\Users\Ido\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\Ido\miniconda3\envs
                          C:\Users\Ido\.conda\envs
                          C:\Users\Ido\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/24.11.2 requests/2.32.3 CPython/3.12.8 Windows/11 Windows/10.0.22631 solver/libmamba conda-libmamba-solver/24.11.1 libmambapy/2.0.5 aau/0.5.0 c/. s/. e/.
          administrator : False
             netrc file : None
           offline mode : False

Conda Config

pip_interop_enabled: True
channel_priority: flexible
channels:
  - pytorch
  - conda-forge
  - defaults

Conda list

# packages in environment at C:\Users\Ido\miniconda3\envs\demo2:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h2466b09_7    conda-forge
ca-certificates           2024.12.14           h56e8100_0    conda-forge
libexpat                  2.6.4                he0c23c2_0    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
liblzma                   5.6.3                h2466b09_1    conda-forge
libsqlite                 3.47.2               h67fdade_0    conda-forge
libzlib                   1.3.1                h2466b09_2    conda-forge
numpy                     2.1.2           py312hf10105a_0    conda-forge
openssl                   3.4.0                h2466b09_0    conda-forge
pip                       24.3.1             pyh8b19718_2    conda-forge
python                    3.12.8          h3f84c4b_1_cpython    conda-forge
setuptools                75.6.0             pyhff2d567_1    conda-forge
tk                        8.6.13               h5226925_1    conda-forge
tzdata                    2024b                hc8b5060_0    conda-forge
ucrt                      10.0.22621.0         h57928b3_1    conda-forge
vc                        14.3                ha32ba9b_23    conda-forge
vc14_runtime              14.42.34433         he29a5d6_23    conda-forge
vs2015_runtime            14.42.34433         hdffcdeb_23    conda-forge
wheel                     0.45.1             pyhd8ed1ab_1    conda-forge

Additional Context

No response

[travishathaway]

@laughingrice,

Thanks for this issue and for providing instructions on how to recreate 🙏. Unfortunately, I was not able to reproduce this error on my macOS laptop using the osx-arm64 architecture.

We will need to wait for someone to try and recreate this on a Windows machine to proceed.

[Manezki]

I'm running into same error while conda env update:ing during a Dockerfile build.

Setup to replicate the error

Dockerfile:

FROM mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu22.04

ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

WORKDIR /app
COPY environment.yml .

RUN conda env update --name base --file environment.yml --prune && \
    conda clean -afy

RUN poetry --version

Environment.yml:

name: conda-build
channels:
  - conda-forge
dependencies:
  - poetry[version='>=1.8']
  - python=3.10

Docker command: docker build --progress plain -t conda-build:latest .

Output:

#0 building with "default" instance using docker driver

conda/conda#1 [internal] load build definition from Dockerfile
conda/conda#1 transferring dockerfile: 645B done
conda/conda#1 DONE 0.0s

conda/conda#2 [internal] load .dockerignore
conda/conda#2 transferring context: 663B done
conda/conda#2 DONE 0.0s

conda/conda#3 [internal] load metadata for mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu22.04:latest
conda/conda#3 DONE 0.0s

conda/conda#4 [1/5] FROM mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu22.04
conda/conda#4 DONE 0.0s

conda/conda#5 [internal] load build context
conda/conda#5 transferring context: 36B done
conda/conda#5 DONE 0.0s

conda/conda#6 [2/5] WORKDIR /app
conda/conda#6 CACHED

conda/conda#7 [3/5] COPY environment.yml .
conda/conda#7 CACHED

conda/conda#8 [4/5] RUN conda env update --name base --file environment.yml --prune &&     conda clean -afy
conda/conda#8 1.144 Retrieving notices: done
conda/conda#8 2.143 Channels:
conda/conda#8 2.143  - conda-forge
conda/conda#8 2.143  - defaults
conda/conda#8 2.143 Platform: linux-64
conda/conda#8 2.143 Collecting package metadata (repodata.json): ...working... done
conda/conda#8 12.39 Solving environment: ...working... done
conda/conda#8 13.92 
                                                      done.
conda/conda#8 14.27 Preparing transaction: done                 
conda/conda#8 14.37 Verifying transaction: done                 
conda/conda#8 14.47 Executing transaction: done
conda/conda#8 21.99 #
conda/conda#8 21.99 # To activate this environment, use
conda/conda#8 21.99 #
conda/conda#8 21.99 #     $ conda activate base
conda/conda#8 21.99 #
conda/conda#8 21.99 # To deactivate an active environment, use
conda/conda#8 21.99 #
conda/conda#8 21.99 #     $ conda deactivate
conda/conda#8 21.99 
conda/conda#8 22.84 Will remove 1 package cache(s).
conda/conda#8 DONE 23.4s

conda/conda#9 [5/5] RUN poetry --version
conda/conda#9 0.482 Traceback (most recent call last):
conda/conda#9 0.482   File "/opt/miniconda/bin/poetry", line 6, in <module>
conda/conda#9 0.482     from poetry.console.application import main
conda/conda#9 0.482   File "/opt/miniconda/lib/python3.10/site-packages/poetry/console/application.py", line 11, in <module>
conda/conda#9 0.482     from cleo.application import Application as BaseApplication
conda/conda#9 0.483 ModuleNotFoundError: No module named 'cleo'
conda/conda#9 ERROR: process "/bin/sh -c poetry --version" did not complete successfully: exit code: 1
------
 > [5/5] RUN poetry --version:
0.482 Traceback (most recent call last):
0.482   File "/opt/miniconda/bin/poetry", line 6, in <module>
0.482     from poetry.console.application import main
0.482   File "/opt/miniconda/lib/python3.10/site-packages/poetry/console/application.py", line 11, in <module>
0.482     from cleo.application import Application as BaseApplication
0.483 ModuleNotFoundError: No module named 'cleo'
------
Dockerfile:17
--------------------
  15 |         conda clean -afy
  16 |     
  17 | >>> RUN poetry --version
  18 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c poetry --version" did not complete successfully: exit code: 1

Workaround

Removing the --prune option from conda env update from the Dockerfile causes the build to succeed.

Fixed Dockerfile:

# The base image originates from the Azure Machine Learning containers repository, see:
# https://github.com/Azure/AzureML-Containers/blob/master/base/cpu/openmpi4.1.0-ubuntu22.04/Dockerfile
FROM mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu22.04

# Writing bytecode is unnecessary in an ephemeral container
ENV PYTHONDONTWRITEBYTECODE=1
# Write STDOUT and STDERR immediately
ENV PYTHONUNBUFFERED=1

WORKDIR /app
COPY environment.yml .

# Install CUDA and poetry (specified in environment.yml)
RUN conda env update --name base --file environment.yml && \
    conda clean -afy

RUN poetry --version

Output:

#0 building with "default" instance using docker driver

conda/conda#1 [internal] load build definition from Dockerfile
conda/conda#1 transferring dockerfile: 637B done
conda/conda#1 DONE 0.1s

conda/conda#2 [internal] load .dockerignore
conda/conda#2 transferring context: 663B done
conda/conda#2 DONE 0.1s

conda/conda#3 [internal] load metadata for mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu22.04:latest
conda/conda#3 DONE 0.0s

conda/conda#4 [1/5] FROM mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu22.04
conda/conda#4 CACHED

conda/conda#5 [internal] load build context
conda/conda#5 transferring context: 142B done
conda/conda#5 DONE 0.1s

conda/conda#6 [2/5] WORKDIR /app
conda/conda#6 DONE 0.1s

conda/conda#7 [3/5] COPY environment.yml .
conda/conda#7 DONE 0.1s

conda/conda#8 [4/5] RUN conda env update --name base --file environment.yml &&     conda clean -afy
conda/conda#8 1.209 Retrieving notices: done
conda/conda#8 2.205 Channels:
conda/conda#8 2.205  - conda-forge
conda/conda#8 2.205  - defaults
conda/conda#8 2.205 Platform: linux-64
conda/conda#8 2.205 Collecting package metadata (repodata.json): ...working... done
conda/conda#8 12.42 Solving environment: ...working... done
conda/conda#8 13.94 
 done                                                ing...
conda/conda#8 16.12 Preparing transaction: done                 
conda/conda#8 16.32 Verifying transaction: done                 
conda/conda#8 16.74 Executing transaction: done                 
conda/conda#8 22.13 #                                           
conda/conda#8 22.13 # To activate this environment, use         
conda/conda#8 22.13 #                                           
conda/conda#8 22.13 #     $ conda activate base                 
conda/conda#8 22.13 #                                           
conda/conda#8 22.13 # To deactivate an active environment, use  
conda/conda#8 22.13 #                                           
conda/conda#8 22.13 #     $ conda deactivate                    
conda/conda#8 22.13                                             
conda/conda#8 23.02 Will remove 1 package cache(s).             
conda/conda#8 DONE 25.9s                                        
conda/conda#8 16.12                                             
conda/conda#9 [5/5] RUN poetry --version                        
conda/conda#9 1.157 Poetry (version 1.8.5)                      
conda/conda#9 DONE 1.2s                                         
                      
conda/conda#10 exporting to image
conda/conda#10 exporting layers
conda/conda#10 exporting layers 2.1s done
conda/conda#10 writing image sha256:fe0037a9cef2c1131aa345e989bd52052c07642177684100c1e21974b69c221a done
conda/conda#10 naming to docker.io/library/conda-build:latest 0.0s done
conda/conda#10 DONE 2.1s

System info

From @travishathaway I assume that the OS might have impact on the behavior. My testing is run on Linux with x86_64.

Additional info

My build broke over time without changes on my build process. My previous successful build has used Conda 24.11.0, and the broken build uses Conda 24.11.2. However, there are other automatically upgrading sources that might have also caused my issue.

[laughingrice]

@travishathaway

@laughingrice,

Thanks for this issue and for providing instructions on how to recreate 🙏. Unfortunately, I was not able to reproduce this error on my macOS laptop using the osx-arm64 architecture.

We will need to wait for someone to try and recreate this on a Windows machine to proceed.

Not sure how recent, but it is a recent regression. It was working fine for me a few weeks ago when I built those environment files.

I just checked on an ubuntu machine, a mac, and two windows machines (all after a conda upgrade --all in the base environment) and it fails on all.

I tried downgrading conda-libmamba-solver, and that solves the problem, so it is one one of the new mamba solver packages (I don't understand enough how the solver works so not sure how to track it down)

conda install conda-libmamba-solver=24.9.0

The following packages will be DOWNGRADED:

conda-libmamba-so~ 24.11.1-pyhd8ed1ab_0 --> 24.9.0-pyhd8ed1ab_0
libmamba 2.0.5-hd41e4cc_0 --> 1.5.11-hd41e4cc_0
libmambapy 2.0.5-py312h0252a60_0 --> 1.5.11-py312h0252a60_0
mamba 2.0.5-hcd709ef_0 --> 1.5.11-py312ha12221d_0

[travishathaway]

@laughingrice,

Thanks for the additional information. I am now able to reliably recreate this issue. I can also report that this does not happen when using the classic solver.

[jaimergp]

I thought --prune was always meant to ignore anything existing on the target environment and only use the environment.yml file as the only source of truth. If that's now what expected, what's the rationale behind using --prune vs not? See the help message:

conda env update --help
usage: conda env update [-h] [-n ENVIRONMENT | -p PATH] [-f FILE] [--prune]
                        [--json] [--console CONSOLE] [-v] [-q]
                        [--solver {classic,libmamba}]
                        [remote_definition]

Update the current environment based on environment file.

positional arguments:
  remote_definition     `remote_definition` is deprecated and will be
                        removed in 25.9. Use `conda env create --file=URL`
                        instead.

options:
  -h, --help            Show this help message and exit.
  -f FILE, --file FILE  environment definition (default: environment.yml)
  --prune               remove installed packages not defined in
                        environment.yml
  --solver {classic,libmamba}
                        Choose which solver backend to use.

[laughingrice]

The way I am reading the help, the way it behaved up to the last version (and based on @travishathaway, how the classic solver behaves)

--prune means remove packages installed by previous version of environment.yml or manually installed packages if they no longer appear in the environment.yml, but do install direct dependencies, otherwise you have a broken environment. That is, replace the current state of the environment with what a new environment created with the given yml would look like (ok, almost, read the last paragraph as well as to how I ended up with a second environment file + prune)

Without --prune, leave everything already installed alone, unless there is a direct conflict requiring an update.

This behavior is useful when managing a team software development and you want to make sure that everyone is using the same environment even when packages were removed along the way. I do not see a scenario where I would want to create an environment without dependencies, and if I did, I would for a nodeps options, or something similar.

The reason I was using this in the first place was that creating a new environment with pip packages specified, first installed the dependencies of the conda packages, and then ignored the pip specification if they were already installed as conda packaged decencies. , This required creating a minimal environment with only the pip packages and then updating the environment with the full specification on top of that (I need pyside6 from pip as the conda version does not include webengine, but having pyqtgraph in the environment file installs the conda pyside6 and then ignores the pip section). That one sounded more like a feature, or at least open to debate as to whether it was a bug.

[jaimergp]

Ah wait I had misread your reproducer. I thought you [python, numpy] was the starting point, which was being updated with python only, so numpy was expected to be left out. Now I see it's the other way around. I'll take a look with some tests. Thank you for the explanation and the intended use case!

[jaimergp]

Opened #596 with a change that fixes the issue and reproducer tests. Let's see if there's an impact in other parts of the test suite.