CI tests failing since 28 Nov

[tsibley]

Error is:

AttributeError: module 'lib' has no attribute 'X509_V_FLAG_NOTIFY_POLICY'

Traceback (most recent call last):
  File "/home/runner/miniconda3/envs/test/bin/nextstrain", line 5, in <module>
    from nextstrain.cli.__main__ import main
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/nextstrain/cli/__init__.py", line 21, in <module>
    from .command     import all_commands, version
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/nextstrain/cli/command/__init__.py", line 1, in <module>
    from . import (
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/nextstrain/cli/command/build.py", line 23, in <module>
    from .. import runner
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/nextstrain/cli/runner/__init__.py", line 7, in <module>
    from . import (
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/nextstrain/cli/runner/aws_batch/__init__.py", line 95, in <module>
    from . import jobs, s3
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/nextstrain/cli/runner/aws_batch/jobs.py", line 10, in <module>
    from ... import aws
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/nextstrain/cli/aws/__init__.py", line 5, in <module>
    import boto3
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/boto3/__init__.py", line 17, in <module>
    from boto3.session import Session
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/boto3/session.py", line 17, in <module>
    import botocore.session
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/botocore/session.py", line 26, in <module>
    import botocore.client
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/botocore/client.py", line 15, in <module>
    from botocore import waiter, xform_name
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/botocore/waiter.py", line 18, in <module>
    from botocore.docs.docstring import WaiterDocstring
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/botocore/docs/__init__.py", line 15, in <module>
    from botocore.docs.service import ServiceDocumenter
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/botocore/docs/service.py", line 14, in <module>
    from botocore.docs.client import (
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/botocore/docs/client.py", line 18, in <module>
    from botocore.docs.example import ResponseExampleDocumenter
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/botocore/docs/example.py", line 13, in <module>
    from botocore.docs.shape import ShapeDocumenter
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/botocore/docs/shape.py", line 19, in <module>
    from botocore.utils import is_json_value_header
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/botocore/utils.py", line 39, in <module>
    import botocore.httpsession
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/botocore/httpsession.py", line 45, in <module>
    from urllib3.contrib.pyopenssl import (
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/urllib3/contrib/pyopenssl.py", line 50, in <module>
    import OpenSSL.crypto
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/OpenSSL/crypto.py", line 1579, in <module>
    class X509StoreFlags(object):
  File "/home/runner/miniconda3/envs/test/lib/python3.9/site-packages/OpenSSL/crypto.py", line 1598, in X509StoreFlags
    NOTIFY_POLICY = _lib.X509_V_FLAG_NOTIFY_POLICY
AttributeError: module 'lib' has no attribute 'X509_V_FLAG_NOTIFY_POLICY'

Example of last good run and first bad run.

[tsibley]

GitHub Actions runner image version remains the same across both (ubuntu-20.04 20241124.1.0), which was my first thought here. But I realized that didn't make sense as the issue is within the ambient runtime set up by our CI tests using Conda, which means it uses its own OpenSSL library instead of the system's.

pyOpenSSL is implicated in the error message, so I checked its release history, and sure enough, 24.3.0 was released (after our normal daily CI run) and tests broke the next morning. I highly suspect that's not spurious correlation. Need to verify though.

[tsibley]

May be deeper:

• AttributeError: module 'lib' has no attribute 'X509_V_FLAG_NOTIFY_POLICY'. Did you mean: 'X509_V_FLAG_EXPLICIT_POLICY'? conda/conda#13619 (comment)
• openssl 3.4.0 not compatible to cryptography 44.0.0 conda-forge/openssl-feedstock#181
• Pin cryptography to versions compatible with pyopenssl conda-forge/conda-forge-repodata-patches-feedstock#913
• (Comments only) Add more context about cryptography pins in pyopenssl conda-forge/conda-forge-repodata-patches-feedstock#919

[tsibley]

Upstream fixed itself it seems; our CI tests seem alright again.