_Pseudomonas aeruginosa_ database error

[erinyoung]

Versions

$ conda list
# packages in environment at /home/eriny/miniconda3/envs/poppunk:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
_x86_64-microarch-level   4               2_sapphirerapids    conda-forge
aom                       3.9.1                hac33072_0    conda-forge
at-spi2-atk               2.38.0               h0630a04_3    conda-forge
at-spi2-core              2.40.3               h0630a04_0    conda-forge
atk-1.0                   2.38.0               h04ea711_2    conda-forge
biopython                 1.85            py312h66e93f0_1    conda-forge
brotli                    1.1.0                hb9d3cd8_2    conda-forge
brotli-bin                1.1.0                hb9d3cd8_2    conda-forge
brotli-python             1.1.0           py312h2ec8cdc_2    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
c-ares                    1.34.4               hb9d3cd8_0    conda-forge
ca-certificates           2024.12.14           hbcca054_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cairo                     1.18.2               h3394656_1    conda-forge
cairomm                   1.16.2               ha770c72_1    conda-forge
cairomm-1.16              1.16.2               h7e731d7_1    conda-forge
certifi                   2024.12.14         pyhd8ed1ab_0    conda-forge
cffi                      1.17.1          py312h06ac9bb_0    conda-forge
charset-normalizer        3.4.1              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_1    conda-forge
contourpy                 1.3.1           py312h68727a3_0    conda-forge
cuda-version              12.8                 h5d125a7_3    conda-forge
cycler                    0.12.1             pyhd8ed1ab_1    conda-forge
dav1d                     1.2.1                hd590300_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
dendropy                  5.0.1              pyhdfd78af_0    bioconda
docopt                    0.6.2              pyhd8ed1ab_2    conda-forge
epoxy                     1.5.10               h166bdaf_1    conda-forge
expat                     2.6.4                h5888daf_0    conda-forge
ffmpeg                    7.1.0           gpl_h099772d_709    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 h77eed37_3    conda-forge
fontconfig                2.15.0               h7e30c49_1    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.55.6          py312h178313f_0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
fribidi                   1.0.10               h36c2ea0_0    conda-forge
gdk-pixbuf                2.42.12              hb9ae30d_0    conda-forge
gettext                   0.22.5               he02047a_3    conda-forge
gettext-tools             0.22.5               he02047a_3    conda-forge
glib-tools                2.82.2               h4833e2c_1    conda-forge
gmp                       6.3.0                hac33072_2    conda-forge
graph-tool                2.88            py312hf67fd38_400    conda-forge
graph-tool-base           2.88            py312h28a2651_400    conda-forge
graphite2                 1.3.13            h59595ed_1003    conda-forge
gtk3                      3.24.43              h021d004_3    conda-forge
h2                        4.1.0              pyhd8ed1ab_1    conda-forge
h5py                      3.12.1          nompi_py312hd203070_103    conda-forge
harfbuzz                  10.2.0               h4bba637_0    conda-forge
hdbscan                   0.8.39          py312hc0a28a1_0    conda-forge
hdf5                      1.14.4          nompi_h2d575fe_105    conda-forge
hicolor-icon-theme        0.17                 ha770c72_2    conda-forge
hpack                     4.1.0              pyhd8ed1ab_0    conda-forge
hyperframe                6.1.0              pyhd8ed1ab_0    conda-forge
icu                       75.1                 he02047a_0    conda-forge
idna                      3.10               pyhd8ed1ab_1    conda-forge
joblib                    1.4.2              pyhd8ed1ab_1    conda-forge
kernel-headers_linux-64   3.10.0              he073ed8_18    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.8           py312h84d6215_0    conda-forge
krb5                      1.21.3               h659f571_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lcms2                     2.16                 hb7c19ff_0    conda-forge
ld_impl_linux-64          2.43                 h712a8e2_2    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
level-zero                1.20.2               h84d6215_0    conda-forge
libabseil                 20240722.0      cxx17_hbbce691_4    conda-forge
libaec                    1.1.3                h59595ed_0    conda-forge
libasprintf               0.22.5               he8f35ee_3    conda-forge
libasprintf-devel         0.22.5               he8f35ee_3    conda-forge
libass                    0.17.3               hba53ac1_1    conda-forge
libblas                   3.9.0           26_linux64_openblas    conda-forge
libboost                  1.86.0               h6c02f8c_3    conda-forge
libboost-python           1.86.0          py312hc39e661_3    conda-forge
libbrotlicommon           1.1.0                hb9d3cd8_2    conda-forge
libbrotlidec              1.1.0                hb9d3cd8_2    conda-forge
libbrotlienc              1.1.0                hb9d3cd8_2    conda-forge
libcblas                  3.9.0           26_linux64_openblas    conda-forge
libcups                   2.3.3                h4637d8d_4    conda-forge
libcurl                   8.11.1               h332b0f4_0    conda-forge
libdeflate                1.23                 h4ddbbb0_0    conda-forge
libdrm                    2.4.124              hb9d3cd8_0    conda-forge
libedit                   3.1.20240808    pl5321h7949ede_0    conda-forge
libegl                    1.7.0                ha4b6fd6_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libexpat                  2.6.4                h5888daf_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc                    14.2.0               h77fa898_1    conda-forge
libgcc-ng                 14.2.0               h69a702a_1    conda-forge
libgettextpo              0.22.5               he02047a_3    conda-forge
libgettextpo-devel        0.22.5               he02047a_3    conda-forge
libgfortran               14.2.0               h69a702a_1    conda-forge
libgfortran5              14.2.0               hd5240d6_1    conda-forge
libgirepository           1.82.0               h0dcfedc_0    conda-forge
libgl                     1.7.0                ha4b6fd6_2    conda-forge
libglib                   2.82.2               h2ff4ddf_1    conda-forge
libglvnd                  1.7.0                ha4b6fd6_2    conda-forge
libglx                    1.7.0                ha4b6fd6_2    conda-forge
libgomp                   14.2.0               h77fa898_1    conda-forge
libhwloc                  2.11.2          default_h0d58e46_1001    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
liblapack                 3.9.0           26_linux64_openblas    conda-forge
liblzma                   5.6.3                hb9d3cd8_1    conda-forge
libnghttp2                1.64.0               h161d5f1_0    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libopenblas               0.3.28          pthreads_h94d23a6_1    conda-forge
libopenvino               2024.6.0             hac27bb2_3    conda-forge
libopenvino-auto-batch-plugin 2024.6.0             h4d9b6c2_3    conda-forge
libopenvino-auto-plugin   2024.6.0             h4d9b6c2_3    conda-forge
libopenvino-hetero-plugin 2024.6.0             h3f63f65_3    conda-forge
libopenvino-intel-cpu-plugin 2024.6.0             hac27bb2_3    conda-forge
libopenvino-intel-gpu-plugin 2024.6.0             hac27bb2_3    conda-forge
libopenvino-intel-npu-plugin 2024.6.0             hac27bb2_3    conda-forge
libopenvino-ir-frontend   2024.6.0             h3f63f65_3    conda-forge
libopenvino-onnx-frontend 2024.6.0             h6363af5_3    conda-forge
libopenvino-paddle-frontend 2024.6.0             h6363af5_3    conda-forge
libopenvino-pytorch-frontend 2024.6.0             h5888daf_3    conda-forge
libopenvino-tensorflow-frontend 2024.6.0             h630ec5c_3    conda-forge
libopenvino-tensorflow-lite-frontend 2024.6.0             h5888daf_3    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpciaccess              0.18                 hd590300_0    conda-forge
libpng                    1.6.46               h943b412_0    conda-forge
libprotobuf               5.28.3               h6128344_1    conda-forge
librsvg                   2.58.4               h49af25d_2    conda-forge
libsqlite                 3.48.0               hee588c1_1    conda-forge
libssh2                   1.11.1               hf672d98_0    conda-forge
libstdcxx                 14.2.0               hc0a3c3a_1    conda-forge
libstdcxx-ng              14.2.0               h4852527_1    conda-forge
libtiff                   4.7.0                hd9ff511_3    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libva                     2.22.0               h8a09558_1    conda-forge
libvpx                    1.14.1               hac33072_0    conda-forge
libwebp-base              1.5.0                h851e524_0    conda-forge
libxcb                    1.17.0               h8a09558_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxkbcommon              1.7.0                h2c5496b_1    conda-forge
libxml2                   2.13.5               h8d12d68_1    conda-forge
libzlib                   1.3.1                hb9d3cd8_2    conda-forge
mandrake                  1.2.2           py312h19fb687_12    conda-forge
matplotlib-base           3.10.0          py312hd3ec401_0    conda-forge
munkres                   1.0.7                      py_1    bioconda
ncurses                   6.5                  h2d0b736_2    conda-forge
networkx                  3.4.2              pyh267e887_2    conda-forge
numpy                     2.2.2           py312h72c5963_0    conda-forge
ocl-icd                   2.3.2                hb9d3cd8_2    conda-forge
openblas                  0.3.28          pthreads_h6ec200e_1    conda-forge
opencl-headers            2024.10.24           h5888daf_0    conda-forge
openh264                  2.5.0                hf92e6e3_0    conda-forge
openjpeg                  2.5.3                h5fbd93e_0    conda-forge
openssl                   3.4.0                h7b32b05_1    conda-forge
packaging                 24.2               pyhd8ed1ab_2    conda-forge
pandas                    2.2.3           py312hf9745cd_1    conda-forge
pango                     1.56.1               h861ebed_0    conda-forge
pcre2                     10.44                hba22ea6_2    conda-forge
pillow                    11.1.0          py312h80c1187_0    conda-forge
pip                       25.0               pyh8b19718_0    conda-forge
pixman                    0.44.2               h29eaf8c_0    conda-forge
plotly                    5.24.1             pyhd8ed1ab_1    conda-forge
poppunk                   2.7.5           py312hc15d307_0    bioconda
pp-sketchlib              2.1.3           py312h8043751_4    conda-forge
pthread-stubs             0.4               hb9d3cd8_1002    conda-forge
pugixml                   1.14                 h59595ed_0    conda-forge
pycairo                   1.27.0          py312h51b637d_0    conda-forge
pycparser                 2.22               pyh29332c3_1    conda-forge
pygobject                 3.50.0          py312hf4b392c_1    conda-forge
pyparsing                 3.2.1              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha55dd90_7    conda-forge
python                    3.12.8          h9e4cc4f_1_cpython    conda-forge
python-dateutil           2.9.0.post0        pyhff2d567_1    conda-forge
python-tzdata             2025.1             pyhd8ed1ab_0    conda-forge
python_abi                3.12                    5_cp312    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
qhull                     2020.2               h434a139_5    conda-forge
rapidnj                   2.3.2                h9948957_6    bioconda
readline                  8.2                  h8228510_1    conda-forge
requests                  2.32.3             pyhd8ed1ab_1    conda-forge
scikit-learn              1.6.1           py312h7a48858_0    conda-forge
scipy                     1.15.1          py312h180e4f1_0    conda-forge
setuptools                75.8.0             pyhff2d567_0    conda-forge
sigcpp-3.0                3.6.0                h59595ed_0    conda-forge
six                       1.17.0             pyhd8ed1ab_0    conda-forge
snappy                    1.2.1                h8bd8927_1    conda-forge
sparsehash                2.0.2                         0    bioconda
svt-av1                   2.3.0                h5888daf_0    conda-forge
sysroot_linux-64          2.17                h0157908_18    conda-forge
tbb                       2022.0.0             hceb3a55_0    conda-forge
tenacity                  9.0.0              pyhd8ed1ab_1    conda-forge
threadpoolctl             3.5.0              pyhc1e730c_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tqdm                      4.67.1             pyhd8ed1ab_1    conda-forge
treeswift                 1.1.45             pyh7e72e81_0    bioconda
tzdata                    2025a                h78e105d_0    conda-forge
unicodedata2              16.0.0          py312h66e93f0_0    conda-forge
urllib3                   2.3.0              pyhd8ed1ab_0    conda-forge
wayland                   1.23.1               h3e06ad9_0    conda-forge
wayland-protocols         1.37                 hd8ed1ab_0    conda-forge
wheel                     0.45.1             pyhd8ed1ab_1    conda-forge
x264                      1!164.3095           h166bdaf_2    conda-forge
x265                      3.5                  h924138e_3    conda-forge
xkeyboard-config          2.43                 hb9d3cd8_0    conda-forge
xorg-libice               1.1.2                hb9d3cd8_0    conda-forge
xorg-libsm                1.2.5                he73a12e_0    conda-forge
xorg-libx11               1.8.10               h4f16b4b_1    conda-forge
xorg-libxau               1.0.12               hb9d3cd8_0    conda-forge
xorg-libxaw               1.0.16               hb9d3cd8_0    conda-forge
xorg-libxcomposite        0.4.6                hb9d3cd8_2    conda-forge
xorg-libxcursor           1.2.3                hb9d3cd8_0    conda-forge
xorg-libxdamage           1.1.6                hb9d3cd8_0    conda-forge
xorg-libxdmcp             1.1.5                hb9d3cd8_0    conda-forge
xorg-libxext              1.3.6                hb9d3cd8_0    conda-forge
xorg-libxfixes            6.0.1                hb9d3cd8_0    conda-forge
xorg-libxi                1.8.2                hb9d3cd8_0    conda-forge
xorg-libxinerama          1.1.5                h5888daf_1    conda-forge
xorg-libxmu               1.2.1                hb9d3cd8_1    conda-forge
xorg-libxpm               3.5.17               hb9d3cd8_1    conda-forge
xorg-libxrandr            1.5.4                hb9d3cd8_0    conda-forge
xorg-libxrender           0.9.12               hb9d3cd8_0    conda-forge
xorg-libxt                1.3.1                hb9d3cd8_0    conda-forge
xorg-libxtst              1.2.5                hb9d3cd8_3    conda-forge
xorg-xextproto            7.3.0             hb9d3cd8_1004    conda-forge
zlib                      1.3.1                hb9d3cd8_2    conda-forge
zstandard                 0.23.0          py312hef9b889_1    conda-forge
zstd                      1.5.6                ha6fb4c9_0    conda-forge

Command used and output returned

# download poppunk reference for Pseudomonas aeruginosa
wget https://ftp.ebi.ac.uk/pub/databases/pp_dbs/Pseudomonas_aeruginosa_v1_refs.tar.bz2

# decompress the file
tar -xvjf Pseudomonas_aeruginosa_v1_refs.tar.bz2

# run poppunk assign
poppunk_assign --db pa_v1_refs --query query_list.txt --output clusters --threads 12

This is the error that I get

PopPUNK: assign
  	(with backend: sketchlib v2.1.4
  	 sketchlib: /home/eriny/miniconda3/envs/poppunk/lib/python3.12/site-packages/[pp_sketchlib.cpython-312-x86_64-linux-gnu.so](http://pp_sketchlib.cpython-312-x86_64-linux-gnu.so/))
  Mode: Assigning clusters of query sequences
  
  
  Graph-tools OpenMP parallelisation enabled: with 12 threads
  Traceback (most recent call last):
    File "/home/eriny/miniconda3/envs/poppunk/bin/poppunk_assign", line 11, in 
      sys.exit(main())
               ^^^^^^
    File "/home/eriny/miniconda3/envs/poppunk/lib/python3.12/site-packages/PopPUNK/assign.py", line 217, in main
      assign_query(dbFuncs,
    File "/home/eriny/miniconda3/envs/poppunk/lib/python3.12/site-packages/PopPUNK/assign.py", line 288, in assign_query
      kmers, sketch_sizes, codon_phased = readDBParams(ref_db)
                                          ^^^^^^^^^^^^^^^^^^^^
    File "/home/eriny/miniconda3/envs/poppunk/lib/python3.12/site-packages/PopPUNK/sketchlib.py", line 188, in readDBParams
      db_kmers = getKmersFromReferenceDatabase(dbPrefix)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/eriny/miniconda3/envs/poppunk/lib/python3.12/site-packages/PopPUNK/sketchlib.py", line 155, in getKmersFromReferenceDatabase
      ref_db = h5py.File(db_file, 'r')
               ^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/eriny/miniconda3/envs/poppunk/lib/python3.12/site-packages/h5py/_hl/files.py", line 561, in __init__
      fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/eriny/miniconda3/envs/poppunk/lib/python3.12/site-packages/h5py/_hl/files.py", line 235, in make_fid
      fid = h5f.open(name, flags, fapl=fapl)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
    File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
    File "h5py/h5f.pyx", line 102, in h5py.h5f.open
  FileNotFoundError: [Errno 2] Unable to synchronously open file (unable to open file: name = 'pa_v1_refs/pa_v1_refs.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

(Note: I also tried this with https://ftp.ebi.ac.uk/pub/databases/pp_dbs/Pseudomonas_aeruginosa_v1_full.tar.bz2)

Describe the bug
I guess poppunk is looking for a file that doesn't exist?

This is everything in the directory I was using

$ ls pa_v1_refs
pa_pruned_threshold_005_clusters.csv  pa_pruned_threshold_005_refined_fit.png  pa_pruned_threshold_005.refs.dists.pkl  pa_pruned_threshold_005_unword_clusters.csv
pa_pruned_threshold_005_fit.npz       pa_pruned_threshold_005.refs             pa_pruned_threshold_005.refs_graph.gt
pa_pruned_threshold_005_fit.pkl       pa_pruned_threshold_005.refs.dists.npy   pa_pruned_threshold_005.refs.h5

[johnlees]

The directory name and prefix need to match. If you rename the directory pa_v1_refs to pa_pruned_threshold_005 I think it should work.

If it does let us know, we should rename this in the downloads on the ftp

[erinyoung]

Thank you for the suggestion!

I tried it and have a NEW error:

  Graph-tools OpenMP parallelisation enabled: with 12 threads
  Sketching 313 genomes using 12 thread(s)
  
  Progress (CPU): 0 / 313
  Progress (CPU): 12 / 313
  Progress (CPU): 16 / 313
  Progress (CPU): 30 / 313
  Progress (CPU): 44 / 313
  Progress (CPU): 54 / 313
  Progress (CPU): 64 / 313
  Progress (CPU): 73 / 313
  Progress (CPU): 86 / 313
  Progress (CPU): 98 / 313
  Progress (CPU): 109 / 313
  Progress (CPU): 121 / 313
  Progress (CPU): 133 / 313
  Progress (CPU): 146 / 313
  Progress (CPU): 158 / 313
  Progress (CPU): 170 / 313
  Progress (CPU): 182 / 313
  Progress (CPU): 194 / 313
  Progress (CPU): 207 / 313
  Progress (CPU): 221 / 313
  Progress (CPU): 231 / 313
  Progress (CPU): 242 / 313
  Progress (CPU): 255 / 313
  Progress (CPU): 270 / 313
  Progress (CPU): 278 / 313
  Progress (CPU): 290 / 313
  Progress (CPU): 302 / 313
  Progress (CPU): 307 / 313
  Progress (CPU): 310 / 313
  Progress (CPU): 313 / 313
  Writing sketches to file
  Traceback (most recent call last):
    File "/home/eriny/miniconda3/envs/poppunk/bin/poppunk_assign", line 11, in 
      sys.exit(main())
               ^^^^^^
    File "/home/eriny/miniconda3/envs/poppunk/lib/python3.12/site-packages/PopPUNK/assign.py", line 217, in main
      assign_query(dbFuncs,
    File "/home/eriny/miniconda3/envs/poppunk/lib/python3.12/site-packages/PopPUNK/assign.py", line 303, in assign_query
      isolateClustering = assign_query_hdf5(dbFuncs,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/eriny/miniconda3/envs/poppunk/lib/python3.12/site-packages/PopPUNK/assign.py", line 425, in assign_query_hdf5
      model = loadClusterFit(model_file + '.pkl',
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/eriny/miniconda3/envs/poppunk/lib/python3.12/site-packages/PopPUNK/models.py", line 104, in loadClusterFit
      with open(pkl_file, 'rb') as pickle_obj:
           ^^^^^^^^^^^^^^^^^^^^
  FileNotFoundError: [Errno 2] No such file or directory: 'pa_pruned_1sd/pa_pruned_1sd_fit.pkl'

[johnlees]

`FileNotFoundError: [Errno 2] No such file or directory: 'pa_pruned_1sd/pa_pruned_1sd_fit.pkl'

This is file pa_pruned_threshold_005_fit.pkl in your original post (and the matching .npz file). You need to copy/rename this into the directory with the pa_pruned_1sd prefix