BUG Input files perceived as NULL while they exist (checked multiple times)

[ulyssebaruchel]

Hi, I have been trying to run zUMIs on a HPC (both through a sbatch job and on an interactive node), but did not succeed. It seems like it does not see my input files as the Smartseq3.zUMIs_YAMLerror.log says:

---

---

WARNING: ignoring environment value of R_HOME
$file1
NULL

$file2
NULL

$file3
NULL

$file4
NULL

[1] ""
[1] ""
[1] ""
[1] ""
[1] "" "" "" ""
[1] ""
[1] ""
[1] ""
[1] ""
[1] "" "" "" ""
[1] "NULL" "NULL" "NULL" "NULL"
$file1
NULL

$file2
NULL

$file3
NULL

$file4
NULL

$file1
NULL

$file2
NULL

$file3
NULL

$file4
NULL

[1] 0

---

---

This is my YAML file:

---

---

project: Smartseq3
sequence_files:
file1:
name: /home/ubaruchel/smart-seq3/data/240814/exp1/1a_cutadapt/Undetermined_S0_L001_trim_R1.fastq.gz
base_definition:
- cDNA(24-75)
- UMI(12-20)
find_pattern: ATTGCGCAATG
file2:
name: /home/ubaruchel/smart-seq3/data/240814/exp1/1a_cutadapt/Undetermined_S0_L001_trim_R2.fastq.gz
base_definition:
- cDNA(1-75)
file3:
name: /home/ubaruchel/smart-seq3/data/240814/exp1/1c_filter_index_reads/filtered_I1.fastq.gz
base_definition:
- BC(1-10)
file4:
name: /home/ubaruchel/smart-seq3/data/240814/exp1/1c_filter_index_reads/filtered_I2.fastq.gz
base_definition:
- BC(1-10)
reference:
STAR_index: /data/scratch/DBC/UBCN/CANCDYN/genomes/homo-sapiens/hg38-ercc/star
GTF_file: /data/scratch/DBC/UBCN/CANCDYN/genomes/homo-sapiens/hg38-ercc/gtf/combined_hg38_ercc.gtf
out_dir: /home/ubaruchel/smart-seq3/data/240814/exp1/2b_zUMIs
num_threads: 24
mem_limit: 50
filter_cutoffs:
BC_filter:
num_bases: 3
phred: 20
UMI_filter:
num_bases: 2
phred: 20
barcodes:
barcode_num: ~
barcode_file: /home/ubaruchel/smart-seq3/data/240814/exp1/0c_prep_well_barcodes/expected_well_barcodes.txt
automatic: no
BarcodeBinning: 1
nReadsperCell: 100
demultiplex: no
counting_opts:
introns: yes
downsampling: '0'
strand: 0
Ham_Dist: 1
write_ham: no
velocyto: no
primaryHit: yes
twoPass: no
make_stats: yes
which_Stage: Filtering
zUMIs_directory: /data/scratch/DBC/UBCN/CANCDYN/software/zUMIs

samtools_exec: samtools
pigz_exec: pigz
STAR_exec: STAR
Rscript_exec: Rscript

---

---

I ran this command through a .sh file that is called through a sbatch script (SLURM):

---

---

#!/bin/bash

Always add these two commands to your scripts when using a environment

eval "$(conda shell.bash hook)"
source $CONDA_PREFIX/etc/profile.d/mamba.sh

Source the parameters file

source ./params_bioinfo_experiments/0_params.sh

Set variables

input_dir=$input_dir_2b
output_dir=$output_dir_2b
log_dir=$log_dir_2b

Create the output and log directories if they don't exist

mkdir -p "$output_dir"
mkdir -p "$log_dir"

Run zUMIs using its own miniconda environment (-c)

and the prepared YAML file (input_dir)

$path_zUMIs/zUMIs.sh -c -y $input_dir

$path_zUMIs/zUMIs.sh -c -y $input_dir

---

I do not know what the problem is. My hypothesis is that maybe the micoconda environment makes it not see in the input files (that do exist and are not empty as verified by the du -sh command). But at the same time it seems to be able to detect a slight discrepancy in the STAR versions used for my index (which means it does not see it as NULL) vs the one used by zUMIs.

Can you help me, please?

I have also tried to make my own mamba (conda) environment to run zUMIs following the vignette https://github.com/sdparekh/zUMIs/wiki/Installation#dependencies but I have not been able to complete the last part of the dependencies installation: devtools::install_github('VPetukhov/ggrastr') (some issues with Cairo)... And Docker is not accepted by HPCs (for security reasons)... Is there anyway you could make it into a Singularity file, please? This would make it much easier to deploy and in particular into pipelines (Nextflow / Snakemake)...

Thank you very much,

Best wishes,

Ulysse

[ulyssebaruchel]

@sdparekh I have noticed the YAMLerror.log has the same error even going back to a few years. Do you know how I may solve this issue, please? Thank you