Release 2.3.0 #137

Daniel-VM · 2024-05-24T10:45:17Z

PR checklist

PR description

This PR is for release 2.3.0

Major changes are:

Implementation of Kmerfinder subworkflow for read quality control, purity assessment and sample grouping based on reference genome estimation.
Enhancement of QUAST allowig to run a general QUAST (without reference genomes) and a reference genome-based quast analysis when kmerfinder is invoked.
Replacement of MULTIQC module by CUSTOM_MULTIQC module allowing custom reports based on assembly type (short, long, hybrid).
Update of the nf-core template v2.14.1

…into buisciii-develop

Implementation of KmerFinder subworkflow Custom Quast, and Custom MultiQC Reports

bump version 2.3.0

github-actions · 2024-05-24T10:47:25Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit dafb9f4

+| ✅ 195 tests passed       |+
#| ❔   4 tests were ignored |#
!| ❗   1 tests had warnings |!

❗ Test warnings:

nextflow_config - Config manifest.version should end in dev: 2.3.0

❔ Tests ignored:

files_exist - File is ignored: conf/igenomes.config
nextflow_config - Config default ignored: params.dfast_config
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-bacass_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-bacass_logo_light.png
files_exist - File found: docs/images/nf-core-bacass_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-bacass_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowBacass.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.assembler= unicycler
nextflow_config - Config default value correct: params.assembly_type= short
nextflow_config - Config default value correct: params.polish_method= medaka
nextflow_config - Config default value correct: params.annotation_tool= prokka
nextflow_config - Config default value correct: params.baktadb_download_args= --type light
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.max_cpus= 16
nextflow_config - Config default value correct: params.max_memory= 128.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.schema_ignore_params= modules,igenomes_base
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-bacass_logo_light.png matches the template
files_unchanged - docs/images/nf-core-bacass_logo_light.png matches the template
files_unchanged - docs/images/nf-core-bacass_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
readme - README Zenodo placeholder was replaced with DOI.
pipeline_todos - No TODO strings found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (231 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/tsv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
modules_config - conf/modules.config found and not ignored.
modules_config - NANOPLOT found in conf/modules.config and Nextflow scripts.
modules_config - PYCOQC found in conf/modules.config and Nextflow scripts.
modules_config - PORECHOP_PORECHOP found in conf/modules.config and Nextflow scripts.
modules_config - UNICYCLER found in conf/modules.config and Nextflow scripts.
modules_config - CANU found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_ALIGN found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_CONSENSUS found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_POLISH found in conf/modules.config and Nextflow scripts.
modules_config - MINIASM found in conf/modules.config and Nextflow scripts.
modules_config - DRAGONFLYE found in conf/modules.config and Nextflow scripts.
modules_config - RACON found in conf/modules.config and Nextflow scripts.
modules_config - MEDAKA found in conf/modules.config and Nextflow scripts.
modules_config - NANOPOLISH found in conf/modules.config and Nextflow scripts.
modules_config - KRAKEN2 found in conf/modules.config and Nextflow scripts.
modules_config - KRAKEN2_LONG found in conf/modules.config and Nextflow scripts.
modules_config - QUAST found in conf/modules.config and Nextflow scripts.
modules_config - PROKKA found in conf/modules.config and Nextflow scripts.
modules_config - DFAST found in conf/modules.config and Nextflow scripts.
modules_config - MULTIQC_CUSTOM found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 2.14.1

Run details

nf-core/tools version 2.14.1
Run at 2024-06-12 14:30:27

FriederikeHanssen

Looks super good. If you could add the missing licenses that would be great. People often just leave a note at the top with MIT or whatever they choose. i can point you to examples if you like

bin/csv_to_yaml.py

FriederikeHanssen · 2024-05-24T14:22:47Z

modules/local/multiqc_custom.nf

+    ## Collect additional files to be included in the report
+    if [ -d extra/ ]; then
+        cp extra/* multiqc_data/
+    fi
+
+    ## Create multiqc custom data
+    multiqc_to_custom_csv.py --assembly_type $params.assembly_type
+
+    ## Avoid the custom Multiqc table when the kmerfinder process is not invoked.
+    if grep ">skip_kmerfinder<" workflow_summary_mqc.yaml; then
+        rm *_assembly_metrics_mqc.csv
+    fi


I was wondering about this when looking at the changelog: so the choice to go with a custom version is to run mqc twice to get modified data and plot it? Just in case you didn't know about this, there is also the nf-core diff command that allows you to modify an nf-core/module, but keep getting the updates, like new versions etc.

First, it generates multiqc_data from the input tools to parse tool logs. Second, it collects extra data (such as from KmerFinder) and formats it to be parsed by the second MultiQC run, creating custom tables for inclusion in the custom MultiQC report. Finally, it runs a second MultiQC with all the generated data and emits a custom report.

It is inspired by and sourced from nf-core/viralrecon custom multiqc

I tried to make a patch, but MultiQC version 1.21 does not properly parse this "file set up", and it might require a deeper refactor. (it worked with version v1.19).

Interesting, might be worth bringing up with the MutliQC devs to see what the issue is and fix it soon.

Sure, that would be great. Is it OK if I add this to a new git issue?

yes absolutely, you can also ask in the #multiqc channel on slack

subworkflows/local/kmerfinder_subworkflow.nf

workflows/bacass.nf

Daniel-VM · 2024-05-25T10:03:00Z

Thank you @FriederikeHanssen :). Reviewer suggestions are in #138

implementation of reviewer suggestions before v2.3.0 release

d4straub

Looks good to me, just low priority comments that I trust you will solve if you deem necessary.

README.md

bin/multiqc_to_custom_csv.py

conf/test_full.config

docs/output.md

nextflow_schema.json

subworkflows/local/kmerfinder_subworkflow.nf

Daniel-VM · 2024-05-30T09:28:17Z

Thanks @d4straub. Due to Git issues I couldn't apply your suggestions directly on this PR. I have created a new PR to add the code review: #139.

Reviewer 2 suggestions before v2.3.0 release

Daniel-VM · 2024-06-04T06:48:04Z

Hi @FriederikeHanssen, I believe all the suggestions have been addressed except for the MultiQC issue. (We can create a specific issue to resolve that.) If there is anything else pending for review, please pin me and I will address it.

bin/kmerfinder_summary.py

Added missing license in `kmerfinder_summary.py`

CHANGELOG.md

Update date of release in CHANGELOG.md

Daniel-VM added 30 commits November 6, 2023 16:55

add kmerfinder for shortreads

ebcd833

add module kmerfinder summary report

68a9c5d

add module to find and download reference genome

fd8a934

fix kmerfinder summary input

c81cc29

update kmerfinder output file extension

f989ae3

add kmerfinder refseqid to meta

20be5f9

fix url in kmerfinder donwload ref

16b5068

temporary commit

10f68ba

group assemblies by refseqid

a48518d

allow global quast and by-refseqid quast

1e436ea

move kmerfinder processing to subworkflow plus output refactoring

7969e11

Merge branch 'bu-isciii_tmp' into buisciii-develop

ef85494

allow quast to standard and byrefseq data

3e8d9f1

add byrefseq quast reports to multiqc and patch quast

53cc7ec

add byrefseq quast reports to multiqc and patch quast

7aff2f6

update multiqc and append fastp metrics to assmebly metrics df

de8e877

add new method to complie kmerfinder results into multiqc report

7103bfa

add long reads assembly metrics to custom multiqc

854782d

Merge branch 'buisciii-develop' of https://github.com/Daniel-VM/bacass …

6fb64b3

…into buisciii-develop

fix custom multiqc when kmerfinder is not invoked

0f3970e

add custom multiqc for hybrid assembly

247f592

add file-check-exist and rename variables

df1403d

update documentation and add save_trimmed option

c42735e

add fastp additional options and fix input sample path

3357785

allow module to emit tsv report

1847f39

add kmerfinder for shortreads

b73e3f0

add module kmerfinder summary report

764a3d0

add module to find and download reference genome

0f07718

fix kmerfinder summary input

e072b2d

update kmerfinder output file extension

8baded1

Daniel-VM and others added 4 commits May 23, 2024 16:51

fix test_long_miniasm git CI test in #135

7b66a5e

Merge pull request #135 from Daniel-VM/dev

7114a6d

Implementation of KmerFinder subworkflow Custom Quast, and Custom MultiQC Reports

bump version 2.3.0

031703d

Merge pull request #136 from Daniel-VM/dev

4f3843f

bump version 2.3.0

Daniel-VM marked this pull request as ready for review May 24, 2024 10:50

FriederikeHanssen reviewed May 24, 2024

View reviewed changes

added reviewer suggestions before v2.3.0 release

156e023

Daniel-VM mentioned this pull request May 25, 2024

implementation of reviewer suggestions before v2.3.0 release #138

Merged

9 tasks

Merge pull request #138 from Daniel-VM/dev

c47b461

implementation of reviewer suggestions before v2.3.0 release

Daniel-VM requested a review from d4straub May 27, 2024 08:50

d4straub approved these changes May 29, 2024

View reviewed changes

reviewer 2 suggestions before v2.3.0 release

6e94fc4

Daniel-VM mentioned this pull request May 30, 2024

Reviewer 2 suggestions before v2.3.0 release #139

Merged

10 tasks

Daniel-VM and others added 2 commits June 3, 2024 08:39

add stable url

cb1c258

Merge pull request #139 from Daniel-VM/dev

6d91eb5

Reviewer 2 suggestions before v2.3.0 release

Daniel-VM requested a review from FriederikeHanssen June 3, 2024 08:38

FriederikeHanssen reviewed Jun 12, 2024

View reviewed changes

bin/kmerfinder_summary.py Outdated Show resolved Hide resolved

added missing liscence

9d48eef

Daniel-VM mentioned this pull request Jun 12, 2024

Added missing license in kmerfinder_summary.py #140

Merged

10 tasks

Merge pull request #140 from Daniel-VM/dev

71ac73a

Added missing license in `kmerfinder_summary.py`

FriederikeHanssen reviewed Jun 12, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

FriederikeHanssen approved these changes Jun 12, 2024

View reviewed changes

Update date of release in CHANGELOG.md #137

7756bd5

Daniel-VM mentioned this pull request Jun 12, 2024

Update date of release in CHANGELOG.md #141

Merged

10 tasks

Merge pull request #141 from Daniel-VM/dev

dafb9f4

Update date of release in CHANGELOG.md

Daniel-VM merged commit d230ebf into master Jun 12, 2024
21 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release 2.3.0 #137

Release 2.3.0 #137

Daniel-VM commented May 24, 2024

github-actions bot commented May 24, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

FriederikeHanssen left a comment

FriederikeHanssen May 24, 2024

Daniel-VM May 25, 2024

Daniel-VM May 25, 2024

FriederikeHanssen May 29, 2024

Daniel-VM May 30, 2024

FriederikeHanssen Jun 12, 2024

Daniel-VM commented May 25, 2024

d4straub left a comment

Daniel-VM commented May 30, 2024

Daniel-VM commented Jun 4, 2024

Release 2.3.0 #137

Release 2.3.0 #137

Conversation

Daniel-VM commented May 24, 2024

PR checklist

PR description

github-actions bot commented May 24, 2024 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

FriederikeHanssen left a comment

Choose a reason for hiding this comment

FriederikeHanssen May 24, 2024

Choose a reason for hiding this comment

Daniel-VM May 25, 2024

Choose a reason for hiding this comment

Daniel-VM May 25, 2024

Choose a reason for hiding this comment

FriederikeHanssen May 29, 2024

Choose a reason for hiding this comment

Daniel-VM May 30, 2024

Choose a reason for hiding this comment

FriederikeHanssen Jun 12, 2024

Choose a reason for hiding this comment

Daniel-VM commented May 25, 2024

d4straub left a comment

Choose a reason for hiding this comment

Daniel-VM commented May 30, 2024

Daniel-VM commented Jun 4, 2024

github-actions bot commented May 24, 2024 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️