Update/locidex #96

mattheww95 · 2024-07-30T15:28:49Z

Draft PR to verify existing unit tests still run.

github-actions · 2024-07-30T15:30:20Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit de0d30e

+| ✅ 223 tests passed       |+
#| ❔  31 tests were ignored |#
!| ❗   3 tests had warnings |!

❗ Test warnings:

nextflow_config - Config manifest.version should end in dev: 0.3.0
readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
schema_lint - Schema $id should be https://raw.githubusercontent.com/phac-nml/mikrokondo/master/nextflow_schema.json
Found https://raw.githubusercontent.com/phac-nml/mikrokondo/main/nextflow_schema.json

❔ Tests ignored:

files_exist - File is ignored: CODE_OF_CONDUCT.md
files_exist - File is ignored: assets/nf-core-mikrokondo_logo_light.png
files_exist - File is ignored: docs/images/nf-core-mikrokondo_logo_light.png
files_exist - File is ignored: docs/images/nf-core-mikrokondo_logo_dark.png
files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: docs/output.md
files_exist - File is ignored: docs/README.md
files_exist - File is ignored: docs/usage.md
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
files_unchanged - File does not exist: CODE_OF_CONDUCT.md
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/config.yml
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/feature_request.yml
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/email_template.html
files_unchanged - File ignored due to lint config: assets/email_template.txt
files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
files_unchanged - File does not exist: assets/nf-core-mikrokondo_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-mikrokondo_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-mikrokondo_logo_dark.png
files_unchanged - File does not exist: docs/README.md
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/mikrokondo/mikrokondo/.github/workflows/awstest.yml
multiqc_config - multiqc_config

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-mikrokondo_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowMikrokondo.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.platform= illumina
nextflow_config - Config default value correct: params.long_read_opt= nanopore
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.max_cpus= 16
nextflow_config - Config default value correct: params.max_memory= 2000.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.tracedir= null/pipeline_info
nextflow_config - Config default value correct: params.stage_in_mode= symlink
nextflow_config - Config default value correct: params.validationS3PathCheck= true
nextflow_config - Config default value correct: params.skip_bakta= true
nextflow_config - Config default value correct: params.skip_ont_header_cleaning= true
nextflow_config - Config default value correct: params.ec_opid= 90.0
nextflow_config - Config default value correct: params.ec_opcov= 90.0
nextflow_config - Config default value correct: params.ec_hpid= 95.0
nextflow_config - Config default value correct: params.ec_hpcov= 50.0
nextflow_config - Config default value correct: params.ec_enable_verification= true
nextflow_config - Config default value correct: params.sr_full_cgmlst= true
nextflow_config - Config default value correct: params.fp_average_quality= 25
nextflow_config - Config default value correct: params.fp_cut_tail_mean_quality= 15
nextflow_config - Config default value correct: params.fp_cut_tail_window_size= 4
nextflow_config - Config default value correct: params.fp_complexity_threshold= 20
nextflow_config - Config default value correct: params.fp_qualified_phred= 15
nextflow_config - Config default value correct: params.fp_unqualified_percent_limit= 40
nextflow_config - Config default value correct: params.fp_polyg_min_len= 10
nextflow_config - Config default value correct: params.fp_polyx_min_len= 10
nextflow_config - Config default value correct: params.fp_illumina_length_min= 35
nextflow_config - Config default value correct: params.fp_illumina_length_max= 400
nextflow_config - Config default value correct: params.fp_single_end_length_min= 1000
nextflow_config - Config default value correct: params.lx_min_evalue= 0.0001
nextflow_config - Config default value correct: params.lx_min_dna_len= 1
nextflow_config - Config default value correct: params.lx_min_aa_len= 1
nextflow_config - Config default value correct: params.lx_max_dna_len= 10000000
nextflow_config - Config default value correct: params.lx_max_aa_len= 10000000
nextflow_config - Config default value correct: params.lx_min_dna_ident= 80.0
nextflow_config - Config default value correct: params.lx_min_aa_ident= 80.0
nextflow_config - Config default value correct: params.lx_min_dna_match_cov= 80.0
nextflow_config - Config default value correct: params.lx_min_aa_match_cov= 80.0
nextflow_config - Config default value correct: params.lx_max_target_seqs= 10
nextflow_config - Config default value correct: params.lx_extraction_mode= raw
nextflow_config - Config default value correct: params.lx_report_mode= normal
nextflow_config - Config default value correct: params.lx_report_prop= locus_name
nextflow_config - Config default value correct: params.lx_report_max_ambig= 0
nextflow_config - Config default value correct: params.lx_report_max_stop= 0
nextflow_config - Config default value correct: params.target_depth= 100
nextflow_config - Config default value correct: params.min_reads= 1000
nextflow_config - Config default value correct: params.ba_min_contig_length= 200
nextflow_config - Config default value correct: params.qt_min_contig_length= 1000
nextflow_config - Config default value correct: params.mh_min_kmer= 10
nextflow_config - Config default value correct: params.flye_read_type= hq
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
pipeline_todos - No TODO strings found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (293 files)
schema_lint - Schema lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: spellcheck.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: doc.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
base_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/base.config and Nextflow scripts.
modules_config - conf/modules.config found and not ignored.
modules_config - PUBLISH_FINAL_READS found in conf/modules.config and Nextflow scripts.
modules_config - PUBLISH_FINAL_ASSEMBLIES found in conf/modules.config and Nextflow scripts.
modules_config - LOCIDEX_EXTRACT found in conf/modules.config and Nextflow scripts.
modules_config - LOCIDEX_SEARCH found in conf/modules.config and Nextflow scripts.
modules_config - LOCIDEX_REPORT found in conf/modules.config and Nextflow scripts.
modules_config - REPORT found in conf/modules.config and Nextflow scripts.
modules_config - IDENTIFY_POINTDB found in conf/modules.config and Nextflow scripts.
modules_config - LOCIDEX_SELECT found in conf/modules.config and Nextflow scripts.
modules_config - LOCIDEX_SUMMARIZE found in conf/modules.config and Nextflow scripts.
modules_config - REPORT_AGGREGATE found in conf/modules.config and Nextflow scripts.
modules_config - BIN_KRAKEN2 found in conf/modules.config and Nextflow scripts.
modules_config - COMBINE_DATA found in conf/modules.config and Nextflow scripts.
modules_config - GZIP_FILES found in conf/modules.config and Nextflow scripts.
modules_config - CHECK_ONT found in conf/modules.config and Nextflow scripts.
modules_config - PARSE_MASH found in conf/modules.config and Nextflow scripts.
modules_config - PARSE_KRAKEN found in conf/modules.config and Nextflow scripts.
modules_config - READ_SCAN found in conf/modules.config and Nextflow scripts.
modules_config - PARSE_FASTP found in conf/modules.config and Nextflow scripts.
modules_config - FASTQC found in conf/modules.config and Nextflow scripts.
modules_config - SEQKIT_STATS found in conf/modules.config and Nextflow scripts.
modules_config - SEQKIT_FILTER found in conf/modules.config and Nextflow scripts.
modules_config - SEQTK_SAMPLE found in conf/modules.config and Nextflow scripts.
modules_config - SEQTK_SIZE found in conf/modules.config and Nextflow scripts.
modules_config - QUAST found in conf/modules.config and Nextflow scripts.
modules_config - CHECKM_LINEAGEWF found in conf/modules.config and Nextflow scripts.
modules_config - BANDAGE_IMAGE found in conf/modules.config and Nextflow scripts.
modules_config - KRAKEN found in conf/modules.config and Nextflow scripts.
modules_config - MASH_ESTIMATE found in conf/modules.config and Nextflow scripts.
modules_config - MLST found in conf/modules.config and Nextflow scripts.
modules_config - STARAMR found in conf/modules.config and Nextflow scripts.
modules_config - MOBSUITE_RECON found in conf/modules.config and Nextflow scripts.
modules_config - MASH_SKETCH found in conf/modules.config and Nextflow scripts.
modules_config - MASH_PASTE found in conf/modules.config and Nextflow scripts.
modules_config - MASH_SCREEN found in conf/modules.config and Nextflow scripts.
modules_config - REMOVE_CONTAMINANTS found in conf/modules.config and Nextflow scripts.
modules_config - FLYE_ASSEMBLE found in conf/modules.config and Nextflow scripts.
modules_config - SPADES_ASSEMBLE found in conf/modules.config and Nextflow scripts.
modules_config - UNICYCLER_ASSEMBLE found in conf/modules.config and Nextflow scripts.
modules_config - FASTP_TRIM found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_INDEX found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_MAP found in conf/modules.config and Nextflow scripts.
modules_config - SAM_TO_BAM found in conf/modules.config and Nextflow scripts.
modules_config - RACON_POLISH found in conf/modules.config and Nextflow scripts.
modules_config - PILON_ITER found in conf/modules.config and Nextflow scripts.
modules_config - MEDAKA_POLISH found in conf/modules.config and Nextflow scripts.
modules_config - BAKTA_DB_DOWNLOAD found in conf/modules.config and Nextflow scripts.
modules_config - BAKTA_ANNOTATE found in conf/modules.config and Nextflow scripts.
modules_config - ABRICATE found in conf/modules.config and Nextflow scripts.
modules_config - ECTYPER found in conf/modules.config and Nextflow scripts.
modules_config - KLEBORATE found in conf/modules.config and Nextflow scripts.
modules_config - SPATYPER found in conf/modules.config and Nextflow scripts.
modules_config - SISTR found in conf/modules.config and Nextflow scripts.
modules_config - LISSERO found in conf/modules.config and Nextflow scripts.
modules_config - SHIGEIFINDER found in conf/modules.config and Nextflow scripts.
modules_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 2.14.1

Run details

nf-core/tools version 2.14.1
Run at 2024-08-23 15:16:21

Perhaps the relevant fields should be sub set out

conf/modules.config

conf/test.config

modules/local/locidex_select.nf

nextflow_schema.json

subworkflows/local/annotate_genomes.nf

subworkflows/local/locidex.nf

tests/data/databases/locidex_dbs/locidex_salm/config.json

kylacochrane

This looks amazing Matthew 😃 I really delved in and learned a lot reviewing it! Thanks for that!

modules/local/locidex_select.nf

kylacochrane

Fantastic work, Matthew! Your detailed comments throughout the code are really appreciated—they make everything so much easier to follow. I found your updates in locidex_search.nf particularly impressive, and the explanations really clarify the changes. Thanks for taking the time to document everything so thoroughly!

apetkau

Thanks so much for your amazing work Matthew. I still have to continue on with some testing, but I have some initial feedback for you below 😄

modules/local/locidex_select.nf

modules/local/locidex_summarize.nf

tests/modules/local/locidex_select/locidex_select.nf.test

tests/nextflow.config

conf/test.config

Thank you Aaron!

apetkau

Thanks so much for your amazing work @mattheww95. And for the time you spend testing/providing feedback @kylacochrane 😄

I have marked as resolved most of the comments I had previously Matthew (thanks) or replied to some of them for additional changes/information.

I did have a few more comments for you as well.

Also, I do have some questions about how passing an allele_scheme will now work in IRIDA Next, but I think that should be a discussion we have.

CHANGELOG.md

tests/nextflow.config

modules/local/locidex_select.nf

modules/local/locidex_summarize.nf

apetkau

Everything looks good to me. Thanks so much Matthew 😄

If all you needed to fix for the tests to pass is to set terminate for one process in the test case, then you can do that and then merge the PR.

mattheww95 added 6 commits July 29, 2024 10:50

starting locidex database integrations

c342ef4

updated added new files

4eff5ce

updated locidex db identification

b06371b

updated automated allele selection

e711e24

updated todos

df4467f

added test data for locidex databases, tests should still function:

caba99c

mattheww95 added 8 commits July 30, 2024 10:36

updated nextflow schema.json

c5659dd

Removed tests for comparing updates for old and new locidex profiles

b172dcb

Perhaps the relevant fields should be sub set out

removed dead test file

41c1893

updated test profile

a32dd11

restrucutured file staging for locidex select

6b4d283

started creation of real tests for locidex workflow

fd6a1a6

updated locidex test data

120b148

updated locidex workflow and tests

37b7a46

mattheww95 requested review from emarinier and kylacochrane August 1, 2024 20:17

updated docker container

384fbce

emarinier requested changes Aug 2, 2024

View reviewed changes

addressed PR issues, still more work to be done

dccd27f

kylacochrane reviewed Aug 2, 2024

View reviewed changes

modules/local/locidex_select.nf Outdated Show resolved Hide resolved

modules/local/locidex_select.nf Outdated Show resolved Hide resolved

modules/local/locidex_select.nf Outdated Show resolved Hide resolved

modules/local/locidex_select.nf Outdated Show resolved Hide resolved

mattheww95 added 9 commits August 2, 2024 15:08

updated database selection functions to get an optimal match

2fb4ac9

added skipped allele calling to test config

1d43ecb

updated test conditions

19fa250

updated test files

21681c2

updated locidex select paths

3cd4a28

removed trailing whitespace

607c1c1

updated tests to include passing cases

dd5c992

added locidex summary process

0dfe87f

updated fixe issues with integer sizing

bae1567

made updates on PR comments

4883b39

mattheww95 marked this pull request as draft August 12, 2024 20:57

mattheww95 added 2 commits August 12, 2024 16:32

normalized output for iridanext across allele scheme input options

673349a

updated test cases

e3d3c8a

kylacochrane reviewed Aug 13, 2024

View reviewed changes

modules/local/locidex_select.nf Outdated Show resolved Hide resolved

kylacochrane approved these changes Aug 13, 2024

View reviewed changes

mattheww95 added 3 commits August 13, 2024 09:42

updated locidex_select tests to match updated interface

d97e6ce

updated test to work in github actions

a341fc6

removed sneaky todo

e938535

mattheww95 marked this pull request as ready for review August 13, 2024 15:46

mattheww95 requested a review from emarinier August 13, 2024 15:47

mattheww95 added this to the Add an Allele Caller milestone Aug 13, 2024

apetkau requested changes Aug 16, 2024

View reviewed changes

updating issues identified and regression in kraken2 header parsing

215a70b

mattheww95 mentioned this pull request Aug 19, 2024

Regression in header parsing #99

Closed

mattheww95 added 8 commits August 19, 2024 16:14

reverted zipped file handling

b9d1b04

fixed regression locidex select

9abcd03

removed repeated variables

2a4c9ba

updated staging of report input to locidex summarize

0e60b8d

updated file parsing in locidex summarize

dba0190

Thank you Aaron!

updated changelog

5d10389

extracted inline code in function

32327f0

bumped locidex container

814e45e

apetkau requested changes Aug 22, 2024

View reviewed changes

updated final todos

034d9c8

apetkau approved these changes Aug 22, 2024

View reviewed changes

updated options to locidex_select test to terminate processes on failure

de0d30e

mattheww95 merged commit ed9cade into dev Aug 23, 2024
5 checks passed

mattheww95 deleted the update/locidex branch August 23, 2024 17:44

mattheww95 mentioned this pull request Aug 26, 2024

Verify that files passed in as values to exec processes operate as expected on cloud executors. #97

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update/locidex #96

Update/locidex #96

mattheww95 commented Jul 30, 2024

github-actions bot commented Jul 30, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

kylacochrane left a comment

kylacochrane left a comment

apetkau left a comment

apetkau left a comment

apetkau left a comment

Update/locidex #96

Update/locidex #96

Conversation

mattheww95 commented Jul 30, 2024

github-actions bot commented Jul 30, 2024 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

kylacochrane left a comment

Choose a reason for hiding this comment

kylacochrane left a comment

Choose a reason for hiding this comment

apetkau left a comment

Choose a reason for hiding this comment

apetkau left a comment

Choose a reason for hiding this comment

apetkau left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 30, 2024 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️