HELP: call-xcor fails with TEST command

[yinyeya]

Describe the bug

When running the newest pipeline (v2.2.0) with TEST command caper run chip.wdl -i https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI_subsampled_chr19_only.json --conda, it returns errors due to could not find function "startsWith" at xcor steps.

I noticed that many issues related to xcor reported can be fixed by re-installing and -installing. But I tried several times, it still can not fix this issue.
In addition, when running the newest encode-atac-seq-pipeline with TEST command, I obtained the same error.
Note python=3.8.13, java=17.0.3.1 and pip=22.1.2 are installed locally, and caper is installed based these.

OS/Platform

• OS/Platform: Red Hat Enterprise Linux Server 7.7 (Maipo)
• Conda version: v4.12.0
• Pipeline version: v2.2.0
• Caper version: v2.2.0

Caper configuration file

Paste contents of ~/.caper/default.conf.

backend=local

# Local directory for localized files and Cromwell's intermediate files.
# If not defined then Caper will make .caper_tmp/ on CWD or `local-out-dir`.
# /tmp is not recommended since Caper store localized data files here.
local-loc-dir=

cromwell=/storage/taowu/home/u237039/.caper/cromwell_jar/cromwell-65.jar
womtool=/storage/taowu/home/u237039/.caper/womtool_jar/womtool-65.jar

Input JSON file

Paste contents of your input JSON file.

{
    "chip.pipeline_type" : "tf",
    "chip.genome_tsv" : "https://storage.googleapis.com/encode-pipeline-genome-data/genome_tsv/v3/hg38_chr19_chrM.tsv",
    "chip.fastqs_rep1_R1" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep1.subsampled.25.fastq.gz"
    ],
    "chip.fastqs_rep2_R1" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep2.subsampled.20.fastq.gz"
    ],
    "chip.ctl_fastqs_rep1_R1" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl1.subsampled.25.fastq.gz"
    ],
    "chip.ctl_fastqs_rep2_R1" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl2.subsampled.25.fastq.gz"
    ],
    "chip.paired_end" : false,
    "chip.title" : "ENCSR000DYI (subsampled 1/25, chr19_chrM only)",
    "chip.description" : "CEBPB ChIP-seq on human A549 produced by the Snyder lab"
}

Troubleshooting result

If you ran caper run without Caper server then Caper automatically runs a troubleshooter for failed workflows. Find troubleshooting result in the bottom of Caper's screen log.

If you ran caper submit with a running Caper server then first find your workflow ID (1st column) with caper list and run caper debug [WORKFLOW_ID].

Paste troubleshooting result.

* Recursively finding failures in calls (tasks)...

==== NAME=atac.xcor, STATUS=RetryableFailure, PARENT=
SHARD_IDX=0, RC=1, JOB_ID=242459
START=2022-06-20T23:33:11.689Z, END=2022-06-20T23:33:27.424Z
STDOUT=/storage/taowu/home/u237039/test/atac/atac/f4f1ee8d-81d6-47af-a1fb-3121489010cd/call-xcor/shard-0/execution/stdout
STDERR=/storage/taowu/home/u237039/test/atac/atac/f4f1ee8d-81d6-47af-a1fb-3121489010cd/call-xcor/shard-0/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/storage/taowu/home/u237039/install/miniconda3/envs/encd-atac-spp/bin/encode_task_xcor.py", line 156, in <module>
    main()
  File "/storage/taowu/home/u237039/install/miniconda3/envs/encd-atac-spp/bin/encode_task_xcor.py", line 144, in main
    args.chip_seq_type, args.exclusion_range_min, args.exclusion_range_max)
  File "/storage/taowu/home/u237039/install/miniconda3/envs/encd-atac-spp/bin/encode_task_xcor.py", line 105, in xcor
    run_shell_cmd(cmd1)
  File "/storage/taowu/home/u237039/install/miniconda3/envs/encd-atac-spp/bin/encode_lib_common.py", line 359, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=242723, PGID=242723, RC=1, DURATION_SEC=0.3
STDERR=Error in eval(expr, envir, enclos) : could not find function "startsWith"
Calls: local -> eval.parent -> eval -> eval -> eval -> eval
Execution halted
STDOUT=

STDERR_BACKGROUND_CONTENTS=
Traceback (most recent call last):
  File "/storage/taowu/home/u237039/install/miniconda3/envs/encd-atac-spp/bin/encode_task_xcor.py", line 156, in <module>
    main()
  File "/storage/taowu/home/u237039/install/miniconda3/envs/encd-atac-spp/bin/encode_task_xcor.py", line 144, in main
    args.chip_seq_type, args.exclusion_range_min, args.exclusion_range_max)
  File "/storage/taowu/home/u237039/install/miniconda3/envs/encd-atac-spp/bin/encode_task_xcor.py", line 105, in xcor
    run_shell_cmd(cmd1)
  File "/storage/taowu/home/u237039/install/miniconda3/envs/encd-atac-spp/bin/encode_lib_common.py", line 359, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=242723, PGID=242723, RC=1, DURATION_SEC=0.3
STDERR=Error in eval(expr, envir, enclos) : could not find function "startsWith"
Calls: local -> eval.parent -> eval -> eval -> eval -> eval
Execution halted
STDOUT=
ln: failed to access ‘/storage/taowu/home/u237039/test/atac/atac/f4f1ee8d-81d6-47af-a1fb-3121489010cd/call-xcor/shard-0/execution/*.cc.plot.png’: No such file or directory
ln: failed to access ‘/storage/taowu/home/u237039/test/atac/atac/f4f1ee8d-81d6-47af-a1fb-3121489010cd/call-xcor/shard-0/execution/*.cc.plot.pdf’: No such file or directory
ln: failed to access ‘/storage/taowu/home/u237039/test/atac/atac/f4f1ee8d-81d6-47af-a1fb-3121489010cd/call-xcor/shard-0/execution/*.cc.qc’: No such file or directory



==== NAME=atac.xcor, STATUS=Failed, PARENT=
SHARD_IDX=0, RC=1, JOB_ID=243126
START=2022-06-20T23:33:29.799Z, END=2022-06-20T23:33:39.861Z
STDOUT=/storage/taowu/home/u237039/test/atac/atac/f4f1ee8d-81d6-47af-a1fb-3121489010cd/call-xcor/shard-0/attempt-2/execution/stdout
STDERR=/storage/taowu/home/u237039/test/atac/atac/f4f1ee8d-81d6-47af-a1fb-3121489010cd/call-xcor/shard-0/attempt-2/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/storage/taowu/home/u237039/install/miniconda3/envs/encd-atac-spp/bin/encode_task_xcor.py", line 156, in <module>
    main()
  File "/storage/taowu/home/u237039/install/miniconda3/envs/encd-atac-spp/bin/encode_task_xcor.py", line 144, in main
    args.chip_seq_type, args.exclusion_range_min, args.exclusion_range_max)
  File "/storage/taowu/home/u237039/install/miniconda3/envs/encd-atac-spp/bin/encode_task_xcor.py", line 105, in xcor
    run_shell_cmd(cmd1)
  File "/storage/taowu/home/u237039/install/miniconda3/envs/encd-atac-spp/bin/encode_lib_common.py", line 359, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=243307, PGID=243307, RC=1, DURATION_SEC=0.1
STDERR=Error in eval(expr, envir, enclos) : could not find function "startsWith"
Calls: local -> eval.parent -> eval -> eval -> eval -> eval
Execution halted
STDOUT=

STDERR_BACKGROUND_CONTENTS=
Traceback (most recent call last):
  File "/storage/taowu/home/u237039/install/miniconda3/envs/encd-atac-spp/bin/encode_task_xcor.py", line 156, in <module>
    main()
  File "/storage/taowu/home/u237039/install/miniconda3/envs/encd-atac-spp/bin/encode_task_xcor.py", line 144, in main
    args.chip_seq_type, args.exclusion_range_min, args.exclusion_range_max)
  File "/storage/taowu/home/u237039/install/miniconda3/envs/encd-atac-spp/bin/encode_task_xcor.py", line 105, in xcor
    run_shell_cmd(cmd1)
  File "/storage/taowu/home/u237039/install/miniconda3/envs/encd-atac-spp/bin/encode_lib_common.py", line 359, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=243307, PGID=243307, RC=1, DURATION_SEC=0.1
STDERR=Error in eval(expr, envir, enclos) : could not find function "startsWith"
Calls: local -> eval.parent -> eval -> eval -> eval -> eval
Execution halted
STDOUT=
ln: failed to access ‘/storage/taowu/home/u237039/test/atac/atac/f4f1ee8d-81d6-47af-a1fb-3121489010cd/call-xcor/shard-0/attempt-2/execution/*.cc.plot.png’: No such file or directory
ln: failed to access ‘/storage/taowu/home/u237039/test/atac/atac/f4f1ee8d-81d6-47af-a1fb-3121489010cd/call-xcor/shard-0/attempt-2/execution/*.cc.plot.pdf’: No such file or directory
ln: failed to access ‘/storage/taowu/home/u237039/test/atac/atac/f4f1ee8d-81d6-47af-a1fb-3121489010cd/call-xcor/shard-0/attempt-2/execution/*.cc.qc’: No such file or directory

[yinyeya]

I just realized that my installed R/R-base in encd-chip-spp is v3.2.2, and the startsWith function is added at R 3.3. This should be the reason for errors in call-xcor. But I would like to further ask if there are any solutions to upgrade R/R-base in this pipeline under the system I indicated. I've tried to arbitrarily update R version in encd-chip-spp environment, but it reports lots of conflicts.

[leepc12]

It works fine with r-base==3.2.2 on my end.
https://github.com/kundajelab/phantompeakqualtools/search?q=startsWith
https://github.com/hms-dbmi/spp/search?q=startsWith
Two relevant packages don't even use startsWith function.
I guess that your Conda environment is messed up?

Please check r-spp version in the environment.

(encd-chip-spp) leepc12@ubuntu:~/code/chip-seq-pipeline2/scripts$ conda list | grep r-spp
r-spp                     1.13                   r3.2.2_0    bioconda

Please try with Singularity (--singularity) or Docker (--docker) or if Conda method doesn't work.

[yinyeya]

@leepc12, thanks for your valuable suggestions.
I checked the r-spp version, and it is the same as yours. I also tried to completely remove and reinstall my conda and related envs, but it still returns the same error.
Now I just figured out that it even returns can not find "startsWith" function error when I run R under encd-chip-spp or even a separate conda envs only including R<3.3.1. It seems that the issues are caused by the server environment.
Yes, I am planning to switch to use singularity or --docker to check.