TMPDIR not set for bismark

[peterch405]

Description of the bug

The Bismark alignment step writes temp files to the local node storage instead of the TMPDIR location. On nodes with small local storage, jobs fail with:

Running Bismark Parallel version. Number of parallel instances to be spawned: 4

Current working directory is: /work/4578865.1.eeskin_pod_36.q/nxf.iLMc9V4b52

...

Created G -> A converted version of the FastQ file H9_naive_5iLA_hTSC_DNAme_rep1_2_val_2.fq.gz.temp.1 (28442995 sequences in total)

Failed to close filehandle No space left on device

Created G -> A converted version of the FastQ file H9_naive_5iLA_hTSC_DNAme_rep1_2_val_2.fq.gz.temp.2 (28442994 sequences in total)

Failed to close filehandle No space left on device

Created G -> A converted version of the FastQ file H9_naive_5iLA_hTSC_DNAme_rep1_2_val_2.fq.gz.temp.3 (28442994 sequences in total)

Failed to close filehandle No space left on device

Created G -> A converted version of the FastQ file H9_naive_5iLA_hTSC_DNAme_rep1_2_val_2.fq.gz.temp.4 (28442994 sequences in total)

Failed to close filehandle No space left on device

Tested with both version 2.6.0 and 1.6.1.

If a node with sufficient local storage is used, all jobs complete successfully, but currently I am not aware of a way to assign to a subset of nodes.

Command used and terminal output

Run command:
nextflow run nf-core/methylseq -r 2.6.0 -work-dir /u/project/kp1/ajcollie/DNAme/PellegriniMila7160_bismark/ -params-file /u/project/kp1/ajcollie/DNAme/PellegriniMila7160_bismark/nf-params.json -resume -c /u/project/kp1/ajcollie/pipelines/nf-core/nf-core-methylseq-1.6.1/configs/conf/hoffman2.config

Relevant files

hoffman2.config:

//Profile config names for nf-core/configs
params {
  config_profile_description = 'University of Los Angeles Hoffman2 cluster profile provided by nf-core/configs.'
  config_profile_contact = 'Peter Chovanec (@peterch405)'
  config_profile_url = 'https://www.hoffman2.idre.ucla.edu/'
}

executor {
  name = "sge"
  queueSize = "50"
}


process {
  clusterOptions = { task.memory ? "-l h_vmem=${task.memory.bytes/task.cpus},highp" : null }
  scratch = true
  penv = { task.cpus > 1 ? "sharedmem" : null }

  // common SGE error statuses
  errorStrategy = {task.exitStatus in [141,143,137,104,134,139,140] ? 'retry' : 'finish'}
  maxErrors = '-1'
  maxRetries = 3

  beforeScript =
  """
  . /u/local/Modules/default/init/modules.sh
  module load apptainer
  export APPTAINER_CACHEDIR="/u/scratch/a/ajcollie"
  export NXF_APPTAINER_CACHEDIR="/u/scratch/a/ajcollie"
  export APPTAINER_TMPDIR="/u/scratch/a/ajcollie"
  export TMPDIR="/u/scratch/a/ajcollie"
  export NXF_TEMP="/u/scratch/a/ajcollie"
  """

}

env {
  TMPDIR="/u/scratch/a/ajcollie"
}

singularity {
  envWhitelist = "TMPDIR"
  runOptions = '-p -B $TMPDIR -B $TMPDIR:/work'
  enabled = true
  autoMounts = true
}

System information

Conda env:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
alsa-lib                  1.2.12               h4ab18f5_0    conda-forge
annotated-types           0.7.0              pyhd8ed1ab_0    conda-forge
attmap                    0.13.2             pyhd8ed1ab_0    conda-forge
attrs                     24.2.0             pyh71513ae_0    conda-forge
biopython                 1.84            py312h9a8786e_0    conda-forge
bleach                    6.1.0              pyhd8ed1ab_0    conda-forge
boltons                   24.0.0             pyhd8ed1ab_0    conda-forge
brotli-python             1.1.0           py312h30efb56_1    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
c-ares                    1.32.3               h4bc722e_0    conda-forge
ca-certificates           2024.7.4             hbcca054_0    conda-forge
cairo                     1.18.0               hbb29018_2    conda-forge
cattrs                    23.2.3             pyhd8ed1ab_0    conda-forge
certifi                   2024.7.4           pyhd8ed1ab_0    conda-forge
cffi                      1.17.0          py312h1671c18_0    conda-forge
cfgv                      3.3.1              pyhd8ed1ab_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
cheetah3                  3.2.6.post2     py312h98912ed_4    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
conda-package-streaming   0.10.0             pyhd8ed1ab_0    conda-forge
coreutils                 9.5                  hd590300_0    conda-forge
curl                      8.9.1                h18eb788_0    conda-forge
distlib                   0.3.8              pyhd8ed1ab_0    conda-forge
docutils                  0.21.2             pyhd8ed1ab_0    conda-forge
eido                      0.2.2              pyhd8ed1ab_0    conda-forge
exceptiongroup            1.2.2              pyhd8ed1ab_0    conda-forge
expat                     2.6.2                h59595ed_0    conda-forge
filelock                  3.15.4             pyhd8ed1ab_0    conda-forge
filetype                  1.2.0              pyhd8ed1ab_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 h77eed37_2    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
future                    1.0.0              pyhd8ed1ab_0    conda-forge
galaxy-tool-util          24.1.1             pyhd8ed1ab_0    conda-forge
galaxy-util               24.1.1             pyhd8ed1ab_0    conda-forge
giflib                    5.2.2                hd590300_0    conda-forge
git                       2.46.0          pl5321hb5640b7_0    conda-forge
gitdb                     4.0.11             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.43             pyhd8ed1ab_0    conda-forge
graphite2                 1.3.13            h59595ed_1003    conda-forge
h2                        4.1.0              pyhd8ed1ab_0    conda-forge
harfbuzz                  8.5.0                hfac3d4d_0    conda-forge
hpack                     4.0.0              pyh9f0ad1d_0    conda-forge
hyperframe                6.0.1              pyhd8ed1ab_0    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
identify                  2.6.0              pyhd8ed1ab_0    conda-forge
idna                      3.7                pyhd8ed1ab_0    conda-forge
importlib-metadata        8.2.0              pyha770c72_0    conda-forge
importlib_resources       6.4.0              pyhd8ed1ab_0    conda-forge
iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
isa-l                     2.31.0               h4bc722e_2    conda-forge
itsdangerous              2.2.0              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.4              pyhd8ed1ab_0    conda-forge
jsonschema                4.23.0             pyhd8ed1ab_0    conda-forge
jsonschema-specifications 2023.12.1          pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.3               h659f571_0    conda-forge
lcms2                     2.16                 hb7c19ff_0    conda-forge
ld_impl_linux-64          2.40                 hf3520f5_7    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libblas                   3.9.0           23_linux64_openblas    conda-forge
libcblas                  3.9.0           23_linux64_openblas    conda-forge
libcups                   2.3.3                h4637d8d_4    conda-forge
libcurl                   8.9.1                hdb1bdb2_0    conda-forge
libdeflate                1.21                 h4bc722e_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libexpat                  2.6.2                h59595ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 14.1.0               h77fa898_0    conda-forge
libgfortran-ng            14.1.0               h69a702a_0    conda-forge
libgfortran5              14.1.0               hc5f4f2c_0    conda-forge
libglib                   2.80.3               h315aac3_2    conda-forge
libgomp                   14.1.0               h77fa898_0    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
liblapack                 3.9.0           23_linux64_openblas    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libopenblas               0.3.27          pthreads_hac2b453_1    conda-forge
libpng                    1.6.43               h2797004_0    conda-forge
libsqlite                 3.46.0               hde9e2c9_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              14.1.0               hc0a3c3a_0    conda-forge
libtiff                   4.6.0                h46a8edc_4    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.4.0                hd590300_0    conda-forge
libxcb                    1.16                 hd590300_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.12.7               h4c95cb1_3    conda-forge
libxslt                   1.1.39               h76b75d6_0    conda-forge
libzlib                   1.3.1                h4ab18f5_1    conda-forge
logmuse                   0.2.6              pyh8c360ce_0    conda-forge
lxml                      5.3.0           py312hf3a79e2_0    conda-forge
markdown                  3.6                pyhd8ed1ab_0    conda-forge
markdown-it-py            3.0.0              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.5           py312h98912ed_0    conda-forge
mdurl                     0.1.2              pyhd8ed1ab_0    conda-forge
ncurses                   6.5                  h59595ed_0    conda-forge
nextflow                  24.04.4              hdfd78af_0    bioconda
nf-core                   2.14.1             pyhdfd78af_1    bioconda
nf-test                   0.9.0                h2a3209d_0    bioconda
nodeenv                   1.9.1              pyhd8ed1ab_0    conda-forge
numpy                     2.0.1           py312h1103770_0    conda-forge
openjdk                   17.0.10              h4260e57_0    conda-forge
openjpeg                  2.5.2                h488ebb8_0    conda-forge
openssl                   3.3.1                h4bc722e_2    conda-forge
oyaml                     1.0                pyhd8ed1ab_0    conda-forge
packaging                 24.1               pyhd8ed1ab_0    conda-forge
pandas                    2.2.2           py312h1d6d2e6_1    conda-forge
pbzip2                    1.1.13               h1fcc475_2    conda-forge
pcre2                     10.44                hba22ea6_2    conda-forge
peppy                     0.40.5             pyhd8ed1ab_0    conda-forge
perl                      5.32.1          7_hd590300_perl5    conda-forge
pigz                      2.8                  h2797004_0    conda-forge
pillow                    10.4.0          py312h287a98d_0    conda-forge
pip                       24.2               pyhd8ed1ab_0    conda-forge
piper                     0.14.2             pyhdfd78af_0    bioconda
pipestat                  0.10.1             pyhd8ed1ab_0    conda-forge
pixman                    0.43.2               h59595ed_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
platformdirs              4.2.2              pyhd8ed1ab_0    conda-forge
pluggy                    1.5.0              pyhd8ed1ab_0    conda-forge
pre-commit                3.8.0              pyha770c72_0    conda-forge
prompt-toolkit            3.0.47             pyha770c72_0    conda-forge
prompt_toolkit            3.0.47               hd8ed1ab_0    conda-forge
psutil                    6.0.0           py312h9a8786e_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pycparser                 2.22               pyhd8ed1ab_0    conda-forge
pydantic                  2.8.2              pyhd8ed1ab_0    conda-forge
pydantic-core             2.20.1          py312hf008fa9_0    conda-forge
pyfaidx                   0.8.1.2            pyhdfd78af_0    bioconda
pygments                  2.18.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.1.2              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
pytest                    8.3.2              pyhd8ed1ab_0    conda-forge
pytest-workflow           2.1.0              pyhd8ed1ab_0    conda-forge
python                    3.12.5          h2ad013b_0_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python-isal               1.7.0           py312h41a817b_0    conda-forge
python-tzdata             2024.1             pyhd8ed1ab_0    conda-forge
python-zlib-ng            0.5.0           py312hbb25f6d_0    conda-forge
python_abi                3.12                    4_cp312    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
pyvcf3                    1.0.3              pyhdfd78af_0    bioconda
pyyaml                    6.0.2           py312h41a817b_0    conda-forge
questionary               2.0.1              pyhd8ed1ab_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
referencing               0.35.1             pyhd8ed1ab_0    conda-forge
refgenconf                0.12.2             pyhdfd78af_0    bioconda
refgenie                  0.12.1             pyhdfd78af_0    bioconda
repoze.lru                0.7                pyhd8ed1ab_1    conda-forge
requests                  2.32.3             pyhd8ed1ab_0    conda-forge
requests-cache            1.2.1              pyhd8ed1ab_0    conda-forge
rich                      13.7.1             pyhd8ed1ab_0    conda-forge
rich-click                1.8.3              pyhd8ed1ab_0    conda-forge
routes                    2.5.1              pyhd8ed1ab_0    conda-forge
rpds-py                   0.20.0          py312hf008fa9_0    conda-forge
setuptools                72.1.0             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     5.0.0              pyhd8ed1ab_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
textual                   0.76.0             pyhd8ed1ab_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
trogon                    0.5.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.12.2               hd8ed1ab_0    conda-forge
typing_extensions         4.12.2             pyha770c72_0    conda-forge
tzdata                    2024a                h0c530f3_0    conda-forge
ubiquerg                  0.8.0              pyhd8ed1ab_0    conda-forge
ujson                     5.10.0          py312h7070661_0    conda-forge
ukkonen                   1.0.1           py312h8572e83_4    conda-forge
url-normalize             1.4.3              pyhd8ed1ab_0    conda-forge
urllib3                   2.2.2              pyhd8ed1ab_1    conda-forge
veracitools               0.1.3                      py_0    conda-forge
virtualenv                20.26.3            pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.13             pyhd8ed1ab_0    conda-forge
webencodings              0.5.1              pyhd8ed1ab_2    conda-forge
wheel                     0.44.0             pyhd8ed1ab_0    conda-forge
whoosh                    2.7.4           py312h7900ff3_8    conda-forge
xopen                     2.0.2              pyh707e725_1    conda-forge
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-inputproto           2.3.2             h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.9                hb711507_1    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-libxi                1.7.10               h4bc722e_1    conda-forge
xorg-libxrender           0.9.11               hd590300_0    conda-forge
xorg-libxt                1.3.0                hd590300_1    conda-forge
xorg-libxtst              1.2.5                h4bc722e_0    conda-forge
xorg-recordproto          1.14.2            h7f98852_1002    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yacman                    0.9.3              pyhd8ed1ab_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zipp                      3.20.0             pyhd8ed1ab_0    conda-forge
zipstream-new             1.1.8              pyhd8ed1ab_0    conda-forge
zlib                      1.3.1                h4ab18f5_1    conda-forge
zlib-ng                   2.2.1                he02047a_0    conda-forge
zstandard                 0.23.0          py312h3483029_0    conda-forge
zstd                      1.5.6                ha6fb4c9_0    conda-forge