Output and Error information pruned during Singularity exec

[atombaby]

It looks like the stderr (and likely stdout) is getting pruned during execution when Singularity is used to run the script:

VERBOSE [U=0,P=3126]       print()                       Set messagelevel to: 5
VERBOSE [U=0,P=3126]       init()                        Starter initialization
DEBUG   [U=0,P=3126]       load_overlay_module()         Trying to load overlay kernel module
DEBUG   [U=0,P=3126]       load_overlay_module()         Overlay seems not supported by the kernel
DEBUG   [U=0,P=3126]       get_pipe_exec_fd()            PIPE_EXEC_FD value: 9
VERBOSE [U=0,P=3126]       is_suid()                     Check if we are running as setuid
VERBOSE [U=0,P=3126]       priv_drop()                   Drop root privileges
DEBUG   [U=34152,P=3126]   init()                        Read engine configuration
tail: shard-1/execution/stderr: file truncated
DEBUG   [U=34152,P=3126]   Master()                      Child exited with exit status 0

Note the "file truncated" message. The lines prior to this is output from singularity -d exec. The resultant stderr file in the execution directory doesn't contain any of the lines above the "file truncated" message.

I believe this happens when the execution script redirects stdout and stderr. The submit script uses Slurm options to save stdout/stderr to files in the execution directory. From a representative output script (script.submit):

   -o /fh/scratch/delete10/_HDC/user/mrg/cromwell-root/cromwell-executions/parseBatchFile/b9e0a739-23e3-485d-8bab-857349697458/call-test/shard-1/execution/stdout \
-e /fh/scratch/delete10/_HDC/user/mrg/cromwell-root/cromwell-executions/parseBatchFile/b9e0a739-23e3-485d-8bab-857349697458/call-test/shard-1/execution/stderr \

When the script is run (script) the following lines are truncating any output occurring between the startup of the job on the node and the execution of the script in the container (particularly Singularity startup messages):

tee '/cromwell-executions/parseBatchFile/b9e0a739-23e3-485d-8bab-857349697458/call-test/shard-1/execution/stdout' < "$outb9e0a739" &
tee '/cromwell-executions/parseBatchFile/b9e0a739-23e3-485d-8bab-857349697458/call-test/shard-1/execution/stderr' < "$errb9e0a739" >&2 &

Note that in the container (where script is run) the path /fh/scratch/delete10/_HDC/user/mrg/cromwell-root/cromwell-executions has been mounted on /cromwell-executions). tee (without other options) will truncate output prior to writing to the file. This removes any output between job start and script execution.

I suspect that adding -a to the tee command will fix this problem.

[vortexing]

Can you at some point check, given the new config in fullConfig if anything about this has changed?