Replace EFS with S3

[munishchouhan]

This PR will Replace usgae of file system with S3

[munishchouhan]

getting this error, while running new buildkit image on k8s

12:08PM WRN Error running in a new user namespace - fork/exec /usr/bin/fusion: invalid argument

12:08PM WRN cannot apply Nextflow profile :: unknown remote store for '' work prefix
/usr/bin/fusermount3: fuse device not found, try 'modprobe fuse' first
12:08PM FTL mount.go:251 > mounting filesystem error="fusermount exited with code 256\n"

[pditommaso]

@fntlnz is the guru

[fntlnz]

@munishchouhan something is off accessing the /dev/fuse device. How are you running this?

[fntlnz]

Found, you need to add

--device /dev/fuse

wave/src/main/groovy/io/seqera/wave/service/builder/DockerBuildStrategy.groovy

Line 105 in 7751c86

'--privileged',

[fntlnz]

If this needs to be done in Kubernetes there are two possible solutions:

• Start the container privileged and mount /dev/fuse inside of it
• Use a device plugin, the one Nextflow uses is https://github.com/nextflow-io/k8s-fuse-plugin -> Add support for custom fuse device plugin nextflow-io/nextflow#4612

[pditommaso]

Better using the fuse plugin

[munishchouhan]

I have started the Daemonset

(base) munish.chouhan@Munishs-MacBook-Pro ~ % kubectl get DaemonSets -n kube-system
NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
fuse-device-plugin-daemonset   1         1         1       1            1           <none>          18m

Add added limit as per https://github.com/nextflow-io/k8s-fuse-plugin/blob/master/README.md#usage
in here

wave/src/main/groovy/io/seqera/wave/service/k8s/K8sServiceImpl.groovy

Line 283 in 88738ee

requests.limits(Map.of("nextflow.io/fuse", new Quantity("1")))

But still getting the same error

2:12PM WRN Error running in a new user namespace - fork/exec /usr/bin/fusion: invalid argument

2:12PM WRN cannot apply Nextflow profile :: unknown remote store for '' work prefix
/usr/bin/fusermount3: mount failed: Operation not permitted
2:12PM FTL mount.go:251 > mounting filesystem error="fusermount exited with code 256\n"

cc @jordeu

[jordeu]

From the invalid argument message I suspect that it's more related to an invalid Fusion command invocation. What is the full command line that it's executed?

[jordeu]

or, can you share an example image that will be executed?

[munishchouhan]

or, can you share an example image that will be executed?

I am using this example to test using wave-cli
wave --conda-package bwa --wave-endpoint http://localhost:9090

[munishchouhan]

From the invalid argument message I suspect that it's more related to an invalid Fusion command invocation. What is the full command line that it's executed?

same command is working with docker, but i will debug further to check this

[jordeu]

From the invalid argument message I suspect that it's more related to an invalid Fusion command invocation. What is the full command line that it's executed?

same command is working with docker, but i will debug further to check this

But with docker, there is no need to clone the namespace, so Fusion behaves differently. If you can get the full command line that is executed I can check if there is something strange.

[munishchouhan]

From the invalid argument message I suspect that it's more related to an invalid Fusion command invocation. What is the full command line that it's executed?

same command is working with docker, but i will debug further to check this

But with docker, there is no need to clone the namespace, so Fusion behaves differently. If you can get the full command line that is executed I can check if there is something strange.

do you want command line for docker or k8s?

[jordeu]

do you want command line for docker or k8s?

It's better if it's the k8s, but they should be the same, so if it's easier for you, send me the docker one.

[jordeu]

Also, the environment variables are important. Send me the docker inspect and we can assume that it's the same on k8s.

[munishchouhan]

It's better if it's the k8s, but they should be the same, so if it's easier for you, send me the docker one.

here is the docker command, I will work on getting you the k8s info

docker run --rm --privileged \
-e AWS_ACCESS_KEY_ID=<AWS_ACCESS_KEY_ID> \
-e AWS_SECRET_ACCESS_KEY=<AWS_SECRET_ACCESS_KEY> \
-e DOCKER_CONFIG=/fusion/s3/s3-bucket/workspace/9a3c69098bd07c17_1 \
--platform linux/amd64 cr.seqera.io/public/wave/buildkit:ef67f15426f36b72 \
buildctl-daemonless.sh build \
--frontend dockerfile.v0 \
--local dockerfile=/fusion/s3/s3-bucket/workspace/9a3c69098bd07c17_1 \
--opt filename=Containerfile \
--local context=/fusion/s3/s3-bucket/workspace/9a3c69098bd07c17_1/context \
--output type=image,name=docker.io/hrma017/dev:bwa--9a3c69098bd07c17,push=true,oci-mediatypes=true \
--opt platform=linux/amd64 \
--export-cache type=registry,image-manifest=true,ref=docker.io/hrma017/cache:9a3c69098bd07c17,mode=max,ignore-error=true,oci-mediatypes=true,compression=gzip,force-compression=false \
--import-cache type=registry,ref=docker.io/hrma017/cache:9a3c69098bd07c17

[munishchouhan]

Also, the environment variables are important. Send me the docker inspect and we can assume that it's the same on k8s.

here is the inspect in file
docker_inspect.txt

[jordeu]

What is this inspect file? In the inspect I see a command different than the one in the docker command line:

"Cmd": [
                "trivy",
                "--quiet",
                "image",
                "--timeout",
                "10m",
                "--format",
                "json",
                "--output",
                "/fusion/s3/s3-bucket/workspace/scan-9f342c61b284/report.json",
                "docker.io/hrma017/dev:bwa--9a3c69098bd07c17"
            ],

[munishchouhan]

@jordeu apologies, i shared the scan inspect, here is for the build
docker_build_inspect.json

[munishchouhan]

hi @jordeu
Any advice or pointers to move forward with k8s-fuse-plugin?