Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add nvidia libraries for hardware-accelerated en-/decoding #34

Open
psych0d0g opened this issue Apr 7, 2021 · 18 comments
Open

Add nvidia libraries for hardware-accelerated en-/decoding #34

psych0d0g opened this issue Apr 7, 2021 · 18 comments

Comments

@psych0d0g
Copy link

Please add librarys required for HW acceleration to work to the container.

Apr 07, 2021 12:12:08.892 [0x7fd516ffd640] ERROR - [Transcode] [FFMPEG] - Cannot load libnvidia-encode.so.1
Apr 07, 2021 12:12:08.892 [0x7fd516ffd640] ERROR - [Transcode] [FFMPEG] - The minimum required Nvidia driver for nvenc is 418.30 or newer
Apr 07, 2021 12:12:08.934 [0x7fd516ffd640] WARN - [Transcode] Codecs: avcodec_open2 returned -1 for encoder 'h264_nvenc'
Apr 07, 2021 12:12:09.164 [0x7fd516ffd640] ERROR - [Transcode] [FFMPEG] - Cannot load libnvidia-encode.so.1
Apr 07, 2021 12:12:09.164 [0x7fd516ffd640] ERROR - [Transcode] [FFMPEG] - The minimum required Nvidia driver for nvenc is 418.30 or newer
Apr 07, 2021 12:12:09.200 [0x7fd516ffd640] WARN - [Transcode] Codecs: avcodec_open2 returned -1 for encoder 'h264_nvenc'
Apr 07, 2021 12:12:09.990 [0x7fd532515640] ERROR - Error issuing curl_easy_perform(handle): 3
Apr 07, 2021 12:12:09.990 [0x7fd532515640] WARN - HTTP error requesting GET /media/providers (3, Error) ()
Apr 07, 2021 12:12:10.188 [0x7fd532515640] ERROR - Unknown metadata type: 
Apr 07, 2021 12:12:11.403 [0x7fd4ff7fe640] ERROR - [Transcoder] [h264 @ 0x393a700] Cannot load libnvcuvid.so.1
Apr 07, 2021 12:12:11.403 [0x7fd4bbfff640] ERROR - [Transcoder] [h264 @ 0x393a700] Failed loading nvcuvid.
Apr 07, 2021 12:12:11.404 [0x7fd516ffd640] ERROR - [Transcoder] [h264 @ 0x393a700] Failed setup for format cuda: hwaccel initialisation returned error.

@psych0d0g
Copy link
Author

psych0d0g commented Apr 7, 2021

not sure how one would add this properly to the container image,
i was able to get my local plex to transcode using nvenc hardware by copying the following files from the host into the container FS:


libnvcuvid.so
libnvidia-allocator.so.460.67
libnvidia-compiler.so.460.67
libnvidia-encode.so
libnvidia-fbc.so.460.67
libnvidia-ifr.so
libnvidia-ml.so.460.67
libnvidia-opticalflow.so
libnvidia-ptxjitcompiler.so.460.39
libnvcuvid.so.1
libnvidia-cbl.so.460.67
libnvidia-container.so.1
libnvidia-encode.so.1
libnvidia-glcore.so.460.67
libnvidia-ifr.so.1
libnvidia-ngx.so.1
libnvidia-opticalflow.so.1
libnvidia-ptxjitcompiler.so.460.67
libnvcuvid.so.460.39
libnvidia-cfg.so
libnvidia-container.so.1.3.3
libnvidia-encode.so.460.39
libnvidia-glsi.so.460.67
libnvidia-ifr.so.460.67
libnvidia-ngx.so.460.67
libnvidia-opticalflow.so.460.39
libnvidia-rtcore.so.460.67
libnvcuvid.so.460.67
libnvidia-cfg.so.1
libnvidia-egl-wayland.so.1
libnvidia-encode.so.460.67
libnvidia-glvkspirv.so.460.67
libnvidia-ml.so
libnvidia-opencl.so.1
libnvidia-opticalflow.so.460.67
libnvidia-tls.so.460.67
libnvidia-allocator.so
libnvidia-cfg.so.460.67
libnvidia-egl-wayland.so.1.1.5
libnvidia-fbc.so
libnvidia-gtk2.so.460.67
libnvidia-ml.so.1
libnvidia-opencl.so.460.39
libnvidia-ptxjitcompiler.so
libnvoptix.so.1
libnvidia-allocator.so.1
libnvidia-compiler.so.460.39
libnvidia-eglcore.so.460.67
libnvidia-fbc.so.1
libnvidia-gtk3.so.460.67
libnvidia-ml.so.460.39
libnvidia-opencl.so.460.67
libnvidia-ptxjitcompiler.so.1
libnvoptix.so.460.67

@frebib
Copy link
Member

frebib commented Apr 7, 2021

I could create a branch that builds onto the base image and adds in those libraries. I'm reluctant to add them to the master/pass branch images though purely due to size. Can I ask what is the rough size of those library files in total?
Also which tag of this image do you use?
I'll see what I can pull together. For now if you wanted a quick fix, something like this (completely untested btw) should get you by until we manage to work out a more permanent solution:

FROM debian:bullseye
RUN apt-get update -y && apt-get install -y <whatever-nvidia-package-has-those-libs>
FROM spritsail/plex-media-server:<tag>
COPY --from=0 /usr/lib/libnvidia*.so* /usr/lib/nvcu*.so* /usr/lib/nvoptix.so* /usr/lib

@frebib frebib changed the title Required librarys for HW Accell using nvidia missing Add nvidia libraries for hardware-accelerated decoding Apr 7, 2021
@psych0d0g
Copy link
Author

if i can help any further i would be happy to.
Sadly, the libs are friggin humoungous (~500mb), ill try and check if i really need the whole collection of nvidia libs and if i can remove some and still have HW transcoding working properly

@frebib
Copy link
Member

frebib commented Apr 7, 2021

It's likely that most of them aren't required. I see two listed in the logs you provided. They might have a few dependencies which you can see with ldd <path/to/lib.so>. Be sure to post your results here and let me know how it goes.
It's probably a good idea to start with those two, and any immediate dependencies they may have. Any others can be added iteratively as you find they're missing

@psych0d0g
Copy link
Author

libnvoptix.so.460.67 can surely be removed from the list, trims down overall space by 220mb already

@psych0d0g
Copy link
Author

psych0d0g commented Apr 7, 2021

ldd libnvidia-encode.so.1
	linux-vdso.so.1 (0x00007ffd48571000)
	libnvcuvid.so.1 => /usr/lib/libnvcuvid.so.1 (0x00007f0491c94000)
	libc.so.6 => /lib/libc.so.6 (0x00007f0491b0f000)
	libdl.so.2 => /lib/libdl.so.2 (0x00007f0491b0a000)
	libm.so.6 => /lib/libm.so.6 (0x00007f04919e3000)
	/usr/lib/ld-linux-x86-64.so.2 (0x00007f049257c000)
ldd libnvcuvid.so.1
	linux-vdso.so.1 (0x00007ffda6ed9000)
	libm.so.6 => /lib/libm.so.6 (0x00007f13b724e000)
	libc.so.6 => /lib/libc.so.6 (0x00007f13b70c9000)
	libdl.so.2 => /lib/libdl.so.2 (0x00007f13b70c4000)
	/usr/lib/ld-linux-x86-64.so.2 (0x00007f13b7a41000)

so it "seems" that we just need those 2 libs, just checked and plex sucessfully transcodes using HW acceleration with only those 2 files

sizewize:

cronix@mediasrv:/mnt/drives/libs$ ls -lah libnvcuvid.so
-rwxr-xr-x 1 root root 4.8M Apr  7 12:48 libnvcuvid.so
cronix@mediasrv:/mnt/drives/libs$ ls -lah libnvidia-encode.so
-rwxr-xr-x 1 root root 111K Apr  7 12:51 libnvidia-encode.so

@psych0d0g
Copy link
Author

from ubuntu packages:

cronix@mediasrv:/mnt/drives/libs$ dpkg -L libnvidia-encode-460
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libnvidia-encode.so.460.39
/usr/share
/usr/share/doc
/usr/share/doc/libnvidia-encode-460
/usr/share/doc/libnvidia-encode-460/changelog.Debian.gz
/usr/share/doc/libnvidia-encode-460/copyright
/usr/lib/x86_64-linux-gnu/libnvidia-encode.so
/usr/lib/x86_64-linux-gnu/libnvidia-encode.so.1

and the cuvid lib is in the corresponding decode package:

cronix@mediasrv:/mnt/drives/libs$ dpkg -L libnvidia-decode-460
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libnvcuvid.so.460.39
/usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.460.39
/usr/lib/x86_64-linux-gnu/vdpau
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.460.39
/usr/share
/usr/share/doc
/usr/share/doc/libnvidia-decode-460
/usr/share/doc/libnvidia-decode-460/changelog.Debian.gz
/usr/share/doc/libnvidia-decode-460/copyright
/usr/lib/x86_64-linux-gnu/libnvcuvid.so
/usr/lib/x86_64-linux-gnu/libnvcuvid.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so
/usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.1
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.1

@frebib
Copy link
Member

frebib commented Apr 7, 2021

Great investigation! Thanks. I'll put something together later

@psych0d0g
Copy link
Author

psych0d0g commented Apr 7, 2021

thanks so much, great work on all these spiritsail containers btw 👍
do you guys have some kind of chat for exchange?

@psych0d0g psych0d0g changed the title Add nvidia libraries for hardware-accelerated decoding Add nvidia libraries for hardware-accelerated en-/decoding Apr 7, 2021
frebib added a commit that referenced this issue Apr 11, 2021
Pull the nvidia encoding and decoding libraries from Debian and include
them in the image.

Fixes #34

Signed-off-by: Joe Groocock <[email protected]>
frebib added a commit that referenced this issue Apr 11, 2021
Pull the nvidia encoding and decoding libraries from Debian and include
them in the image.

Fixes #34

Signed-off-by: Joe Groocock <[email protected]>
@frebib
Copy link
Member

frebib commented Apr 11, 2021

I've pushed some images to Docker Hub if you want to give them a try. The source for that image is on the nvidia branch.
https://hub.docker.com/r/spritsail/plex-media-server/tags?page=1&ordering=last_updated&name=nvidia

Let me know if it works or not. I don't have Docker set up with nvidia yet. Last I saw it was a bit of a pain to try to get working. I might try setting it up if this image doesn't work for you.

As for chat, I don't think we do really. You can always open a GitHub issue for problems (I'd always prefer these for tracking). You can email us @spritsail.io, the email addresses should be in the maintainer labels for each of the images. I guess you could always hit me up on Matrix @frebib:chat.nerdhouse.io or Telegram @frebib. I'll have a think about the chat thing though, I think it could be a good idea

@frebib
Copy link
Member

frebib commented Apr 27, 2021

Bump @psych0d0g

frebib added a commit that referenced this issue May 5, 2021
Pull the nvidia encoding and decoding libraries from Debian and include
them in the image.

Fixes #34

Signed-off-by: Joe Groocock <[email protected]>
@psych0d0g
Copy link
Author

ill check the build soon, sorry had a bit of private troubles lately that held me back from experimenting

frebib added a commit that referenced this issue May 19, 2021
Pull the nvidia encoding and decoding libraries from Debian and include
them in the image.

Fixes #34

Signed-off-by: Joe Groocock <[email protected]>
@lwndow
Copy link

lwndow commented Jun 23, 2021

I stumbled across this tagged image looking for something similar. Would you be able to bump the pass version with these libs @frebib and I'll be another tester?

frebib added a commit that referenced this issue Jun 23, 2021
Pull the nvidia encoding and decoding libraries from Debian and include
them in the image.

Fixes #34

Signed-off-by: Joe Groocock <[email protected]>
@frebib
Copy link
Member

frebib commented Jun 23, 2021

@lwndow I'm 99.9% sure that this is broken since Plex moved to musl as it's libc, but I've rebased the nvidia branch and re-pushed it but I think CI will likely fail: https://drone.spritsail.io/spritsail/plex-media-server/695

I'm not sure how to make this work with musl as I expect nvidia don't provide anything other than the glibc-based libraries that Debian has. I briefly looked for source/upstream for these a while back but couldn't find it. Presumably this does still work with other images though so it'll require a little more work on my part

Edit: I made it build, but it won't work, at least with the Debian-packaged libraries

/usr/lib # /lib/ld-musl-x86_64.so.1 --list libnvcuvid.so.465.31
	/lib/ld-musl-x86_64.so.1 (0x7fbfaddf4000)
	libm.so.6 => /lib/ld-musl-x86_64.so.1 (0x7fbfaddf4000)
	libc.so.6 => /lib/ld-musl-x86_64.so.1 (0x7fbfaddf4000)
	libdl.so.2 => /lib/ld-musl-x86_64.so.1 (0x7fbfaddf4000)
Error relocating libnvcuvid.so.465.31: dlvsym: symbol not found
/usr/lib # /lib/ld-musl-x86_64.so.1 --list libnvidia-encode.so.465.31
	/lib/ld-musl-x86_64.so.1 (0x7fd21b383000)
Error loading shared library libnvcuvid.so.1: No such file or directory (needed by libnvidia-encode.so.465.31)
	libc.so.6 => /lib/ld-musl-x86_64.so.1 (0x7fd21b383000)
	libdl.so.2 => /lib/ld-musl-x86_64.so.1 (0x7fd21b383000)
Error relocating libnvidia-encode.so.465.31: dlvsym: symbol not found

frebib added a commit that referenced this issue Jun 23, 2021
Pull the nvidia encoding and decoding libraries from Debian and include
them in the image.

Fixes #34

Signed-off-by: Joe Groocock <[email protected]>
frebib added a commit that referenced this issue Jun 23, 2021
Pull the nvidia encoding and decoding libraries from Debian and include
them in the image.

This change is only a test and doesn't build the base Plex image first,
which the CI pipeline should do when this is merged into pass.

Fixes #34

Signed-off-by: Joe Groocock <[email protected]>
frebib added a commit that referenced this issue Jun 23, 2021
Pull the nvidia encoding and decoding libraries from Debian and include
them in the image.

This change is only a test and doesn't build the base Plex image first,
which the CI pipeline should do when this is merged into pass.

Fixes #34

Signed-off-by: Joe Groocock <[email protected]>
@frebib
Copy link
Member

frebib commented Jun 23, 2021

NVIDIA/nvidia-docker#509 (comment)

Time to buy an AMD card I think 😁

@lwndow
Copy link

lwndow commented Jun 23, 2021

NVIDIA/nvidia-docker#509 (comment)

Time to buy an AMD card I think 😁

DEFINITELY

@Drikani
Copy link

Drikani commented Aug 29, 2022

Is it at all possible to use a nvidia card with this container or do I need to go for the official pms-docker or linuxserver image?
I use it on UnRAID and plan on getting a used GTX 1060 for transcoding with tdarr as well and everywhere I look it only tells me to add a few variables and runtime options and the rest will be done by the plugin in UnRAID?
Does this Issue here only apply to those who do not use the nvidia-docker container runtime?

@frebib
Copy link
Member

frebib commented Apr 10, 2023

There's not much we can do here until nvidia either provide the source for their libraries (never going to happen) or they provide binaries compiled against musl instead of glibc. We could also bundle glibc into our image with these libraries which I think should work, but that's a bit of work to do. I'll accept a PR if anyone wants to give it a go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants