Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cuda-modules: fixed output derivations and new modules #306172

Draft
wants to merge 71 commits into
base: master
Choose a base branch
from

Conversation

ConnorBaker
Copy link
Contributor

@ConnorBaker ConnorBaker commented Apr 23, 2024

Description of changes

TODO

  • CUDA 12.5's run file installer, GDB, and NVCC need revisions to their overrides because they no longer build due to missing libraries or invalid patches.
  • The deeply nested JSON is meant mostly to minimize the amount of JSON being vendored in Nixpkgs. It does complicate things on my end though -- what would the the design look like without trying to minimize it?
  • Clean (and simplify?) the modules
  • Fix build errors in-tree
  • Test eval on aarch64-linux and *-darwin
  • Documentation
  • Check whether cudaPackages.setupCudaHook: <crt/host_defines.h> not visible to the host compiler #279872 is resolved with the changes to setupCudaHook

Files with references to CUDA that I need to fix up or create separate issues for:

Docs

  • ./doc/languages-frameworks/cuda.section.md
  • Everything in ./pkgs/development/cuda-modules

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@ConnorBaker ConnorBaker changed the title Feat/cuda packages fixed output derivations cuda-modules: fixed output derivations and new modules Apr 24, 2024
@ConnorBaker ConnorBaker force-pushed the feat/cudaPackages-fixed-output-derivations branch from c47c18a to c25d607 Compare April 30, 2024 01:36
@ConnorBaker
Copy link
Contributor Author

Starting to firm up!

$ nix run -L --override-input nixpkgs "path:///home/connorbaker/Packages/nixpkgs" --show-trace .#nix-cuda-test
warning: not writing modified lock file of flake 'git+file:///home/connorbaker/Packages/nix-cuda-test':
• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/3b7dfff0e6eea9e17167a3d1870642da4f416c5f?narHash=sha256-NSMHm0DmnH6Ax6QDLdhKNq7PHK28f/8VfjylJlDg2g4%3D' (2024-04-30)
  → 'path:/home/connorbaker/Packages/nixpkgs?lastModified=1714527095&narHash=sha256-ycXJuyvZhbdd7J5TZV7Y%2B0h6RJbVjokusdKAPs7D9Nk%3D' (2024-05-01)
trace: warning: cudaPackages.autoAddDriverRunpath is deprecated, use pkgs.autoAddDriverRunpath instead
trace: warning: cudaPackages.autoAddDriverRunpath is deprecated, use pkgs.autoAddDriverRunpath instead
trace: warning: cudaPackages.cudaFlags is deprecated, use cudaPackages.flags instead
Seed set to 42
Using bfloat16 Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Files already downloaded and verified
Files already downloaded and verified
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

  | Name      | Type             | Params
-----------------------------------------------
0 | criterion | CrossEntropyLoss | 0     
1 | model     | ViT              | 86.3 M
-----------------------------------------------
86.3 M    Trainable params
0         Non-trainable params
86.3 M    Total params
345.317   Total estimated model params size (MB)
Epoch 9: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 781/781 [01:29<00:00,  8.70it/s, v_num=10, train_loss=2.340, val_loss=2.320]`Trainer.fit` stopped: `max_epochs=10` reached.                                                                                                                                                                     
Epoch 9: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 781/781 [01:31<00:00,  8.49it/s, v_num=10, train_loss=2.340, val_loss=2.320]

@ConnorBaker ConnorBaker force-pushed the feat/cudaPackages-fixed-output-derivations branch from 8cfbadc to ce3a887 Compare May 1, 2024 02:37
@ConnorBaker
Copy link
Contributor Author

@SomeoneSerge here's a sample of some of the errors I'm seeing as of commit 97073a6. Mostly seem to be failures to detect the libraries or headers in downstream projects.

/nix/store/b3vddlbsfjvk6nh0kx05q711jvn25n9l-llama-cpp-grpc.drv
error: builder for '/nix/store/b3vddlbsfjvk6nh0kx05q711jvn25n9l-llama-cpp-grpc.drv' failed with exit code 1;
       last 50 log lines:
       > -- x86 detected
       > CMake Warning at common/CMakeLists.txt:29 (message):
       >   Git repository not found; to enable automatic generation of build info,
       >   make sure Git is installed and the project is a Git repository.
       >
       > 
       > -- Found ZLIB: /nix/store/rrhw8p4hdrbh3pq7p2hlc1z2ixxzcxcb-zlib-1.3.1/lib/libz.so (found version "1.3.1")
       > -- Found OpenSSL: /nix/store/9xgpmwqw0881kxa9sl498qfmzy7z2ndn-openssl-3.0.13/lib/libcrypto.so (found version "3.0.13")
       > -- Found c-ares: /nix/store/vga9mw01vkkkg06alnnr64ldbbjzij4r-c-ares-1.27.0-dev/lib/cmake/c-ares/c-ares-config.cmake (found version "1.27.0")
       > -- Found RE2 via CMake.
       > -- Using protobuf version 24.4.0 | Protobuf_INCLUDE_DIRS:  | CMAKE_CURRENT_BINARY_DIR: /build/source/build/examples/grpc-server
       > -- Configuring done (4.4s)
       > CMake Error at CMakeLists.txt:1184 (target_link_libraries):
       >   Target "ggml" links to:
       >
       >     CUDA::cuda_driver
       >
       >   but the target was not found.  Possible reasons include:
       >
       >     * There is a typo in the target name.
       >     * A find_package call is missing for an IMPORTED target.
       >     * An ALIAS target is missing.
       >
       >
       > 
       > CMake Error at CMakeLists.txt:1208 (target_link_libraries):
       >   Target "llama" links to:
       >
       >     CUDA::cuda_driver
       >
       >   but the target was not found.  Possible reasons include:
       >
       >     * There is a typo in the target name.
       >     * A find_package call is missing for an IMPORTED target.
       >     * An ALIAS target is missing.
       >
       >
       > 
       > -- Generating done (0.1s)
       > CMake Warning:
       >   Manually-specified variables were not used by the project:
       >
       >     BUILD_SHARED_SERVER
       >     CMAKE_EXPORT_NO_PACKAGE_REGISTRY
       >     CMAKE_POLICY_DEFAULT_CMP0025
       >     CUDAToolkit_INCLUDE_DIR
       >     CUDA_HOST_COMPILER
       >
       > 
       > CMake Generate step failed.  Build files cannot be regenerated correctly.
       For full logs, run 'nix log /nix/store/b3vddlbsfjvk6nh0kx05q711jvn25n9l-llama-cpp-grpc.drv'.
/nix/store/52rlljxdgp5l0v2g5q5jw4jb863r202k-dcgm-3.2.5.drv
error: builder for '/nix/store/52rlljxdgp5l0v2g5q5jw4jb863r202k-dcgm-3.2.5.drv' failed with exit code 1;
       last 49 log lines:
       > Sourcing auto-add-driver-runpath-hook
       > Using autoAddDriverRunpath
       > Sourcing fix-elf-files.sh
       > Running phase: unpackPhase
       > unpacking source archive /nix/store/yhn5z0fijgp6kf4jhafp45w4v16lvvgg-source
       > source root is source
       > Running phase: patchPhase
       > Running phase: updateAutotoolsGnuConfigScriptsPhase
       > Running phase: configurePhase
       > fixing cmake files...
       > cmake flags: -DCMAKE_FIND_USE_SYSTEM_PACKAGE_REGISTRY=OFF -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF -DCMAKE_INSTALL_LOCALEDIR=/nix/store/fw29ankk4cplig3m7135kd8yl1p6ajmr-dcgm-3.2.5/share/locale -DCMAKE_INSTALL_LIBEXECDIR=/nix/store/fw29ankk4cplig3m7135kd8yl1p6ajmr-dcgm-3.2.5/libexec -DCMAKE_INSTALL_LIBDIR=/nix/store/fw29ankk4cplig3m7135kd8yl1p6ajmr-dcgm-3.2.5/lib -DCMAKE_INSTALL_DOCDIR=/nix/store/fw29ankk4cplig3m7135kd8yl1p6ajmr-dcgm-3.2.5/share/doc/dcgm -DCMAKE_INSTALL_INFODIR=/nix/store/fw29ankk4cplig3m7135kd8yl1p6ajmr-dcgm-3.2.5/share/info -DCMAKE_INSTALL_MANDIR=/nix/store/fw29ankk4cplig3m7135kd8yl1p6ajmr-dcgm-3.2.5/share/man -DCMAKE_INSTALL_OLDINCLUDEDIR=/nix/store/fw29ankk4cplig3m7135kd8yl1p6ajmr-dcgm-3.2.5/include -DCMAKE_INSTALL_INCLUDEDIR=/nix/store/fw29ankk4cplig3m7135kd8yl1p6ajmr-dcgm-3.2.5/include -DCMAKE_INSTALL_SBINDIR=/nix/store/fw29ankk4cplig3m7135kd8yl1p6ajmr-dcgm-3.2.5/sbin -DCMAKE_INSTALL_BINDIR=/nix/store/fw29ankk4cplig3m7135kd8yl1p6ajmr-dcgm-3.2.5/bin -DCMAKE_INSTALL_NAME_DIR=/nix/store/fw29ankk4cplig3m7135kd8yl1p6ajmr-dcgm-3.2.5/lib -DCMAKE_POLICY_DEFAULT_CMP0025=NEW -DCMAKE_OSX_SYSROOT= -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_STRIP=/nix/store/sqj66pwj8wgf1xcply3gfb7qwfndxw6y-gcc-wrapper-11.4.0/bin/strip -DCMAKE_RANLIB=/nix/store/sqj66pwj8wgf1xcply3gfb7qwfndxw6y-gcc-wrapper-11.4.0/bin/ranlib -DCMAKE_AR=/nix/store/sqj66pwj8wgf1xcply3gfb7qwfndxw6y-gcc-wrapper-11.4.0/bin/ar -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_INSTALL_PREFIX=/nix/store/fw29ankk4cplig3m7135kd8yl1p6ajmr-dcgm-3.2.5
       > -- The C compiler identification is GNU 11.4.0
       > -- The CXX compiler identification is GNU 11.4.0
       > -- Detecting C compiler ABI info
       > -- Detecting C compiler ABI info - done
       > -- Check for working C compiler: /nix/store/sqj66pwj8wgf1xcply3gfb7qwfndxw6y-gcc-wrapper-11.4.0/bin/gcc - skipped
       > -- Detecting C compile features
       > -- Detecting C compile features - done
       > -- Detecting CXX compiler ABI info
       > -- Detecting CXX compiler ABI info - done
       > -- Check for working CXX compiler: /nix/store/sqj66pwj8wgf1xcply3gfb7qwfndxw6y-gcc-wrapper-11.4.0/bin/g++ - skipped
       > -- Detecting CXX compile features
       > -- Detecting CXX compile features - done
       > -- Performing Test CC_HAS_WNO_CAST_FUNCTION_TYPE
       > -- Performing Test CC_HAS_WNO_CAST_FUNCTION_TYPE - Success
       > -- Performing Test CC_HAS_WUNREACHABLE_CODE_LOOP_INCREMENT
       > -- Performing Test CC_HAS_WUNREACHABLE_CODE_LOOP_INCREMENT - Failed
       > DCGM_LIB_INSTALL_PREFIX=/nix/store/fw29ankk4cplig3m7135kd8yl1p6ajmr-dcgm-3.2.5/lib
       > fatal: not a git repository (or any parent up to mount point /)
       > Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
       > fatal: not a git repository (or any parent up to mount point /)
       > Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
       > RT_PROPS: arch=x86_64;branch=;changelist=(dirty);os=linux;platform=linux-x86_64;version=3.2.5;component_name=dcgm
       > RT_PROPS: arch=x86_64;branch=;changelist=(dirty);os=linux;platform=linux-x86_64;version=3.2.5.0;component_name=dcgm_config
       > version:3.2.5;arch:x86_64;buildtype:Release;buildid:;builddate:2024-05-02;commit:(dirty);branch:;buildplatform:Linux 6.5.0-1019-azure #20-Ubuntu SMP Wed Apr  3 01:46:33 UTC 2024 x86_64;;crc:7accf220123e67e5b18968c542c8b286
       > /nix/store/d2wn8rd6hc3r2dy4nkp49l4jsffci33w-cuda-combined-10/targets/x86_64-linux/include
       > libcublas10-NOTFOUND
       > /nix/store/as2w5avsg0s99xjrz8hkpibaw0mkkbqc-cudatoolkit-10.2.89-lib/lib/libcudart_static.a
       > libculibos10-NOTFOUND
       > libcublaslt10-NOTFOUND
       > libcuda10-NOTFOUND
       > CMake Error at cmake/FindCuda.cmake:92 (message):
       >   Could NOT find Cuda 10
       > Call Stack (most recent call first):
       >   cmake/FindCuda.cmake:108 (load_cuda)
       >   cublas_proxy/CMakeLists.txt:14 (find_package)
       >
       > 
       > -- Configuring incomplete, errors occurred!
       For full logs, run 'nix log /nix/store/52rlljxdgp5l0v2g5q5jw4jb863r202k-dcgm-3.2.5.drv'.
/nix/store/9yima1h47f3asrx98pfmi23lc5s9pvd9-faiss-1.7.4.drv
error: builder for '/nix/store/9yima1h47f3asrx98pfmi23lc5s9pvd9-faiss-1.7.4.drv' failed with exit code 1;
       last 50 log lines:
       > -- Could NOT find MKL (missing: MKL_LIBRARIES)
       > -- Looking for sgemm_
       > -- Looking for sgemm_ - not found
       > -- Looking for sgemm_
       > -- Looking for sgemm_ - found
       > -- Found BLAS: /nix/store/cy6dh661j89s69mxlbi604z06arqb474-blas-3/lib/libblas.so
       > -- Looking for cheev_
       > -- Looking for cheev_ - found
       > -- Found LAPACK: /nix/store/cy6dh661j89s69mxlbi604z06arqb474-blas-3/lib/libblas.so;-lm;-ldl
       > -- Found CUDAToolkit: /nix/store/ah7yxsk3g2bcsv8jgc2z0va3l6fyl3k9-cuda_nvcc-12.4.131-dev/include (found version "12.4.131")
       > -- Found SWIG: /nix/store/ddf3jpb2nciwf10jcjzh55bk1c3hrq67-swig-4.0.2/bin/swig (found version "4.0.2") found components: python
       > -- Found Python: /nix/store/glfr70gi7hfaj50mwj2431p8bg60fhqw-python3-3.11.9/include/python3.11 (found version "3.11.9") found components: Development NumPy Interpreter Development.Module Development.Embed
       > -- Configuring done (7.2s)
       > CMake Error at faiss/gpu/CMakeLists.txt:180 (target_link_libraries):
       >   Target "faiss" links to:
       >
       >     CUDA::cublas
       >
       >   but the target was not found.  Possible reasons include:
       >
       >     * There is a typo in the target name.
       >     * A find_package call is missing for an IMPORTED target.
       >     * An ALIAS target is missing.
       >
       >
       > 
       > CMake Error at faiss/gpu/CMakeLists.txt:181 (target_link_libraries):
       >   Target "faiss_avx2" links to:
       >
       >     CUDA::cublas
       >
       >   but the target was not found.  Possible reasons include:
       >
       >     * There is a typo in the target name.
       >     * A find_package call is missing for an IMPORTED target.
       >     * An ALIAS target is missing.
       >
       >
       > 
       > -- Generating done (0.1s)
       > CMake Warning:
       >   Manually-specified variables were not used by the project:
       >
       >     CMAKE_EXPORT_NO_PACKAGE_REGISTRY
       >     CMAKE_POLICY_DEFAULT_CMP0025
       >     CUDAToolkit_INCLUDE_DIR
       >     CUDA_HOST_COMPILER
       >
       > 
       > CMake Generate step failed.  Build files cannot be regenerated correctly.
       For full logs, run 'nix log /nix/store/9yima1h47f3asrx98pfmi23lc5s9pvd9-faiss-1.7.4.drv'.
/nix/store/1gy4fjcigrwgpz2nkr40jlk3gl6bdfrs-llama-cpp-2746.drv
error: builder for '/nix/store/1gy4fjcigrwgpz2nkr40jlk3gl6bdfrs-llama-cpp-2746.drv' failed with exit code 1;
       last 50 log lines:
       >   Target "ggml" links to:
       >
       >     CUDA::cuda_driver
       >
       >   but the target was not found.  Possible reasons include:
       >
       >     * There is a typo in the target name.
       >     * A find_package call is missing for an IMPORTED target.
       >     * An ALIAS target is missing.
       >
       >
       > 
       > CMake Error at CMakeLists.txt:1187 (target_link_libraries):
       >   Target "ggml_shared" links to:
       >
       >     CUDA::cuda_driver
       >
       >   but the target was not found.  Possible reasons include:
       >
       >     * There is a typo in the target name.
       >     * A find_package call is missing for an IMPORTED target.
       >     * An ALIAS target is missing.
       >
       >
       > 
       > CMake Error at CMakeLists.txt:1204 (target_link_libraries):
       >   Target "llama" links to:
       >
       >     CUDA::cuda_driver
       >
       >   but the target was not found.  Possible reasons include:
       >
       >     * There is a typo in the target name.
       >     * A find_package call is missing for an IMPORTED target.
       >     * An ALIAS target is missing.
       >
       >
       > 
       > -- Generating done (0.1s)
       > CMake Warning:
       >   Manually-specified variables were not used by the project:
       >
       >     BUILD_SHARED_SERVER
       >     CMAKE_EXPORT_NO_PACKAGE_REGISTRY
       >     CMAKE_POLICY_DEFAULT_CMP0025
       >     CUDAToolkit_INCLUDE_DIR
       >     CUDA_HOST_COMPILER
       >
       > 
       > CMake Generate step failed.  Build files cannot be regenerated correctly.
       For full logs, run 'nix log /nix/store/1gy4fjcigrwgpz2nkr40jlk3gl6bdfrs-llama-cpp-2746.drv'.
/nix/store/59fqjbrvh264ygp0nixj61vpkdk35y5j-cuquantum-23.03.0.20.drv
error: builder for '/nix/store/59fqjbrvh264ygp0nixj61vpkdk35y5j-cuquantum-23.03.0.20.drv' failed with exit code 1;
       last 50 log lines:
       > Moving /nix/store/saw5dgcnq7mbla25h5c9l59d8l5d24wi-cuquantum-23.03.0.20-lib/lib/libcutensornet_static.a to /nix/store/4934fzsp07kk7ywpsssjg9r5wcwd440y-cuquantum-23.03.0.20-static/lib/libcutensornet_static.a
       > Running phase: fixupPhase
       > Moving /nix/store/aawm2zl7ylrpyi8i7pq78j05m76rskq1-cuquantum-23.03.0.20/include to /nix/store/4zi88r7fhdqazb04hh7xyv00pdj5j900-cuquantum-23.03.0.20-dev/include
       > shrinking RPATHs of ELF executables and libraries in /nix/store/4zi88r7fhdqazb04hh7xyv00pdj5j900-cuquantum-23.03.0.20-dev
       > checking for references to /build/ in /nix/store/4zi88r7fhdqazb04hh7xyv00pdj5j900-cuquantum-23.03.0.20-dev...
       > patching script interpreter paths in /nix/store/4zi88r7fhdqazb04hh7xyv00pdj5j900-cuquantum-23.03.0.20-dev
       > shrinking RPATHs of ELF executables and libraries in /nix/store/saw5dgcnq7mbla25h5c9l59d8l5d24wi-cuquantum-23.03.0.20-lib
       > shrinking /nix/store/saw5dgcnq7mbla25h5c9l59d8l5d24wi-cuquantum-23.03.0.20-lib/lib/libcustatevec.so.1.3.0.20
       > shrinking /nix/store/saw5dgcnq7mbla25h5c9l59d8l5d24wi-cuquantum-23.03.0.20-lib/lib/libcutensornet.so.2.1.0.20
       > checking for references to /build/ in /nix/store/saw5dgcnq7mbla25h5c9l59d8l5d24wi-cuquantum-23.03.0.20-lib...
       > patching script interpreter paths in /nix/store/saw5dgcnq7mbla25h5c9l59d8l5d24wi-cuquantum-23.03.0.20-lib
       > stripping (with command strip and flags -S -p) in  /nix/store/saw5dgcnq7mbla25h5c9l59d8l5d24wi-cuquantum-23.03.0.20-lib/lib
       > shrinking RPATHs of ELF executables and libraries in /nix/store/aawm2zl7ylrpyi8i7pq78j05m76rskq1-cuquantum-23.03.0.20
       > checking for references to /build/ in /nix/store/aawm2zl7ylrpyi8i7pq78j05m76rskq1-cuquantum-23.03.0.20...
       > patching script interpreter paths in /nix/store/aawm2zl7ylrpyi8i7pq78j05m76rskq1-cuquantum-23.03.0.20
       > shrinking RPATHs of ELF executables and libraries in /nix/store/4934fzsp07kk7ywpsssjg9r5wcwd440y-cuquantum-23.03.0.20-static
       > checking for references to /build/ in /nix/store/4934fzsp07kk7ywpsssjg9r5wcwd440y-cuquantum-23.03.0.20-static...
       > patching script interpreter paths in /nix/store/4934fzsp07kk7ywpsssjg9r5wcwd440y-cuquantum-23.03.0.20-static
       > stripping (with command strip and flags -S -p) in  /nix/store/4934fzsp07kk7ywpsssjg9r5wcwd440y-cuquantum-23.03.0.20-static/lib
       > Adding dev to out's propagated-build-inputs
       > Adding lib to out's propagated-build-inputs
       > Adding static to out's propagated-build-inputs
       > automatically fixing dependencies for ELF files
       > {'append_rpaths': [PosixPath('$ORIGIN')],
       >  'extra_args': [],
       >  'ignore_missing': ['libcuda.so', 'libcuda.so.*'],
       >  'libs': [PosixPath('/nix/store/f1ii69v7p27z1r5zybmlbld3bdzm6a5f-gcc-13.2.0-lib/lib'),
       >           PosixPath('/nix/store/wzp1i1ivq0zsaqlp00rc7lmlawb0cqhc-libcutensor-2.0.1.2-lib/lib'),
       >           PosixPath('/nix/store/k8gb9zxy7aph71n5sm4l1vv73r2sl75v-libcublas-12.4.5.8-lib/lib'),
       >           PosixPath('/nix/store/qc1p3jgcvkc5d0fshzq8zgrmrbcl9kfk-libcusolver-11.6.1.9-lib/lib')],
       >  'paths': [PosixPath('/nix/store/4zi88r7fhdqazb04hh7xyv00pdj5j900-cuquantum-23.03.0.20-dev'),
       >            PosixPath('/nix/store/saw5dgcnq7mbla25h5c9l59d8l5d24wi-cuquantum-23.03.0.20-lib'),
       >            PosixPath('/nix/store/aawm2zl7ylrpyi8i7pq78j05m76rskq1-cuquantum-23.03.0.20'),
       >            PosixPath('/nix/store/4934fzsp07kk7ywpsssjg9r5wcwd440y-cuquantum-23.03.0.20-static')],
       >  'recursive': True,
       >  'runtime_dependencies': []}
       > searching for dependencies of /nix/store/saw5dgcnq7mbla25h5c9l59d8l5d24wi-cuquantum-23.03.0.20-lib/lib/libcustatevec.so.1.3.0.20
       >     libcublas.so.12 -> found: /nix/store/k8gb9zxy7aph71n5sm4l1vv73r2sl75v-libcublas-12.4.5.8-lib/lib
       >     libgcc_s.so.1 -> found: /nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib
       > setting RPATH to: /nix/store/k8gb9zxy7aph71n5sm4l1vv73r2sl75v-libcublas-12.4.5.8-lib/lib:/nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib:$ORIGIN
       > searching for dependencies of /nix/store/saw5dgcnq7mbla25h5c9l59d8l5d24wi-cuquantum-23.03.0.20-lib/lib/libcutensornet.so.2.1.0.20
       >     libcutensor.so.1 -> not found!
       >     libcublas.so.12 -> found: /nix/store/k8gb9zxy7aph71n5sm4l1vv73r2sl75v-libcublas-12.4.5.8-lib/lib
       >     libcusolver.so.11 -> found: /nix/store/qc1p3jgcvkc5d0fshzq8zgrmrbcl9kfk-libcusolver-11.6.1.9-lib/lib
       >     libgcc_s.so.1 -> found: /nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib
       > setting RPATH to: /nix/store/k8gb9zxy7aph71n5sm4l1vv73r2sl75v-libcublas-12.4.5.8-lib/lib:/nix/store/qc1p3jgcvkc5d0fshzq8zgrmrbcl9kfk-libcusolver-11.6.1.9-lib/lib:/nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib:$ORIGIN
       > auto-patchelf: 1 dependencies could not be satisfied
       > error: auto-patchelf could not satisfy dependency libcutensor.so.1 wanted by /nix/store/saw5dgcnq7mbla25h5c9l59d8l5d24wi-cuquantum-23.03.0.20-lib/lib/libcutensornet.so.2.1.0.20
       > auto-patchelf failed to find all the required dependencies.
       > Add the missing dependencies to --libs or use `--ignore-missing="foo.so.1 bar.so etc.so"`.
       For full logs, run 'nix log /nix/store/59fqjbrvh264ygp0nixj61vpkdk35y5j-cuquantum-23.03.0.20.drv'.
/nix/store/cqr10szkz2n56iq18l2f02n3gk745rhq-python3.11-cupy-13.0.0.drv
error: builder for '/nix/store/cqr10szkz2n56iq18l2f02n3gk745rhq-python3.11-cupy-13.0.0.drv' failed with exit code 1;
       last 50 log lines:
       > command '/nix/store/4kwk49k8rc8cg8c387kbbjcrl8i0y693-gcc-wrapper-13.2.0/bin/g++' failed with exit code 1
       > **************************************************
       >
       > ************************************************************
       > * CuPy Configuration Summary                               *
       > ************************************************************
       >
       > Build Environment:
       >   Include directories: ['/build/cupy-13.0.0/cupy/_core/include/cupy/_cccl/libcudacxx', '/build/cupy-13.0.0/cupy/_core/include/cupy/_cccl/thrust', '/build/cupy-13.0.0/cupy/_core/include/cupy/_cccl/cub', '/build/cupy-13.0.0/cupy/_core/include', '/nix/store/aprgv60d2gsin62j1iyfs55ww0kkcj6s-cudatoolkit-joined-11.8/include']
       >   Library directories: ['/nix/store/aprgv60d2gsin62j1iyfs55ww0kkcj6s-cudatoolkit-joined-11.8/lib']
       >   nvcc command       : ['/nix/store/zrx93j5a9rn260ighn3mbrvyxz59ybzr-cuda_nvcc-11.8.89/bin/nvcc']
       >   hipcc command      : (not found)
       >
       > Environment Variables:
       >   CFLAGS          : (none)
       >   LDFLAGS         : -L/nix/store/k7010jfalw8vijsrr2fcw262gd7as6ws-cuda_cudart-11.8.89/lib/stubs
       >   LIBRARY_PATH    : (none)
       >   CUDA_PATH       : /nix/store/aprgv60d2gsin62j1iyfs55ww0kkcj6s-cudatoolkit-joined-11.8
       >   NVCC            : /nix/store/zrx93j5a9rn260ighn3mbrvyxz59ybzr-cuda_nvcc-11.8.89/bin/nvcc
       >   HIPCC           : (none)
       >   ROCM_HOME       : (none)
       >
       > Modules:
       >   cuda      : No
       >     -> Include files not found: ['cublas_v2.h', 'cuda.h', 'cuda_profiler_api.h', 'cuda_runtime.h', 'cufft.h', 'curand.h', 'cusparse.h']
       >     -> Check your CFLAGS environment variable.
       >
       > ERROR: CUDA could not be found on your system.
       >
       > HINT: You are trying to build CuPy from source, which is NOT recommended for general use.
       >       Please consider using binary packages instead.
       >
       > Please refer to the Installation Guide for details:
       > https://docs.cupy.dev/en/stable/install.html
       >
       > ************************************************************
       >
       > Traceback (most recent call last):
       >   File "/build/cupy-13.0.0/nix_run_setup", line 8, in <module>
       >     exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
       >   File "setup.py", line 93, in <module>
       >     ext_modules = cupy_setup_build.get_ext_modules(True, ctx)
       >                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       >   File "/build/cupy-13.0.0/install/cupy_builder/cupy_setup_build.py", line 473, in get_ext_modules
       >     extensions = make_extensions(ctx, compiler, use_cython)
       >                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       >   File "/build/cupy-13.0.0/install/cupy_builder/cupy_setup_build.py", line 328, in make_extensions
       >     raise Exception('Your CUDA environment is invalid. '
       > Exception: Your CUDA environment is invalid. Please check above error log.
       > /nix/store/9wnvhjyxjykwn5y06xc9a2h8rs5fbfia-stdenv-linux/setup: line 1579: pop_var_context: head of shell_variables not a function context
       For full logs, run 'nix log /nix/store/cqr10szkz2n56iq18l2f02n3gk745rhq-python3.11-cupy-13.0.0.drv'.
/nix/store/wc8sv0zrva7fdplw5c34c98v9vqwnk00-python3.11-lightgbm-4.3.0.drv
error: builder for '/nix/store/wc8sv0zrva7fdplw5c34c98v9vqwnk00-python3.11-lightgbm-4.3.0.drv' failed with exit code 1;
       last 50 log lines:
       > * Getting build dependencies for wheel...
       > * Building wheel...
       > 2024-05-02 00:49:23,846 - scikit_build_core - INFO - RUN: /nix/store/0xpv46mb188z226j7cs211zmmi5xw9nj-cmake-3.29.1/bin/cmake --version
       > 2024-05-02 00:49:23,887 - scikit_build_core - INFO - CMake version: 3.29.1
       > *** scikit-build-core 0.8.2 using CMake 3.29.1 (wheel)
       > 2024-05-02 00:49:23,896 - scikit_build_core - INFO - Build directory: /build/tmpcdhm3hu6/build
       > *** Configuring CMake...
       > 2024-05-02 00:49:24,039 - scikit_build_core - INFO - RUN: /nix/store/wvxpsf8xbyz31j0j2azjkndn1wbmh5ia-ninja-1.11.1/bin/ninja --version
       > 2024-05-02 00:49:24,059 - scikit_build_core - INFO - Ninja version: 1.11.1
       > 2024-05-02 00:49:24,237 - scikit_build_core - INFO - RUN: /nix/store/0xpv46mb188z226j7cs211zmmi5xw9nj-cmake-3.29.1/bin/cmake -S. -B/build/tmpcdhm3hu6/build -DCMAKE_BUILD_TYPE:STRING=Release -C/build/tmpcdhm3hu6/build/CMakeInit.txt -DCMAKE_MAKE_PROGRAM=/nix/store/wvxpsf8xbyz31j0j2azjkndn1wbmh5ia-ninja-1.11.1/bin/ninja -DUSE_CUDA=ON -D__BUILD_FOR_PYTHON:BOOL=ON
       > loading initial cache file /build/tmpcdhm3hu6/build/CMakeInit.txt
       > -- The C compiler identification is GNU 13.2.0
       > -- The CXX compiler identification is GNU 13.2.0
       > -- Detecting C compiler ABI info
       > -- Detecting C compiler ABI info - done
       > -- Check for working C compiler: /nix/store/4kwk49k8rc8cg8c387kbbjcrl8i0y693-gcc-wrapper-13.2.0/bin/gcc - skipped
       > -- Detecting C compile features
       > -- Detecting C compile features - done
       > -- Detecting CXX compiler ABI info
       > -- Detecting CXX compiler ABI info - done
       > -- Check for working CXX compiler: /nix/store/4kwk49k8rc8cg8c387kbbjcrl8i0y693-gcc-wrapper-13.2.0/bin/g++ - skipped
       > -- Detecting CXX compile features
       > -- Detecting CXX compile features - done
       > -- The CUDA compiler identification is NVIDIA 12.4.131
       > -- Detecting CUDA compiler ABI info
       > -- Detecting CUDA compiler ABI info - done
       > -- Check for working CUDA compiler: /nix/store/jpr10dkcpvv56dn3hqq7710lbrbalvvg-cuda_nvcc-12.4.131-bin/bin/nvcc - skipped
       > -- Detecting CUDA compile features
       > -- Detecting CUDA compile features - done
       > -- Found OpenMP_C: -fopenmp (found version "4.5")
       > -- Found OpenMP_CXX: -fopenmp (found version "4.5")
       > -- Found OpenMP: TRUE (found version "4.5")
       > CMake Warning (dev) at CMakeLists.txt:201 (find_package):
       >   Policy CMP0146 is not set: The FindCUDA module is removed.  Run "cmake
       >   --help-policy CMP0146" for policy details.  Use the cmake_policy command to
       >   set the policy and suppress this warning.
       >
       > This warning is for project developers.  Use -Wno-dev to suppress it.
       > 
       > CMake Error at /nix/store/0xpv46mb188z226j7cs211zmmi5xw9nj-cmake-3.29.1/share/cmake-3.29/Modules/FindCUDA.cmake:883 (message):
       >   Specify CUDA_TOOLKIT_ROOT_DIR
       > Call Stack (most recent call first):
       >   CMakeLists.txt:201 (find_package)
       >
       > 
       > -- Configuring incomplete, errors occurred!
       >
       > *** CMake configuration failed
       >
       > ERROR Backend subprocess exited when trying to invoke build_wheel
       For full logs, run 'nix log /nix/store/wc8sv0zrva7fdplw5c34c98v9vqwnk00-python3.11-lightgbm-4.3.0.drv'.
/nix/store/axcv3wlw3nz9wygacj8dzyk6jsgjp39s-bazel-build-jaxlib-0.4.24-deps.tar.gz.drv
error: builder for '/nix/store/axcv3wlw3nz9wygacj8dzyk6jsgjp39s-bazel-build-jaxlib-0.4.24-deps.tar.gz.drv' failed with exit code 2;
       last 50 log lines:
       >          config = find_cuda_config(repository_ctx, ["cuda", "cudnn"])
       >   File "/build/output/external/tsl/third_party/gpus/cuda_configure.bzl", line 693, column 26, in find_cuda_config
       >                exec_result = execute(repository_ctx, [python_bin, repository_ctx.attr._find_cuda_config] + cuda_libraries)
       >    File "/build/output/external/tsl/third_party/remote_config/common.bzl", line 230, column 13, in execute
       >                fail(
       > Error in fail: Repository command failed
       > Could not find any nvml.h in any subdirectory:
       >         ''
       >         'include'
       >         'include/cuda'
       >         'include/*-linux-gnu'
       >         'extras/CUPTI/include'
       >         'include/cuda/CUPTI'
       >         'local/cuda/extras/CUPTI/include'
       >         'targets/x86_64-linux/include'
       > of:
       >         '/nix/store/sy3977fpiq8rjn7vnrrcxw657i8p4q35-cuda-build-deps-joined'
       > INFO: Repository rules_cc instantiated at:
       >   /build/source/WORKSPACE:15:15: in <toplevel>
       >   /build/output/external/xla/workspace1.bzl:12:19: in workspace
       >   /build/output/external/tsl/workspace1.bzl:30:14: in workspace
       >   /build/output/external/com_github_grpc_grpc/bazel/grpc_deps.bzl:158:21: in grpc_deps
       > Repository rule http_archive defined at:
       >   /build/output/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
       > ERROR: Skipping '@xla//xla/python:enable_gpu': no such package '@local_config_cuda//cuda': Repository command failed
       > Could not find any nvml.h in any subdirectory:
       >         ''
       >         'include'
       >         'include/cuda'
       >         'include/*-linux-gnu'
       >         'extras/CUPTI/include'
       >         'include/cuda/CUPTI'
       >         'local/cuda/extras/CUPTI/include'
       >         'targets/x86_64-linux/include'
       > of:
       >         '/nix/store/sy3977fpiq8rjn7vnrrcxw657i8p4q35-cuda-build-deps-joined'
       > WARNING: Target pattern parsing failed.
       > ERROR: @xla//xla/python:enable_gpu :: Error loading option @xla//xla/python:enable_gpu: no such package '@local_config_cuda//cuda': Repository command failed
       > Could not find any nvml.h in any subdirectory:
       >         ''
       >         'include'
       >         'include/cuda'
       >         'include/*-linux-gnu'
       >         'extras/CUPTI/include'
       >         'include/cuda/CUPTI'
       >         'local/cuda/extras/CUPTI/include'
       >         'targets/x86_64-linux/include'
       > of:
       >         '/nix/store/sy3977fpiq8rjn7vnrrcxw657i8p4q35-cuda-build-deps-joined'
       > 
       For full logs, run 'nix log /nix/store/axcv3wlw3nz9wygacj8dzyk6jsgjp39s-bazel-build-jaxlib-0.4.24-deps.tar.gz.drv'.
/nix/store/wi844yixix3pxn0z8vcw0gxsg8p20ja5-gromacs-2024.1.drv
error: builder for '/nix/store/wi844yixix3pxn0z8vcw0gxsg8p20ja5-gromacs-2024.1.drv' failed with exit code 1;
       last 50 log lines:
       > -- Looking for include file mm_malloc.h
       > -- Looking for include file mm_malloc.h - found
       > -- Looking for include file malloc.h
       > -- Looking for include file malloc.h - found
       > -- Checking for _mm_malloc()
       > -- Checking for _mm_malloc() - supported
       > -- Looking for posix_memalign
       > -- Looking for posix_memalign - found
       > -- Looking for memalign
       > -- Looking for memalign - not found
       > -- Using default binary suffix: ""
       > -- Using default library suffix: ""
       > -- Looking for HWLOC
       > -- Looking for hwloc_topology_init
       > -- Looking for hwloc_topology_init - found
       > -- hwloc version:
       > -- Found HWLOC: /nix/store/dfhk5bji8bsq422k3kxdjjcir51w23bp-hwloc-2.10.0-lib/lib/libhwloc.so (found suitable version "2.10.0", minimum required is "1.5")
       > -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
       > -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
       > -- Found Threads: TRUE
       > -- Looking for C++ include pthread.h
       > -- Looking for C++ include pthread.h - found
       > -- Performing Test TEST_ATOMICS
       > -- Performing Test TEST_ATOMICS - Success
       > -- Atomic operations found
       > -- Performing Test PTHREAD_SETAFFINITY
       > -- Performing Test PTHREAD_SETAFFINITY - Success
       > -- Performing Test C_msse4_1_FLAG_ACCEPTED
       > -- Performing Test C_msse4_1_FLAG_ACCEPTED - Success
       > -- Performing Test C_msse4_1_COMPILE_WORKS
       > -- Performing Test C_msse4_1_COMPILE_WORKS - Success
       > -- Performing Test CXX_msse4_1_FLAG_ACCEPTED
       > -- Performing Test CXX_msse4_1_FLAG_ACCEPTED - Success
       > -- Performing Test CXX_msse4_1_COMPILE_WORKS
       > -- Performing Test CXX_msse4_1_COMPILE_WORKS - Success
       > -- Enabling SSE4.1 SIMD instructions using CXX flags:  -msse4.1
       > -- Performing Test _callconv___vectorcall
       > -- Performing Test _callconv___vectorcall - Failed
       > -- Performing Test _callconv___regcall
       > -- Performing Test _callconv___regcall - Failed
       > -- Performing Test _callconv_
       > -- Performing Test _callconv_  - Success
       > CMake Error at /nix/store/0xpv46mb188z226j7cs211zmmi5xw9nj-cmake-3.29.1/share/cmake-3.29/Modules/FindCUDA.cmake:883 (message):
       >   Specify CUDA_TOOLKIT_ROOT_DIR
       > Call Stack (most recent call first):
       >   cmake/gmxManageCuda.cmake:45 (find_package)
       >   CMakeLists.txt:699 (include)
       >
       > 
       > -- Configuring incomplete, errors occurred!
       For full logs, run 'nix log /nix/store/wi844yixix3pxn0z8vcw0gxsg8p20ja5-gromacs-2024.1.drv'.
/nix/store/q11g2v43vz615l777dgxqib1zb847ksd-python3.11-tensorrt-10.0.1.6.drv
error: builder for '/nix/store/q11g2v43vz615l777dgxqib1zb847ksd-python3.11-tensorrt-10.0.1.6.drv' failed with exit code 2;
       last 18 log lines:
       > Sourcing python-remove-tests-dir-hook
       > Sourcing python-catch-conflicts-hook.sh
       > Sourcing python-remove-bin-bytecode-hook.sh
       > Sourcing wheel setup hook
       > Sourcing pypa-install-hook
       > Using pypaInstallPhase
       > Sourcing python-imports-check-hook.sh
       > Using pythonImportsCheckPhase
       > Sourcing python-namespaces-hook
       > Sourcing python-catch-conflicts-hook.sh
       > Sourcing auto-add-driver-runpath-hook
       > Using autoAddDriverRunpath
       > Sourcing fix-elf-files.sh
       > Running phase: unpackPhase
       > tar: /nix/store/n66l0ssg31s13crbijhrc9gkmjxmmrli-TensorRT-10.0.1.6.Linux.x86_64-gnu.cuda-11.8-source: Cannot read: Is a directory
       > tar: At beginning of tape, quitting now
       > tar: Error is not recoverable: exiting now
       > /nix/store/9wnvhjyxjykwn5y06xc9a2h8rs5fbfia-stdenv-linux/setup: line 131: pop_var_context: head of shell_variables not a function context
       For full logs, run 'nix log /nix/store/q11g2v43vz615l777dgxqib1zb847ksd-python3.11-tensorrt-10.0.1.6.drv'.
/nix/store/gn8j87m2x3a8ja60bdvsyl77qpk2ffjs-python3.11-torch-2.2.2.drv
error: builder for '/nix/store/gn8j87m2x3a8ja60bdvsyl77qpk2ffjs-python3.11-torch-2.2.2.drv' failed with exit code 1;
       last 50 log lines:
       >     libcublasLt.so.12 -> found: /nix/store/k8gb9zxy7aph71n5sm4l1vv73r2sl75v-libcublas-12.4.5.8-lib/lib
       >     libcudnn.so.8 -> not found!
       >     libnccl.so.2 -> found: /nix/store/dblzql0ckypj5jpk16gkcrql84wmhrrr-nccl-2.21.5-1/lib
       >     libc10.so -> found: /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib
       >     libtorch_cpu.so -> found: /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib
       >     libstdc++.so.6 -> found: /nix/store/f1ii69v7p27z1r5zybmlbld3bdzm6a5f-gcc-13.2.0-lib/lib
       >     libgcc_s.so.1 -> found: /nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib
       > setting RPATH to: /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib:/nix/store/sc86fzwf05x1qsqxh87zl14azmfimlxw-cuda_cudart-12.4.127-lib/lib:/nix/store/p2pbkq2i2qdy62h30xv5i8k8ds91jy1i-libcusparse-12.3.1.170-lib/lib:/nix/store/7kvi8bn6vhk1jlsssvzgmajfp84k8cka-libcufft-11.2.1.3-lib/lib:/nix/store/d9s9sdgry1jb8rhabxlafsg7finxqh88-cuda_nvtx-12.4.127-lib/lib:/nix/store/hhvahl1kjs0zailh1bk0nsvhn5rkxgz1-libcurand-10.3.5.147-lib/lib:/nix/store/k8gb9zxy7aph71n5sm4l1vv73r2sl75v-libcublas-12.4.5.8-lib/lib:/nix/store/dblzql0ckypj5jpk16gkcrql84wmhrrr-nccl-2.21.5-1/lib:/nix/store/f1ii69v7p27z1r5zybmlbld3bdzm6a5f-gcc-13.2.0-lib/lib:/nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib
       > searching for dependencies of /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib/libtorch_cuda_linalg.so
       >     libtorch_cpu.so -> found: /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib
       >     libtorch_cuda.so -> found: /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib
       >     libcusolver.so.11 -> found: /nix/store/qc1p3jgcvkc5d0fshzq8zgrmrbcl9kfk-libcusolver-11.6.1.9-lib/lib
       >     libc10_cuda.so -> found: /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib
       >     libcudart.so.12 -> found: /nix/store/sc86fzwf05x1qsqxh87zl14azmfimlxw-cuda_cudart-12.4.127-lib/lib
       >     libnvToolsExt.so.1 -> found: /nix/store/d9s9sdgry1jb8rhabxlafsg7finxqh88-cuda_nvtx-12.4.127-lib/lib
       >     libc10.so -> found: /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib
       >     libcublas.so.12 -> found: /nix/store/k8gb9zxy7aph71n5sm4l1vv73r2sl75v-libcublas-12.4.5.8-lib/lib
       >     libcusparse.so.12 -> found: /nix/store/p2pbkq2i2qdy62h30xv5i8k8ds91jy1i-libcusparse-12.3.1.170-lib/lib
       >     libstdc++.so.6 -> found: /nix/store/f1ii69v7p27z1r5zybmlbld3bdzm6a5f-gcc-13.2.0-lib/lib
       >     libgcc_s.so.1 -> found: /nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib
       > setting RPATH to: /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib:/nix/store/qc1p3jgcvkc5d0fshzq8zgrmrbcl9kfk-libcusolver-11.6.1.9-lib/lib:/nix/store/sc86fzwf05x1qsqxh87zl14azmfimlxw-cuda_cudart-12.4.127-lib/lib:/nix/store/d9s9sdgry1jb8rhabxlafsg7finxqh88-cuda_nvtx-12.4.127-lib/lib:/nix/store/k8gb9zxy7aph71n5sm4l1vv73r2sl75v-libcublas-12.4.5.8-lib/lib:/nix/store/p2pbkq2i2qdy62h30xv5i8k8ds91jy1i-libcusparse-12.3.1.170-lib/lib:/nix/store/f1ii69v7p27z1r5zybmlbld3bdzm6a5f-gcc-13.2.0-lib/lib:/nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib
       > searching for dependencies of /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib/libtorch_global_deps.so
       >     libcudart.so.12 -> found: /nix/store/sc86fzwf05x1qsqxh87zl14azmfimlxw-cuda_cudart-12.4.127-lib/lib
       >     libnvToolsExt.so.1 -> found: /nix/store/d9s9sdgry1jb8rhabxlafsg7finxqh88-cuda_nvtx-12.4.127-lib/lib
       >     libgomp-a34b3233.so.1 -> found: /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib
       > setting RPATH to: /nix/store/sc86fzwf05x1qsqxh87zl14azmfimlxw-cuda_cudart-12.4.127-lib/lib:/nix/store/d9s9sdgry1jb8rhabxlafsg7finxqh88-cuda_nvtx-12.4.127-lib/lib:/nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib
       > searching for dependencies of /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib/libtorch_python.so
       >     libshm.so -> found: /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib
       >     libtorch.so -> found: /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib
       >     libnvToolsExt.so.1 -> found: /nix/store/d9s9sdgry1jb8rhabxlafsg7finxqh88-cuda_nvtx-12.4.127-lib/lib
       >     libtorch_cpu.so -> found: /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib
       >     libtorch_cuda.so -> found: /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib
       >     libc10_cuda.so -> found: /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib
       >     libc10.so -> found: /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib
       >     libcudart.so.12 -> found: /nix/store/sc86fzwf05x1qsqxh87zl14azmfimlxw-cuda_cudart-12.4.127-lib/lib
       >     libcudnn.so.8 -> not found!
       >     libstdc++.so.6 -> found: /nix/store/f1ii69v7p27z1r5zybmlbld3bdzm6a5f-gcc-13.2.0-lib/lib
       >     libgcc_s.so.1 -> found: /nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib
       > setting RPATH to: /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib:/nix/store/d9s9sdgry1jb8rhabxlafsg7finxqh88-cuda_nvtx-12.4.127-lib/lib:/nix/store/sc86fzwf05x1qsqxh87zl14azmfimlxw-cuda_cudart-12.4.127-lib/lib:/nix/store/f1ii69v7p27z1r5zybmlbld3bdzm6a5f-gcc-13.2.0-lib/lib:/nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib
       > searching for dependencies of /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib/libgomp-a34b3233.so.1
       > searching for dependencies of /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib/libcusparseLt-f8b4a9fb.so.0
       >     libstdc++.so.6 -> found: /nix/store/f1ii69v7p27z1r5zybmlbld3bdzm6a5f-gcc-13.2.0-lib/lib
       >     libgcc_s.so.1 -> found: /nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib
       > setting RPATH to: /nix/store/f1ii69v7p27z1r5zybmlbld3bdzm6a5f-gcc-13.2.0-lib/lib:/nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib
       > auto-patchelf: 3 dependencies could not be satisfied
       > warn: auto-patchelf ignoring missing libcuda.so.1 wanted by /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib/libcaffe2_nvrtc.so
       > error: auto-patchelf could not satisfy dependency libcudnn.so.8 wanted by /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so
       > error: auto-patchelf could not satisfy dependency libcudnn.so.8 wanted by /nix/store/jlmz070r9gxb8861ggdjwlsyzkb8klyr-python3.11-torch-2.2.2/lib/python3.11/site-packages/torch/lib/libtorch_python.so
       > auto-patchelf failed to find all the required dependencies.
       > Add the missing dependencies to --libs or use `--ignore-missing="foo.so.1 bar.so etc.so"`.
       For full logs, run 'nix log /nix/store/gn8j87m2x3a8ja60bdvsyl77qpk2ffjs-python3.11-torch-2.2.2.drv'.
/nix/store/xf4nswmzx3jni7lcnvb4iv92jmfd6nkl-opencv-4.9.0.drv
error: builder for '/nix/store/xf4nswmzx3jni7lcnvb4iv92jmfd6nkl-opencv-4.9.0.drv' failed with exit code 2;
       last 50 log lines:
       >             instantiation of "void cv::cudev::gridReduceToColumn<Reductor,SrcPtr,ResType>(const SrcPtr &, cv::cudev::GpuMat_<ResType> &, cv::cuda::Stream &) [with Reductor=cv::cudev::Min<double4>, SrcPtr=cv::cudev::GpuMat_<uchar4>, ResType=double4]" at line 104 of /build/source/opencv_contrib/cudaarithm/src/cuda/reduce.cu
       >             instantiation of "void <unnamed>::reduceToColumnImpl_<T,S,D>(const cv::cuda::GpuMat &, cv::cuda::GpuMat &, int, cv::cuda::Stream &) [with T=uchar4, S=double4, D=double4]" at line 123 of /build/source/opencv_contrib/cudaarithm/src/cuda/reduce.cu
       >             instantiation of "void <unnamed>::reduceToColumnImpl<T,S,D>(const cv::cuda::GpuMat &, cv::cuda::GpuMat &, int, cv::cuda::Stream &) [with T=uchar, S=double, D=double]" at line 232 of /build/source/opencv_contrib/cudaarithm/src/cuda/reduce.cu
       >
       > make[2]: *** [modules/cudaarithm/CMakeFiles/opencv_cudaarithm.dir/build.make:287: modules/cudaarithm/CMakeFiles/cuda_compile_1.dir/src/cuda/cuda_compile_1_generated_sum.cu.o] Error 1
       > CMake Error at cuda_compile_1_generated_minmaxloc.cu.o.Release.cmake:280 (message):
       >   Error generating file
       >   /build/source/build/modules/cudaarithm/CMakeFiles/cuda_compile_1.dir/src/cuda/./cuda_compile_1_generated_minmaxloc.cu.o
       >
       > 
       > CMake Error at cuda_compile_1_generated_polar_cart.cu.o.Release.cmake:280 (message):
       >   Error generating file
       >   /build/source/build/modules/cudaarithm/CMakeFiles/cuda_compile_1.dir/src/cuda/./cuda_compile_1_generated_polar_cart.cu.o
       >
       > 
       > make[2]: *** [modules/cudaarithm/CMakeFiles/opencv_cudaarithm.dir/build.make:210: modules/cudaarithm/CMakeFiles/cuda_compile_1.dir/src/cuda/cuda_compile_1_generated_minmaxloc.cu.o] Error 1
       > make[2]: *** [modules/cudaarithm/CMakeFiles/opencv_cudaarithm.dir/build.make:245: modules/cudaarithm/CMakeFiles/cuda_compile_1.dir/src/cuda/cuda_compile_1_generated_normalize.cu.o] Error 1
       > make[2]: *** [modules/cudaarithm/CMakeFiles/opencv_cudaarithm.dir/build.make:266: modules/cudaarithm/CMakeFiles/cuda_compile_1.dir/src/cuda/cuda_compile_1_generated_split_merge.cu.o] Error 1
       > make[2]: *** [modules/cudaarithm/CMakeFiles/opencv_cudaarithm.dir/build.make:252: modules/cudaarithm/CMakeFiles/cuda_compile_1.dir/src/cuda/cuda_compile_1_generated_polar_cart.cu.o] Error 1
       > /build/source/opencv_contrib/cudev/include/opencv2/cudev/grid/detail/reduce_to_column.hpp(91): error: no instance of overloaded function "cv::cudev::blockReduce" matches the argument list
       >             argument types are: (cuda::std::__4::tuple<volatile double *, volatile double *, volatile double *, volatile double *>, cuda::std::__4::tuple<double &, double &, double &, double &>, const unsigned int, cuda::std::__4::tuple<cv::cudev::Max<double>, cv::cudev::Max<double>, cv::cudev::Max<double>, cv::cudev::Max<double>>)
       >               blockReduce<BLOCK_SIZE>(smem_tuple(smem[0], smem[1], smem[2], smem[3]), tie(myVal.x, myVal.y, myVal.z, myVal.w), threadIdx.x, make_tuple(op, op, op, op));
       >               ^
       > /build/source/opencv_contrib/cudev/include/opencv2/cudev/block/reduce.hpp(72): note #3327-D: candidate function template "cv::cudev::blockReduce<N,P0,P1,P2,P3,P4,P5,P6,P7,P8,P9,R0,R1,R2,R3,R4,R5,R6,R7,R8,R9,Op0,Op1,Op2,Op3,Op4,Op5,Op6,Op7,Op8,Op9>(const thrust::THRUST_200302_750_NS::tuple<P0, P1, P2, P3, P4, P5, P6, P7, P8, P9> &, const thrust::THRUST_200302_750_NS::tuple<R0, R1, R2, R3, R4, R5, R6, R7, R8, R9> &, uint, const thrust::THRUST_200302_750_NS::tuple<Op0, Op1, Op2, Op3, Op4, Op5, Op6, Op7, Op8, Op9> &)" failed deduction
       >                             void blockReduce(const tuple<P0, P1, P2, P3, P4, P5, P6, P7, P8, P9>& smem,
       >                                  ^
       > /build/source/opencv_contrib/cudev/include/opencv2/cudev/block/reduce.hpp(63): note #3327-D: candidate function template "cv::cudev::blockReduce<N,T,Op>(volatile T *, T &, uint, const Op &)" failed deduction
       >                             void blockReduce(volatile T* smem, T& val, uint tid, const Op& op)
       >                                  ^
       >           detected during:
       >             instantiation of "void cv::cudev::grid_reduce_to_vec_detail::Reduce<BLOCK_SIZE, work_type, work_elem_type, Reductor, 4>::call(work_elem_type (*)[BLOCK_SIZE], work_type &) [with BLOCK_SIZE=256, work_type=double4, work_elem_type=double, Reductor=cv::cudev::Max<double4>]" at line 118
       >             instantiation of "void cv::cudev::grid_reduce_to_vec_detail::reduceToColumn<Reductor,BLOCK_SIZE,SrcPtr,ResType,MaskPtr>(SrcPtr, ResType *, MaskPtr, int) [with Reductor=cv::cudev::Max<double4>, BLOCK_SIZE=256, SrcPtr=cv::cudev::GlobPtr<uchar4>, ResType=double4, MaskPtr=cv::cudev::WithOutMask]" at line 135
       >             instantiation of "void cv::cudev::grid_reduce_to_vec_detail::reduceToColumn<Reductor,Policy,SrcPtr,ResType,MaskPtr>(const SrcPtr &, ResType *, const MaskPtr &, int, int, cudaStream_t) [with Reductor=cv::cudev::Max<double4>, Policy=cv::cudev::DefaultReduceToVecPolicy, SrcPtr=cv::cudev::GlobPtr<uchar4>, ResType=double4, MaskPtr=cv::cudev::WithOutMask]" at line 206 of /build/source/opencv_contrib/cudev/include/opencv2/cudev/grid/reduce_to_vec.hpp
       >             instantiation of "void cv::cudev::gridReduceToColumn_<Reductor,Policy,SrcPtr,ResType>(const SrcPtr &, cv::cudev::GpuMat_<ResType> &, cv::cuda::Stream &) [with Reductor=cv::cudev::Max<double4>, Policy=cv::cudev::DefaultReduceToVecPolicy, SrcPtr=cv::cudev::GpuMat_<uchar4>, ResType=double4]" at line 228 of /build/source/opencv_contrib/cudev/include/opencv2/cudev/grid/reduce_to_vec.hpp
       >             instantiation of "void cv::cudev::gridReduceToColumn<Reductor,SrcPtr,ResType>(const SrcPtr &, cv::cudev::GpuMat_<ResType> &, cv::cuda::Stream &) [with Reductor=cv::cudev::Max<double4>, SrcPtr=cv::cudev::GpuMat_<uchar4>, ResType=double4]" at line 108 of /build/source/opencv_contrib/cudaarithm/src/cuda/reduce.cu
       >             instantiation of "void <unnamed>::reduceToColumnImpl_<T,S,D>(const cv::cuda::GpuMat &, cv::cuda::GpuMat &, int, cv::cuda::Stream &) [with T=uchar4, S=double4, D=double4]" at line 123 of /build/source/opencv_contrib/cudaarithm/src/cuda/reduce.cu
       >             instantiation of "void <unnamed>::reduceToColumnImpl<T,S,D>(const cv::cuda::GpuMat &, cv::cuda::GpuMat &, int, cv::cuda::Stream &) [with T=uchar, S=double, D=double]" at line 232 of /build/source/opencv_contrib/cudaarithm/src/cuda/reduce.cu
       >
       > 36 errors detected in the compilation of "/build/source/opencv_contrib/cudaarithm/src/cuda/reduce.cu".
       > CMake Error at cuda_compile_1_generated_reduce.cu.o.Release.cmake:280 (message):
       >   Error generating file
       >   /build/source/build/modules/cudaarithm/CMakeFiles/cuda_compile_1.dir/src/cuda/./cuda_compile_1_generated_reduce.cu.o
       >
       > 
       > make[2]: *** [modules/cudaarithm/CMakeFiles/opencv_cudaarithm.dir/build.make:259: modules/cudaarithm/CMakeFiles/cuda_compile_1.dir/src/cuda/cuda_compile_1_generated_reduce.cu.o] Error 1
       > make[1]: *** [CMakeFiles/Makefile2:3788: modules/cudaarithm/CMakeFiles/opencv_cudaarithm.dir/all] Error 2
       > make[1]: *** Waiting for unfinished jobs....
       > [ 12%] Built target opencv_flann
       > [ 12%] Built target opencv_imgproc
       > make: *** [Makefile:166: all] Error 2
       For full logs, run 'nix log /nix/store/xf4nswmzx3jni7lcnvb4iv92jmfd6nkl-opencv-4.9.0.drv'.
/nix/store/z2rsczhx28qd2yak5zqf2w69i4pxply8-magma-2.6.2.drv
error: builder for '/nix/store/z2rsczhx28qd2yak5zqf2w69i4pxply8-magma-2.6.2.drv' failed with exit code 1;
       last 50 log lines:
       > /nix/store/7yab2py1qd0p4qqrcvcb3hqv7wnqn3c1-cuda_cudart-11.8.89-dev/include/driver_types.h:59:10: fatal error: crt/host_defines.h: No such file or directory
       >    59 | #include "crt/host_defines.h"
       >       |          ^~~~~~~~~~~~~~~~~~~~
       > compilation terminated.
       > [635/3376] Building CXX object CMakeFiles/magma.dir/src/zhetrf_nopiv.cpp.o
       > FAILED: CMakeFiles/magma.dir/src/zhetrf_nopiv.cpp.o
       > /nix/store/s8i61f6l2xr63kc1wfba5065wwdkayxa-gfortran-wrapper-13.2.0/bin/g++ -Dmagma_EXPORTS -I/build/magma-2.6.2/build/include -I/build/magma-2.6.2/include -I/build/magma-2.6.2/control -I/build/magma-2.6.2/magmablas -I/build/magma-2.6.2/sparse/include -I/build/magma-2.6.2/sparse/control -I/build/magma-2.6.2/testing -DADD_ -std=c++11 -fopenmp -Wall -Wno-unused-function -O3 -DNDEBUG -fPIC -MD -MT CMakeFiles/magma.dir/src/zhetrf_nopiv.cpp.o -MF CMakeFiles/magma.dir/src/zhetrf_nopiv.cpp.o.d -o CMakeFiles/magma.dir/src/zhetrf_nopiv.cpp.o -c /build/magma-2.6.2/src/zhetrf_nopiv.cpp
       > In file included from /nix/store/k29gv2mc1sm8fnnnd4wa16y6arr1nrla-libcublas-11.11.3.6-dev/include/cublas_api.h:72,
       >                  from /nix/store/k29gv2mc1sm8fnnnd4wa16y6arr1nrla-libcublas-11.11.3.6-dev/include/cublas_v2.h:65,
       >                  from /build/magma-2.6.2/include/magma_types.h:72,
       >                  from /build/magma-2.6.2/include/magma_copy.h:12,
       >                  from /build/magma-2.6.2/include/magmablas.h:12,
       >                  from /build/magma-2.6.2/include/magma_v2.h:22,
       >                  from /build/magma-2.6.2/control/magma_internal.h:63,
       >                  from /build/magma-2.6.2/src/zhetrf_nopiv.cpp:13:
       > /nix/store/7yab2py1qd0p4qqrcvcb3hqv7wnqn3c1-cuda_cudart-11.8.89-dev/include/driver_types.h:59:10: fatal error: crt/host_defines.h: No such file or directory
       >    59 | #include "crt/host_defines.h"
       >       |          ^~~~~~~~~~~~~~~~~~~~
       > compilation terminated.
       > [636/3376] Building CXX object CMakeFiles/magma.dir/src/zhetrf_nopiv_cpu.cpp.o
       > FAILED: CMakeFiles/magma.dir/src/zhetrf_nopiv_cpu.cpp.o
       > /nix/store/s8i61f6l2xr63kc1wfba5065wwdkayxa-gfortran-wrapper-13.2.0/bin/g++ -Dmagma_EXPORTS -I/build/magma-2.6.2/build/include -I/build/magma-2.6.2/include -I/build/magma-2.6.2/control -I/build/magma-2.6.2/magmablas -I/build/magma-2.6.2/sparse/include -I/build/magma-2.6.2/sparse/control -I/build/magma-2.6.2/testing -DADD_ -std=c++11 -fopenmp -Wall -Wno-unused-function -O3 -DNDEBUG -fPIC -MD -MT CMakeFiles/magma.dir/src/zhetrf_nopiv_cpu.cpp.o -MF CMakeFiles/magma.dir/src/zhetrf_nopiv_cpu.cpp.o.d -o CMakeFiles/magma.dir/src/zhetrf_nopiv_cpu.cpp.o -c /build/magma-2.6.2/src/zhetrf_nopiv_cpu.cpp
       > In file included from /nix/store/k29gv2mc1sm8fnnnd4wa16y6arr1nrla-libcublas-11.11.3.6-dev/include/cublas_api.h:72,
       >                  from /nix/store/k29gv2mc1sm8fnnnd4wa16y6arr1nrla-libcublas-11.11.3.6-dev/include/cublas_v2.h:65,
       >                  from /build/magma-2.6.2/include/magma_types.h:72,
       >                  from /build/magma-2.6.2/include/magma_copy.h:12,
       >                  from /build/magma-2.6.2/include/magmablas.h:12,
       >                  from /build/magma-2.6.2/include/magma_v2.h:22,
       >                  from /build/magma-2.6.2/control/magma_internal.h:63,
       >                  from /build/magma-2.6.2/src/zhetrf_nopiv_cpu.cpp:11:
       > /nix/store/7yab2py1qd0p4qqrcvcb3hqv7wnqn3c1-cuda_cudart-11.8.89-dev/include/driver_types.h:59:10: fatal error: crt/host_defines.h: No such file or directory
       >    59 | #include "crt/host_defines.h"
       >       |          ^~~~~~~~~~~~~~~~~~~~
       > compilation terminated.
       > [637/3376] Building CXX object CMakeFiles/magma.dir/src/zsytrf_nopiv_cpu.cpp.o
       > FAILED: CMakeFiles/magma.dir/src/zsytrf_nopiv_cpu.cpp.o
       > /nix/store/s8i61f6l2xr63kc1wfba5065wwdkayxa-gfortran-wrapper-13.2.0/bin/g++ -Dmagma_EXPORTS -I/build/magma-2.6.2/build/include -I/build/magma-2.6.2/include -I/build/magma-2.6.2/control -I/build/magma-2.6.2/magmablas -I/build/magma-2.6.2/sparse/include -I/build/magma-2.6.2/sparse/control -I/build/magma-2.6.2/testing -DADD_ -std=c++11 -fopenmp -Wall -Wno-unused-function -O3 -DNDEBUG -fPIC -MD -MT CMakeFiles/magma.dir/src/zsytrf_nopiv_cpu.cpp.o -MF CMakeFiles/magma.dir/src/zsytrf_nopiv_cpu.cpp.o.d -o CMakeFiles/magma.dir/src/zsytrf_nopiv_cpu.cpp.o -c /build/magma-2.6.2/src/zsytrf_nopiv_cpu.cpp
       > In file included from /nix/store/k29gv2mc1sm8fnnnd4wa16y6arr1nrla-libcublas-11.11.3.6-dev/include/cublas_api.h:72,
       >                  from /nix/store/k29gv2mc1sm8fnnnd4wa16y6arr1nrla-libcublas-11.11.3.6-dev/include/cublas_v2.h:65,
       >                  from /build/magma-2.6.2/include/magma_types.h:72,
       >                  from /build/magma-2.6.2/include/magma_copy.h:12,
       >                  from /build/magma-2.6.2/include/magmablas.h:12,
       >                  from /build/magma-2.6.2/include/magma_v2.h:22,
       >                  from /build/magma-2.6.2/control/magma_internal.h:63,
       >                  from /build/magma-2.6.2/src/zsytrf_nopiv_cpu.cpp:15:
       > /nix/store/7yab2py1qd0p4qqrcvcb3hqv7wnqn3c1-cuda_cudart-11.8.89-dev/include/driver_types.h:59:10: fatal error: crt/host_defines.h: No such file or directory
       >    59 | #include "crt/host_defines.h"
       >       |          ^~~~~~~~~~~~~~~~~~~~
       > compilation terminated.
       > ninja: build stopped: subcommand failed.
       For full logs, run 'nix log /nix/store/z2rsczhx28qd2yak5zqf2w69i4pxply8-magma-2.6.2.drv'.
/nix/store/iiyx04vk64xxhkr30f9phh68r3s586ba-nsight_compute-2024.1.1.4.drv
error: builder for '/nix/store/iiyx04vk64xxhkr30f9phh68r3s586ba-nsight_compute-2024.1.1.4.drv' failed with exit code 1;
       last 50 log lines:
       >     libssl.so.10 -> found: /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX
       >     libstdc++.so.6 -> found: /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/linux-desktop-glibc_2_11_3-x64
       >     libgcc_s.so.1 -> found: /nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib
       > setting RPATH to: /nix/store/nwsbnbj4xgfq2x45df9xg2b77953h2zz-curl-8.7.1/lib:/nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64:/nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX:/nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/linux-desktop-glibc_2_11_3-x64:/nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib:$ORIGIN
       > searching for dependencies of /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/lib/providers/libsysfs_provider.so
       >     libssl.so.10 -> found: /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX
       >     libcrypto.so.10 -> found: /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX
       >     libstdc++.so.6 -> found: /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/linux-desktop-glibc_2_11_3-x64
       >     libgcc_s.so.1 -> found: /nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib
       > setting RPATH to: /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX:/nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/linux-desktop-glibc_2_11_3-x64:/nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib:$ORIGIN
       > searching for dependencies of /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/lib/providers/libevents_switch_provider.so
       >     libibmad.so.5 -> not found!
       >     libibumad.so.3 -> not found!
       >     libibverbs.so.1 -> not found!
       >     libssl.so.10 -> found: /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX
       >     libcrypto.so.10 -> found: /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX
       >     libstdc++.so.6 -> found: /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/linux-desktop-glibc_2_11_3-x64
       >     libgcc_s.so.1 -> found: /nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib
       > setting RPATH to: /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX:/nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/linux-desktop-glibc_2_11_3-x64:/nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib:$ORIGIN
       > searching for dependencies of /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/python/packages/nsys_recipe/third_party/_sqlite3.cpython-310-x86_64-linux-gnu.so
       >     libsqlite3.so.0 -> found: /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/linux-desktop-glibc_2_11_3-x64
       > setting RPATH to: /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/linux-desktop-glibc_2_11_3-x64:$ORIGIN
       > setting interpreter of /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/python/bin/python
       > searching for dependencies of /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/python/bin/python
       > setting RPATH to: $ORIGIN
       > searching for dependencies of /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/extras/python/_ncu_report.so
       >     libgcc_s.so.1 -> found: /nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib
       > setting RPATH to: /nix/store/d8c0am4jck9188ajr9swx6kmcqrh8zkl-gcc-13.2.0-libgcc/lib:$ORIGIN
       > auto-patchelf: 19 dependencies could not be satisfied
       > error: auto-patchelf could not satisfy dependency libQt6WlShellIntegration.so.6 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/linux-desktop-glibc_2_11_3-x64/Plugins/wayland-shell-integration/libwl-shell-plugin.so
       > warn: auto-patchelf ignoring missing libcuda.so.1 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/nvgpucs
       > error: auto-patchelf could not satisfy dependency libibverbs.so.1 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/clx
       > error: auto-patchelf could not satisfy dependency libibumad.so.3 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/clx
       > error: auto-patchelf could not satisfy dependency libibmad.so.5 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/clx
       > error: auto-patchelf could not satisfy dependency libucp.so.0 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/clx
       > error: auto-patchelf could not satisfy dependency libuct.so.0 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/clx
       > error: auto-patchelf could not satisfy dependency libucs.so.0 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/clx
       > error: auto-patchelf could not satisfy dependency libucm.so.0 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/clx
       > error: auto-patchelf could not satisfy dependency libibmad.so.5 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/pmc
       > error: auto-patchelf could not satisfy dependency libibumad.so.3 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/pmc
       > error: auto-patchelf could not satisfy dependency libibverbs.so.1 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/pmc
       > error: auto-patchelf could not satisfy dependency libcom_err.so.2 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/libssl.so.10
       > error: auto-patchelf could not satisfy dependency libibverbs.so.1 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/libclx_api.so
       > error: auto-patchelf could not satisfy dependency libibumad.so.3 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/libclx_api.so
       > error: auto-patchelf could not satisfy dependency libibmad.so.5 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/libclx_api.so
       > error: auto-patchelf could not satisfy dependency libibmad.so.5 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/lib/providers/libevents_switch_provider.so
       > error: auto-patchelf could not satisfy dependency libibumad.so.3 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/lib/providers/libevents_switch_provider.so
       > error: auto-patchelf could not satisfy dependency libibverbs.so.1 wanted by /nix/store/c7kswpmyicww9ddsm56j5hkc3n8lvgcp-nsight_compute-2024.1.1.4/nsight-compute/2024.1.1/host/target-linux-x64/CollectX/lib/providers/libevents_switch_provider.so
       > auto-patchelf failed to find all the required dependencies.
       > Add the missing dependencies to --libs or use `--ignore-missing="foo.so.1 bar.so etc.so"`.
       For full logs, run 'nix log /nix/store/iiyx04vk64xxhkr30f9phh68r3s586ba-nsight_compute-2024.1.1.4.drv'.

@ConnorBaker ConnorBaker force-pushed the feat/cudaPackages-fixed-output-derivations branch from d00b56c to daa1c75 Compare May 9, 2024 21:38
@ConnorBaker ConnorBaker force-pushed the feat/cudaPackages-fixed-output-derivations branch 2 times, most recently from d18490a to d51c8d3 Compare May 15, 2024 15:43
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/overlaying-packages-using-cuda-to-use-zluda/45374/3

@ConnorBaker ConnorBaker force-pushed the feat/cudaPackages-fixed-output-derivations branch from dfe9568 to ee0b90c Compare May 18, 2024 12:02
@ConnorBaker ConnorBaker force-pushed the feat/cudaPackages-fixed-output-derivations branch from a993636 to dabbb63 Compare May 21, 2024 15:11
@ConnorBaker ConnorBaker force-pushed the feat/cudaPackages-fixed-output-derivations branch from 23ab722 to 92f3c30 Compare May 22, 2024 23:29
@ConnorBaker ConnorBaker force-pushed the feat/cudaPackages-fixed-output-derivations branch 2 times, most recently from f94e623 to cdc307e Compare May 24, 2024 04:44
@ConnorBaker
Copy link
Contributor Author

In a6ab7c8 I applied recurseIntoAttrs to all the CUDA package sets to verify that it did cause nixpkgs-review to build all of them.

But in doing that, I saw and caught a number of different bugs/breakages. That made me think -- maybe we should have recurseIntoAttrs on all of them?

Some initial thoughts:

Pros:

  • Find breakages we wouldn't otherwise see

Cons:

  • Many, many more packages to evaluate and large binaries to add to the store (though binary caches can help here).

@ConnorBaker ConnorBaker force-pushed the feat/cudaPackages-fixed-output-derivations branch 2 times, most recently from a70d25c to 9df641f Compare May 28, 2024 17:37
TODO-PR-306172.md Outdated Show resolved Hide resolved
@ConnorBaker ConnorBaker force-pushed the feat/cudaPackages-fixed-output-derivations branch from eaed9b8 to 71ebe67 Compare September 23, 2024 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment