ci.nix isBuildable fails on broken in-tree root packages

[drewrisinger]

I'm running into an issue in drewrisinger#133. Basically, I have a default.nix similar to:

{ pkgs ? import <nixpkgs> {}}:

with pkgs;
{ python3Packages = recurseIntoAttrs rec {
    rootpkg = python3.buildPythonPackage {
      # ...
      meta.broken = lib.versionOlder lib.version "21.05";
    };
    leafpkg = python3.buildPythonPackage {
      # ...
      propagatedBuildInputs = [ rootpkg ];
    };
  };
}

When I run nix-build -A cacheOutputs ci.nix, I always get an error b/c it's trying to build leafpkg (which has evidently made it into cacheOutputs because it's not broken itself), even though rootpkg is obviously broken. I think the issue is the isBuildable check isn't comprehensive enough (i.e. it doesn't check dependencies).

Proposed solutions:

• modify isBuildable to do a recursive dependency check, and return false if any dependencies are broken. I considered this, but I'm struggling to come up with a better expression than

  nur-packages-template/ci.nix

  Line 18 in cc8c782

  isBuildable = p: !(p.meta.broken or false) && p.meta.license.free or true;

  . I assume I need to come up with some sort of folding and recursive dependency check.
• Mark leafpkg (in my case, about 10ish packages) as broken. Obviously a simpler solution, but it seems silly to do this when a future update could fix rootpkg and then you need to manually unmark all leaf packages.

[Mic92]

Also not sure how to preform this check efficiently. @infinisil any ideas?

[infinisil]

Hydra solves this by actually trying to instantiate all packages. If they fail to evaluate, it logs the error and continues to the next one.

CI could make use of this too by using the hydra-eval-jobs command, which takes a hydra-compatible release.nix and returns a json of all the derivations that successfully instantiated or inspectable failures.

[fgaz]

We could use tryEval. Something like

isBuildable = p: (builtins.tryEval (builtins.deepSeq p p)).success;

But this, like the hydra script, catches every evaluation error. Is this something we want? Is the job of ci catching eval errors too or should it only build the packages?

edit: nevermind the code above, deepseq is too much

[Mic92]

We also have https://github.com/nix-community/nix-eval-jobs now, which is derived from hydra's code. It could do the instantiation part

[drewrisinger]

Here's the current solution, that I implemented today thanks to the above helpful comments:

1. Change the GitHub CI workflow to generate an evaluation using nix-eval-jobs. (nix-eval-jobs --option restrict-eval true default.nix > evaluations.json).
2. Read the JSON (-like, b/c the line-delineated format is not exactly JSON) in ci.nix, then use the attributes when checking if the package is buildable.

evaluatedPackages = builtins.map builtins.fromJSON (pkgs.lib.splitString "\n" (pkgs.lib.fileContents ./evaluations.json));

canEvaluate = p: elem: ( elem ? "attr" && (pkgs.lib.hasInfix p.pname elem.attr) && (! (elem ? "error")));
isBuildable = p: any (canEvaluate p) evaluatedPackages;

Feedback would be appreciated, I can submit a PR if people are interested. I tried to make a derivation in ci.nix to do step 1 above, but I was running into sandboxing issues when trying to pass the current nixpkgs for evaluation.

[Mic92]

Here's the current solution, that I implemented today thanks to the above helpful comments:

1. Change the GitHub CI workflow to generate an evaluation using `nix-eval-jobs`. (`nix-eval-jobs --option restrict-eval true default.nix > evaluations.json`).

2. Read the JSON (-like, b/c the line-delineated format is not exactly JSON) in `ci.nix`, then use the attributes when checking if the package is buildable.

evaluatedPackages = builtins.map builtins.fromJSON (pkgs.lib.splitString "\n" (pkgs.lib.fileContents ./evaluations.json));

canEvaluate = p: elem: ( elem ? "attr" && (pkgs.lib.hasInfix p.pname elem.attr) && (! (elem ? "error")));
isBuildable = p: any (canEvaluate p) evaluatedPackages;

Feedback would be appreciated, I can submit a PR if people are interested. I tried to make a derivation in ci.nix to do step 1 above, but I was running into sandboxing issues when trying to pass the current nixpkgs for evaluation.

Sounds like a good first step also I think we could use the the json-line output to stream into a ci runner and hence already start building while the rest is still evaluating:

nix-eval-jobs --option restrict-eval true default.nix | xargs --no-run-if-empty -n1 -P$(nproc) nix-build -E '$some expression'

The only thing we are missing from nix-build-uncached is that it will also build things that are already in the cache.
On the upside however package builds can fail individually without stopping the build of the working once.
@adisbladis how did you end up using the output of nix-eval-jobs?

[Mic92]

nix-eval-jobs can now check the cache state of a derivation like nix-build-uncached did before.