buildDotnetModule: allow fetching nuget dependencies at compile time

doc/languages-frameworks/dotnet.section.md

@@ -2,7 +2,7 @@
 
 ## Local Development Workflow {#local-development-workflow}
 
-For local development, it's recommended to use nix-shell to create a dotnet environment:
+For local development, it's recommended to use `nix-shell` to create a dotnet environment:
 
 ```nix
 # shell.nix
@@ -93,11 +93,11 @@ The `dotnetCorePackages.sdk` contains both a runtime and the full sdk of a given
 To package Dotnet applications, you can use `buildDotnetModule`. This has similar arguments to `stdenv.mkDerivation`, with the following additions:
 
 * `projectFile` is used for specifying the dotnet project file, relative to the source root. These have `.sln` (entire solution) or `.csproj` (single project) file extensions. This can be a list of multiple projects as well. When omitted, will attempt to find and build the solution (`.sln`). If running into problems, make sure to set it to a file (or a list of files) with the `.csproj` extension - building applications as entire solutions is not fully supported by the .NET CLI.
-* `nugetDeps` takes either a path to a `deps.nix` file, or a derivation. The `deps.nix` file can be generated using the script attached to `passthru.fetch-deps`. If the argument is a derivation, it will be used directly and assume it has the same output as `mkNugetDeps`.
+* `nugetDeps` takes either a path to a `deps.nix` file, or a derivation, in case we cannot deterministically do so. The `deps.nix` file can be generated using the script attached to `passthru.fetch-deps`. If the argument is a derivation, it will be used directly and assume it has the same output as `mkNugetDeps`. This option is mutually exclusive with the `nugetSha256` attribute.
 ::: {.note}
 For more detail about managing the `deps.nix` file, see [Generating and updating NuGet dependencies](#generating-and-updating-nuget-dependencies)
 :::
-
+* `nugetSha256` is used to specify the hash of the generated Nix-compatible lockfile containing all NuGet dependencies.
 * `packNupkg` is used to pack project as a `nupkg`, and installs it to `$out/share`. If set to `true`, the derivation can be used as a dependency for another dotnet project by adding it to `projectReferences`.
 * `projectReferences` can be used to resolve `ProjectReference` project items. Referenced projects can be packed with `buildDotnetModule` by setting the `packNupkg = true` attribute and passing a list of derivations to `projectReferences`. Since we are sharing referenced projects as NuGets they must be added to csproj/fsproj files as `PackageReference` as well.
  For example, your project has a local dependency:
@@ -126,7 +126,15 @@ For more detail about managing the `deps.nix` file, see [Generating and updating
 * `dotnetPackFlags` can be used to pass flags to `dotnet pack`. Used only if `packNupkg` is set to `true`.
 * `dotnetFlags` can be used to pass flags to all of the above phases.
 
-When packaging a new application, you need to fetch its dependencies. Create an empty `deps.nix`, set `nugetDeps = ./deps.nix`, then run `nix-build -A package.fetch-deps` to generate a script that will build the lockfile for you.
+When packaging a new application, you need to fetch and lock its Nuget dependencies. `buildDotnetModule` has two ways of handling this:
+* Using `nugetSha256`. We try to generate a Nix-compatible lockfile at build time, of which you can specify the hash with `nugetSha256`. When you try to build the derivation, everything gets fetched automatically and the expected hash gets printed out. We try this by default when neither `nugetSha256` nor `nugetDeps` are set. This is the recommended way to resolve dependencies. The passthru attribute `nuget-lockfile` is exposed which builds this lockfile for you. The downside of this method is that it only works if no floating versions are requested upstream – otherwise an error will be thrown and you will need to use `nugetDeps` instead.
+* Using `nugetDeps`. This method requires you to manually generate a Nix-compatible lockfile, which has to be checked into source control. You can run `nix-build -A package.fetch-deps` to build a script which generates the lockfile. After running said script you can set the `nugetDeps` attribute to the path of the generated lockfile. Note that if you change any flags affecting the restore process, you will need to regenerate this script. This takes more work to maintain, so you should use `nugetSha256` instead when possible.
+
+Note that both of these methods use the flags and attributes from your derivation, and attempt to mirror the restore process from the package itself. That means that if you for example set `dotnetRestoreFlags`, it will be used both for fetching dependencies and for building the derivation.
+
+If you change the `dotnet-sdk` attribute, or set a flag effecting `dotnet restore`, you will most likely need to regenerate the lockfile.
+
+When updating an existing application, you should *always* regenerate its lockfile.
 
 Here is an example `default.nix`, using some of the previously discussed arguments:
 ```nix
@@ -141,21 +149,22 @@ in buildDotnetModule rec {
   src = ./.;
 
   projectFile = "src/project.sln";
-  # File generated with `nix-build -A package.passthru.fetch-deps`.
-  # To run fetch-deps when this file does not yet exist, set nugetDeps to null
-  nugetDeps = ./deps.nix;
+
+  nugetSha256 = "sha256-8FBfEhXmnfk/Y2Cd5jWoKgSS/HbEJyBXV5jOTyR+ft8="; # Generated automatically.
+  nugetDeps = ./deps.nix; # File generated with `nix-build -A package.passthru.fetch-deps`, this is *not* recommanded.
 
   projectReferences = [ referencedProject ]; # `referencedProject` must contain `nupkg` in the folder structure.
 
-  dotnet-sdk = dotnetCorePackages.sdk_6_0;
-  dotnet-runtime = dotnetCorePackages.runtime_6_0;
+  dotnet-sdk = dotnetCorePackages.sdk_7_0;
+  dotnet-runtime = dotnetCorePackages.runtime_7_0;
+  dotnetFlags = [ "--runtime linux-x64" ]; # Note: this usually is not required.
 
   executables = [ "foo" ]; # This wraps "$out/lib/$pname/foo" to `$out/bin/foo`.
   executables = []; # Don't install any executables.
 
   packNupkg = true; # This packs the project as "foo-0.1.nupkg" at `$out/share`.
 
-  runtimeDeps = [ ffmpeg ]; # This will wrap ffmpeg's library path into `LD_LIBRARY_PATH`.
+  runtimeDeps = [ ffmpeg ]; # This will wrap ffmpeg into `LD_LIBRARY_PATH`.
 }
 ```
 

pkgs/build-support/dotnet/build-dotnet-module/default.nix

@@ -5,20 +5,24 @@
 , srcOnly
 , linkFarmFromDrvs
 , symlinkJoin
+, nuget-to-nix
+, writeText
 , makeWrapper
 , dotnetCorePackages
 , mkNugetSource
 , mkNugetDeps
-, nuget-to-nix
 , cacert
 , coreutils
 , runtimeShellPackage
 }:
 
 { name ? "${args.pname}-${args.version}"
 , pname ? name
+, meta ? { platforms = dotnet-sdk.meta.platforms; maintainers = []; }
+
 , enableParallelBuilding ? true
 , doCheck ? false
+
   # Flags to pass to `makeWrapper`. This is done to avoid double wrapping.
 , makeWrapperArgs ? [ ]
 
@@ -64,6 +68,13 @@
   # platforms in meta.platforms which are supported by the sdk.
 , runtimeId ? null
 
+# The hash of the generated nix-based lockfile containing all Nuget dependencies. This depends on no floating version ranges being requested upstream, as we cannot deterministically fetch those.
+# If any floating version is requested, an error will be thrown. Note that this option is mutually exclusive with the `nugetDeps` attribute.
+, lockfileSha256 ? null
+# Don't expose any Nuget source to `dotnet restore`.
+, dontSetNugetSource ? false
+
+
   # Tests to disable. This gets passed to `dotnet test --filter "FullyQualifiedName!={}"`, to ensure compatibility with all frameworks.
   # See https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test#filter-option-details for more details.
 , disabledTests ? [ ]
@@ -78,43 +89,52 @@
   # Whether to use an alternative wrapper, that executes the application DLL using the dotnet runtime from the user environment. `dotnet-runtime` is provided as a default in case no .NET is installed
   # This is useful for .NET tools and applications that may need to run under different .NET runtimes
 , useDotnetFromEnv ? false
-  # Whether to explicitly enable UseAppHost when building. This is redundant if useDotnetFromEnv is enabledz
+  # Whether to explicitly enable UseAppHost when building. This is redundant if useDotnetFromEnv is enabled.
 , useAppHost ? true
   # The dotnet SDK to use.
 , dotnet-sdk ? dotnetCorePackages.sdk_6_0
   # The dotnet runtime to use.
 , dotnet-runtime ? dotnetCorePackages.runtime_6_0
-, ...
-} @ args:
+
+, ... } @ args:
+
+#assert projectFile == null -> throw "Defining the `projectFile` attribute is required. This is usually an `.csproj`, or `.sln` file.";
+assert (nugetDeps != null && lockfileSha256 != null) -> throw "The attributes `nugetDeps` and 'lockfileSha256' are mutually exclusive!";
 
 let
   platforms =
     if args ? meta.platforms
     then lib.intersectLists args.meta.platforms dotnet-sdk.meta.platforms
     else dotnet-sdk.meta.platforms;
+  # Nuget packages provided by the SDK, these get excluded from the generated lockfile
+  sdkExclusions = writeText "${dotnet-sdk.name}-exclusions" (lib.concatStringsSep "\n" (dotnet-sdk.passthru.packages_list { fetchNuGet = attrs: attrs.pname; }));
 
   inherit (callPackage ./hooks {
-    inherit dotnet-sdk disabledTests nuget-source dotnet-runtime runtimeDeps buildType;
+    inherit dotnet-sdk disabledTests nuget-source dotnet-runtime runtimeDeps buildType dontSetNugetSource;
     runtimeId =
       if runtimeId != null
       then runtimeId
       else dotnetCorePackages.systemToDotnetRid stdenvNoCC.targetPlatform.system;
-  }) dotnetConfigureHook dotnetBuildHook dotnetCheckHook dotnetInstallHook dotnetFixupHook;
+  }) dotnetConfigureHook dotnetValidateLockfileHook dotnetBuildHook dotnetCheckHook dotnetInstallHook dotnetFixupHook;
+
+  fetchImpureDeps = callPackage ./fetch-deps.nix {
+    inherit name meta dotnet-sdk lockfileSha256 projectFile testProjectFile dotnetFlags dotnetRestoreFlags enableParallelBuilding dotnetValidateLockfileHook sdkExclusions;
+    src = srcOnly args;
+  };
 
   localDeps =
     if (projectReferences != [ ])
     then linkFarmFromDrvs "${name}-project-references" projectReferences
     else null;
 
   _nugetDeps =
-    if (nugetDeps != null) then
-      if lib.isDerivation nugetDeps
+    let
+      importedDeps = if (nugetDeps == null)
+        then fetchImpureDeps
+        else nugetDeps;
+    in if lib.isDerivation nugetDeps
       then nugetDeps
-      else mkNugetDeps {
-        inherit name;
-        sourceFile = nugetDeps;
-      }
-    else throw "Defining the `nugetDeps` attribute is required, as to lock the NuGet dependencies. This file can be generated by running the `passthru.fetch-deps` script.";
+      else mkNugetDeps { inherit name; nugetDeps = import importedDeps; };
 
   # contains the actual package dependencies
   dependenciesSource = mkNugetSource {
@@ -188,6 +208,7 @@ stdenvNoCC.mkDerivation (args // {
   propagatedSandboxProfile = toString dotnet-runtime.__propagatedSandboxProfile;
 
   passthru = {
+    nuget-lockfile = fetchImpureDeps;
     inherit nuget-source;
   } // lib.optionalAttrs (!lib.isDerivation nugetDeps) {
     fetch-deps =
@@ -209,7 +230,6 @@ stdenvNoCC.mkDerivation (args // {
       in
       writeShellScript "fetch-${pname}-deps" ''
         set -euo pipefail
-
         export PATH="${lib.makeBinPath [ coreutils runtimeShellPackage dotnet-sdk (nuget-to-nix.override { inherit dotnet-sdk; }) ]}"
 
         for arg in "$@"; do

pkgs/build-support/dotnet/build-dotnet-module/fetch-deps.nix

@@ -0,0 +1,76 @@
+{ lib
+, buildDotnetModule
+, dotnetValidateLockfileHook
+, nuget-to-nix
+, lockfileSha256
+
+, src
+, name
+, meta
+, dotnet-sdk
+, projectFile
+, testProjectFile
+, dotnetFlags
+, dotnetRestoreFlags
+, enableParallelBuilding
+, sdkExclusions
+} @ args:
+
+buildDotnetModule rec {
+  name = "${args.name}-nuget-lockfile";
+
+  inherit src dotnet-sdk projectFile testProjectFile dotnetFlags dotnetRestoreFlags enableParallelBuilding;
+
+  nativeBuildInputs = [
+    dotnetValidateLockfileHook
+    (nuget-to-nix.override {inherit dotnet-sdk;})
+    ];
+
+  generateLockfile = true;
+  dontSetNugetSource = true;
+  dontDotnetFixup = true;
+
+  impureEnvVars = lib.fetchers.proxyImpureEnvVars;
+  outputHashAlgo = "sha256";
+  outputHashMode = "flat";
+  outputHash = if (lockfileSha256 != null)
+    then lockfileSha256
+    else ""; # This needs to be set for networking, an empty string prints the "empty hash found" warning
+
+  preConfigure = ''
+    dotnetRestoreFlags+=(
+      --packages "$HOME/nuget-pkgs"
+    )
+
+    echo "Evaluating and fetching Nuget dependencies"
+  '';
+
+  buildPhase = ''
+    runHook preBuild
+
+    NUGET_DEPS="$HOME/deps.nix"
+    echo "Writing lockfile..."
+    nuget-to-nix "$HOME/nuget-pkgs" "${sdkExclusions}" > "$NUGET_DEPS"
+
+    runHook postBuild
+  '';
+
+  installPhase = ''
+    runHook preInstall
+
+    cp "$NUGET_DEPS" $out
+    echo "Installed lockfile to: $out"
+
+    runHook postInstall
+  '';
+
+  # This is the last phase that runs before we error out about the hash being wrong
+  postFixup = lib.optionalString (lockfileSha256 == null) ''
+    echo "Please set lockfileSha256 to the hash below!"
+  '';
+
+  meta = {
+    description = "A lockfile containing the Nuget dependencies for ${name}";
+    inherit (args.meta) maintainers platforms;
+  };
+}

pkgs/build-support/dotnet/build-dotnet-module/hooks/default.nix

@@ -7,13 +7,15 @@
 , callPackage
 , makeSetupHook
 , makeWrapper
+, jq
 , dotnet-sdk
 , disabledTests
 , nuget-source
 , dotnet-runtime
 , runtimeDeps
 , buildType
 , runtimeId
+, dontSetNugetSource
 }:
 assert (builtins.isString runtimeId);
 
@@ -24,8 +26,9 @@ in
   dotnetConfigureHook = makeSetupHook
     {
       name = "dotnet-configure-hook";
+      propagatedBuildInputs = [ dotnet-sdk ];
       substitutions = {
-        nugetSource = nuget-source;
+        nugetSource = lib.optional (!dontSetNugetSource) nuget-source;
         dynamicLinker = "${stdenv.cc}/nix-support/dynamic-linker";
         libPath = lib.makeLibraryPath [
           stdenv.cc.cc.lib
@@ -39,6 +42,12 @@ in
     }
     ./dotnet-configure-hook.sh;
 
+  dotnetValidateLockfileHook =  makeSetupHook
+    {
+      name = "dotnet-validate-lockfile-hook";
+      propagatedBuildInputs = [ jq ];
+    } ./dotnet-validate-lockfile.sh;
+
   dotnetBuildHook = makeSetupHook
     {
       name = "dotnet-build-hook";

pkgs/build-support/dotnet/build-dotnet-module/hooks/dotnet-build-hook.sh

@@ -46,6 +46,7 @@ dotnetBuildHook() {
             -p:BuildInParallel=$parallelBuildFlag \
             -p:ContinuousIntegrationBuild=true \
             -p:Deterministic=true \
+            -p:UseAppHost=true \
             --configuration "@buildType@" \
             --no-restore \
             ${versionFlags[@]} \

pkgs/build-support/dotnet/build-dotnet-module/hooks/dotnet-configure-hook.sh

@@ -13,16 +13,27 @@ dotnetConfigureHook() {
         local -r parallelFlag="--disable-parallel"
     fi
 
+    if [ -z "${dontSetNugetSource-}" ]; then
+        nugetSourceFlag="--source @nugetSource@/lib"
+    fi
+
+    if [ "${generateLockfile-}" ]; then
+        lockfilePath="$HOME/lockfile.json"
+        lockfileFlag="--use-lock-file --lock-file-path ${lockfilePath}"
+    fi
+
     dotnetRestore() {
         local -r project="${1-}"
         dotnet restore ${project-} \
             -p:ContinuousIntegrationBuild=true \
             -p:Deterministic=true \
             --runtime "@runtimeId@" \
-            --source "@nugetSource@/lib" \
+            ${nugetSourceFlag-} \
+            ${lockfileFlag-} \
             ${parallelFlag-} \
             ${dotnetRestoreFlags[@]} \
             ${dotnetFlags[@]}
+
     }
 
     # Generate a NuGet.config file to make sure everything,
@@ -43,7 +54,7 @@ EOF
     find -name paket.dependencies -exec sed -i 's+source .*+source @nugetSource@/lib+g' {} \;
     find -name paket.lock -exec sed -i 's+remote:.*+remote: @nugetSource@/lib+g' {} \;
 
-    dotnet tool restore --add-source "@nugetSource@/lib"
+    #dotnet tool restore --add-source "@nugetSource@/lib"
 
     (( "${#projectFile[@]}" == 0 )) && dotnetRestore