Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buildDotnetModule: allow fetching nuget dependencies at compile time #314990

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

purepani
Copy link
Contributor

A (quick) attempt to rebase #162548

I have tested building jellyfin so far, but nothing else to see if it broke anything. I haven't had a chance to update the warning or docs, but i wanted to at least get nixpkgs-review running on the PR. I also haven't had a chance to see if the hash specification still works, but i wanted to at least get nixpkgs-review running on it to make sure nothing else broke.

Pinging people from the last PR:
@jonringer @SuperSandro2000 @winterqt @IvarWithoutBones @Mic92
Hope these are reasonable pings; sorry if not!

Also I'm not quite so sure why almost everyone's name was removed from authoring the pr? I just noticed that, so I'll see if I can fix that tomorrow. There's probably a bunch documentation stuff that needs to be updated as well.

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@github-actions github-actions bot added the 8.has: documentation This PR adds or changes documentation label May 27, 2024
, ... } @ args:

assert projectFile == null -> throw "Defining the `projectFile` attribute is required. This is usually an `.csproj`, or `.sln` file.";
assert (nugetDeps != null && nugetSha256 != null) -> throw "The attributes `nugetDeps` and 'nugetSha256' are mutually exclusive!";
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Triggers with buildDotnetGlobalTools since it passes through the same attr.

dotnetValidateLockfileHook = makeSetupHook
{
name = "dotnet-validate-lockfile-hook";
deps = [ jq ];
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Depreciated; replace with propagagedBuildInputs

@purepani
Copy link
Contributor Author

Result of nixpkgs-review pr 314990 run on x86_64-linux 1

61 packages built:
  • ArchiSteamFarm
  • BeatSaberModManager
  • alttpr-opentracker
  • am2rlauncher
  • audiobookshelf
  • avalonia-ilspy
  • bicep
  • boogie
  • btcpayserver
  • btcpayserver-altcoins
  • cavalier
  • celeste64
  • certdump
  • dafny
  • denaro
  • depotdownloader
  • discordchatexporter-cli
  • dotnet-outdated
  • eventstore
  • famistudio
  • formula
  • fsautocomplete
  • galaxy-buds-client
  • garnet
  • git-credential-manager
  • github-runner
  • ilspycmd
  • inklecate
  • jackett
  • jellyfin
  • kavita
  • libation
  • lubelogger
  • lumafly
  • marksman
  • mqttmultimeter
  • msbuild
  • naps2
  • nbxplorer
  • netcoredbg
  • omnisharp-roslyn
  • opentabletdriver
  • openutau
  • osu-lazer
  • pablodraw
  • parabolic
  • pinta
  • ps3-disc-dumper
  • pupdate
  • retrospy
  • roslyn
  • roslyn-ls
  • ryujinx
  • scarab
  • slskd
  • space-station-14-launcher
  • technitium-dns-server
  • tone
  • torrentstream
  • wasabibackend
  • xivlauncher

@Mic92
Copy link
Member

Mic92 commented May 27, 2024

Have you any package this new hash can be tested with? Maybe we can also remove the lock file for a package in nixpkgs for testing?

@purepani
Copy link
Contributor Author

I just tried it with jellyfin with no success; looks like some more work needs to be done.


# This is the last phase that runs before we error out about the hash being wrong
postFixup = lib.optionalString (nugetSha256 == null) ''
echo "Please set nugetSha256 to the hash below!"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New things should ideally only support hash

@superherointj superherointj added the 6.topic: dotnet Language: .NET label Jun 4, 2024
This allows user to specify a hash for the dependencies using
the "nugetSha256" attribute, instead of having to manually generate
a lockfile.

This is done by generating a lockfile with `dotnet restore`, and then
parsing the requested version ranges to see if anything is floating.

Afterwards we generate a nix-based lockfile containing hashes and stable
downloads for each dependency, which we can IFD. The checksum of this
lockfile is specified with "nugetSha256".

We want to use the checksum of the nix-based lockfile instead of hashing
the entire nuget source so that we can independantly fetch all
dependencies, and re-use them across derivations.

Co-authored-by: Valentin Gagarin <[email protected]>
@purepani purepani force-pushed the dotnet-module-fetch-impure-deps-new branch from e2c84cf to c84d0a1 Compare June 12, 2024 03:30
@purepani
Copy link
Contributor Author

purepani commented Jun 12, 2024

Have you any package this new hash can be tested with? Maybe we can also remove the lock file for a package in nixpkgs for testing?

Motivated to get this working again due to the recent ascii regression.

It is unclean, and probably needs to be looked at a lot closer to pull any more recent updates that have been created from the original pr, but it now works, successfully building jellyfin!
Is there a better name for nugetSha256 that I can use? It seems like that attribute is already taken by build-dotnet-global-tools I believe. For now I'll change it to lockfileSha256 just to get it the CI working.

@purepani purepani force-pushed the dotnet-module-fetch-impure-deps-new branch from c84d0a1 to 5987390 Compare June 12, 2024 03:52
Copy link
Contributor Author

@purepani purepani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still some updates needed, particularly documentation

@@ -43,7 +54,7 @@ EOF
find -name paket.dependencies -exec sed -i 's+source .*+source @nugetSource@/lib+g' {} \;
find -name paket.lock -exec sed -i 's+remote:.*+remote: @nugetSource@/lib+g' {} \;

dotnet tool restore --add-source "@nugetSource@/lib"
#dotnet tool restore --add-source "@nugetSource@/lib"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commented out due to making a network call; I believe that telemetry will just need to be disabled with export DOTNET_CLI_TELEMETRY_OPTOUT=1 now that I think about it.

@@ -136,6 +136,7 @@ mkCommon type rec {

passthru = {
inherit icu;
packages_list = packages;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wasn't sure how to get the list of projects so I just readded this since the previous pr did that. If there's a better way do get the packages already retrieved from dotnet, then we can change this.

@purepani
Copy link
Contributor Author

purepani commented Jun 12, 2024

Oh so I guess I didn't notice that the original used IFD(Was it not banned back then?). Next step is to remove that.

@purepani
Copy link
Contributor Author

Should we wait for NuGet/Home#13288 to be considered before implementing something like this?

@wegank wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Jul 4, 2024
@wegank wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.status: merge conflict This PR has merge conflicts with the target branch 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 6.topic: dotnet Language: .NET 8.has: documentation This PR adds or changes documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants