[WIP] Set absolute sonames

[lheckemann]

Motivation for this change

#24844

Most stuff I've tried building with this works. However, nixos won't build with this yet because the extra-tools initrd stuff doesn't build successfully. I'm getting the… interesting error message

Inconsistency detected by ld.so: dl-version.c: 205: _dl_check_map_versions: Assertion `needed != NULL' failed!

when it tries to test the udevadm in extra-utils. Help would be appreciated!

Also, thanks very much to @grahamc for having made the aarch64 community builder available. It's been invaluable for testing this, what with all the gcc rebuilds!

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Fits CONTRIBUTING.md.

[Mic92]

Can you add this option also in doc/stdenv.xml?

[lheckemann]

Done!

[lheckemann]

This should also only run on linux, I just realised. Not sure how exactly to do this though — it should probably be in pkgs/stdenv/linux somewhere rather than pkgs/stdenv/generic/default.nix and guarded by isLinux?

[Ericson2314]

How close does this get us to being able to remove the --rpath stuff from cc-wrapper?

[lheckemann]

Not much at all. This change doesn't affect dlopen.

[Mic92]

I suppose we will uncover bugs, where we implicitly covered dlopen with our rpath? In any case we would also be closer to the macOS port so we have the same behavior on both platforms, which is a good thing.

[dezgeg]

I presume this breaks LD_LIBRARY_PATH?

IIRC at some point I tried a similar thing which instead of making the DT_SONAME directly absolute, it would only patchelf all DT_NEEDED to absolute paths. I think changing DT_SONAME itself makes me slightly more nervous given that AFAICT DT_SONAME is sometimes used as some sort of 'cookie' (e.g. if you try to load both /nix/store/foo/libsomething.so and /nix/store/bar/libsomething.so, if the both have the same DT_SONAME only one of them will be loaded).

But, that solution breaks LD_LIBRARY_PATH just as well. So nowadays I think the easiest way to solve the O(n^2) lookup problem would be to cache the contents of directories in RPATH in the dynamic linker.

[Mic92]

When a library was found in the dependency closure and its soname is replaced by an absolute path in the binary, why would we still need LD_LIBRARY_PATH for the same library?

[lheckemann]

#24844 (comment)

We can still use LD_PRELOAD when necessary.

[vcunat]

Fixed the GitHub UI.

[dtzWill]

This seems to rebase pretty easily, but may need to be updated for non-conflict reasons such as marking other gcc versions noAbsoluteSoname (why do these need special handling? do other compilers or runtimes?)

Sure would be great to have this, it's silly for nix to know precisely what libraries to use but every program does a (depending on how you count, n^2) search to rediscover this information.

Anyway I probably don't need to argue for this to the folks on this PR already anyway haha but just a reminder :D.

[ambrop72]

I'm interested in getting this further. What is the current status? From what I understand it needs to be rebased and there is a problem with the initrd build.

Libraries that may need to be overridden (e.g. libGL for drivers that don't support libglvnd) can still be linked the old way (normal soname and directory in RUNPATH). I also think that the same should be done for libc because having more than one instance of libc loaded supposedly is not supported and does not work.

[lheckemann]

@ambrop72 I've started to think that this isn't actually a good approach, and patching the NEEDED entries may be better. I'm still unsure though.

[stale]

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

[Mic92]

@lheckemann do you still plan to finish this?

[lheckemann]

I definitely still want this to happen — if nobody else beats me to it, I'll have a look at reworking this to patch DT_NEEDED entries At Some Point™.

[Profpatsch]

Would love to see us putting absolute paths into `DT_NEEDED`, that looks like the cleanest solution to me.

…

On Tue, Jun 2, 2020 at 12:15 PM Linus Heckemann ***@***.***> wrote: I definitely still want this to happen — if nobody else beats me to it, I'll have a look at reworking this to patch DT_NEEDED entries At Some Point™. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#45105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAYB5ZULC7EZIKZKFVSH5VDRUTGKLANCNFSM4FP5XAXQ> .

[stale]

I marked this as stale due to inactivity. → More info

[lheckemann]

NixOS/patchelf#357 implements setting absolute NEEDED entries, and seems to be quite promising.