Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

jemalloc (jemallocator) C toolchain / compilation issues #370494

Open
dpc opened this issue Jan 3, 2025 · 6 comments · May be fixed by tikv/jemallocator#116
Open

jemalloc (jemallocator) C toolchain / compilation issues #370494

dpc opened this issue Jan 3, 2025 · 6 comments · May be fixed by tikv/jemallocator#116
Labels
0.kind: bug Something is broken

Comments

@dpc
Copy link
Contributor

dpc commented Jan 3, 2025

Describe the bug

It seems like Rust tikv-jemalloc-sys crate fails on compilation of vendored C jemalloc code,
but only on NixOS, only with newer C toolchains and only when optimizations are NOT enabled.

The developers investigating it point out to odd behavior of the toolchains suggesting some misconfiguration/misbehavior. I have been experiencing it with clang, some other user with gcc, but both of us are on NixOS, and problem doesn't repro on other distros as far as we can tell. (Edit: it does repro on other distros if Nix toolchain is used e.g. from a nix develop shell).

Steps To Reproduce

Compile a Rust project using tikv-jemalloc crate in debug mode on NixOS.

Expected behavior

It should compile.

Screenshots

NA

Additional context

NA

Metadata

NA

Notify maintainers


Note for maintainers: Please tag this issue in your PR.


Add a 👍 reaction to issues you find important.

@jr1221
Copy link

jr1221 commented Jan 3, 2025

It looks like systemd carries a patch to build on musl, could be related:

(musl-patches + "/0012-distinguish-XSI-compliant-strerror_r-from-GNU-specif.patch")

@bkchr
Copy link
Contributor

bkchr commented Jan 6, 2025

#370498 this looks related.

@anoadragon453
Copy link
Member

anoadragon453 commented Jan 17, 2025

This issue unfortunately still occurs after #370881 (which addressed #370498) merged and hit nixos-unstable.

@anoadragon453
Copy link
Member

I was able to work around this for now by ignoring the error when compiling:

env.CFLAGS = "-Wno-error=int-conversion";

@bkchr
Copy link
Contributor

bkchr commented Jan 20, 2025

have been experiencing it with clang, some other user with gcc, but both of us are on NixOS, and problem doesn't repro on other distros as far as we can tell.

BTW, locally I could fix the issue by reverting the default gcc to gcc13.

@th7nder
Copy link
Contributor

th7nder commented Jan 30, 2025

Best (in my opinion) workaround

is setting CFLAGS=-DJEMALLOC_STRERROR_R_RETURNS_CHAR_WITH_GNU_SOURCE.

    devShells.default = mkShell {
      inherit buildInputs;

      # [...]
      CFLAGS = "-DJEMALLOC_STRERROR_R_RETURNS_CHAR_WITH_GNU_SOURCE";
    };

I think it's the best one, as it's specific to this issue and doesn't break/cause other functionalities.

Other possible ways:

  • @anoadragon453's env.CFLAGS = "-Wno-error=int-conversion"; - I think it's not ideal, because it disregards a warning which actually has it merits. The warning can cause an undefined behaviour - wrong codepath is used and it erases error message.
  • @bkchr's downgrade to gcc13, I didn't test.

Real solution

Something is setting the _FORTIFY_SOURCE and it causes the test compilation to fail, we can:

  • understand why _FORTIFY_SOURCE is there, maybe get rid of it _FORTIFY_SOURCE somehow
  • force CFLAGS=-O1 even when we're running a debug build in the build.rs. (btw. there might be a bug in there as it is setting LDFLAGS=CFLAGS, which causes a lot of weird stuff when looking at the config.log of the tikv-jemallocator-build).
  • maybe don't pass those CFLAGS in the jemalloc-sys/build.rs at all? Not sure whether we should compile dependency with the same flags as cargo build in the debug mode. We'd let the autoconfigure figure out flags on its own, so not passing -O0 (disabling optimization flag) and it'd compile even with _FORTIFY_SOURCE, no matter the debug or release.

_FORTIFY_SOURCE.

Looks like nix gcc wrapper,

else remove "pie" knownHardeningFlags);
sets _FORTIFY_SOURCE by default.
When I run it on the latest nixos-unstable nixpkgs environment:

th7nder@stormheim ~/w/e/jemallocator (main) [0|1]> gcc -dM -E - < /dev/null | grep FORTIFY
#define _FORTIFY_SOURCE 3

You could disable this by specifying in your flake.nix:

hardeningDisable = [ "fortify" ];

But I think disabling it at a nix level, is not a solution, because we'd disable fortify on everything that's compiled.
It's essentially a problem with how the underlying C binary is build when building the crate.

So if we wanted to keep building debug jemalloc when building debug crate tikv-jemalloc-sys with nix-wrapped gcc, we could detect it and add -U_FORTIFY_SOURCE. It'd effectively get rid of this define, because it won't work if the crate is compiled without optimizations.
It's alternative to not passing the debug CFLAGS at all.

What do you think guys?
@BusyJay

Deeper

In the current nix:

  1. Something is setting _FORTIFY_SOURCE
  2. It causes a warning, because detection program for strerror is compiled with-O0 flag
  3. Because -Wall, warning is an error and compilation of a test program (see point 3 in explanation) fails.
  4. It does not set the flag JEMALLOC_STRERROR_R_RETURNS_CHAR_WITH_GNU_SOURCE) which causes the next warning about the conversion to int in the actual code, and again, compilation fails.

The Deepest

  1. jemalloc adapts to different versions of strerror_r based on the defs of _GNU_SOURCE and JEMALLOC_STRERROR_R_RETURNS_CHAR_WITH_GNU_SOURCE.
  2. I was looking through the compilation logs in nix and _GNU_SOURCE is set, so only is not set JEMALLOC_STRERROR_R_RETURNS_CHAR_WITH_GNU_SOURCE.
  3. jemalloc autoconfigure sets this flags based on whether a program compiles.
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
], [
  char *buffer = (char *) malloc(100);
  char *error = strerror_r(EINVAL, buffer, 100);
  printf("%s\n", error);
  1. In local non-nix compilation (basically clone https://github.com/tikv/jemallocator and cargo -vv build with GCC 11, looking at the config.log. It passes:
configure:14110: checking whether strerror_r returns char with gnu source is compilable
configure:14135: cc -o conftest -std=gnu11 -Wall -Wextra -Wsign-compare -Wundef -Wno-format-zero-length -Wpointer-arith -Wno-missing-braces -Wno-missing-field-initializers -Wno-missing-attributes -pipe -g3 -fvisibility=hidden -Wimplicit-fallthrough -O3 -funroll-loops -D_GNU_SOURCE -Werror -herror_on_warning -O0 -ffunction-sections -fdata-sections -fPIC -gdwarf-4 -fno-omit-frame-pointer -m64 -Wall -O0 -ffunction-sections -fdata-sections -fPIC -gdwarf-4 -fno-omit-frame-pointer -m64 -Wall -D_GNU_SOURCE -D_REENTRANT -O0 -ffunction-sections -fdata-sections -fPIC -gdwarf-4 -fno-omit-frame-pointer -m64 -Wall conftest.c -lm  -pthread >&5
configure:14135: $? = 0
configure:14143: result: yes
  1. In the nix compilation, compilation fails, but not because of the wrong header and strerror compatibility. It fails because of some warning about FORTIFY_SOURCE being enabled, while optimisation disabled.
configure:14110: checking whether strerror_r returns char with gnu source is compilable
configure:14135: gcc -o conftest -std=gnu11 -Wall -Wextra -Wsign-compare -Wundef -Wno-format-zero-length -Wpointer-arith -Wno-missing-braces -Wno-missing-field-initializers -Wno-missing-attributes -pipe -g3 -fvisibility=hidden -Wimplicit-fallthrough -O3 -funroll-loops -D_GNU_SOURCE -Werror -O0 -ffunction-sections -fdata-sections -fPIC -m64 -Wall -O0 -ffunction-sections -fdata-sections -fPIC -m64 -Wall -D_GNU_SOURCE -D_REENTRANT -O0 -ffunction-sections -fdata-sections -fPIC -m64 -Wall conftest.c -lm  -pthread >&5
In file included from /nix/store/hn4s3cq7368jkb2db02gzxhdzfa3g9zp-glibc-2.40-36-dev/include/errno.h:25,
                 from conftest.c:101:
/nix/store/hn4s3cq7368jkb2db02gzxhdzfa3g9zp-glibc-2.40-36-dev/include/features.h:422:4: error: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Werror=cpp]
  422 | #  warning _FORTIFY_SOURCE requires compiling with optimization (-O)
      |    ^~~~~~~
cc1: all warnings being treated as errors
configure:14135: $? = 1
configure: failed program was:
| /* confdefs.h */

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.kind: bug Something is broken
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants