[WIP] Prototype run-make test to check core::ffi::c_* types against clang
#133944
Conversation
|
Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @tgross35 (or someone else) some time within the next two weeks. Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (
|
rustbot
added
A-run-make
| ("c_char", "__CHAR_BIT__"), // Character width | ||
| ("char_signedness", "__CHAR_UNSIGNED__"), | ||
| ("c_double", "__SIZEOF_DOUBLE__"), // Double precision floating-point | ||
| ("c_float", "__SIZEOF_FLOAT__"), // Single precision floating-point | ||
| ("c_int", "__SIZEOF_INT__"), // Signed integer | ||
| ("c_long", "__SIZEOF_LONG__"), // Signed long integer | ||
| ("c_longlong", "__SIZEOF_LONG_LONG__"), // Signed long long integer | ||
| ("c_schar", "__SIZEOF_CHAR__"), // Signed char | ||
| ("c_short", "__SIZEOF_SHORT__"), // Signed short integer | ||
| ("c_uchar", "__SIZEOF_CHAR__"), // Unsigned char | ||
| ("c_uint", "__SIZEOF_INT__"), // Unsigned integer | ||
| ("c_ulong", "__SIZEOF_LONG__"), // Unsigned long integer | ||
| ("c_ulonglong", "__SIZEOF_LONG_LONG__"), // Unsigned long long integer | ||
| ("c_ushort", "__SIZEOF_SHORT__"), // Unsigned short integer | ||
| ("c_size_t", "__SIZEOF_SIZE_T__"), // Size type | ||
| ("c_ptrdiff_t", "__SIZEOF_PTRDIFF_T__"), // Pointer difference type]; |
There was a problem hiding this comment.
We should either prefer the standard macros of CHAR_WIDTH and so on where available (which will be in bits instead of bytes, but eh, just hit it with >> 3), or simply use the actual sizeof operator within the generated C code.
This comment has been minimized.
This comment has been minimized.
try a program like this: (and so on for the other types.) From there you can parse them into a map like you have for LLVM. to get a list of the available types you could use something like |
|
Oh hey, thanks for working on this! There are two interesting bits here:
What I mentioned in #133058 (comment) works, I was able to prototype this out relatively easily. Solution to #1Rust has the internal // smallcore.rs (within this run-make directory)
#![allow(internal_features)]
#![feature(decl_macro)]
#![feature(intrinsics)]
#![feature(lang_items)]
#![feature(link_cfg)]
#![feature(no_core)]
#![feature(rustc_attrs)]
#![no_core]
#![no_std]
// This is in `tests/auxiliary/minicore.rs`
extern crate minicore;
use minicore::Sized;
// Copy the body from https://github.com/rust-lang/rust/blob/df5b8e39b7def660696340b199ae395a869b3064/library/core/src/internal_macros.rs#L149
macro_rules! cfg_if { /* ... */ }
macro_rules! panic {
($msg:literal) => { $crate::panic(&$msg) };
}
/* Intrinsics are function signatures that `rustc` does something magic with.
* The body doesn't matter. */
#[rustc_intrinsic]
#[rustc_intrinsic_const_stable_indirect]
#[rustc_intrinsic_must_be_overridden]
pub const fn size_of<T>() -> usize {
loop {}
}
#[rustc_intrinsic]
#[rustc_intrinsic_must_be_overridden]
pub const fn abort() -> ! {
loop {}
}
/* Lang items are similar to intrinsics but aren't limited to function signatures */
// This is what `panic!()` usually eventually expands to
#[lang = "panic"]
#[rustc_const_panic_str]
const fn panic(_expr: &&'static str) -> ! {
abort();
}
/* We need to reimplement some basic traits so rustc knows what to do with `==` and `!` */
#[lang = "eq"]
pub trait PartialEq<Rhs: ?Sized = Self> {
fn eq(&self, other: &Rhs) -> bool;
fn ne(&self, other: &Rhs) -> bool {
!self.eq(other)
}
}
#[lang = "not"]
pub trait Not {
type Output;
fn not(self) -> Self::Output;
}
impl PartialEq for usize {
fn eq(&self, other: &usize) -> bool {
(*self) == (*other)
}
}
impl Not for bool {
type Output = bool;
fn not(self) -> Self {
!self
}
}(@jieyouxu any interest in just putting some of this in minicore?) Solution to #2We can't build # Start with core/src/ffi copied to a temporary directory
# Remove stability attrbutes since they can't be used outside of std
perl -i -0777 -pe 's/#!?\[(un)?stable.*?\]//gs' ffi/mod.rs
# Remove doc attributes since we'll be removing some items they apply to
perl -i -0777 -pe 's/#\[doc.*?\]//gs' ffi/mod.rs
# Remove lang item indicators so they don't conflict if `minicore` gets updated
perl -i -0777 -pe 's/#\[lang.*?\]//gs' ffi/mod.rs
# Remove non-inline modules since we don't need `cstr` or `va_list`
perl -i -0777 -pe 's/.*mod.*;//g' ffi/mod.rs
# Remove reexports that go with those modules
perl -i -0777 -pe 's/.*use.*;//g' ffi/mod.rs
# Remove the `Debug` implementation for `c_void`. This is done in two steps
# to avoid regex tripping over `{`/`}`
perl -i -0777 -pe 's/fn fmt.*?\{.*?\}//gs' ffi/mod.rs
perl -i -0777 -pe 's/impl fmt::Debug for.*?\{.*?\}//gs' ffi/mod.rs
# Just for sanity sake when looking at the generated files, eliminate repeated empty lines
perl -i -0777 -pe 's/\n{3,}/\n\n/gs' ffi/mod.rsWhat might actually be better is to move the types we care about into a new TestsWrite a test file that asserts each size at compile time, using what we have in // tests.rs
use super::*; // `super` will include everything from `smallcore` once glued together
pub const TEST_C_CHAR_SIZE: () = if size_of::<ffi::c_char>() != CLANG_C_CHAR_SIZE {
panic!("wrong c_char size");
};
pub const TEST_C_LONG_SIZE: () = if size_of::<ffi::c_long>() != CLANG_C_LONG_SIZE {
panic!("wrong c_long size");
};GlueThis is the only target-specific part: each target needs to generate its own file where you fill in the /* boilerplate part */
#![allow(unused)]
// include! rather than `mod` with `path` since this one has crate-level
// attributes
include!("path/to/smallcore.rs");
// Path to the simplified FFI file
#[path = "path/to/ffi.rs"]
mod ffi;
#[path = "path/to/tests.rs"]
mod tests;
/* end boilerplate */
/* Per-target: append to the template based on Clang's output */
const CLANG_C_CHAR_SIZE: usize = 1;
const CLANG_C_LONG_SIZE: usize = 8;
// ...Then just have rustc build the file for the relevant target; if our sizes are wrong, we'll get a failure at const eval. You can even pass |
|
I'll take a look later today, thanks for the detailed writeup. |
|
For checking the sign, you could do add something like this to trait Signed {
const SIGNED: bool;
}
impl Signed for i8 {
const SIGNED: bool = true;
}
impl Signed for u8 {
const SIGNED: bool = false;
}
const TEST_C_CHAR_SIGNED: () = if ffi::c_char::SIGNED ^ CLANG_C_CHAR_SIGNED {
panic("mismatched c_char sign");
};This requires adding a |
Thanks! This is less nice than having core available ofc, but this builds a Most fragile thing is probably the text processing of |
|
One other thought: this PR shouldn't fix anything that doesn't currently line up, so there should be xfails. It might be easiest to just do this in // tests that always pass don't need a cfg_if
cfg_if! {
if #[cfg(target_os = "foobar")] {
// FIXME: long isn't long enough on foobar
const XFAIL_C_LONG_SIZE: usize = 16;
pub const TEST_C_LONG_SIZE: () = if size_of::<ffi::c_long>() != XFAIL_C_LONG_SIZE {
panic("wrong c_long size");
};
} else {
// Default test
pub const TEST_C_LONG_SIZE: () = if size_of::<ffi::c_long>() != CLANG_C_LONG_SIZE {
panic("wrong c_long size");
};
}
} |
|
Thanks for the write up, I appreciate it! Will read it and look to understand it tonight :). Just a quick question that might be off topic but I created the following function for testing purposes: I wanted to save the sizes for the c types for the current target but I get 3 errors for the following types: c_ptrdiff_t The error is as follows: use of unstable library feature The open issue is issue #88345 My concern is that in the tests we are using size_of::ffi::TYPE(). If we input the type as c_size_t or the other 2 we will get an error. Should this effect the run-make test at all or is it fine? Edit: I am assuming adding #![feature(c_size_t)] allows the run-make test not to be affected by this. |
|
disregard this commit, gonna be following what you outlined for the solution. |
This comment has been minimized.
This comment has been minimized.
|
I was eepy yesterday, going to revisit this in a moment today. I'll need to revisit what the test requirements are. |
Just for reference, this would be correct if importing from |
Yes, if these items are only in core (not alloc, not std-only), please do add them to
Moving to
I don't actually know what |
rustbot
added
S-waiting-on-author
|
@tgross35 |
|
@tgross35
this is how I attempt to build it but I receive this error: error[E0463]: can't find crate for I have tried to fix this problem through defining #![no_std] in the target file but that does not seem right and does not work. Smallcore is still the same as what you outlined and I created a mod.rs file like you explained. |
This comment has been minimized.
This comment has been minimized.
That file also needs Also as @jieyouxu mentioned above, you can add whatever is missing to |
| let rustc_output = rustc() | ||
| .arg("-Z") | ||
| .arg("unstable-options") | ||
| .arg("--target") | ||
| .arg(target) | ||
| .arg(&file_name) | ||
| .run(); |
There was a problem hiding this comment.
You can add --emit=metadata so it just checks without building
| } | ||
| } | ||
|
|
||
| // default sizing here, not sure if this is necessary tho or should be included. |
There was a problem hiding this comment.
Do you know which specific targets are missing these types? I think it is probably better to not have a fallback since that might mean things pass when they shouldn't if clang's output is weird for some reason.
There was a problem hiding this comment.
good point, will delete the defaulting portion.
There was a problem hiding this comment.
This will eventually need to be generated rather than checked in (I assume this is for testing, just making sure)
| for target in targets.lines() { | ||
| // Run clang to get target info | ||
|
|
||
| //TODO: someone please test riscv targets as clang on my machine does not have that llvm |
| if target.starts_with("riscv") { | ||
| continue; | ||
| } | ||
| if target.starts_with("wasm") { | ||
| continue; | ||
| } | ||
| if target.starts_with("xtensa") { | ||
| continue; | ||
| } |
There was a problem hiding this comment.
Could this instead be a const SKIPPED_TARGETS: &[&str] = &[...] at the top of the file, with the full exact target names? So we don't cross off more than is needed.
|
@tgross35 It seems that just generating a file that has include!("path/to/smallcore.rs");
#[path = "./ffi/mod.rs"]
mod ffi;
#[path = "./tests.rs"]
mod tests;does not glue it together properly as it does not find the custom macros (panic, cfg_if, etc...) and the macro include does not exist for this scope. Could I just directly inline the parsed mod.rs and tests.rs and then use a modified minicore with the added checks that are in smallcore? What I am thinking code wise: let mut rmake_content = String::from(
r#"
#![no_std]
#![no_core]
#![feature(no_core)]
#![allow(unused)]
extern crate minicore; //would contain the added macros from smallcore.rs
use minicore::Sized;
"#,
);
// direct inline of mod.rs content (FFI types)
let mod_content = rfs::read_to_string("./ffi/mod.rs");
rmake_content.push_str(&mod_content);
rmake_content.push_str(&format!(
"
const CLANG_C_CHAR_SIZE: usize = {};
const CLANG_C_CHAR_SIGNED: bool = {};
const CLANG_C_SHORT_SIZE: usize = {};
const CLANG_C_INT_SIZE: usize = {};
const CLANG_C_LONG_SIZE: usize = {};
const CLANG_C_LONGLONG_SIZE: usize = {};
const CLANG_C_FLOAT_SIZE: usize = {};
const CLANG_C_DOUBLE_SIZE: usize = {};
",
parse_size(&defines, "CHAR"),
parse_signed(&defines, "CHAR"),
parse_size(&defines, "SHORT"),
parse_size(&defines, "INT"),
parse_size(&defines, "LONG"),
parse_size(&defines, "LONG_LONG"),
parse_size(&defines, "FLOAT"),
parse_size(&defines, "DOUBLE"),
));
// direct inline of tests.rs
let test_content = rfs::read_to_string("tests.rs"); //super is deleted from tests.rs
rmake_content.push_str(&test_content);what do you think? Maybe the structure or something else is messed up but I am unsure on how else it can all be glued together in order to use the newly specified macros as well as include tests.rs. I would still like to use super and make it clean instead of writing out literally everything to the target file. I will continue to think about it but if you have any insight it is greatly appreciated. Thanks! EDIT: this code does not fix the issues nor does it really change anything. |
Would you mind pushing if code is updated from what is here currently, or pasting the error messages? It should work as long as I think this test will need
Using |
They do in two(?) specific runners, I believe it's |
jieyouxu
changed the title
core::ffi::c_* types against clang
|
@tgross35 Alrighty, I pushed the code. Here are a few examples of some of the errors (42 output when I run): size_of not found not finding the types in ffi panic macro not found alias not found |
| r#" | ||
| #![no_std] | ||
| #![no_core] | ||
| #![feature(no_core)] | ||
| #![allow(unused)] | ||
| #[path = "ffi/mod.rs"] | ||
| mod ffi; | ||
| #[path = "tests.rs"] | ||
| mod tests; | ||
| "#, |
There was a problem hiding this comment.
Missing an include!("../../path/to/minicore.rs")? :)
You should be able to populate the path with something like run_make_support::source_root.join("tests/auxiliary/minicore.rs").
There was a problem hiding this comment.
Could I do it like this?
let minicore_path = run_make_support::source_root().join("tests/auxiliary/minicore.rs");
// For each target:
for target in targets.lines() {
// Run clang to get target info
//TODO: someone please test riscv targets as clang on my machine does not have that llvm
//target
if SKIPPED_TARGETS.iter().any(|prefix| target.starts_with(prefix)) {
continue;
}
let clang_output =
clang().args(&["-E", "-dM", "-x", "c", "/dev/null", "-target", target]).run();
let defines = String::from_utf8(clang_output.stdout()).expect("Invalid UTF-8");
// TODO: mod rs file will need to be generated based off of core_ffi - can use regex
// availble in run-make to handle.
let minicore_content = rfs::read_to_string(&minicore_path);
let mut rmake_content = format!(
r#"
#![no_std]
#![no_core]
#![feature(no_core)]
#![allow(unused)]
{}
#[path = "ffi/mod.rs"]
mod ffi;
#[path = "tests.rs"]
mod tests;
"#,
minicore_content
);
the issue I am seeing with the include! is it does not exist because the macro is undefined. Hence, I just decided to directly insert the file contents into here.
There was a problem hiding this comment.
Ah I forgot that cyclic dependency; yeah, just combining it there should work. Otherwise you could redefine the macro like https://doc.rust-lang.org/1.84.0/src/core/macros/mod.rs.html#1540, or build it separately, but I think what you suggested is easiest.
|
I have most of the test now working properly! Just a few issues with specific targets. To verify, these are the types that have mismatches: const SKIPPED_TARGETS: &[&str] = &[
"riscv",
"wasm", //error: unknown target triple 'wasm32v1-none'
"xtensa", //error: unknown target triple 'xtensa-esp32-espidf'
"aarch64",
"arm",
"avr",
"csky",
"hexagon",
"msp430",
"thumb",
"x86_64-unknown-l4re-uclibc", //idk how to define this one
];
//riscv32 unsigned char
//aarch64 unsigned char
//arm unsigned char
//csky unsigned char
//hexagon unsigned char
//thumb unsigned char
//x86_64-unknown-l4re-uclibc unsigned char
//avr incorrect c_int size
//msp430 incorrect c_int sizeWith these skipped the test passes. I don't want to skip them but instead handle these properly in tests.rs as you highlighted with the following: cfg_if! {
if #[cfg(any(
all(
target_os = "darwin",
target_arch = "aarch64"
)
))] {
const XFAIL_C_CHAR_SIGNED: bool = true;
pub const TEST_C_CHAR_UNSIGNED: () = if ffi::c_char::SIGNED ^ XFAIL_C_CHAR_SIGNED {
panic!("mismatched c_char sign");
};
} else {
pub const TEST_C_CHAR_UNSIGNED: () = if ffi::c_char::SIGNED ^ CLANG_C_CHAR_SIGNED {
panic!("mismatched c_char sign");
};
}
}but for some reason it is not matching up the target and executing that if branch (just testing with the aarch64 target atm). I have tried multiple ways to define it and highlight it but it always executes the else branch and I am stumped on why. Any ideas? |
Is this the actual code that is failing? If so, For future reference, you can run |
|
Oh nice. Thanks, that fixed it. Will do this for the rest and then the PR should be ready :) . |
In regards to this tho, would it be better to just include it into mod.rs? mod c_char_definition {
cfg_if! {
// These are the targets on which c_char is unsigned.
if #[cfg(any(
all(
target_os = "linux",
any(
target_arch = "aarch64",
target_arch = "arm",
target_arch = "hexagon",
target_arch = "powerpc",
target_arch = "powerpc64",
target_arch = "s390x",
target_arch = "riscv64",
target_arch = "riscv32",
target_arch = "csky"
)
),
all(target_os = "android", any(target_arch = "aarch64", target_arch = "arm")),
all(target_os = "l4re", target_arch = "x86_64"),
all(
any(target_os = "freebsd", target_os = "openbsd", target_os = "rtems"),
any(
target_arch = "aarch64",
target_arch = "arm",
target_arch = "powerpc",
target_arch = "powerpc64",
target_arch = "riscv64"
)
),
all(
target_os = "netbsd",
any(
target_arch = "aarch64",
target_arch = "arm",
target_arch = "powerpc",
target_arch = "riscv64"
)
),
all(
target_os = "vxworks",
any(
target_arch = "aarch64",
target_arch = "arm",
target_arch = "powerpc64",
target_arch = "powerpc"
)
),
all(
target_os = "fuchsia",
any(target_arch = "aarch64", target_arch = "riscv64")
),
all(target_os = "nto", target_arch = "aarch64"),
target_os = "horizon",
target_os = "aix",
))] {
pub type c_char = u8;
} else {
// On every other target, c_char is signed.
pub type c_char = i8;
}
}
}Basically in this just add the additional targets (os as well) I found that have an unsigned char. |
|
That file was updated recently so it's as bad, rust/library/core/src/ffi/mod.rs Lines 158 to 179 in dd333ca However, it shouldn't matter right? Once you have that autogenerated from |
|
@tgross35 I noticed that the most recent mod.rs does not handle different sizes for doubles (ex: avr having a size of 4 bytes for a double leading to a mismatch in sizes). Would it be a good idea to add this to the Edit: |
…fied mod.rs creation
|
The job Click to see the possible cause of the failure (guessed by this bot) |
Don't change the definitions in |
Hello,
I have been working on issue #133058 for a bit of time now and would love some feedback as I seem to be stuck and not sure if I am approaching this correctly.
I currently have the following setup:
Get the rust target list
Use rust target to query the llvm target
Get clang definitions through querying the clang command with llvm target. I only save the necessary defines. Here is an example of the saved info (code can easily be modified to store Width as well):
Target: riscv64-unknown-linux-musl
CHAR_BIT = 8
CHAR_UNSIGNED = 1
SIZEOF_DOUBLE = 8
SIZEOF_INT = 4
SIZEOF_LONG = 8
SIZEOF_PTRDIFF_T = 8
SIZEOF_SIZE_T = 8
SIZEOF_FLOAT = 4
SIZEOF_LONG_LONG = 8
SIZEOF_SHORT = 2
Target: riscv64-unknown-fuchsia
CHAR_UNSIGNED = 1
SIZEOF_SHORT = 2
CHAR_BIT = 8
SIZEOF_INT = 4
SIZEOF_SIZE_T = 8
SIZEOF_FLOAT = 4
SIZEOF_LONG = 8
SIZEOF_DOUBLE = 8
SIZEOF_PTRDIFF_T = 8
SIZEOF_LONG_LONG = 8
This is where it gets a bit shaky as I have been brainstorming ways to get the available c types in core::ffi to verify the size of the defined types but do not think I have the expertise to do this.
For the current implementation I specifically focus on the c_char type (unsigned vs signed). The test is currently failing as there are type mismatches which are expected (issue #129945 highlights this). I just do not know how to continue executing tests even with the type mismatches as it creates an error when running the run-make test. Or maybe I am doing something wrong in generating the test? Not too sure but would love your input. Thanks
r? @tgross35 @jieyouxu