Progress towards tier 1 support for FreeBSD x86_64 #1859

mgxm · 2018-12-27T00:13:06Z

Continuing the work already done by @tiehuis and @myfreeweb, I have completed
some missing points to finish the Tier 1 support. One of the
main points was that we get rid of all the syscalls and now we are using the
libc interface, which is the stable ABI.

Almost all tests are passing, missing only one to fix and which I'm working
on a lot of fixes. The said test is the test-cli, which works on FreeBSD 12 but fails on
FreeBSD 11.

FreeBSD 11 Stack Trace

error: NameTooLong
/usr/home/marcio/build/zig/build/lib/zig/std/os/index.zig:1040:28: 0x234bff in ??? (cli)
            posix.EPERM => return error.AccessDenied,
                           ^
/usr/home/marcio/build/zig/build/lib/zig/std/os/index.zig:1014:9: 0x22bc9c in ??? (cli)
        return deleteFileC(&file_path_c);
        ^
/usr/home/marcio/build/zig/build/lib/zig/std/os/index.zig:513:42: 0x2202f9 in ??? (cli)
    if (file_path.len >= posix.PATH_MAX) return error.NameTooLong;
                                         ^
/usr/home/marcio/build/zig/build/lib/zig/std/os/index.zig:1013:29: 0x22bc1b in ??? (cli)
        const file_path_c = try toPosixPath(file_path);
                            ^
/usr/home/marcio/build/zig/build/lib/zig/std/os/index.zig:1442:16: 0x229355 in ??? (cli)
            => return err,
               ^
/usr/home/marcio/build/zig/build/lib/zig/std/os/index.zig:1484:17: 0x2297ad in ??? (cli)
                try deleteTree(allocator, full_entry_path);
                ^
/usr/home/marcio/build/zig/test/cli.zig:40:9: 0x228411 in ??? (cli)
        try os.deleteTree(a, dir_path);

Since the stable kernel ABI is through libc ziglang#1759

All supported versions of FreeBSD have libc++ in the base system.

Remove all syscalls references * dup2() * chdir() * execve() * fork() * getcwd() * isatty() * readlink() * mkdir() * mmap() * munmap() * read() * rmdir() * symlink() * pread() * write() * pwrite() * rename() * open() * close() * lseek() * exit() * unlink() * waitpid() * nanosleep() * setreuid() * setregid() * raise() * fstat() * pipe() * added pipe2() extern c fn

The system call getrandom(2) just landed on FreeBSD 12, so if we want to support some earlier version or at least FreeBSD 11, we can't depend on the system call.

FreeBSD doesn't mount procfs as default on the base system, so we can't depend on it to get the current path, In this case, we use sysctl(3) to retrieves the system information and get the same information. - CTL_KERN: High kernel limits - KERN_PROC: Return selected information about specific running processes. - KERN_PROC_PATHNAME: The path of the process - Process ID: a process ID of -1 implies the current process.

Stack trace is partially working, with only a few symbols missing. Uses the same functions that we use in Linux to get the necessary debug info.

Fix dirent/stat, add preadv

Use libc interface for: - getdents - kill - openat - setgid - setuid

Prior to this fix, the compare-outputs test suite was showing a strange behavior, the tests always stopped between tests 6-8 and had a stack track similar to each other. ``` Test 8/68 compare-output multiple files with private function (ReleaseSmall)...OK /usr/home/mgxm/dev/zig/zig-cache/source.zig:7:2: error: invalid token: '&' }&(getStdOut() catch unreachable).outStream().stream; ^ The following command exited with error code 1: ``` With the wrong O_* flags, the source code was being written in append mode which resulted in an invalid file ```zig use @import("foo.zig"); use @import("bar.zig"); pub fn main() void { foo_function(); bar_function(); }&(getStdOut() catch unreachable).outStream().stream; stdout.print("OK 2\n") catch unreachable; } fn privateFunction() void { printText(); } ```

andrewrk · 2018-12-27T01:34:54Z

Looks like the problem is in trying to cross compile, targeting FreeBSD.

lld: error: cannot open /usr/lib/Scrt1.o: No such file or directory
lld: error: cannot open /usr/lib/crti.o: No such file or directory
lld: error: cannot open /usr/lib/crtbegin.o: No such file or directory
lld: error: unable to find library -lgcc
lld: error: unable to find library -lgcc_s
lld: error: unable to find library -lc
lld: error: unable to find library -lm
lld: error: unable to find library -lgcc
lld: error: unable to find library -lgcc_s
lld: error: cannot open /usr/lib/crtend.o: No such file or directory
lld: error: cannot open /usr/lib/crtn.o: No such file or directory

We'll have to figure out how to target FreeBSD on non-FreeBSD operating systems to continue zig's promise of "use any target to build for any target".

However you don't have to solve this problem in this PR. One step at a time is OK, and this PR already does a lot.

mgxm · 2018-12-27T03:20:00Z

We'll have to figure out how to target FreeBSD on non-FreeBSD operating systems to continue zig's promise of "use any target to build for any target".

I completely forgot about that, I will work on it too

There's anything I can do?
should I remove Freebsd from the test matrix?

andrewrk · 2018-12-27T03:42:44Z

There's anything I can do?
should I remove Freebsd from the test matrix?

Yeah, for now, remove it from the test matrix, and manually add the native tests in the CI script:

bin/zig test ../test/behavior.zig
bin/zig test ../std/index.zig
bin/zig test ../std/special/compiler_rt/index.zig

And then do these with --release-fast, --release-small, and --library c variants.

valpackett · 2018-12-27T10:29:05Z

works on FreeBSD 12 but fails on FreeBSD 11

Don't expect anything that touches the filesystem to work on both when the filesystem structures are defined for 12 :) I think it's fine to not support versions older than 12 in a young language.

target FreeBSD on non-FreeBSD operating systems

You can't cross-link without having the libraries. Try extracting the system https://download.freebsd.org/ftp/snapshots/amd64/12.0-STABLE/base.txz into some directory and passing --sysroot=/that/directory to lld.

(You can only extract lib directories: bsdtar -xf ... -C /that/directory ./lib ./usr/lib ./usr/libdata ./usr/include)

(btw do you target macOS from non-macOS operating systems?)

valpackett · 2018-12-27T10:30:47Z

std/os/index.zig

@@ -116,7 +116,7 @@ pub fn getRandomBytes(buf: []u8) !void {
                else => return unexpectedErrorPosix(errno),
            }
        },
-        Os.macosx, Os.ios => return getRandomBytesDevURandom(buf),
+        Os.macosx, Os.ios, Os.freebsd => return getRandomBytesDevURandom(buf),


note: FreeBSD 12 has getrandom, I used it on purpose

Why not use getrandom?

because getrandom just landed on FreeBSD 12 and I was considering we were going to support version 11, but as @myfreeweb said if it's ok to no support, we can bring it back

I see, thanks.

I think it's fine to not support versions older than 12 in a young language.

I agree with this statement.

andrewrk · 2018-12-27T14:45:34Z

(btw do you target macOS from non-macOS operating systems?)

Yes, and it works, although LLD is not great at mach-o files. The newest darwin versions do not require the crt1.o startup files or anything like that.

You can't cross-link without having the libraries.

We're going to have to provide these in source form (probably ported to zig, just like we do with compiler-rt) and have the ability to build them for any target.

valpackett · 2018-12-27T16:18:08Z

provide these in source form (probably ported to zig, just like we do with compiler-rt) and have the ability to build them for any target

A whole libc for any target? That sounds… very ambitious (and how does it align with "libc is the stable ABI"?)

andrewrk · 2018-12-27T16:59:59Z

A whole libc for any target is #514. But that's not needed for FreeBSD cross compilation. Only crt1.o & friends, compatible with the FreeBSD libc is necessary. That's a much smaller component. These .o files are statically linked into executables when building against FreeBSD libc. Try it for yourself with the system compiler and look at the linker line. So your ABI question would apply to using C with the system compiler as much as it would apply to Zig here.

- Manually add all the native tests to CI manifest.

mgxm · 2018-12-28T13:46:09Z

I removed the FreeBSD from the test matrix and added the native's tests as
recommended. For this last one, please tell me if it's the correct way or I got
it wrong

andrewrk · 2018-12-28T16:25:03Z

Looks good! I'm ready to merge this if the tests passed for you locally. Looks like the CI failure was a temporary hiccup.

mgxm · 2018-12-28T17:22:16Z

test ../std/index.zig will fail, clock-related functions need to be implemented yet.

andrewrk · 2018-12-28T18:02:05Z

That's ok, let's disable that test for this PR and then part of tier 1 support is working on those tests.

andrewrk · 2018-12-29T06:18:22Z

I'm ready to merge this, do you consider it ready @mgxm?

mgxm · 2018-12-29T16:02:05Z

Yes, but because I made a mess running the tests in the beginning, I think we
should rename the PR to "Initial Tier 1 support" or something like that. I was
using all the tests running with ./bin/zig build --build-file ../build.zig test as a reference and these tests are all passing on FreeBSD 12 as I didn't
know about and didn't go through the tests with bin/zig test ../std/index.zig,
I could not verify all the missing fixes, so the cross-compilation is not yet
done and the std library has some work to be done.

My apologies for the confusion.

mgxm · 2018-12-29T16:15:28Z

Issue #1759 have a checklist that you could ticked off.

switch to libc
stack traces

The issue #1217 I could not reproduce on the master branch

andrewrk · 2018-12-29T17:45:23Z

No worries! This patch still represents significant progress. Thank you for your contribution.

mgxm and others added 21 commits December 19, 2018 18:41

freebsd: remove syscall files

666b153

freebsd: add getdirentries

5ea37f6

freebsd: add access

0273fbf

freebsd: link against libc

1fc56b8

Since the stable kernel ABI is through libc ziglang#1759

freebsd: link against libc++

6cfcdbd

All supported versions of FreeBSD have libc++ in the base system.

freebsd: remove getrandom dependency from libc

1811e7e

The system call getrandom(2) just landed on FreeBSD 12, so if we want to support some earlier version or at least FreeBSD 11, we can't depend on the system call.

freebsd: initial stack trace

e5b4748

Stack trace is partially working, with only a few symbols missing. Uses the same functions that we use in Linux to get the necessary debug info.

Fix stat/timespec definitions for FreeBSD

054c7ab

Add preadv/pwritev on FreeBSD

76efc46

Merge pull request #1 from myfreeweb/fbsd2

ac1b2a3

Fix dirent/stat, add preadv

freebsd: use realpath() to resolve symbolic links

46a0f60

freebsd: add realpath to freebsd/index.zig

a6f33e3

freebsd: remove system linker hack

c156d51

freebsd: fix Stat mode type

c26f543

Merge branch 'master' into fbsd2

773bf80

freebsd: remove syscall and use libc

682815f

Use libc interface for: - getdents - kill - openat - setgid - setuid

freebsd: implement std.os.time.sleep

de473d4

tests: add FreeBSD to the test matrix

c29cae7

mgxm changed the title ~~Add Tier 1 FreeBSD Support for x86_64~~ [WIP] Add Tier 1 FreeBSD Support for x86_64 Dec 27, 2018

valpackett reviewed Dec 27, 2018

View reviewed changes

tests: remove freebsd from the test matrix

4ff23b6

- Manually add all the native tests to CI manifest.

ci: update freebsd manifest

2e08bd6

andrewrk changed the title ~~[WIP] Add Tier 1 FreeBSD Support for x86_64~~ Progress towards tier 1 support for FreeBSD x86_64 Dec 29, 2018

andrewrk merged commit d8b6fa9 into ziglang:master Dec 29, 2018

mgxm deleted the fbsd2 branch January 4, 2019 21:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Progress towards tier 1 support for FreeBSD x86_64 #1859

Progress towards tier 1 support for FreeBSD x86_64 #1859

mgxm commented Dec 27, 2018 •

edited

Loading

andrewrk commented Dec 27, 2018

mgxm commented Dec 27, 2018

andrewrk commented Dec 27, 2018

valpackett commented Dec 27, 2018

valpackett Dec 27, 2018

andrewrk Dec 27, 2018

mgxm Dec 27, 2018

andrewrk Dec 27, 2018

andrewrk commented Dec 27, 2018

valpackett commented Dec 27, 2018

andrewrk commented Dec 27, 2018

mgxm commented Dec 28, 2018

andrewrk commented Dec 28, 2018

mgxm commented Dec 28, 2018

andrewrk commented Dec 28, 2018

andrewrk commented Dec 29, 2018

mgxm commented Dec 29, 2018

mgxm commented Dec 29, 2018

andrewrk commented Dec 29, 2018

Progress towards tier 1 support for FreeBSD x86_64 #1859

Progress towards tier 1 support for FreeBSD x86_64 #1859

Conversation

mgxm commented Dec 27, 2018 • edited Loading

andrewrk commented Dec 27, 2018

mgxm commented Dec 27, 2018

andrewrk commented Dec 27, 2018

valpackett commented Dec 27, 2018

valpackett Dec 27, 2018

Choose a reason for hiding this comment

andrewrk Dec 27, 2018

Choose a reason for hiding this comment

mgxm Dec 27, 2018

Choose a reason for hiding this comment

andrewrk Dec 27, 2018

Choose a reason for hiding this comment

andrewrk commented Dec 27, 2018

valpackett commented Dec 27, 2018

andrewrk commented Dec 27, 2018

mgxm commented Dec 28, 2018

andrewrk commented Dec 28, 2018

mgxm commented Dec 28, 2018

andrewrk commented Dec 28, 2018

andrewrk commented Dec 29, 2018

mgxm commented Dec 29, 2018

mgxm commented Dec 29, 2018

andrewrk commented Dec 29, 2018

mgxm commented Dec 27, 2018 •

edited

Loading