Zig: Tier 1 FreeBSD Support for x86_64

[andrewrk]

Here's the issue to track FreeBSD support.

History:

• @tiehuis started an initial freebsd branch.
• @myfreeweb picked up the figurative baton in #1661 and that work is merged into freebsd2 branch.
  
  
  
  • See this PR description and comments for some useful information.
  
  
  • I believe the current state of this branch is that the tests pass in debug mode but not release mode. I haven't personally tried it yet.
  
  
• freebsd2 is now merged into master, and FreeBSD has Tier 2 Support status.

Checklist:

• [x] switch to using libc as the stable kernel ABI for freebsd. citation
• [x] #1217 Bus error on simple code
• [x] get all the tests passing for freebsd.
• [x] reduce stage1 memory to within sr.ht limits, so that CI tests can pass (#3210)
• [x] (@andrewrk) audit the freebsd2 branch
• [x] CI for FreeBSD. We actually have a few options:
  
  
  
  • get my raspberry pi (aarch64) hooked up to the CI somehow
  
  
  • use https://builds.sr.ht/ which has freebsd x86_64 support
  
  
  • on azure dev ops, use qemu and a custom freebsd image as demonstrated by @andrewchambers in #941 (although this was for openbsd not freebsd). This should be possible because we get a full 6 hours to run the tests with azure dev ops.
  
  
• [x] make sure stack traces are working
• [ ] audit the structs of std/os/freebsd.zig to make sure the ABIs are correct and match the system header files.
• [ ] #2876 Solve cross compilation when targeting FreeBSD

Once this is complete, we can mark FreeBSD x86_64 as having Tier 1 support.

[bnoordhuis]

get my raspberry pi (aarch64) hooked up to the CI somehow

That's a raspberry pi 3? We have a small armada of those in the Node.js CI. They're a lot faster than the pi 1 and 2 but still so much slower than regular server hardware that we cross-compile and shard the test suite across 10 or 12 machines.

That's a long-winded way of saying they're not really an option unless you're _really_ patient. :-)

[andrewrk]

That's a raspberry pi 3?

Yes it is. Thanks for the heads up!

[myfreeweb]

re: CI, I'm "starting" a buildbot cluster at https://porting.club — currently it's just a Scaleway 4-core ThunderX VPS (the master as well as the first worker, which is a jail on the same box).
The "plan" is to have a cluster with many different OSes and CPU architectures specifically for stuff that has portability needs — compilers/interpreters/VMs, libraries that touch OS specific stuff, etc.
The idea is to not use throwaway containers or VMs, but instead set up persistent environments — so that they can be used not just for CI tests or nightly builds, but also for development (i.e. project developers should be able to ssh into the workers).

(btw ThunderX cores are comparable to Cortex-A72s / significantly faster than the A53s the RPi has)

(@bnoordhuis I guess Node.js builds its own copy of V8, while Zig wouldn't need to build any monster dependencies, and the compiler itself is kinda lightweight. Oh wait, LLVM can be a memory hog though, and the Pi only has 1GB.)

[andrewrk]

@myfreeweb that's exciting. Keep me updated.

In other news I updated the support table in the README to use a "tier" system: https://github.com/ziglang/zig#support-table

I think what I'll do is review the freebsd2 branch, make any changes that I see fit, and then merge it into master. Then freebsd becomes Tier 2 for x86_64 and i386. And then this issue can remain open to track Tier 1 support.

[andrewrk]

I just merged the freebsd2 branch into master, and so now FreeBSD has Tier 2 Support. This issue is tracking the effort to achieve Tier 1 Support.

[ararslan]

FreeBSD is now available on Cirrus CI. I set it up on one of my repositories to test it out. As an example, you can see the config here and a build log here.

[andrewrk]

I rented a freebsd server for an hour and poked at it. Here's what I accomplished:

• Fixed bug in os_self_exe_path which was making the build cache not work
• Fixed a bunch of incorrect declarations in std/os/freebsd/index.zig. For example NSIG was copied from linux and it was incorrect. Be careful copying this stuff from other systems, we have to hand-verify that the declarations are correct for every single one that we add. Also the Sigaction struct was knowingly incorrect. I didn't catch this in the pull request. Similarly some syscall functions returned success even though they did nothing. I removed these. I think it's better that they are compile errors until we can have the actual implementations for them.
• Implemented raise syscall and hence made abort work. I might have found a bug in freebsd libc: https://lists.freebsd.org/pipermail/freebsd-hackers/2018-December/053712.html Depending on the answer to that mailing list question, we need to add signal handling protection in raise or abort.
• Updated syscall code and fixed some bugs

Now Hello World works and the behavior tests pass in debug mode. ReleaseFast mode doesn't pass, and the standard library tests require more porting work in order to compile. But they're in a state where I think it's easy to contribute to.

[andrewrk]

One of the developers from FreeBSD stopped by and clarified that the stable kernel ABI is in fact through libc, and so we will now switch to doing it that way, like we do for MacOS.

[andrewrk]

Thanks @ararslan and @mgxm for the CI pull requests. We now have sr.ht building successfully on x86_64 FreeBSD on every pull request and master branch commit. However, all tests are disabled except for the behavioral tests in debug mode (which all pass).

So - we've now ensured that FreeBSD won't regress accidentally. And we can progress on Tier 1 Support by getting more tests to pass, and then making sure those tests are run by CI. Once we get all the tests passing, I'll work on static FreeBSD builds being available on ziglang.org/download.

Progress!

[andrewrk]

See this comment for a proof of concept of cross compiling glibc: https://github.com/ziglang/zig/issues/514#issuecomment-470957284

With regards to FreeBSD here are the next steps:

• [ ] Follow the example set by glibc, using the wiki article and the new tool libc/process_headers.zig, creating or obtaining builds of FreeBSD libc for all the target architectures that it supports (and that Zig also supports - see zig targets). Commit those header files to Zig source repository.
• [ ] On FreeBSD, link hello_world.c in verbose mode and observe the link options. Find out what startup files are needed. Anything with gcc in it should be covered by zig's compiler_rt.a and libunwind.a. crtbegin.o and crtend.o are unnecessary. crt1.o, crti.o, etc, these need to be built from source by zig for the target.
• [ ] Look at the build commands FreeBSD libc uses to build the startup files, and extract the minimum necessary source files from FreeBSD libc - patching them if necessary - and include them in Zig source repository and make it so that Zig can build them from source for all the targets supported by both FreeBSD libc and Zig. Follow the example set by glibc, in codegen.cpp and link.cpp.
• [ ] update the wiki article with maintenance instructions for the next person to follow when FreeBSD libc is updated.

[myfreeweb]

observe the link options. Find out what startup files are needed

 "/usr/bin/cc" -cc1 -triple x86_64-unknown-freebsd13.0 -emit-obj -mrelax-all -disable-free -main-file-name basenamewtf.c -mrelocation-model static -mthread-model posix -mdisable-fp-elim -masm-verbose -mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu x86-64 -dwarf-column-info -debugger-tuning=gdb -v -resource-dir /usr/lib/clang/7.0.1 -fdebug-compilation-dir /home/greg/src/localhost -ferror-limit 19 -fmessage-length 284 -fobjc-runtime=gnustep -fdiagnostics-show-option -fcolor-diagnostics -o /tmp/basenamewtf-eff904.o -x c basenamewtf.c -faddrsig
 "/usr/bin/ld" --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o basenamewtf /usr/lib/crt1.o /usr/lib/crti.o /usr/lib/crtbegin.o -L/usr/lib /tmp/basenamewtf-eff904.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/crtend.o /usr/lib/crtn.o

keep in mind the funny build process for crt1 required for the ABI version metadata

crtbegin.o and crtend.o are unnecessary

hm. Do ctors/dtors from shared libraries work without crtbegin/end in the executable?

[andrewrk]

funny build process

I've been staring at glibc code for 7 days now, that "funny build process" looks incredibly clean and easy to understand :rofl:

hm. Do ctors/dtors from shared libraries work without crtbegin/end in the executable?

Based on my understanding, crtbegin.o and crtend.o are the "old way" of doing things and aren't necessary with newer C runtime libraries. I haven't tested it yet though.

Another option we have is saying "that's not supported". I think you would have trouble finding someone who thinks shared library constructors/destructors are a good idea.

Finally a third option is to stop fighting it, and support building crtbegin.o and crtend.o. I looked at the code in gcc, and it wouldn't be straightforward. But I have a few ideas to try that could solve it.

[andrewrk]

By the way what ABI does zig say is native for you on freebsd?

zig targets

Which one do you think makes sense to be the native one? I'm trying to understand this triple: x86_64-unknown-freebsd13.0 and I wonder if FreeBSD has a patch to clang to add another ABI?

[daurnimator]

I think you would have trouble finding someone who thinks shared library constructors/destructors are a good idea.

✋ why would they be a bad idea? How else should a shared library e.g. initialise an initialisation mutex?

[andrewrk]

How else should a shared library e.g. initialise an initialisation mutex?

like this

[myfreeweb]

crtbegin.o and crtend.o are the "old way" of doing things and aren't necessary with newer C runtime libraries

oh, right, ctors/dtors are the old way, and the new way is init_array/fini_array. (And FreeBSD on new architectures like aarch64 doesn't support ctors/dtors at all.) I always forget which one is the new one lol. Seems like the _arrays are handled by rtld, so we're fine here!

I'm trying to understand this triple: x86_64-unknown-freebsd13.0 and I wonder if FreeBSD has a patch to clang to add another ABI?

Look at the Triple class in LLVM, everything is parsed with StartsWith and the version is parsed into numbers.

Nothing prevents me from doing

cc -cc1 -triple x86_64-unknown-freebsd69.420 …

:)

[andrewrk]

I started working on making FreeBSD builds available in the llvm8 branch. I ran into a snag. This command:

release/bin/zig build-exe ../example/hello_world/hello_libc.zig --library c

when I run it manually in a freebsd server that I'm renting, works fine. But in the sr.ht freebsd CI, it prints:

/usr/home/build/zig/example/hello_world/hello_libc.zig:1:11: error: C import failed

and nothing else. It's hard to tell what the problem is because as far as I understand the environments should be the same, and it works fine in the environment that I have shell access to.

[andrewrk]

FreeBSD binary builds available on the download page are also blocking on this sr.ht issue.

[andrewrk]

The sr.ht issue is resolved and I have pushed a commit to master branch enabling FreeBSD CI builds to automatically go on the download page when tests pass.

[andrewrk]

FreeBSD builds are now available on https://ziglang.org/download/. We still don't have all the tests passing yet though.

[andrewrk]

New TODO item for FreeBSD: audit the structs in std/os/freebsd.zig. See https://github.com/ziglang/zig/pull/2326/files#r279576473. Some of the structs were copied from std/os/linux.zig rather than system header files, which results in ABI mismatches. They're really annoying to troubleshoot, and there are probably other mistakes as well.

[dch]

• [x] WIP port of zig 0.4.0 to FreeBSD https://reviews.freebsd.org/D20437 RPATH fixed thanks IRC <3

[andrewrk]

Good news for FreeBSD folks, I added some memory profiling code in #3482, and it looks like there are some straightforward changes to make which will greatly reduce memory usage. Hopefully this will put us within sr.ht's requirements so that we can enable all the tests. Then Tier 1 Support will basically be down to supporting cross compiling for FreeBSD's libc.

[andrewrk]

More progress: #4458
All the test suite is enabled in the CI now except for std lib tests, which are hitting OOM. However, stage1 has seen a handful of improvements recently regarding memory consumption, and we're only a couple hundred mebibytes away from all freebsd tests passing on sr.ht's 4 GiB VM image.