This cache directory can be re-used by everything and everyone. There is
nothing bazel or hermetic_cc_toolchain specific there. So let's make it
clear.
Also, if there are any more zig-based toolchains on top of Bazel or
other build systems where they cannot rely on $HOME, but need an
absolute path, this feels like a reasonable choice.
From https://github.com/marler8997/zig-unofficial-releases
I also updated the "Release Process" wiki instructing how to use it.
Long live Mr. Marler!
The launcher got a facelift because of this error:
/code/zig-linux-x86_64-0.11.0-dev.2619+bd3e248c7/lib/std/fmt.zig:2013:9: error: function called at runtime cannot return value at comptime
return &buf;
^~~~~~~~~~~
referenced by:
test.launcher:parseArgs: toolchain/launcher.zig:334:31
remaining reference traces hidden; use '-freference-trace' to see all reference traces
Zig gets confused by the requirement in that test in a runtime context.
This log message has been seen in a Github Actions worker:
running <...>/c++ failed: signal: illegal instruction (core dumped)
I conclude that the launcher was compiled on a newer CPU than used on
the worker at the time.
Reported-by: mp@edgeless.systemsFixes#22
We are using only a single trivial function `paths`, which can be
bundled. Makes things like test-zigcc[1] easier: less dependencies to
worry about.
[1]: https://git.jakstys.lt/motiejus/test-zig-cc/
Sometimes the launcher fails to compile with the following error
messsage:
```
error: FileNotFound
```
We cannot reproduce this in a controlled environment, but see it
happening in the wild often enough to receive repeated questions.
Since this has been escalated to Zig Software Foundation, the most
meaningful thing we can ask our users to do is apply a workaround and
wait. Let's do just that.
`res_search` became a proper symbol only in glibc 2.34. Until that it
was re-defined in headers, hitting the (in-)famous
https://github.com/ziglang/zig/issues/9485
glibc 2.34+:
resolv/resolv.h:int res_search (const char *, int, int, unsigned char *, int)
Old glibc:
resolv/resolv.h:#define res_search __res_search
Also, remove the toplevel includes, as they should never be included in
the first place: the headers are included explicitly when needed.
Until now we needed to maintain two versions of the zig launcher: one
for Windows and one for everything else. This was problematic for two
reasons:
1. I do not know powershell and thus keep breaking the Windows wrapper
all the time (see git history of Fabian fixing stuff that I broke).
2. This makes bazel-zig-cc dependent on the system shell, making it not
really hermetic. So the recently added
`--experimental_use_hermetic_linux_sandbox` does not work with
bazel-zig-cc, unless we bind-mount a bunch of stuff: `/usr`, `/bin`,
`/lib`, `/usr/lib:/lib`, `/usr/lib64:/lib64` and `/proc`.
Switching to a Zig-based wrapper solves both issues, and we can do this:
bazel build "$@" \
--experimental_use_hermetic_linux_sandbox \
--sandbox_add_mount_pair=/proc \
<...>
Zig itself still depends on `/proc` for `/proc/self/exe`, so we need to
keep that. I will look into reducing even that dependency separately.
Not all is nice and shiny though: this commit replaces ~80 LOC worth of
shell scripts wrappers with a singe ~300 LOC zig program, which is
arguably harder to understand. However, it is easier to change, at least
for me, because it's a single file with unit tests! Most importantly,
the gnarly code (which resolves paths and sets environment variables) is
cross-platform.
Thanks to Fabian Hahn for testing this on Windows and pointing out
errors.
* cc: bazel supports only cpp
* wasm-ld: we do not support wasm here. Can be re-added with proper target config.
* coff: we use ld64.lld for some reason (I don't know enough about Windows to tell)
Also, now creating the tools where it makes sense to add for the target only.
While the original intention to be "xdg-friendly" was honorable, it
never worked in practice. Bazel has a tendency to remove almost all
environment variables during the build, causing only the fallback to
remain (i.e. all zig's cache to be put to /tmp/bazel-zig-cc).
If we just accept the world as is, we can get rid of half of the shell
wrappers.
We already know which headers are necessary, so it's wasteful to put the
full zig_sdk to every sandbox.
This reduces the number of files in `external/zig_sdk` from 15k to 4k or
6k, depending on the action.
Just like we know the path to zig_lib_dir, we do know the path to
zig_exe. Lets use that.
This was surfaced by experimenting with
`--experimental_use_hermetic_linux_sandbox`.
`repository_ctx.path("zig")` would return the real file system path to
zig (outside of the sandbox), which is not good when the sandbox is
hermetic.
Co-developed-by: Fabian Hahn <fabian@hahn.graphics>
Previously the toolchain wrappers used to assume the zig_lib_dir is
always in "lib". However, it depends on the OS: macos-x86_64 and
linux-aarch64 have it in "lib/zig".
Update the wrapper scripts to reflect that.
Zig does not always share `libc++.a` and the glibc shims. We have observed
to be caused for 2 reasons:
- For some commands Go `chdir`s to `/tmp/`. This causes `ZIG_LIB_DIR` to be
absolute, which blows the cache key to find the right `libc++.a` (because Zig
thinks we are using a different lib dir to compile libc++). This is currently
worked around in `toolchain/defs.bzl` by overfitting to Go and returning
early from that particular invocation.
- Sometimes Bazel's sandbox messes up Zig's cache keys. If one runs without the
sandbox (`--spawn_strategy=standalone`), the cache hit rate and thus the
build time are much better.
This is actively investigated. I am adding `--spawn_strategy=standalone`
to CI to see the speedup that it provides. I will rollback it later.
Go ignores CFLAGS for some commands. That causes unnecessary build of
glibc shims (and libc++.a).
In the GCC days cross-compiler toolchains used to expose a "tool" per
architecture. Let's do the same here. Then Go cannot cheat with skipping
CFLAGS which we normally *always* expect.
The wrapper code is getting gnarly, I know. But it still fits in my head
somehow, so we can still keep adding to it.
'common' is not a thing in Windows; but it needs to be defined.
This makes v1.0.0-rc3 broken for Windows, and not useful (yet) for
anyone else but me. Revert the README update too.
Empirically these need to come from most specfic to least specific.
The error message is as follows:
In file included from test/c/main.c:1:
In file included from external/zig_sdk/lib/libcxx/include/stdio.h:107:
In file included from external/zig_sdk/lib/libc/include/generic-glibc/stdio.h:38:
external/zig_sdk/lib/libc/include/generic-glibc/bits/types.h:139:3: error:
# error
^
external/zig_sdk/lib/libc/include/generic-glibc/bits/types.h:145:1: error: unknown type name '__STD_TYPE'
__STD_TYPE __DEV_T_TYPE __dev_t; /* Type of device numbers. */
Dissected `generic-glibc/bits/types.h:#error`:
#if __WORDSIZE == 32
<...>
# define __STD_TYPE __extension__ typedef
#elif __WORDSIZE == 64
<...>
# define __STD_TYPE typedef
#else
# error
#endif
So we do not have the `__WORDSIZE`. Where does that come from? Probably from a
directory that has an `x86_64` in it. How does that get included? Let's start
with `lib/libcxx/include/stdio.h`:
16 #include_next <stdio.h>
Now previously our `c++` command line looked like this:
external/zig_sdk/tools/c++ \
<...>
-Iexternal/zig_sdk/lib/include \
-Iexternal/zig_sdk/lib/libcxx/include \
-Iexternal/zig_sdk/lib/libcxxabi/include \
-Iexternal/zig_sdk/lib/libunwind/include \
-Iexternal/zig_sdk/lib/libc/include/generic-glibc \
-Iexternal/zig_sdk/lib/libc/include/any-linux-any \
-Iexternal/zig_sdk/lib/libc/include/x86_64-linux-gnu \
-Iexternal/zig_sdk/lib/libc/include/x86_64-linux-any \
-Iexternal/zig_sdk/lib/libc/include/x86-linux-any \
-Iexternal/zig_sdk/glibc-hacks \
<...>
So the next place it will find `stdio.h` is in `generic-glibc`, which already
uses the `__WORDSIZE`. If we make the "next" include to be the arch-specific
one instead of the generic-glibc, things start compiling again.
Fix the same fo musl.
Motiejus Jakštys