1
Fork 0
Commit Graph

163 Commits

Author SHA1 Message Date
Motiejus Jakštys 1fb93e089c toolchain wrapper: zig_exe path is relative
Just like we know the path to zig_lib_dir, we do know the path to
zig_exe. Lets use that.

This was surfaced by experimenting with
`--experimental_use_hermetic_linux_sandbox`.
`repository_ctx.path("zig")` would return the real file system path to
zig (outside of the sandbox), which is not good when the sandbox is
hermetic.

Co-developed-by: Fabian Hahn <fabian@hahn.graphics>
2022-12-11 18:28:38 +02:00
Fabian Hahn 43b6e1c0c1 fix windows include root path 2022-12-11 18:28:19 +02:00
Motiejus Jakštys 70ef4dd54e remove zig_include_root
It is now consistent across all 5 platforms we support:

    $ for x in *324*.tar.xz; do tar -tf $x | grep libc/ | head -5; done
    zig-0.11.0-dev.324+f61c5f3f5/lib/libc/
    zig-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/
    zig-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/
    zig-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/ntddstor-uuid.c
    zig-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/bth-uuid.c
    zig-linux-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/
    zig-linux-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/
    zig-linux-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/
    zig-linux-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/ntddstor-uuid.c
    zig-linux-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/bth-uuid.c
    zig-linux-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/
    zig-linux-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/
    zig-linux-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/
    zig-linux-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/ntddstor-uuid.c
    zig-linux-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/bth-uuid.c
    zig-macos-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/
    zig-macos-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/
    zig-macos-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/
    zig-macos-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/ntddstor-uuid.c
    zig-macos-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/bth-uuid.c
    zig-macos-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/
    zig-macos-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/
    zig-macos-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/
    zig-macos-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/ntddstor-uuid.c
    zig-macos-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/bth-uuid.c
    $ for x in *324*.zip; do unzip -l $x | grep libc/ | head -5; done
            0  2022-11-28 19:26   zig-windows-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/
            0  2022-11-28 19:26   zig-windows-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/darwin/
       301018  2022-11-28 19:26   zig-windows-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/darwin/libSystem.11.tbd
       309693  2022-11-28 19:26   zig-windows-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/darwin/libSystem.12.tbd
       326079  2022-11-28 19:26   zig-windows-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/darwin/libSystem.13.tbd
2022-12-07 12:17:44 +02:00
Motiejus Jakštys 6bd588c07b upgrade to zig 0.11.0-dev.324+f61c5f3f5
includes a zig upstream fix for absolute directories: https://github.com/ziglang/zig/issues/13050
2022-11-29 13:51:57 +02:00
Motiejus Jakštys 88e7e47ed2 zig_include_root: fixes
- do not pass where it's unnecessary
- remove trailing slash. That conflicts with ziglang/zig/pull/13596
2022-11-21 06:08:22 +02:00
Motiejus Jakštys dd4239407e lint 2022-11-17 15:47:46 +02:00
Motiejus Jakštys 87fb993408 zig build: add -target automatically 2022-11-17 14:04:25 +02:00
Motiejus Jakštys d195402030 zig build-exe, build-lib, build-obj
This change makes it possible to build zig sources from Bazel rules by
calling the right tool.
2022-11-17 13:51:31 +02:00
Motiejus Jakštys 50103497f6 upgrade zig sdk
The uber1 was broken on x86_64-macos
2022-10-19 11:38:46 +03:00
Motiejus Jakštys ae2746ebeb zig_include_root: support windows too
No real change, just make the logic consistent with Linux/Darwin.

This prepares us for windows-aarch64, which will probably have lib in
lib/zig.
2022-10-13 06:11:25 +03:00
Motiejus Jakštys fed463e1d0 bump zig to 4301+uber1
This is a patched zig release with 13051[1]. While we are waiting for a
resolution on #13050 (which @andrewrk is working on).

[1]: https://patch-diff.githubusercontent.com/raw/ziglang/zig/pull/13051
2022-10-13 05:57:56 +03:00
Motiejus Jakštys e5eff0c911 zig include root: it depends on the OS
Previously the toolchain wrappers used to assume the zig_lib_dir is
always in "lib". However, it depends on the OS: macos-x86_64 and
linux-aarch64 have it in "lib/zig".

Update the wrapper scripts to reflect that.
2022-10-11 14:57:44 +03:00
Motiejus Jakštys 5743865c26 Revert "resolve ZIG_LIB_DIR before passing it to zig"
This reverts commit 3a8c2eecfe.

Causes the dependency paths to be absolute.
2022-10-05 14:48:19 +03:00
Motiejus Jakštys 3a8c2eecfe resolve ZIG_LIB_DIR before passing it to zig
That way the zig's cache keys will be the same for all sandboxes.

Also see https://github.com/ziglang/zig/issues/13050#issuecomment-1268074032
2022-10-05 14:24:19 +03:00
Motiejus Jakštys 50ca49181f Zig cache hacks/optimizations
Zig does not always share `libc++.a` and the glibc shims. We have observed
to be caused for 2 reasons:

- For some commands Go `chdir`s to `/tmp/`. This causes `ZIG_LIB_DIR` to be
  absolute, which blows the cache key to find the right `libc++.a` (because Zig
  thinks we are using a different lib dir to compile libc++). This is currently
  worked around in `toolchain/defs.bzl` by overfitting to Go and returning
  early from that particular invocation.
- Sometimes Bazel's sandbox messes up Zig's cache keys. If one runs without the
  sandbox (`--spawn_strategy=standalone`), the cache hit rate and thus the
  build time are much better.

This is actively investigated. I am adding `--spawn_strategy=standalone`
to CI to see the speedup that it provides. I will rollback it later.
2022-10-03 05:50:53 +03:00
Motiejus Jakštys f30a3afdde remove redundant flags
- -target
- linkopts
2022-10-01 23:27:58 +03:00
Motiejus Jakštys b1209d0165 gohack only in tools/<arch>/c++ 2022-10-01 18:53:38 +03:00
Motiejus Jakštys 8919c5b703 fix tool paths for different OSes/environments 2022-10-01 14:39:59 +03:00
Motiejus Jakštys 9b510810cc split all tools to per-codename subdirectories
Go ignores CFLAGS for some commands. That causes unnecessary build of
glibc shims (and libc++.a).

In the GCC days cross-compiler toolchains used to expose a "tool" per
architecture. Let's do the same here. Then Go cannot cheat with skipping
CFLAGS which we normally *always* expect.

The wrapper code is getting gnarly, I know. But it still fits in my head
somehow, so we can still keep adding to it.
2022-10-01 14:35:27 +03:00
Motiejus Jakštys e96f5bb59c lint 2022-10-01 05:31:50 +03:00
Motiejus Jakštys e9638443c6 tools wrapper: fix for windows
'common' is not a thing in Windows; but it needs to be defined.

This makes v1.0.0-rc3 broken for Windows, and not useful (yet) for
anyone else but me. Revert the README update too.
2022-10-01 05:12:45 +03:00
Motiejus Jakštys c4ab1cfaea buildifier 2022-09-30 15:17:49 +03:00
Motiejus Jakštys 9e8d89b40d CGo-specific: reduce unneeded compilation
This adds an ugly workaround for
https://go-review.googlesource.com/c/go/+/436884 ; fingers crossed.

Impact: ~20-25s reduced first build time for a particular
architecture+glibc.
2022-09-30 14:24:56 +03:00
Fabian Hahn 04b95526ae fix Windows host support with relative paths 2022-09-29 11:32:19 +03:00
Motiejus Jakštys 08b02cb7d2 rename absolute_path to zig_path
this is not an absolute path.
2022-09-27 15:37:28 +03:00
Motiejus Jakštys 0302fb630a defs: fix import paths
Empirically these need to come from most specfic to least specific.

The error message is as follows:

    In file included from test/c/main.c:1:
    In file included from external/zig_sdk/lib/libcxx/include/stdio.h:107:
    In file included from external/zig_sdk/lib/libc/include/generic-glibc/stdio.h:38:
    external/zig_sdk/lib/libc/include/generic-glibc/bits/types.h:139:3: error:
    # error
      ^
    external/zig_sdk/lib/libc/include/generic-glibc/bits/types.h:145:1: error: unknown type name '__STD_TYPE'
    __STD_TYPE __DEV_T_TYPE __dev_t;        /* Type of device numbers.  */

Dissected `generic-glibc/bits/types.h:#error`:

    #if __WORDSIZE == 32
    <...>
    # define __STD_TYPE		__extension__ typedef
    #elif __WORDSIZE == 64
    <...>
    # define __STD_TYPE		typedef
    #else
    # error
    #endif

So we do not have the `__WORDSIZE`. Where does that come from? Probably from a
directory that has an `x86_64` in it. How does that get included? Let's start
with `lib/libcxx/include/stdio.h`:

	16 #include_next <stdio.h>

Now previously our `c++` command line looked like this:
    external/zig_sdk/tools/c++ \
    <...>
    -Iexternal/zig_sdk/lib/include \
    -Iexternal/zig_sdk/lib/libcxx/include \
    -Iexternal/zig_sdk/lib/libcxxabi/include \
    -Iexternal/zig_sdk/lib/libunwind/include \
    -Iexternal/zig_sdk/lib/libc/include/generic-glibc \
    -Iexternal/zig_sdk/lib/libc/include/any-linux-any \
    -Iexternal/zig_sdk/lib/libc/include/x86_64-linux-gnu \
    -Iexternal/zig_sdk/lib/libc/include/x86_64-linux-any \
    -Iexternal/zig_sdk/lib/libc/include/x86-linux-any \
    -Iexternal/zig_sdk/glibc-hacks \
    <...>

So the next place it will find `stdio.h` is in `generic-glibc`, which already
uses the `__WORDSIZE`. If we make the "next" include to be the arch-specific
one instead of the generic-glibc, things start compiling again.

Fix the same fo musl.
2022-09-27 10:15:02 +03:00
Ken Micklas ab59c18d60 WIP: use relative paths for tools 2022-09-27 10:14:59 +03:00
Motiejus Jakštys f6b16c386f upgrade to zig 0.10.0-dev.4166+cae76d829
- llvm 15
- uses less space for /tmp/bazel-zig-cc
2022-09-25 13:51:59 +03:00
Motiejus Jakštys 9dbba15064 partially revert c023c217a5
The `toolchain/private/cc_toolchains.bzl` changes were added by accident.
2022-09-22 13:32:17 +03:00
Motiejus Jakštys 026d234bc2 Revert "bump to zig 0.10.0-dev.3999+fda6d4477"
This reverts commit f161caf870.

This version has a bug that's been fixed upstream, but a new nightly did
not come out yet: https://github.com/ziglang/zig/issues/12858
2022-09-22 11:57:39 +03:00
Motiejus Jakštys c023c217a5 contrib/own_zig.md: clarify zig-bootstrap instructions 2022-09-22 11:39:39 +03:00
Ken Micklas 64b6c702db Fix include root path for macos-aarch64 2022-09-15 22:19:06 +03:00
Ken Micklas ba0b0376ea Move Label dependencies above archive fetching to reduce restarts 2022-09-15 22:18:43 +03:00
Motiejus Jakštys f161caf870 bump to zig 0.10.0-dev.3999+fda6d4477
llvm 15
2022-09-14 11:54:04 +03:00
Motiejus Jakštys bf32b5ee92 repository_rule: depend on $BAZEL_ZIG_CC_CACHE_PREFIX
Otherwise the repository doesn't get re-downloaded/wiped when this
parameter changes.
2022-09-12 13:46:07 +03:00
Motiejus Jakštys 2f0ae4affa target_musl: remove linkopts
These flags were added in this commit:

    commit 58a04fbfec
    Author: Motiejus Jakštys <motiejus@jakstys.lt>
    Date:   Tue Jun 15 09:10:15 2021 +0300

        bump zig

I am still quite puzzled on why I did this and in such a commit: it
bumps the zig version and changes the linker flag.

Working alone, experimentation time? Probably something like this.
Anyhow, this is a good example of the real value of code reviews.
2022-09-02 13:21:39 +03:00
Motiejus Jakštys e1d699f7ed upgrade to 0.10.0-dev.3838+77f31ebbb 2022-09-01 09:28:43 +03:00
Motiejus Jakštys 79674a1d96 replace /usr/bin/env sh with /bin/sh
/bin/sh exists in NixOS and in fact is the only file in `/bin`:

    $ docker run -ti --rm nixos/nix ls /bin /usr/bin
    /bin:
    sh

    /usr/bin:
    env

More context in
https://lists.sr.ht/~motiejus/bazel-zig-cc/%3CCAFVMu-pvNx%2BpeQYdge_fvmSGrHDmn78VmoYwTxkDgMjbzfVAwQ%40mail.gmail.com%3E
2022-08-30 09:21:27 +03:00
Motiejus Jakštys 342b239bb9 nit: style of cache prefix definition 2022-08-29 14:44:38 +03:00
Motiejus Jakštys e861c5b4be cache_prefix: clean up OS logic
'Windows' and 'Other' and guessing of cache prefix was convoluted
2022-08-29 13:01:09 +03:00
Fabian Hahn a410141ca4 add cache prefix configuration for Windows 2022-08-28 20:18:21 +03:00
Motiejus Jakštys 15f9578591 use a specific template if cache prefix is known 2022-08-26 10:55:29 +03:00
Ken Micklas 0612c6ed9c Support configuring cache prefix with an environment variable
The user may want to build with `--sandbox_tmpfs_path=/tmp`, in which
case the Zig cache will not be shared between sandboxes, massively
slowing down the build. With this feature, the user can pass flags
like `--repo_env=BAZEL_ZIG_CC_CACHE_PREFIX=$HOME/.cache/bazel-zig-cc`
together with `--sandbox_writable_path=$HOME/.cache/bazel-zig-cc` to
put the cache somewhere else which can be shared between sandboxes.
2022-08-26 10:47:25 +03:00
Motiejus Jakštys 23d9b9d39e [macos] remove libc/include/{}-macos.{}-gnu
This seems to be absent in the Zig SDK for macos targets.

Reported-by: Mike Eastham <meastham@tecton.ai>
2022-08-23 18:12:02 +03:00
Luis Holanda f13e59a30a Add missing include path for MacOS platforms
While integrating the project into our Bazel workspace, we had some
problems while building protoc in an Intel MacBook.

From my testing, the issue seems to be caused by missing include path
in the toolchain configuration to the `-none` variant of the headers.
More details below.

The issue can be reproduced directly inside `bazel-zig-cc` by including
the protobuf workspace as a dependency:

+http_archive(
+    name = "com_google_protobuf",
+    sha256 = "2d9084d3dd13b86ca2e811d2331f780eb86f6d7cb02b405426e3c80dcbfabf25",
+    strip_prefix = "protobuf-3.21.1",
+    url = "https://github.com/protocolbuffers/protobuf/archive/v3.21.1.zip",
+)
+
+load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
+
+protobuf_deps()

And then executing:

bazel build --platforms @zig_sdk//platform:darwin_amd64 @com_google_protobuf//:protoc

The command fails with the same error both in native compilation and
cross-compiling from Linux. The error:

ERROR: <output_base>/external/zlib/BUILD.bazel:37:11: Compiling compress.c failed: undeclared inclusion(s) in rule '@zlib//:zlib':
this rule is missing dependency declarations for the following files included by 'compress.c':
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/sys/cdefs.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/sys/_symbol_aliasing.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/machine/limits.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/i386/limits.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/i386/_limits.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/sys/syslimits.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/machine/types.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/i386/types.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/i386/_types.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/sys/_types/_int8_t.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/sys/_types/_uintptr_t.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/machine/_types.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/sys/_pthread/_pthread_types.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/machine/endian.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/i386/endian.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/libkern/_OSByteOrder.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/libkern/i386/_OSByteOrder.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/sys/_types/_fd_def.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/Availability.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/AvailabilityInternal.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/sys/_pthread/_pthread_attr_t.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/sys/_pthread/_pthread_cond_t.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/sys/_pthread/_pthread_condattr_t.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/sys/_pthread/_pthread_rwlock_t.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/sys/_pthread/_pthread_rwlockattr_t.h'
  '<output_base>/external/zig_sdk/lib/libc/include/x86_64-macos.10-none/sys/_pthread/_pthread_t.h'
Target @com_google_protobuf//:protoc failed to build

(<output_base> replaced manually to keep the description sane)

Note that the specific rule included in the error message will vary due
to parallelism, but the error is always the same and the missing paths
always have the same prefix (up to `x86_64-macos.10-none`).

After applying the patch, we confirmed that both native and cross compilations
works as expected and can properly build protoc.

Signed-off-by: Luis Holanda <luiscmholanda@gmail.com>
2022-08-23 18:11:54 +03:00
Motiejus Jakštys db7beefbe4 c++: fall back to cache dir in /tmp
rules_go is making calls that do `env - <...>` and are setting neither
TMPDIR nor HOME. So we should always fallback somewhere.
2022-08-12 09:25:37 +03:00
Motiejus Jakštys e50d1597f4 upgrade zig sdk to one of last night 2022-08-12 05:38:34 +03:00
Motiejus Jakštys 771fcc5bc6 bump to go 1.19
Removing --no-gc-sections, because that's now in Go. Whoo!
2022-08-12 05:21:24 +03:00
Luis Holanda c4125cf13b Support NixOS and others non-FHS Linux distros
As it expects Bash to be at `/bin/bash`, the current implementation
fails to execute on NixOS, given that there bash is in a non-standard
path (i.g. `/nix/store/<hash>-bash-interactive-<version>/bin/bash`).

This patch specifically changes `/bin/bash` paths to use `/usr/bin/env bash`,
which should give the correct path in every Unix system.

Signed-off-by: Luis Holanda <luiscmholanda@gmail.com>
2022-07-28 12:44:57 -07:00
Motiejus Jakštys 64a7007a23 fastbuild: remove -O0
this is the default in zig cc now
2022-07-12 14:53:02 +03:00