motiejus/hermetic_cc_toolchain
1
Fork 0
Code Releases Activity
nix
hermetic_cc_toolchain/ci/launcher

28 lines
742 B
Plaintext
Raw Permalink Normal View History

zig launcher: replace shell wrappers with a binary Until now we needed to maintain two versions of the zig launcher: one for Windows and one for everything else. This was problematic for two reasons: 1. I do not know powershell and thus keep breaking the Windows wrapper all the time (see git history of Fabian fixing stuff that I broke). 2. This makes bazel-zig-cc dependent on the system shell, making it not really hermetic. So the recently added `--experimental_use_hermetic_linux_sandbox` does not work with bazel-zig-cc, unless we bind-mount a bunch of stuff: `/usr`, `/bin`, `/lib`, `/usr/lib:/lib`, `/usr/lib64:/lib64` and `/proc`. Switching to a Zig-based wrapper solves both issues, and we can do this: bazel build "$@" \ --experimental_use_hermetic_linux_sandbox \ --sandbox_add_mount_pair=/proc \ <...> Zig itself still depends on `/proc` for `/proc/self/exe`, so we need to keep that. I will look into reducing even that dependency separately. Not all is nice and shiny though: this commit replaces ~80 LOC worth of shell scripts wrappers with a singe ~300 LOC zig program, which is arguably harder to understand. However, it is easier to change, at least for me, because it's a single file with unit tests! Most importantly, the gnarly code (which resolves paths and sets environment variables) is cross-platform. Thanks to Fabian Hahn for testing this on Windows and pointing out errors.
2022-12-20 17:00:50 +02:00
#!/usr/bin/env bash
Re-license portions of the code to Apache 2.0 Your old version of the software released under the old license can still be used under the terms of the old license. I have received the following statement from the past contributors Luis Holanda, Jeremy Volkman and Fabian Hahn: I hereby confirm that I am the copyright holder or authorized by the copyright holder of this contribution. As such I hereby confirm that all contributions made to bazel-zig-cc by me or on behalf of me, hereby is licensed under the Apache 2.0 License [http://www.apache.org/licenses/LICENSE-2.0]. I am aware that my previous contributions will still be available under the MIT license as well. I confirm that I am aware of the bazel-zig-cc teams intention to release bazel-zig-cc under the Apache 2.0 License from release [1.0] and onwards, and that the bazel-zig-cc no longer will accept contributions made under the MIT and that any future submissions from myself will be considered to be licensed under Apache 2.0 unless I expressly state otherwise. Copyright and re-licensing for Google and Uber employees has been resolved internally. Since not all contributors took action to re-license the code yet, portions of bazel-zig-cc remain licensed as MIT. I have started this project during my personal time with my personal resources. I am now handing over all copyrights to this code to Uber Technologies, Inc.
2023-03-02 23:42:35 +02:00
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the Apache License, Version 2.0
zig launcher: replace shell wrappers with a binary Until now we needed to maintain two versions of the zig launcher: one for Windows and one for everything else. This was problematic for two reasons: 1. I do not know powershell and thus keep breaking the Windows wrapper all the time (see git history of Fabian fixing stuff that I broke). 2. This makes bazel-zig-cc dependent on the system shell, making it not really hermetic. So the recently added `--experimental_use_hermetic_linux_sandbox` does not work with bazel-zig-cc, unless we bind-mount a bunch of stuff: `/usr`, `/bin`, `/lib`, `/usr/lib:/lib`, `/usr/lib64:/lib64` and `/proc`. Switching to a Zig-based wrapper solves both issues, and we can do this: bazel build "$@" \ --experimental_use_hermetic_linux_sandbox \ --sandbox_add_mount_pair=/proc \ <...> Zig itself still depends on `/proc` for `/proc/self/exe`, so we need to keep that. I will look into reducing even that dependency separately. Not all is nice and shiny though: this commit replaces ~80 LOC worth of shell scripts wrappers with a singe ~300 LOC zig program, which is arguably harder to understand. However, it is easier to change, at least for me, because it's a single file with unit tests! Most importantly, the gnarly code (which resolves paths and sets environment variables) is cross-platform. Thanks to Fabian Hahn for testing this on Windows and pointing out errors.
2022-12-20 17:00:50 +02:00
set -xeuo pipefail
More CI (#3) - ci only: use a separate zig cache dir for each run - use `tools/bazel` everywhere. - remove arm64 tests. They don't give enough value to be worth the brittle environment. - remove windows execution, except for launcher. Ditto. I will probably bring them back.
2023-03-06 17:32:48 +02:00
echo "--- which zig"
wip nix $ ./ci/test bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8) + cache_prefix=/tmp/bazel-zig-cc + echo '--- build a single target with very hermetic sandbox' --- build a single target with very hermetic sandbox + bazel build --experimental_use_hermetic_linux_sandbox --sandbox_writable_path=/tmp/bazel-zig-cc --sandbox_add_mount_pair=/proc //test/c:which_libc_linux_amd64_gnu.2.19 /nix/store/96ky1zdkpq871h2dlk198fz0zvklr1dr-bash-5.1-p16/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8) /nix/store/96ky1zdkpq871h2dlk198fz0zvklr1dr-bash-5.1-p16/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8) INFO: Analyzed target //test/c:which_libc_linux_amd64_gnu.2.19 (0 packages loaded, 0 targets configured). INFO: Found 1 target... Target //test/c:which_libc_linux_amd64_gnu.2.19 up-to-date: bazel-bin/test/c/which_libc_/libc_aware/platform:linux_amd64_gnu.2.19 INFO: Elapsed time: 1.147s, Critical Path: 0.02s INFO: 1 process: 1 internal. INFO: Build completed successfully, 1 total action + echo '--- bazel test ...' --- bazel test ... + bazel test ... /nix/store/96ky1zdkpq871h2dlk198fz0zvklr1dr-bash-5.1-p16/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8) /nix/store/96ky1zdkpq871h2dlk198fz0zvklr1dr-bash-5.1-p16/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8) ERROR: /home/motiejus/.cache/bazel/_bazel_motiejus/6ec14a6813f13f2c8f1d8cce5637da2e/external/local_jdk/BUILD.bazel:2:10: in fail_rule rule @local_jdk//:jdk: Traceback (most recent call last): File "/home/motiejus/.cache/bazel/_bazel_motiejus/6ec14a6813f13f2c8f1d8cce5637da2e/external/bazel_tools/tools/jdk/fail_rule.bzl", line 19, column 13, in _fail_rule_impl fail("%s %s" % (ctx.attr.header, ctx.attr.message)) Error in fail: Auto-Configuration Error: Cannot find Java binary bin/java in /home/motiejus/.cache/bazel/_bazel_motiejus/install/1349a6142c7ce93610b5832496a88870/embedded_tools/tools/jdk/nosystemjdk; either correct your JAVA_HOME, PATH or specify Java from remote repository (e.g. --java_runtime_version=remotejdk_11 ERROR: /home/motiejus/.cache/bazel/_bazel_motiejus/6ec14a6813f13f2c8f1d8cce5637da2e/external/local_jdk/BUILD.bazel:2:10: Analysis of target '@local_jdk//:jdk' failed ERROR: Analysis of target '//.direnv/flake-inputs/hfjyanh5shjsx6m3h1k8gw61s3azkk68-source/test/cgo:cgo_test_linux_amd64_gnu.2.19' failed; build aborted: INFO: Elapsed time: 1.853s INFO: 0 processes. FAILED: Build did NOT complete successfully (70 packages loaded, 11766 targets configured) Fetching repository @nixpkgs_go; Building Nix derivation ERROR: Couldn't start the build. Unable to run tests motiejus@mtwork:/code/bazel-zig-cc$
2023-03-22 13:25:20 +02:00
ZIG=${ZIG:-$(bazel run "$@" --run_under=echo @zig_sdk//:zig)}
zig launcher: replace shell wrappers with a binary Until now we needed to maintain two versions of the zig launcher: one for Windows and one for everything else. This was problematic for two reasons: 1. I do not know powershell and thus keep breaking the Windows wrapper all the time (see git history of Fabian fixing stuff that I broke). 2. This makes bazel-zig-cc dependent on the system shell, making it not really hermetic. So the recently added `--experimental_use_hermetic_linux_sandbox` does not work with bazel-zig-cc, unless we bind-mount a bunch of stuff: `/usr`, `/bin`, `/lib`, `/usr/lib:/lib`, `/usr/lib64:/lib64` and `/proc`. Switching to a Zig-based wrapper solves both issues, and we can do this: bazel build "$@" \ --experimental_use_hermetic_linux_sandbox \ --sandbox_add_mount_pair=/proc \ <...> Zig itself still depends on `/proc` for `/proc/self/exe`, so we need to keep that. I will look into reducing even that dependency separately. Not all is nice and shiny though: this commit replaces ~80 LOC worth of shell scripts wrappers with a singe ~300 LOC zig program, which is arguably harder to understand. However, it is easier to change, at least for me, because it's a single file with unit tests! Most importantly, the gnarly code (which resolves paths and sets environment variables) is cross-platform. Thanks to Fabian Hahn for testing this on Windows and pointing out errors.
2022-12-20 17:00:50 +02:00
More CI (#3) - ci only: use a separate zig cache dir for each run - use `tools/bazel` everywhere. - remove arm64 tests. They don't give enough value to be worth the brittle environment. - remove windows execution, except for launcher. Ditto. I will probably bring them back.
2023-03-06 17:32:48 +02:00
echo "--- compile launcher.zig for various architectures"
CI: fix tests
2023-01-31 04:01:01 +02:00
for target in \
aarch64-linux-gnu.2.19 \
aarch64-macos-none \
x86_64-linux-gnu.2.19 \
x86_64-macos-none \
x86_64-windows-gnu
do
$ZIG build-exe -fno-emit-bin -target $target toolchain/launcher.zig
done
More CI (#3) - ci only: use a separate zig cache dir for each run - use `tools/bazel` everywhere. - remove arm64 tests. They don't give enough value to be worth the brittle environment. - remove windows execution, except for launcher. Ditto. I will probably bring them back.
2023-03-06 17:32:48 +02:00
echo "--- zig fmt --check toolchain/launcher.zig"
launcher: remove --color=on
2023-02-24 12:01:49 +02:00
$ZIG fmt --check toolchain/launcher.zig
ci/launcher: check fmt
2023-02-24 11:53:10 +02:00
More CI (#3) - ci only: use a separate zig cache dir for each run - use `tools/bazel` everywhere. - remove arm64 tests. They don't give enough value to be worth the brittle environment. - remove windows execution, except for launcher. Ditto. I will probably bring them back.
2023-03-06 17:32:48 +02:00
echo "--- zig test toolchain/launcher.zig"
zig launcher: replace shell wrappers with a binary Until now we needed to maintain two versions of the zig launcher: one for Windows and one for everything else. This was problematic for two reasons: 1. I do not know powershell and thus keep breaking the Windows wrapper all the time (see git history of Fabian fixing stuff that I broke). 2. This makes bazel-zig-cc dependent on the system shell, making it not really hermetic. So the recently added `--experimental_use_hermetic_linux_sandbox` does not work with bazel-zig-cc, unless we bind-mount a bunch of stuff: `/usr`, `/bin`, `/lib`, `/usr/lib:/lib`, `/usr/lib64:/lib64` and `/proc`. Switching to a Zig-based wrapper solves both issues, and we can do this: bazel build "$@" \ --experimental_use_hermetic_linux_sandbox \ --sandbox_add_mount_pair=/proc \ <...> Zig itself still depends on `/proc` for `/proc/self/exe`, so we need to keep that. I will look into reducing even that dependency separately. Not all is nice and shiny though: this commit replaces ~80 LOC worth of shell scripts wrappers with a singe ~300 LOC zig program, which is arguably harder to understand. However, it is easier to change, at least for me, because it's a single file with unit tests! Most importantly, the gnarly code (which resolves paths and sets environment variables) is cross-platform. Thanks to Fabian Hahn for testing this on Windows and pointing out errors.
2022-12-20 17:00:50 +02:00
# until bazel-zig-cc gets a zig toolchain, run launcher's unit tests here.
$ZIG test toolchain/launcher.zig
Copy Permalink