109 lines
4.6 KiB
Markdown
109 lines
4.6 KiB
Markdown
bazel-zig-cc and llvm
|
|
---------------------
|
|
|
|
bazel-zig-cc has a performance issue when compiling many files. This repository
|
|
reproduces that.
|
|
|
|
The test
|
|
--------
|
|
|
|
Compiles 64 small binaries with [bazel-zig-cc][1] and [llvm14][2]. The tests
|
|
were run on an x86_64 8-core machine running Ubuntu 22.04. The llvm's sysroot
|
|
is this:
|
|
|
|
sha256 = "84656a6df544ecef62169cfe3ab6e41bb4346a62d3ba2a045dc5a0a2ecea94a3",
|
|
urls = ["https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/2202c161310ffde63729f29d27fe7bb24a0bc540/debian_stretch_amd64_sysroot.tar.xz"],
|
|
|
|
... but with extra ~8k files, so the number of files in the sysroot is exactly
|
|
the same as in zig sdk.
|
|
|
|
Baseline
|
|
--------
|
|
|
|
Baseline llvm14 without sandbox: 12 seconds:
|
|
|
|
bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
|
|
bazel clean; bazel shutdown
|
|
bazel --batch build --spawn_strategy=local --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux ...
|
|
INFO: Elapsed time: 11.971s, Critical Path: 0.41s
|
|
|
|
zig cc without the sandbox. 15 seconds:
|
|
|
|
bazel build --spawn_strategy=local --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
|
|
bazel clean; bazel shutdown
|
|
bazel --batch build --spawn_strategy=local --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
|
|
INFO: Elapsed time: 15.631s, Critical Path: 1.99s
|
|
|
|
Cost of Sandbox
|
|
---------------
|
|
|
|
llvm 14:
|
|
|
|
bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
|
|
bazel clean; bazel shutdown
|
|
bazel --batch build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux ...
|
|
INFO: Elapsed time: 109.677s, Critical Path: 13.60s
|
|
|
|
zig cc:
|
|
|
|
bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
|
|
bazel clean; bazel shutdown
|
|
bazel --batch build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
|
|
INFO: Elapsed time: 116.370s, Critical Path: 16.23s
|
|
|
|
Possible mitigation: Sandboxfs
|
|
------------------------------
|
|
|
|
No dice.
|
|
|
|
llvm 14: does not build at all:
|
|
|
|
$ bazel build --experimental_sandboxfs_path=sandboxfs --experimental_use_sandboxfs --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
|
|
ERROR: /code/test-zigcc/BUILD:4:14: Compiling main.cc failed: (Exit 5): cc_wrapper.sh failed: error executing command external/llvm_toolchain_with_sysroot/bin/cc_wrapper.sh -U_FORTIFY_SOURCE '--target=x86_64-unknown-linux-gnu' -U_FORTIFY_SOURCE -fstack-protector -fno-omit-frame-pointer -fcolor-diagnostics -Wall ... (remaining 29 arguments skipped)
|
|
|
|
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
|
|
ERROR: could not find clang; PWD="/proc/self/cwd"; PATH="<...>".
|
|
Target //:zigzag-0 failed to build
|
|
Use --verbose_failures to see the command lines of failed build steps.
|
|
INFO: Elapsed time: 7.578s, Critical Path: 0.26s
|
|
INFO: 3 processes: 3 internal.
|
|
FAILED: Build did NOT complete successfully
|
|
|
|
|
|
zig cc:
|
|
|
|
bazel build --experimental_sandboxfs_path=sandboxfs --experimental_use_sandboxfs --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
|
|
bazel clean; bazel shutdown
|
|
bazel --batch build --experimental_sandboxfs_path=sandboxfs --experimental_use_sandboxfs --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
|
|
INFO: Elapsed time: 126.613s, Critical Path: 17.63s
|
|
|
|
|
|
Possible mitigation: `--experimental_reuse_sandbox_directories`
|
|
---------------------------------------------------------------
|
|
|
|
Dice.
|
|
|
|
llvm 14:
|
|
|
|
bazel build --experimental_reuse_sandbox_directories --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
|
|
bazel clean; bazel shutdown
|
|
bazel --batch build --experimental_reuse_sandbox_directories --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux ...
|
|
INFO: Elapsed time: 25.252s, Critical Path: 4.44s
|
|
|
|
zig cc:
|
|
|
|
bazel build --experimental_reuse_sandbox_directories --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
|
|
bazel clean; bazel shutdown
|
|
bazel --batch build --experimental_reuse_sandbox_directories --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
|
|
INFO: Elapsed time: 29.132s, Critical Path: 3.88s
|
|
|
|
|
|
Flame graphs and discussion
|
|
---------------------------
|
|
|
|
Flame graphs and more tests are in results/. As of 2022-12-11 most of the
|
|
overhead comes from creating and deleting the sandboxes.
|
|
|
|
[1]: https://sr.ht/~motiejus/bazel-zig-cc
|
|
[2]: https://github.com/grailbio/bazel-toolchain
|