bazel-zig-cc and llvm

bazel-zig-cc has a performance issue when compiling many files. This repository reproduces that.

The test

Compiles 64 small binaries with bazel-zig-cc and llvm14. The tests were run on an x86_64 8-core machine running Ubuntu 22.04. The llvm's sysroot is this:

sha256 = "84656a6df544ecef62169cfe3ab6e41bb4346a62d3ba2a045dc5a0a2ecea94a3",
urls = ["https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/2202c161310ffde63729f29d27fe7bb24a0bc540/debian_stretch_amd64_sysroot.tar.xz"],

... but with extra ~8k files, so the number of files in the sysroot is exactly the same as in zig sdk.

Baseline

Baseline llvm14 without sandbox: 12 seconds:

bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
bazel clean; bazel shutdown
bazel --batch build --spawn_strategy=local --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux ...
INFO: Elapsed time: 13.268s, Critical Path: 1.19s

zig cc without the sandbox. 15 seconds:

bazel build --spawn_strategy=local --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
bazel clean; bazel shutdown
bazel --batch build --spawn_strategy=local --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
INFO: Elapsed time: 23.836s, Critical Path: 3.47s

Cost of Sandbox

llvm 14:

bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
bazel clean; bazel shutdown
bazel --batch build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux ...
INFO: Elapsed time: 89.776s, Critical Path: 12.01s

zig cc:

bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
bazel clean; bazel shutdown
bazel --batch build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
INFO: Elapsed time: 58.919s, Critical Path: 8.38s

Possible mitigation: Sandboxfs

No dice.

llvm 14: does not build at all:

$ bazel build --experimental_sandboxfs_path=sandboxfs --experimental_use_sandboxfs --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
ERROR: /code/test-zigcc/BUILD:4:14: Compiling main.cc failed: (Exit 5): cc_wrapper.sh failed: error executing command external/llvm_toolchain_with_sysroot/bin/cc_wrapper.sh -U_FORTIFY_SOURCE '--target=x86_64-unknown-linux-gnu' -U_FORTIFY_SOURCE -fstack-protector -fno-omit-frame-pointer -fcolor-diagnostics -Wall ... (remaining 29 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
ERROR: could not find clang; PWD="/proc/self/cwd"; PATH="<...>".
Target //:zigzag-0 failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 7.578s, Critical Path: 0.26s
INFO: 3 processes: 3 internal.
FAILED: Build did NOT complete successfully

zig cc:

bazel build --experimental_sandboxfs_path=sandboxfs --experimental_use_sandboxfs --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
bazel clean; bazel shutdown
bazel --batch build --experimental_sandboxfs_path=sandboxfs --experimental_use_sandboxfs --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
INFO: Elapsed time: 126.613s, Critical Path: 17.63s

Possible mitigation: --experimental_reuse_sandbox_directories

Dice.

llvm 14:

bazel build --experimental_reuse_sandbox_directories --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
bazel clean; bazel shutdown
bazel --batch build --experimental_reuse_sandbox_directories --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux ...
INFO: Elapsed time: 18.335s, Critical Path: 2.00s

zig cc:

bazel build --experimental_reuse_sandbox_directories --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
bazel clean; bazel shutdown
bazel --batch build --experimental_reuse_sandbox_directories --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
INFO: Elapsed time: 34.777s, Critical Path: 4.71s

Flame graphs and discussion

Flame graphs and more tests are in results/. As of 2022-12-11 most of the overhead comes from creating and deleting the sandboxes.