bazel-zig-cc and llvm

bazel-zig-cc has a performance issue when compiling many files. This repository reproduces that.

The test

Compiles 64 small binaries with bazel-zig-cc and llvm14. The tests were run on an x86_64 8-core machine running Ubuntu 22.04. The llvm's sysroot is this:

sha256 = "84656a6df544ecef62169cfe3ab6e41bb4346a62d3ba2a045dc5a0a2ecea94a3",
urls = ["https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/2202c161310ffde63729f29d27fe7bb24a0bc540/debian_stretch_amd64_sysroot.tar.xz"],

... but with extra ~8k files, so the number of files in the sysroot is exactly the same as in zig sdk.

Baseline

Baseline llvm14 without sandbox: 12 seconds:

bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
bazel clean; bazel shutdown
bazel --batch build --spawn_strategy=local --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux ...
INFO: Elapsed time: 11.971s, Critical Path: 0.41s

zig cc without the sandbox. 15 seconds:

bazel build --spawn_strategy=local --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
bazel clean; bazel shutdown
bazel --batch build --spawn_strategy=local --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
INFO: Elapsed time: 15.631s, Critical Path: 1.99s

Cost of Sandbox

llvm 14:

bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
bazel clean; bazel shutdown
bazel --batch build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux ...
INFO: Elapsed time: 109.677s, Critical Path: 13.60s

zig cc:

bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
bazel clean; bazel shutdown
bazel --batch build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
INFO: Elapsed time: 116.370s, Critical Path: 16.23s

Possible mitigation: Sandboxfs

No dice.

llvm 14:

bazel build --experimental_sandboxfs_path=sandboxfs --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
bazel clean; bazel shutdown
bazel --batch build --experimental_sandboxfs_path=sandboxfs --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux ...
INFO: Elapsed time: 96.803s, Critical Path: 12.42s

zig cc:

bazel build --experimental_sandboxfs_path=sandboxfs --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
bazel clean; bazel shutdown
bazel --batch build --experimental_sandboxfs_path=sandboxfs --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
INFO: Elapsed time: 109.347s, Critical Path: 13.94s

Possible mitigation: --experimental_reuse_sandbox_directories

Dice.

llvm 14:

bazel build --experimental_reuse_sandbox_directories --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
bazel clean; bazel shutdown
bazel --batch build --experimental_reuse_sandbox_directories --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux ...
INFO: Elapsed time: 25.252s, Critical Path: 4.44s

zig cc:

bazel build --experimental_reuse_sandbox_directories --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
bazel clean; bazel shutdown
bazel --batch build --experimental_reuse_sandbox_directories --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
INFO: Elapsed time: 29.132s, Critical Path: 3.88s

Flame graphs and discussion

Flame graphs and more tests are in results/. As of 2022-12-11 most of the overhead comes from creating and deleting the sandboxes.