bazel-zig-cc and llvm

bazel-zig-cc has a performance issue when compiling many files. This repository reproduces that.

The test

Compiles 64 small binaries with bazel-zig-cc and llvm14. The tests were run on an x86_64 8-core machine running Ubuntu 22.04.

Steps to reproduce

Baseline is llvm14: 12-13 seconds:

bazel --batch clean; bazel --batch build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux ...
INFO: Elapsed time: 12.454s, Critical Path: 1.11s

zig cc without the sandbox. Which means different invocations of zig cc will see that all files in zig_lib_dir are the same file. 17 seconds:

bazel --batch clean; bazel --batch build --spawn_strategy=local  --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
INFO: Elapsed time: 17.021s, Critical Path: 1.67s

zig cc with --sandbox_base=/dev/shm: all sandbox files, including the zig_sdk, are put to /dev/shm. However, they are all symlinks, which means zig will re-hash all it's dependencies. 42 seconds:

bazel --batch clean; bazel --batch build --sandbox_base=/dev/shm --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
INFO: Elapsed time: 42.251s, Critical Path: 5.26s

zig cc plain: 142 seconds. Sandbox is on a real disk, which means it will take even longer to re-hash all its dependencies:

bazel --batch clean; bazel --batch build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
INFO: Elapsed time: 142.264s, Critical Path: 20.65s

Similar to --config=hermetic-sandbox. Which means there is something else at play, not only re-hashing the sandbox files:

bazel --batch clean; bazel --batch build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
INFO: Elapsed time: 136.957s, Critical Path: 18.14s