bazel-zig-cc and llvm

bazel-zig-cc has a performance issue when compiling many files. This repository reproduces that.

The test

Compiles 64 small binaries with bazel-zig-cc and llvm14. The tests were run on an x86_64 8-core machine running Ubuntu 22.04.

Results

Benchmark 1: bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=local //...
  Time (mean ± σ):      9.257 s ±  0.287 s    [User: 0.083 s, System: 0.026 s]
  Range (min … max):    8.818 s …  9.515 s    5 runs

Benchmark 2: bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --spawn_strategy=local //...
  Time (mean ± σ):      9.613 s ±  0.214 s    [User: 0.077 s, System: 0.038 s]
  Range (min … max):    9.418 s …  9.866 s    5 runs

Benchmark 3: bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=sandboxed --experimental_reuse_sandbox_directories //...
  Time (mean ± σ):     13.314 s ±  0.201 s    [User: 0.109 s, System: 0.028 s]
  Range (min … max):   13.093 s … 13.536 s    5 runs

Benchmark 4: bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --spawn_strategy=sandboxed --experimental_reuse_sandbox_directories //...
  Time (mean ± σ):     13.829 s ±  0.376 s    [User: 0.093 s, System: 0.046 s]
  Range (min … max):   13.573 s … 14.489 s    5 runs

Benchmark 5: bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=sandboxed //...
  Time (mean ± σ):     35.101 s ±  0.365 s    [User: 0.103 s, System: 0.028 s]
  Range (min … max):   34.747 s … 35.673 s    5 runs

Benchmark 6: bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --spawn_strategy=sandboxed //...
  Time (mean ± σ):     21.071 s ±  0.378 s    [User: 0.091 s, System: 0.034 s]
  Range (min … max):   20.611 s … 21.635 s    5 runs

Summary
                                  'bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=local //...' ran
    1.04 ± 0.04 times faster than 'bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28              --spawn_strategy=local //...'
    1.44 ± 0.05 times faster than 'bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=sandboxed --experimental_reuse_sandbox_directories //...'
    1.49 ± 0.06 times faster than 'bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28              --spawn_strategy=sandboxed --experimental_reuse_sandbox_directories //...'
    2.28 ± 0.08 times faster than 'bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28              --spawn_strategy=sandboxed //...'
    3.79 ± 0.12 times faster than 'bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=sandboxed //...'

Command	Mean [s]	Min [s]	Max [s]	Relative
bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=local //...	9.257 ± 0.287	8.818	9.515	1.00
bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --spawn_strategy=local //...	9.613 ± 0.214	9.418	9.866	1.04 ± 0.04
bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=sandboxed --experimental_reuse_sandbox_directories //...	13.314 ± 0.201	13.093	13.536	1.44 ± 0.05
bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --spawn_strategy=sandboxed --experimental_reuse_sandbox_directories //...	13.829 ± 0.376	13.573	14.489	1.49 ± 0.06
bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=sandboxed //...	35.101 ± 0.365	34.747	35.673	3.79 ± 0.12
bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --spawn_strategy=sandboxed //...	21.071 ± 0.378	20.611	21.635	2.28 ± 0.08

Explanation:

• --spawn_strategy=local is the baseline. This is the fastest possible approach (no sandbox), but does never runs in production.
• --spawn_strategy=sandboxed is the default for Bazel. This is what happens if one does not pass any arguments.
• --spawn_strategy=sandboxed --experimental_reuse_sandbox_directories is an optimization which is meaningful on sandboxes with many files. As of Bazel 6 this has been promoted to stable, so safe to use and, as one can see in the numbers, recommended with both bazel-zig-cc and a nontrivial sysroot. As a result, this is the most important benchmark to look at.

Flame graphs and discussion

Flame graphs in results/. Only of historical interest, because the original performance issues have been resolved with a combination of zig and bazel-zig-cc changes.