bazel-zig-cc and llvm
---------------------

bazel-zig-cc has a performance issue when compiling many files. This repository
reproduces that.

The test
--------

Compiles 64 small binaries with [bazel-zig-cc][1] and [llvm14][2]. The tests
were run on an x86_64 8-core machine running Ubuntu 22.04.

TLDR
----

Extracting the fastest meaningful benchmarks in all categories (from the
section below):

                                  'bazel build                                                                             --experimental_reuse_sandbox_directories //...' ran
    2.77 ± 0.08 times faster than 'bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --experimental_reuse_sandbox_directories //...'
    2.85 ± 0.07 times faster than 'bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28              --experimental_reuse_sandbox_directories //...'

This demonstrates that **adding a hermetic toolchain to a project caused 2.77x
slowdown on this project**. Note that we are not counting `llvm_toolchain//...`
as a real toolchain -- one needs a sysroot to compile anything meaningful
anyway (which zig bundles with the compiler).

Once we have determined that a hermetic C++ toolchain is required, bazel-zig-cc
is about 2.8% slower than its contender plain llvm14.

Results
-------

    Benchmark 1: bazel build  --spawn_strategy=local //...
      Time (mean ± σ):      4.813 s ±  0.140 s    [User: 0.015 s, System: 0.009 s]
      Range (min … max):    4.529 s …  5.007 s    10 runs
     
    Benchmark 2: bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux --spawn_strategy=local //...
      Time (mean ± σ):      6.921 s ±  0.238 s    [User: 0.013 s, System: 0.011 s]
      Range (min … max):    6.513 s …  7.279 s    10 runs
     
    Benchmark 3: bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=local //...
      Time (mean ± σ):      9.336 s ±  0.347 s    [User: 0.013 s, System: 0.012 s]
      Range (min … max):    8.968 s … 10.220 s    10 runs
     
    Benchmark 4: bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --spawn_strategy=local //...
      Time (mean ± σ):      9.311 s ±  0.190 s    [User: 0.014 s, System: 0.010 s]
      Range (min … max):    9.050 s …  9.667 s    10 runs
     
    Benchmark 5: bazel build  --experimental_reuse_sandbox_directories //...
      Time (mean ± σ):      4.726 s ±  0.051 s    [User: 0.011 s, System: 0.011 s]
      Range (min … max):    4.653 s …  4.793 s    10 runs
     
    Benchmark 6: bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux --experimental_reuse_sandbox_directories //...
      Time (mean ± σ):      7.674 s ±  0.370 s    [User: 0.012 s, System: 0.012 s]
      Range (min … max):    7.000 s …  8.492 s    10 runs
     
    Benchmark 7: bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --experimental_reuse_sandbox_directories //...
      Time (mean ± σ):     13.098 s ±  0.329 s    [User: 0.018 s, System: 0.006 s]
      Range (min … max):   12.365 s … 13.510 s    10 runs
     
    Benchmark 8: bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --experimental_reuse_sandbox_directories //...
      Time (mean ± σ):     13.465 s ±  0.292 s    [User: 0.014 s, System: 0.012 s]
      Range (min … max):   13.033 s … 13.897 s    10 runs
     
    Benchmark 9: bazel build  --spawn_strategy=sandboxed //...
      Time (mean ± σ):      4.766 s ±  0.179 s    [User: 0.014 s, System: 0.008 s]
      Range (min … max):    4.553 s …  5.224 s    10 runs
     
    Benchmark 10: bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux --spawn_strategy=sandboxed //...
      Time (mean ± σ):      8.666 s ±  0.116 s    [User: 0.014 s, System: 0.009 s]
      Range (min … max):    8.504 s …  8.883 s    10 runs
     
    Benchmark 11: bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=sandboxed //...
      Time (mean ± σ):     33.827 s ±  0.630 s    [User: 0.019 s, System: 0.010 s]
      Range (min … max):   32.121 s … 34.412 s    10 runs
     
      Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
     
    Benchmark 12: bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --spawn_strategy=sandboxed //...
      Time (mean ± σ):     20.396 s ±  0.330 s    [User: 0.016 s, System: 0.011 s]
      Range (min … max):   19.966 s … 20.939 s    10 runs
     
    Summary
                                      'bazel build                                                                             --experimental_reuse_sandbox_directories //...' ran
        1.01 ± 0.04 times faster than 'bazel build                                                                             --spawn_strategy=sandboxed               //...'
        1.02 ± 0.03 times faster than 'bazel build                                                                             --spawn_strategy=local                   //...'
        1.46 ± 0.05 times faster than 'bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux              --spawn_strategy=local                   //...'
        1.62 ± 0.08 times faster than 'bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux              --experimental_reuse_sandbox_directories //...'
        1.83 ± 0.03 times faster than 'bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux              --spawn_strategy=sandboxed               //...'
        1.97 ± 0.05 times faster than 'bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28              --spawn_strategy=local                   //...'
        1.98 ± 0.08 times faster than 'bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=local                   //...'
        2.77 ± 0.08 times faster than 'bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --experimental_reuse_sandbox_directories //...'
        2.85 ± 0.07 times faster than 'bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28              --experimental_reuse_sandbox_directories //...'
        4.32 ± 0.08 times faster than 'bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28              --spawn_strategy=sandboxed               //...'
        7.16 ± 0.15 times faster than 'bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=sandboxed               //...'


| Command                                                                                                                                  | Mean [s]       | Min [s] | Max [s] | Relative    |
|:-----------------------------------------------------------------------------------------------------------------------------------------|---------------:|--------:|--------:|------------:|
| `bazel build                                                                             --experimental_reuse_sandbox_directories //...` |  4.726 ± 0.051 |   4.653 |   4.793 | 1.00        |
| `bazel build                                                                             --spawn_strategy=sandboxed               //...` |  4.766 ± 0.179 |   4.553 |   5.224 | 1.01 ± 0.04 |
| `bazel build                                                                             --spawn_strategy=local                   //...` |  4.813 ± 0.140 |   4.529 |   5.007 | 1.02 ± 0.03 |
| `bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux              --spawn_strategy=local                   //...` |  6.921 ± 0.238 |   6.513 |   7.279 | 1.46 ± 0.05 |
| `bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux              --experimental_reuse_sandbox_directories //...` |  7.674 ± 0.370 |   7.000 |   8.492 | 1.62 ± 0.08 |
| `bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux              --spawn_strategy=sandboxed               //...` |  8.666 ± 0.116 |   8.504 |   8.883 | 1.83 ± 0.03 |
| `bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28              --spawn_strategy=local                   //...` |  9.311 ± 0.190 |   9.050 |   9.667 | 1.97 ± 0.05 |
| `bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=local                   //...` |  9.336 ± 0.347 |   8.968 |  10.220 | 1.98 ± 0.08 |
| `bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --experimental_reuse_sandbox_directories //...` | 13.098 ± 0.329 |  12.365 |  13.510 | 2.77 ± 0.08 |
| `bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28              --experimental_reuse_sandbox_directories //...` | 13.465 ± 0.292 |  13.033 |  13.897 | 2.85 ± 0.07 |
| `bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28              --spawn_strategy=sandboxed               //...` | 20.396 ± 0.330 |  19.966 |  20.939 | 4.32 ± 0.08 |
| `bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=sandboxed               //...` | 33.827 ± 0.630 |  32.121 |  34.412 | 7.16 ± 0.15 |

Explanation:

* `--spawn_strategy=local` is the baseline. This is the theoretically fastest
  approach (no sandbox), but unfit for production.
* `--spawn_strategy=sandboxed` is the default for Bazel. This is what happens
  if one does not pass any arguments.
* `--experimental_reuse_sandbox_directories` is an optimization which is
  meaningful on sandboxes with many files. As of Bazel 6 this has been
  [promoted to stable][3], so safe to use and, as one can see in the numbers,
  recommended with both bazel-zig-cc and a nontrivial sysroot. As a result,
  this is the most important benchmark to look at.

Flame graphs and discussion
---------------------------

Flame graphs in results/. Only of historical interest, because the biggest
performance issues have been resolved with a combination of zig and
bazel-zig-cc changes.

[1]: https://sr.ht/~motiejus/bazel-zig-cc
[2]: https://github.com/grailbio/bazel-toolchain
[3]: https://github.com/bazelbuild/bazel/issues/16138
