test-zig-cc/README.md

bazel-zig-cc and llvm
---------------------

bazel-zig-cc has a performance issue when compiling many files. This repository
reproduces that.

The test
--------

Compiles 64 small binaries with [bazel-zig-cc][1] and [llvm14][2]. The tests
were run on an x86_64 8-core machine running Ubuntu 22.04. The llvm's sysroot
is this:

    sha256 = "84656a6df544ecef62169cfe3ab6e41bb4346a62d3ba2a045dc5a0a2ecea94a3",
    urls = ["https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/2202c161310ffde63729f29d27fe7bb24a0bc540/debian_stretch_amd64_sysroot.tar.xz"],

... but with extra ~8k files, so the number of files in the sysroot is exactly
the same as in zig sdk.

Baseline
--------

Baseline llvm14 without sandbox: 12 seconds:

    bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
    bazel clean; bazel shutdown
    bazel --batch build --spawn_strategy=local --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux ...
    INFO: Elapsed time: 11.971s, Critical Path: 0.41s

zig cc without the sandbox. 15 seconds:

    bazel build --spawn_strategy=local --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
    bazel clean; bazel shutdown
    bazel --batch build --spawn_strategy=local --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
    INFO: Elapsed time: 15.631s, Critical Path: 1.99s

Cost of Sandbox
---------------

llvm 14:

    bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
    bazel clean; bazel shutdown
    bazel --batch build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux ...
    INFO: Elapsed time: 109.677s, Critical Path: 13.60s

zig cc:

    bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
    bazel clean; bazel shutdown
    bazel --batch build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
    INFO: Elapsed time: 116.370s, Critical Path: 16.23s

Possible mitigation: Sandboxfs
------------------------------

No dice.

llvm 14:

    bazel build --experimental_sandboxfs_path=sandboxfs --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
    bazel clean; bazel shutdown
    bazel --batch build --experimental_sandboxfs_path=sandboxfs --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux ...
    INFO: Elapsed time: 96.803s, Critical Path: 12.42s

zig cc:

    bazel build --experimental_sandboxfs_path=sandboxfs --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
    bazel clean; bazel shutdown
    bazel --batch build --experimental_sandboxfs_path=sandboxfs --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
    INFO: Elapsed time: 109.347s, Critical Path: 13.94s

Possible mitigation: `--experimental_reuse_sandbox_directories`
---------------------------------------------------------------

Dice.

llvm 14:

    bazel build --experimental_reuse_sandbox_directories --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
    bazel clean; bazel shutdown
    bazel --batch build --experimental_reuse_sandbox_directories --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux ...
    INFO: Elapsed time: 25.252s, Critical Path: 4.44s

zig cc:

    bazel build --experimental_reuse_sandbox_directories --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
    bazel clean; bazel shutdown
    bazel --batch build --experimental_reuse_sandbox_directories --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
    INFO: Elapsed time: 29.132s, Critical Path: 3.88s


Flame graphs and discussion
---------------------------

Flame graphs and more tests are in results/. As of 2022-12-11 most of the
overhead comes from creating and deleting the sandboxes.

[1]: https://sr.ht/~motiejus/bazel-zig-cc
[2]: https://github.com/grailbio/bazel-toolchain