test-zig-cc/README.md

bazel-zig-cc and llvm
---------------------

bazel-zig-cc has a performance issue when compiling many files. This repository
reproduces that.

The test
--------

Compiles 64 small binaries with [bazel-zig-cc][1] and [llvm14][2]. The tests
were run on an x86_64 8-core machine running Ubuntu 22.04. The llvm's sysroot
is this:

    sha256 = "84656a6df544ecef62169cfe3ab6e41bb4346a62d3ba2a045dc5a0a2ecea94a3",
    urls = ["https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/2202c161310ffde63729f29d27fe7bb24a0bc540/debian_stretch_amd64_sysroot.tar.xz"],

... but with extra ~8k files, so the number of files in the sysroot is exactly
the same as in zig sdk.

Baseline
--------

Baseline llvm14 without sandbox: 12 seconds:

    bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
    bazel clean; bazel shutdown
    bazel --batch build --spawn_strategy=local --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux ...
    INFO: Elapsed time: 11.971s, Critical Path: 0.41s

zig cc without the sandbox. 15 seconds:

    bazel build --spawn_strategy=local --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
    bazel clean; bazel shutdown
    bazel --batch build --spawn_strategy=local --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
    INFO: Elapsed time: 15.631s, Critical Path: 1.99s

Cost of Sandbox
---------------

llvm 14:

    bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
    bazel clean; bazel shutdown
    bazel --batch build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux ...
    INFO: Elapsed time: 109.677s, Critical Path: 13.60s

zig cc:

    bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
    bazel clean; bazel shutdown
    bazel --batch build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
    INFO: Elapsed time: 116.370s, Critical Path: 16.23s

Possible mitigation: Sandboxfs
------------------------------

No dice.

llvm 14: does not build at all:

    $ bazel build --experimental_sandboxfs_path=sandboxfs --experimental_use_sandboxfs --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
    ERROR: /code/test-zigcc/BUILD:4:14: Compiling main.cc failed: (Exit 5): cc_wrapper.sh failed: error executing command external/llvm_toolchain_with_sysroot/bin/cc_wrapper.sh -U_FORTIFY_SOURCE '--target=x86_64-unknown-linux-gnu' -U_FORTIFY_SOURCE -fstack-protector -fno-omit-frame-pointer -fcolor-diagnostics -Wall ... (remaining 29 arguments skipped)

    Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
    ERROR: could not find clang; PWD="/proc/self/cwd"; PATH="<...>".
    Target //:zigzag-0 failed to build
    Use --verbose_failures to see the command lines of failed build steps.
    INFO: Elapsed time: 7.578s, Critical Path: 0.26s
    INFO: 3 processes: 3 internal.
    FAILED: Build did NOT complete successfully


zig cc:

    bazel build --experimental_sandboxfs_path=sandboxfs --experimental_use_sandboxfs --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
    bazel clean; bazel shutdown
    bazel --batch build --experimental_sandboxfs_path=sandboxfs --experimental_use_sandboxfs --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
    INFO: Elapsed time: 126.613s, Critical Path: 17.63s


Possible mitigation: `--experimental_reuse_sandbox_directories`
---------------------------------------------------------------

Dice.

llvm 14:

    bazel build --experimental_reuse_sandbox_directories --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux //:zigzag-0
    bazel clean; bazel shutdown
    bazel --batch build --experimental_reuse_sandbox_directories --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux ...
    INFO: Elapsed time: 25.252s, Critical Path: 4.44s

zig cc:

    bazel build --experimental_reuse_sandbox_directories --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //:zigzag-0
    bazel clean; bazel shutdown
    bazel --batch build --experimental_reuse_sandbox_directories --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 ...
    INFO: Elapsed time: 29.132s, Critical Path: 3.88s


Flame graphs and discussion
---------------------------

Flame graphs and more tests are in results/. As of 2022-12-11 most of the
overhead comes from creating and deleting the sandboxes.

[1]: https://sr.ht/~motiejus/bazel-zig-cc
[2]: https://github.com/grailbio/bazel-toolchain