motiejus/test-zig-cc
1
Fork 0
Code Releases Activity
f9ed9f6215
test-zig-cc/README.md

136 lines
10 KiB
Markdown
Raw Normal View History

let it be so
2022-10-19 11:35:55 +03:00
bazel-zig-cc and llvm
---------------------
bazel-zig-cc has a performance issue when compiling many files. This repository
reproduces that.
use `bazel build --batch`
2022-11-21 06:56:12 +02:00
The test
--------
Compiles 64 small binaries with [bazel-zig-cc][1] and [llvm14][2]. The tests
update results
2022-12-16 05:10:01 +02:00
were run on an x86_64 8-core machine running Ubuntu 22.04.
move TLDR upwards
2022-12-20 12:56:17 +02:00
TLDR
----
Extracting the fastest meaningful benchmarks in all categories (from the
section below):
'bazel build --experimental_reuse_sandbox_directories //...' ran
2.77 ± 0.08 times faster than 'bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --experimental_reuse_sandbox_directories //...'
2.85 ± 0.07 times faster than 'bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --experimental_reuse_sandbox_directories //...'
% -> times
2022-12-20 13:07:37 +02:00
This demonstrates that **adding a hermetic toolchain to a project caused 2.77x
move TLDR upwards
2022-12-20 12:56:17 +02:00
slowdown on this project**. Note that we are not counting `llvm_toolchain//...`
as a real toolchain -- one needs a sysroot to compile anything meaningful
anyway (which zig bundles with the compiler).
Once we have determined that a hermetic C++ toolchain is required, bazel-zig-cc
is about 2.8% slower than its contender plain llvm14.
update results
2022-12-16 05:10:01 +02:00
Results
-------
update results with more test data
2022-12-20 12:40:46 +02:00
Benchmark 1: bazel build --spawn_strategy=local //...
Time (mean ± σ): 4.813 s ± 0.140 s [User: 0.015 s, System: 0.009 s]
Range (min … max): 4.529 s … 5.007 s 10 runs
Benchmark 2: bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux --spawn_strategy=local //...
Time (mean ± σ): 6.921 s ± 0.238 s [User: 0.013 s, System: 0.011 s]
Range (min … max): 6.513 s … 7.279 s 10 runs
Benchmark 3: bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=local //...
Time (mean ± σ): 9.336 s ± 0.347 s [User: 0.013 s, System: 0.012 s]
Range (min … max): 8.968 s … 10.220 s 10 runs
Benchmark 4: bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --spawn_strategy=local //...
Time (mean ± σ): 9.311 s ± 0.190 s [User: 0.014 s, System: 0.010 s]
Range (min … max): 9.050 s … 9.667 s 10 runs
Benchmark 5: bazel build --experimental_reuse_sandbox_directories //...
Time (mean ± σ): 4.726 s ± 0.051 s [User: 0.011 s, System: 0.011 s]
Range (min … max): 4.653 s … 4.793 s 10 runs
Benchmark 6: bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux --experimental_reuse_sandbox_directories //...
Time (mean ± σ): 7.674 s ± 0.370 s [User: 0.012 s, System: 0.012 s]
Range (min … max): 7.000 s … 8.492 s 10 runs
Benchmark 7: bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --experimental_reuse_sandbox_directories //...
Time (mean ± σ): 13.098 s ± 0.329 s [User: 0.018 s, System: 0.006 s]
Range (min … max): 12.365 s … 13.510 s 10 runs
Benchmark 8: bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --experimental_reuse_sandbox_directories //...
Time (mean ± σ): 13.465 s ± 0.292 s [User: 0.014 s, System: 0.012 s]
Range (min … max): 13.033 s … 13.897 s 10 runs
Benchmark 9: bazel build --spawn_strategy=sandboxed //...
Time (mean ± σ): 4.766 s ± 0.179 s [User: 0.014 s, System: 0.008 s]
Range (min … max): 4.553 s … 5.224 s 10 runs
Benchmark 10: bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux --spawn_strategy=sandboxed //...
Time (mean ± σ): 8.666 s ± 0.116 s [User: 0.014 s, System: 0.009 s]
Range (min … max): 8.504 s … 8.883 s 10 runs
Benchmark 11: bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=sandboxed //...
Time (mean ± σ): 33.827 s ± 0.630 s [User: 0.019 s, System: 0.010 s]
Range (min … max): 32.121 s … 34.412 s 10 runs
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
Benchmark 12: bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --spawn_strategy=sandboxed //...
Time (mean ± σ): 20.396 s ± 0.330 s [User: 0.016 s, System: 0.011 s]
Range (min … max): 19.966 s … 20.939 s 10 runs
update results
2022-12-16 05:10:01 +02:00
Summary
format README
2022-12-20 12:46:05 +02:00
'bazel build --experimental_reuse_sandbox_directories //...' ran
1.01 ± 0.04 times faster than 'bazel build --spawn_strategy=sandboxed //...'
1.02 ± 0.03 times faster than 'bazel build --spawn_strategy=local //...'
1.46 ± 0.05 times faster than 'bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux --spawn_strategy=local //...'
update results with more test data
2022-12-20 12:40:46 +02:00
1.62 ± 0.08 times faster than 'bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux --experimental_reuse_sandbox_directories //...'
format README
2022-12-20 12:46:05 +02:00
1.83 ± 0.03 times faster than 'bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux --spawn_strategy=sandboxed //...'
1.97 ± 0.05 times faster than 'bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --spawn_strategy=local //...'
1.98 ± 0.08 times faster than 'bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=local //...'
update results with more test data
2022-12-20 12:40:46 +02:00
2.77 ± 0.08 times faster than 'bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --experimental_reuse_sandbox_directories //...'
2.85 ± 0.07 times faster than 'bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --experimental_reuse_sandbox_directories //...'
format README
2022-12-20 12:46:05 +02:00
4.32 ± 0.08 times faster than 'bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --spawn_strategy=sandboxed //...'
7.16 ± 0.15 times faster than 'bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=sandboxed //...'
update results with more test data
2022-12-20 12:40:46 +02:00
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:-----------------------------------------------------------------------------------------------------------------------------------------|---------------:|--------:|--------:|------------:|
| `bazel build --experimental_reuse_sandbox_directories //...` | 4.726 ± 0.051 | 4.653 | 4.793 | 1.00 |
| `bazel build --spawn_strategy=sandboxed //...` | 4.766 ± 0.179 | 4.553 | 5.224 | 1.01 ± 0.04 |
| `bazel build --spawn_strategy=local //...` | 4.813 ± 0.140 | 4.529 | 5.007 | 1.02 ± 0.03 |
| `bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux --spawn_strategy=local //...` | 6.921 ± 0.238 | 6.513 | 7.279 | 1.46 ± 0.05 |
| `bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux --experimental_reuse_sandbox_directories //...` | 7.674 ± 0.370 | 7.000 | 8.492 | 1.62 ± 0.08 |
| `bazel build --extra_toolchains=@llvm_toolchain//:cc-toolchain-x86_64-linux --spawn_strategy=sandboxed //...` | 8.666 ± 0.116 | 8.504 | 8.883 | 1.83 ± 0.03 |
| `bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --spawn_strategy=local //...` | 9.311 ± 0.190 | 9.050 | 9.667 | 1.97 ± 0.05 |
| `bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=local //...` | 9.336 ± 0.347 | 8.968 | 10.220 | 1.98 ± 0.08 |
| `bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --experimental_reuse_sandbox_directories //...` | 13.098 ± 0.329 | 12.365 | 13.510 | 2.77 ± 0.08 |
| `bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --experimental_reuse_sandbox_directories //...` | 13.465 ± 0.292 | 13.033 | 13.897 | 2.85 ± 0.07 |
| `bazel build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 --spawn_strategy=sandboxed //...` | 20.396 ± 0.330 | 19.966 | 20.939 | 4.32 ± 0.08 |
| `bazel build --extra_toolchains=@llvm_toolchain_with_sysroot//:cc-toolchain-x86_64-linux --spawn_strategy=sandboxed //...` | 33.827 ± 0.630 | 32.121 | 34.412 | 7.16 ± 0.15 |
update results
2022-12-16 05:10:01 +02:00
Explanation:
wording
2022-12-16 05:13:15 +02:00
* `--spawn_strategy=local` is the baseline. This is the theoretically fastest
approach (no sandbox), but unfit for production.
update results
2022-12-16 05:10:01 +02:00
* `--spawn_strategy=sandboxed` is the default for Bazel. This is what happens
if one does not pass any arguments.
update results with more test data
2022-12-20 12:40:46 +02:00
* `--experimental_reuse_sandbox_directories` is an optimization which is
meaningful on sandboxes with many files. As of Bazel 6 this has been
[promoted to stable][3], so safe to use and, as one can see in the numbers,
recommended with both bazel-zig-cc and a nontrivial sysroot. As a result,
this is the most important benchmark to look at.
trying --experimental_use_hermetic_linux_sandbox
2022-12-11 07:03:10 +02:00
tests with sysroot
2022-12-11 15:55:21 +02:00
Flame graphs and discussion
---------------------------
add more prose to README
2022-12-20 12:54:45 +02:00
Flame graphs in results/. Only of historical interest, because the biggest
update results
2022-12-16 05:10:01 +02:00
performance issues have been resolved with a combination of zig and
bazel-zig-cc changes.
tests with sysroot
2022-12-11 15:55:21 +02:00
use `bazel build --batch`
2022-11-21 06:56:12 +02:00
[1]: https://sr.ht/~motiejus/bazel-zig-cc
[2]: https://github.com/grailbio/bazel-toolchain
update results
2022-12-16 05:10:01 +02:00
[3]: https://github.com/bazelbuild/bazel/issues/16138
Copy Permalink