motiejus/hermetic_cc_toolchain
1
Fork 0
Code Releases Activity

Zig cache hacks/optimizations

Browse Source 
Zig does not always share `libc++.a` and the glibc shims. We have observed
to be caused for 2 reasons:

- For some commands Go `chdir`s to `/tmp/`. This causes `ZIG_LIB_DIR` to be
  absolute, which blows the cache key to find the right `libc++.a` (because Zig
  thinks we are using a different lib dir to compile libc++). This is currently
  worked around in `toolchain/defs.bzl` by overfitting to Go and returning
  early from that particular invocation.
- Sometimes Bazel's sandbox messes up Zig's cache keys. If one runs without the
  sandbox (`--spawn_strategy=standalone`), the cache hit rate and thus the
  build time are much better.

This is actively investigated. I am adding `--spawn_strategy=standalone`
to CI to see the speedup that it provides. I will rollback it later.
This commit is contained in:
Motiejus Jakštys 2022-10-03 05:50:53 +03:00
parent f30a3afdde
commit 50ca49181f
2 changed files with 23 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
README.md
View File

@ -297,6 +297,20 @@ unlikely to implement them any time soon, but patches implementing those will
be accepted. See [Questions & Contributions](#questions-amp-contributions) on
how to contribute.
## Sharing of libc shims/libc++.a
Zig does not always share `libc++.a` and the glibc shims. We have observed
to be caused for 2 reasons:
- For some commands Go `chdir`s to `/tmp/`. This causes `ZIG_LIB_DIR` to be
absolute, which blows the cache key to find the right `libc++.a` (because Zig
thinks we are using a different lib dir to compile libc++). This is currently
worked around in `toolchain/defs.bzl` by overfitting to Go and returning
early from that particular invocation.
- Sometimes Bazel's sandbox messes up Zig's cache keys. If one runs without the
sandbox (`--spawn_strategy=standalone`), the cache hit rate and thus the
build time are much better.
## Zig cache location
Currently zig cache is in `$HOME`, so `bazel clean --expunge` does not clear

13
toolchain/defs.bzl
View File

@ -133,7 +133,7 @@ export ZIG_LIB_DIR
export ZIG_LOCAL_CACHE_DIR="{cache_prefix}/bazel-zig-cc"
export ZIG_GLOBAL_CACHE_DIR="{cache_prefix}/bazel-zig-cc"
{maybe_gohack}
exec "{zig}" "{zig_tool}" {maybe_target} "$@" $maybe_o2
exec "{zig}" "{zig_tool}" {maybe_target} "$@"
"""
_ZIG_TOOL_WRAPPER_CACHE_GUESS = """#!/bin/sh
@ -158,7 +158,7 @@ export ZIG_LIB_DIR
export ZIG_LOCAL_CACHE_DIR="$_cache_prefix/bazel-zig-cc"
export ZIG_GLOBAL_CACHE_DIR=$ZIG_LOCAL_CACHE_DIR
{maybe_gohack}
exec "{zig}" "{zig_tool}" {maybe_target} "$@" $maybe_o2
exec "{zig}" "{zig_tool}" {maybe_target} "$@"
"""
# The abomination below adds "-O2" to Go's link-prober command. Saves around
@ -169,9 +169,14 @@ exec "{zig}" "{zig_tool}" {maybe_target} "$@" $maybe_o2
_ZIG_TOOL_GOHACK = """
quote(){ echo "$1" | sed -e "s,','\\\\'',g"; }
for arg in "$@"; do saved="${saved:+$saved }'$(quote "$arg")'"; done
maybe_o2=
while [ "$#" -gt 6 ]; do shift; done
[ "$*" = "-Wl,--no-gc-sections -x c - -o /dev/null" ] && maybe_o2="-O2"
if [ "$*" = "-Wl,--no-gc-sections -x c - -o /dev/null" ]; then
# This command probes if `--no-gc-sections` is accepted by the linker.
# Since it is executed in /tmp, the ZIG_LIB_DIR is absolute,
# glibc stubs and libc++ cannot be shared with other invocations (which use
# a relative ZIG_LIB_DIR).
exit 0;
fi
eval set -- "$saved"
"""