Subject: bazel-zig-cc embeds bazel cache path in go binary

From: Malte Poll <[email-masked]>
Date: Tue, 7 Mar 2023 16:15:08 +0000

Hello,

I was checking if bazel-zig-cc produces reproducible go binaries if used as a compiler for cgo.
As a test, I compiled the same binary on the same commit on two different systems (Ubuntu, Fedora) and in two different paths at different times.

The binaries are not reproducible. At least some differences seem to be coming from bazel-zig-cc:

The paths to pthread.h and signal.h on my build system are embedded in the resulting binary. See below for an instance of this behavior.
Do you think this can be prevented to make the resulting binary more hermetic / reproducible?

 -  [15809be]  /root/.cache/bazel/_bazel_root/378578e4f433132b2cb209458a9b56bc/sandbox/processwrapper-sandbox/44/execroot/__main__/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libc/include/generic-glibc/pthread.h
│ -  [1580a8e]  gcc_libinit.c
│ -  [1580a9c]  pthread_create failed: %s
│ -  [1580ab6]  gcc_linux_amd64.c
│ -  [1580ac8]  /root/.cache/bazel/_bazel_root/378578e4f433132b2cb209458a9b56bc/sandbox/processwrapper-sandbox/44/execroot/__main__/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libc/include/generic-glibc/pthread.h
│ -  [1580b98]  /root/.cache/bazel/_bazel_root/378578e4f433132b2cb209458a9b56bc/sandbox/processwrapper-sandbox/44/execroot/__main__/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libc/include/generic-glibc/signal.h
│ -  [1580c67]  malloc failed: %s
│ -  [1580c79]  bad stack bounds: lo=%p hi=%p\n
│ -  [1580c98]  pthread_create failed: %s
│ -  [1580cb2]  gcc_util.c
│ -  [1580cbd]  runtime/cgo: out of memory in thread_start\n
│ +  [15809be]  gcc_libinit.c
│ +  [15809cc]  /home/malte/.cache/bazel/_bazel_malte/bb1175a64d2f0f71785aea697a0484ea/sandbox/linux-sandbox/224/execroot/__main__/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libc/include/generic-glibc/pthread.h
│ +  [1580a9b]  pthread_create failed: %s
│ +  [1580ab5]  gcc_linux_amd64.c
│ +  [1580ac7]  malloc failed: %s
│ +  [1580ad9]  /home/malte/.cache/bazel/_bazel_malte/bb1175a64d2f0f71785aea697a0484ea/sandbox/linux-sandbox/224/execroot/__main__/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libc/include/generic-glibc/pthread.h
│ +  [1580ba8]  bad stack bounds: lo=%p hi=%p\n
│ +  [1580bc7]  /home/malte/.cache/bazel/_bazel_malte/bb1175a64d2f0f71785aea697a0484ea/sandbox/linux-sandbox/224/execroot/__main__/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libc/include/generic-glibc/signal.h
│ +  [1580c95]  pthread_create failed: %s
│ +  [1580caf]  gcc_util.c
│ +  [1580cba]  runtime/cgo: out of memory in thread_start\n

Attached is the full output of diffoscope.

Kind regards,
Malte Poll

---

From: Motiejus Jakštys <[email-masked]>
Date: Wed, 8 Mar 2023 06:55:43 +0200

> Do you think this can be prevented to make the resulting binary more hermetic / reproducible?

Hi Malte,

Thanks for the report. I am 99% confident the problem is with Go,
because Go compiles CGo files 2 ways:
1. in the Bazel sandbox.
2. in /tmp/work-<...>, outside the sandbox.

>  -  [15809be]  /root/.cache/bazel/_bazel_root/378578e4f433132b2cb209458a9b56bc/sandbox/processwrapper-sandbox/44/execroot/__main__/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../lib/libc/includ…

When we invoke "zig c++", the ZIG_LIB_DIR needs to be passed to Zig.
It needs to be a relative directory, but it can't always do that. This
is how it's determined:
https://git.sr.ht/~motiejus/bazel-zig-cc/tree/v1.0.1/item/toolchain/launcher.zig#L190-215

Here are the relevant lines with comments. First, determine the "root"
of the zig sdk:

    const root = blk: {
        var dir = cwd.openDir(
            "external" ++ sep ++ "zig_sdk" ++ sep ++ "lib",
            .{ .access_sub_paths = false, .no_follow = true },
        );

        if (dir) |*dir_exists| {
            dir_exists.close();
            break :blk "external" ++ sep ++ "zig_sdk";
        } else |_| {}

If "external/zig_sdk/lib" exists, then "root" is exactly that. If it
does not exist:

        // directory does not exist or there was an error opening it
        const here = fs.path.dirname(arg0) orelse ".";
        break :blk try fs.path.join(arena, &[_][]const u8{ here, "..", ".." });
    };

... then the zig sdk root will be the full path to the executed file
(e.g. /root/.cache/bazel/_bazel_root/378578e4f433132b2cb209458a9b56bc/sandbox/processwrapper-sandbox/44/execroot/__main__/external/zig_sdk/tools/x86_64-linux-gnu.2.34/c++),
it's dirname, plus "../../". Then append the "lib" to that path:

    const zig_lib_dir = try fs.path.join(arena, &[_][]const u8{ root, "lib" });
    <...>

And then use that path for ZIG_LIB_DIR:

    try env.put("ZIG_LIB_DIR", zig_lib_dir);

So following the logic ZIG_LIB_DIR in your example will be
/root/.cache/bazel/_bazel_root/378578e4f433132b2cb209458a9b56bc/sandbox/processwrapper-sandbox/44/execroot/__main__/external/zig_sdk/tools/x86_64-linux-gnu.2.34/../../
; and this is how Zig will embed that path.

Other than changing how Go builds it's CGO binaries, I don't see a
good way to work around this. At the point Go is invoking the external
linker we are already in $WORK, so we don't have control over that
working directory.

The horrible part is, of course, changing the working directory at the
bottom leaves of the compilation chain. The good part about this
design is that Bazel does not see non-hermeticity (at this point Bazel
lost control). So as far as Bazel is concerned, it's the same
dependency hash. The bad part, is, of course, non-reproducible
binaries.

> <...>

Thanks, TIL.

Are the paths still there if you strip the binaries (are they
stripped? hard to tell, because your diffoscope output does not
include all sections)? IIRC filenames are only present in binaries
compiled with debug info. But I am not sure if `strip` removes strings
from rodata. Please let us know.

Motiejus