Subject: [PATCH bazel-zig-cc 0/2] Ignore undefined symbols when building dynamic libraries on darwin

From: ~jvolkman <[email-masked]>
Date: Wed, 25 May 2022 01:41:49 +0000

Hi,

I'm working on a rule set to build Python modules using bazel cc
toolchains, and found that with bazel-zig-cc, I'm unable to build native
modules on macos. The issue turns out to be differences in handling
undefined symbols at link time in linux vs. macos. When building for a
linux target, zig ignores undefined symbols, but for macos they generate
errors. In the case of Python, native modules never link against
libpython. Instead, its symbols are provided when the module is loaded
by the cpython interpreter (at least that's my understanding).

See some discussion of this at
https://github.com/ziglang/zig/issues/8180.

This change adds dynamic_library_linkopts to zig_cc_toolchain_config
which apply only to the cpp_link_dynamic_library and
cpp_link_nodeps_dynamic_library. For darwin targets, I specify the
-Wl,-undefined=dynamic_lookup flag. This logic is adapted from bazel's
built-in macos toolchain [1].

Thanks!

[1]
https://cs.github.com/bazelbuild/bazel/blob/ca40a514b8bc609478f769194909dd0be0afa1f9/tools/osx/crosstool/cc_toolchain_config.bzl#L1125-L1131

Jeremy Volkman (2):
  Adds dynamic_library_linkopts to zig_cc_toolchain_config
  Pass "-Wl,-undefined=dynamic_lookup" for dynamic libraries on darwin.

 toolchain/private/cc_toolchains.bzl |  2 ++
 toolchain/private/defs.bzl          |  3 +++
 toolchain/zig_toolchain.bzl         | 17 ++++++++++++++---
 3 files changed, 19 insertions(+), 3 deletions(-)

-- 
2.34.2

---

From: ~jvolkman <[email-masked]>
Date: Tue, 24 May 2022 17:14:35 -0700

From: Jeremy Volkman <[email-masked]>

If specified, the flags in dynamic_library_linkopts are included in
cpp_link_dynamic_library and cpp_link_nodeps_dynamic_library actions.
---
 toolchain/private/cc_toolchains.bzl |  2 ++
 toolchain/private/defs.bzl          |  3 +++
 toolchain/zig_toolchain.bzl         | 17 ++++++++++++++---
 3 files changed, 19 insertions(+), 3 deletions(-)

diff --git a/toolchain/private/cc_toolchains.bzl b/toolchain/private/cc_toolchains.bzl
index 64e99f1..3a1e1cb 100644
--- a/toolchain/private/cc_toolchains.bzl
+++ b/toolchain/private/cc_toolchains.bzl
@@ -32,6 +32,7 @@ def declare_cc_toolchains(absolute_path, zig_include_root):
             absolute_tool_paths[name] = "%s/%s" % (absolute_path, tool_path)
 
         linkopts = target_config.linkopts
+        dynamic_library_linkopts = target_config.dynamic_library_linkopts
         copts = target_config.copts
         for s in getattr(target_config, "linker_version_scripts", []):
             linkopts = linkopts + ["-Wl,--version-script,%s/%s" % (absolute_path, s)]
@@ -45,6 +46,7 @@ def declare_cc_toolchains(absolute_path, zig_include_root):
             cxx_builtin_include_directories = cxx_builtin_include_directories,
             copts = copts,
             linkopts = linkopts,
+            dynamic_library_linkopts = dynamic_library_linkopts,
             target_cpu = target_config.bazel_target_cpu,
             target_system_name = "unknown",
             target_libc = "unknown",
diff --git a/toolchain/private/defs.bzl b/toolchain/private/defs.bzl
index 6059065..053641c 100644
--- a/toolchain/private/defs.bzl
+++ b/toolchain/private/defs.bzl
@@ -54,6 +54,7 @@ def _target_darwin(gocpu, zigcpu):
             "libc/include/any-macos-any",
         ],
         linkopts = [],
+        dynamic_library_linkopts = [],
         copts = [],
         bazel_target_cpu = "darwin",
         constraint_values = [
@@ -84,6 +85,7 @@ def _target_linux_gnu(gocpu, zigcpu, glibc_version):
         compiler_extra_includes = ["glibc-hacks/glibchack-fcntl.h"] if fcntl_hack else [],
         linker_version_scripts = ["glibc-hacks/fcntl.map"] if fcntl_hack else [],
         linkopts = ["-lc++", "-lc++abi"],
+        dynamic_library_linkopts = [],
         copts = [],
         bazel_target_cpu = "k8",
         constraint_values = [
@@ -105,6 +107,7 @@ def _target_linux_musl(gocpu, zigcpu):
             "libc/include/{}-linux-any".format(zigcpu),
         ] + (["libc/include/x86-linux-any"] if zigcpu == "x86_64" else []),
         linkopts = ["-s", "-w"],
+        dynamic_library_linkopts = [],
         copts = ["-D_LIBCPP_HAS_MUSL_LIBC", "-D_LIBCPP_HAS_THREAD_API_PTHREAD"],
         bazel_target_cpu = "k8",
         constraint_values = [
diff --git a/toolchain/zig_toolchain.bzl b/toolchain/zig_toolchain.bzl
index aa14ec1..8676471 100644
--- a/toolchain/zig_toolchain.bzl
+++ b/toolchain/zig_toolchain.bzl
@@ -14,6 +14,11 @@ all_link_actions = [
     ACTION_NAMES.cpp_link_nodeps_dynamic_library,
 ]
 
+dynamic_library_link_actions = [
+    ACTION_NAMES.cpp_link_dynamic_library,
+    ACTION_NAMES.cpp_link_nodeps_dynamic_library,
+]
+
 compile_and_link_actions = [
     ACTION_NAMES.c_compile,
     ACTION_NAMES.cpp_compile,
@@ -79,15 +84,20 @@ def _zig_cc_toolchain_config_impl(ctx):
         flag_sets = [
             flag_set(
                 actions = all_link_actions,
-                flag_groups = ([
+                flag_groups = [
                     flag_group(
                         flags = ["-target", ctx.attr.target] +
                                 no_gc_sections +
                                 ctx.attr.linkopts,
                     ),
-                ]),
+                ],
             ),
-        ],
+        ] + [
+            flag_set(
+                actions = dynamic_library_link_actions,
+                flag_groups = [flag_group(flags = ctx.attr.dynamic_library_linkopts)],
+            ),
+        ] if ctx.attr.dynamic_library_linkopts else [],
     )
 
     features = [
@@ -119,6 +129,7 @@ zig_cc_toolchain_config = rule(
     attrs = {
         "cxx_builtin_include_directories": attr.string_list(),
         "linkopts": attr.string_list(),
+        "dynamic_library_linkopts": attr.string_list(),
         "copts": attr.string_list(),
         "tool_paths": attr.string_dict(),
         "target": attr.string(),
-- 
2.34.2

---

From: ~jvolkman <[email-masked]>
Date: Tue, 24 May 2022 17:17:33 -0700

This linker flag causes undefined symbols to be ignored, which is the
default behavior on linux but not macos. This is required when building
shared libraries to use as e.g. Python modules. This also matches what
Bazel's built-in macos cc toolchain does.
---
 toolchain/private/defs.bzl | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/toolchain/private/defs.bzl b/toolchain/private/defs.bzl
index 053641c..42aaaeb 100644
--- a/toolchain/private/defs.bzl
+++ b/toolchain/private/defs.bzl
@@ -54,7 +54,7 @@ def _target_darwin(gocpu, zigcpu):
             "libc/include/any-macos-any",
         ],
         linkopts = [],
-        dynamic_library_linkopts = [],
+        dynamic_library_linkopts = ["-Wl,-undefined=dynamic_lookup"],
         copts = [],
         bazel_target_cpu = "darwin",
         constraint_values = [
-- 
2.34.2

---

From: "builds.sr.ht" <[email-masked]>
Date: Wed, 25 May 2022 01:45:04 +0000

bazel-zig-cc/patches/.build.yml: FAILED in 3m12s

[Ignore undefined symbols when building dynamic libraries on darwin][0] from [~jvolkman][1]

[0]: https://lists.sr.ht/~motiejus/bazel-zig-cc/patches/32501
[1]: mailto:[email-masked]

✗ #765889 FAILED bazel-zig-cc/patches/.build.yml https://builds.sr.ht/~motiejus/job/765889

---

From: Motiejus Jakštys <[email-masked]>
Date: Wed, 25 May 2022 05:50:12 +0300

>   Pass "-Wl,-undefined=dynamic_lookup" for dynamic libraries on darwin.

Hi Jeremy,

looks like the patch broke ELF[1] on x86_64. Thus I did not reach your
patch carefully. Please let me know when I should take action.

Motiejus

[1]: https://builds.sr.ht/~motiejus/job/765889