1
Fork 0

Compare commits

...

154 Commits
v0.9.1 ... main

Author SHA1 Message Date
Motiejus Jakštys b3d3be2049 maintainers: update the section
We have a new maintainer. Also clarify what they can and cannot do.
2023-04-28 21:01:53 +03:00
Motiejus Jakštys e319ac4635
zig cache dir: move to `/tmp/zig-cache` (#52)
This cache directory can be re-used by everything and everyone. There is
nothing bazel or hermetic_cc_toolchain specific there. So let's make it
clear.

Also, if there are any more zig-based toolchains on top of Bazel or
other build systems where they cannot rely on $HOME, but need an
absolute path, this feels like a reasonable choice.
2023-04-28 09:05:34 -07:00
Motiejus Jakštys 9855851dd0 remove PUBLICITY.md
We've outgrown this by now.
2023-04-27 10:18:27 +03:00
Jonathan Baker e8c37be732 Update README.md
Fixed URL references

Signed-off-by: Jonathan Baker <jonbaker@uber.com>
2023-04-27 09:34:34 +03:00
Jonathan Baker 7179113b66 Rename Publicity.md to PUBLICITY.md
Signed-off-by: Jonathan Baker <jonbaker@uber.com>
2023-04-27 09:34:34 +03:00
Jonathan Baker 9dc94fb435 Create Publicity.md
Signed-off-by: Jonathan Baker <jonbaker@uber.com>
2023-04-27 09:34:34 +03:00
Jonathan Baker 372b746490 Update README.md
Updated references to use the plural "team".  Corrected grammar in the Project Origin, and added highlight.  Moved Previous Communication under Communication.

Signed-off-by: Jonathan Baker <jonbaker@uber.com>
2023-04-27 09:34:34 +03:00
Motiejus Jakštys 1deefddc95 Merge branch '47' 2023-04-26 09:59:36 +03:00
Jeremy Volkman f35f5d3eeb buildifier 2023-04-26 07:32:43 +03:00
Jeremy Volkman 5a7e58af4e Move -Wl,-S to a separate strip_debug_symbols feature 2023-04-26 07:32:43 +03:00
Motiejus Jakštys b1b66ce88c Merge tag 'v2.0.0-rc1'
This is a whoops of me forgetting to push the main branch after the
release.
2023-04-25 09:39:53 +03:00
Motiejus Jakštys 46289440a3 releaser: add MODULE.bazel
To make bzlMod work.

Test:

    $ tar -tvf hermetic_cc_toolchain-v99.0.0.tar.gz | grep MODULE.bazel
    -rw-rw-r-- root/root       561 2023-04-24 21:03 MODULE.bazel
2023-04-24 21:37:48 +03:00
Motiejus Jakštys c2701497d0 update mailing list archive
There will be no more emails coming in there.
2023-04-24 21:07:53 +03:00
Motiejus Jakštys fa951e262e
Merge pull request #33 from jvolkman/dev/linkopts
Re-add linkopts and set -Wl,-headerpad_max_install_names on macos
2023-04-24 20:41:44 +03:00
Jeremy Volkman cff2b2660b Re-add linkopts and set -Wl,-headerpad_max_install_names on macos 2023-04-24 09:48:27 -07:00
Motiejus Jakštys 9e7efd9a1b remaining linker_version_scripts cleanup
This was missed in #30
2023-04-24 16:16:00 +03:00
Motiejus Jakštys 1f50f76cc6 bump zig sdk to 0.11.0-dev.2619+bd3e248c7
From https://github.com/marler8997/zig-unofficial-releases

I also updated the "Release Process" wiki instructing how to use it.
Long live Mr. Marler!

The launcher got a facelift because of this error:

    /code/zig-linux-x86_64-0.11.0-dev.2619+bd3e248c7/lib/std/fmt.zig:2013:9: error: function called at runtime cannot return value at comptime
            return &buf;
            ^~~~~~~~~~~
    referenced by:
        test.launcher:parseArgs: toolchain/launcher.zig:334:31
        remaining reference traces hidden; use '-freference-trace' to see all reference traces

Zig gets confused by the requirement in that test in a runtime context.
2023-04-24 15:48:07 +03:00
Motiejus Jakštys 491144c721 remove rules/rules_go
1. Go needed quite a few hacks to be made work with Go. From
   https://github.com/ziglang/zig/pull/15060 less hacks are needed (but
   still not zero, though now documented).
2. This file was never meant to be exposed as part of
   `hermetic_cc_toolchain`. I probably broke someone. Sorry.
2023-04-24 14:59:15 +03:00
Motiejus Jakštys 3e07f72a8b bump zig sdk to 0.11.0-dev.2619+bd3e248c7
From https://github.com/marler8997/zig-unofficial-releases

I also updated the "Release Process" wiki instructing how to use it.
Long live Mr. Marler!

The launcher got a facelift because of this error:

    /code/zig-linux-x86_64-0.11.0-dev.2619+bd3e248c7/lib/std/fmt.zig:2013:9: error: function called at runtime cannot return value at comptime
            return &buf;
            ^~~~~~~~~~~
    referenced by:
        test.launcher:parseArgs: toolchain/launcher.zig:334:31
        remaining reference traces hidden; use '-freference-trace' to see all reference traces

Zig gets confused by the requirement in that test in a runtime context.
2023-04-24 14:46:23 +03:00
Motiejus Jakštys cb46744cb8 launcher: use `-mcpu=baseline`
This log message has been seen in a Github Actions worker:

      running <...>/c++ failed: signal: illegal instruction (core dumped)

I conclude that the launcher was compiled on a newer CPU than used on
the worker at the time.

Reported-by: mp@edgeless.systems

Fixes #22
2023-04-24 12:38:23 +03:00
Motiejus Jakštys 765fd04587 README: bring the compat name of http_archive #40
The recent repository rename broke the assumption about the repository
name.

We will soon (hopefully next week) release v2 with all the occurrences
of `bazel-zig-cc` removed, and all of this will be over. Sorry for the
confusion/mess until that happens.

Fixes #39
2023-04-24 12:12:34 +03:00
Motiejus Jakštys ef820f5e59 README.md: use the correct badge 2023-04-24 12:10:33 +03:00
Goni Zahavy f164afe89a Removed leftovers of `compiler_extra_includes` 2023-04-24 12:09:35 +03:00
Goni Zahavy 19c735c41a remove glibc hacks and update Zig SDK 2023-04-24 12:09:35 +03:00
Motiejus Jakštys 6ba4b61d05 Update MODULE.bazel
Signed-off-by: Motiejus Jakštys <motiejus@jakstys.lt>
2023-04-24 12:07:33 +03:00
zplin 3569fbd0fd dev_dependency 2023-04-24 12:07:33 +03:00
zplin fa7a3d8e9f Adding MODULE.bazel 2023-04-24 12:07:33 +03:00
Motiejus Jakštys 92067b5d85 remove @bazel_skylib dependency (again)
This was accidentally included in
d61aab183e434d48783dbaa2fdb9e39714217637
2023-04-24 12:02:35 +03:00
Motiejus Jakštys 0e12e84103 replace run_shell with run 2023-04-24 10:13:53 +03:00
Motiejus Jakštys 3630f62053 rules/ remove run_under
rules/ were never meant to be added to the release. Now that we are not
*running* multi-arch binaries, we can remove the infrastructure that was
necessary to do so.

I may get back to this -- but we need to dust quite a bit off before
that happens.
2023-04-24 10:13:53 +03:00
Motiejus Jakštys 9779650a78 Work around a known race in Zig build system
More information in the commit message and
https://github.com/ziglang/zig/issues/14815
2023-04-24 10:10:58 +03:00
Motiejus Jakštys 97a4846b58 remove `@bazel_skylib` dependency
We are using only a single trivial function `paths`, which can be
bundled. Makes things like test-zigcc[1] easier: less dependencies to
worry about.

[1]: https://git.jakstys.lt/motiejus/test-zig-cc/
2023-04-24 10:10:30 +03:00
Motiejus Jakštys 62f6b80479 README: fix the download URL
We need to preserve the old mirror URL until the new release is cut.
2023-04-21 19:40:16 +03:00
Motiejus Jakštys e0e7a4ca46
Rename bazel-zig-cc to hermetic_cc_toolchain (#36)
As it says on the tin.

Long live hermetic_cc_toolchain!
2023-04-21 10:00:03 -04:00
Motiejus Jakštys 73a9ceccfb
revert "Re-license portions of the code to Apache 2.0" (#35)
This (partially) reverts commit dfdb1f2680
2023-04-19 20:23:25 -04:00
Motiejus Jakštys 4ddd3b7a62
Make Project Origin more prominent (#34)
The link to the historic email threads seems to be a good spot for it.
2023-04-19 20:22:48 -04:00
Zhongpeng Lin 041d7f26ab
Fixing the url template in readme (#27) 2023-03-24 14:10:17 -04:00
Jonathan Baker 477f3d3690
Fixed release URL in readme (#26) 2023-03-24 09:47:42 -04:00
Motiejus Jakštys 6d2ee8cad0 replace zopfli with encoding/gzip 2023-03-14 20:25:57 +02:00
Motiejus Jakštys d3a2d43356 releaser
This is a work in progress. Next steps:

1. Add instructions to the wiki.
2. Try the new tarball on a real repository.
3. Cut the actual release.

Test output for an upcoming `v1.0.2`:

    $ bazel run //tools/releaser -- -skip_upgrades=true -tag v1.0.2
    INFO: Analyzed target //tools/releaser:releaser (1 packages loaded, 29 targets configured).
    INFO: Found 1 target...
    Target //tools/releaser:releaser up-to-date:
      bazel-bin/tools/releaser/releaser_/releaser
    INFO: Elapsed time: 1.978s, Critical Path: 1.81s
    INFO: 3 processes: 1 internal, 2 linux-sandbox.
    INFO: Build completed successfully, 3 total actions
    INFO: Running command line: bazel-bin/tools/releaser/releaser_/releaser '-skip_upgrades=true' -tag v1.0.2
    Running pre-release checks:
    - SKIPPING: go update commands
    - gazelle
    - checking if repository is clean
    Creating tag v1.0.2
    Creating archive bazel-zig-cc-v1.0.2.tar
    Compressing bazel-zig-cc-v1.0.2.tar
    Written /code/bazel-zig-cc/bazel-zig-cc-v1.0.2.tar.gz
    Release boilerplate:
    -----
    load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

    http_archive(
        name = "bazel-zig-cc",
        sha256 = "b0e857f8b32062a112305931437c5a7e1762287e27379c6d2d7173f0fa74e270",
        urls = [
            "https://mirror.bazel.build/github.com/uber/bazel-zig-cc/releases/download/v1.0.2/bazel-zig-cc-v1.0.2.tar.gz",
            "https://github.com/uber/bazel-zig-cc/releases/download/v1.0.2/bazel-zig-cc-v1.0.2.tar.gz",
        ],
    )

    load("@bazel-zig-cc//toolchain:defs.bzl", zig_toolchains = "toolchains")

    # Argument-free will pick reasonable defaults.
    zig_toolchains()

    # version, url_formats and host_platform_sha256 are can be set for those who
    # wish to control their Zig SDK version and where it is downloaded from
    zig_toolchains(
        version = "<...>",
        url_formats = [
            "https://example.org/zig/zig-{host_platform}-{version}.{_ext}",
        ],
        host_platform_sha256 = { ... },
    )
2023-03-14 20:25:57 +02:00
Motiejus Jakštys 4a42b46a99 launcher miscompilation workaround
Sometimes the launcher fails to compile with the following error
messsage:

```
error: FileNotFound
```

We cannot reproduce this in a controlled environment, but see it
happening in the wild often enough to receive repeated questions.

Since this has been escalated to Zig Software Foundation, the most
meaningful thing we can ask our users to do is apply a workaround and
wait. Let's do just that.
2023-03-14 13:52:13 +02:00
Motiejus Jakštys 5cccfb1ce1 move bin/ to tools/
The directories are not that different now. Also, clean up .bazelrc
2023-03-10 10:37:43 +02:00
Motiejus Jakštys 9b68f5ff1e
Add the mailing list archive (#8)
I will be shutting down ~motiejus/bazel-zig-cc@lists.sr.ht within days.
Now that comms are in Slack and github issues/PRs, let's add the
archive for historical reference.

It can be opened and read through like this:

  mutt -R -f mailing-list-archive.mbox
2023-03-08 09:16:22 -08:00
Motiejus Jakštys 3a204583f3
all commands: use /tmp/bazel-zig-cc (#6)
Different Bazel instances can share the zig cache all right.
2023-03-08 17:33:59 +02:00
Motiejus Jakštys f239f467a0
go mod tidy (#7) 2023-03-07 21:06:56 -08:00
Motiejus Jakštys 95ee5b0d3a
update README (#5)
- remove mentions of wine and aarch64
- contributions are now straightforward
- update maintainers (do we need this section at all?)
2023-03-07 06:15:51 +02:00
Motiejus Jakštys 3f4b4ef88e
remove contrib/ (#4)
dead code; our release and testing processes changed quite a
bit.
2023-03-06 07:37:32 -08:00
Motiejus Jakštys af9360366b
More CI (#3)
- ci only: use a separate zig cache dir for each run
- use `tools/bazel` everywhere.
- remove arm64 tests. They don't give enough value to be worth the
  brittle environment.
- remove windows execution, except for launcher. Ditto. I will probably
  bring them back.
2023-03-06 17:32:48 +02:00
Zhongpeng Lin fef28b9706 Enable Lint on CI (#2) 2023-03-06 11:46:02 +02:00
zplin a1b1be2303 Setup buildkite 2023-03-06 11:13:33 +02:00
Motiejus Jakštys e08cf8e6bf Update release notes for v1.0.1 2023-03-06 11:11:55 +02:00
Motiejus Jakštys b92dc6a533 release: sort the RC versions correctly 2023-03-06 11:11:44 +02:00
Motiejus Jakštys dfdb1f2680
Re-license portions of the code to Apache 2.0
Your old version of the software released under the old license can
still be used under the terms of the old license.

I have received the following statement from the past contributors
Luis Holanda, Jeremy Volkman and Fabian Hahn:

  I hereby confirm that I am the copyright holder or authorized by the
  copyright holder of this contribution. As such I hereby confirm that
  all contributions made to bazel-zig-cc by me or on behalf of me,
  hereby is licensed under the Apache 2.0 License
  [http://www.apache.org/licenses/LICENSE-2.0]. I am aware that my
  previous contributions will still be available under the MIT license
  as well. I confirm that I am aware of the bazel-zig-cc teams intention
  to release bazel-zig-cc under the Apache 2.0 License from release
  [1.0] and onwards, and that the bazel-zig-cc no longer will accept
  contributions made under the MIT and that any future submissions from
  myself will be considered to be licensed under Apache 2.0 unless I
  expressly state otherwise.

Copyright and re-licensing for Google and Uber employees has been
resolved internally.

Since not all contributors took action to re-license the code yet,
portions of bazel-zig-cc remain licensed as MIT.

I have started this project during my personal time with my personal
resources. I am now handing over all copyrights to this code to Uber
Technologies, Inc.
2023-03-06 11:05:13 +02:00
Motiejus Jakštys 9621da8ad6 zig sdk: use mirror.bazel.build
The current version of Zig SDK was uploaded to mirror.bazel.build via
https://github.com/bazelbuild/bazel/issues/17635 . Thanks Google!
2023-03-01 16:25:16 +02:00
Motiejus Jakštys 0d0441b9c7 ci/test: show how bad the artifact situation is 2023-02-24 12:41:32 +02:00
Motiejus Jakštys 4647e8436c launcher: remove --color=on 2023-02-24 12:01:49 +02:00
Motiejus Jakštys 28b46bfb96 ci/launcher: check fmt 2023-02-24 11:53:10 +02:00
Motiejus Jakštys 6ebe4747ae upgrade zig 2023-02-24 11:50:41 +02:00
Motiejus Jakštys 52b4f4b436 Merge branch 'motiejus_go1.20' 2023-02-24 11:43:00 +02:00
Motiejus Jakštys b4d067adbe launcher: support multi-for loop syntax
At the moment we need both old and new zig, so doing a way that's
compatible with both.
2023-02-23 09:50:53 +02:00
Motiejus Jakštys 53d89ba627 wip go 1.20 2023-02-10 15:44:56 +02:00
Motiejus Jakštys 0fd73783db ci/launcher: colors 2023-01-30 19:11:20 -08:00
Motiejus Jakštys 13a7d6800e ci/launcher: make it go faster 2023-01-30 19:08:30 -08:00
Motiejus Jakštys 11f7e1ad82 CI: fix tests 2023-01-30 18:01:01 -08:00
Motiejus Jakštys 90867df2b7 update README
- add more test commands
- add the Laurynas BazelCon talk
2023-01-30 17:18:11 -08:00
Motiejus Jakštys 3ba59fcb78 launcher: formatting
no real changes. Code looks a bit less ugly now.
2023-01-30 17:17:51 -08:00
Motiejus Jakštys f92ef2d8d4 Update release notes for v1.0.0 2023-01-24 10:58:51 +02:00
Motiejus Jakštys c03a602023 add gazelle to CI targets 2023-01-18 22:43:33 +02:00
Motiejus Jakštys 8d1e1c9fa6 res_search: fix for aarch64
On linux_aarch64 `res_search` is `__res_search@GLIBC_2.17`, which is
different from the x86_64 counterpart `__res_search@GLIBC_2.2.5`.

Also fix tests.
2023-01-18 22:39:35 +02:00
Motiejus Jakštys a47c7abc96 upgrade gazelle, add -lresolv
does not fix the test
2023-01-18 16:58:08 +02:00
Motiejus Jakštys 7b0de33070 work-around res_search on older glibcs
`res_search` became a proper symbol only in glibc 2.34. Until that it
was re-defined in headers, hitting the (in-)famous
https://github.com/ziglang/zig/issues/9485

glibc 2.34+:

    resolv/resolv.h:int             res_search (const char *, int, int, unsigned char *, int)

Old glibc:

    resolv/resolv.h:#define res_search              __res_search

Also, remove the toplevel includes, as they should never be included in
the first place: the headers are included explicitly when needed.
2023-01-18 15:45:47 +02:00
Motiejus Jakštys 90f6c07178 Merge branch 'motiejus_launcher_zig' 2023-01-03 12:55:00 +02:00
Motiejus Jakštys eced0109ca zig launcher: replace shell wrappers with a binary
Until now we needed to maintain two versions of the zig launcher: one
for Windows and one for everything else. This was problematic for two
reasons:
1. I do not know powershell and thus keep breaking the Windows wrapper
   all the time (see git history of Fabian fixing stuff that I broke).
2. This makes bazel-zig-cc dependent on the system shell, making it not
   really hermetic. So the recently added
   `--experimental_use_hermetic_linux_sandbox` does not work with
   bazel-zig-cc, unless we bind-mount a bunch of stuff: `/usr`, `/bin`,
   `/lib`, `/usr/lib:/lib`, `/usr/lib64:/lib64` and `/proc`.

Switching to a Zig-based wrapper solves both issues, and we can do this:

    bazel build "$@" \
        --experimental_use_hermetic_linux_sandbox \
        --sandbox_add_mount_pair=/proc \
        <...>

Zig itself still depends on `/proc` for `/proc/self/exe`, so we need to
keep that. I will look into reducing even that dependency separately.

Not all is nice and shiny though: this commit replaces ~80 LOC worth of
shell scripts wrappers with a singe ~300 LOC zig program, which is
arguably harder to understand. However, it is easier to change, at least
for me, because it's a single file with unit tests! Most importantly,
the gnarly code (which resolves paths and sets environment variables) is
cross-platform.

Thanks to Fabian Hahn for testing this on Windows and pointing out
errors.
2023-01-03 11:46:58 +02:00
Motiejus Jakštys 265644b3b0 license: copyright is owned by contributors.
Now that we have more than one contributor, it's fair be explicit about
it. I never asked for CLA and who owns the copyright has been clearly
documented in the README.

Note that this is not a policy change.
2022-12-23 23:45:21 +02:00
Motiejus Jakštys 872bf302b0 reduce number of zig tools
* cc: bazel supports only cpp
* wasm-ld: we do not support wasm here. Can be re-added with proper target config.
* coff: we use ld64.lld for some reason (I don't know enough about Windows to tell)

Also, now creating the tools where it makes sense to add for the target only.
2022-12-18 06:13:32 +02:00
Motiejus Jakštys bd243aad71 simplify zig wrapper
While the original intention to be "xdg-friendly" was honorable, it
never worked in practice. Bazel has a tendency to remove almost all
environment variables during the build, causing only the fallback to
remain (i.e. all zig's cache to be put to /tmp/bazel-zig-cc).

If we just accept the world as is, we can get rid of half of the shell
wrappers.
2022-12-18 05:51:55 +02:00
Jeremy Volkman cc8f113489 Use artifact_name_pattern to specify proper macos and windows artifact names.
On macos, dynamic libraries are generated as "libfoo.dylib".

On windows, executables end with ".exe", static libraries end with ".lib",
and dynamic libraries end with ".dll".
2022-12-18 05:29:04 +02:00
Jeremy Volkman 6ee01496be Use -mcpu=apple_m1 when targeting macos + aarch64 2022-12-18 05:12:24 +02:00
Jeremy Volkman cba7c5fcae Rename _target_darwin to _target_macos
This aligns with bazel's @platforms//os:macos
2022-12-18 05:11:05 +02:00
Jeremy Volkman 0db5d2d9e6 Adds supports_dynamic_linker feature
With this feature, cc_library generates a .so as well as a .a and the
"dynamic_library" output group is enabled.
2022-12-18 05:07:59 +02:00
Motiejus Jakštys 947402c843 bump zig to 0.11.0-dev.811+8ff9284c4
includes 8ff9284c46, a fix for cache poisoning when building glibc stubs
2022-12-15 22:10:09 +02:00
Motiejus Jakštys 775227fde2 Update release notes for v1.0.0-rc4 2022-12-14 14:54:41 +02:00
Motiejus Jakštys 79812dc567 includes: add `libcxxabi`
Otherwise v8 complains.
2022-12-14 00:02:23 +02:00
Motiejus Jakštys 66b9635ca8 lint 2022-12-13 18:26:56 +02:00
Motiejus Jakštys 00e126ec85 don't forget libcxx and ar 2022-12-13 17:19:42 +02:00
Motiejus Jakštys 65a8d7c4bb toolchain files: reduce the number of files in the sandbox
We already know which headers are necessary, so it's wasteful to put the
full zig_sdk to every sandbox.

This reduces the number of files in `external/zig_sdk` from 15k to 4k or
6k, depending on the action.
2022-12-13 16:40:03 +02:00
Motiejus Jakštys 0e48fc3cb4 tests: sandboxing
The slowness of bazel-zig-cc comes from setup and teardown of sandbox
directories[1]. Turning on `experimental_reuse_sandbox_directories`
speeds it up significantly.

Depending on how the tests with this go in the next few days, I will add
this recommendation to README.

This has been marked as safe to use and no longer experimental as of
Bazel master[2].

[1]: https://git.sr.ht/~motiejus/test-zigcc
[2]: https://github.com/bazelbuild/bazel/pull/16490
2022-12-12 10:24:52 +02:00
Motiejus Jakštys 4db3c242c1 Revert "remove binfmt-support"
This reverts commit 88b623287c.

Although binfmt-support is a dependency, it does not get installed on CI
2022-12-12 06:06:00 +02:00
Motiejus Jakštys 88b623287c remove binfmt-support
this is an implicit dependency of qemu-user-static
2022-12-11 22:12:07 +02:00
Motiejus Jakštys 9afee48d44 buildifier 2022-12-11 22:10:37 +02:00
Motiejus Jakštys 47d54b8099 update rules_go
this is used only in tests.
2022-12-11 22:06:28 +02:00
Motiejus Jakštys 1fb93e089c toolchain wrapper: zig_exe path is relative
Just like we know the path to zig_lib_dir, we do know the path to
zig_exe. Lets use that.

This was surfaced by experimenting with
`--experimental_use_hermetic_linux_sandbox`.
`repository_ctx.path("zig")` would return the real file system path to
zig (outside of the sandbox), which is not good when the sandbox is
hermetic.

Co-developed-by: Fabian Hahn <fabian@hahn.graphics>
2022-12-11 18:28:38 +02:00
Fabian Hahn 43b6e1c0c1 fix windows include root path 2022-12-11 18:28:19 +02:00
Motiejus Jakštys 8bc78753e2 ci/test: quote {}
This is recommended in the man page
2022-12-11 07:20:14 +02:00
Motiejus Jakštys 6feba23aee build.yml: add file
this will fix the last test in ci/test
2022-12-11 07:12:43 +02:00
Motiejus Jakštys 70ef4dd54e remove zig_include_root
It is now consistent across all 5 platforms we support:

    $ for x in *324*.tar.xz; do tar -tf $x | grep libc/ | head -5; done
    zig-0.11.0-dev.324+f61c5f3f5/lib/libc/
    zig-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/
    zig-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/
    zig-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/ntddstor-uuid.c
    zig-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/bth-uuid.c
    zig-linux-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/
    zig-linux-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/
    zig-linux-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/
    zig-linux-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/ntddstor-uuid.c
    zig-linux-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/bth-uuid.c
    zig-linux-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/
    zig-linux-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/
    zig-linux-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/
    zig-linux-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/ntddstor-uuid.c
    zig-linux-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/bth-uuid.c
    zig-macos-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/
    zig-macos-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/
    zig-macos-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/
    zig-macos-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/ntddstor-uuid.c
    zig-macos-aarch64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/bth-uuid.c
    zig-macos-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/
    zig-macos-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/
    zig-macos-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/
    zig-macos-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/ntddstor-uuid.c
    zig-macos-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/mingw/libsrc/bth-uuid.c
    $ for x in *324*.zip; do unzip -l $x | grep libc/ | head -5; done
            0  2022-11-28 19:26   zig-windows-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/
            0  2022-11-28 19:26   zig-windows-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/darwin/
       301018  2022-11-28 19:26   zig-windows-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/darwin/libSystem.11.tbd
       309693  2022-11-28 19:26   zig-windows-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/darwin/libSystem.12.tbd
       326079  2022-11-28 19:26   zig-windows-x86_64-0.11.0-dev.324+f61c5f3f5/lib/libc/darwin/libSystem.13.tbd
2022-12-07 12:17:44 +02:00
Motiejus Jakštys 9eaebe0ad2 rules_go: cherry-pick a gorace workaround 2022-11-30 21:10:08 +02:00
Motiejus Jakštys bf88ddb2d5 ci/test: fail on error 2022-11-29 14:23:45 +02:00
Motiejus Jakštys 6bd588c07b upgrade to zig 0.11.0-dev.324+f61c5f3f5
includes a zig upstream fix for absolute directories: https://github.com/ziglang/zig/issues/13050
2022-11-29 13:51:57 +02:00
Motiejus Jakštys 0f981d591e WORKSPACE: bump go to 1.19.3 2022-11-28 11:06:03 +02:00
Motiejus Jakštys 88e7e47ed2 zig_include_root: fixes
- do not pass where it's unnecessary
- remove trailing slash. That conflicts with ziglang/zig/pull/13596
2022-11-21 06:08:22 +02:00
Motiejus Jakštys dd4239407e lint 2022-11-17 15:47:46 +02:00
Motiejus Jakštys 87fb993408 zig build: add -target automatically 2022-11-17 14:04:25 +02:00
Motiejus Jakštys d195402030 zig build-exe, build-lib, build-obj
This change makes it possible to build zig sources from Bazel rules by
calling the right tool.
2022-11-17 13:51:31 +02:00
Motiejus Jakštys 50103497f6 upgrade zig sdk
The uber1 was broken on x86_64-macos
2022-10-19 11:38:46 +03:00
Motiejus Jakštys abf00a525c contrib/build-and-archive: fix lib dir for macos-x86_64 2022-10-13 15:50:19 +03:00
Motiejus Jakštys 557b726d62 shellcheck 2022-10-13 12:48:43 +03:00
Motiejus Jakštys 7383c4984e ci/test: add tests for exact expected artifacts.
Cache reuse in zig is currently quite fragile (we are using a forked
version with github.com/ziglang/zig/issues/13051), so ensure that it
does not regress when zig SDK upgrades.
2022-10-13 12:41:07 +03:00
Motiejus Jakštys eba4adbbb4 WORKSPACE: upgrade to go1.19.2 2022-10-13 12:30:06 +03:00
Motiejus Jakštys ae2746ebeb zig_include_root: support windows too
No real change, just make the logic consistent with Linux/Darwin.

This prepares us for windows-aarch64, which will probably have lib in
lib/zig.
2022-10-13 06:11:25 +03:00
Motiejus Jakštys fed463e1d0 bump zig to 4301+uber1
This is a patched zig release with 13051[1]. While we are waiting for a
resolution on #13050 (which @andrewrk is working on).

[1]: https://patch-diff.githubusercontent.com/raw/ziglang/zig/pull/13051
2022-10-13 05:57:56 +03:00
Motiejus Jakštys e5eff0c911 zig include root: it depends on the OS
Previously the toolchain wrappers used to assume the zig_lib_dir is
always in "lib". However, it depends on the OS: macos-x86_64 and
linux-aarch64 have it in "lib/zig".

Update the wrapper scripts to reflect that.
2022-10-11 14:57:44 +03:00
Motiejus Jakštys 359cb0fe6d README: fix copy/pasta 2022-10-08 15:01:00 +03:00
Motiejus Jakštys c0a58691eb README: rollback to v0.9.2
v1.0-rc* all have a performance regression due to
https://github.com/ziglang/zig/issues/13050; follow along there.
2022-10-07 12:28:07 +02:00
Motiejus Jakštys 5743865c26 Revert "resolve ZIG_LIB_DIR before passing it to zig"
This reverts commit 3a8c2eecfe.

Causes the dependency paths to be absolute.
2022-10-05 14:48:19 +03:00
Motiejus Jakštys 3a8c2eecfe resolve ZIG_LIB_DIR before passing it to zig
That way the zig's cache keys will be the same for all sandboxes.

Also see https://github.com/ziglang/zig/issues/13050#issuecomment-1268074032
2022-10-05 14:24:19 +03:00
Motiejus Jakštys c5e3c25015 tests: add linux_arm64_musl, simplify BUILD 2022-10-03 16:24:59 +03:00
Motiejus Jakštys c9d2442143 tests: remove linux_amd64_gnu.2.31
It does not add anything beyond linux_amd64_gnu.2.28.
2022-10-03 06:48:12 +03:00
Motiejus Jakštys 50ca49181f Zig cache hacks/optimizations
Zig does not always share `libc++.a` and the glibc shims. We have observed
to be caused for 2 reasons:

- For some commands Go `chdir`s to `/tmp/`. This causes `ZIG_LIB_DIR` to be
  absolute, which blows the cache key to find the right `libc++.a` (because Zig
  thinks we are using a different lib dir to compile libc++). This is currently
  worked around in `toolchain/defs.bzl` by overfitting to Go and returning
  early from that particular invocation.
- Sometimes Bazel's sandbox messes up Zig's cache keys. If one runs without the
  sandbox (`--spawn_strategy=standalone`), the cache hit rate and thus the
  build time are much better.

This is actively investigated. I am adding `--spawn_strategy=standalone`
to CI to see the speedup that it provides. I will rollback it later.
2022-10-03 05:50:53 +03:00
Motiejus Jakštys f30a3afdde remove redundant flags
- -target
- linkopts
2022-10-01 23:27:58 +03:00
Motiejus Jakštys b1209d0165 gohack only in tools/<arch>/c++ 2022-10-01 18:53:38 +03:00
Motiejus Jakštys 8919c5b703 fix tool paths for different OSes/environments 2022-10-01 14:39:59 +03:00
Motiejus Jakštys 9b510810cc split all tools to per-codename subdirectories
Go ignores CFLAGS for some commands. That causes unnecessary build of
glibc shims (and libc++.a).

In the GCC days cross-compiler toolchains used to expose a "tool" per
architecture. Let's do the same here. Then Go cannot cheat with skipping
CFLAGS which we normally *always* expect.

The wrapper code is getting gnarly, I know. But it still fits in my head
somehow, so we can still keep adding to it.
2022-10-01 14:35:27 +03:00
Motiejus Jakštys e96f5bb59c lint 2022-10-01 05:31:50 +03:00
Motiejus Jakštys e9638443c6 tools wrapper: fix for windows
'common' is not a thing in Windows; but it needs to be defined.

This makes v1.0.0-rc3 broken for Windows, and not useful (yet) for
anyone else but me. Revert the README update too.
2022-10-01 05:12:45 +03:00
Motiejus Jakštys 53975784ea ci: catch easy problems early 2022-09-30 15:18:21 +03:00
Motiejus Jakštys c4ab1cfaea buildifier 2022-09-30 15:17:49 +03:00
Motiejus Jakštys 873a703fc1 Update release notes for v1.0.0-rc3 2022-09-30 15:03:51 +03:00
Motiejus Jakštys 9e8d89b40d CGo-specific: reduce unneeded compilation
This adds an ugly workaround for
https://go-review.googlesource.com/c/go/+/436884 ; fingers crossed.

Impact: ~20-25s reduced first build time for a particular
architecture+glibc.
2022-09-30 14:24:56 +03:00
Motiejus Jakštys ae57dac00d ci/lint: make it fail in CI
It would "fix" the files on CI and not fail. ci/lint should verify they
are OK first, and give guidance to fix them.

also, `shellcheck -x` is no longer necessary; we can use the plain one
now.
2022-09-29 12:11:34 +03:00
Motiejus Jakštys b9ded4caeb Update release notes for v1.0.0-rc2 2022-09-29 11:32:57 +03:00
Fabian Hahn 04b95526ae fix Windows host support with relative paths 2022-09-29 11:32:19 +03:00
Motiejus Jakštys d841763745 update versions and sizes 2022-09-27 15:43:24 +03:00
Motiejus Jakštys 530dd1a825 Update release notes for v1.0.0-rc1 2022-09-27 15:41:06 +03:00
Motiejus Jakštys 08b02cb7d2 rename absolute_path to zig_path
this is not an absolute path.
2022-09-27 15:37:28 +03:00
Motiejus Jakštys 0302fb630a defs: fix import paths
Empirically these need to come from most specfic to least specific.

The error message is as follows:

    In file included from test/c/main.c:1:
    In file included from external/zig_sdk/lib/libcxx/include/stdio.h:107:
    In file included from external/zig_sdk/lib/libc/include/generic-glibc/stdio.h:38:
    external/zig_sdk/lib/libc/include/generic-glibc/bits/types.h:139:3: error:
    # error
      ^
    external/zig_sdk/lib/libc/include/generic-glibc/bits/types.h:145:1: error: unknown type name '__STD_TYPE'
    __STD_TYPE __DEV_T_TYPE __dev_t;        /* Type of device numbers.  */

Dissected `generic-glibc/bits/types.h:#error`:

    #if __WORDSIZE == 32
    <...>
    # define __STD_TYPE		__extension__ typedef
    #elif __WORDSIZE == 64
    <...>
    # define __STD_TYPE		typedef
    #else
    # error
    #endif

So we do not have the `__WORDSIZE`. Where does that come from? Probably from a
directory that has an `x86_64` in it. How does that get included? Let's start
with `lib/libcxx/include/stdio.h`:

	16 #include_next <stdio.h>

Now previously our `c++` command line looked like this:
    external/zig_sdk/tools/c++ \
    <...>
    -Iexternal/zig_sdk/lib/include \
    -Iexternal/zig_sdk/lib/libcxx/include \
    -Iexternal/zig_sdk/lib/libcxxabi/include \
    -Iexternal/zig_sdk/lib/libunwind/include \
    -Iexternal/zig_sdk/lib/libc/include/generic-glibc \
    -Iexternal/zig_sdk/lib/libc/include/any-linux-any \
    -Iexternal/zig_sdk/lib/libc/include/x86_64-linux-gnu \
    -Iexternal/zig_sdk/lib/libc/include/x86_64-linux-any \
    -Iexternal/zig_sdk/lib/libc/include/x86-linux-any \
    -Iexternal/zig_sdk/glibc-hacks \
    <...>

So the next place it will find `stdio.h` is in `generic-glibc`, which already
uses the `__WORDSIZE`. If we make the "next" include to be the arch-specific
one instead of the generic-glibc, things start compiling again.

Fix the same fo musl.
2022-09-27 10:15:02 +03:00
Ken Micklas ab59c18d60 WIP: use relative paths for tools 2022-09-27 10:14:59 +03:00
Motiejus Jakštys f6b16c386f upgrade to zig 0.10.0-dev.4166+cae76d829
- llvm 15
- uses less space for /tmp/bazel-zig-cc
2022-09-25 13:51:59 +03:00
Motiejus Jakštys 9dbba15064 partially revert c023c217a5
The `toolchain/private/cc_toolchains.bzl` changes were added by accident.
2022-09-22 13:32:17 +03:00
Motiejus Jakštys 026d234bc2 Revert "bump to zig 0.10.0-dev.3999+fda6d4477"
This reverts commit f161caf870.

This version has a bug that's been fixed upstream, but a new nightly did
not come out yet: https://github.com/ziglang/zig/issues/12858
2022-09-22 11:57:39 +03:00
Motiejus Jakštys b3a8b5b169 update README
- update fixed upstream issues
- add the mailing list e-mail address
2022-09-22 11:57:13 +03:00
Motiejus Jakštys c023c217a5 contrib/own_zig.md: clarify zig-bootstrap instructions 2022-09-22 11:39:39 +03:00
Ken Micklas 64b6c702db Fix include root path for macos-aarch64 2022-09-15 22:19:06 +03:00
Ken Micklas ba0b0376ea Move Label dependencies above archive fetching to reduce restarts 2022-09-15 22:18:43 +03:00
Motiejus Jakštys f161caf870 bump to zig 0.10.0-dev.3999+fda6d4477
llvm 15
2022-09-14 11:54:04 +03:00
Motiejus Jakštys f0f2ce597e Update release notes for v0.9.2 2022-09-12 13:46:54 +03:00
Motiejus Jakštys bf32b5ee92 repository_rule: depend on $BAZEL_ZIG_CC_CACHE_PREFIX
Otherwise the repository doesn't get re-downloaded/wiped when this
parameter changes.
2022-09-12 13:46:07 +03:00
Motiejus Jakštys 2f0ae4affa target_musl: remove linkopts
These flags were added in this commit:

    commit 58a04fbfec
    Author: Motiejus Jakštys <motiejus@jakstys.lt>
    Date:   Tue Jun 15 09:10:15 2021 +0300

        bump zig

I am still quite puzzled on why I did this and in such a commit: it
bumps the zig version and changes the linker flag.

Working alone, experimentation time? Probably something like this.
Anyhow, this is a good example of the real value of code reviews.
2022-09-02 13:21:39 +03:00
Motiejus Jakštys e1d699f7ed upgrade to 0.10.0-dev.3838+77f31ebbb 2022-09-01 09:28:43 +03:00
Motiejus Jakštys 79674a1d96 replace /usr/bin/env sh with /bin/sh
/bin/sh exists in NixOS and in fact is the only file in `/bin`:

    $ docker run -ti --rm nixos/nix ls /bin /usr/bin
    /bin:
    sh

    /usr/bin:
    env

More context in
https://lists.sr.ht/~motiejus/bazel-zig-cc/%3CCAFVMu-pvNx%2BpeQYdge_fvmSGrHDmn78VmoYwTxkDgMjbzfVAwQ%40mail.gmail.com%3E
2022-08-30 09:21:27 +03:00
Motiejus Jakštys 342b239bb9 nit: style of cache prefix definition 2022-08-29 14:44:38 +03:00
Motiejus Jakštys e861c5b4be cache_prefix: clean up OS logic
'Windows' and 'Other' and guessing of cache prefix was convoluted
2022-08-29 13:01:09 +03:00
Fabian Hahn a410141ca4 add cache prefix configuration for Windows 2022-08-28 20:18:21 +03:00
Motiejus Jakštys a0fefdb15b Update release notes for v0.9.1 2022-08-26 11:35:12 +03:00
66 changed files with 18920 additions and 724 deletions

View File

@ -4,5 +4,6 @@ test --test_output=errors
build --verbose_failures
build --worker_sandboxing
build --experimental_reuse_sandbox_directories
build --incompatible_enable_cc_toolchain_resolution
build --action_env BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1

View File

@ -1 +1 @@
5.2.0
6.0.0

View File

@ -1,35 +0,0 @@
image: debian/stable
packages:
- direnv
- shellcheck
- qemu-user-static
- binfmt-support
- moreutils
- wine64
sources:
- https://git.sr.ht/~motiejus/bazel-zig-cc
environment:
CC: /usr/bin/false
triggers:
- action: email
condition: failure
to: motiejus+srht@jakstys.lt
tasks:
- setup: |
sudo apt-get purge gcc -y && sudo apt-get autoremove -y
sudo dpkg --add-architecture arm64
sudo apt-get update
sudo apt-get install libc6:arm64 -y
- list_toolchains_platforms: |
cd bazel-zig-cc; . .envrc
./ci/list_toolchains_platforms
- test: |
cd bazel-zig-cc; . .envrc
./ci/test --color=yes --curses=yes
- lint: |
cd bazel-zig-cc; . .envrc
./ci/lint
git diff --exit-code
- test_release: |
cd bazel-zig-cc; . .envrc
./ci/release

34
.buildkite/pipeline.yml Normal file
View File

@ -0,0 +1,34 @@
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
steps:
- label: "Test"
command: ci/test
- label: "Lint"
command: ci/lint
- label: "List Platforms"
command: ci/list_toolchains_platforms
- label: "Test Release and Launcher scripts"
command: |
git config --global user.email "buildkite@example.com"
git config --global user.name "Buildkite Bot"
echo "--- ci/release"
ci/release
echo "--- ci/launcher"
ci/launcher
- label: "Test Launcher on wine64"
plugins:
- docker#v5.5.0:
image: "debian:stable"
command: |
apt-get update && apt-get install --no-install-recommends -y \
wine64 python3 ca-certificates
ci/launcher-wine64
- label: "mod-tidy and update-repos"
command: |
tools/mod-tidy
git diff --exit-code
agents:
- "queue=init"
- "docker=*"

46
.envrc
View File

@ -1,45 +1,7 @@
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
set -eu
BIN_DIR="$(git rev-parse --show-toplevel)/bin"
BIN_DIR="$(git rev-parse --show-toplevel)/tools"
export PATH="$BIN_DIR:$PATH"
_u_bzl=https://github.com/bazelbuild/bazelisk/releases/download/v1.12.0/bazelisk-
_u_bldf=https://github.com/bazelbuild/buildtools/releases/download/5.1.0/buildifier-
if [[ "${PRINT_TOOL_HASHES:-no}" = "yes" ]]; then
for os in linux darwin; do
for arch in amd64 arm64; do
hash_bzl=$(direnv fetchurl "${_u_bzl}$os-$arch")
hash_bldf=$(direnv fetchurl "${_u_bldf}$os-$arch")
echo -e "bzl: $os-$arch\t$hash_bzl"
echo -e "bldf: $os-$arch\t$hash_bldf"
done
done
fi
# to fetch the hashes, run:
# $ PRINT_TOOL_HASHES=yes bash .envrc
case "$(uname | tr A-Z a-z)-$(uname -m)" in
linux-x86_64)
bzl=$(direnv fetchurl "${_u_bzl}linux-amd64" sha256-awvLLqFbyhb/+r5v2nWANEA3U1TAhUgP42HSy/MlAds=)
bldf=$(direnv fetchurl "${_u_bldf}linux-amd64" sha256-Ur9rECy0+IRk4ZfKrAbWl5P6KwX1rVCn579vvWVmSKM=)
;;
linux-aarch64)
bzl=$(direnv fetchurl "${_u_bzl}linux-arm64" sha256-KdhhykjfJKPo3sV/sAUIumZKMZIQR7JobDjPmiDUY58=)
bldf=$(direnv fetchurl "${_u_bldf}linux-arm64" sha256-kX1ZnbsEDmOuen4a23ENIFeBGQL9yeNczpJev9lm7rg=)
;;
darwin-x86_64)
bzl=$(direnv fetchurl "${_u_bzl}darwin-amd64" sha256-cM9/50gI0WQY03H+uMzU58RCFdsD0sT/x1t2e3ZUCfs=)
bldf=$(direnv fetchurl "${_u_bldf}darwin-amd64" sha256-yTeNn0KT/DjsVKCPvHTnqdKJFNrmiRM0QB5Z849uZdw=)
;;
darwin-arm64)
bzl=$(direnv fetchurl "${_u_bzl}darwin-arm64" sha256-NFu4uQDWue90I06enkE67Tj7Ke8lXkrhisYb9KYQLYQ=)
bldf=$(direnv fetchurl "${_u_bldf}darwin-arm64" sha256-dF/rXqlstv85p2soIcV1kf1wtSgyVWJIbUe10IkA4uQ=)
;;
*)
>&2 echo "unsupported architecture tuple $(uname | tr A-Z a-z)-$(uname -m)"
exit 1;;
esac
ln -sf "${bzl}" "$BIN_DIR/bazel"
ln -sf "${bldf}" "$BIN_DIR/buildifier"

6
.gitignore vendored
View File

@ -1,11 +1,7 @@
*~
*.sw[op]
/bin/bazel
/bin/buildifier
/bin/bazelisk-*
/bazel-bazel-zig-cc
/bazel-hermetic_cc_toolchain
/bazel-bin
/bazel-out
/bazel-testlogs

19
BUILD
View File

@ -1,7 +1,20 @@
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
load("@bazel_gazelle//:def.bzl", "gazelle")
# gazelle:map_kind go_binary go_binary //rules:rules_go.bzl
# gazelle:build_file_name BUILD
# gazelle:prefix git.sr.ht/~motiejus/bazel-zig-cc
# gazelle:prefix github.com/uber/hermetic_cc_toolchain
# gazelle:exclude tools.go
gazelle(name = "gazelle")
gazelle(
name = "gazelle-update-repos",
args = [
"-from_file=go.mod",
"-to_macro=repositories.bzl%go_repositories",
"-prune",
],
command = "update-repos",
)

View File

@ -1,4 +1,5 @@
Copyright 2021 Motiejus Jakštys
Copyright 2021-2022 bazel-zig-cc contributors
Copyright 2023 hermetic_cc_toolchain contributors
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in

26
MODULE.bazel Normal file
View File

@ -0,0 +1,26 @@
module(
name = "hermetic_cc_toolchain",
version = "1.0.1",
)
bazel_dep(name = "rules_go", version = "0.38.1", dev_dependency = True)
go_sdk = use_extension(
"@rules_go//go:extensions.bzl",
"go_sdk",
dev_dependency = True,
)
use_repo(go_sdk, "go_default_sdk")
bazel_dep(name = "gazelle", version = "0.29.0", dev_dependency = True)
go_deps = use_extension(
"@gazelle//:extensions.bzl",
"go_deps",
dev_dependency = True,
)
go_deps.from_file(go_mod = "//:go.mod")
use_repo(
go_deps,
"com_github_bazelbuild_buildtools",
)

274
README.md
View File

@ -1,42 +1,55 @@
[![builds.sr.ht status](https://builds.sr.ht/~motiejus/bazel-zig-cc.svg)](https://builds.sr.ht/~motiejus/bazel-zig-cc)
[![Build status](https://badge.buildkite.com/58cd1ecad012ad0ddee9a868ec11464025a979045318a0bc3f.svg)](https://buildkite.com/uberopensource/hermetic-cc-toolchain)
# Bazel zig cc toolchain
# Hermetic CC toolchain
This is a C/C++ toolchain that can (cross-)compile C/C++ programs. It contains
clang-14, musl, glibc 2-2.34, all in a ~50MB package. Read
This is a C/C++ toolchain that can (cross-)compile C/C++ programs on top of
`zig cc`. It contains clang-16, musl, glibc 2-2.34, all in a ~40MB package.
Read
[here](https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html)
about zig-cc; the rest of the README will present how to use this toolchain
from Bazel.
Configuring toolchains in Bazel is complex, under-documented, and fraught with
peril. I, the co-author of bazel-zig-cc, am still confused on how this all
works, and often wonder why it works at all. That aside, we made the our best
effort to make bazel-zig-cc usable for your C/C++/CGo projects, with as many
guardrails as we could install.
peril. We, the team behind `hermetic_cc_toolchain`,are still confused on how
this all works, and often wonder why it works at all. That aside, we made
our best effort to make `hermetic_cc_toolchain` usable for your C/C++/CGo
projects, with as many guardrails as we could install.
While copy-pasting the code in your project, attempt to read and understand the
text surrounding the code snippets. This will save you hours of head
scratching, I promise.
scratching.
# Usage
## Project Origin
This repository is cloned from and is based on Adam Bouhenguel's [bazel-zig-cc][ajbouh],
and was later developed at `sr.ht/~motiejus/bazel-zig-cc`. After a while this repository
was moved to [the Uber GitHub repository](https://github.com/uber) and renamed to `hermetic_cc_toolchain`.
> **Our special thanks to Adam for coming up with the idea - and creating the original version of `bazel-zig-cc`
> and publishing it. His idea and work helped make the concept of using Zig with
> Bazel a reality; now we all can benefit from it.**
## Usage
Add this to your `WORKSPACE`:
```
BAZEL_ZIG_CC_VERSION = "v0.9.0"
HERMETIC_CC_TOOLCHAIN_VERSION = "v1.0.1"
http_archive(
name = "bazel-zig-cc",
sha256 = "7f1a9ebfd2a68965f7c0744ae6398ba51193e8b264031e74a3c96462427ad302",
strip_prefix = "bazel-zig-cc-{}".format(BAZEL_ZIG_CC_VERSION),
urls = ["https://git.sr.ht/~motiejus/bazel-zig-cc/archive/{}.tar.gz".format(BAZEL_ZIG_CC_VERSION)],
sha256 = "e9f82bfb74b3df5ca0e67f4d4989e7f1f7ce3386c295fd7fda881ab91f83e509",
strip_prefix = "bazel-zig-cc-{}".format(HERMETIC_CC_TOOLCHAIN_VERSION),
urls = [
"https://mirror.bazel.build/github.com/uber/bazel-zig-cc/releases/download/{0}/{0}.tar.gz".format(HERMETIC_CC_TOOLCHAIN_VERSION),
"https://github.com/uber/hermetic_cc_toolchain/releases/download/{0}/{0}.tar.gz".format(HERMETIC_CC_TOOLCHAIN_VERSION),
],
)
load("@bazel-zig-cc//toolchain:defs.bzl", zig_toolchains = "toolchains")
# version, url_formats and host_platform_sha256 are optional, but highly
# recommended. Zig SDK is by default downloaded from dl.jakstys.lt, which is a
# tiny server in the closet of Yours Truly.
# version, url_formats and host_platform_sha256 are optional for those who
# want to control their Zig SDK version.
zig_toolchains(
version = "<...>",
url_formats = [
@ -56,13 +69,13 @@ The snippets above will download the zig toolchain and make the bazel
toolchains available for registration and usage. If you do nothing else, this
may work. The `.bazelrc` snippet instructs Bazel to use the registered "new
kinds of toolchains". All above are required regardless of how wants to use it.
The next steps depend on how one wants to use bazel-zig-cc. The descriptions
The next steps depend on how one wants to use `hermetic_cc_toolchain`. The descriptions
below is a gentle introduction to C++ toolchains from "user's perspective" too.
## Use case: manually build a single target with a specific zig cc toolchain
### Use case: manually build a single target with a specific zig cc toolchain
This option is least disruptive to the workflow compared to no hermetic C++
toolchain, and works best when trying out or getting started with bazel-zig-cc
toolchain, and works best when trying out or getting started with `hermetic_cc_toolchain`
for a subset of targets.
To request Bazel to use a specific toolchain (compatible with the specified
@ -77,7 +90,7 @@ bazel build \
There are a few things going on here, let's try to dissect them.
### Option `--platforms @zig_sdk//platform:linux_arm64`
#### Option `--platforms @zig_sdk//platform:linux_arm64`
Specifies that the our target platform is `linux_arm64`, which resolves into:
@ -96,7 +109,7 @@ platform(
compatible with (in Bazelspeak, `target_compatible_with`) **all of the**
`["@platforms//os:linux", "@platforms//cpu:aarch64"]`.
### Option `--toolchains=@zig_sdk//toolchain:linux_arm64_musl`
#### Option `--toolchains=@zig_sdk//toolchain:linux_arm64_musl`
Inspect first (`@platforms//cpu:aarch64` is an alias to
`@platforms//cpu:arm64`):
@ -110,7 +123,7 @@ toolchain(
generator_location = "toolchain/BUILD:7:19",
toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
target_compatible_with = ["@platforms//os:linux", "@platforms//cpu:aarch64", "@zig_sdk//libc:unconstrained"],
toolchain = "@zig_sdk//private:aarch64-linux-musl_cc",
toolchain = "@zig_sdk//:aarch64-linux-musl_cc",
)
```
@ -125,7 +138,7 @@ which will compile and link the C/C++ code with musl.
`@zig_sdk//libc:unconstrained` will become important later.
### Same as above, less typing (with `--config`)
#### Same as above, less typing (with `--config`)
Specifying the platform and toolchain for every target may become burdensome,
so they can be put used via `--config`. For example, append this to `.bazelrc`:
@ -141,7 +154,7 @@ And then building to linux-arm64-musl boils down to:
bazel build --config=linux_arm64_musl //test/go:go
```
## Use case: always compile with zig cc
### Use case: always compile with zig cc
Instead of adding the toolchains to `.bazelrc`, they can be added
unconditionally. Append this to `WORKSPACE` after `zig_toolchains(...)`:
@ -166,7 +179,7 @@ build --action_env BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1
From Bazel's perspective, this is almost equivalent to always specifying
`--extra_toolchains` on every `bazel <...>` command-line invocation. It also
means there is no way to disable the toolchain with the command line. This is
useful if you find bazel-zig-cc useful enough to compile for all of your
useful if you find `hermetic_cc_toolchain` useful enough to compile for all of your
targets and tools.
With `BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1` Bazel stops detecting the default
@ -175,7 +188,7 @@ auto-detection (read: fallback to non-hermetic toolchain) is a footgun best
avoided. This option is not documented in bazel, so may break. If you intend to
use the hermetic toolchain exclusively, it won't hurt.
## Use case: zig-cc for targets for multiple libc variants
### Use case: zig-cc for targets for multiple libc variants
When some targets need to be build with different libcs (either different
versions of glibc or musl), use a linux toolchain from
@ -185,7 +198,7 @@ when building for a specific libc. For example, in `WORKSPACE`:
```
register_toolchains(
"@zig_sdk//libc_aware/toolchain:linux_amd64_gnu.2.19",
"@zig_sdk//libc_aware/toolchain:linux_amd64_gnu.2.28",
"@zig_sdk//libc_aware/toolchain:linux_arm64_gnu.2.28",
"@zig_sdk//libc_aware/toolchain:x86_64-linux-musl",
)
```
@ -251,9 +264,9 @@ $ bazel query "attr(constraint_setting, @zig_sdk//libc:variant, @zig_sdk//...)"
`@zig_sdk//libc:unconstrained` is a special value that indicates that no value
for the constraint is specified. The non libc aware linux toolchains are only
compatible with this value to prevent accidental silent fallthrough to them.
This is a guardrail. Thanks, future me!
This is a guardrail.
# Note: Naming
## Note: Naming
Both Go and Bazel naming schemes are accepted. For convenience with
Go, the following Go-style toolchain aliases are created:
@ -272,12 +285,12 @@ used, run:
$ bazel query @zig_sdk//toolchain/...
```
# Incompatibilities with clang and gcc
## Incompatibilities with clang and gcc
`zig cc` is *almost* a drop-in replacement for clang/gcc. This section lists
some of the discovered differences and ways to live with them.
## UBSAN and "SIGILL: Illegal Instruction"
### UBSAN and "SIGILL: Illegal Instruction"
`zig cc` differs from "mainstream" compilers by [enabling UBSAN by
default][ubsan1]. Which means your program may compile successfully and crash
@ -290,19 +303,19 @@ SIGILL: illegal instruction
This flag encourages program authors to fix the undefined behavior. There are
[many ways][ubsan2] to find the undefined behavior.
# Known Issues In bazel-zig-cc
## Known Issues In `hermetic_cc_toolchain`
These are the things you may stumble into when using bazel-zig-cc. I am
These are the things you may stumble into when using `hermetic_cc_toolchain`. We are
unlikely to implement them any time soon, but patches implementing those will
be accepted. See [Questions & Contributions](#questions-amp-contributions) on
how to contribute.
## Zig cache location
### Zig cache location
Currently zig cache is in `$HOME`, so `bazel clean --expunge` does not clear
the zig cache. Zig's cache should be stored somewhere in the project's path.
## zig cc concurrency
### zig cc concurrency
- Bazel spawns up to `nproc` workers.
- For each of those, Go may spawn up to `nproc` processes while compiling.
@ -312,53 +325,50 @@ the zig cache. Zig's cache should be stored somewhere in the project's path.
Tracked in [ziglang/zig #12101 RFC: -j/--jobs for zig
subcommands](https://github.com/ziglang/zig/issues/12101).
## zig cc cache
Both Zig and Bazel cache the artifacts, requiring double disk space for cache.
Zig may remove caching for external builds. Tracked in [ziglang/zig #12317
Possibility to disable caching for user
code](https://github.com/ziglang/zig/issues/12317)
## OSX: sysroot
### OSX: sysroot
For non-trivial programs (and for all darwin/arm64 cgo programs) MacOS SDK may
be necessary. Read [Jakub's comment][sysroot] about it. Support for OSX sysroot
is currently not implemented.
## OSX: different OS targets (Catalina -- Monterey)
### OSX: different OS targets (Catalina -- Monterey)
[Zig 0.9.0](https://ziglang.org/download/0.9.0/release-notes.html#macOS) may
target macos.10 (Catalina), macos.11 (Big Sur) or macos.12 (Monterey). It
currently targets the lowest version, without ability to change it.
## Windows only: output file extensions
## Known Issues In Upstream
For Windows targets Bazel uses Unix extensions for output binaries. Those may
need to be renamed before deploying to the Windows system. Here is a primer:
This section lists issues that we have stumbled into when using `zig cc`, and is
outside of `hermetic_cc_toolchain`'s control.
| Binary type | Bazel extension | Windows extension |
|----------------|-----------------|-------------------|
| Static library | .a | .lib |
| Shared library | .so | .dll |
| Executable | (no extension) | .exe |
### using glibc 2.27 or older
# Known Issues In Upstream
This section lists issues that I've stumbled into when using `zig cc`, and is
outside of bazel-zig-cc's control.
## using glibc 2.27 or older
**Severity: Low**
**Severity: Medium**
Task: [ziglang/zig #9485 glibc 2.27 or older: fcntl64 not found, but zig's glibc headers refer it](https://github.com/ziglang/zig/issues/9485)
Background: when glibc 2.27 or older is selected, it may miss `fcntl64`. A
workaround is applied for `x86_64`, but not for aarch64. The same workaround
may apply to aarch64, but the author didn't find a need to test it (yet).
may apply to aarch64, our team did not find a need to test it (yet).
# Closed Upstream Issues
In September 2022 the severity has been bumped to Medium, because glibc header
updates cause a lot of churn when upgrading the SDK, when it shouldn't cause
any at all.
Feel free to track [Universal headers][universal-headers] project for a fix.
### Number of libc stubs with Go 1.20+
Until Go 1.19 the number of glibc stubs that needed to be compiled was strictly
controlled. Go 1.20 no longer ships with pre-compiled archive files for the
standard library, and it generates them on the fly, causing many extraneous
libc stubs. Therefore, the initial compilation will take longer until those
stubs are pre-cached.
## Closed Upstream Issues
- [ziglang/zig #12317 Possibility to disable caching for user](https://github.com/ziglang/zig/issues/12317) (CLOSED, thanks andrewrk and motiejus)
- [golang/go #52690 Go linker does not put libc onto the linker line](https://github.com/golang/go/issues/52690) (CLOSED, thanks andrewrk and motiejus)
- [ziglang/zig #10386 zig cc regression in 0.9.0](https://github.com/ziglang/zig/issues/10386) (CLOSED, thanks Xavier)
- [ziglang/zig #10312 macho: fail if requested -framework is not found](https://github.com/ziglang/zig/pull/10312) (CLOSED, thanks kubkon)
@ -373,7 +383,9 @@ may apply to aarch64, but the author didn't find a need to test it (yet).
- [rules/go #2894 Per-arch_target linker flags](https://github.com/bazelbuild/rules_go/issues/2894) (CLOSED, thanks mjonaitis)
- [golang/go #46644 cmd/link: with CC=zig: SIGSERV when cross-compiling to darwin/amd64](https://github.com/golang/go/issues/46644) (CLOSED, thanks kubkon)
# Host Environments
... and more.
## Host Environments
This repository is used on the following (host) platforms:
@ -383,120 +395,78 @@ This repository is used on the following (host) platforms:
- `darwin_arm64`, the M1.
- `windows_amd64`, a.k.a. `x64`.
The tests are running (CId) on linux-amd64, and are assuming the kernel is
configured to run `linux_arm64` and `windows_amd64` binaries.
The tests are running (CId) on linux-amd64.
There are two reasonably convenient ways to configure `linux_arm64` emulation:
### Transient docker environment
1. Install and configure [`binfmt_misc`][binfmt_misc]:
```
apt install qemu-user-static binfmt-support
```
2. Magic of the internet:
```
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
```
## Transient docker environment
A standalone Docker environment to play with bazel-zig-cc:
A standalone Docker environment to play with `hermetic_cc_toolchain`:
```
$ docker run -e CC=/usr/bin/false -ti --rm -v "$PWD:/x" -w /x debian:bullseye-slim
# apt update
# apt install --no-install-recommends -y direnv git shellcheck ca-certificates
# eval "$(direnv hook bash)" && direnv allow
# ./ci/test
# apt update && apt install --no-install-recommends -y shellcheck ca-certificates python3
# ./ci/lint
```
Some of the tests rely on `qemu-aarch64` to run arm64 binaries and wine for
Windows binaries. Therefore, with the setup above, these tests will fail.
To install *all* dependencies, so all tests can pass:
```
$ docker run -e CC=/usr/bin/false -ti --rm -v "$PWD:/x" -w /x debian:bullseye-slim
# dpkg --add-architecture arm64 && apt update
# apt install --no-install-recommends -y direnv git shellcheck ca-certificates libc6:arm64 qemu-user-static wine64
# eval "$(direnv hook bash)" && direnv allow
# ./ci/launcher
# ./ci/test
# ./ci/lint
```
# Questions & Contributions
## Communication
Project's mailing list is [~motiejus/bazel-zig-cc][mailing-list]. The mailing
list is used for:
We maintain two channels for comms:
- Github issues and pull requests.
- Slack: `#zig` in bazel.slack.com.
- announcements (I am aiming to send an email with every release).
- user discussions.
- raising issues.
- contributions.
### Previous Commuications
I will generally respond to emails about issues. I may even be able to fix
them. However, no promises: you are much more likely (and welcome!) to get it
fixed by submitting a patch.
Previous communications were done in an email list; the past archive is in
`mailing-list-archive.mbox`. It can be accessed like this:
To contribute, send your patches to the mailing list, as described in
[git-send-email.io][git-send-email] or via [Sourcehut web UI][video].
mutt -R -f mailing-list-archive.mbox
Copyright is retained by the contributors.
## Maintainers
# Maintainers
This section lists the driving forces behind `hermetic_cc_toolchain`.
Committers have write access, maintainers own their areas. Should make it
easier to understand our interests when reading patches or mailing lists.
This section lists the driving forces behind bazel-zig-cc. Committers have push
access, maintainers have their areas. Should make it easier to understand our
interests when reading patches or mailing lists.
- Maintainers: Motiejus Jakštys, Laurynas Lubys, Zhongpeng Lin, Sung Yoon Whang
and Jeremy Volkman.
- Maintainer for Windows: Fabian Hahn.
- Owner: Motiejus Jakštys. Applies others' patches, writes documentation,
emails, and occasionally contributes. Signs releases.
- Committer: Laurynas Lubys. Bazel expert with regards to tests, transitions
and overall structure. Rewrote bazel-zig-cc to cater for platforms when libc
platforms were added.
- Committer: Ken Micklas. Ken is leading hermetic toolchain effort at Uber, of
which bazel-zig-cc is a part of.
- Maintainer for Windows: Fabian Hahn. If you make a change that breaks
Windows, Fabian will find you. Please don't break Windows, so Fabian doesn't
have to look for you. Instead, send him your patches first.
Guidelines for maintainers:
You may find contact information of the individuals in the commit logs.
$ zig zen
* Communicate intent precisely.
* Edge cases matter.
* Favor reading code over writing code.
* Only one obvious way to do things.
* Runtime crashes are better than bugs.
* Compile errors are better than runtime crashes.
* Incremental improvements.
* Avoid local maximums.
* Reduce the amount one must remember.
* Focus on code rather than style.
* Resource allocation may fail; resource deallocation must succeed.
* Memory is a resource.
* Together we serve the users.
# Publicity
On a more practical note:
This section lists notable uses or mentions of bazel-zig-cc.
- 2022-05-23 [How Zig is used at Uber (youtube)][yt-how-zig-is-used-at-uber]:
Yours Truly (the author) talks about how bazel-zig-cc came to existence and
how it's used at Uber in Milan Zig Meetup.
- 2022-05-23 [How Uber uses Zig][how-uber-uses-zig]: text version of the above.
- 2022-03-30 [Google Open Source Peer Bonus Program][google-award] awarded the
author $250 for bazel-zig-cc.
- 2022-01-13 [bazel-zig-cc building Envoy][zig-cc-envoy].
If you'd like your blog post, company or a project added here, do not hesitate
and send a patch.
# Thanks
Many thanks to Adam Bouhenguel and his [bazel-zig-cc][ajbouh], the parent of
this repository. Also, the Zig team for making this all possible and handling
the issues promptly.
- Maintainers can merge others' pull requests following their best judgement.
They may or may not ask for feedback from other maintainers. Follow the Zen
of Zig.
- Releases are cut by Uber employees, because they can test the
version-to-be-released with our [Go Monorepo][go-monorepo]. If you use
`hermetic_cc_toolchain` in any serious capacity, we encourage you to make
yourself known, so we can work together to validate it before cutting the
release.
[^1]: a [mathematical subset][subset]: both can be equal.
[binfmt_misc]: https://en.wikipedia.org/wiki/Binfmt_misc
[mailing-list]: https://lists.sr.ht/~motiejus/bazel-zig-cc
[ajbouh]: https://github.com/ajbouh/bazel-zig-cc/
[git-send-email]: https://git-send-email.io/
[video]: https://spacepub.space/w/no6jnhHeUrt2E5ST168tRL
[sysroot]: https://github.com/ziglang/zig/issues/10299#issuecomment-989153750
[ubsan1]: https://github.com/ziglang/zig/issues/4830#issuecomment-605491606
[ubsan2]: https://github.com/ziglang/zig/issues/5163
[transitions]: https://docs.bazel.build/versions/main/skylark/config.html#user-defined-transitions
[subset]: https://en.wikipedia.org/wiki/Subset
[yt-how-zig-is-used-at-uber]: https://www.youtube.com/watch?v=SCj2J3HcEfc
[how-uber-uses-zig]: https://jakstys.lt/2022/how-uber-uses-zig/
[zig-cc-envoy]: https://github.com/envoyproxy/envoy/issues/19535
[google-award]: https://opensource.googleblog.com/2022/03/Announcing-First-Group-of-Google-Open-Source-Peer-Bonus-Winners-in-2022.html
[go-gc-sections]: https://go-review.googlesource.com/c/go/+/407814
[universal-headers]: https://github.com/ziglang/universal-headers
[go-monorepo]: https://www.uber.com/blog/go-monorepo-bazel/

View File

@ -1,24 +1,24 @@
workspace(
name = "bazel-zig-cc",
name = "hermetic_cc_toolchain",
)
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "io_bazel_rules_go",
sha256 = "16e9fca53ed6bd4ff4ad76facc9b7b651a89db1689a2877d6fd7b82aa824e366",
sha256 = "dd926a88a564a9246713a9c00b35315f54cbd46b31a26d5d8fb264c07045f05d",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.34.0/rules_go-v0.34.0.zip",
"https://github.com/bazelbuild/rules_go/releases/download/v0.34.0/rules_go-v0.34.0.zip",
"https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.38.1/rules_go-v0.38.1.zip",
"https://github.com/bazelbuild/rules_go/releases/download/v0.38.1/rules_go-v0.38.1.zip",
],
)
http_archive(
name = "bazel_gazelle",
sha256 = "5982e5463f171da99e3bdaeff8c0f48283a7a5f396ec5282910b9e8a49c0dd7e",
sha256 = "ecba0f04f96b4960a5b250c8e8eeec42281035970aa8852dda73098274d14a1d",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/bazel-gazelle/releases/download/v0.25.0/bazel-gazelle-v0.25.0.tar.gz",
"https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.25.0/bazel-gazelle-v0.25.0.tar.gz",
"https://mirror.bazel.build/github.com/bazelbuild/bazel-gazelle/releases/download/v0.29.0/bazel-gazelle-v0.29.0.tar.gz",
"https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.29.0/bazel-gazelle-v0.29.0.tar.gz",
],
)
@ -30,7 +30,7 @@ go_rules_dependencies()
# use latest stable.
go_download_sdk(
name = "go_sdk",
version = "1.19",
version = "1.20.3",
)
go_register_toolchains()
@ -68,3 +68,17 @@ register_toolchains(
"@zig_sdk//libc_aware/toolchain:linux_arm64_gnu.2.28",
"@zig_sdk//libc_aware/toolchain:linux_arm64_musl",
)
http_archive(
name = "com_google_protobuf",
sha256 = "d0f5f605d0d656007ce6c8b5a82df3037e1d8fe8b121ed42e536f569dec16113",
strip_prefix = "protobuf-3.14.0",
urls = [
"https://mirror.bazel.build/github.com/protocolbuffers/protobuf/archive/v3.14.0.tar.gz",
"https://github.com/protocolbuffers/protobuf/archive/v3.14.0.tar.gz",
],
)
load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
protobuf_deps()

View File

@ -1,10 +0,0 @@
#!/usr/bin/env bash
set -xeuo pipefail
cd "$(git rev-parse --show-toplevel)/"
bazel run @go_sdk//:bin/go -- mod tidy
exec bazel run //:gazelle -- update-repos \
-from_file=go.mod \
-prune \
-to_macro=repositories.bzl%go_repositories

32
ci/launcher Executable file
View File

@ -0,0 +1,32 @@
#!/usr/bin/env bash
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
set -xeuo pipefail
echo "--- which zig"
ZIG=${ZIG:-$(tools/bazel run "$@" --run_under=echo @zig_sdk//:zig)}
echo "--- compile launcher.zig for various architectures"
for target in \
aarch64-linux-gnu.2.19 \
aarch64-macos-none \
x86_64-linux-gnu.2.19 \
x86_64-macos-none \
x86_64-windows-gnu
do
if [[ $target == aarch64-macos-none ]]; then
mcpu=apple_a14
else
mcpu=baseline
fi
$ZIG build-exe -fno-emit-bin -target $target -mcpu=$mcpu toolchain/launcher.zig
done
echo "--- zig fmt --check toolchain/launcher.zig"
$ZIG fmt --check toolchain/launcher.zig
echo "--- zig test toolchain/launcher.zig"
# until hermetic_cc_toolchain gets a zig toolchain, run launcher's unit tests here.
$ZIG test toolchain/launcher.zig

18
ci/launcher-wine64 Executable file
View File

@ -0,0 +1,18 @@
#!/usr/bin/env bash
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
set -xeuo pipefail
echo "--- which zig"
ZIG=${ZIG:-$(tools/bazel run "$@" --run_under=echo @zig_sdk//:zig)}
echo "--- test toolchain/launcher.zig via wine64"
# ReleaseSafe because of https://github.com/ziglang/zig/issues/14036
$ZIG test \
-OReleaseSafe \
-target x86_64-windows-gnu \
--test-cmd wine64-stable \
--test-cmd-bin \
toolchain/launcher.zig

31
ci/lint
View File

@ -1,21 +1,26 @@
#!/usr/bin/env bash
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
set -euo pipefail
cd "$(git rev-parse --show-toplevel)"
mapfile -t files < \
<(git ls-files)
mapfile -t scripts < \
<(awk '/#!\/usr\/bin\/env/&&FNR==1{print FILENAME}' "${files[@]}")
mapfile -t buildfiles < \
<(find . \( -name 'WORKSPACE' -o -name 'BUILD' -o -name '*.bzl' \))
if command -v shellcheck &> /dev/null; then
mapfile -t files < \
<(git ls-files)
mapfile -t scripts < \
<(awk '/#!(\/usr\/bin\/env bash|\/bin\/sh)/&&FNR==1{print FILENAME}' "${files[@]}")
>&2 echo "shellcheck"
for f in "${scripts[@]}"; do >&2 echo " $f"; done
shellcheck "${scripts[@]}"
>&2 echo -e "OK\n"
fi
>&2 echo "shellcheck"
for f in "${scripts[@]}"; do >&2 echo " $f"; done
shellcheck -x "${scripts[@]}"
>&2 echo "--- buildifier :bazel:"
tools/buildifier -r -mode check "$PWD"
>&2 echo -e "OK\n"
>&2 echo "buildifier"
for f in "${buildfiles[@]}"; do >&2 echo " $f"; done
buildifier "${buildfiles[@]}"
>&2 echo -e "OK\n"
>&2 echo "--- Gazelle :goat:"
tools/bazel run //:gazelle -- -mode diff

View File

@ -1,15 +1,19 @@
#!/usr/bin/env bash
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
set -euo pipefail
indent() { sed 's/^/ /'; }
echo "Available toolchains:"
bazel query --noshow_progress '@zig_sdk//toolchain:*' | indent
echo "Available platforms:"
bazel query --noshow_progress '@zig_sdk//platform:*' | indent
echo "Available libc aware toolchains:"
bazel query --noshow_progress '@zig_sdk//libc_aware/toolchain:*' | indent
echo "Available libc aware platforms:"
bazel query --noshow_progress '@zig_sdk//libc_aware/platform:*' | indent
echo "Available libc variants:"
bazel query --noshow_progress "attr(constraint_setting, @zig_sdk//libc:variant, @zig_sdk//...)" | indent
echo "--- Available toolchains:"
tools/bazel query --noshow_progress '@zig_sdk//toolchain:*' | indent
echo "--- Available platforms:"
tools/bazel query --noshow_progress '@zig_sdk//platform:*' | indent
echo "--- Available libc aware toolchains:"
tools/bazel query --noshow_progress '@zig_sdk//libc_aware/toolchain:*' | indent
echo "--- Available libc aware platforms:"
tools/bazel query --noshow_progress '@zig_sdk//libc_aware/platform:*' | indent
echo "--- Available libc variants:"
tools/bazel query --noshow_progress "attr(constraint_setting, @zig_sdk//libc:variant, @zig_sdk//...)" | indent

View File

@ -1,20 +1,15 @@
#!/usr/bin/env bash
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
set -xeuo pipefail
cd "$(git rev-parse --show-toplevel)"
prev_ref=$(git rev-parse HEAD)
git commit --allow-empty -m "this is a test commit"
./release --nosign v99.0
cleanup() { git tag -d v99.0; git reset --hard "$prev_ref"; }
tools/bazel run //tools/releaser -- -tag v99.0.0
cleanup() { git tag -d v99.0.0; git reset --hard "$prev_ref"; }
trap cleanup EXIT
want=" 1 file changed, 2 insertions(+), 2 deletions(-)"
got=$(git show --shortstat HEAD | tail -1)
if [[ "$want" != "$got" ]]; then
echo wanted:
echo \ \ "$want"
echo got:
echo \ \ "$got"
exit 1
fi
file hermetic_cc_toolchain-v99.0.0.tar.gz | grep -q 'gzip compressed data'

40
ci/test
View File

@ -1,2 +1,38 @@
#!/usr/bin/env sh
exec bazel test "$@" ...
#!/usr/bin/env bash
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
set -xeuo pipefail
cache_prefix="${HERMETIC_CC_TOOLCHAIN_CACHE_PREFIX:-/tmp/zig-cache}"
# check a very hermetic setup with a single target. Re-building all of
# them takes a long time, so using only one. If we ever decide to build all
# targets, we will need to exclude Go, since go dynamically links to glibc on
# linux.
echo "--- build a single target with very hermetic sandbox"
tools/bazel build "$@" \
--experimental_use_hermetic_linux_sandbox \
--sandbox_writable_path="$cache_prefix" \
--sandbox_add_mount_pair=/proc \
//test/c:which_libc
# then test everything else with the standard sandbox
echo "--- bazel test $* ..."
tools/bazel test "$@" ...
echo "--- ensure github.com/ziglang/zig/issues/13050 does not regress"
find "$cache_prefix" \
-name mutex_destructor.o -execdir file '{}' \; \
| sort \
| uniq -c \
| sort -rn > /tmp/got_cache
diff -u ci/testdata/want_cache /tmp/got_cache || {
>&2 echo "ERROR: unexpected artifacts. This is TODO."
# TODO: Go 1.20 regressed this. Find a way to re-enable. See README.
#exit 1
exit 0
}

3
ci/testdata/want_cache vendored Normal file
View File

@ -0,0 +1,3 @@
5 ./mutex_destructor.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
3 ./mutex_destructor.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), not stripped
1 ./mutex_destructor.o: Mach-O 64-bit x86_64 object, flags:<|SUBSECTIONS_VIA_SYMBOLS>

View File

@ -1,28 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
zigdir=out/zig-x86_64-linux-musl-x86_64_v3
if [[ ! "$PWD" =~ /zig-bootstrap$ ]]; then
>&2 echo "expected to be in zig-bootstrap directory. Bailing"
exit 1
fi
if [[ ! -f "$zigdir/bin/zig" ]]; then
>&2 echo "$zigdir/bin/zig not found. Please run:"
>&2 echo " ./build -j\$(nproc) x86_64-linux-musl x86_64_v3"
exit 1
fi
pushd "$zigdir"
vsn=$(bin/zig version)
outdir="zig-linux-x86_64-$vsn"
mkdir -p "$outdir"
cp -r "bin/zig" "$outdir"
cp -r "lib/zig" "$outdir/lib"
tar -cf "$outdir.tar" "$outdir"
xz -vk -9 -T0 "$outdir.tar"
popd
echo "$zigdir/$outdir.tar.xz is ready for use"

View File

@ -1,39 +0,0 @@
How to test a different version of zig
--------------------------------------
Assume you want to test an unreleased version of zig. Here's how:
1. Clone zig-bootstrap:
$ git clone https://github.com/ziglang/zig-bootstrap
$ cd zig-bootstrap
2. Copy over zig/ from ~/zig:
$ rm -fr zig
$ tar -C ~/zig archive --format=tar --prefix=zig/ master | tar -xv
3. Build it (assuming `x86_64-linux`):
$ vim build # edit ZIG_VERSION
$ ./build -j$(nproc) x86_64-linux-musl baseline
4. Pack the release tarball:
$ ~/code/bazel-zig-cc/makerel
This gives us a usable Zig SDK. Now:
- Send the .tar.gz it to your mirror.
- Point toolchain/defs.bzl to the new version.
- Run tests.
Links
-----
- [ziglang/release-cutter][1], a script that creates binaries for [ziglang.org/download][2].
- [ziglang/zig-bootstrap][3], a set of scripts that compile a static Zig.
[1]: https://github.com/ziglang/release-cutter/blob/master/script
[2]: https://ziglang.org/download
[3]: https://github.com/ziglang/zig-bootstrap

11
go.mod
View File

@ -1,3 +1,10 @@
module git.sr.ht/~motiejus/bazel-zig-cc
module github.com/uber/hermetic_cc_toolchain
go 1.18
go 1.19
require github.com/bazelbuild/buildtools v0.0.0-20230302165817-a6ca93fd072d
require (
github.com/golang/protobuf v1.5.2 // indirect
google.golang.org/protobuf v1.28.1 // indirect
)

82
go.sum
View File

@ -0,0 +1,82 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/bazelbuild/buildtools v0.0.0-20230302165817-a6ca93fd072d h1:AfqOWCQPEd/qAi41zI5ovrK0d5oQbXdX+l5VonjZq7A=
github.com/bazelbuild/buildtools v0.0.0-20230302165817-a6ca93fd072d/go.mod h1:689QdV3hBP7Vo9dJMmzhoYIyo/9iMhEmHkJcnaPRCbo=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.5 h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
go.starlark.net v0.0.0-20210223155950-e043a3d3c984/go.mod h1:t3mmBBPzAVvK0L0n1drDmrQsJ8FoIx4INCqVMTr/Zo0=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
google.golang.org/protobuf v1.28.1 h1:d0NfwRgPtno5B1Wa6L2DAG+KivqkdutMf1UhdNx175w=
google.golang.org/protobuf v1.28.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=

16593
mailing-list-archive.mbox Normal file

File diff suppressed because it is too large Load Diff

34
release
View File

@ -1,34 +0,0 @@
#!/usr/bin/env bash
set -xeuo pipefail
sign=(-u motiejus@jakstys.lt)
[[ $1 == "--nosign" ]] && { sign=(); shift; }
_err(){ >&2 echo "ERROR: $*"; exit 1; }
git status --porcelain | grep -q "" &&
_err "working tree is dirty, commit your changes first."
[[ "$1" =~ ^v([0-9]+)\.([0-9]+)(\.([0-9]+))?(-rc([0-9]+))?$ ]] || \
_err "arg1 accepts the following formats: v1.0 v1.0.0 v1.0-rc1 v1.0.1-rc1"
git tag | grep -q "^$1$" &&
_err "tag $1 already exists"
last_tag=$(git tag | tail -1)
{
echo bazel-zig-cc "$1"
echo
echo Changelog since "$last_tag":
git log --pretty=format:"- [%an] %s" "$last_tag"..HEAD | \
grep -v "Update release notes for $last_tag"
} | git tag "${sign[@]}" -F - "$1"
shasum=$(git archive --prefix="bazel-zig-cc-$1/" --format=tar "$1" | \
gzip -n | sha256sum | cut -f1 -d" ")
./relnotes.awk -v tag="$1" -v sha256sum="$shasum" README.md | sponge README.md
git add README.md
git commit -m "Update release notes for $1"

View File

@ -1,34 +0,0 @@
#!/usr/bin/awk -f
BEGIN {stage=0};
!/```/ && stage==0 {
print
}
/```/ && stage==0 {
print "```"
print "BAZEL_ZIG_CC_VERSION = \""tag"\""
print ""
print "http_archive("
print " name = \"bazel-zig-cc\","
print " sha256 = \""sha256sum"\","
print " strip_prefix = \"bazel-zig-cc-{}\".format(BAZEL_ZIG_CC_VERSION),"
print " urls = [\"https://git.sr.ht/~motiejus/bazel-zig-cc/archive/{}.tar.gz\".format(BAZEL_ZIG_CC_VERSION)],"
print ")"
stage=1
next
}
!/^)$/ && stage==1 {
next
};
/^)$/ && stage==1 {
stage=2
next
};
stage==2 {
print;
};

View File

@ -1,2 +1,195 @@
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
load("@bazel_gazelle//:deps.bzl", "go_repository")
def go_repositories():
pass
go_repository(
name = "co_honnef_go_tools",
importpath = "honnef.co/go/tools",
sum = "h1:/hemPrYIhOhy8zYrNj+069zDB68us2sMGsfkFJO0iZs=",
version = "v0.0.0-20190523083050-ea95bdfd59fc",
)
go_repository(
name = "com_github_bazelbuild_buildtools",
importpath = "github.com/bazelbuild/buildtools",
sum = "h1:AfqOWCQPEd/qAi41zI5ovrK0d5oQbXdX+l5VonjZq7A=",
version = "v0.0.0-20230302165817-a6ca93fd072d",
)
go_repository(
name = "com_github_burntsushi_toml",
importpath = "github.com/BurntSushi/toml",
sum = "h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=",
version = "v0.3.1",
)
go_repository(
name = "com_github_census_instrumentation_opencensus_proto",
importpath = "github.com/census-instrumentation/opencensus-proto",
sum = "h1:glEXhBS5PSLLv4IXzLA5yPRVX4bilULVyxxbrfOtDAk=",
version = "v0.2.1",
)
go_repository(
name = "com_github_chzyer_logex",
importpath = "github.com/chzyer/logex",
sum = "h1:Swpa1K6QvQznwJRcfTfQJmTE72DqScAa40E+fbHEXEE=",
version = "v1.1.10",
)
go_repository(
name = "com_github_chzyer_readline",
importpath = "github.com/chzyer/readline",
sum = "h1:fY5BOSpyZCqRo5OhCuC+XN+r/bBCmeuuJtjz+bCNIf8=",
version = "v0.0.0-20180603132655-2972be24d48e",
)
go_repository(
name = "com_github_chzyer_test",
importpath = "github.com/chzyer/test",
sum = "h1:q763qf9huN11kDQavWsoZXJNW3xEE4JJyHa5Q25/sd8=",
version = "v0.0.0-20180213035817-a1ea475d72b1",
)
go_repository(
name = "com_github_client9_misspell",
importpath = "github.com/client9/misspell",
sum = "h1:ta993UF76GwbvJcIo3Y68y/M3WxlpEHPWIGDkJYwzJI=",
version = "v0.3.4",
)
go_repository(
name = "com_github_envoyproxy_go_control_plane",
importpath = "github.com/envoyproxy/go-control-plane",
sum = "h1:4cmBvAEBNJaGARUEs3/suWRyfyBfhf7I60WBZq+bv2w=",
version = "v0.9.1-0.20191026205805-5f8ba28d4473",
)
go_repository(
name = "com_github_envoyproxy_protoc_gen_validate",
importpath = "github.com/envoyproxy/protoc-gen-validate",
sum = "h1:EQciDnbrYxy13PgWoY8AqoxGiPrpgBZ1R8UNe3ddc+A=",
version = "v0.1.0",
)
go_repository(
name = "com_github_golang_glog",
importpath = "github.com/golang/glog",
sum = "h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58=",
version = "v0.0.0-20160126235308-23def4e6c14b",
)
go_repository(
name = "com_github_golang_mock",
importpath = "github.com/golang/mock",
sum = "h1:G5FRp8JnTd7RQH5kemVNlMeyXQAztQ3mOWV95KxsXH8=",
version = "v1.1.1",
)
go_repository(
name = "com_github_golang_protobuf",
importpath = "github.com/golang/protobuf",
sum = "h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=",
version = "v1.5.2",
)
go_repository(
name = "com_github_google_go_cmp",
importpath = "github.com/google/go-cmp",
sum = "h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU=",
version = "v0.5.5",
)
go_repository(
name = "com_github_prometheus_client_model",
importpath = "github.com/prometheus/client_model",
sum = "h1:gQz4mCbXsO+nc9n1hCxHcGA3Zx3Eo+UHZoInFGUIXNM=",
version = "v0.0.0-20190812154241-14fe0d1b01d4",
)
go_repository(
name = "com_google_cloud_go",
importpath = "cloud.google.com/go",
sum = "h1:e0WKqKTd5BnrG8aKH3J3h+QvEIQtSUcf2n5UZ5ZgLtQ=",
version = "v0.26.0",
)
go_repository(
name = "net_starlark_go",
importpath = "go.starlark.net",
sum = "h1:xwwDQW5We85NaTk2APgoN9202w/l0DVGp+GZMfsrh7s=",
version = "v0.0.0-20210223155950-e043a3d3c984",
)
go_repository(
name = "org_golang_google_appengine",
importpath = "google.golang.org/appengine",
sum = "h1:/wp5JvzpHIxhs/dumFmF7BXTf3Z+dd4uXta4kVyO508=",
version = "v1.4.0",
)
go_repository(
name = "org_golang_google_genproto",
importpath = "google.golang.org/genproto",
sum = "h1:+kGHl1aib/qcwaRi1CbqBZ1rk19r85MNUf8HaBghugY=",
version = "v0.0.0-20200526211855-cb27e3aa2013",
)
go_repository(
name = "org_golang_google_grpc",
importpath = "google.golang.org/grpc",
sum = "h1:rRYRFMVgRv6E0D70Skyfsr28tDXIuuPZyWGMPdMcnXg=",
version = "v1.27.0",
)
go_repository(
name = "org_golang_google_protobuf",
importpath = "google.golang.org/protobuf",
sum = "h1:d0NfwRgPtno5B1Wa6L2DAG+KivqkdutMf1UhdNx175w=",
version = "v1.28.1",
)
go_repository(
name = "org_golang_x_crypto",
importpath = "golang.org/x/crypto",
sum = "h1:VklqNMn3ovrHsnt90PveolxSbWFaJdECFbxSq0Mqo2M=",
version = "v0.0.0-20190308221718-c2843e01d9a2",
)
go_repository(
name = "org_golang_x_exp",
importpath = "golang.org/x/exp",
sum = "h1:c2HOrn5iMezYjSlGPncknSEr/8x5LELb/ilJbXi9DEA=",
version = "v0.0.0-20190121172915-509febef88a4",
)
go_repository(
name = "org_golang_x_lint",
importpath = "golang.org/x/lint",
sum = "h1:XQyxROzUlZH+WIQwySDgnISgOivlhjIEwaQaJEJrrN0=",
version = "v0.0.0-20190313153728-d0100b6bd8b3",
)
go_repository(
name = "org_golang_x_net",
importpath = "golang.org/x/net",
sum = "h1:oWX7TPOiFAMXLq8o0ikBYfCJVlRHBcsciT5bXOrH628=",
version = "v0.0.0-20190311183353-d8887717615a",
)
go_repository(
name = "org_golang_x_oauth2",
importpath = "golang.org/x/oauth2",
sum = "h1:vEDujvNQGv4jgYKudGeI/+DAX4Jffq6hpD55MmoEvKs=",
version = "v0.0.0-20180821212333-d2e6202438be",
)
go_repository(
name = "org_golang_x_sync",
importpath = "golang.org/x/sync",
sum = "h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU=",
version = "v0.0.0-20190423024810-112230192c58",
)
go_repository(
name = "org_golang_x_sys",
importpath = "golang.org/x/sys",
sum = "h1:+Nyd8tzPX9R7BWHguqsrbFdRx3WQ/1ib8I44HXV5yTA=",
version = "v0.0.0-20200930185726-fdedc70b468f",
)
go_repository(
name = "org_golang_x_text",
importpath = "golang.org/x/text",
sum = "h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=",
version = "v0.3.0",
)
go_repository(
name = "org_golang_x_tools",
importpath = "golang.org/x/tools",
sum = "h1:5Beo0mZN8dRzgrMMkDp0jc8YXQKx9DiJ2k1dkvGsn5A=",
version = "v0.0.0-20190524140312-2c0ae7006135",
)
go_repository(
name = "org_golang_x_xerrors",
importpath = "golang.org/x/xerrors",
sum = "h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=",
version = "v0.0.0-20200804184101-5ec99f83aff1",
)

View File

@ -1 +1,4 @@
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
package(default_visibility = ["//test:__pkg__"])

View File

@ -1,9 +1,5 @@
def _vars_script(env, run_under, cmd):
ret = ["#!/bin/sh"]
for k, v in env.items():
ret += ['export {}="{}"'.format(k, v)]
ret += ['exec {} {} "$@"'.format(run_under, cmd)]
return "\n".join(ret) + "\n" # trailing newline is easier on the eyes
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
def _platform_transition_impl(settings, attr):
_ignore = settings
@ -20,22 +16,22 @@ _platform_transition = transition(
)
def _platform_binary_impl(ctx):
source_info = ctx.attr.src[DefaultInfo]
executable = None
if source_info.files_to_run and source_info.files_to_run.executable:
command = _vars_script(ctx.attr.env, ctx.attr.run_under, source_info.files_to_run.executable.short_path)
executable = ctx.actions.declare_file("{}_{}".format(ctx.file.src.basename, ctx.attr.platform))
ctx.actions.write(
output = executable,
content = command,
is_executable = True,
)
platform_sanitized = ctx.attr.platform.replace("/", "_").replace(":", "_")
dstname = "{}-{}".format(
_paths_basename(ctx.file.src.path),
platform_sanitized,
)
dst = ctx.actions.declare_file(dstname)
src = ctx.file.src
ctx.actions.run(
outputs = [dst],
inputs = [src],
executable = "cp",
arguments = [src.path, dst.path],
)
return [DefaultInfo(
executable = executable,
files = depset([executable]),
runfiles = ctx.runfiles(files = ctx.files.src),
files = depset([dst]),
executable = dst,
)]
_attrs = {
@ -47,12 +43,6 @@ _attrs = {
"platform": attr.string(
doc = "The platform to build the target for.",
),
"run_under": attr.string(
doc = "wrapper executable",
),
"env": attr.string_dict(
doc = "Environment variables for the test",
),
"_allowlist_function_transition": attr.label(
default = "@bazel_tools//tools/allowlists/function_transition_allowlist",
),
@ -73,3 +63,7 @@ platform_test = rule(
attrs = _attrs,
test = True,
)
## Copied from https://github.com/bazelbuild/bazel-skylib/blob/1.4.1/lib/paths.bzl#L22
def _paths_basename(p):
return p.rpartition("/")[-1]

View File

@ -1,21 +0,0 @@
load("@io_bazel_rules_go//go:def.bzl", go_binary_rule = "go_binary")
"""
go_binary overrides go_binary from rules_go and provides default
gc_linkopts values that are needed to compile for macos target.
To use it, add this map_kind gazelle directive to your BUILD.bazel files
where target binary needs to be compiled with zig toolchain.
Example: if this toolchain is registered as bazel-zig-cc in your WORKSPACE, add this to
your root BUILD file
# gazelle:map_kind go_binary go_binary @bazel-zig-cc//rules:rules_go.bzl
"""
_MACOS_GC_LINKOPTS = ["-s", "-w", "-buildmode=pie"]
def go_binary(**kwargs):
kwargs["gc_linkopts"] = select({
"@platforms//os:macos": _MACOS_GC_LINKOPTS,
"//conditions:default": [],
}) + kwargs.pop("gc_linkopts", [])
go_binary_rule(**kwargs)

View File

@ -1,4 +1,4 @@
load("@bazel-zig-cc//rules:platform.bzl", "platform_binary")
load("@hermetic_cc_toolchain//rules:platform.bzl", "platform_binary")
cc_binary(
name = "which_libc",
@ -13,9 +13,7 @@ cc_binary(
platform_binary(
name = "which_libc_{}".format(name),
src = "which_libc",
env = {"QEMU_LD_PREFIX": "/usr/aarch64-linux-gnu"} if is_arm64 else {},
platform = platform,
run_under = "qemu-aarch64-static" if is_arm64 else "",
),
sh_test(
name = "test_libc_{}".format(name),
@ -27,12 +25,10 @@ cc_binary(
},
),
)
for name, platform, want, is_arm64 in [
("linux_amd64_musl", "//libc_aware/platform:linux_amd64_musl", "non-glibc", False),
("linux_amd64_gnu.2.19", "//libc_aware/platform:linux_amd64_gnu.2.19", "glibc_2.19", False),
("linux_amd64_gnu.2.28", "//libc_aware/platform:linux_amd64_gnu.2.28", "glibc_2.28", False),
("linux_amd64_gnu.2.31", "//libc_aware/platform:linux_amd64_gnu.2.31", "glibc_2.31", False),
("linux_amd64", "//platform:linux_amd64", "glibc_2.19", False),
("linux_arm64", "//platform:linux_arm64", "glibc_2.28", True),
for name, platform, want in [
("linux_amd64_musl", "//libc_aware/platform:linux_amd64_musl", "non-glibc"),
("linux_amd64_gnu.2.19", "//libc_aware/platform:linux_amd64_gnu.2.19", "glibc_2.19"),
("linux_amd64_gnu.2.28", "//libc_aware/platform:linux_amd64_gnu.2.28", "glibc_2.28"),
("linux_amd64", "//platform:linux_amd64", "glibc_2.19"),
]
]

View File

@ -1,3 +1,6 @@
// Copyright 2023 Uber Technologies, Inc.
// Licensed under the MIT License
#include <stdio.h>
#include <features.h>

View File

@ -1,6 +1,9 @@
#!/usr/bin/env bash
#!/bin/sh
set -euo pipefail
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
set -eu
# shellcheck disable=SC2153
want=$WANT
@ -8,7 +11,7 @@ want=$WANT
binary=$BINARY
got=$($binary)
if [[ "$got" != "$want" ]]; then
if [ "$got" != "$want" ]; then
echo wanted:
echo \ \ "$want"
echo got:

View File

@ -1,12 +1,14 @@
load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")
load("//rules:rules_go.bzl", "go_binary")
load("@bazel-zig-cc//rules:platform.bzl", "platform_binary", "platform_test")
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library", "go_test")
load("@hermetic_cc_toolchain//rules:platform.bzl", "platform_binary", "platform_test")
go_library(
name = "cgo_lib",
srcs = ["cgo.go"],
cgo = True,
importpath = "git.sr.ht/~motiejus/bazel-zig-cc/test/cgo",
importpath = "github.com/uber/hermetic_cc_toolchain/test/cgo",
visibility = ["//visibility:private"],
)
@ -19,6 +21,13 @@ go_test(
go_binary(
name = "cgo",
embed = [":cgo_lib"],
gc_linkopts = select({
"@platforms//os:macos": [
"-w", # https://github.com/ziglang/zig/issues/15439
"-buildmode=pie", # https://github.com/ziglang/zig/issues/15438
],
"//conditions:default": [],
}),
visibility = ["//visibility:public"],
)
@ -41,14 +50,10 @@ go_binary(
platform_test(
name = "cgo_test_{}".format(name),
src = "cgo_test",
env = {"QEMU_LD_PREFIX": "/usr/aarch64-linux-gnu"} if is_arm64 else {},
platform = platform,
run_under = "qemu-aarch64-static" if is_arm64 else "",
)
for name, platform, is_arm64 in [
("linux_amd64_musl", "//libc_aware/platform:linux_amd64_musl", False),
("linux_amd64_gnu.2.19", "//libc_aware/platform:linux_amd64_gnu.2.19", False),
("linux_arm64_musl", "//libc_aware/platform:linux_arm64_musl", True),
("linux_arm64_gnu.2.28", "//libc_aware/platform:linux_arm64_gnu.2.28", True),
]
]

View File

@ -1,11 +1,11 @@
// Copyright 2023 Uber Technologies, Inc.
// Licensed under the MIT License
package main
// #define _FILE_OFFSET_BITS 64
// #include <unistd.h>
// #include <fcntl.h>
// #include <stdio.h>
// char* hello() { return "hello, world"; }
// void phello() { printf("%s, your lucky number is %p\n", hello(), fcntl); }
// void phello() { printf("%s\n", hello()); }
import "C"
func main() {

View File

@ -1,3 +1,6 @@
// Copyright 2023 Uber Technologies, Inc.
// Licensed under the MIT License
package main
import (

25
test/glibc_hacks/BUILD Normal file
View File

@ -0,0 +1,25 @@
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
load("@hermetic_cc_toolchain//rules:platform.bzl", "platform_binary")
cc_binary(
name = "main",
srcs = ["main.c"],
)
[
(
platform_binary(
name = "main_{}".format(name),
src = "main",
platform = platform,
),
)
for name, platform in [
("linux_amd64_musl", "//libc_aware/platform:linux_amd64_musl"),
("linux_amd64_gnu.2.19", "//libc_aware/platform:linux_amd64_gnu.2.19"),
("linux_amd64_gnu.2.28", "//libc_aware/platform:linux_amd64_gnu.2.28"),
("linux_arm64_musl", "//libc_aware/platform:linux_arm64_musl"),
]
]

16
test/glibc_hacks/main.c Normal file
View File

@ -0,0 +1,16 @@
// Copyright 2023 Uber Technologies, Inc.
// Licensed under the MIT License
// This file tests that problematic functions (glibc-hacks) work.
// Also see https://github.com/ziglang/zig/issues/9485
#define _FILE_OFFSET_BITS 64
#include <unistd.h>
#include <fcntl.h>
#include <resolv.h>
#include <stdio.h>
int main() {
printf("Your lucky numbers are %p and %p\n", fcntl, res_search);
return 0;
}

View File

@ -1,12 +1,14 @@
load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")
load("//rules:rules_go.bzl", "go_binary")
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library", "go_test")
go_library(
name = "gorace_lib",
srcs = ["main.go"],
# keep
cgo = True,
importpath = "git.sr.ht/~motiejus/bazel-zig-cc/test/gorace",
importpath = "github.com/uber/hermetic_cc_toolchain/test/gorace",
visibility = ["//visibility:private"],
)

View File

@ -1,3 +1,6 @@
// Copyright 2023 Uber Technologies, Inc.
// Licensed under the MIT License
//
// Package main tests that Zig can compile race-enabled tests.
//
// As of writing, this fails:
@ -6,7 +9,7 @@
// More context: https://github.com/ziglang/zig/issues/11398
//
// This fails, because `zig cc` adds `--gc-sections` to the linker
// flag by default, which is incompatible with cgo. bazel-zig-cc
// flag by default, which is incompatible with cgo. hermetic_cc_toolchain
// adds a workaround for it.
package main

View File

@ -1,3 +1,6 @@
// Copyright 2023 Uber Technologies, Inc.
// Licensed under the MIT License
package main
import "testing"

View File

@ -1,4 +1,7 @@
load("@bazel-zig-cc//rules:platform.bzl", "platform_binary", "platform_test")
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
load("@hermetic_cc_toolchain//rules:platform.bzl", "platform_binary", "platform_test")
cc_binary(
name = "winver",
@ -6,12 +9,10 @@ cc_binary(
tags = ["manual"],
)
platform_test(
platform_binary(
name = "winver_windows_amd64",
src = "winver",
platform = "//platform:windows_amd64",
run_under = "wine64-stable",
tags = ["no-sandbox"],
)
platform_binary(

View File

@ -1,3 +1,6 @@
// Copyright 2023 Uber Technologies, Inc.
// Licensed under the MIT License
#include <stdio.h>
#include <windows.h>

View File

@ -1,17 +1,18 @@
load("@bazel-zig-cc//toolchain:defs.bzl", "declare_files")
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
load("@hermetic_cc_toolchain//toolchain:defs.bzl", "declare_files")
load("@hermetic_cc_toolchain//toolchain/private:cc_toolchains.bzl", "declare_cc_toolchains")
package(
default_visibility = ["//visibility:public"],
)
declare_files(
os = {os},
zig_include_root = {zig_include_root},
)
exports_files([
"glibc-hacks/fcntl.map",
"glibc-hacks/glibchack-fcntl.h",
])
declare_cc_toolchains(
os = {os},
zig_sdk_path = {zig_sdk_path},
)

View File

@ -1,45 +1,33 @@
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
load("@bazel_tools//tools/build_defs/repo:utils.bzl", "read_user_netrc", "use_netrc")
load("@bazel-zig-cc//toolchain/private:defs.bzl", "DEFAULT_INCLUDE_DIRECTORIES", "target_structs", "zig_tool_path")
load("@hermetic_cc_toolchain//toolchain/private:defs.bzl", "target_structs", "zig_tool_path")
_fcntl_map = """
GLIBC_2.2.5 {
fcntl;
};
"""
_fcntl_h = """
#ifdef __ASSEMBLER__
.symver fcntl64, fcntl@GLIBC_2.2.5
#else
__asm__(".symver fcntl64, fcntl@GLIBC_2.2.5");
#endif
"""
# Directories that `zig c++` includes behind the scenes.
_DEFAULT_INCLUDE_DIRECTORIES = [
"libcxx/include",
"libcxxabi/include",
"libunwind/include",
]
# Official recommended version. Should use this when we have a usable release.
URL_FORMAT_RELEASE = "https://ziglang.org/download/{version}/zig-{host_platform}-{version}.{_ext}"
# Caution: nightly releases are purged from ziglang.org after ~90 days. A real
# solution would be to allow the downstream project specify their own mirrors.
# This is explained in
# https://sr.ht/~motiejus/bazel-zig-cc/#alternative-download-urls and is
# awaiting my attention or your contribution.
# Caution: nightly releases are purged from ziglang.org after ~90 days. Use the
# Bazel mirror or your own.
URL_FORMAT_NIGHTLY = "https://ziglang.org/builds/zig-{host_platform}-{version}.{_ext}"
# Author's mirror that doesn't purge the nightlies so aggressively. I will be
# cleaning those up manually only after the artifacts are not in use for many
# months in bazel-zig-cc. dl.jakstys.lt is a small x86_64 server with an NVMe
# drive sitting in my home closet on a 1GB/s symmetric residential connection,
# which, as of writing, has been quite reliable.
URL_FORMAT_JAKSTYS = "https://dl.jakstys.lt/zig/zig-{host_platform}-{version}.{_ext}"
# Official Bazel's mirror with selected Zig SDK versions. Bazel community is
# generous enough to host the artifacts, which we use.
URL_FORMAT_BAZELMIRROR = "https://mirror.bazel.build/" + URL_FORMAT_NIGHTLY.lstrip("https://")
_VERSION = "0.10.0-dev.3529+44a6172ed"
_VERSION = "0.11.0-dev.2619+bd3e248c7"
_HOST_PLATFORM_SHA256 = {
"linux-aarch64": "7fc31ebc02de51091ba9d0176b0abb3a374f6f45cf111f7924becc67943bd854",
"linux-x86_64": "11753bbb58acb5bbc133b678f9b8b8edf2c3603f1c5cd493afb3026f2ccd81f9",
"macos-aarch64": "aa138ec924106e0cdad49cf4710efb91ff9ccef187a8b524e2747795aaff8f71",
"macos-x86_64": "f098c41ab617718564f61fdd31f09066cd8fc1f1612f44c6b3c941a1faa6306f",
"windows-x86_64": "d6a919c78d1856cbac3c16e65f68f25cfecb5deaafcc3c107229b60fca076ae5",
"linux-aarch64": "e72aedc7b9ecf20164dc5aae952499f03402383ca9fb84e72bbb8598f45f693f",
"linux-x86_64": "019dbe76a9035ae4c775483ccb860a740759e22c8bbebd0234f8eaa2a7458cd7",
"macos-aarch64": "3a62f5a535c532978c6a7d248a1f141004de0812fa4b432d50f5dcc9e29e2a55",
"macos-x86_64": "eb4e409cc84991dc0ea8e3e550edb2254d0b15be5f59f2cff4bdc406a9b1ec46",
"windows-x86_64": "01004e422e7e7d48f1df403e368422ce7150428f827f3d579cd44e28f53dba2c",
}
_HOST_PLATFORM_EXT = {
@ -50,9 +38,35 @@ _HOST_PLATFORM_EXT = {
"windows-x86_64": "zip",
}
_MCPU = {
"linux-aarch64": "baseline",
"linux-x86_64": "baseline",
"macos-aarch64": "apple_a14",
"macos-x86_64": "baseline",
"windows-x86_64": "baseline",
}
_compile_failed = """
Compilation of launcher.zig failed:
command={compile_cmd}
return_code={return_code}
stderr={stderr}
stdout={stdout}
You stumbled into a problem with Zig SDK that bazel-zig-cc was not able to fix.
Please file a new issue to github.com/uber/bazel-zig-cc with:
- Full output of this Bazel run, including the Bazel command.
- Version of the Zig SDK if you have a non-default.
- Version of bazel-zig-cc.
Note: this *may* have been https://github.com/ziglang/zig/issues/14815, for
which bazel-zig-cc has a workaround and you may have been "struck by lightning"
three times in a row.
"""
def toolchains(
version = _VERSION,
url_formats = [URL_FORMAT_NIGHTLY, URL_FORMAT_JAKSTYS],
url_formats = [URL_FORMAT_BAZELMIRROR, URL_FORMAT_NIGHTLY],
host_platform_sha256 = _HOST_PLATFORM_SHA256,
host_platform_ext = _HOST_PLATFORM_EXT):
"""
@ -67,52 +81,11 @@ def toolchains(
url_formats = url_formats,
host_platform_sha256 = host_platform_sha256,
host_platform_ext = host_platform_ext,
host_platform_include_root = {
"linux-aarch64": "lib/zig/",
"linux-x86_64": "lib/",
"macos-aarch64": "lib/zig/",
"macos-x86_64": "lib/zig/",
"windows-x86_64": "lib/",
},
)
ZIG_TOOL_WRAPPER_CACHE_KNOWN = """#!/usr/bin/env sh
_cache_prefix="{cache_prefix}"
export ZIG_LOCAL_CACHE_DIR="$_cache_prefix/bazel-zig-cc"
export ZIG_GLOBAL_CACHE_DIR=$ZIG_LOCAL_CACHE_DIR
exec "{zig}" "{zig_tool}" "$@"
"""
ZIG_TOOL_WRAPPER_CACHE_GUESS = """#!/usr/bin/env sh
set -e
if [ -n "$TMPDIR" ]; then
_cache_prefix=$TMPDIR
elif [ -n "$HOME" ]; then
if [ "$(uname)" = Darwin ]; then
_cache_prefix="$HOME/Library/Caches"
else
_cache_prefix="$HOME/.cache"
fi
else
_cache_prefix=/tmp
fi
export ZIG_LOCAL_CACHE_DIR="$_cache_prefix/bazel-zig-cc"
export ZIG_GLOBAL_CACHE_DIR=$ZIG_LOCAL_CACHE_DIR
exec "{zig}" "{zig_tool}" "$@"
"""
ZIG_TOOL_WRAPPER_WINDOWS = """@echo off
"{zig}" "{zig_tool}" %*
"""
_ZIG_TOOLS = [
"c++",
"cc",
"ar",
"ld.lld", # ELF
"ld64.lld", # Mach-O
"lld-link", # COFF
"wasm-ld", # WebAssembly
]
def _quote(s):
@ -132,7 +105,6 @@ def _zig_repository_impl(repository_ctx):
host_platform = "{}-{}".format(os, arch)
zig_include_root = repository_ctx.attr.host_platform_include_root[host_platform]
zig_sha256 = repository_ctx.attr.host_platform_sha256[host_platform]
zig_ext = repository_ctx.attr.host_platform_ext[host_platform]
format_vars = {
@ -141,47 +113,11 @@ def _zig_repository_impl(repository_ctx):
"host_platform": host_platform,
}
urls = [uf.format(**format_vars) for uf in repository_ctx.attr.url_formats]
repository_ctx.download_and_extract(
auth = use_netrc(read_user_netrc(repository_ctx), urls, {}),
url = urls,
stripPrefix = "zig-{host_platform}-{version}/".format(**format_vars),
sha256 = zig_sha256,
)
for zig_tool in _ZIG_TOOLS:
cache_prefix = repository_ctx.os.environ.get("BAZEL_ZIG_CC_CACHE_PREFIX", "")
if os == "windows":
zig_tool_wrapper = ZIG_TOOL_WRAPPER_WINDOWS.format(
zig = str(repository_ctx.path("zig")).replace("/", "\\") + ".exe",
zig_tool = zig_tool,
)
elif cache_prefix:
zig_tool_wrapper = ZIG_TOOL_WRAPPER_CACHE_KNOWN.format(
zig = str(repository_ctx.path("zig")),
zig_tool = zig_tool,
cache_prefix = cache_prefix,
)
else:
zig_tool_wrapper = ZIG_TOOL_WRAPPER_CACHE_GUESS.format(
zig = str(repository_ctx.path("zig")),
zig_tool = zig_tool,
)
repository_ctx.file(
zig_tool_path(os).format(zig_tool = zig_tool),
zig_tool_wrapper,
)
repository_ctx.file(
"glibc-hacks/fcntl.map",
content = _fcntl_map,
)
repository_ctx.file(
"glibc-hacks/glibchack-fcntl.h",
content = _fcntl_h,
)
# Fetch Label dependencies before doing download/extract.
# The Bazel docs are not very clear about this behavior but see:
# https://bazel.build/extending/repo#when_is_the_implementation_function_executed
# and a related rules_go PR:
# https://github.com/bazelbuild/bazel-gazelle/pull/1206
for dest, src in {
"platform/BUILD": "//toolchain/platform:BUILD",
"toolchain/BUILD": "//toolchain/toolchain:BUILD",
@ -193,27 +129,116 @@ def _zig_repository_impl(repository_ctx):
for dest, src in {
"BUILD": "//toolchain:BUILD.sdk.bazel",
"private/BUILD": "//toolchain/private:BUILD.sdk.bazel",
# "private/BUILD": "//toolchain/private:BUILD.sdk.bazel",
}.items():
repository_ctx.template(
dest,
Label(src),
executable = False,
substitutions = {
"{absolute_path}": _quote(str(repository_ctx.path(""))),
"{zig_sdk_path}": _quote("external/zig_sdk"),
"{os}": _quote(os),
"{zig_include_root}": _quote(zig_include_root),
},
)
urls = [uf.format(**format_vars) for uf in repository_ctx.attr.url_formats]
repository_ctx.download_and_extract(
auth = use_netrc(read_user_netrc(repository_ctx), urls, {}),
url = urls,
stripPrefix = "zig-{host_platform}-{version}/".format(**format_vars),
sha256 = zig_sha256,
)
cache_prefix = repository_ctx.os.environ.get("HERMETIC_CC_TOOLCHAIN_CACHE_PREFIX", "")
if cache_prefix == "":
if os == "windows":
cache_prefix = "C:\\\\Temp\\\\hermetic_cc_toolchain"
else:
cache_prefix = "/tmp/zig-cache"
repository_ctx.template(
"tools/launcher.zig",
Label("//toolchain:launcher.zig"),
executable = False,
substitutions = {
"{HERMETIC_CC_TOOLCHAIN_CACHE_PREFIX}": cache_prefix,
},
)
compile_env = {
"ZIG_LOCAL_CACHE_DIR": cache_prefix,
"ZIG_GLOBAL_CACHE_DIR": cache_prefix,
}
compile_cmd = [
_paths_join("..", "zig"),
"build-exe",
"-mcpu={}".format(_MCPU[host_platform]),
"-OReleaseSafe",
"launcher.zig",
] + (["-static"] if os == "linux" else [])
# The elaborate code below is a workaround for ziglang/zig#14815:
# Sometimes, when Zig's cache is empty, compiling the launcher may fail
# with `error: FileNotFound`. The remedy is to clear the cache and try
# again. Until this change, we have been asking users to clear the Zig
# cache themselves and re-run the Bazel command.
#
# We can do better than that: if we detect the launcher failed, we can
# purge the zig cache and retry the compilation. It will be retried for up
# to two times.
launcher_success = True
launcher_err_msg = ""
for _ in range(3):
# Do not remove the cache_prefix itself, because it is not controlled
# by this script. Instead, clear the cache subdirs that we know Zig
# populates.
zig_cache_dirs = ["h", "o", "tmp", "z"]
if not launcher_success:
print("Launcher compilation failed. Clearing %s/{%s} and retrying" %
(cache_prefix, ",".join(zig_cache_dirs)))
for d in zig_cache_dirs:
repository_ctx.delete(_paths_join(cache_prefix, d))
ret = repository_ctx.execute(
compile_cmd,
working_directory = "tools",
environment = compile_env,
)
if ret.return_code == 0:
launcher_success = True
break
launcher_success = False
full_cmd = [k + "=" + v for k, v in compile_env.items()] + compile_cmd
launcher_err_msg = _compile_failed.format(
compile_cmd = " ".join(full_cmd),
return_code = ret.return_code,
stdout = ret.stdout,
stderr = ret.stderr,
cache_prefix = cache_prefix,
)
if not launcher_success:
fail(launcher_err_msg)
exe = ".exe" if os == "windows" else ""
for target_config in target_structs():
for zig_tool in _ZIG_TOOLS + target_config.tool_paths.values():
tool_path = zig_tool_path(os).format(
zig_tool = zig_tool,
zigtarget = target_config.zigtarget,
)
repository_ctx.symlink("tools/launcher{}".format(exe), tool_path)
zig_repository = repository_rule(
attrs = {
"version": attr.string(),
"host_platform_sha256": attr.string_dict(),
"url_formats": attr.string_list(allow_empty = False),
"host_platform_include_root": attr.string_dict(),
"host_platform_ext": attr.string_dict(),
},
environ = ["HERMETIC_CC_TOOLCHAIN_CACHE_PREFIX"],
implementation = _zig_repository_impl,
)
@ -221,7 +246,8 @@ def filegroup(name, **kwargs):
native.filegroup(name = name, **kwargs)
return ":" + name
def declare_files(os, zig_include_root):
def declare_files(os):
filegroup(name = "all", srcs = native.glob(["**"]))
filegroup(name = "empty")
if os == "windows":
native.exports_files(["zig.exe"], visibility = ["//visibility:public"])
@ -229,11 +255,90 @@ def declare_files(os, zig_include_root):
else:
native.exports_files(["zig"], visibility = ["//visibility:public"])
filegroup(name = "lib/std", srcs = native.glob(["lib/std/**"]))
lazy_filegroups = {}
for target_config in target_structs():
for d in DEFAULT_INCLUDE_DIRECTORIES + target_config.includes:
d = zig_include_root + d
all_includes = [native.glob(["lib/{}/**".format(i)]) for i in target_config.includes]
cxx_tool_label = ":" + zig_tool_path(os).format(
zig_tool = "c++",
zigtarget = target_config.zigtarget,
)
filegroup(
name = "{}_includes".format(target_config.zigtarget),
srcs = _flatten(all_includes),
)
filegroup(
name = "{}_compiler_files".format(target_config.zigtarget),
srcs = [
":zig",
":{}_includes".format(target_config.zigtarget),
cxx_tool_label,
],
)
filegroup(
name = "{}_linker_files".format(target_config.zigtarget),
srcs = [
":zig",
":{}_includes".format(target_config.zigtarget),
cxx_tool_label,
] + native.glob([
"lib/libc/{}/**".format(target_config.libc),
"lib/libcxx/**",
"lib/libcxxabi/**",
"lib/libunwind/**",
"lib/compiler_rt/**",
"lib/std/**",
"lib/*.zig",
"lib/*.h",
]),
)
filegroup(
name = "{}_ar_files".format(target_config.zigtarget),
srcs = [
":zig",
":" + zig_tool_path(os).format(
zig_tool = "ar",
zigtarget = target_config.zigtarget,
),
],
)
filegroup(
name = "{}_all_files".format(target_config.zigtarget),
srcs = [
":{}_linker_files".format(target_config.zigtarget),
":{}_compiler_files".format(target_config.zigtarget),
":{}_ar_files".format(target_config.zigtarget),
],
)
for d in _DEFAULT_INCLUDE_DIRECTORIES + target_config.includes:
d = "lib/" + d
if d not in lazy_filegroups:
lazy_filegroups[d] = filegroup(name = d, srcs = native.glob([d + "/**"]))
def _flatten(iterable):
result = []
for element in iterable:
result += element
return result
## Copied from https://github.com/bazelbuild/bazel-skylib/blob/1.4.1/lib/paths.bzl#L59-L98
def _paths_is_absolute(path):
return path.startswith("/") or (len(path) > 2 and path[1] == ":")
def _paths_join(path, *others):
result = path
for p in others:
if _paths_is_absolute(p):
result = p
elif not result or result.endswith("/"):
result += p
else:
result += "/" + p
return result

436
toolchain/launcher.zig Normal file
View File

@ -0,0 +1,436 @@
// Copyright 2023 Uber Technologies, Inc.
// Licensed under the MIT License
//
// A wrapper for `zig` subcommands.
//
// In simple cases it is usually enough to:
//
// zig c++ -target <triple> <...>
//
// However, there are some caveats:
//
// * Sometimes toolchains (looking at you, Go, see an example in
// https://github.com/golang/go/pull/55966) skip CFLAGS to the underlying
// compiler. Doing that may carry a huge cost, because zig may need to spend
// ~30s compiling libc++ for an innocent feature test. Having an executable per
// target platform (like GCC does things, e.g. aarch64-linux-gnu-<tool>) is
// what most toolchains are designed to work with. So we need a wrapper per
// zig sub-command per target. As of writing, the layout is:
// tools/
// x86_64-linux-gnu.2.34
//    ar
//    c++
//    ld.lld
// x86_64-linux-musl
//    ar
//    c++
//    ld.lld
// x86_64-macos-none
//    ar
//    c++
//    ld64.lld
// ...
// * ZIG_LIB_DIR controls the output of `zig c++ -MF -MD <...>`. Bazel uses
// command to understand which input files were used to the compilation. If any
// of the files are not in `external/<...>/`, Bazel will understand and
// complain that the compiler is using undeclared directories on the host file
// system. We do not declare prerequisites using absolute paths, because that
// busts Bazel's remote cache.
// * HERMETIC_CC_TOOLCHAIN_CACHE_PREFIX is configurable per toolchain instance, and
// ZIG_GLOBAL_CACHE_DIR and ZIG_LOCAL_CACHE_DIR must be set to its value for
// all `zig` invocations.
//
// Originally this was a Bash script, then a POSIX shell script, then two
// scripts (one with pre-defined HERMETIC_CC_TOOLCHAIN_CACHE_PREFIX, one
// without). Then Windows came along with two PowerShell scripts (ports of the
// POSIX shell scripts), which I kept breaking. Then Bazel 6 came with
// `--experimental_use_hermetic_linux_sandbox`, which hermetizes the sandbox to
// the extreme: the sandbox has nothing that is not declared. /bin/sh and its
// dependencies (/lib/x86_64-linux-gnu/libc.so.6 on my system) are obviously
// not declared. So one can either declare those dependencies, bundle a shell
// to execute the wrapper, or port the shell logic to a cross-platform program
// that compiles to a static binary. By a chance we happen to already ship a
// toolchain of a language that could compile such program. And behold, the
// program is below.
const builtin = @import("builtin");
const std = @import("std");
const fs = std.fs;
const mem = std.mem;
const process = std.process;
const ChildProcess = std.ChildProcess;
const ArrayListUnmanaged = std.ArrayListUnmanaged;
const sep = fs.path.sep_str;
const EXE = switch (builtin.target.os.tag) {
.windows => ".exe",
else => "",
};
const CACHE_DIR = "{HERMETIC_CC_TOOLCHAIN_CACHE_PREFIX}";
const usage_cpp =
\\
\\Usage: <...>/tools/<target-triple>/{[zig_tool]s}{[exe]s} <args>...
\\
\\Wraps the "zig" multi-call binary. It determines the target platform from
\\the directory where it was called. Then sets ZIG_LIB_DIR,
\\ZIG_GLOBAL_CACHE_DIR, ZIG_LOCAL_CACHE_DIR and then calls:
\\
\\ zig c++ -target <target-triple> <args>...
;
const usage_other =
\\Usage: <...>/tools/<target-triple>/{[zig_tool]s}{[exe]s} <args>...
\\
\\Wraps the "zig" multi-call binary. It sets ZIG_LIB_DIR,
\\ZIG_GLOBAL_CACHE_DIR, ZIG_LOCAL_CACHE_DIR, and then calls:
\\
\\ zig {[zig_tool]s} <args>...
;
const Action = enum {
err,
exec,
};
const ExecParams = struct {
args: ArrayListUnmanaged([]const u8),
env: process.EnvMap,
};
const ParseResults = union(Action) {
err: []const u8,
exec: ExecParams,
};
pub fn main() u8 {
const allocator = if (builtin.link_libc)
std.heap.c_allocator
else blk: {
var gpa = std.heap.GeneralPurposeAllocator(.{}){};
break :blk gpa.allocator();
};
var arena_allocator = std.heap.ArenaAllocator.init(allocator);
defer arena_allocator.deinit();
const arena = arena_allocator.allocator();
var argv_it = process.argsWithAllocator(arena) catch |err|
return fatal("error parsing args: {s}\n", .{@errorName(err)});
const action = parseArgs(arena, fs.cwd(), &argv_it) catch |err|
return fatal("error: {s}\n", .{@errorName(err)});
switch (action) {
.err => |msg| return fatal("{s}", .{msg}),
.exec => |params| {
if (builtin.os.tag == .windows)
return spawnWindows(arena, params)
else
return execUnix(arena, params);
},
}
}
fn spawnWindows(arena: mem.Allocator, params: ExecParams) u8 {
var proc = ChildProcess.init(params.args.items, arena);
proc.env_map = &params.env;
const ret = proc.spawnAndWait() catch |err|
return fatal(
"error spawning {s}: {s}\n",
.{ params.args.items[0], @errorName(err) },
);
switch (ret) {
.Exited => |code| return code,
else => |other| return fatal("abnormal exit: {any}\n", .{other}),
}
}
fn execUnix(arena: mem.Allocator, params: ExecParams) u8 {
const err = process.execve(arena, params.args.items, &params.env);
std.debug.print(
"error execing {s}: {s}\n",
.{ params.args.items[0], @errorName(err) },
);
return 1;
}
// argv_it is an object that has such method:
// fn next(self: *Self) ?[]const u8
// in non-testing code it is *process.ArgIterator.
// Leaks memory: the name of the first argument is arena not by chance.
fn parseArgs(
arena: mem.Allocator,
cwd: fs.Dir,
argv_it: anytype,
) error{OutOfMemory}!ParseResults {
const arg0 = argv_it.next() orelse
return parseFatal(arena, "error: argv[0] cannot be null", .{});
const zig_tool = blk: {
const b = fs.path.basename(arg0);
if (builtin.target.os.tag == .windows and
std.ascii.eqlIgnoreCase(".exe", b[b.len - 4 ..]))
break :blk b[0 .. b.len - 4];
break :blk b;
};
const maybe_target = getTarget(arg0) catch |err| switch (err) {
error.BadParent => {
const fmt_args = .{ .zig_tool = zig_tool, .exe = EXE };
if (mem.eql(u8, zig_tool, "c++"))
return parseFatal(arena, usage_cpp, fmt_args)
else
return parseFatal(arena, usage_other, fmt_args);
},
else => |e| return e,
};
const root = blk: {
var dir = cwd.openDir(
"external" ++ sep ++ "zig_sdk" ++ sep ++ "lib",
.{ .access_sub_paths = false, .no_follow = true },
);
if (dir) |*dir_exists| {
dir_exists.close();
break :blk "external" ++ sep ++ "zig_sdk";
} else |_| {}
// directory does not exist or there was an error opening it
const here = fs.path.dirname(arg0) orelse ".";
break :blk try fs.path.join(arena, &[_][]const u8{ here, "..", ".." });
};
const zig_lib_dir = try fs.path.join(arena, &[_][]const u8{ root, "lib" });
const zig_exe = try fs.path.join(
arena,
&[_][]const u8{ root, "zig" ++ EXE },
);
var env = process.getEnvMap(arena) catch |err|
return parseFatal(arena, "error getting env: {s}", .{@errorName(err)});
try env.put("ZIG_LIB_DIR", zig_lib_dir);
try env.put("ZIG_LOCAL_CACHE_DIR", CACHE_DIR);
try env.put("ZIG_GLOBAL_CACHE_DIR", CACHE_DIR);
// args is the path to the zig binary and args to it.
var args = ArrayListUnmanaged([]const u8){};
try args.appendSlice(arena, &[_][]const u8{ zig_exe, zig_tool });
if (maybe_target) |target|
try args.appendSlice(arena, &[_][]const u8{ "-target", target });
while (argv_it.next()) |arg|
try args.append(arena, arg);
return ParseResults{ .exec = .{ .args = args, .env = env } };
}
fn parseFatal(
arena: mem.Allocator,
comptime fmt: []const u8,
args: anytype,
) error{OutOfMemory}!ParseResults {
const msg = try std.fmt.allocPrint(arena, fmt ++ "\n", args);
return ParseResults{ .err = msg };
}
pub fn fatal(comptime fmt: []const u8, args: anytype) u8 {
std.debug.print(fmt, args);
return 1;
}
fn getTarget(self_exe: []const u8) error{BadParent}!?[]const u8 {
const here = fs.path.dirname(self_exe) orelse return error.BadParent;
const triple = fs.path.basename(here);
// Validating the triple now will help users catch errors even if they
// don't yet need the target. yes yes the validation will miss things
// strings `is.it.x86_64?-stallinux,macos-`; we are trying to aid users
// that run things from the wrong directory, not trying to punish the ones
// having fun.
var it = mem.split(u8, triple, "-");
const arch = it.next() orelse return error.BadParent;
if (mem.indexOf(u8, "aarch64,x86_64", arch) == null)
return error.BadParent;
const got_os = it.next() orelse return error.BadParent;
if (mem.indexOf(u8, "linux,macos,windows", got_os) == null)
return error.BadParent;
// ABI triple is too much of a moving target
if (it.next() == null) return error.BadParent;
// but the target needs to have 3 dashes.
if (it.next() != null) return error.BadParent;
if (mem.eql(u8, "c++" ++ EXE, fs.path.basename(self_exe)))
return triple
else
return null;
}
const testing = std.testing;
pub const TestArgIterator = struct {
index: usize = 0,
argv: []const [:0]const u8,
pub fn next(self: *TestArgIterator) ?[:0]const u8 {
if (self.index == self.argv.len) return null;
defer self.index += 1;
return self.argv[self.index];
}
};
fn compareExec(
res: ParseResults,
want_args: []const [:0]const u8,
want_env_zig_lib_dir: []const u8,
) !void {
try testing.expectEqual(want_args.len, res.exec.args.items.len);
for (want_args, res.exec.args.items) |want_arg, got_arg|
try testing.expectEqualStrings(want_arg, got_arg);
try testing.expectEqualStrings(
want_env_zig_lib_dir,
res.exec.env.get("ZIG_LIB_DIR").?,
);
}
test "launcher:parseArgs" {
// not using testing.allocator, because parseArgs is designed to be used
// with an arena.
var gpa = std.heap.GeneralPurposeAllocator(.{}){};
var allocator = gpa.allocator();
const tests = [_]struct {
args: []const [:0]const u8,
precreate_dir: ?[]const u8 = null,
want_result: union(Action) {
err: []const u8,
exec: struct {
args: []const [:0]const u8,
env_zig_lib_dir: []const u8,
},
},
}{
.{
.args = &[_][:0]const u8{"ar" ++ EXE},
.want_result = .{
.err = comptime std.fmt.comptimePrint(usage_other ++ "\n", .{
.zig_tool = "ar",
.exe = EXE,
}),
},
},
.{
.args = &[_][:0]const u8{"c++" ++ EXE},
.want_result = .{
.err = comptime std.fmt.comptimePrint(usage_cpp ++ "\n", .{
.zig_tool = "c++",
.exe = EXE,
}),
},
},
.{
.args = &[_][:0]const u8{
"tools" ++ sep ++ "x86_64-linux-musl" ++ sep ++ "c++" ++ EXE,
"main.c",
"-o",
"/dev/null",
},
.want_result = .{
.exec = .{
.args = &[_][:0]const u8{
"tools" ++ sep ++ "x86_64-linux-musl" ++ sep ++
".." ++ sep ++ ".." ++ sep ++ "zig" ++ EXE,
"c++",
"-target",
"x86_64-linux-musl",
"main.c",
"-o",
"/dev/null",
},
.env_zig_lib_dir = "tools" ++ sep ++ "x86_64-linux-musl" ++
sep ++ ".." ++ sep ++ ".." ++ sep ++ "lib",
},
},
},
.{
.args = &[_][:0]const u8{
"tools" ++ sep ++ "x86_64-linux-musl" ++ sep ++ "ar" ++ EXE,
"-rcs",
"all.a",
"main.o",
"foo.o",
},
.want_result = .{
.exec = .{
.args = &[_][:0]const u8{
"tools" ++ sep ++ "x86_64-linux-musl" ++ sep ++ ".." ++
sep ++ ".." ++ sep ++ "zig" ++ EXE,
"ar",
"-rcs",
"all.a",
"main.o",
"foo.o",
},
.env_zig_lib_dir = "tools" ++ sep ++ "x86_64-linux-musl" ++
sep ++ ".." ++ sep ++ ".." ++ sep ++ "lib",
},
},
},
.{
.args = &[_][:0]const u8{
"external_zig_sdk" ++ sep ++ "tools" ++ sep ++
"x86_64-linux-gnu.2.28" ++ sep ++ "c++" ++ EXE,
"main.c",
"-o",
"/dev/null",
},
.precreate_dir = "external" ++ sep ++ "zig_sdk" ++ sep ++ "lib",
.want_result = .{
.exec = .{
.args = &[_][:0]const u8{
"external" ++ sep ++ "zig_sdk" ++ sep ++ "zig" ++ EXE,
"c++",
"-target",
"x86_64-linux-gnu.2.28",
"main.c",
"-o",
"/dev/null",
},
.env_zig_lib_dir = "external" ++ sep ++ "zig_sdk" ++
sep ++ "lib",
},
},
},
};
for (tests) |tt| {
var tmp = testing.tmpDir(.{});
defer tmp.cleanup();
if (tt.precreate_dir) |dir|
try tmp.dir.makePath(dir);
var res = try parseArgs(allocator, tmp.dir, &TestArgIterator{
.argv = tt.args,
});
switch (tt.want_result) {
.err => |want_msg| try testing.expectEqualStrings(
want_msg,
res.err,
),
.exec => |want| {
try compareExec(res, want.args, want.env_zig_lib_dir);
},
}
}
}

View File

@ -1,4 +1,7 @@
load("@bazel-zig-cc//toolchain/libc:defs.bzl", "declare_libcs")
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
load("@hermetic_cc_toolchain//toolchain/libc:defs.bzl", "declare_libcs")
package(
default_visibility = ["//visibility:public"],

View File

@ -1,4 +1,7 @@
load("@bazel-zig-cc//toolchain/private:defs.bzl", "LIBCS")
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
load("@hermetic_cc_toolchain//toolchain/private:defs.bzl", "LIBCS")
def declare_libcs():
for libc in LIBCS:

View File

@ -1,4 +1,7 @@
load("@bazel-zig-cc//toolchain/platform:defs.bzl", "declare_libc_aware_platforms")
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
load("@hermetic_cc_toolchain//toolchain/platform:defs.bzl", "declare_libc_aware_platforms")
package(
default_visibility = ["//visibility:public"],

View File

@ -1,4 +1,7 @@
load("@bazel-zig-cc//toolchain/toolchain:defs.bzl", "declare_libc_aware_toolchains")
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
load("@hermetic_cc_toolchain//toolchain/toolchain:defs.bzl", "declare_libc_aware_toolchains")
package(
default_visibility = ["//visibility:public"],

View File

@ -1,4 +1,4 @@
load("@bazel-zig-cc//toolchain/platform:defs.bzl", "declare_platforms")
load("@hermetic_cc_toolchain//toolchain/platform:defs.bzl", "declare_platforms")
package(
default_visibility = ["//visibility:public"],

View File

@ -1,4 +1,4 @@
load("@bazel-zig-cc//toolchain/private:defs.bzl", "LIBCS")
load("@hermetic_cc_toolchain//toolchain/private:defs.bzl", "LIBCS")
_CPUS = (("x86_64", "amd64"), ("aarch64", "arm64"))
_OS = {

View File

@ -1,7 +0,0 @@
load("@bazel-zig-cc//toolchain/private:cc_toolchains.bzl", "declare_cc_toolchains")
declare_cc_toolchains(
os = {os},
absolute_path = {absolute_path},
zig_include_root = {zig_include_root},
)

View File

@ -1,5 +1,5 @@
load(":defs.bzl", "DEFAULT_INCLUDE_DIRECTORIES", "target_structs", "zig_tool_path")
load("@bazel-zig-cc//toolchain:zig_toolchain.bzl", "zig_cc_toolchain_config")
load(":defs.bzl", "target_structs", "zig_tool_path")
load("@hermetic_cc_toolchain//toolchain:zig_toolchain.bzl", "zig_cc_toolchain_config")
DEFAULT_TOOL_PATHS = {
"ar": "ar",
@ -11,33 +11,30 @@ DEFAULT_TOOL_PATHS = {
"strip": "/usr/bin/false",
}.items()
def declare_cc_toolchains(os, absolute_path, zig_include_root):
def declare_cc_toolchains(os, zig_sdk_path):
for target_config in target_structs():
gotarget = target_config.gotarget
zigtarget = target_config.zigtarget
cxx_builtin_include_directories = []
for d in DEFAULT_INCLUDE_DIRECTORIES + target_config.includes:
d = zig_include_root + d
cxx_builtin_include_directories.append(absolute_path + "/" + d)
for d in getattr(target_config, "toplevel_include", []):
cxx_builtin_include_directories.append(absolute_path + "/" + d)
absolute_tool_paths = {}
for name, path in target_config.tool_paths.items() + DEFAULT_TOOL_PATHS:
if path[0] == "/":
absolute_tool_paths[name] = path
continue
tool_path = zig_tool_path(os).format(zig_tool = path)
absolute_tool_paths[name] = "%s/%s" % (absolute_path, tool_path)
tool_path = zig_tool_path(os).format(
zig_tool = path,
zigtarget = zigtarget,
)
absolute_tool_paths[name] = tool_path
linkopts = target_config.linkopts
dynamic_library_linkopts = target_config.dynamic_library_linkopts
copts = target_config.copts
for s in getattr(target_config, "linker_version_scripts", []):
linkopts = linkopts + ["-Wl,--version-script,%s/%s" % (absolute_path, s)]
for incl in getattr(target_config, "compiler_extra_includes", []):
copts = copts + ["-include", absolute_path + "/" + incl]
linkopts = target_config.linkopts
# We can't pass a list of structs to a rule, so we use json encoding.
artifact_name_patterns = getattr(target_config, "artifact_name_patterns", [])
artifact_name_pattern_strings = [json.encode(p) for p in artifact_name_patterns]
zig_cc_toolchain_config(
name = zigtarget + "_cc_config",
@ -53,18 +50,21 @@ def declare_cc_toolchains(os, absolute_path, zig_include_root):
compiler = "clang",
abi_version = "unknown",
abi_libc_version = "unknown",
artifact_name_patterns = artifact_name_pattern_strings,
visibility = ["//visibility:private"],
)
native.cc_toolchain(
name = zigtarget + "_cc",
toolchain_identifier = zigtarget + "-toolchain",
toolchain_config = ":%s_cc_config" % zigtarget,
all_files = "@zig_sdk//:zig",
ar_files = "@zig_sdk//:zig",
compiler_files = "@zig_sdk//:zig",
linker_files = "@zig_sdk//:zig",
all_files = "@zig_sdk//:%s_all_files" % zigtarget,
ar_files = "@zig_sdk//:%s_ar_files" % zigtarget,
compiler_files = "@zig_sdk//:%s_compiler_files" % zigtarget,
linker_files = "@zig_sdk//:%s_linker_files" % zigtarget,
dwp_files = "@zig_sdk//:empty",
objcopy_files = "@zig_sdk//:empty",
strip_files = "@zig_sdk//:empty",
supports_param_files = 0,
visibility = ["//visibility:private"],
)

View File

@ -1,10 +1,4 @@
DEFAULT_INCLUDE_DIRECTORIES = [
"include",
"libcxx/include",
"libcxxabi/include",
]
_ZIG_TOOL_PATH = "tools/{zig_tool}"
_ZIG_TOOL_PATH = "tools/{zigtarget}/{zig_tool}"
# Zig supports even older glibcs than defined below, but we have tested only
# down to 2.17.
@ -28,28 +22,37 @@ _GLIBCS = [
"2.34",
]
_INCLUDE_TAIL = [
"libcxx/include",
"libcxxabi/include",
"include",
]
LIBCS = ["musl"] + ["gnu.{}".format(glibc) for glibc in _GLIBCS]
def zig_tool_path(os):
if os == "windows":
return _ZIG_TOOL_PATH + ".bat"
return _ZIG_TOOL_PATH + ".exe"
else:
return _ZIG_TOOL_PATH
def target_structs():
ret = []
for zigcpu, gocpu in (("x86_64", "amd64"), ("aarch64", "arm64")):
ret.append(_target_darwin(gocpu, zigcpu))
ret.append(_target_macos(gocpu, zigcpu))
ret.append(_target_windows(gocpu, zigcpu))
ret.append(_target_linux_musl(gocpu, zigcpu))
for glibc in _GLIBCS:
ret.append(_target_linux_gnu(gocpu, zigcpu, glibc))
return ret
def _target_darwin(gocpu, zigcpu):
min_os = "10"
def _target_macos(gocpu, zigcpu):
min_os = "11"
copts = []
if zigcpu == "aarch64":
min_os = "11"
copts = ["-mcpu=apple_m1"]
return struct(
gotarget = "darwin_{}".format(gocpu),
zigtarget = "{}-macos-none".format(zigcpu),
@ -59,16 +62,24 @@ def _target_darwin(gocpu, zigcpu):
"libc/include/{}-macos.{}-none".format(zigcpu, min_os),
"libc/include/any-macos.{}-any".format(min_os),
"libc/include/any-macos-any",
],
linkopts = [],
] + _INCLUDE_TAIL,
linkopts = ["-Wl,-headerpad_max_install_names"],
dynamic_library_linkopts = ["-Wl,-undefined=dynamic_lookup"],
copts = [],
copts = copts,
libc = "darwin",
bazel_target_cpu = "darwin",
constraint_values = [
"@platforms//os:macos",
"@platforms//cpu:{}".format(zigcpu),
],
tool_paths = {"ld": "ld64.lld"},
artifact_name_patterns = [
{
"category_name": "dynamic_library",
"prefix": "lib",
"extension": ".dylib",
},
],
)
def _target_windows(gocpu, zigcpu):
@ -76,43 +87,58 @@ def _target_windows(gocpu, zigcpu):
gotarget = "windows_{}".format(gocpu),
zigtarget = "{}-windows-gnu".format(zigcpu),
includes = [
"libc/mingw",
"libunwind/include",
"libc/include/any-windows-any",
],
] + _INCLUDE_TAIL,
linkopts = [],
dynamic_library_linkopts = [],
copts = [],
libc = "mingw",
bazel_target_cpu = "x64_windows",
constraint_values = [
"@platforms//os:windows",
"@platforms//cpu:{}".format(zigcpu),
],
tool_paths = {"ld": "ld64.lld"},
artifact_name_patterns = [
{
"category_name": "static_library",
"prefix": "",
"extension": ".lib",
},
{
"category_name": "dynamic_library",
"prefix": "",
"extension": ".dll",
},
{
"category_name": "executable",
"prefix": "",
"extension": ".exe",
},
],
)
def _target_linux_gnu(gocpu, zigcpu, glibc_version):
glibc_suffix = "gnu.{}".format(glibc_version)
# https://github.com/ziglang/zig/issues/5882#issuecomment-888250676
# fcntl_hack is only required for glibc 2.27 or less.
fcntl_hack = glibc_version < "2.28"
return struct(
gotarget = "linux_{}_{}".format(gocpu, glibc_suffix),
zigtarget = "{}-linux-{}".format(zigcpu, glibc_suffix),
includes = [
"libunwind/include",
"libc/include/generic-glibc",
"libc/include/{}-linux-gnu".format(zigcpu),
"libc/include/generic-glibc",
] +
# x86_64-linux-any is x86_64-linux and x86-linux combined.
(["libc/include/x86-linux-any"] if zigcpu == "x86_64" else []) +
(["libc/include/{}-linux-any".format(zigcpu)] if zigcpu != "x86_64" else []) + [
"libc/include/any-linux-any",
"libc/include/{}-linux-gnu".format(zigcpu),
"libc/include/{}-linux-any".format(zigcpu),
] + (["libc/include/x86-linux-any"] if zigcpu == "x86_64" else []),
toplevel_include = ["glibc-hacks"] if fcntl_hack else [],
compiler_extra_includes = ["glibc-hacks/glibchack-fcntl.h"] if fcntl_hack else [],
linker_version_scripts = ["glibc-hacks/fcntl.map"] if fcntl_hack else [],
linkopts = ["-lc++", "-lc++abi"],
] + _INCLUDE_TAIL,
linkopts = [],
dynamic_library_linkopts = [],
copts = [],
libc = "glibc",
bazel_target_cpu = "k8",
constraint_values = [
"@platforms//os:linux",
@ -120,6 +146,7 @@ def _target_linux_gnu(gocpu, zigcpu, glibc_version):
],
libc_constraint = "@zig_sdk//libc:{}".format(glibc_suffix),
tool_paths = {"ld": "ld.lld"},
artifact_name_patterns = [],
)
def _target_linux_musl(gocpu, zigcpu):
@ -127,14 +154,18 @@ def _target_linux_musl(gocpu, zigcpu):
gotarget = "linux_{}_musl".format(gocpu),
zigtarget = "{}-linux-musl".format(zigcpu),
includes = [
"libc/include/generic-musl",
"libc/include/{}-linux-musl".format(zigcpu),
"libc/include/generic-musl",
] +
# x86_64-linux-any is x86_64-linux and x86-linux combined.
(["libc/include/x86-linux-any"] if zigcpu == "x86_64" else []) +
(["libc/include/{}-linux-any".format(zigcpu)] if zigcpu != "x86_64" else []) + [
"libc/include/any-linux-any",
"libc/include/{}-linux-musl".format(zigcpu),
"libc/include/{}-linux-any".format(zigcpu),
] + (["libc/include/x86-linux-any"] if zigcpu == "x86_64" else []),
linkopts = ["-s", "-w"],
] + _INCLUDE_TAIL,
linkopts = [],
dynamic_library_linkopts = [],
copts = ["-D_LIBCPP_HAS_MUSL_LIBC", "-D_LIBCPP_HAS_THREAD_API_PTHREAD"],
libc = "musl",
bazel_target_cpu = "k8",
constraint_values = [
"@platforms//os:linux",
@ -142,4 +173,5 @@ def _target_linux_musl(gocpu, zigcpu):
],
libc_constraint = "@zig_sdk//libc:musl",
tool_paths = {"ld": "ld.lld"},
artifact_name_patterns = [],
)

View File

@ -1,4 +1,4 @@
load("@bazel-zig-cc//toolchain/toolchain:defs.bzl", "declare_toolchains")
load("@hermetic_cc_toolchain//toolchain/toolchain:defs.bzl", "declare_toolchains")
package(
default_visibility = ["//visibility:public"],

View File

@ -1,4 +1,4 @@
load("@bazel-zig-cc//toolchain/private:defs.bzl", "target_structs")
load("@hermetic_cc_toolchain//toolchain/private:defs.bzl", "target_structs")
def declare_toolchains():
for target_config in target_structs():
@ -32,7 +32,7 @@ def _declare_toolchain(gotarget, zigtarget, target_compatible_with):
name = gotarget,
exec_compatible_with = None,
target_compatible_with = target_compatible_with,
toolchain = "@zig_sdk//private:%s_cc" % zigtarget,
toolchain = "@zig_sdk//:%s_cc" % zigtarget,
toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
)
@ -41,6 +41,6 @@ def _declare_toolchain(gotarget, zigtarget, target_compatible_with):
name = zigtarget,
exec_compatible_with = None,
target_compatible_with = target_compatible_with,
toolchain = "@zig_sdk//private:%s_cc" % zigtarget,
toolchain = "@zig_sdk//:%s_cc" % zigtarget,
toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
)

View File

@ -1,6 +1,7 @@
load("@bazel_tools//tools/build_defs/cc:action_names.bzl", "ACTION_NAMES")
load(
"@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl",
"artifact_name_pattern",
"feature",
"feature_set",
"flag_group",
@ -68,6 +69,7 @@ def _compilation_mode_features(ctx):
],
)
# fastbuild also gets the strip_debug_symbols flags by default.
fastbuild_feature = feature(
name = "fastbuild",
flag_sets = [
@ -75,7 +77,7 @@ def _compilation_mode_features(ctx):
actions = actions,
flag_groups = [
flag_group(
flags = ["-fno-lto", "-Wl,-S"],
flags = ["-fno-lto"],
),
],
),
@ -93,8 +95,6 @@ def _zig_cc_toolchain_config_impl(ctx):
"-I" + d
for d in ctx.attr.cxx_builtin_include_directories
] + [
"-target",
ctx.attr.target + ctx.attr.target_suffix,
"-no-canonical-prefixes",
"-Wno-builtin-macro-redefined",
"-D__DATE__=\"redacted\"",
@ -115,37 +115,62 @@ def _zig_cc_toolchain_config_impl(ctx):
],
)
link_flag_sets = []
if ctx.attr.linkopts:
link_flag_sets.append(
flag_set(
actions = all_link_actions,
flag_groups = [flag_group(flags = ctx.attr.linkopts)],
),
)
if ctx.attr.dynamic_library_linkopts:
dynamic_library_flag_sets = [
link_flag_sets.append(
flag_set(
actions = dynamic_library_link_actions,
flag_groups = [flag_group(flags = ctx.attr.dynamic_library_linkopts)],
),
]
else:
dynamic_library_flag_sets = []
)
default_linker_flags = feature(
name = "default_linker_flags",
enabled = True,
flag_sets = link_flag_sets,
)
supports_dynamic_linker = feature(
name = "supports_dynamic_linker",
enabled = True,
)
strip_debug_symbols_feature = feature(
name = "strip_debug_symbols",
flag_sets = [
flag_set(
actions = all_link_actions,
flag_groups = [
flag_group(
flags = ["-target", ctx.attr.target] +
ctx.attr.linkopts,
flags = ["-Wl,-S"],
expand_if_available = "strip_debug_symbols",
),
],
),
] + dynamic_library_flag_sets,
],
)
features = [
compile_and_link_flags,
default_linker_flags,
supports_dynamic_linker,
strip_debug_symbols_feature,
] + _compilation_mode_features(ctx)
artifact_name_patterns = [
artifact_name_pattern(**json.decode(p))
for p in ctx.attr.artifact_name_patterns
]
return cc_common.create_cc_toolchain_config_info(
ctx = ctx,
features = features,
@ -162,6 +187,7 @@ def _zig_cc_toolchain_config_impl(ctx):
for name, path in ctx.attr.tool_paths.items()
],
cxx_builtin_include_directories = ctx.attr.cxx_builtin_include_directories,
artifact_name_patterns = artifact_name_patterns,
)
zig_cc_toolchain_config = rule(
@ -180,6 +206,7 @@ zig_cc_toolchain_config = rule(
"compiler": attr.string(),
"abi_version": attr.string(),
"abi_libc_version": attr.string(),
"artifact_name_patterns": attr.string_list(),
},
provides = [CcToolchainConfigInfo],
)

5
tools.go Normal file
View File

@ -0,0 +1,5 @@
package bazel_zig_cc
import (
_ "github.com/bazelbuild/buildtools/buildifier"
)

493
tools/bazel Executable file
View File

@ -0,0 +1,493 @@
#!/usr/bin/env python3
"""
Copyright 2018 Google Inc. All rights reserved.
Licensed under the MIT License (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""
# Origin: https://github.com/bazelbuild/bazelisk/blob/fc3e3d68c42744dc1c01739f9710cc52f4a8258c/bazelisk.py
import base64
from contextlib import closing
import hashlib
import json
import netrc
import os
import os.path
import platform
import re
import shutil
import subprocess
import sys
import tempfile
import time
try:
from urllib.parse import urlparse
from urllib.request import urlopen, Request
from urllib.error import HTTPError
except ImportError:
# Python 2.x compatibility hack.
# http://python-future.org/compatible_idioms.html?highlight=urllib#urllib-module
from urlparse import urlparse
from urllib2 import urlopen, Request, HTTPError
FileNotFoundError = IOError
ONE_HOUR = 1 * 60 * 60
LATEST_PATTERN = re.compile(r"latest(-(?P<offset>\d+))?$")
LAST_GREEN_COMMIT_BASE_PATH = (
"https://storage.googleapis.com/bazel-untrusted-builds/last_green_commit/"
)
LAST_GREEN_COMMIT_PATH_SUFFIXES = {
"last_green": "github.com/bazelbuild/bazel.git/bazel-bazel",
"last_downstream_green": "downstream_pipeline",
}
BAZEL_GCS_PATH_PATTERN = (
"https://storage.googleapis.com/bazel-builds/artifacts/{platform}/{commit}/bazel"
)
SUPPORTED_PLATFORMS = {"linux": "ubuntu1404", "windows": "windows", "darwin": "macos"}
TOOLS_BAZEL_PATH = "./tools/bazel"
BAZEL_REAL = "BAZEL_REAL"
BAZEL_UPSTREAM = "bazelbuild"
def decide_which_bazel_version_to_use():
# Check in this order:
# - env var "USE_BAZEL_VERSION" is set to a specific version.
# - env var "USE_NIGHTLY_BAZEL" or "USE_BAZEL_NIGHTLY" is set -> latest
# nightly. (TODO)
# - env var "USE_CANARY_BAZEL" or "USE_BAZEL_CANARY" is set -> latest
# rc. (TODO)
# - the file workspace_root/tools/bazel exists -> that version. (TODO)
# - workspace_root/.bazelversion exists -> read contents, that version.
# - workspace_root/WORKSPACE contains a version -> that version. (TODO)
# - fallback: latest release
if "USE_BAZEL_VERSION" in os.environ:
return os.environ["USE_BAZEL_VERSION"]
workspace_root = find_workspace_root()
if workspace_root:
bazelversion_path = os.path.join(workspace_root, ".bazelversion")
if os.path.exists(bazelversion_path):
with open(bazelversion_path, "r") as f:
return f.read().strip()
return "latest"
def find_workspace_root(root=None):
if root is None:
root = os.getcwd()
if os.path.exists(os.path.join(root, "WORKSPACE")):
return root
new_root = os.path.dirname(root)
return find_workspace_root(new_root) if new_root != root else None
def resolve_version_label_to_number_or_commit(bazelisk_directory, version):
"""Resolves the given label to a released version of Bazel or a commit.
Args:
bazelisk_directory: string; path to a directory that can store
temporary data for Bazelisk.
version: string; the version label that should be resolved.
Returns:
A (string, bool) tuple that consists of two parts:
1. the resolved number of a Bazel release (candidate), or the commit
of an unreleased Bazel binary,
2. An indicator for whether the returned version refers to a commit.
"""
suffix = LAST_GREEN_COMMIT_PATH_SUFFIXES.get(version)
if suffix:
return get_last_green_commit(suffix), True
if "latest" in version:
match = LATEST_PATTERN.match(version)
if not match:
raise Exception(
'Invalid version "{}". In addition to using a version '
'number such as "0.20.0", you can use values such as '
'"latest" and "latest-N", with N being a non-negative '
"integer.".format(version)
)
history = get_version_history(bazelisk_directory)
offset = int(match.group("offset") or "0")
return resolve_latest_version(history, offset), False
return version, False
def get_last_green_commit(path_suffix):
return read_remote_text_file(LAST_GREEN_COMMIT_BASE_PATH + path_suffix).strip()
def get_releases_json(bazelisk_directory):
"""Returns the most recent versions of Bazel, in descending order."""
releases = os.path.join(bazelisk_directory, "releases.json")
# Use a cached version if it's fresh enough.
if os.path.exists(releases):
if abs(time.time() - os.path.getmtime(releases)) < ONE_HOUR:
with open(releases, "rb") as f:
try:
return json.loads(f.read().decode("utf-8"))
except ValueError:
print("WARN: Could not parse cached releases.json.")
pass
with open(releases, "wb") as f:
body = read_remote_text_file("https://api.github.com/repos/bazelbuild/bazel/releases")
f.write(body.encode("utf-8"))
return json.loads(body)
def read_remote_text_file(url):
with closing(urlopen(url)) as res:
body = res.read()
try:
return body.decode(res.info().get_content_charset("iso-8859-1"))
except AttributeError:
# Python 2.x compatibility hack
return body.decode(res.info().getparam("charset") or "iso-8859-1")
def get_version_history(bazelisk_directory):
return sorted(
(
release["tag_name"]
for release in get_releases_json(bazelisk_directory)
if not release["prerelease"]
),
# This only handles versions with numeric components, but that is fine
# since prerelease versions have been excluded.
key=lambda version: tuple(int(component)
for component in version.split('.')),
reverse=True,
)
def resolve_latest_version(version_history, offset):
if offset >= len(version_history):
version = "latest-{}".format(offset) if offset else "latest"
raise Exception(
'Cannot resolve version "{}": There are only {} Bazel '
"releases.".format(version, len(version_history))
)
# This only works since we store the history in descending order.
return version_history[offset]
def get_operating_system():
operating_system = platform.system().lower()
if operating_system not in ("linux", "darwin", "windows"):
raise Exception(
'Unsupported operating system "{}". '
"Bazel currently only supports Linux, macOS and Windows.".format(operating_system)
)
return operating_system
def determine_executable_filename_suffix():
operating_system = get_operating_system()
return ".exe" if operating_system == "windows" else ""
def determine_bazel_filename(version):
operating_system = get_operating_system()
supported_machines = get_supported_machine_archs(version, operating_system)
machine = normalized_machine_arch_name()
if machine not in supported_machines:
raise Exception(
'Unsupported machine architecture "{}". Bazel {} only supports {} on {}.'.format(
machine, version, ", ".join(supported_machines), operating_system.capitalize()
)
)
filename_suffix = determine_executable_filename_suffix()
bazel_flavor = "bazel"
if os.environ.get("BAZELISK_NOJDK", "0") != "0":
bazel_flavor = "bazel_nojdk"
return "{}-{}-{}-{}{}".format(bazel_flavor, version, operating_system, machine, filename_suffix)
def get_supported_machine_archs(version, operating_system):
supported_machines = ["x86_64"]
versions = version.split(".")[:2]
if len(versions) == 2:
# released version
major, minor = int(versions[0]), int(versions[1])
if (
operating_system == "darwin"
and (major > 4 or major == 4 and minor >= 1)
or operating_system == "linux"
and (major > 3 or major == 3 and minor >= 4)
):
# Linux arm64 was supported since 3.4.0.
# Darwin arm64 was supported since 4.1.0.
supported_machines.append("arm64")
elif operating_system in ("darwin", "linux"):
# This is needed to run bazelisk_test.sh on Linux and Darwin arm64 machines, which are
# becoming more and more popular.
# It works because all recent commits of Bazel support arm64 on Darwin and Linux.
# However, this would add arm64 by mistake if the commit is too old, which should be
# a rare scenario.
supported_machines.append("arm64")
return supported_machines
def normalized_machine_arch_name():
machine = platform.machine().lower()
if machine == "amd64":
machine = "x86_64"
elif machine == "aarch64":
machine = "arm64"
return machine
def determine_url(version, is_commit, bazel_filename):
if is_commit:
sys.stderr.write("Using unreleased version at commit {}\n".format(version))
# No need to validate the platform thanks to determine_bazel_filename().
return BAZEL_GCS_PATH_PATTERN.format(
platform=SUPPORTED_PLATFORMS[platform.system().lower()], commit=version
)
# Split version into base version and optional additional identifier.
# Example: '0.19.1' -> ('0.19.1', None), '0.20.0rc1' -> ('0.20.0', 'rc1')
(version, rc) = re.match(r"(\d*\.\d*(?:\.\d*)?)(rc\d+)?", version).groups()
if "BAZELISK_BASE_URL" in os.environ:
return "{}/{}/{}".format(os.environ["BAZELISK_BASE_URL"], version, bazel_filename)
else:
return "https://releases.bazel.build/{}/{}/{}".format(
version, rc if rc else "release", bazel_filename
)
def trim_suffix(string, suffix):
if string.endswith(suffix):
return string[: len(string) - len(suffix)]
else:
return string
def download_bazel_into_directory(version, is_commit, directory):
bazel_filename = determine_bazel_filename(version)
bazel_url = determine_url(version, is_commit, bazel_filename)
filename_suffix = determine_executable_filename_suffix()
bazel_directory_name = trim_suffix(bazel_filename, filename_suffix)
destination_dir = os.path.join(directory, bazel_directory_name, "bin")
maybe_makedirs(destination_dir)
destination_path = os.path.join(destination_dir, "bazel" + filename_suffix)
if not os.path.exists(destination_path):
download(bazel_url, destination_path)
os.chmod(destination_path, 0o755)
sha256_path = destination_path + ".sha256"
expected_hash = ""
if not os.path.exists(sha256_path):
try:
download(bazel_url + ".sha256", sha256_path)
except HTTPError as e:
if e.code == 404:
sys.stderr.write(
"The Bazel mirror does not have a checksum file; skipping checksum verification."
)
return destination_path
raise e
with open(sha256_path, "r") as sha_file:
expected_hash = sha_file.read().split()[0]
sha256_hash = hashlib.sha256()
with open(destination_path, "rb") as bazel_file:
for byte_block in iter(lambda: bazel_file.read(4096), b""):
sha256_hash.update(byte_block)
actual_hash = sha256_hash.hexdigest()
if actual_hash != expected_hash:
os.remove(destination_path)
os.remove(sha256_path)
print(
"The downloaded Bazel binary is corrupted. Expected SHA-256 {}, got {}. Please try again.".format(
expected_hash, actual_hash
)
)
# Exiting with a special exit code not used by Bazel, so the calling process may retry based on that.
# https://docs.bazel.build/versions/0.21.0/guide.html#what-exit-code-will-i-get
sys.exit(22)
return destination_path
def download(url, destination_path):
sys.stderr.write("Downloading {}...\n".format(url))
request = Request(url)
if "BAZELISK_BASE_URL" in os.environ:
parts = urlparse(url)
creds = None
try:
creds = netrc.netrc().hosts.get(parts.netloc)
except Exception:
pass
if creds is not None:
auth = base64.b64encode(("%s:%s" % (creds[0], creds[2])).encode("ascii"))
request.add_header("Authorization", "Basic %s" % auth.decode("utf-8"))
with closing(urlopen(request)) as response, open(destination_path, "wb") as file:
shutil.copyfileobj(response, file)
def get_bazelisk_directory():
bazelisk_home = os.environ.get("BAZELISK_HOME")
if bazelisk_home is not None:
return bazelisk_home
operating_system = get_operating_system()
base_dir = None
if operating_system == "windows":
base_dir = os.environ.get("LocalAppData")
if base_dir is None:
raise Exception("%LocalAppData% is not defined")
elif operating_system == "darwin":
base_dir = os.environ.get("HOME")
if base_dir is None:
raise Exception("$HOME is not defined")
base_dir = os.path.join(base_dir, "Library/Caches")
elif operating_system == "linux":
base_dir = os.environ.get("XDG_CACHE_HOME")
if base_dir is None:
base_dir = os.environ.get("HOME")
if base_dir is None:
raise Exception("neither $XDG_CACHE_HOME nor $HOME are defined")
base_dir = os.path.join(base_dir, ".cache")
else:
raise Exception("Unsupported operating system '{}'".format(operating_system))
return os.path.join(base_dir, "bazelisk")
def maybe_makedirs(path):
"""
Creates a directory and its parents if necessary.
"""
try:
os.makedirs(path)
except OSError as e:
if not os.path.isdir(path):
raise e
def delegate_tools_bazel(bazel_path):
"""Match Bazel's own delegation behavior in the builds distributed by most
package managers: use tools/bazel if it's present, executable, and not this
script.
"""
root = find_workspace_root()
if root:
wrapper = os.path.join(root, TOOLS_BAZEL_PATH)
if os.path.exists(wrapper) and os.access(wrapper, os.X_OK):
try:
if not os.path.samefile(wrapper, __file__):
return wrapper
except AttributeError:
# Python 2 on Windows does not support os.path.samefile
if os.path.abspath(wrapper) != os.path.abspath(__file__):
return wrapper
return None
def prepend_directory_to_path(env, directory):
"""
Prepend binary directory to PATH
"""
if "PATH" in env:
env["PATH"] = directory + os.pathsep + env["PATH"]
else:
env["PATH"] = directory
def make_bazel_cmd(bazel_path, argv):
env = os.environ.copy()
wrapper = delegate_tools_bazel(bazel_path)
if wrapper:
env[BAZEL_REAL] = bazel_path
bazel_path = wrapper
directory = os.path.dirname(bazel_path)
prepend_directory_to_path(env, directory)
return {
"exec": bazel_path,
"args": argv,
"env": env,
}
def execute_bazel(bazel_path, argv):
cmd = make_bazel_cmd(bazel_path, argv)
# We cannot use close_fds on Windows, so disable it there.
p = subprocess.Popen([cmd["exec"]] + cmd["args"], close_fds=os.name != "nt", env=cmd["env"])
while True:
try:
return p.wait()
except KeyboardInterrupt:
# Bazel will also get the signal and terminate.
# We should continue waiting until it does so.
pass
def get_bazel_path():
bazelisk_directory = get_bazelisk_directory()
maybe_makedirs(bazelisk_directory)
bazel_version = decide_which_bazel_version_to_use()
bazel_version, is_commit = resolve_version_label_to_number_or_commit(
bazelisk_directory, bazel_version
)
# TODO: Support other forks just like Go version
bazel_directory = os.path.join(bazelisk_directory, "downloads", BAZEL_UPSTREAM)
return download_bazel_into_directory(bazel_version, is_commit, bazel_directory)
def main(argv=None):
if argv is None:
argv = sys.argv
bazel_path = get_bazel_path()
argv = argv[1:]
if argv and argv[0] == "--print_env":
cmd = make_bazel_cmd(bazel_path, argv)
env = cmd["env"]
for key in env:
print("{}={}".format(key, env[key]))
return 0
return execute_bazel(bazel_path, argv)
if __name__ == "__main__":
sys.exit(main())

8
tools/buildifier Executable file
View File

@ -0,0 +1,8 @@
#!/bin/sh
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
set -xeu
exec "$(git rev-parse --show-toplevel)/tools/bazel" run @com_github_bazelbuild_buildtools//buildifier -- "$@"

12
tools/mod-tidy Executable file
View File

@ -0,0 +1,12 @@
#!/bin/sh
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
set -xeu
cd "$(git rev-parse --show-toplevel)"
echo "--- go mod tidy"
tools/bazel run @go_sdk//:bin/go -- mod tidy "$@"
echo "--- gazelle-update-repos"
exec tools/bazel run //:gazelle-update-repos

23
tools/releaser/BUILD Normal file
View File

@ -0,0 +1,23 @@
# Copyright 2023 Uber Technologies, Inc.
# Licensed under the MIT License
load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library", "go_test")
go_library(
name = "releaser_lib",
srcs = ["main.go"],
importpath = "github.com/uber/hermetic_cc_toolchain/tools/releaser",
visibility = ["//visibility:private"],
)
go_binary(
name = "releaser",
embed = [":releaser_lib"],
visibility = ["//visibility:public"],
)
go_test(
name = "releaser_test",
srcs = ["main_test.go"],
embed = [":releaser_lib"],
)

1
tools/releaser/WORKSPACE Normal file
View File

@ -0,0 +1 @@
workspace(name = "hermetic_cc_toolchain")

168
tools/releaser/main.go Normal file
View File

@ -0,0 +1,168 @@
// Copyright 2023 Uber Technologies, Inc.
// Licensed under the MIT License
// releaser is a tool for managing part of the process to release a new version of hermetic_cc_toolchain.
package main
import (
"bytes"
"compress/gzip"
"crypto/sha256"
"errors"
"flag"
"fmt"
"io"
"os"
"os/exec"
"path"
"regexp"
"strings"
)
var (
// Paths to be included to the release
_paths = []string{
"LICENSE",
"README.md",
"MODULE.bazel",
"toolchain/*",
}
// regexp for valid tags
tagRegexp = regexp.MustCompile(`^v([0-9]+)\.([0-9]+)(\.([0-9]+))(-rc([0-9]+))?$`)
errTag = errors.New("tag accepts the following formats: v1.0.0 v1.0.1-rc1")
)
func main() {
if err := run(); err != nil {
fmt.Fprintf(os.Stderr, "error: %s\n", err)
os.Exit(1)
}
}
func log(msg string, format ...any) {
fmt.Fprintf(flag.CommandLine.Output(), msg+"\n", format...)
}
func run() (_err error) {
var (
repoRoot string
tag string
)
flag.StringVar(&repoRoot, "repo_root", os.Getenv("BUILD_WORKSPACE_DIRECTORY"), "root directory of hermetic_cc_toolchain repo")
flag.StringVar(&tag, "tag", "", "tag for this release")
flag.Usage = func() {
fmt.Fprint(flag.CommandLine.Output(), `usage: bazel run //tools/releaser -- -go_version <version> -tag <tag>
This utility is intended to handle many of the steps to release a new version.
`)
flag.PrintDefaults()
}
flag.Parse()
if tag == "" {
return fmt.Errorf("tag is required")
}
if !tagRegexp.MatchString(tag) {
return errTag
}
// commands that Must Not Fail
cmds := [][]string{
{"git", "diff", "--stat", "--exit-code"},
{"git", "tag", tag},
}
log("Cutting a release:")
for _, c := range cmds {
cmd := exec.Command(c[0], c[1:]...)
cmd.Dir = repoRoot
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf(
"run %s: %w\n%s",
strings.Join(c, " "),
err,
out,
)
}
}
fpath := path.Join(repoRoot, fmt.Sprintf("hermetic_cc_toolchain-%s.tar.gz", tag))
tgz, err := os.Create(fpath)
if err != nil {
return err
}
defer func() {
if _err != nil {
os.Remove(fpath)
}
}()
hashw := sha256.New()
gzw, err := gzip.NewWriterLevel(io.MultiWriter(tgz, hashw), gzip.BestCompression)
if err != nil {
return fmt.Errorf("create gzip writer: %w", err)
}
log("- creating %s", fpath)
var stderr bytes.Buffer
cmd := exec.Command(
"git",
append([]string{
"archive",
"--format=tar",
// WORKSPACE in the resulting tarball needs to be much
// smaller than of hermetic_cc_toolchain. See #15.
"--add-file=tools/releaser/WORKSPACE",
tag,
}, _paths...)...,
)
cmd.Dir = repoRoot
cmd.Stdout = gzw
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
var exitError *exec.ExitError
errors.As(err, &exitError)
return fmt.Errorf("create git archive: %w\n%s", err, stderr.Bytes())
}
if err := gzw.Close(); err != nil {
return fmt.Errorf("close gzip stream: %w", err)
}
if err := tgz.Close(); err != nil {
return err
}
log("- wrote %s", fpath)
log("Release:\n-----\n" + genBoilerplate(tag, fmt.Sprintf("%x", hashw.Sum(nil))))
return nil
}
func genBoilerplate(version, shasum string) string {
return fmt.Sprintf(`load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "hermetic_cc_toolchain",
sha256 = "%[2]s",
urls = [
"https://mirror.bazel.build/github.com/uber/hermetic_cc_toolchain/releases/download/%[1]s/hermetic_cc_toolchain-%[1]s.tar.gz",
"https://github.com/uber/hermetic_cc_toolchain/releases/download/%[1]s/hermetic_cc_toolchain-%[1]s.tar.gz",
],
)
load("@hermetic_cc_toolchain//toolchain:defs.bzl", zig_toolchains = "toolchains")
# plain zig_toolchains() will pick reasonable defaults. See
# toolchain/defs.bzl:toolchains on how to change the Zig SDK path and version.
zig_toolchains()`, version, shasum)
}

View File

@ -0,0 +1,36 @@
// Copyright 2023 Uber Technologies, Inc.
// Licensed under the MIT License
package main
import (
"fmt"
"testing"
)
func TestRegex(t *testing.T) {
tests := []struct {
tag string
good bool
}{
{good: true, tag: "v1.0.0"},
{good: true, tag: "v99.99.99"},
{good: true, tag: "v1.0.1-rc1"},
{good: true, tag: "v1.0.99-rc99"},
{good: false, tag: ""},
{good: false, tag: "v1.0"},
{good: false, tag: "1.0.0"},
{good: false, tag: "1.0.99-rc99"},
}
for _, tt := range tests {
t.Run(fmt.Sprintf("tag=%s good=%s", tt.tag, tt.good), func(t *testing.T) {
matched := tagRegexp.MatchString(tt.tag)
if tt.good && !matched {
t.Errorf("expected %s to be a valid tag, but it was not", tt.tag)
} else if !tt.good && matched {
t.Errorf("expected %s to be an invalida tag, but it was", tt.tag)
}
})
}
}