* add Module instances for each package's build.zig and attach it to the
dependencies.zig module with the hash digest hex string as the name.
* fix incorrectly skipping the wrong packages for creating
dependencies.zig
* a couple more renaming of "package" to "module"
Only problem is that it looks like `has_build_zig` is being false when
it should be true.
After that is fixed then main.zig needs to create the `@dependencies`
module from the generated source code.
Finish the work started in 4c4fb839972f66f55aa44fc0aca5f80b0608c731.
Now the compiler compiles again.
Wire up dependency tree fetching code in the CLI for `zig build`.
Everything is hooked up except for `createDependenciesModule` is not yet
implemented.
* start renaming "package" to "module" (see #14307)
- build system gains `main_mod_path` and `main_pkg_path` is still
there but it is deprecated.
* eliminate the object-oriented memory management style of what was
previously `*Package`. Now it is `*Package.Module` and all pointers
point to externally managed memory.
* fixes to get the new Fetch.zig code working. The previous commit was
work-in-progress. There are still two commented out code paths, the
one that leads to `Compilation.create` and the one for `zig build`
that fetches the entire dependency tree and creates the required
modules for the build runner.
Organize everything around a Fetch task which does a bunch of stuff in a
worker thread without touching any shared state, and then queues up
Fetch tasks for its dependencies.
This isn't the theoretical optimal package fetching performance because
CPU cores don't necessarily map 1:1 with I/O tasks, and each fetch task
contains a mixture of computations and I/O. However, it is expected for
this to significantly outperform master branch, which fetches everything
recursively with only one thread.
The logic is now a lot more linear and easy to follow. Everything that
is embarassingly parallel is done on the thread pool, and then after
everything is fetched, the worker threads are joined and the main thread
does the finishing touches of stitching together the dependencies.zig
import files. There is only one tiny little critical section and it does
not even have any error handling in it.
This also lays the groundwork for #14281 because in system mode, all
this fetching logic will be skipped, but the "finishing touches"
mentioned above still need to be done. With this branch, that logic is
separated out and no longer recursively tangled with fetching stuff.
Additionally, this branch:
* Implements inclusion directives in `build.zig.zon` for deciding which
files belong the package (#14311).
* Adds basic documentation for `build.zig.zon` files.
* Adds support for fetching dependencies with the `file://` protocol
scheme (#17364).
* Adds a workaround for a Linux/btrfs file system bug (#17282).
This commit is a work-in-progress. Still todo:
1. Hook up the CLI to the new system.
2. Restore the module table creation logic after all the fetching is
done.
3. Fix compilation errors, get the tests passing, and regression test
against real world projects.
Until now, we would pass `candidate: NativeTargetInfo` which creates
a copy of the `NativeTargetInfo.DynamicLinker` buffer. We would then
return this buffer in `bad_dl: []const u8` which would goes out-of-scope
the moment we leave this function frame yielding garbage. To fix this,
we just need to remember to pass by const-pointer
`candidate: *const NativeTargetInfo`.
zig fetch [options] <url>
zig fetch [options] <path>
Fetches a package which is found at <url> or <path> into the global
cache directory, printing the package hash to stdout.
Closes#16972
Related to #14280
Additionally, this commit:
* Adds uncompressed .tar support to package fetching
* Introduces symlink support to package fetching
This commit introduces `--debug-incremental` so that we can start
playing around with incremental compilation while it is still being
developed, and before it is enabled by default.
Currently it saves InternPool data, and has TODO comments for the
remaining things. Deserialization is not implemented yet, which will
require some post-processing such as to build a string map out of
null-terminated string table bytes.
The saved compiler state is stored in a file called <root-name>.zcs
alongside <root-name>.o, <root-name>.pdb, <root-name>.exe, etc. In case
of using the zig build system, these files are all in a zig-cache
directory.
For the self-hosted compiler, here is one data point on the performance
penalty of saving this data:
```
Benchmark 1 (3 runs): zig build-exe ...
measurement mean ± σ min … max outliers delta
wall_time 51.1s ± 354ms 50.7s … 51.4s 0 ( 0%) 0%
peak_rss 3.91GB ± 354KB 3.91GB … 3.91GB 0 ( 0%) 0%
cpu_cycles 212G ± 3.17G 210G … 216G 0 ( 0%) 0%
instructions 274G ± 57.5M 274G … 275G 0 ( 0%) 0%
cache_references 13.1G ± 97.6M 13.0G … 13.2G 0 ( 0%) 0%
cache_misses 1.12G ± 24.6M 1.10G … 1.15G 0 ( 0%) 0%
branch_misses 1.53G ± 1.46M 1.53G … 1.53G 0 ( 0%) 0%
Benchmark 2 (3 runs): zig build-exe ... --debug-incremental
measurement mean ± σ min … max outliers delta
wall_time 51.8s ± 271ms 51.5s … 52.1s 0 ( 0%) + 1.3% ± 1.4%
peak_rss 3.91GB ± 317KB 3.91GB … 3.91GB 0 ( 0%) - 0.0% ± 0.0%
cpu_cycles 213G ± 398M 212G … 213G 0 ( 0%) + 0.3% ± 2.4%
instructions 275G ± 79.1M 275G … 275G 0 ( 0%) + 0.1% ± 0.1%
cache_references 13.1G ± 26.9M 13.0G … 13.1G 0 ( 0%) - 0.1% ± 1.2%
cache_misses 1.12G ± 5.66M 1.11G … 1.12G 0 ( 0%) - 0.6% ± 3.6%
branch_misses 1.53G ± 1.75M 1.53G … 1.54G 0 ( 0%) + 0.2% ± 0.2%
```
At the end of each compilation with `--debug-incremental`, we end up
with a 43 MiB `zig.zcs` file that contains all of the InternPool data
serialized.
Of course, it will necessarily be more expensive to save the state than
to not save the state. However, this data point shows just how cheap the
save state operation is, with all of the groundwork laid for using a
serialization-friendly in-memory data layout.
When compiling for *-windows-msvc, find the native libc_installation and
add the lib dirs to lib_dirs, so that system libs can be found.
Previously, `version` and `ole32` were detected via the mingw.libExists logic,
even on .msvc, which was a false positive. This detection logic for mingw doesn't
find uuid.lib, which was the failure that triggered this bugfix.
Only build the issue_5825 test if the native target is x86_64-windows-msvc,
since it requires the .msvc abi.
The include directories used when preprocessing .rc files are now separate from the target, and by default will use the system MSVC include paths if the MSVC + Windows SDK are present, otherwise it will fall back to the MinGW includes distributed with Zig. This default behavior can be overridden by the `-rcincludes` option (possible values: any (the default), msvc, gnu, or none).
This behavior is useful because Windows resource files may `#include` files that only exist with in the MSVC include dirs (e.g. in `<MSVC install directory>/atlmfc/include` which can contain other .rc files, images, icons, cursors, etc). So, by defaulting to the `any` behavior (MSVC if present, MinGW fallback), users will by default get behavior that is most-likely-to-work.
It also should be okay that the include directories used when compiling .rc files differ from the include directories used when compiling the main binary, since the .res format is not dependent on anything ABI-related. The only relevant differences would be things like `#define` constants being different values in the MinGW headers vs the MSVC headers, but any such differences would likely be a MinGW bug.
The new `@depedencies` module contains generated code like the
following (where strings like "abc123" represent hashes):
```zig
pub const root_deps = [_]struct { []const u8, []const u8 }{
.{ "foo", "abc123" },
};
pub const packages = struct {
pub const abc123 = struct {
pub const build_root = "/home/mlugg/.cache/zig/blah/abc123";
pub const build_zig = @import("abc123");
pub const deps = [_]struct { []const u8, []const u8 }{
.{ "bar", "abc123" },
.{ "name", "ghi789" },
};
};
};
```
Each package contains a build root string, the build.zig import, and a
mapping from dependency names to package hashes. There is also such a
mapping for the root package dependencies.
In theory, we could now remove the `dep_prefix` field from `std.Build`,
since its main purpose is now handled differently. I believe this is a
desirable goal, as it doesn't really make sense to assign a single FQN
to any package (because it may appear in many different places in the
package hierarchy). This commit does not remove that field, as it's used
non-trivially in a few places in the build runner and compiler tests:
this will be a future enhancement.
Resolves: #16354Resolves: #17135
Regressed in 2006add849.
References to native_darwin_sdk are no longer kept in the frontend.
Instead the darwin SDK is detected as part of NativePaths and as part of
LibCInstallation.
The `null` value here was missed in
0a4d4eb252a73555a568a532902951a13284d2ef. I hope it is the cause of the
CI failures on Windows.
The fact that libc++ depends on libc is not important for the CLI and
Compilation.create already handles that logic.
chicken out and make -l match the status quo behavior, where it looks
for dynamic libraries and then falls back to static libraries.
library resolution is still done in the CLI now though, and these
options are added:
-search_static_first Search for static libs in all library search
paths, then dynamic libs.
-search_dylibs_only Only search for dynamic libs.
-search_static_only Only search for static libs.
this matches the already existing options below:
-search_paths_first For each library search path, check for dynamic
lib then static lib before proceeding to next path.
-search_dylibs_first Search for dynamic libs in all library search
So, it is still possible to get the strict behavior by passing
`-search_dylibs_only` or `-search_static_only`.
This commit also makes -dynamic and -static affect the preferred link
mode and search strategy.
