The old logic only decremented `remaining_prelink_tasks` if `bin_file`
was not `null`. This meant that on `-fno-emit-bin` builds with
registered prelink tasks (e.g. C source files), we exited from
`Compilation.performAllTheWorkInner` early, assuming a prelink error.
Instead, when `bin_file` is `null`, we still decrement
`remaining_prelink_tasks`; we just don't do any actual work.
Resolves: #22682
When determining the type of RC compiler, meson passes `/?` or `--version` and then reads from `stdout` looking for particular string(s) anywhere in the output.
So, by adding the string "Microsoft Resource Compiler" to the `/?` output, meson will recognize `zig rc` as rc.exe and give it the correct options, which works fine since `zig rc` is drop-in CLI compatible with rc.exe.
This allows using `zig rc` with meson for (cross-)compiling, by either:
- Setting WINDRES="zig rc" or putting windres = ['zig', 'rc'] in the cross-file
+ This will work like rc.exe, so it will output .res files. This will only link successfully if you are using a linker that can do .res -> .obj conversion (so something like zig cc, MSVC, lld)
- Setting WINDRES="zig rc /:output-format coff" or putting windres = ['zig', 'rc', '/:output-format', 'coff'] in the cross-file
+ This will make meson pass flags as if it were rc.exe, but it will cause the resulting .res file to actually be a COFF object file, meaning it will work with any linker that handles COFF object files
Example cross file that uses `zig cc` (which can link `.res` files, so `/:output-format coff` is not necessary) and `zig rc`:
```
[binaries]
c = ['zig', 'cc', '--target=x86_64-windows-gnu']
windres = ['zig', 'rc']
[target_machine]
system = 'windows'
cpu_family = 'x86_64'
cpu = 'x86_64'
endian = 'little'
```
LLD expects the library file name (minus extension) to be exactly libmingw32. By
calling it mingw32 previously, we prevented it from being detected as being in
LLD's list of libraries that are excluded from the MinGW-specific auto-export
mechanism.
b9d27ac252/lld/COFF/MinGW.cpp (L30-L56)
As a result, a DLL built for *-windows-gnu with Zig would export a bunch of
internal MinGW symbols. This sometimes worked out fine, but it could break at
link or run time when linking an EXE with a DLL, where both are targeting
*-windows-gnu and thus linking separate copies of mingw32.lib. In #23204, this
manifested as the linker getting confused about _gnu_exception_handler() because
it was incorrectly exported by the DLL while also being defined in the
mingw32.lib that was being linked into the EXE.
Closes#23204.
This commit reworks how Sema handles arithmetic on comptime-known
values, fixing many bugs in the process.
The general pattern is that arithmetic on comptime-known values is now
handled by the new namespace `Sema.arith`. Functions handling comptime
arithmetic no longer live on `Value`; this is because some of them can
emit compile errors, so some *can't* go on `Value`. Only semantic
analysis should really be doing arithmetic on `Value`s anyway, so it
makes sense for it to integrate more tightly with `Sema`.
This commit also implements more coherent rules surrounding how
`undefined` interacts with comptime and mixed-comptime-runtime
arithmetic. The rules are as follows.
* If an operation cannot trigger Illegal Behavior, and any operand is
`undefined`, the result is `undefined`. This includes operations like
`0 *| undef`, where the LHS logically *could* be used to determine a
defined result. This is partly to simplify the language, but mostly to
permit codegen backends to represent `undefined` values as completely
invalid states.
* If an operation *can* trigger Illegal Behvaior, and any operand is
`undefined`, then Illegal Behavior results. This occurs even if the
operand in question isn't the one that "decides" illegal behavior; for
instance, `undef / 1` is undefined. This is for the same reasons as
described above.
* An operation which would trigger Illegal Behavior, when evaluated at
comptime, instead triggers a compile error. Additionally, if one
operand is comptime-known undef, such that the other (runtime-known)
operand isn't needed to determine that Illegal Behavior would occur,
the compile error is triggered.
* The only situation in which an operation with one comptime-known
operand has a comptime-known result is if that operand is undefined,
in which case the result is either undefined or a compile error per
the above rules. This could potentially be loosened in future (for
instance, `0 * rt` could be comptime-known 0 with a runtime assertion
that `rt` is not undefined), but at least for now, defining it more
conservatively simplifies the language and allows us to easily change
this in future if desired.
This commit fixes many bugs regarding the handling of `undefined`,
particularly in vectors. Along with a collection of smaller tests, two
very large test cases are added to check arithmetic on `undefined`.
The operations which have been rewritten in this PR are:
* `+`, `+%`, `+|`, `@addWithOverflow`
* `-`, `-%`, `-|`, `@subWithOverflow`
* `*`, `*%`, `*|`, `@mulWithOverflow`
* `/`, `@divFloor`, `@divTrunc`, `@divExact`
* `%`, `@rem`, `@mod`
Other arithmetic operations are currently unchanged.
Resolves: #22743Resolves: #22745Resolves: #22748Resolves: #22749Resolves: #22914
The code did one useless thing and two wrong things:
- ref counting was basically a noop
- last_dir_fd was chosen from the wrong index and also under the wrong
condition
This caused regular crashes on macOS which are now gone.
On updates with failed files, we should refrain from doing any semantic
analysis, or even touching codegen/link. That way, incremental
compilation state is untouched for when the user fixes the AstGen
errors.
Resolves: #23205
* use `tmp.dir.realpathAlloc()` to get full path into tmpDir instances
* use `testing.allocator` where that simplifies things (vs. manual ArenaAllocator for 1 or 2 allocs)
* Trust `TmpDir.cleanup()` to clean up contained files and sub-trees
* Remove some unnecessary absolute paths (enabling WASI to run the tests)
* Drop some no-longer necessary `[_][]const u8` casts
* Add scopes to reduce `var` usage in favor of `const`
This reverts commit 7e0c25eccd.
The `--git-dir` argument is relative to the `-C` argument, making this
patch OK after all.
I added a comment to go along with this since I found it confusing.
Apologies for the revert.
Sometimes Zig is built not from a git repository (e.g. from tarball), but inside another git repository (e.g. distro package repository). Make sure that the version check tries to parse a tag of Zig, and not of a parent directory.
This should be a lot easier to maintain. It's also a small step towards
eventually making the builder API parse the data layout string in order to
answer layout questions that we need to ask during code generation.
Clang's integrated Arm assembler doesn't understand -mabi yet, so this results
in "unused command line argument" warnings when building musl code and glibc
stubs, for example.
This commits adds the following distinct integer types to std.zig.Ast:
- OptionalTokenIndex
- TokenOffset
- OptionalTokenOffset
- Node.OptionalIndex
- Node.Offset
- Node.OptionalOffset
The `Node.Index` type has also been converted to a distinct type while
`TokenIndex` remains unchanged.
`Ast.Node.Data` has also been changed to a (untagged) union to provide
safety checks.
This function checks for various possibilities that are never produced
by the parser.
Given that lastToken is unsafe to call on an Ast with errors, I also
removed code paths that would be reachable on an Ast with errors.
The bitcode format always uses little endian words. Prior to this commit, a
bitcode file produced on e.g. aarch64_be or s390x would fail to be loaded by
LLVM.
Windows is a ridiculous operating system designed by toddlers, and so
requires us to close all file handles in the `tmp/xxxxxxx` cache dir
before renaming it into `o/xxxxxxx`. We have a hack in place to handle
this for the main output file, but the MachO linker also outputs a file
with debug symbols, and we weren't closing it! This led to a fuckton of
CI failures when we enabled `.whole` cache mode by default for
self-hosted backends.
thanks jacob for figuring this out while i sat there
The Bernstein-Yang inversion code was meant to be used only with the
fields we currently use for the NIST curves.
But people copied that code and were confused that it didn't work as
expected with other field sizes.
It doesn't cost anything to make it work with other field sizes,
that may support in the future. So let's do it.
This also reduces the diff with the example zig code in fiat crypto.
Suggested by @Rexicon226 -- Thank you!
Simliarly to shl_with_overflow, we first SHL/SAL the integer, then
SHR/SAR it back to compare if overflow happens.
If overflow happened, set result to the upper limit to make it saturating.
Bug: #17645
Co-authored-by: Jacob Young <jacobly0@users.noreply.github.com>
Signed-off-by: Bingwu Zhang <xtex@aosc.io>
This reverts commit dea72d15da, reversing
changes made to ab381933c8.
The changeset does not work as advertised and does not have sufficient
test coverage.
Reopens#22822
mainly this addresses the following use case:
1. Someone creates a template with build.zig.zon, id field included
(note that zig init does not create this problem since it generates
fresh id every time it runs).
2. User A uses the template, changing package name to "example" but not
id field.
3. User B uses the same template, changing package name also to
"example", also not changing the id field.
Here, both packages have unintentional conflicting logical ids.
By making the field a combination of name checksum + random id, this
accident is avoided. "nonce" is an OK name for this.
Also relaxes errors on remote packages when using `zig fetch`.
Introduces the `id` field to `build.zig.zon`.
Together with name, this represents a globally unique package
identifier. This field should be initialized with a 16-bit random number
when the package is first created, and then *never change*. This allows
Zig to unambiguously detect when one package is an updated version of
another.
When forking a Zig project, this id should be regenerated with a new
random number if the upstream project is still maintained. Otherwise,
the fork is *hostile*, attempting to take control over the original
project's identity.
`0x0000` is invalid because it obviously means a random number wasn't
used.
`0xffff` is reserved to represent "naked" packages.
Tracking issue #14288
Additionally:
* Fix bad path in error messages regarding build.zig.zon file.
* Manifest validates that `name` and `version` field of build.zig.zon
are maximum 32 bytes.
* Introduce error for root package to not switch to enum literal for
name.
* Introduce error for root package to omit `id`.
* Update init template to generate `id`
* Update init template to populate `minimum_zig_version`.
* New package hash format changes:
- name and version limited to 32 bytes via error rather than truncation
- truncate sha256 to 192 bits rather than 40 bits
- include the package id
This means that, given only the package hashes for a complete dependency
tree, it is possible to perform version selection and know the final
size on disk, without doing any fetching whatsoever. This prevents
wasted bandwidth since package versions not selected do not need to be
fetched.
This branch regressed from master by switching to binary rather than hex
digest, allowing null bytes to end up in identifiers in the zig file.
This commit fixes it by changing the "hash" to be literally equal to the
sub_path (with a prefix '/' to indicate "global") if it can fit. If it
is too long then it is actually hashed, and that value used instead.
Inheriting allow-deprecation from parent modules doesn't make too much
sense, so instead make them default to disallow unless otherwise
specified. This allows build system to avoid redundant
`-fno-allow-deprecated` args.
This makes the generated CLIs smaller, and makes zig1.wasm update not
needed.
Also represented `is_root` differently (moved to field of graph).
Problem here is if zig is asked to create multiple static libraries, it
will build the runtime multiple times and then they will conflict.
Instead we want to build the runtime exactly once.
Unlike `compiler-rt`, `ubsan` uses the standard library quite a lot.
Using a similar approach to how `compiler-rt` is handled today, where it's
compiled into its own object and then linked would be sub-optimal as we'd
be introducing a lot of code bloat.
This approach always "imports" `ubsan` if the ZCU, if it exists. If it doesn't
such as the case where we're compiling only C code, then we have no choice other
than to compile it down to an object and link. There's still a tiny optimization
we can do in that case, which is when compiling to a static library, there's no
need to construct an archive with a single object. We'd only go back and parse out
ubsan from the archive later in the pipeline. So we compile it to an object instead
and link that to the static library.
TLDR;
- `zig build-exe foo.c` -> build `libubsan.a` and links
- `zig build-obj foo.c` -> doesn't build anything, just emits references to ubsan runtime
- `zig build-lib foo.c -static` -> build `ubsan.o` and link it
- `zig build-exe foo.zig bar.c` -> import `ubsan-rt` into the ZCU
- `zig build-obj foo.zig bar.c` -> import `ubsan-rt` into the ZCU
- `zig build-lib foo.zig bar.c` -> import `ubsan-rt` into the ZCU
Instead of hardcoding a call to defaultRandomSeed() use the customizable
std.options.cryptoRandomSeed() like in the rest of the function.
Closes#19943.
In #22522 I said:
> RC="zig rc" will now work in combination with zig cc and CMake. Here's an example of cross-compiling a simple Windows GUI CMake project
>
> $ RC="zig rc" CC="zig cc --target=x86_64-windows-gnu" cmake .. -DCMAKE_SYSTEM_NAME=Windows -G Ninja
However, I didn't realize that the time that this only works because of the `-G Ninja` part. When not using Ninja as the build tool, CMake adds a workaround for 'very long lists of object files' where it takes all object files and runs them through `ar` to combine them into one archive:
4a11fd8dde/Modules/Platform/Windows-GNU.cmake (L141-L158)
This is a problem for the Windows resource use-case, because `ar` doesn't know how to deal with `.res` files and so this object combining step fails with:
unknown file type: foo.rc.res
Only the linker knows what to do with .res files (since it has its own `.res` -> `.obj` ('cvtres') conversion mechanism). So, when using Ninja, this object file combining step is skipped and the .res file gets passed to the linker and everyone is happy.
Note: When CMake thinks that its using `windres` as the Windows resource compiler, it will pass `-O coff` to windres which causes it to output a COFF object file instead of a `.res` file, which means that the `ar` step can succeed because it's only working on actual object files.
---
This commit gives `zig rc` the ability to output COFF object files directly when `/:output-format coff` is provided as an argument. This effectively matches what happens when CMake uses `windres` for resource compilation, but requires the argument to be provided explicitly.
So, after this change, the following CMake cross-compilation use case will work, even when not using Ninja as the generator:
RC="zig rc /:output-format coff" CC="zig cc --target=x86_64-windows-gnu" cmake .. -DCMAKE_SYSTEM_NAME=Windows
These were previously incremental tests, so weren't running. They didn't
*need* to be incremental. They worked under the old runner because of
how it directly integrated with the compiler so tracked error messages
differently.
Updated solution is future proof for arbitary size integer handling for both strategies .br_table lowering if switch case is dense, .br_if base jump table if values are too sparse.
This is a correctness issue: When -fno-builtin is used, we must assume that we
could be compiling the memcpy/memset implementations, so generating calls to
them is problematic.
This can also be extended to ELF later as it means roughly the same thing there.
This addresses the main issue in #21721 but as I don't have a macOS machine to
do further testing on, I can't confirm whether zig cc is able to pass the entire
cgo test suite after this commit. It can, however, cross-compile a basic program
that uses cgo to x86_64-macos-none which previously failed due to lack of -x
support. Unlike previously, the resulting symbol table does not contain local
symbols (such as C static functions).
I believe this satisfies the related donor bounty: https://ziglang.org/news/second-donor-bounty
* Adds startTupleField/startStructField, makes pattern in print targets less verbose
* Makes some enums into strings
* Start/finish renamed to begin/end
I feel bad changing this, but I don't know why I named them this way in the first place.
Begin/end is consistent with the json API, and with other APIs in the wild that follow this pattern.
Better to change now than later.
When runtime safety is turned on, `Ed25519.fromSecretKey()` can
currently hit an assertion if the format of the secret key is
invalid.
Return an error instead, so that applications can recover.
Unfortunately some duplicate files must remain in lib/libc/wasi/libc-top-half
because they include internal headers *in the same directory* which have edits
relative to upstream musl. Because C is an amazing language, there is no way to
make it so that e.g. upstream musl's src/stdio/fputc.c includes wasi-libc's
src/stdio/putc.h instead of the upstream putc.h. The preprocessor always
searches the current directory first for quote includes.
Anyway, this still takes us from 2.9M to 1.4M for the combination of
lib/libc/wasi and lib/libc/include/wasm-wasi-musl, so I still call it a win.
`EACCES` is returned if the file mode bit (i.e., user/group/other rwx
bits) disallow access. `EPERM` is returned if something else denies
access (immutable bit, SELinux, capabilities, etc). This somewhat subtle
no-access distinction is part of POSIX. For now map both to
`error.PermissionDenied` to keep the error signature unchanged. See
duopoly.
This PR is effecitvely an update/simplification of PR #19193.
Tested locally with an immutable file.
Fixes#22733 and #19162.
- allow `-fsanitize-coverage-trace-pc-guard` to be used on its own without enabling the fuzzer.
(note that previouly, while the flag was only active when fuzzing, the fuzzer itself doesn't use it, and the code will not link as is.)
- add stub functions in the fuzzer to link with instrumented C code (previously fuzzed tests failed to link if they were calling into C):
while the zig compile unit uses a custom `EmitOptions.Coverage` with features disabled,
the C code is built calling into the clang driver with "-fsanitize=fuzzer-no-link" that automatically enables the default features.
(see de06978ebc/clang/lib/Driver/SanitizerArgs.cpp (L587))
- emit `-fsanitize-coverage=trace-pc-guard` instead of `-Xclang -fsanitize-coverage-trace-pc-guard` so that edge coverrage is enabled by clang driver. (previously, it was enabled only because the fuzzer was)
readAtLeast is greedy and will read the entire length of the buffer if it can. However, reading past the end of the cert in this case is useless, so reading the full length of the buffer just puts an increasingly large (due to the growth algorithm of ArrayList) collection of wasted bytes after each cert in cb.bytes.
In practical terms, this ends up saving potentially millions of bytes of wasted reads/allocations. In my testing, after reading the keychain files on my machine, cb.bytes ends up with these capacities:
- Before: cb.bytes.capacity = 32720747
- After: cb.bytes.capacity = 251937
That's a decrease of 99.2%
Additionally, swaps to readNoEof since it should be an error to hit EOF without reading the full cert size.
On my machine, the defaults are 5 seconds (LLDB) and 2 seconds (GDB). These are
too low on the CI machines during high load, and the CI system itself already
enforces a timeout on jobs anyway, so just disable the timeout altogether.
* std.crypto: add the ability to explicitly tag a value as secret
It turns out that Valgrind can be a very useful tool to check that secrets
are not leaked via side channels involving lookups or conditional jumps.
Valgrind tracks uninitialized data, and memcheck reports operations
involving uninitialized values. By permanently or temporarily telling
Valgrind that a memory region containing secrets is uninitialized, we can
detect common side-channel vulnerabilities.
For example, the following code snippets would immediately report that the
result is not computed in constant time:
```zig
classify(&key);
const len = std.mem.indexOfScalar(u8, &key, 0);
```
```zig
classify(&key);
const idx = key[0];
x += idx;
```
```zig
var x: [4]u8 = undefined;
std.crypto.random.bytes(&x);
classify(&x);
if (std.mem.eql(u8, "test", &x)) return;
```
This is not fool-proof, but it can help a lot to detect unwanted compiler
optimizations.
Also, right now, this is relying on Valgrind primitives, but these
annotations can be used to do more interesting things later, especially with
our own code generation backends.
* Update for Zig 0.14
* Remove checks for Valgrind enablement
Four tests in lib/std/posix/test.zig were disabled because they created
fixed-name files in the current working directory, and this caused
problems if tests were running in parallel with other build's tests.
This PR fixes those tests to all use `std.testing.tmpDir` to create unique
temporary names and directories.
Also clean the tests up to more consistently use `defer` to clean up, or
to just rely on tmpDir cleanup to remove individual files.
Working on these tests revealed a bunch of stale WASI code paths in
posix.zig, fixed by replacing stale `wast.AT.FDCWD` references with just
`AT.FDCWD`.
Fixes#14968.
* bcrypt: make silently_truncate_password a member of Params
This removes the need for having both `bcrypt()` and
`bcryptWithTruncation()` in the public API.
And whether truncation happens or not becomes even more explicit.
* Update crypto benchmark
In the original PR that implemented this (https://github.com/ziglang/zig/pull/14325), it included a list of references for the keychain format. Multiple of those references include the checks that are added in this commit, and empirically this fixes the loading of a real keychain file that was previously failing (it had both a record with offset 0 and a record with cert_size 0).
Fixes#22870
Functions like isMinGW() and isGnuLibC() have a good reason to exist: They look
at multiple components of the target. But functions like isWasm(), isDarwin(),
isGnu(), etc only exist to save 4-8 characters. I don't think this is a good
enough reason to keep them, especially given that:
* It's not immediately obvious to a reader whether target.isDarwin() means the
same thing as target.os.tag.isDarwin() precisely because isMinGW() and similar
functions *do* look at multiple components.
* It's not clear where we would draw the line. The logical conclusion before
this commit would be to also wrap Arch.isX86(), Os.Tag.isSolarish(),
Abi.isOpenHarmony(), etc... this obviously quickly gets out of hand.
* It's nice to just have a single correct way of doing something.
* arm_apcs is the long dead "OABI" which we never had working support for.
* arm_aapcs16_vfp is for arm-watchos-none which is a dead target that we've
dropped support for.
In the MAC finalization function, concatenated tags at odd positions
were not absorbed into the correct lane.
Spotted by a Tigerbeetle regression test and reported by Rafael Batiati
(@batiati) — Thanks!
The check is not needed, since we are already checking for the os
at line 847 and returning at 916 when the check succeeds.
Therefore, at 926, we know the os is not windows.
Using the browser's `console.error`, etc. functions instead of `console.log` produces prettier output in the console. Additionally, `console.error` in particular includes a stack trace, which is useful for debugging where the error occurred.
Additionally, this commit leverages the enhanced logging to delete the separate `panic` function from the JS code and write it in Zig instead.
Additionally, this commit streamlines the way unparseable files are handled, by giving them the AST of an empty file. This avoids bugs in the rest of the Autodoc logic trying to work with invalid ASTs.
I observed a stack overflow during x86_64 CodeGen in a debug compiler
compiled by the llvm backend. This happens while compiling
`main.buildOutputType` due to the Air being nested almost 500 levels.
This is all of the expected 0.14.0 progress on #21530, which can now be
postponed once this commit is merged.
This required rewriting the (un)wrap operations since the original
implementations were extremely buggy.
Also adds an easy way to retrigger Sema OPV bugs so that I don't have to
keep updating #22419 all the time.
The current zig fetch help docs tell the user to specify a package's URL, but it's unclear what the URL should be.
This change expands the help output to explain what URLs the zig fetch command can handle and provides examples of valid URLs.
Related: #20096
A git bundle file seems to be the more accurate term, as it's what git uses in its documentation: https://git-scm.com/docs/git-bundle
When using the LLVM backend, array copies were lowered as calls to
`llvm.memcpy.*` builtin which could cause recursive calls to memcpy to
be generated (observed with `-target x86_64-linux -mcpu x86_64+avx512vl
--debug-rt`).
By instead performing these small fixed-size copies with integers or
vectors the LLVM backend does not generate calls to the `llvm.memcpy`
builtin, and so (with `-fno-builtin`) recursive calls to memcpy will
not be generated by LLVM.
The assertions and (test build) runtime safety have been removed as they
may cause (mutually) recursive calls to memcpy in debug builds since the
panic handler generates calls to llvm.memcpy.
breaking change to the fuzz testing API; it now passes a type-safe
context parameter to the fuzz function.
libfuzzer is reworked to select inputs from the entire corpus.
I tested that it's roughly as good as it was before in that it can find
the panics in the simple examples, as well as achieve decent coverage on
the tokenizer fuzz test.
however I think the next step here will be figuring out why so many
points of interest are missing from the tokenizer in both Debug and
ReleaseSafe modes.
does not quite close#20803 yet since there are some more important
things to be done, such as opening the previous corpus, continuing
fuzzing after finding bugs, storing the length of the inputs, etc.
Currently zig fails to build while linking the system LLVM/C++ libraries
on my Chimera Linux system due to the fact that libc++.so is a linker
script with the following contents:
INPUT(libc++.so.1 -lc++abi -lunwind)
Prior to this commit, zig would try to convert "ambiguous names" in
linker scripts such as libc++.so.1 in this example into -lfoo style
flags. This fails in this case due to the so version number as zig
checks for exactly the .so suffix.
Furthermore, I do not think that this conversion is semantically correct
since converting libfoo.so to -lfoo could theoretically end up resulting
in libfoo.a getting linked which seems wrong when a different file is
specified in the linker script.
With this patch, this attempted conversion is removed. Instead, zig
always first checks if the exact file/path in the linker script exists
relative to the current working directory.
If the file is classified as a library (including versioned shared
objects such as libfoo.so.1), zig then falls back to checking if
the exact file/path in the linker script exists relative to each
directory in the library search path, selecting the first match or
erroring out if none is found.
This behavior fixes the regression that prevents building zig while
linking the system LLVM/C++ libraries on Chimera Linux.
std.c.NI was never used in the source, so let's finally use it and make
the function more clear!
This is a breaking change, although a minor one: If you previously passed 0 here
(meaning no flags), then now you have to pass an empty struct (.{}) instead.
Otherwise, you probably used @bitCast() shenanigans here (like
@bitCast(c.NI { .NUMERICHOST = true }) and that will still work, but you can
also get rid of the @bitCast() now!
Zig's copy of the `SYMLINK_{NO,}FOLLOW` constants from wasi-musl was
wrong, as were the `IFIFO` and `IFSOCK` file type flags. Fix these up,
and add comments pointing to exactly where they come from (as the
wasi-musl source has lots of unused, different definitions of these
constants).
Add tests for the Zig convention that WASM preopen 3 is the current
working directory. This is true for WASM with or without libc.
Enable several fs and posix tests that are now passing (not necessarily
because of this change) on wasm targets.
Fixes#20890.
Freelist length accounting in alloc had a negative impact, especially
with the integer type bumped up to u16, so I changed the system to be
based on counting slabs rather than total allocations.
* slab length reduced to 64K
* track freelist length with u8s
* on free(), rotate if freelist length exceeds max_freelist_len
Prevents memory leakage in the scenario where one thread only allocates
and another thread only frees.
This ensure capacity call does not match the number of
appendAssumeCapacity() calls that follow it. Fix this.
This was discovered due to hitting the assertion failure in
appendAssumeCapacity() while building river.
I'm not sure how to isolate a minimal reproducer for a test.
The previous language using subsets was really just stating a couple
of the properties of the union of a group of sets, and with a minor
error at that.
Windows-only, depends on kernel32 in violation of zig std lib policy,
and redundant with other cross-platform APIs that perform the same
functionality.
In larger small buckets, the comptime logic that computed slot count did
not verify that the number it produced was valid. Now it verifies it,
which made this bug into a compile error. Then I fixed the bug by
introducing a "minimum slots per bucket" declaration.
Reversal on the decision: the Allocator interface is the correct place
for the memset to undefined because it allows Allocator implementations
to bypass the interface and use a backing allocator directly, skipping
the performance penalty of memsetting the entire allocation, which may
be very large, as well as having valuable zeroes on them.
closes#4298
I don't think these belong in std, at least not in their current form.
If someone wants to add these back I'd like to review the patch before
it lands.
Reverts 629e2e7844
This one changes the size of an allocation, allowing it to be relocated.
However, the implementation will still return `null` if it would be
equivalent to
new = alloc
memcpy(new, old)
free(old)
Mainly this prepares for taking advantage of `mremap` which I thought
would be a bigger deal but apparently is only available on Linux. Still,
we should use it on Linux.
no longer causes compilation failure.
This also addresses the problem of high map count causing OOM by
choosing a page size of 2MiB for most targets when the page_size_max is
smaller than this number.
This allocator now supports alignments greater than page size, with the
same implementation as it used before.
This is a partial revert of ceb0a632cf.
It looks like VirtualAlloc2 has better solutions to this problem,
including features such as MEM_RESERVE_PLACEHOLDER and MEM_LARGE_PAGES.
This possibility can be investigated as a follow-up task.
This keeps the implementation matching master branch, however,
introduces a compile error that applications can work around by
explicitly setting page_size_max and page_size_min to match their
computer's settings, in the case that those values are not already
equal.
I plan to rework this allocator in a follow-up enhancement with the goal
of reducing total active memory mappings.
* fix merge conflicts
* rename the declarations
* reword documentation
* extract FixedBufferAllocator to separate file
* take advantage of locals
* remove the assertion about max alignment in Allocator API, leaving it
Allocator implementation defined
* fix non-inline function call in start logic
The GeneralPurposeAllocator implementation is totally broken because it
uses global state but I didn't address that in this commit.
heap.zig: define new default page sizes
heap.zig: add min/max_page_size and their options
lib/std/c: add miscellaneous declarations
heap.zig: add pageSize() and its options
switch to new page sizes, especially in GPA/stdlib
mem.zig: remove page_size
The spec is ambiguous, and it's too late to change it.
So the most reasonable thing to do in order to avoid generating
strings that could be parsed differently by other implementations
is to forbid parameters named "v" at compile-time.
See https://github.com/P-H-C/phc-string-format/issues/8
Unfortunately, I can't easily add a test for this, because the repro
depends on some details of DWARF layout; but I've confirmed that it
fixes a bug repro on another branch.
Clearing the analysis roots was very clever and all, but not actually
valid. We need to avoid *any* reference to the analysis errors if there
were any fatal files, and that includes sorting the errors!
Resolves: #22774
Sometimes we emit runtime instructions in comptime scopes. These
instructions will be discarded, but they allow comptime blocks to
contain intermediate runtime-known values, which is necessary for
expressions like `runtime_array.len` to work.
Since we will always throw away these runtime instructions, including
safety checks is a time waste at best and trips an assertion at worst!
Resolves: #20064
The changes from a few commits earlier, where semantic analysis no
longer occurs if any Zig files failed to lower to ZIR, mean `file`
dependencies are no longer necessary! However, we now need them for ZON
files, to be invalidated whenever a ZON file changes.
This came with a big cleanup to `Zcu.PerThread.updateFile` (formerly
`astGenFile`).
Also, change how the cache manifest works for files in the import table.
Instead of being added to the manifest when we call `semaFile` on them,
we iterate the import table after running the AstGen workers and add all
the files to the cache manifest then.
The downside is that this is a bit more eager to include files in the
manifest; in particular, files which are imported but not actually
referenced are now included in analysis. So, for instance, modifying any
standard library file will invalidate all Zig compilations using that
standard library, even if they don't use that file.
The original motivation here was simply that the old logic in `semaFile`
didn't translate nicely to ZON. However, it turns out to actually be
necessary for correctness. Because `@import("foo.zig")` is an
AstGen-level error if `foo.zig` does not exist, we need to invalidate
the cache when an imported but unreferenced file is removed to make sure
this error is triggered when it needs to be.
Resolves: #22746
This is mainly in preparation for integrating ZonGen into the pipeline
properly, although these names are better because `astGenFile` isn't
*necessarily* running AstGen; it may determine that the current ZIR is
up-to-date, or load cached ZIR.
Instead, `source`, `tree`, and `zir` should all be optional. This is
precisely what we're actually trying to model here; and `File` isn't
optimized for memory consumption or serializability anyway, so it's fine
to use a couple of extra bytes on actual optionals here.
The ZON PR (#20271) is causing these tests to inexplicably fail. It
doesn't seem like that PR is what's breaking GPA, so these tests are now
disabled. This is tracked by #22731.
This commit allows using ZON (Zig Object Notation) in a few ways.
* `@import` can be used to load ZON at comptime and convert it to a
normal Zig value. In this case, `@import` must have a result type.
* `std.zon.parse` can be used to parse ZON at runtime, akin to the
parsing logic in `std.json`.
* `std.zon.stringify` can be used to convert arbitrary data structures
to ZON at runtime, again akin to `std.json`.
This commit effectively reverts 9e683f0, and hence un-accepts #19777.
While nice in theory, this proposal turned out to have a few problems.
Firstly, supplying a result type implicitly coerces the operand to this
type -- that's the main point of result types! But for `try`, this is
actually a bad idea; we want a redundant `try` to be a compile error,
not to silently coerce the non-error value to an error union. In
practice, this didn't always happen, because the implementation was
buggy anyway; but when it did, it was really quite silly. For instance,
`try try ... try .{ ... }` was an accepted expression, with the inner
initializer being initially coerced to `E!E!...E!T`.
Secondly, the result type inference here didn't play nicely with
`return`. If you write `return try`, the operand would actually receive
a result type of `E!E!T`, since the `return` gave a result type of `E!T`
and the `try` wrapped it in *another* error union. More generally, the
problem here is that `try` doesn't know when it should or shouldn't
nest error unions. This occasionally broke code which looked like it
should work.
So, this commit prevents `try` from propagating result types through to
its operand. A key motivation for the original proposal here was decl
literals; so, as a special case, `try .foo(...)` is still an allowed
syntax form, caught by AstGen and specially lowered. This does open the
doors to allowing other special cases for decl literals in future, such
as `.foo(...) catch ...`, but those proposals are for another time.
Resolves: #21991Resolves: #22633
This check isn't valid in such cases, because the source and destination
pointers both refer to zero bits of memory, meaning they effectively
never alias.
Resolves: #21655
Sema is arbitrarily scalarizing some operations, which means that when I
try to implement vectorized versions of those operations in a backend,
they are impossible to test due to Sema not producing them. Now, I can
implement them and then temporarily enable the new feature for that
backend in order to test them. Once the backend supports all of them,
the feature can be permanently enabled.
This also deletes the Air instructions `int_from_bool` and
`int_from_ptr`, which are just bitcasts with a fixed result type, since
changing `un_op` to `ty_op` takes up the same amount of memory.
- patch authored by Jacob Young
- tested on alpine-aarch64, 3.21.0, qemu-system 9.2.0
- issue manifested on Alpine Linux aarch64 under qemu-system where
zig2 fails during bootstrap: error.ProcessFdQuotaExceeded
This instruction is like `intcast`, but includes two safety checks:
* Checks that the int is in range of the destination type
* If the destination type is an exhaustive enum, checks that the int
is a named enum value
This instruction is locked behind the `safety_checked_instructions`
backend feature; if unsupported, Sema will emit a fallback, as with
other safety-checked instructions.
This instruction is used to add a missing safety check for `@enumFromInt`
truncating bits. This check also has a fallback for backends which do
not yet support `safety_checked_instructions`.
Resolves: #21946
Currently -freference-trace only works when running from a terminal.
This is annoying if you're running in another environment or if you redirect the output.
But -freference-trace also works fine without the color, so change how the build runner is interpreting this option.
This allocator has no purpose since it cannot truly fulfill the role of
page allocation, and std.heap.wasm_allocator is better both in terms of
performance and code size.
This commit redefines `std.heap.page_allocator` to be less strict:
"On operating systems that support memory mapping, this allocator makes
a syscall directly for every allocation and free. Otherwise, it falls
back to the preferred singleton for the target. Thread-safe."
This now matches how it was actually being implemented, and matches its
use sites - which are mainly as the backing allocator for
`std.heap.ArenaAllocator`.
These are system DLLs, most of which MinGW provides .def files for. It just so
happens that MinGW also has some static libraries by the same name which link in
some GUID definitions.
The remaining non-MinGW library names represent libraries that are always
statically linked, so if those are requested by the user, it makes sense to
error if libc is not linked. A future enhancement could be to compile those
independent of mingw32.lib, however.
Closes#22560.
Otherwise the disk just keeps getting filled up.
Also remove some pointless cleanup commands since the actions/checkout workflow
step already cleans the repository by default.
I recently saw a user hit the "comptime call of extern function" error,
and get confused because they didn't know why the scope was `comptime`.
So, use `explainWhyBlockIsComptime` on this and related errors to add
all the relevant notes.
The added test case shows the motivating situation.
* The langspec definition of `@memcpy` has been changed so that the
source and destination element types must be in-memory coercible,
allowing all such calls to be raw copying operations, not actually
applying any coercions.
* Implement aliasing check for comptime `@memcpy`; a compile error will
now be emitted if the arguments alias.
* Implement more efficient comptime `@memcpy` by loading and storing a
whole array at once, similar to how `@memset` is implemented.
This is a stupid Clang-ism:
❯ cat test.c
int main() {
int value = 42;
int const *value_ptr = &value;
int location;
__atomic_store(&location, value_ptr, __ATOMIC_SEQ_CST);
}
❯ gcc test.c -fsyntax-only
❯ clang test.c -fsyntax-only
test.c:5:31: warning: passing 'const int *' to parameter of type 'int *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
5 | __atomic_store(&location, value_ptr, __ATOMIC_SEQ_CST);
| ^~~~~~~~~
1 warning generated.
I have no idea why Clang doesn't define these builtins as taking const pointers
for the parameters that are only read from. Anyway, after the next zig1.wasm
update, this change should shut up these warnings that we've been seeing in CI
during bootstrap for ages.
Turns out this was already fixed in #21964.
I have no idea why GitHub showed an incorrect diff in #21273, or how applying the diff to master was even possible, but here we are.
* compiler-rt and mingw32 have both run into LLD bugs, and LLVM disables LTO for
its compiler-rt, so disable LTO for these.
* While we haven't run into any bugs in it, LLVM disables LTO for its libtsan,
so follow suit just to be safe.
* Allow LTO for libfuzzer as LLVM does.
If this isn't done, LTO can completely miscompile the input bitcode modules for
certain targets where we need to explicitly set these ABIs (because LLVM's
defaults are bad).
The real problem here is that Git for Windows has horrendous defaults
which convert LF to CRLF. However, rather than changing this
configuration on the CI runners, it's worth supporting inexplicable CRLF
in these files so that anyone else cloning Zig on Windows doesn't get
unexpected test failures.
Uses of `@embedFile` register dependencies on the corresponding
`Zcu.EmbedFile`. At the start of every update, we iterate all embedded
files and update them if necessary, and invalidate the dependencies if
they changed.
In order to properly integrate with the lazy analysis model, failed
embed files are now reported by the `AnalUnit` which actually used
`@embedFile`; the filesystem error is stored in the `Zcu.EmbedFile`.
An incremental test is added covering incremental updates to embedded
files, and I have verified locally that dependency invalidation is
working correctly.
and remove faulty assertion. When a prelink task fails, the
completed_prelink_tasks counter will not decrement.
A future improvement will be needed to make the pipeline fully robust
and handle failed prelink tasks, followed by updates in which those
tasks succeed, and compilation proceeds like normal.
Currently if a prelink task fails, the Compilation will be left in a
state unrecoverable by an incremental update.
* `std.builtin.Panic` -> `std.builtin.panic`, because it is a namespace.
* `root.Panic` -> `root.panic` for the same reason. There are type
checks so that we still allow the legacy `pub fn panic` strategy in
the 0.14.0 release.
* `std.debug.SimplePanic` -> `std.debug.simple_panic`, same reason.
* `std.debug.NoPanic` -> `std.debug.no_panic`, same reason.
* `std.debug.FormattedPanic` is now a function `std.debug.FullPanic`
which takes as input a `panicFn` and returns a namespace with all the
panic functions. This handles the incredibly common case of just
wanting to override how the message is printed, whilst keeping nice
formatted panics.
* Remove `std.builtin.panic.messages`; now, every safety panic has its
own function. This reduces binary bloat, as calls to these functions
no longer need to prepare any arguments (aside from the error return
trace).
* Remove some legacy declarations, since a zig1.wasm update has
happened. Most of these were related to the panic handler, but a quick
grep for "zig1" brought up a couple more results too.
Also, add some missing type checks to Sema.
Resolves: #22584
formatted -> full
When using the self-hosted backends, especially in incremental mode, the
.eh_frame_hdr section may be incomplete, so we can't treat it as authoritative.
Instead, if we started out intending to use .eh_frame_hdr but find that it's
incomplete, load .eh_frame/.debug_frame on demand and use that info going
forward.
This moves the default value logic to Package.Module.create() instead and makes
it so that Compilation.Config.any_unwind_tables is computed similarly to
any_sanitize_thread, any_fuzz, etc. It turns out that for any_unwind_tables, we
only actually care if unwind tables are enabled at all, not at what level.
Allows the stack trace tests to be additionally compiled and run with
`.use_llvm = false, .use_lld = false` depending on the host target. This
is currently enabled for x86_64 targets emitting ELF.
Self-hosted backends emit slightly different DWARF info to the LLVM
backend, so the checking logic (and the tests themselves) had to be
tweaked slightly to support both backends at once.
This is more verbose, but at least we now get a compile error instead of UB when
a new feature is added to std.Target.wasm.Feature but not to link.Wasm.Feature.
This will mainly be used when targeting our wasm2c implementation which has no
problem with zero-length bulk memory operations, as a non-standard extension.
I think it would be better if this invisible doc comments is top-level
doc comments rather than doc comments. Because it is at the start of a
source file. This makes the doc comments visible.
However, the last paragraph replaces doc comments with normal comments.
This is because I thought that normal comments would be more
appropriate than top-level doc comments given the context of the last
paragraph.
I think it would be better if this invisible doc comments is top-level
doc comments rather than doc comments. Because it is at the start of a
source file. This makes the doc comments visible.
It's now unnecessary to explicitly pass this, because it receives an
implicit error trace parameter anyway, so can just use
`@errorReturnTrace()`. The previous commit updated Sema to expect this
new interface. This saves an AIR instruction at all `returnError` call
sites.
Acts as a replacement for `addSharedLibrary` and `addStaticLibrary`, but
linking mode can be changed more easily in build.zig, for example:
In library:
```zig
const linkage = b.option(std.builtin.LinkMode, "linkage", "Link mode for a foo_bar library") orelse .static; // or other default
const lib = b.addLibrary(.{
.linkage = linkage,
.name = "foo_bar",
.root_module = mod,
});
```
In consumer:
```zig
const dep_foo_bar = b.dependency("foo_bar", .{
.target = target,
.optimize = optimize,
.linkage = .static // or dynamic
});
mod.linkLibrary(dep_foor_bar.artifact("foo_bar"));
```
It also matches nicely with `linkLibrary` name.
Signed-off-by: Eric Joldasov <bratishkaerik@landless-city.net>
See: https://github.com/WebAssembly/tool-conventions/pull/235
This is not *quite* using the same features as the spec'd lime1 model because
LLVM 19 doesn't have the level of feature granularity that we need for that.
This will be fixed once we upgrade to LLVM 20.
Part of #21818.
As discussed in #21818, generic is a poor baseline model because that model is a
moving target in LLVM. Instead, use mvp, which has no features enabled.
Apparently the WebAssembly spec requires these instructions to trap if the
computed memory access could be out of bounds, even if the length is zero.
Really a rather bizarre design choice.
This check doesn't make sense with the modern Allocator API; it's left over
from when realloc could change alignment. It's statically known (but not
comptime-known) to be true always. This check was one of the things
blocking Allocator from being used at comptime (related: #1291).
It caused an assertion failure when building Zig from source.
This reverts commit 0595feb341, reversing
changes made to 744771d330.
closes#22566closes#22568
Turns out that even modern Debian aarch64 glibc libc_nonshared.a has
references to _init, meaning that the previous commit caused a
regression when trying to build any -lc executable on that target.
This commit backs out the changes to LibCInstallation.
There is still a fork in the road coming up when the self-hosted ELF
linker becomes load bearing on that target.
crti.o/crtn.o is a legacy strategy for calling constructor functions
upon object loading that has been superseded by the
init_array/fini_array mechanism.
Zig code depends on neither, since the language intentionally has no way
to initialize data at runtime, but alas the Zig linker still must
support this feature since popular languages depend on it.
Anyway, the way it works is that crti.o has the machine code prelude of
two functions called _init and _fini, each in their own section with the
respective name. crtn.o has the machine code instructions comprising the
exitlude for each function. In between, objects use the .init and .fini
link section to populate the function body.
This function is then expected to be called upon object initialization
and deinitialization.
This mechanism is depended on by libc, for example musl and glibc, but
only for older ISAs. By the time the libcs gained support for newer
ISAs, they had moved on to the init_array/fini_array mechanism instead.
For the Zig linker, we are trying to move the linker towards
order-independent objects which is incompatible with the legacy
crti/crtn mechanism.
Therefore, this commit drops support entirely for crti/crtn mechanism,
which is necessary since the other commits in this branch make it
nondeterministic in which order the libc objects and the other link
inputs are sent to the linker.
The linker is still expected to produce a deterministic output, however,
by ignoring object input order for the purposes of symbol resolution.
This means doing more work in parallel which is already good, but it's
also a correctnes fix because we need link_task_wait_group.wait() to
ensure that no more linker inputs will be generated.
The original motivation here was to fix regressions caused by #22414.
However, while working on this, I ended up discussing a language
simplification with Andrew, which changes things a little from how they
worked before #22414.
The main user-facing change here is that any reference to a prior
function parameter, even if potentially comptime-known at the usage
site or even not analyzed, now makes a function generic. This applies
even if the parameter being referenced is not a `comptime` parameter,
since it could still be populated when performing an inline call. This
is a breaking language change.
The detection of this is done in AstGen; when evaluating a parameter
type or return type, we track whether it referenced any prior parameter,
and if so, we mark this type as being "generic" in ZIR. This will cause
Sema to not evaluate it until the time of instantiation or inline call.
A lovely consequence of this from an implementation perspective is that
it eliminates the need for most of the "generic poison" system. In
particular, `error.GenericPoison` is now completely unnecessary, because
we identify generic expressions earlier in the pipeline; this simplifies
the compiler and avoids redundant work. This also entirely eliminates
the concept of the "generic poison value". The only remnant of this
system is the "generic poison type" (`Type.generic_poison` and
`InternPool.Index.generic_poison_type`). This type is used in two
places:
* During semantic analysis, to represent an unknown result type.
* When storing generic function types, to represent a generic parameter/return type.
It's possible that these use cases should instead use `.none`, but I
leave that investigation to a future adventurer.
One last thing. Prior to #22414, inline calls were a little inefficient,
because they re-evaluated even non-generic parameter types whenever they
were called. Changing this behavior is what ultimately led to #22538.
Well, because the new logic will mark a type expression as generic if
there is any change its resolved type could differ in an inline call,
this redundant work is unnecessary! So, this is another way in which the
new design reduces redundant work and complexity.
Resolves: #22494Resolves: #22532Resolves: #22538
We can still often determine a comptime result based on the type, even
if the pointer is runtime-known.
Also, we previously used load -> is non null instead of AIR
`is_non_null_ptr` if the pointer is comptime-known, but that's a bad
heuristic. Instead, we should check for the pointer to be
comptime-known, *and* for the load to be comptime-known, and only in
that case should we call `Sema.analyzeIsNonNull`.
Resolves: #22556
The new memcpy function aims to be more generic than the previous
implementation which was adapted from an implementation optimized for
x86_64 avx2 machines. Even on x86_64 avx2 machines this implementation
should be generally be faster due to fewer branches in the small length
cases and generating less machine code.
Note that the new memcpy function no longer acts as a memmove.
On x86_64, the `@divFloor` change is a strict improvement, and the
`@mod` change adds one zero latency instruction. In return, once we
upgrade to LLVM 20, when the optimizer discovers one of these operations
has a power-of-two constant rhs, it will be able to optimize the entire
operation into an `ashr` or `and`, respectively.
#I CPL CPT
old `@divFloor` | 8 | 15 | .143 |
new `@divFloor` | 7 | 15 | .148 |
old `@mod` | 9 | 17 | .134 | (rip llvm
new `@mod` | 10 | 17 | .138 | scheduler)
`std.Build.Step.Run` makes the very reasonable assumption that
`error.InvalidExe` will be reported on `spawn` if it will happen.
However, this property does not currently hold on POSIX targets. This
is, through a slightly convoluted series of events, partially
responsible for the sporadic `BrokenPipe` errors we've been seeing more
and more in CI runs.
Making `spawn` wait for the child to exec in the POSIX path introduces
a block of up to 400us. So, instead of doing that, we add a new API for
this particular case: `waitForSpawn`. This function is a nop on Windows,
but on POSIX it blocks until the child successfully (or otherwise) calls
`execvpe`, and reports the error if necessary. `std.Build.Step.Run`
calls this function, so that it can get `error.InvalidExe` when it wants
it.
I'm not convinced that this API is optimal. However, I think this entire
API needs to be either heavily refactored or straight-up redesigned
(related: #22504), so I'm not too worried about hitting the perfect API:
I'd rather just fix this bug for now, and figure out the long-term goal
a bit later.
The previous logic here was trying to assume that custom test runners
never used `std.zig.Server` to communicate with the build runner;
however, it was flawed, because modifying the `test_runner` field on
`Step.Compile` would not update this flag. That might have been
intentional (allowing a way for the user to specify a custom test runner
which *does* use the compiler server protocol), but if so, it was a
flawed API, since it was too easy to update one field without updating
the other.
Instead, bundle these two pieces of state into a new type
`std.Build.Step.Compile.TestRunner`. When passing a custom test runner,
you are now *provided* to specify whether it is a "simple" runner, or
whether it uses the compiler server protocol.
This is a breaking change, but is unlikely to affect many people, since
custom test runners are seldom used in the wild.
The blocker for enabling this feature was my need to debug the emitted
assembly without debug info and having to manually inspect memory to
determine struct contents. However, we now have debug info!
(lldb) v -L foo bar
0x00007fffffffda20: (repro.repro.Foo) foo = {
0x00007fffffffda24: .x = 12
0x00007fffffffda20: .y = 34
}
0x00007fffffffda28: (repro.repro.Bar) bar = {
0x00007fffffffda28: .x = 56
0x00007fffffffda2c: .y = 78
}
Updates #21530
Note: This mostly matches resinator v0.1.0 rather than the latest master version, since the latest master version focuses on adding support for .res -> .obj conversion which is not necessary for the future planned relationship of zig and resinator (resinator will likely be moved out of the compiler and into the build system, a la translate-c).
So, ultimately the changes here consist mostly of bug fixes for obscure edge cases.
This PR adds support for handling ZIP64 format in local file headers,
when a zip file contains entries where the compressed or uncompressed
size fields are set to 0xFFFFFFFF, and the extra field contains ZIP64
extended information tag (0x0001)
The code now:
Reads the actual sizes from the ZIP64 extra field data
Validates these sizes against the entry's compressed and uncompressed sizes
Zip file format spec.: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
This change allows proper extraction of ZIP files that use ZIP64 format in their
local file headers.
Fixes: #22329
This reverts commit 133abdeda2 but keeps
the tests disabled for the wasm target, which is the only configuration
that seems to fail, even though the error looks like a frontend error.
For representing struct field default values and array/pointer type
sentinel values, we use `*const anyopaque`, since there is no way for
`std.builtin.Type.StructField` etc to refer back to its `type` field.
However, when introspecting a type, this is quite awkward due to the
pointer casts necessary.
As such, this commit renames the `sentinel` fields to `sentinel_ptr`,
and the `default_value` field to `default_value_ptr`, and introduces
helper methods `sentinel()` and `defaultValue()` to load the values.
These methods are marked as `inline` because their return value, which
is always comptime-known, is very often required at comptime by use
sites, so this avoids having to annotate such calls with `comptime`.
This is a breaking change, although note that 0.14.0 is already a
breaking release for all users of `std.builtin.Type` due to the union
fields being renamed.
Implementing the changes from the prior commit, to prepare for the
following commit.
This also means that zig1 now uses the new value interpret mode, so
that adding and removing fields from `std.builtin` types is easier.
Signed-off-by: mlugg <mlugg@mlugg.co.uk>
This was done by regex substitution with `sed`. I then manually went
over the entire diff and fixed any incorrect changes.
This diff also changes a lot of `callconv(.C)` to `callconv(.c)`, since
my regex happened to also trigger here. I opted to leave these changes
in, since they *are* a correct migration, even if they're not the one I
was trying to do!
This matches established naming conventions. Now is an opportune time to
make this change, since we're already performing breaking changes to
`std.builtin.Type`.
this logic has not yet been ported to the new design, but the logic is
safe and sound in the git history and does not need to also live as
commented out code
fix calculation of alignment and size
include __tls_align and __tls_size globals along with __tls_base
include them only if the TLS segment is emitted
add missing reloc logic for memory_addr_tls_sleb
fix name of data segments to include only the prefix
Object being linked has neither functions nor globals named "foo" or
"bar" and so these names correctly fail to be exported when creating an
executable.
I intentionally simplified the target features functionality to use the
target features that are explicitly specified to the linker and ignore
the "tooling conventions"
this makes the wasm linker behave the same as ELF, COFF, and MachO.
this tests for importing a function table, but the example source does
not try to use an imported table, so it's a useless check. it's unclear
what the behavior is even supposed to do in this case.
the other two cases are left alone.
The purpose of this test is unclear. It checks for the existence of bss
section which is completely unnecessary since those zeroes can be
omitted from the binary.
Furthermore the code generated for __wasm_init_memory looks wrong.
Finally, the CheckObject DSL is brittle, it only checks for exact
matches of entire lines in an ad-hoc text format. Conclusion, it's a bad
test, delete it.
and detect passive inits from Zcu
don't forget to intern function type for __wasm_init_memory
make that function the start function if it is present
don't skip emitting passive data segment data to the binary
codegen can be called which contains calls to navs which have only their
type resolved. this means the indirect function table needs to track nav
indexes not ip indexes.
instead of recursion, callers of the function are responsible for
checking the respective tables that might have new entries in them and
then calling lowerZcuData again.
Recognize three distinct phases:
* before prelink ("object phase")
* after prelink, before flush ("zcu phase")
* during flush ("flush phase")
With this setup, we create data structures during the object phase, then
mutate them during the zcu phase, and then further mutate them during
the flush phase. In order to make the flush phase repeatable, the data
structures are copied just before starting the flush phase.
Further Zcu updates occur against the non-copied data structures.
What's not implemented is frontend garbage collection, in which case
some more changes will be needed in this linker logic to achieve a valid
state with data invariants intact.
fix some compilation errors for reworked Emit now that it's actually
referenced
introduce DataSegment.Id for sorting data both from object files and
from the Zcu.
introduce optimization: data segment sorting includes a descending sort
on reference count so that references to data can be smaller integers
leading to better LEB encodings. this optimization is skipped for object
files.
implement uav address access function which is based on only 1 hash
table lookup to find out the offset after sorting.
* function resolution now links to zcu_funcs, not navs_exe/navs_obj
* updateFunc now adds things to output functions
* updateNav now handles function aliases correctly
* only report start symbol missing when it is unresolved
mainly, rework how relocations works. This is the point at which symbol
indexes are known - not before. And don't emit unnecessary relocations!
They're only needed when emitting an object file.
Changes wasm linker to keep MIR around long-lived so that fixups can be
reapplied after linker garbage collection.
use labeled switch while we're at it
Makes linker functions have small error sets, required to report
diagnostics properly rather than having a massive error set that has a
lot of codes.
Other linker implementations are not ported yet.
Also the branch is not passing semantic analysis yet.
The goals of this branch are to:
* compile faster when using the wasm linker and backend
* enable saving compiler state by directly copying in-memory linker
state to disk.
* more efficient compiler memory utilization
* introduce integer type safety to wasm linker code
* generate better WebAssembly code
* fully participate in incremental compilation
* do as much work as possible outside of flush(), while continuing to do
linker garbage collection.
* avoid unnecessary heap allocations
* avoid unnecessary indirect function calls
In order to accomplish this goals, this removes the ZigObject
abstraction, as well as Symbol and Atom. These abstractions resulted
in overly generic code, doing unnecessary work, and needless
complications that simply go away by creating a better in-memory data
model and emitting more things lazily.
For example, this makes wasm codegen emit MIR which is then lowered to
wasm code during linking, with optimal function indexes etc, or
relocations are emitted if outputting an object. Previously, this would
always emit relocations, which are fully unnecessary when emitting an
executable, and required all function calls to use the maximum size LEB
encoding.
This branch introduces the concept of the "prelink" phase which occurs
after all object files have been parsed, but before any Zcu updates are
sent to the linker. This allows the linker to fully parse all objects
into a compact memory model, which is guaranteed to be complete when Zcu
code is generated.
This commit is not a complete implementation of all these goals; it is
not even passing semantic analysis.
`Type.hasWellDefinedLayout` was in disagreement with pointer loading
logic about auto-layout structs with zero fields, `struct {}`. For
consistency, these types should not have a well-defined layout.
This is technically a breaking change.
This replaces the msdos-stub binary with a fully documented
byte array with inline comments to make it easy to understand what
every byte actually means.
`Sema.explainWhyValueContainsReferenceToComptimeVar` (concise name!)
adds notes to an error explaining how to get from a given `Value` to a
pointer to some `comptime var` (or a comptime field). Previously, this
error could be very opaque in any case where it wasn't obvious where the
comptime var pointer came from; particularly for type captures. Now, the
error notes explain this to the user.
Before the prior commit, the maximum comptime recursion depth on my
system was 4062. After the prior commit, it decreased to 2854. This
commit increases the compiler's stack size enough so that the recursion
depth limit is no less than it was before the `Sema.analyzeCall`
rewrite, preventing this from being a breaking change. Specifically,
this stack size increases my observed maximum comptime recursion depth
to 4105.
This rewrite improves some error messages, hugely simplifies the logic,
and fixes several bugs. One of these bugs is technically a new rule
which Andrew and I agreed on: if a parameter has a comptime-only type
but is not declared `comptime`, then the corresponding call argument
should not be *evaluated* at comptime; only resolved. Implementing this
required changing how function types work a little, which in turn
required allowing a new kind of function coercion for some generic use
cases: function coercions are now allowed to implicitly *remove*
`comptime` annotations from parameters with comptime-only types. This is
okay because removing the annotation affects only the call site.
Resolves: #22262
The relocation range issues will happen eventually as we add more code to the
standard library and test suites, so we may as well just deal with this now.
@MasonRemaley ran into this in #20271, for example.
* MSDN documentation page covering what resource IDs manifests should have:
https://learn.microsoft.com/en-us/windows/win32/sbscs/using-side-by-side-assemblies-as-a-resource
* This change ensures shared libraries that embed win32 manifests use the
proper ID of 2 instead of 1, which is only allowed for .exes. If the manifest
uses the wrong ID, it will not be found and is essentially ignored.
Rather than `Zcu.BuiltinDecl.Memoized` being a struct with fields, it
can instead just be an array, indexed by the enum. This allows runtime
indexing, avoiding a few now-unnecessary `inline` switch cases.
This commit reworks how values like the panic handler function are
memoized during a compiler invocation. Previously, the value was
resolved by whichever analysis requested it first, and cached on `Zcu`.
This is problematic for incremental compilation, as after the initial
resolution, no dependencies are marked by users of this memoized state.
This is arguably acceptable for `std.builtin`, but it's definitely not
acceptable for the panic handler/messages, because those can be set by
the user (`std.builtin.Panic` checks `@import("root").Panic`).
So, here we introduce a new kind of `AnalUnit`, called `memoized_state`.
There are 3 such units:
* `.{ .memoized_state = .va_list }` resolves the type `std.builtin.VaList`
* `.{ .memoized_state = .panic }` resolves `std.Panic`
* `.{ .memoized_state = .main }` resolves everything else we want
These units essentially "bundle" the resolution of their corresponding
declarations, storing the results into fields on `Zcu`. This way, when,
for instance, a function wants to call the panic handler, it simply runs
`ensureMemoizedStateResolved`, registering one dependency, and pulls the
values from the `Zcu`. This "bundling" minimizes dependency edges. The 3
units are separated to allow them to act independently: for instance,
the panic handler can use `std.builtin.Type` without triggering a
dependency loop.
`Zcu.PerThead.ensureTypeUpToDate` is set up in such a way that it only
returns the updated type the first time it is called. In general, that's
okay; however, the exception is that we want the function to continue
returning `error.AnalysisFail` when the type has been lost, or its
number of captures changed.
Therefore, the check for this case now happens before the up-to-date
success return.
For simplicity, the number of captures is now handled by intentionally
losing the instruction in `Zcu.mapOldZirToNew`, since there is nothing
to gain from tracking a type when old instances of it can never be
reused.
The old lowering was kind of neat, but it unintentionally allowed the
syntax `for (123) |_| { ... }`, and there wasn't really a way to fix
that. So, instead, we include both the start and the end of the range in
the `for_len` instruction (each operand to `for` now has *two* entries
in this multi-op instruction). This slightly increases the size of ZIR
for loops of predominantly indexables, but the difference is small
enough that it's not worth complicating ZIR to try and fix it.
Previously, logic in `Compilation.getAllErrorsAlloc` was corrupting the
`failed_analysis` hashmap. This meant that on updates after the initial
update, attempts to remove entries from this map (because the `AnalUnit`
in question is being re-analyzed) silently failed. This resulted in
compile errors from earlier updates wrongly getting "stuck", i.e. never
being removed.
This commit also adds a few log calls which helped me to find this bug.
Most calls to `requireRuntimeBlock` in Sema are not correct. This
function doesn't deal with all of them, but it does deal with ones which
have, in combination with the past few commits, introduced real-world
regressions.
Related: #22353
Some sub-expressions should always be evaluated at comptime -- in
particular, type expressions, e.g. `E` in `E!T`. However, bugs in this
logic are easy to miss, because the parent scope is usually comptime
anyway!
This fixes a bug which exposed a compiler implementation detail (ZIR
alloc elision). Previously, `const` declarations with a runtime-known
value in a comptime scope were permitted only if AstGen was able to
elide the alloc in ZIR, since the error was reported by storing to the
comptime alloc.
This just adds a new instruction to also emit this error when the alloc
is elided.
To avoid this PR regressing error messages, most of the work here has
gone towards improving error notes for why code was comptime-evaluated.
ZIR `block_comptime` now stores a "comptime reason", the enum for which
is also used by Sema. There are two types in Sema:
* `ComptimeReason` represents the reason we started evaluating something
at comptime.
* `BlockComptimeReason` represents the reason a given block is evaluated
at comptime; it's either a `ComptimeReason` with an attached source
location, or it's because we're in a function which was called at
comptime (and that function's `Block` should be consulted for the
"parent" reason).
Every `Block` stores a `?BlockComptimeReason`. The old `is_comptime`
field is replaced with a trivial `isComptime()` method which returns
whether that reason is non-`null`.
Lastly, the handling for `block_comptime` has been simplified. It was
previously going through an unnecessary runtime-handling path; now, it
is a trivial sub block exited through a `break_inline` instruction.
Resolves: #22296
The `.empty` map in a shard is weird: it claims to have capacity 1, but
you're not actually allowed to actually use that capacity. That's fine
for the normal insertion algorithm, because it always resizes to a
higher capacity when inserting the initial element. However,
`rehashTrackedInsts` was not aware of this caveat, so sometimes tried to
store to the single element of the `empty` map.
This system exists to avoid an extra branch in the main resizing logic
(since `new_cap = old_cap * 2` only works if the capacity is never
non-zero). However, it's fine for `rehashTrackedInsts` to have an extra
branch to handle this case, since it's literally called once per update.
This commit separates semantic analysis of the annotated type vs value
of a global declaration, therefore allowing recursive and mutually
recursive values to be declared.
Every `Nav` which undergoes analysis now has *two* corresponding
`AnalUnit`s: `.{ .nav_val = n }` and `.{ .nav_ty = n }`. The `nav_val`
unit is responsible for *fully resolving* the `Nav`: determining its
value, linksection, addrspace, etc. The `nav_ty` unit, on the other
hand, resolves only the information necessary to construct a *pointer*
to the `Nav`: its type, addrspace, etc. (It does also analyze its
linksection, but that could be moved to `nav_val` I think; it doesn't
make any difference).
Analyzing a `nav_ty` for a declaration with no type annotation will just
mark a dependency on the `nav_val`, analyze it, and finish. Conversely,
analyzing a `nav_val` for a declaration *with* a type annotation will
first mark a dependency on the `nav_ty` and analyze it, using this as
the result type when evaluating the value body.
The `nav_val` and `nav_ty` units always have references to one another:
so, if a `Nav`'s type is referenced, its value implicitly is too, and
vice versa. However, these dependencies are trivial, so, to save memory,
are only known implicitly by logic in `resolveReferences`.
In general, analyzing ZIR `decl_val` will only analyze `nav_ty` of the
corresponding `Nav`. There are two exceptions to this. If the
declaration is an `extern` declaration, then we immediately ensure the
`Nav` value is resolved (which doesn't actually require any more
analysis, since such a declaration has no value body anyway).
Additionally, if the resolved type has type tag `.@"fn"`, we again
immediately resolve the `Nav` value. The latter restriction is in place
for two reasons:
* Functions are special, in that their externs are allowed to trivially
alias; i.e. with a declaration `extern fn foo(...)`, you can write
`const bar = foo;`. This is not allowed for non-function externs, and
it means that function types are the only place where it is possible
for a declaration `Nav` to have a `.@"extern"` value without actually
being declared `extern`. We need to identify this situation
immediately so that the `decl_ref` can create a pointer to the *real*
extern `Nav`, not this alias.
* In certain situations, such as taking a pointer to a `Nav`, Sema needs
to queue analysis of a runtime function if the value is a function. To
do this, the function value needs to be known, so we need to resolve
the value immediately upon `&foo` where `foo` is a function.
This restriction is simple to codify into the eventual language
specification, and doesn't limit the utility of this feature in
practice.
A consequence of this commit is that codegen and linking logic needs to
be more careful when looking at `Nav`s. In general:
* When `updateNav` or `updateFunc` is called, it is safe to assume that
the `Nav` being updated (the owner `Nav` for `updateFunc`) is fully
resolved.
* Any `Nav` whose value is/will be an `@"extern"` or a function is fully
resolved; see `Nav.getExtern` for a helper for a common case here.
* Any other `Nav` may only have its type resolved.
This didn't seem to be too tricky to satisfy in any of the existing
codegen/linker backends.
Resolves: #131
The `Cau` abstraction originated from noting that one of the two primary
roles of the legacy `Decl` type was to be the subject of comptime
semantic analysis. However, the data stored in `Cau` has always had some
level of redundancy. While preparing for #131, I went to remove that
redundany, and realised that `Cau` now had exactly one field: `owner`.
This led me to conclude that `Cau` is, in fact, an unnecessary level of
abstraction over what are in reality *fundamentally different* kinds of
analysis unit (`AnalUnit`). Types, `Nav` vals, and `comptime`
declarations are all analyzed in different ways, and trying to treat
them as the same thing is counterproductive!
So, these 3 cases are now different alternatives in `AnalUnit`. To avoid
stealing bits from `InternPool`-based IDs, which are already a little
starved for bits due to the sharding datastructures, `AnalUnit` is
expanded to 64 bits (30 of which are currently unused). This doesn't
impact memory usage too much by default, because we don't store
`AnalUnit`s all too often; however, we do store them a lot under
`-fincremental`, so a non-trivial bump to peak RSS can be observed
there. This will be improved in the future when I made
`InternPool.DepEntry` less memory-inefficient.
`Zcu.PerThread.ensureCauAnalyzed` is split into 3 functions, for each of
the 3 new types of `AnalUnit`. The new logic is much easier to
understand, because it avoids conflating the logic of these
fundamentally different cases.
The new representation is often more compact. It is also more
straightforward to understand: for instance, `extern` is represented on
the `declaration` instruction itself rather than using a special
instruction. The same applies to `var`, making both of these far more
compact.
This commit also separates the type and value bodies of a `declaration`
instruction. This is a prerequisite for #131.
In general, `declaration` now directly encodes details of the syntax
form used, and the embedded ZIR bodies are for actual expressions. The
only exception to this is functions, where ZIR is effectively designed
as if we had #1717. `extern fn` declarations are modeled as
`extern const` with a function type, and normal `fn` definitions are
modeled as `const` with a `func{,_fancy,_inferred}` instruction. This
may change in the future, but improving on this was out of scope for
this commit.
looking at `man getgroups` and `info getgroups` this is given as an
example:
```c
// Here's how to use ‘getgroups’ to read all the supplementary group
// IDs:
gid_t *
read_all_groups (void)
{
int ngroups = getgroups (0, NULL);
gid_t *groups
= (gid_t *) xmalloc (ngroups * sizeof (gid_t));
int val = getgroups (ngroups, groups);
if (val < 0)
{
free (groups);
return NULL;
}
return groups;
}
```
getgroups(0, NULL) is used to get the count of groups so that the
correct count can be used to allocate a list of gid_t. This small changes makes this
possible.
equivalent example in Zig after the change:
```zig
// get the group count
const ngroups: usize = std.os.linux.getgroups(0, null);
if (ngroups <= 0) {
return error.GetGroupsError;
}
std.debug.print("number of groups: {d}\n", .{ngroups});
const groups_gids: []u32 = try alloc.alloc(u32, ngroups);
// populate an array of gid_t
_ = std.os.linux.getgroups(ngroups, @ptrCast(groups_gids));
```
There's been some proliferation of dependency URLs that reference
mutable data such as links to git branches that can change. This has
resulted in broken projects, i.e.
* 9eef9de94c/build.zig.zon
* 4b64353e9c
There's also disagreement about whether it's fine for URL's to point to
git branches, i.e.
https://github.com/Not-Nik/raylib-zig/pull/130
This updates the docs to mention that zig won't be able to use URLs if
their content changes.
compiler: disallow `callconv` etc from depending on function parameters
Also, disallow `align`/`linksection`/`addrspace` annotations on container-level declarations with comptime-only types.
This includes function aliases, but not function declarations.
Also, re-introduce a target check for function alignment which was
inadvertently removed in the prior commit.
This commit amends `std.Build.ExecutableOptions` etc to have a new
field, `root_module`, which allows artifacts to be created whose root
module is an existing `*Module` rather than a freshly constructed one.
This API can be far more versatile, allowing construction of complex
module graphs before creating any compile steps, and therefore also
allowing easy reuse of modules.
The fields which correspond to module options, such as
`root_source_file`, are all considered deprecated. They may not be
populated at the same time as the `root_module` field. In the next
release cycle, these deprecated fields will be removed, and the
`root_module` field made non-optional.
At the expense of a slight special case in the build runner, we can make
the handling of dependencies between modules a little shorter and much
easier to follow.
When module and step graphs are being constructed during the "configure"
phase, we do not set up step dependencies triggered by modules. Instead,
after the configure phase, the build runner traverses the whole
step/module graph, starting from the root top-level steps, and
configures all step dependencies implied by modules. The "make" phase
then proceeds as normal. Also, the old `Module.dependencyIterator` logic
is replaced by two separate iterables. `Module.getGraph` takes the root
module of a compilation, and returns all modules in its graph; while
`Step.Compile.getCompileDependencies` takes a `*Step.Compile` and
returns all `*Step.Compile` it depends on, recursively, possibly
excluding dynamic libraries. The old `Module.dependencyIterator`
combined these two functions into one unintuitive iterator; they are now
separated, which in particular helps readability at the usage sites
which only need one or the other.
This commit changes the `root_module` field of `std.Build.Step.Compile`
to be a `*Module` rather than a `Module`. This is a breaking change, but
an incredibly minor one (the full potential extent of the breakage can
be seen in the modified standalone test).
This change will be necessary for an upcoming improvement, so it was
convenient to make it here.
(With the exception of x86 since that was available from the beginning.)
These were determined by analyzing the full, reconstructed Git history of the
Linux kernel here: https://landley.net/kdocs/fullhist
Previously, if multiple navs owned the same type due to being the same
zir node and having the same captures, they would overwrite each other.
Now, the navs codegenned later emit a decl alias to the first one.
Currently, `zig ast-check` fails on ZON files, because it tries to
interpret the file as Zig source code. This commit introduces a new
verification pass, `std.zig.ZonGen`, which applies to an AST in ZON
mode.
Like `AstGen`, this pass also converts the AST into a more helpful
format. Rather than a sequence of instructions like `Zir`, the output
format of `ZonGen` is a new datastructure called `Zoir`. This type is
essentially a simpler form of AST, containing only the information
required for consumers of ZON. It is also far more compact than
`std.zig.Ast`, with the size generally being comparable to the size of
the well-formatted source file.
The emitted `Zoir` is currently not used aside from the `-t` option to
`ast-check` which causes it to be dumped to stdout. However, in future,
it can be used for comptime `@import` of ZON files, as well as for
simpler handling of files like `build.zig.zon`, and even by other parts
of the Zig Standard Library.
Resolves: #22078
The previous commit exposed some missing `const` qualifiers in a few
places. These mutable slices could have been used to store invalid
values into memory!
The error messages here aren't amazing yet, but this is an improvement
on status quo, because the current behavior allows false negative
compile errors, so effectively miscompiles.
Resolves: #15874
This code was left over from the legacy Autodoc implementation. No
component of the compiler pipeline actually requires doc comments, so it
is a waste of time and space to store them in ZIR.
This is, at least today, a very broken target: It doesn't actually build either
musl or wasi-libc even if you use -lc. It does give you musl headers, but that's
it. Those headers are not terribly useful, however, without any implementation
code. You can sort of call some math functions because they just so happen to
have implementations in compiler-rt. But that's only true for a small subset,
and I don't think users should be relying on the ABI surface of a library that
is an implementation detail of the compiler.
Clearly, a freestanding-capable libc of sorts is a useful thing as evidenced by
newlib, picolibc, etc existing. However, calling it "musl" is misleading when it
isn't actually musl-compatible, nor can it ever be because the musl API surface
is inextricably tied to the Linux kernel. In the discussion on #20690, there was
agreement that once we split up the API and ABI components in the target string,
the API component should be about compatibility, not whether you literally get a
particular implementation of it. Also, we decided that Linux musl and wasi-libc
musl shouldn't use the same API tag precisely because they're not actually
compatible.
(And besides, how would any syscall even be implemented in freestanding? Who or
what would we be calling?)
So I think we should remove this triple for now. If we decide to reintroduce
something like this, especially once #2879 gets going, we should come up with a
bespoke name for it rather than using "musl".
The goal of this commit is to get rid of some "unused command line argument"
warnings that Clang would give for various file types previously. This cleanup
also has the side effect of making the order of flags more understandable,
especially as it pertains to include paths.
Since a lot of code was shuffled around in this commit, I recommend reviewing
the old and new versions of the function side-by-side rather than trying to make
sense of the diff.
There are several test decls inside `/src` that are not currently being
tested and have bitrotted as a result. This commit revives those tests
and adds the `test-compiler-internals` set of tests which tests
everything reachable from `/src/main.zig`.
In cf88cf2657 the eql function provided in
The context of ArrayHashMap was changed to also include the key index,
but this wasn't properly updated in the documentation.
Since a flat `usize` is unintuitive, I've tried to explain the function
of the parameter as best I can based on the original commit.
Finally, I didn't do an extensive search if this eql definition is
incorrectly stated anywhere outside of these 2 spots. But I somewhat
doubt an file outside of `array_hash_map` would
The previous commit cast doubt upon the initial report about macOS
kernel behavior, identifying another reason that ENOENT could be
returned from file creation.
However, it is demonstrable that ENOENT can be returned for both cases:
1. create file race
2. handle refers to deleted directory
This commit re-introduces the workaround for the file creation race on
macOS however it does not unconditionally retry - it first tries again
with O_EXCL to disambiguate the error condition that has occurred.
Previous commits
2b0929929d4ea2f441df
had this text:
> There are no dir components, so you would think that this was
> unreachable, however we have observed on macOS two processes racing to
> do openat() with O_CREAT manifest in ENOENT.
This appears to have been a misunderstanding based on the issue
report #12138 and corresponding PR #12139 in which the steps to
reproduce removed the cache directory in a loop which also executed
detached Zig compiler processes.
There is no evidence for the macOS kernel bug however the ENOENT is
easily explained by the removal of the cache directory.
This commit reverts those commits, ultimately reporting the ENOENT as an
error rather than repeating the create file operation. However this
commit also adds an explicit error set to `std.Build.Cache.hit` as well
as changing the `failed_file_index` to a proper diagnostic field that
fully communicates what failed, leading to more informative error
messages on failure to check the cache.
The equivalent failure when occuring for AstGen performs a fatal process
kill, reasoning being that the compiler has an invariant of the cache
directory not being yanked out from underneath it while executing. This
could be made a more granular error in the future but I suspect such
thing is not valuable to pursue.
Related to #18340 but does not solve it.
Whatever was in the frame pointer register prior to clone() will no longer be
valid in the child process, so zero it to protect FP-based unwinders. This is
just an extension of what was already done for i386 and x86_64. Only applied
to architectures where the _start() code also zeroes the frame pointer.
Whatever was in the frame pointer register prior to clone() will no longer be
valid in the child process, so zero it to protect FP-based unwinders. Similarly,
mark the link register as undefined to protect DWARF-based unwinders.
This is only zeroing the frame pointer(s) on Arm/Thumb because of an LLVM
assembler bug: https://github.com/llvm/llvm-project/issues/115891
The goal here is to support both levels of unwind tables (sync and async) in
zig cc and zig build. Previously, the LLVM backend always used async tables
while zig cc was partially influenced by whatever was Clang's default.
NetBSD has long since migrated to the EABI and doesn't officially support the
OABI anymore. The ABI selection logic in LLVM only actually picks OABI for
NetBSD as a last resort if the EABI isn't selected. That fallback is likely to
be removed in the future. So just remove this support in Zig entirely.
While here, I also removed some leftover 32-bit Arm and 32-bit x86 code for
Apple targets, which are long dead and unsupported by Zig.
Both of these instructions were previously under a special case in
`rvalue` which resulted in every reference to such an instruction adding
a new `ref` instruction. This had the effect that, for instance,
`&a != &a` for parameters. Deduplicating these `ref` instructions was
problematic for different reasons.
For `alloc_inferred`, the problem was that it's not valid to `ref` the
alloc until the allocation has been resolved (`resolve_inferred_alloc`),
but `AstGen.appendBodyWithFixups` would place the `ref` directly after
the `alloc_inferred`. This is solved by bringing
`resolve_inferred_alloc` in line with `make_ptr_const` by having it
*return* the final pointer, rather than modifying `sema.inst_map` of the
original `alloc_inferred`. That way, the `ref` refers to the
`resolve_inferred_alloc` instruction, so is placed immediately after it,
avoiding this issue.
For `param`, the problem is a bit trickier: `param` instructions live in
a body which must contain only `param` instructions, then a
`func{,_inferred,_fancy}`, then a `break_inline`. Moreover, `param`
instructions may be referenced not only by the function body, but also
by other parameters, the return type expression, etc. Each of these
bodies requires separate `ref` instructions. This is solved by pulling
entries out of `ref_table` after evaluating each component of the
function declaration, and appending the refs later on when actually
putting the bodies together. This gives way to another issue: if you
write `fn f(x: T) @TypeOf(x.foo())`, then since `x.foo()` takes a
reference to `x`, this `ref` instruction is now in a comptime context
(outside of the `@TypeOf` ZIR body), so emits a compile error. This is
solved by loosening the rules around `ref` instructions; because they
are not side-effecting, it is okay to allow `ref` of runtime values at
comptime, resulting in a runtime-known value in a comptime scope. We
already apply this mechanism in some cases; for instance, it's why
`runtime_array.len` works in a `comptime` context. In future, we will
want to give similar treatment to many operations in Sema: in general,
it's fine to apply runtime operations at comptime provided they don't
have side effects!
Resolves: #22140
Frame pointers make both debugging and profiling work better, and the overhead
is reportedly 1% or less for typical programs [0]. I think the pros outweigh the
cons here. People who *really* care about that 1% can simply use the
-fomit-frame-pointer option to reclaim it. For ReleaseSmall, though, it makes
sense to omit frame pointers by default for the sake of code size, as we already
strip the binary in this case anyway.
Closes#22161.
[0] https://www.brendangregg.com/blog/2024-03-17/the-return-of-the-frame-pointers.html
* This warning's wording is actually inaccurate when using the -fno-compiler-rt
or -rtlib=none options.
* It's not all that helpful; it's already understood that these libraries are
part of the compiler, so printing a warning is just noise. In practice, this
warning would always happen when building upstream musl, for example.
* We don't warn when we satisfy -lunwind using our bundled libunwind either, or
various libc libraries using our bundled libc, or when providing libc++, etc.
So I really don't think we should be warning here either.
This is GCC's equivalent to compiler-rt. The two libraries have a huge overlap
in exported symbols, so we may as well satisfy it this way to increase
compatibility with build systems in the wild.
The real libgcc_s is a compiler-provided library; it works just fine with both
glibc and musl. There's no reason that I can see for this check to be limited to
glibc-based targets.
This is a library that ships with GCC and provides fallback implementations of
atomic intrinsics where necessary. Since we do the same in our compiler-rt
implementation, and since some build systems insist on passing -latomic even
for Clang (which zig cc masquerades as), just satisfy this dependency by way of
compiler-rt.
Closes#22165.
The introduction of the `extended(astgen_error())` instruction allows a
`test` declaration to be unresolved, i.e. the declaration doesn't even
contain a `func`. I could modify AstGen to not do this, but it makes
more sense to just handle this case when collecting test functions.
Note that tests under incremental compilation are currently broken if
you ever remove all references to a test; this is tracked as a subtask
of #21165.
The main change here is to partition tracked instructions found within a
declaration. It's very unlikely that, for instance, a `struct { ... }`
type declaration was intentionally turned into a reification or an
anonymous initialization, so it makes sense to track things in a few
different arrays.
In particular, this fixes an issue where a `func` instruction could
wrongly be mapped to something else if the types of function parameters
changed. This would cause huge problems further down the pipeline; we
expect that if a `declaration` is tracked, and it previously contained a
`func`/`func_inferred`/`func_fancy`, then this instruction is either
tracked to another `func`/`func_inferred`/`func_fancy` instruction, or
is lost.
Also, this commit takes the opportunity to rename the functions actually
doing this logic. `Zir.findDecls` was a name that might have made sense
at some point, but nowadays, it's definitely not finding declarations,
and it's not *exclusively* finding type declarations. Instead, the point
is to find instructions which we want to track; hence the new name,
`Zir.findTrackable`.
Lastly, a nice side effect of partitioning the output of `findTrackable`
is that `Zir.declIterator` no longer needs to accept input instructions
which aren't type declarations (e.g. `reify`, `func`).
The previous commit exposed some bugs in incremental compilation. This
commit fixes those, and adds a little more logging for debugging
incremental compilation.
Also, allow `ast-check -t` to dump ZIR when there are non-fatal AstGen
errors.
This commit enhances AstGen to introduce a form of error resilience
which allows valid ZIR to be emitted even when AstGen errors occur.
When a non-fatal AstGen error (e.g. `appendErrorNode`) occurs, ZIR
generation is not affected; the error is added to `astgen.errors` and
ultimately to the errors stored in `extra`, but that doesn't stop us
getting valid ZIR. Fatal AstGen errors (e.g. `failNode`) are a bit
trickier. These errors return `error.AnalysisFail`, which is propagated
up the stack. In theory, any parent expression can catch this error and
handle it, continuing ZIR generation whilst throwing away whatever was
lost. For now, we only do this in one place: when creating declarations.
If a call to `fnDecl`, `comptimeDecl`, `globalVarDecl`, etc, returns
`error.AnalysisFail`, the `declaration` instruction is still created,
but its body simply contains the new `extended(astgen_error())`
instruction, which instructs Sema to terminate semantic analysis with a
transitive error. This means that a fatal AstGen error causes the
innermost declaration containing the error to fail, but the rest of the
file remains intact.
If a source file contains parse errors, or an `error.AnalysisFail`
happens when lowering the top-level struct (e.g. there is an error in
one of its fields, or a name has multiple declarations), then lowering
for the entire file fails. Alongside the existing `Zir.hasCompileErrors`
query, this commit introduces `Zir.loweringFailed`, which returns `true`
only in this case.
The end result here is that files with AstGen failures will almost
always still emit valid ZIR, and hence can undergo semantic analysis on
the parts of the file which are (from AstGen's perspective) valid. This
is a noteworthy improvement to UX, but the main motivation here is
actually incremental compilation. Previously, AstGen failures caused
lots of semantic analysis work to be thrown out, because all `AnalUnit`s
in the file required re-analysis so as to trigger necessary transitive
failures and remove stored compile errors which would no longer make
sense (because a fresh compilation of this code would not emit those
errors, as the units those errors applied to would fail sooner due to
referencing a failed file). Now, this case only applies when a file has
severe top-level errors, which is far less common than something like
having an unused variable.
Lastly, this commit changes a few errors in `AstGen` to become fatal
when they were previously non-fatal and vice versa. If there is still a
reasonable way to continue AstGen and lower to ZIR after an error, it is
non-fatal; otherwise, it is fatal. For instance, `comptime const`, while
redundant syntax, has a clear meaning we can lower; on the other hand,
using an undeclared identifer has no sane lowering, so must trigger a
fatal error.
It doesn't appear that targeting bridgeOS is meaningfully supported by Apple.
Even LLVM/Clang appear to have incomplete support for it, suggesting that Apple
never bothered to upstream that support. So there's really no sense in us
pretending to support this.
This fix doesn't matter at all in the grand scheme of things, but I
think the story behind it is perhaps curious, as it might point at a
design flaw in the Sema's error reporting API. So, a story:
On lobsters, there's a rather heated discussion on the merits on RAII vs
defer. I don't really like participating in heating discussions, but
also sort of can't stop thinking about this.
My own personal experience with Zig's defer and errdefer is that they
are fiddly to get right consistency --- if a program has a lot of
resource management to do, I _always_ mess up at least one
defer/errdefer. I've found my internal peace by just avoiding
spread-out, "pox" resource management, and instead centralizing resource
ownership under one of the following patterns:
* Either the thing is acquired and released in main
* Or main allocates N instances of thing, and then the rest of the code
explicitly juggles this finite pool of N. Notably, this juggling
typically doesn't involve defer/errdefer at all, as, at this level of
precision, there are no `try`s left, so you only code the happy path
* Or there's some sort of arena thing, where a bunch of resources have a
single owner, the user's don' bother cleaning up their resources, and
instead the owner does it once at the end.
So I wanted to make a lobster.rs comment in the vein of "yeah, if your
program is mostly about resource management, then Zig could be kinda a
pain, but that's friction tells you something: perhaps your program
shouldn't be about resource management, and instead it should be doing
what it is supposed to do?". And, as an evidence for my claim, I wanted
to point out some large body of Zig code which doesn't have a lot of
errdefers.
So, I cracked opened Sema.zig, `ctrl+f` for `defer`, saw whopping 400
something occupancies, and my heart skipped a bit. Looking at the
occurrences, _some_ of them were non-resource-related usages of defer.
But a lot of them were the following pattern:
```zig
const msg = try sema.errMsg(src, "comptime control flow inside runtime block", .{});
errdefer msg.destroy(sema.gpa);
```
This is exactly the thing that I know _I_ can't get right consistently!
So, at this point, I made a prediction that at least one of `errdefer`s
is missing. So, I looked at the first few `const msg = try` and of
course found one without `errdefer`.
I am at 0.8 that, even with this PR applied, the claim will still stand
--- there will be `errdefer` missing. So it feels like some API
re-design is in order, to make sure individual error messages are not
resources.
Could Sema just own all partially-constructed error messages, and, at a
few known safe-points:
* if the control flow is normal, assert that there are no in-progress
error messages
* if we are throwing an error, go and release messages immediately?
I am unlikely to do the actual refactor here, but I think it's worth
highlighting the overall pattern here.
PS: I am only 0.9 sure that what I've found is indeed a bug! I don't
understand the code, I did a dumb text search, so I _could_ have made a
fool of myself here :P
If present, these headers are usable even when compiling for older C language
versions.
Most notably, this enables zig.h atomics to work with slimcc and TinyCC in C99
mode (and earlier).
The freestanding and other OS targets by default need to just @trap in the
default Panic implementation.
And `isValidMemory` won't work with freestanding or other targets.
Update the unwind_freestanding.zig test case to also run on the 'other' OS
target, too. This should keep the Zig's stacktrace generation from
regressing on the standalone targets.
Without doing this, we don't actually test whether the data layout string we
generate matches LLVM's.
A number of targets had to be commented out due to this change:
* Some are using a non-working experimental LLVM backend (arc, csky, ...).
* Some don't have working LLD support (lanai, sparc, ...).
* Some don't have working self-hosted linker support (nvptx).
* Some are using ABIs that haven't been standardized (loongarch32).
Finally, all non-x86 uefi targets are hopelessly broken and can't really be
fixed until we change our emit logic to lower *-uefi-* verbatim rather than to
*-windows-*. See: https://github.com/ziglang/zig/issues/21630
We have deduced that it seems the sporadic BrokenPipe failures happening
on the CI runners (e.g.
https://github.com/ziglang/zig/actions/runs/12035916948/job/33555963190)
are likely caused by the test runner's stdin pipe abnormally closing,
likely due to the process crashing. Here, we introduce error handling
for this case, so that if these writes fail, the step is marked as
failed correctly, and we still collect the child's stderr to report.
This won't fix the CI issues, but it should promote them to proper error
messages including child stderr, which -- at least in theory -- should
allow us to ultimately track down where the errors come from.
Note that this change is desirable regardless of bugs in the test runner
or similar, since the child process could terminate abnormally for any
number of reasons (e.g. a crashing test), and such cases should be
correctly reported by the build runner.
Apple has already dropped support for macOS 12.
GitHub Actions is dropping macOS 12 support now.
The Zig project is also dropping macOS 12 support now.
This commit also bumps default minimum macos version to 13.
To my knowledge there isn't an implementation of `sse4.2` that doesn't have `crc32`.
The Clang driver also sets `crc32` to be implicitly enabled when an explicit `-crc32`
wasn't provided. This matches that behaviour.
We need this behaviour to compile libraries like `rocksdb` which currently guard against
`crc32` intrinsics by checking for `sse4.2`.
* Cleanup the argument handling logic to allow for optional arguments.
* Add a filter for which `llvm_target` to process.
* Switch to using a threadpool, needed for skipping llvm targets cleanly
and better distributes the work.
* Remove a seemingly useless piece of logic. I re-ran the script and it gave identical outputs.
Uses the non rational solution of a quadratic, I made it work up to 256
bits, added Mathematica code in case anyone wants to verify the magic
constant.
integers between sizes 3...15 were affected by fatal bias, it is best to
make them pass through the generic solution.
Thanks to RetroDev256 & Andrew feedback.
In the parent commit, I handled odd bit sizes by upcasting and
truncating. However it seems the else branch is intended to handle
those cases instead, so this commit reverts that behavior.
also
* allow signed ints, simply bitcast them to unsigned
* handle odd bit sizes by upcasting and then truncating
* naming conventions
* remove redundant code
* better use of testing API
Before, the default bit mixer was very biased, and after a
lot of searching it turns out that selecting a better solution is hard.
I wrote a custom statistical analysis taylored for bit mixers in order
to select the best one at each size (u64/u32/u16), compared a lot of
mixers, and packaged the best ones in this commit.
It wasn't immediately clear from the implementation whether passing
zero-length memory to free() was undefined behavior or intentionally
supported. Since ArrayList and other core data structures rely on
this behavior working correctly, this should be explicitly documented
as part of the public API contract.
And make the initialization less error prone by removing a default for
iter, which is required for a functional parser
std: Add a brief doc comment for `std.fmt.Parser`
Previously, stepping from the single statement within the loop would
always exit the loop because all of the code unrolled from the loop is
associated with the same line and treated by the debugger as one line.
/home/alexrp/.cache/zig/b/18236e302af25e3fb99bc6a232ddc447/builtin.zig:6:5: error: TODO (SPIR-V): Implement unsigned composite int type of 64 bits
pub const zig_backend = std.builtin.CompilerBackend.stage2_spirv64;
~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is necessary since isGnuLibC() is true for hurd, so we need to be able to
represent a glibc version for it.
Also add an Os.TaggedVersionRange.gnuLibCVersion() convenience function.
The tool will now skip over undefined symbols. These can only occur as a result
of building musl without compiler-rt, i.e. -rtlib=none. Thanks to this, it's no
longer necessary to patch Zig's compiler-rt, nor is it necessary to maintain a
symbol blacklist.
See the updated instructions here: https://github.com/ziglang/zig/wiki/Updating-libc#updating-the-libcs-file
Additionally, the tool now produces slightly more compact output by recognizing
symbols that are defined for a single arch, for a family of arches, or only for
arches using 32-bit or 64-bit time as their primary ABI.
Finally, the tool now supports all architectures that we can emit code for, with
the single exception of x86_64-linux-muslx32. (x32 currently fails with a ton of
relocation errors, leading me to believe that it might be an LLVM or LLD bug.)
The `if (@sizeOf(T) == 0) return true;` check simply doesn't work for a number
of cases so that is removed and changed into `@sizeOf(T) != 0` and then
used in the `eqlBytes` check chain to make comparing `enum{}`s possible.
The rest special-cases for comptime-only types and undefined to make
those comparisons possible as well.
Fixes#19929
They way OpenSSH does key derivation to protect keys using a password
is not the standard PBKDF2, but something funky, picking key material
non-linearly.
* std.crypto.aes: introduce AES block vectors
Modern Intel CPUs with the VAES extension can handle more than a
single AES block per instruction.
So can some ARM and RISC-V CPUs. Software implementations with
bitslicing can also greatly benefit from this.
Implement low-level operations on AES block vectors, and the
parallel AEGIS variants on top of them.
AMD Zen4:
aegis-128x4: 73225 MiB/s
aegis-128x2: 51571 MiB/s
aegis-128l: 25806 MiB/s
aegis-256x4: 46742 MiB/s
aegis-256x2: 30227 MiB/s
aegis-256: 8436 MiB/s
aes128-gcm: 5926 MiB/s
aes256-gcm: 5085 MiB/s
AES-GCM, and anything based on AES-CTR are also going to benefit
from this later.
* Make AEGIS-MAC twice a fast
* crypto.keccak.State: don't unconditionally permute after a squeeze()
Now, squeeze() behaves like absorb()
Namely,
squeeze(x[0..t]);
squeeze(x[t..n)); with t <= n
becomes equivalent to squeeze(x[0..n]).
* keccak: in debug mode, track transitions to prevent insecure ones.
Fixes#22019
When using Build.Step.Run.captureStdOut with a program that prints more
than 10 megabytes of output, the build process hangs.
This is because evalGeneric returns an error without reading child's
stdin till the end, so we subsequently get stuck in `try child.wait()`.
To fix this, make sure to kill the child in case of an error!
Output before this change:
λ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6
[3/8] steps
└─ run gh
^C
λ # an hour of debugging
Output after this change:
λ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6
install
└─ install generated to ../tigerbeetle
└─ run build_mutliversion (tigerbeetle)
└─ run unzip
└─ run gh failure
error: unable to spawn gh: StdoutStreamTooLong
Build Summary: 3/8 steps succeeded; 1 failed (disable with --summary none)
install transitive failure
└─ install generated to ../tigerbeetle transitive failure
└─ run build_mutliversion (tigerbeetle) transitive failure
└─ run unzip transitive failure
└─ run gh failure
error: the following build command failed with exit code 1:
/home/matklad/p/tb/work/.zig-cache/o/c0e3f5e66ff441cd16f9a1a7e1401494/build /home/matklad/p/tb/work/zig/zig /home/matklad/p/tb/work /home/matklad/p/tb/work/.zig-cache /home/matklad/.cache/zig --seed 0xc1d4efc8 -Zaecc61299ff08765 -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6
Our key pair creation API was ugly and inconsistent between ecdsa
keys and other keys.
The same `generate()` function can now be used to generate key pairs,
and that function cannot fail.
For deterministic keys, a `generateDeterministic()` function is
available for all key types.
Fix comments and compilation of the benchmark by the way.
Fixes#21002
If the same dependency is first found as lazy and then later as eager,
the existing entry needs to be updated to eager in order for
`b.dependency()` to work.
This command being JITed leads to a substantially worse first-time user
experience, since you have to wait for upwards of 20 seconds for
`fmt.zig` to build. This is especially bad when your editor is
configured to run `zig fmt` on save and does so in a blocking manner. As
such, it makes sense from a usability perspective to not JIT this
particular command.
This is similar to the old `llvm/shift_right_plus_left` case, which was
disabled by 1b1c78c. The case is not enabled on the LLVM backend, since
incremental compilation support for this backend is a work in progress
and is tracked by #21165. It passes on the x86_64-linux target with the
self-hosted backend.
Resolves: #12288
These cases have been disabled for a while, and we have transitioned to
using a compact file format for incremental test cases.
I was originally planning to port all of these cases, but the vast
majority aren't testing anything interesting, so it wasn't worth the
effort. I did look through each one; anything interesting being tested
has been extracted into a new case in `test/incremental/`.
Two of the new tests are currently failing with the self-hosted ELF
linker, and thus are currently only enabled with the C backend.
Resolves: #12844
This seems to be required for ptr_elem_ptr with storage buffers. Note that
this does not imply that the pointer can be regarded as physical too.
Some variants of ptr_elem_ptr will need to be forbidden
This is the same mode used by openssh for private keys. This does not
change the mode of an existing file, so users who need something
different can pre-create the file with their designed permissions or
change them after the fact, and running another process that writes to
the key log will not change it back.
By default, programs built in debug mode that open a https connection
will append secrets to the file specified in the SSLKEYLOGFILE
environment variable to allow protocol debugging by external programs.
This was preventing TLSv1.2 from working in some cases, because servers
are allowed to send multiple handshake messages in the first handshake
record, whereas this inital loop was assuming that it only contained a
server hello.
This is mostly nfc cleanup as I was bisecting the client hello to find
the problematic part, and the only bug fix ended up being
key_share.x25519_kp.public_key ++
key_share.ml_kem768_kp.public_key.toBytes()
to
key_share.ml_kem768_kp.public_key.toBytes() ++
key_share.x25519_kp.public_key)
and the same swap in `KeyShare.exchange` as per some random blog that
says "a hybrid keyshare, constructed by concatenating the public KEM key
with the public X25519 key". I also note that based on the same blog
post, there was a draft version of this method that indeed had these
values swapped, and that used to be supported by this code, but it was
not properly fixed up when this code was updated from the draft spec.
Closes#21747
Note that the removed `error.TlsIllegalParameter` case is still caught
below when it is compared to a fixed-length string, but after checking
the proper protocol version requirement first.
From `zig build-exe --help`:
-fno-builtin Disable implicit builtin knowledge of functions
It seems entirely reasonable and even expected that this option should imply
both no-builtins on functions (which disables transformation of recognized code
patterns to libcalls) and nobuiltin on call sites (which disables transformation
of libcalls to intrinsics). We now match Clang's behavior for -fno-builtin.
In both cases, we're painting with a fairly broad brush by applying this to an
entire module, but it's better than nothing. #21833 proposes a more fine-grained
way to apply nobuiltin.
This option, by its very nature, needs to be attached to a module. If it isn't,
the code in a module could break at random when compiled into an application
that doesn't have this option set.
After this change, skip_linker_dependencies no longer implies no_builtin in the
LLVM backend.
The former prevents recognizing code patterns and turning them into libcalls,
which is what we want for compiler-rt. The latter is meant to be used on call
sites to prevent them from being turned into intrinsics.
Context: https://github.com/ziglang/zig/issues/21833
* Make it work for thumb and aarch64.
* Clean up std.os.windows.teb() a bit.
I also updated stage1/zig.h since the changes are backwards-compatible and are
necessary due to the std.os.windows changes that call the newly-added functions.
The -lldmingw option affects a lot of logic throughout LLD. We need to pass it
for *-windows-gnu even when we're not actually linking MinGW since we're still
using the MinGW ABI with all that that entails. (One particular problem we would
run into is missing handling of stdcall-decorated symbols for 32-bit x86.) Also,
various other LLD options are sensitive to this option, so it's best to pass it
as early as possible.
Closes#11817.
The old isARM() function was a portability trap. With the name it had, it seemed
like the obviously correct function to use, but it didn't include Thumb. In the
vast majority of cases where someone wants to ask "is the target Arm?", Thumb
*should* be included.
There are exactly 3 cases in the codebase where we do actually need to exclude
Thumb, although one of those is in Aro and mirrors a check in Clang that is
itself likely a bug. These rare cases can just add an extra isThumb() check.
Once we upgrade to LLVM 20, these should be lowered verbatim rather than to
simply musl. Similarly, the special case in llvmMachineAbi() should go away.
Like d1d95294fd, this is more Apple nonsense where
they abused the arch component of the triple to encode what's really an ABI.
Handling this correctly in Zig's target triple model would take quite a bit of
work. Fortunately, the last Armv7-based Apple Watch was released in 2017 and
these targets are now considered legacy. By the time Zig hits 1.0, they will be
a distant memory. So just remove them.
Don't use the reader interface
Avoid unnecessary heap allocations
At first I started working on incorporating the Archive fields into the
Wasm data model, however, I realized a better strategy: simply omit
Archive data from the serialized linker state. Those files can be
trivially reparsed on next compiler process start. If they haven't
changed, great. Otherwise if they have, the prelink phase needs to be
restarted anyway.
Before, the wasm struct had a string table, the ZigObject had a string
table, and each Object had a string table.
Now there is just the one. This makes for more efficient use of memory
and simplifies logic, particularly with regards to linker state
serialization.
This commit additionally adds significantly more integer type safety.
- Rename GPU address spaces to match with SPIR-V spec.
- Emit `Block` Decoration for Uniform/PushConstant variables.
- Don't emit `OpTypeForwardPointer` for non-opencl targets.
(there's still a false-positive about recursive structs)
Signed-off-by: Ali Cheraghi <alichraghi@proton.me>
I was just bitten by this footgun, where I actually wanted
`sliceAsBytes` but unintentionally used `asBytes`, which in practice
ignored all but the first element. Just add a comptime assertion to
trigger a compile error in this case.
This commit reworks how anonymous struct literals and tuples work.
Previously, an untyped anonymous struct literal
(e.g. `const x = .{ .a = 123 }`) was given an "anonymous struct type",
which is a special kind of struct which coerces using structural
equivalence. This mechanism was a holdover from before we used
RLS / result types as the primary mechanism of type inference. This
commit changes the language so that the type assigned here is a "normal"
struct type. It uses a form of equivalence based on the AST node and the
type's structure, much like a reified (`@Type`) type.
Additionally, tuples have been simplified. The distinction between
"simple" and "complex" tuple types is eliminated. All tuples, even those
explicitly declared using `struct { ... }` syntax, use structural
equivalence, and do not undergo staged type resolution. Tuples are very
restricted: they cannot have non-`auto` layouts, cannot have aligned
fields, and cannot have default values with the exception of `comptime`
fields. Tuples currently do not have optimized layout, but this can be
changed in the future.
This change simplifies the language, and fixes some problematic
coercions through pointers which led to unintuitive behavior.
Resolves: #16865
On Linux, File.metadata calls the statx syscall directly. As such, the
return value is the error code. Previously, it handled the error with
`posix.errno`, which when libc is linked, treats the return value as a
value set to -1 if there is an error with the error code in errno. If
libc wasn't linked, it would be handled correctly.
In the Linux with libc linked case, this would cause the error result to
always be treated as success (err val != -1), even when an error
occurred.
Primarily, this moves linker input parsing from flush() into the linker
task queue, which is executed simultaneously with the frontend.
I also made it avoid redundantly opening the same archive file N times
for each object file inside. Furthermore, hard code fixed buffer stream
rather than using a generic stream type.
Finally, I fixed the error handling of the Wasm.Archive.parse function.
Please pay attention to this pattern of returning a struct rather than
accepting a mutable struct as an argument. This ensures function-level
atomicity and makes resource management straightforward.
Deletes the file and path fields from Archive and Object.
Removed a well-meaning but ultimately misguided suggestion about how to
think about ZigObject since thinking about it that way has led to
problematic anti-DOD patterns.
Removes the `files` field from the Wasm linker, storing the ZigObject
as its own field instead using a tagged union.
This removes a layer of indirection when accessing the ZigObject, and
untangles logic so that we can introduce a "pre-link" phase that
prepares the linker state to handle only incremental updates to the
ZigObject and then minimize logic inside flush().
Furthermore, don't make array elements store their own indexes, that's
always a waste.
Flattens some of the file system hierarchy and unifies variable names
for easier refactoring.
Introduces type safety for optional object indexes.
* AIX has its own bespoke format.
* Handle all Apple platforms.
* FreeBSD and OpenBSD both use the GNU format in LLVM.
* Windows has since been switched to the COFF format by default in LLVM.
LLVM recently introduced new Triple::ArchType members in 19.1.3 which broke our
static assertions in zig_llvm.cpp. When implementing a fix for that, I realized
that we don't even need a lot of the stuff we have in zig_llvm.(cpp,h) anymore.
This commit trims the interface down considerably.
According to a comment in mold, this is the expected (and desired)
condition by the linkers, except for some architectures (RISCV and
Loongarch) where this condition does not have to upheld.
If you follow the changes in this patch and in particular doc comments
I have linked the comment/code in mold that explains and implements
this.
I have also modified `testEhFrameRelocatable` test to now test both
cases such that `zig ld -r a.o b.o -o c.o` and `zig ld -r b.o a.o -o
d.o`. In both cases, `c.o` and `d.o` should produce valid object
files which was not the case before this patch.
Xcode requires target arm64_32 (aarch64-watchos-ilp32) in order to
build code for Apple Watches. This commit fixes compilation errors
that appear when compiling with that target.
It appears that ReadFile returns ERROR_BROKEN_PIPE for a broken pipe, but WriteFile returns ERROR_NO_DATA.
Co-authored-by: Alex Rønne Petersen <alex@alexrp.com>
This caused a missing reference for u16 to not be emitted. Triggered
after removing something from start.zig which transitively added u16
to the module.
Also, start using labeled switch statements when dispatching
maybe-runtime instructions like condbr to comptime-only variants like
condbr_inline.
This can't be merged until we get a zig1.wasm update due to #21385.
Resolves: #21405
The print order of error sets depends on the order that the compiler
adds names to its internal state. These names can be anything, and
do not necessarily need to be from the same error set or be errors
at all. When the last remaining reference to builtin.cpu.arch was
removed in start.zig in 9b42bc1ce5, this order changed. Likely there
is something that has the name 'C' that is referenced somewhere
recursively from builtin.cpu.arch.
This all causes these few tests to fail, and hence the expected
order is simply updated now. Perhaps there is a better way to
add this.
Under some architecture/operating system combinations it is forbidden
to return a pointer from a merge, as these pointers must point to a
location at compile time. This adds a check for those cases when
returning a pointer from a block merge.
* Use builtin.zig_backend instead of builtin.cpu.arch, the latter
does not yet compile under VK.
* Don't call regular _start for either opencl or vulkan. We might
even want to disable these completely.
Its semantics are now documented in terms of DynamicLinker.kind(os.tag).
The idea here is two-fold:
* The term "standard" actually means something; we shouldn't return a valid
dynamic linker path for a triple for which it hasn't *actually* been
standardized. That's just incorrect. For example, previously, this function
would happily return a path for x86_64-linux-androideabi, csky-macos-gnu, or
aarch64-hurd-msvc, and other such obvious nonsense.
* Callers that use the return value from this function to do host probing (such
as std.zig.system.detectAbiAndDynamicLinker()) can now do so with greater
confidence because DynamicLinker.standard() will eagerly reject nonsensical
target triples.
Since we exclude Abi.none from the list of ABIs to be tested, it means that
Abi.gnu, which happens to be the first in the list, always gets picked for hosts
where the dynamic linker path does not depend on the ABI component of the
triple. Such hosts include all the BSDs, Haiku, Serenity, Solaris, etc.
To fix this, use DynamicLinker.kind() to determine whether this whole exercise
even makes sense. If it doesn't, as is the case on every OS other than Linux and
Hurd, we'll just fall back to Abi.default() which will try to pick a sensible
default based on the arch and OS components. This detection logic still has
plenty of room for improvement, but is at least a notable step up from
confusingly detecting Abi.gnu ~everywhere.
Closes#9089.
hasDynamicLinker() was just kind of lying in the case of Darwin platforms for
the benefit of std.zig.system.detectAbiAndDynamicLinker(). A better name would
have been hasElfDynamicLinker() or something. It also got the answer wrong for a
bunch of platforms that don't actually use ELF. Anyway, this was clearly the
wrong layer to do this at, so remove this function and instead use
DynamicLinker.kind() + an isDarwin() check in detectAbiAndDynamicLinker().
I took a slightly unconventional approach to detecting endianness here. We have
no compiler/platform-specific preprocessor checks in the stage1 C code today,
and I think that's a property worth maintaining.
This makes no difference presently, but if LLVM ever starts modeling features
for these, we would not get them by default for our baseline if we use the
generic model.
This matches Clang's defaults. That also means these CPU models tend to get more
testing, so they're a safer baseline choice. Anecdotally, the oldest MIPS
hardware that I've seen anyone run Zig on was also r2.
ppc64le remains the baseline CPU model. Note that there's nothing about little
endian, 64-bit PowerPC that requires the features in the ppc64le model; the
reason it exists is that 64-bit PowerPC wasn't really used in little endian mode
prior to those features being commonplace. That makes the ppc64le model a good
baseline model, but not the right choice for a generic model.
```
std/os/uefi/protocol/simple_text_input.zig:10:63: error: no field named 'Win64' in enum '@typeInfo(builtin.CallingConvention).@"union".tag_type.?'
std/builtin.zig:169:31: note: enum declared here
std/os/uefi/protocol/simple_text_output.zig:9:64: error: no field named 'Win64' in enum '@typeInfo(builtin.CallingConvention).@"union".tag_type.?'
std/builtin.zig:169:31: note: enum declared here
std/os/uefi/tables/runtime_services.zig:26:86: error: no field named 'Win64' in enum '@typeInfo(builtin.CallingConvention).@"union".tag_type.?'
std/builtin.zig:169:31: note: enum declared here
```
using `.C` in Sema is incorrect since it will be resolved under the target that Zig was compiled with, not the target build configuration. This is easily solved by just calling `cCallingConvention` on the target to resolve it.
it doesn't detect and remove no longer watched things yet
it also isn't aware of any file names reported by kqueue. I'm unsure if
that functionality exists.
these tasks have some shared data dependencies so they cannot be done
simultaneously. Future work should untangle these data dependencies so
that more can be done in parallel.
for now this commit ensures correctness by making linker input parsing
and codegen tasks part of the same queue.
Unfortunately it's not a complete solution, so a follow-up commit will
need to do something more drastic like not do the linker task queue at
the same time as codegen task queue.
From that point, it is possible to do more work at the same time but
that should be a separate branch. This one has gotten big enough.
If the "is darwin" check is moved below the libc_installation check
below, error.LibCInstallationMissingCrtDir is returned from
lci.resolveCrtPaths().
This should be revisited because it makes sense to check
libc_installation first even on darwin.
Anyway for now this more closely matches logic from master branch.
By making it a field of link.Elf, it is now accessible without a data
dependency on `files`, fixing a race condition with the codegen thread
and linker thread.
* Compilation.objects changes to Compilation.link_inputs which stores
objects, archives, windows resources, shared objects, and strings
intended to be put directly into the dynamic section. Order is now
preserved between all of these kinds of linker inputs. If it is
determined the order does not matter for a particular kind of linker
input, that item should be moved to a different array.
* rename system_libs to windows_libs
* untangle library lookup from CLI types
* when doing library lookup, instead of using access syscalls, go ahead
and open the files and keep the handles around for passing to the
cache system and the linker.
* during library lookup and cache file hashing, use positioned reads to
avoid affecting the file seek position.
* library directories are opened in the CLI and converted to Directory
objects, warnings emitted for those that cannot be opened.
The compiler defaults this value to off so that users whose system
shared libraries are all ELF files don't have to pay the cost of
checking every file to find out if it is a text file instead.
When a GNU ld script is encountered, the error message instructs users
about the CLI flag that will immediately solve their problem.
along with the relevant logic, making the libraries within subject to
the same search criteria as all the other libraries.
this unfortunately means doing file system access on all .so files when
targeting ELF to determine if they are linker scripts, however, I have a
plan to address this.
thread-sanitizer reports data races here when running test-link. I tried
only removing the ones that triggered races, but after 10 back and
forths with the compiler and tsan, I got impatient and removed all of
them.
next time, let's be sure the test suite runs tsan-clean before merging
any changes that add parallelism.
after this commit, `zig build test-link` completes without any tsan
warnings.
closes#21778
In end(), the freelist pointer is owned so the bare store would be ok.
However, there is a load in start() that can happen at the same time, if
another start() and end() pair grabs that same index.
I don't think this fixes#21663 actually because even if the data race
corrupts the value for `next`, the cmpxchg protects the value from being
stored there.
This was the cause of aarch64-windows shared libraries causing "bad image" errors
during load-time linking. I also re-enabled the tests that were surfacing this bug.
minFunctionAlignment() is something we can know ahead of time for any given
target because it's a matter of ABI. However, defaultFunctionAlignment() is a
matter of optimization and every backend can do it differently depending on any
number of factors. For example, LLVM will base the choice on the CPU model in
its aarch64 backend. So just don't use this value in the frontend.
defaultFunctionAlignment() can be made more sophisticated over time based on the
CPU model and/or features. For now, I've picked some reasonable values for the
CPUs that are most commonly used in practice. (Values are sourced from LLVM.)
The whole motivation behind this proposal in the first place was that
the LLVM backend disagrees with the self-hosted backends on what
`@setAlignStack` meant, so we can't just translate the old logic to the
new system! These backends can introduce support for overriding
`incoming_stack_alignment` later on.
These only worked before because our lowering of the `AAPCS` calling
convention was incorrect in a way which happened to match the ABI of
these functions. The tests aren't actually very helpful -- there are
already tests for `divmoddi4` etc -- so rather than using inline asm on
the caller side to match the ABI, we just delete these two tests.
We were for some reason missing a direct test for `__udivmodsi4`, so one
has been added.
There are several more that we could support here, but I didn't feel
like going down the rabbit-hole of figuring them out. In particular,
some of the Clang enum fields aren't specific enough for us, so we'll
have to switch on the target to figure out how to translate-c them. That
can be a future enhancement.
This commit finishes implementing #21209 by removing the
`@setAlignStack` builtin in favour of `CallingConvention` payloads. The
x86_64 backend is updated to use the stack alignment given in the
calling convention (the LLVM backend was already updated in a previous
commit).
Resolves: #21209
The old `CallingConvention` type is replaced with the new
`NewCallingConvention`. References to `NewCallingConvention` in the
compiler are updated accordingly. In addition, a few parts of the
standard library are updated to use the new type correctly.
As well as being necessary for the `CallingConvention` changes, this
update includes the following notable changes:
* Fix unlabeled `break` targeting the wrong scope in the presence of
labeled continue, unblocking #21422
* Implement `@FieldType`
* Implement `@splat` on arrays
Signed-off-by: mlugg <mlugg@mlugg.co.uk>
This commit begins implementing accepted proposal #21209 by making
`std.builtin.CallingConvention` a tagged union.
The stage1 dance here is a little convoluted. This commit introduces the
new type as `NewCallingConvention`, keeping the old `CallingConvention`
around. The compiler uses `std.builtin.NewCallingConvention`
exclusively, but when fetching the type from `std` when running the
compiler (e.g. with `getBuiltinType`), the name `CallingConvention` is
used. This allows a prior build of Zig to be used to build this commit.
The next commit will update `zig1.wasm`, and then the compiler and
standard library can be updated to completely replace
`CallingConvention` with `NewCallingConvention`.
The second half of #21209 is to remove `@setAlignStack`, which will be
implemented in another commit after updating `zig1.wasm`.
* fix compilation errors for fs and fs.Dir
* mem.span instead of mem.sliceTo
* Updating symLinkAbsoluteW function parameters
* Update with expected rename semantics
These are really answering questions about the Zig compiler's capacity to
provide a libc/libc++ implementation. As such, std.zig.target seems like a more
fitting place for these.
The re-analysis here is a little coarse; it'd be nice in the future to
have a way for an AstGen failure to preserve *all* analysis which
depends on the last success, and just hide the compile errors which
depend on it somehow. But I'm not sure how we'd achieve that, so this
works fine for now.
Resolves: #21223
This not only simplifies the error bundling logic, but also improves
efficiency by allowing the result to be cached between, for instance,
multiple calls to `totalErrorCount`.
This isn't exactly the case provided in #11290, but is a slightly
simpler case which I know would have triggered the same bug in the old
implementation of incremental compilation.
Resolves: #11290
Ideally we'd like to use whatever alignment glibc actually ends up using in the
real libc.so.6. But we don't really have a way of getting at that information at
the moment, and it's not present in the abilist files. I haven't yet seen a
symbol that wasn't word-aligned, though, so I think this should be good enough
for 99% of symbols, if not actually 100%.
This prevents LLVM from...cleverly...merging all of the global variable stub
symbols that we emit under certain circumstances. This was observed in practice
when using zig-bootstrap for arm-linux-gnueabi(hf).
Make shared_objects a StringArrayHashMap so that deduping does not
need to happen in flush. That deduping code also was using an O(N^2)
algorithm, which is not allowed in this codebase. There is another
violation of this rule in resolveSymbols but this commit does not
address it.
This required reworking shared object parsing, breaking it into
independent components so that we could access soname earlier.
Shared object parsing had a few problems that I noticed and fixed in
this commit:
* Many instances of incorrect use of align(1).
* `shnum * @sizeOf(elf.Elf64_Shdr)` can overflow based on user data.
* `@divExact` can cause illegal behavior based on user data.
* Strange versyms logic that wasn't present in mold nor lld. The logic
was not commented and there is no git blame information in ziglang/zig
nor kubkon/zld. I changed it to match mold and lld instead.
* Use of ArrayList for slices of memory that are never resized.
* finding DT_VERDEFNUM in a different loop than finding DT_SONAME.
Ultimately I think we should follow mold's lead and ignore this
integer, relying on null termination instead.
* Doing logic based on VER_FLG_BASE rather than ignoring it like mold
and LLD do. No comment explaining why the behavior is different.
* Mutating the original ELF symbols rather than only storing the mangled
name on the new Symbol struct.
I noticed something that I didn't try to address in this commit: Symbol
stores a lot of redundant information that is already present in the ELF
symbols. I suspect that the codebase could benefit from reworking Symbol
to not store redundant information.
Additionally:
* Add some type safety to std.elf.
* Eliminate 1-3 file system reads for determining the kind of input
files, by taking advantage of file name extension and handling error
codes properly.
* Move more error handling methods to link.Diags and make them
infallible and thread-safe
* Make the data dependencies obvious in the parameters of
parseSharedObject. It's now clear that the first two steps (Header and
Parsed) can be done during the main Compilation pipeline, rather than
waiting for flush().
Some compilers such as Go reference the end of a section (addr + size)
which cannot be contained in any non-zero atom (since then this atom
would exceed section boundaries). In order to facilitate this behaviour,
we create a dummy zero-sized atom at section end (addr + size).
By organizing linker diagnostics into this struct, it becomes possible
to share more code between linker backends, and more importantly it
becomes possible to pass only the Diag struct to some functions, rather
than passing the entire linker state object in. This makes data
dependencies more obvious, making it easier to rearrange code and to
multithread.
Also fix MachO code abusing an atomic variable. Not only was it using
the wrong atomic operation, it is unnecessary additional state since
the state is already being protected by a mutex.
In order to reduce the logic that happens in flush() we need to see
which data is being accessed by all this logic, so we can see which
operations depend on each other.
`check_pie_supported` only uses the `OUTPUT_VARIABLE` to to signify errors
if PIE is actually supported is signaled by `CMAKE_<lang>_LINK_PIE_SUPPORTED`.
Checking if `OUTPUT_VARIABLE` is empty is not enough either since the check
is bypassed if its results are cached but the output variable is not cached.
When errors occurred during flush(), incremental cache mode was still
writing a successful cache manifest, making subsequent compilations fail
because they would get a cache hit only to find invalid data.
Embrace the Path abstraction, doing more operations based on directory
handles rather than absolute file paths. Most of the diff noise here
comes from this one.
Fix sorting of crtbegin/crtend atoms. Previously it would look at all
path components for those strings.
Make the C runtime path detection partially a pure function, and move
some logic to glibc.zig where it belongs.
The initAtoms function now only uses the `elf_file` parameter for
reporting linker error messages, making it easier to see that the
function has no data dependencies other than the Object struct itself,
making it easier to parallelize or otherwise move that logic around.
Also removed an indirect call via `addExtra` since we already know the
atom's file is the current Object instance. All calls to `Atom.addExtra`
should be audited for similar reasons.
Also removed unjustified use of `inline fn`.
Special symbols include explictly force undefined symbols passed via -u
flag, missing entry point symbol, missing 'dyld_stub_binder' symbol, or
missing '_objc_msgsend' symbol.
flush() must not do anything more than necessary. Determining the type
of input files must be done only once, before flush. Fortunately, we
don't even need any file system accesses to do this since that
information is statically known in most cases, and in the rest of the
cases can be determined by file extension alone.
This commit also updates the nearby code to conform to the convention
for error handling where there is exactly one error code to represent
the fact that error messages have already been emitted. This had the
side effect of improving the error message for a linker script parse
error.
"positionals" is not a linker concept; it is a command line interface
concept. Zig's linker implementation should not mention "positionals".
This commit deletes that array list in favor of directly making function
calls, eliminating that heap allocation during flush().
The goal is to minimize as much as possible how much logic is inside
flush(). So let's start by moving out obvious stuff. This data can be
preformatted before flush().
This commit changes how `std.io.poll` is implemented on Windows. The new
implementation unfortunately incurs a little extra system call overhead,
but fixes several bugs in the old implementation:
* The `lpNumberOfBytesRead` parameter of `ReadFile` was used with
overlapped I/O. This is explicitly disallowed by the documentation, as
the value written to this pointer is "potentially erroneous"; instead,
`GetOverlappedResult` must always be used, even if the operation
immediately returns. Documentation states that `lpNumberOfBytesRead`
cannot be passed as null on Windows 7, so for compatibility, the
parameter is passed as a pointer to a dummy global.
* If the initial `ReadFile` returned data, and the next read returned
`BROKEN_PIPE`, the received data was silently ignored in the sense
that `pollWindows` did not `return`, instead waiting for data to come
in on another file (or for all files to close).
* The asynchronous `ReadFile` calls which were left pending between
calls to `pollWindows` pointed to a potentially unstable buffer, since
the user of `poll` may use part of the `LinearFifo` API which rotate
its ring buffer. This race condition was causing CI failures in some
uses of the compiler server protocol.
These issues are all resolved. Now, `pollWindows` will queue an initial
read to a small (128-byte) stable buffer per file. When this read is
completed, reads directly into the FIFO's writable slice are performed
until one is left pending, at which point that read is cancelled (with a
check to see if it was completed between the `ReadFile` and `CancelIo`
calls) and the next read into the small stable buffer is queued. These
small buffer reads are the ones left pending between `pollWindows`
calls, avoiding the race condition described above.
Related: #21565
Throw another target in there just to spice things up a little!
Running the incremental cases with the C backend is pretty slow due to
the need to recompile the whole output from scratch on every update; for
this reason, we probably don't want to keep many of these targeting CBE
long-term. However, for now, while we have relatively few tests and
things are still changing quite a lot, it's better to have this little
bit of extra test coverage.
If no external executor is available for a successful binary, its
execution is silently skipped. This allows the CI to test, to the
fullest extent possible, incremental cross-compilation to targets whose
binaries can't be executed on the host.
This is contained in the `test` step, so is tested by CI.
This commit also includes some enhancements to the `incr-check` tool to
make this work correctly.
* Adds new cpu architectures propeller1 and propeller2.
These cpu architectures allow targeting the Parallax Propeller 1 and Propeller 2, which are both very special microcontrollers with 512 registers and 8 cpu cores.
Resolves#21559
* Adds std.elf.EM.PROPELLER and std.elf.EM.PROPELLER2
* Fixes missing switch prongs in src/codegen/llvm.zig
* Fixes order in std.Target.Arch
---------
Co-authored-by: Felix "xq" Queißner <git@random-projects.net>
On s390x-linux, fstat() does not have nanosecond precision, but fstatat() does.
As a result, comparing Stat structs returned from these syscalls is doomed to
fail.
In theory, this should work for v68+. In practice, it runs into an LLVM
assertion when using a `freeze` instruction on `f16` values, similar to the
issue we had for LoongArch.
This works around the fact that LLVM and LLD both have broken support for the
small data area, yet the feature is on by default for all Hexagon CPUs.
I want to eventually replace this hack with a flag in update_cpu_features.zig
for marking features that should always be off by default and not be accessible
to users. That way, the compiler will have full control over them.
Until we actually figure out a version range for this, we shouldn't classify
this as using semver. Doing so results in a panic when trying to access the
version range since it's not in fact a semver value.
Only targets for which we don't source CPU models/features from LLVM should use
the fallback prong in this switch. Also make it exhaustive to catch similar
mistakes in the future.
At bare minimum, the compiler expects std.Target.<arch>.cpu.generic to work for
any given architecture when generating the builtin module. So rather than try to
hack that into working when the affected modules are omitted, just actually keep
them.
This affected lanai and xcore.
* Added error message 'ProcessNotFound' for reading and writing in a Linux
process.
This error occurs if the process to be read from or written to no longer exists.
Fixes#19875
* Added error message "ProcessNotFound" for error forwarding.
* Add error messgae for forwarding.
* Added message for forwarding.
* Error set completed.
* Fixed format error.
* Changed comments to doc comments.
These have no callers outside std.elf. Even if the standard library should
provide functions like these, std.elf is probably not the place, given how
general they are.
The dependency cache is shared amongst all Build objects. This is currently
done by allocating a single instance and storing a reference to it in each
Build object. However, the Graph object already exists to host shared
state so by moving it there we reuse the same pattern for shared state
and avoid an extra object on the heap.
This experimental target has no recent active maintainer. It's the only
linker backend complaining about this branch and I can't make sense of
the stack trace.
This can be fixed asynchronously by anyone who wants to maintain plan9
support. It does not need to block this branch.
although they would also pass simply reverted to master branch because
I made the deprecated API still work for now (to be removed after 0.14.0
is tagged)
Introduces `std.builtin.Panic` which is a complete interface for
panicking. Provide `std.debug.FormattedPanic` and
`std.debug.SimplePanic` and let the user choose, or make their own.
This is a breaking change which updates the `rtattr.type` from `IFLA` to
`union { IFLA, IFA }`. `IFLA` is for the `RTM_*LINK` messages and `IFA`
is for the `RTM_*ADDR` messages.
Fixes#21446
Both UefiPoolAllocator and UefiRawPoolAllocator were
passing the value of `log2_ptr_align` directly to
`mem.alignAllocLen` which expects a alignment value.
Both of these calls to `mem.alignAllocLen` are pointless
and the result of the alignment both always true, and
was thrown away anyway.
I have removed these calls entirely.
memcpy requires non-overlapping arguments.
fifo.realign() handles this case correctly and tries to provide an
optimized implementation.
This probably wasn't hit in practice, as, in a typical usage, fifo's
head is not advanced.
cortex-m85 already has 8msecext, which is not the same as trustzone. The former
is for the M profile, while the latter is for the A profile.
Revert of a small part of #18498.
Abi.android on its own is not enough to know whether soft float or hard float
should be used. In the C world, androideabi is typically used for the soft float
case, so let's go with that.
Note that Android doesn't have a hard float ABI, so no androideabihf.
Closes#21488.
The previous implementation of buffered_reader always reads from the
unbuffered reader into the internal buffer, and then dumps the data onto
the destination. This is inefficient, as sometimes it's possible to read
directly into the destination. The previous strategy generates more
memory copies and unbuffered reads than necessary.
See: https://devblogs.microsoft.com/directx/directx-adopting-spir-v
Since we never hooked up the (experimental) DirectX LLVM backend, we've never
actually supported targeting DXIL in Zig. With Microsoft moving away from DXIL,
that seems very unlikely to change.
Upgrades the LLVM, Clang, and LLD dependencies to LLVM 19.x
Related to #16270
Big thanks to Alex Rønne Petersen for doing the bulk of the upgrade work
in this branch.
This works around a problem that started happening with LLD around
version 18.1.8:
```
lld-link: error: duplicate symbol: .weak.__nexf2.default
>>> defined at CMakeFiles/zig2.dir/compiler_rt.c.obj
>>> defined at compiler_rt.lib(compiler_rt.lib.obj)
```
This reverts commit 7e66b6d0684fb1b3aa76381486e655e4d13bc0a5.
I don't think this is needed, I don't get any errors locally when I
bootstrap windows without this change.
Windows does not really have weak symbols. So when we bootstrap with `zig cc`
and link both Zig's compiler-rt and the CBE's `compiler_rt.c` we end up with
duplicate symbol errors at link time.
* Add `ProcessorAlias` support.
* Bump output buffer size.
* Include `i` extension in RISC-V baselines.
* Update evaluation branch quota for RISC-V.
* Retain some CPU features that LLVM removed.
* Flatten more 'meta-features' used for CPU models.
* Remove some superfluous dependencies.
There is one minor language change here, which is that comparisons of
the form `comptime_inf < runtime_f32` have their results comptime-known.
This is consistent with comparisons against comptime NaN for instance,
which are always comptime known. A corresponding behavior test is added.
This fixes a bug with int comparison elision which my previous commit
somehow triggered. `Sema.compareIntsOnlyPossibleResult` is much cleaner
now!
PR #20927 made some improvements to the `binarySearch` API, but one
change I found surprising was the relationship between the left-hand and
right-hand parameters of `compareFn` was inverted. This is different
from how comparison functions typically behave, both in other parts of
Zig (e.g. `std.math.order`) and in other languages (e.g. C's `bsearch`).
Unless a strong reason can be identified and documented for doing
otherwise, I think it'll be better to stick with convention.
While writing this patch and changing things back to the way they were,
the predicates of `lowerBound` and `upperBound` seemed to be the only
areas that benefited from the inversion. I don't think that benefit is
worth the cost, personally. Calling `Order.invert()` in the predicates
accomplishes the same goal.
I've seen implementations in the wild that send 'Upgrade: WebSocket',
which currently fails the handshake.
From https://datatracker.ietf.org/doc/html/rfc6455:
"If the response lacks an |Upgrade| header field or the |Upgrade| header
field contains a value that is not an ASCII case-insensitive match for
the value "websocket", the client MUST _Fail the WebSocket Connection_."
The previous API used `std.testing.fuzzInput(.{})` however that has the
problem that users call it multiple times incorrectly, and there might
be work happening to obtain the corpus which should not be included in
coverage analysis, and which must not slow down iteration speed.
This commit restructures it so that the main loop lives in libfuzzer and
directly calls the "test one" function.
In this commit I was a little too aggressive because I made the test
runner export `fuzzer_one` for this purpose. This was motivated by
performance, but it causes "exported symbol collision: fuzzer_one" to
occur when more than one fuzz test is provided.
There are three ways to solve this:
1. libfuzzer needs to be passed a function pointer instead. Possible
performance downside.
2. build runner needs to build a different process per fuzz test.
Potentially wasteful and unclear how to isolate them.
3. test runner needs to perform a relocation at runtime to point the
function call to the relevant unit test. Portability issues and
dubious performance gains.
Closes#21358Closes#21360
This commit modifies the `multiline_string_literal_line`, `doc_comment`,
and `container_doc_comment` tokens to no longer include the line ending
as part of the token. This makes it easier to handle line endings (which
may be LF, CRLF, or in edge cases possibly nonexistent) consistently.
In the two issues linked above, Autodoc was already assuming this for
doc comments, and yielding incorrect results when handling files with
CRLF line endings (both in Markdown parsing and source rendering).
Applying the same simplification for multiline string literals also
brings `zig fmt` into conformance with
https://github.com/ziglang/zig-spec/issues/38 regarding formatting of
multiline strings with CRLF line endings: the spec says that `zig fmt`
should remove the CR from such line endings, but this was not previously
the case.
Broadly speaking, versions 6, 7, and 8 are the ones that are in common use. Of
these, v7 is what you'll typically see for 32-bit Arm today. So let's actually
make sure that that's what we're testing.
`ensureCauAnalyzed` adds the anal_unit to the transitive failures, so we don't need to do anything here. The errors will be handled after this function.
This fixes the function for riscv32 where the old nanosleep() is not available.
clock_nanosleep() has been available since Linux 2.6 and glibc 2.1 anyway.
Closes#21311
The sign of the result `r` needs to be initialized before the correction
`r.addScalar(r.toConst(), -1)`, or the intended end result could be off
by 2 (depending on the original sign of `r`).
Based on:
* `include/elf/common.h` in binutils
* `include/uapi/linux/elf-em.h` in Linux
* https://www.sco.com/developers/gabi/latest/ch4.eheader.html
I opted to use the tag naming of binutils because it seems to be by far the most
complete and authoritative source at this point in time.
`.loop` is also a block, so the block_depth must be stored *after* block
creation, ensuring a correct block_depth to jump back to when receiving
`.repeat`.
This also un-regresses `switch_br` which now correctly handles ranges
within cases. It supports it for both jump tables as well as regular
conditional branches.
The parse of `fn foo(a: switch (...) { ... })` was previously handled
incorrectly; `a` was treated as both the parameter name and a label.
The same issue exists for `for` and `while` expressions -- they should
be fixed too, and the grammar amended appropriately. This commit does
not do this: it only aims to avoid introducing regressions from labeled
switch syntax.
This commit introduces a new AIR instruction, `repeat`, which causes
control flow to move back to the start of a given AIR loop. `loop`
instructions will no longer automatically perform this operation after
control flow reaches the end of the body.
The motivation for making this change now was really just consistency
with the upcoming implementation of #8220: it wouldn't make sense to
have this feature work significantly differently. However, there were
already some TODOs kicking around which wanted this feature. It's useful
for two key reasons:
* It allows loops over AIR instruction bodies to loop precisely until
they reach a `noreturn` instruction. This allows for tail calling a
few things, and avoiding a range check on each iteration of a hot
path, plus gives a nice assertion that validates AIR structure a
little. This is a very minor benefit, which this commit does apply to
the LLVM and C backends.
* It should allow for more compact ZIR and AIR to be emitted by having
AstGen emit `repeat` instructions more often rather than having
`continue` statements `break` to a `block` which is *followed* by a
`repeat`. This is done in status quo because `repeat` instructions
only ever cause the direct parent block to repeat. Now that AIR is
more flexible, this flexibility can be pretty trivially extended to
ZIR, and we can then emit better ZIR. This commit does not implement
this.
Support for this feature is currently regressed on all self-hosted
native backends, including x86_64. This support will be added where
necessary before this branch is merged.
This commit modifies the representation of the AIR `switch_br`
instruction to represent ranges in cases. Previously, Sema emitted
different AIR in the case of a range, where the `else` branch of the
`switch_br` contained a simple `cond_br` for each such case which did a
simple range check (`x > a and x < b`). Not only does this add
complexity to Sema, which we would like to minimize, but it also gets in
the way of the implementation of #8220. That proposal turns certain
`switch` statements into a looping construct, and for optimization
purposes, we want to lower this to AIR fairly directly (i.e. without
involving a `loop` instruction). That means we would ideally like a
single instruction to represent the entire `switch` statement, so that
we can dispatch back to it with a different operand as in #8220. This is
not really possible to do correctly under the status quo system.
This commit implements lowering of this new `switch_br` usage in the
LLVM and C backends. The C backend just turns any case containing ranges
entirely into conditionals, as before. The LLVM backend is a little
smarter, and puts scalar items into the `switch` instruction, only using
conditionals for the range cases (which direct to the same bb). All
remaining self-hosted backends are temporarily regressed in the presence
of switch range cases. This functionality will be restored for at least
the x86_64 backend before merge.
Very simply add the format specifier to the print statement.
Since debug.print is hard coded I couldn't come up with a reasonalble
way to add a test, and since this function is simple enough I doubt it's
useful.
fixes one part of #21094
Both glibc and musl use time64 as the base ABI for riscv32. This fixes the
`sleep` test in `std.time` hanging forever due to the libc functions reading
bogus values.
The kernel does define the struct, it just doesn't use it. Yet both glibc and
musl expose it directly as their public stat struct, and std.c takes it from
std.os.linux. So just define it after all.
Passing it by value means that bringup on new architectures is harder for no
real benefit. Passing it by pointer allows to get the compiler running without
needing to figure out the C calling convention details first. This manifested in
practice on LoongArch, for example.
Most of the required renames here are net wins for readaibility, I'd
say. The ones in `arch` are a little more verbose, but I think better. I
didn't bother renaming the non-conflicting functions in
`arch/arm/bits.zig` and `arch/aarch64/bits.zig`, since these backends
are pretty bit-rotted anyway AIUI.
These names aren't matching any formal specification; they're mostly
just ripped from LLVM code. Therefore, we should definitely follow Zig
naming conventions here.
Most of these changes seem like improvements. The PDB thing had a TODO
saying it used to crash; I anticipate it works now, we'll see what CI
does.
The `std.os.uefi` field renames are a notable breaking change.
because it marks the linker section, preventing garbage collection.
Also, name the members because that is required by this intrinsic.
Also, enable the StackDepth option in the sancov pass as a workaround
for https://github.com/llvm/llvm-project/pull/106464, otherwise, LLVM
enables TracePCGuard even though we explicitly disable it.
This matches what LLVM's sancov pass does and is required so that
optimization passes do not delete the instrumentation.
However, this is currently triggering an error: "members of
llvm.compiler.used must be named" so the next commit will add names to
those globals.
It's useful to have TraceCmp based on the results of LLVM optimizations,
while the code coverage bits were emitted by Zig manually, allowing more
careful correlation to points of interest in the source code.
This re-enables the sancov pass in `-ffuzz` mode, but only TraceCmp.
Notably, IndirectCalls is off, which needs to be implemented manually in
the LLVM backend, and StackDepth remains off, because it is not used by
libfuzzer or AFL either.
If stack depth is re-introduced, it can be done with better performance
characteristics by being function call graph aware, and only lowered in
call graph cycles, where its heuristic properties come in useful.
Fixes the fuzzing regression.
instead of relying on the LLVM sancov pass. The LLVM pass is still
executed if trace_pc_guard is requested, disabled otherwise. The LLVM
backend emits the instrumentation directly.
It uses `__sancov_pcs1` symbol name instead of `__sancov_pcs` because
each element is 1 usize instead of 2.
AIR: add CoveragePoint to branch hints which indicates whether those
branches are interesting for code coverage purposes.
Update libfuzzer to use the new instrumentation. It's simplified since
we no longer need the constructor and the pcs are now in a continguous
list.
This is a regression in the fuzzing functionality because the
instrumentation for comparisons is no longer emitted, resulting in worse
fuzzer inputs generated. A future commit will add that instrumentation
back.
These won't live in the parent namespace as decls which causes problems
later in this function, and tests are guaranteed not to be referenced at
comptime anyway, so there's actually no need to run this logic.
The compiler actually doesn't need any functional changes for this: Sema
does reification based on the tag indices of `std.builtin.Type` already!
So, no zig1.wasm update is necessary.
This change is necessary to disallow name clashes between fields and
decls on a type, which is a prerequisite of #9938.
Grepping for `NO_THUMB` in glibc suggests that glibc does not actually support
pure Thumb-2 mode. This is the mode that is implied by these target triples;
mixed Arm/Thumb mode should just use the regular `arm*-linux-gnueabi*` triples.
Pointer subtraction on `void *` or function pointers is UB by the C
spec, but is permitted by GCC and Clang as an extension. So, avoid
crashing translate-c in such cases, and follow the extension behavior --
there's nothing else that could really be intended.
Implements the accepted proposal to introduce `@branchHint`. This
builtin is permitted as the first statement of a block if that block is
the direct body of any of the following:
* a function (*not* a `test`)
* either branch of an `if`
* the RHS of a `catch` or `orelse`
* a `switch` prong
* an `or` or `and` expression
It lowers to the ZIR instruction `extended(branch_hint(...))`. When Sema
encounters this instruction, it sets `sema.branch_hint` appropriately,
and `zirCondBr` etc are expected to reset this value as necessary. The
state is on `Sema` rather than `Block` to make it automatically
propagate up non-conditional blocks without special handling. If
`@panic` is reached, the branch hint is set to `.cold` if none was
already set; similarly, error branches get a hint of `.unlikely` if no
hint is explicitly provided. If a condition is comptime-known, `cold`
hints from the taken branch are allowed to propagate up, but other hints
are discarded. This is because a `likely`/`unlikely` hint just indicates
the direction this branch is likely to go, which is redundant
information when the branch is known at comptime; but `cold` hints
indicate that control flow is unlikely to ever reach this branch,
meaning if the branch is always taken from its parent, then the parent
is also unlikely to ever be reached.
This branch information is stored in AIR `cond_br` and `switch_br`. In
addition, `try` and `try_ptr` instructions have variants `try_cold` and
`try_ptr_cold` which indicate that the error case is cold (rather than
just unlikely); this is reachable through e.g. `errdefer unreachable` or
`errdefer @panic("")`.
A new API `unwrapSwitch` is introduced to `Air` to make it more
convenient to access `switch_br` instructions. In time, I plan to update
all AIR instructions to be accessed via an `unwrap` method which returns
a convenient tagged union a la `InternPool.indexToKey`.
The LLVM backend lowers branch hints for conditional branches and
switches as follows:
* If any branch is marked `unpredictable`, the instruction is marked
`!unpredictable`.
* Any branch which is marked as `cold` gets a
`llvm.assume(i1 true) [ "cold"() ]` call to mark the code path cold.
* If any branch is marked `likely` or `unlikely`, branch weight metadata
is attached with `!prof`. Likely branches get a weight of 2000, and
unlikely branches a weight of 1. In `switch` statements, un-annotated
branches get a weight of 1000 as a "middle ground" hint, since there
could be likely *and* unlikely *and* un-annotated branches.
For functions, a `cold` hint corresponds to the `cold` function
attribute, and other hints are currently ignored -- as far as I can tell
LLVM doesn't really have a way to lower them. (Ideally, we would want
the branch hint given in the function to propagate to call sites.)
The compiler and standard library do not yet use this new builtin.
Resolves: #21148
mlugg: this is cherry-picked from Andrew's nosanitize branch (with
Jacob's fixes squashed in) since I needed this for `unpredictable` and
`prof` metadata. The nosanitize-specific changes are reverted in the
next commit.
Co-authored-by: Jacob Young <jacobly0@users.noreply.github.com>
before this, calls to `resolveTypeFieldsStruct` (now renamed to the more correct `resolveStructFieldTypes`) would just throw away the sema that `resolveStructInner` created and create its own. There is no reason to do this, and we fix it to preserve the sema through it all.
My main gripes with this design were that it was incorrectly namespaced, the naming was inconsistent and a bit wrong (`fooAlign` vs `fooAlignment`).
This commit moves all the logic from `PerThread.zig` to use the zcu + tid system that the previous couple commits introduce.
I've organized and merged the functions to be a bit more specific to their own purpose.
- `fieldAlignment` takes a struct or union type, an index, and a Zcu (or the Sema version which takes a Pt), and gives you the alignment of the field at the index.
- `structFieldAlignment` takes the field type itself, and provides the logic to handle special cases, such as externs.
A design goal I had in mind was to avoid using the word 'struct' in the function name, when it worked for things that aren't structs, such as unions.
I don't recall why I put these checks here -- they aren't correct. We
can freely recreate a type even if its fields have changed, because we
are going to re-do all type resolution.
The only conditions for recreations are (a) the ZIR index must not be
lost and (b) the number of captures must be the same. These conditions
are permissible because if either is violated, we can guarantee that
analysis of a valid `zirStructDecl` (etc) will never reference this
type (since the ZIR index has just been tracked, and the captures have
just been created based on the ZIR).
Adds a corresponding test case.
Resolves: #21185
Implements the base that should usually work that is
- Check LD_LIBRARY_PATH if the binary is no setuid setgid binary
- Check /lib, /usr/lib, in that order
The missing parts are:
- DT_RPATH and DT_RUNPATH handling from the calling executable
- Reading /etc/ld.so.cache
For more details check man page of dlopen(3)
This reverts commit cb5a6be41a.
I deeply apologize for the churn.
This change is problematic given that we do not have ranged integers
(yet? see #3806).
In the meantime, this type needs to be `usize`, matching the length and
index types for all std lib data structures.
Users who want to save memory should not use heap-allocated BoundedArray
values, since it is inherently memory-inefficient. Use a different
memory layout instead.
If #3806 is accepted and implemented, the length value can become an
integer with the appropriate range, without the footgun. If that
proposal is not accepted, len type will remain a usize.
Ensures that all runned command are visible when using `--verbose` flag,
for example `pkg-config` from Step.Compile or `git describe` from build.zig.
Signed-off-by: Eric Joldasov <bratishkaerik@landless-city.net>
This is necessary to inform the real, non-stub glibc that a program built with
Zig is using a modern `FILE` structure, i.e. glibc 2.1+. This is particularly
important on lesser-used architectures where the legacy code is poorly tested;
for example, glibc 2.40 introduced a regression for the legacy case in the
libio cleanup code, causing all Zig-compiled MIPS binaries to crash on exit.
These are fundamentally incapable of producing accurate information for reasons
I've laid out in #20771. Since our only use of these functions is to check that
object files have the correct machine type, and since #21020 made
`std.Target.to{Coff,Elf}Machine()` more accurate, just switch these checks over
to that and compare the machine type tags instead.
Closes#20771.
* reduce iteration cost by not tracking unused entries
* avoid emitting unused abbrevs to `.debug_abbrev`
* get the compiler executable passing `llvm-dwarfdump --verify`
* make it possible to skip `.debug_line` padding much more quickly
* Indices of referenced captures
* Line and column of `@src()`
The second point aligns with a reversal of the "incremental compilation"
section of https://github.com/ziglang/zig/issues/2029#issuecomment-645793168.
This reversal was already done as #17688 (46a6d50), with the idea to
push incremental compilation down the line. My proposal is to keep it as
comptime-known, and simply re-analyze uses of `@src()` whenever their
line/column change.
I think this decision is reasonable for a few reasons:
* The Zig compiler is quite fast. Occasionally re-analyzing a few
functions containing `@src()` calls is perfectly acceptable and won't
noticably impact update times.
* The system described by Andrew in #2029 is currently vaporware.
* The system described by Andrew in #2029 is non-trivial to implement.
In particular, it requires some way to have backends update a single
global in certain cases, without re-doing semantic analysis. There is
no other part of incremental compilation which requires this.
* Having `@src().line` be comptime-known is useful. For instance, #17688
was justified by broken Tracy integration because the source line
couldn't be comptime-known.
In a `memoized_call`, store how many backwards braches the call
performs. Add this to `sema.branch_count` when using a memoized call. If
this exceeds the quota, perform a non-memoized call to get a correct
"exceeded X backwards branches" error.
Also, do not memoize calls which do `@setEvalBranchQuota` or similar, as
this affects global state which must apply to the caller.
Change some eval branch quotas so that the compiler itself still builds correctly.
This commit manually changes a file in Aro which is automatically
generated. The sources which generate the file are not in this repo.
Upstream Aro should make the suitable changes on their end before the
next sync of Aro sources into the Zig repo.
Closes#21132
According to the XDG Base Directory specification
(https://specifications.freedesktop.org/basedir-spec/latest/#variables),
empty values for these environment variables should be treated the same
as if they are unset. Specifically, for the instances changed in this
commit,
> $XDG_DATA_HOME defines the base directory relative to which
> user-specific data files should be stored. If $XDG_DATA_HOME is either
> not set **or empty**, a default equal to $HOME/.local/share should be
> used.
and
> $XDG_CACHE_HOME defines the base directory relative to which
> user-specific non-essential data files should be stored. If
> $XDG_CACHE_HOME is either not set **or empty**, a default equal to
> $HOME/.cache should be used.
(emphasis mine)
In addition to the case mentioned in the linked issue, all other uses of
XDG environment variables were corrected.
* std.c.darwin: add missing CPUFAMILY fields
* std.zig.system.detectNativeCpuAndFeatures: add missing darwin fields
* add comment so the prong isnt lost and easily discoverable during next llvm upgrade
This type is exactly the same as std.Build.Cache.Path, except for
one function which is not used anymore. Therefore we can replace
it without consequences.
A compilation build step for which the binary is not required could not
be compiled previously. There were 2 issues that caused this:
- The compiler communicated only the results of the emitted binary and
did not properly communicate the result if the binary was not emitted.
This is fixed by communicating the final hash of the artifact path (the
hash of the corresponding /o/<hash> directory) and communicating this
instead of the entire path. This changes the zig build --listen protocol
to communicate hashes instead of paths, and emit_bin_path is accordingly
renamed to emit_digest.
- There was an error related to the default llvm object path when
CacheUse.Whole was selected. I'm not really sure why this didn't manifest
when the binary is also emitted.
This was fixed by improving the path handling related to flush() and
emitLlvmObject().
In general, this commit also improves some of the path handling throughout
the compiler and standard library.
This replaces the constant `Zir.Inst.Ref` tags (and the analagous tags
in `Air.Inst.Ref`, `InternPool.Index`) referring to types in
`std.builtin` with a ZIR instruction `extended(builtin_type(...))` which
instructs Sema to fetch such a type, effectively as if it were a
shorthand for the ZIR for `@import("std").builtin.xyz`.
Previously, this was achieved through constant tags in `Ref`. The
analagous `InternPool` indices began as `simple_type` values, and were
later rewritten to the correct type information. This system was kind of
brittle, and more importantly, isn't compatible with incremental
compilation of std, since incremental compilation relies on the ability
to recreate types at different indices when they change. Replacing the
old system with this instruction slightly increases the size of ZIR, but
it simplifies logic and allows incremental compilation to work correctly
on the standard library.
This shouldn't have a significant impact on ZIR size or compiler
performance, but I will take measurements in the PR to confirm this.
Without this, incremental updates which would change inferred error sets
fail, since they assume the IES is resolved and equals the old set,
resulting in false positive compile errors when e.g. coercing to an IES.
The kernel sets r7 to 0 (success) or -1 (error), and stores the result in r2.
When r7 is -1 and the result is positive, it needs to be negated to get the
errno value that higher-level code, such as errnoFromSyscall(), expects to see.
The old code was missing the check that r2 is positive, but was also doing the
r7 check incorrectly; since it can only be set to 0 or -1, the blez instruction
would always branch.
In practice, this fix is necessary for e.g. the ENOSYS error to be interpreted
correctly. This manifested as hitting an unreachable branch when calling
process_vm_readv() in std.debug.MemoryAccessor.
This function now has to allocate anyway to resolve references, so we
may as well just build the error bundle and check its length.
Also remove some unnecessary calls of this function for efficiency.
The old logic here had bitrotted, largely because there were some
incorrect `else` cases. This is now implemented correctly for all
current ZIR instructions. This prevents instructions being lost in
incremental updates, which is important for updates to be minimal.
Another big commit, sorry! This commit makes all fixes necessary for
incremental updates of the compiler itself (specifically, adding a
breakpoint to `zirCompileLog`) to succeed, at least on the frontend.
The biggest change here is a reform to how types are handled. It works
like this:
* When a type is first created in `zirStructDecl` etc, its namespace is
scanned. If the type requires resolution, an `interned` dependency is
declared for the containing `AnalUnit`.
* `zirThis` also declared an `interned` dependency for its `AnalUnit` on
the namespace's owner type.
* If the type's namespace changes, the surrounding source declaration
changes hash, so `zirStructDecl` etc will be hit again. We check
whether the namespace has been scanned this generation, and re-scan it
if not.
* Namespace lookups also check whether the namespace in question
requires a re-scan based on the generation. This is because there's no
guarantee that the `zirStructDecl` is re-analyzed before the namespace
lookup is re-analyzed.
* If a type's structure (essentially its fields) change, then the type's
`Cau` is considered outdated. When the type is re-analyzed due to
being outdated, or the `zirStructDecl` is re-analyzed by being
transitively outdated, or a corresponding `zirThis` is re-analyzed by
being transitively outdated, the struct type is recreated at a new
`InternPool` index. The namespace's owner is updated (but not
re-scanned, since that is handled by the mechanisms above), and the
old type, while remaining a valid `Index`, is removed from the map
metadata so it will never be found by lookups. `zirStructDecl` and
`zirThis` store an `interned` dependency on the *new* type.
When a type becomes outdated, there will still be lingering references
to the old index -- for instance, any declaration whose value was that
type holds a reference to that index. These references may live for an
arbitrarily long time in some cases. So, we can't just remove the type
from the pool -- the old `Index` must remain valid!
Instead, we want to preserve the old `Index`, but avoid it from ever
appearing in lookups. (It's okay if analysis of something referencing
the old `Index` does weird stuff -- such analysis are guaranteed by the
incremental compilation model to always be unreferenced.) So, we use the
new `InternPool.putKeyReplace` to replace the shard entry for this index
with the newly-created index.
An enum type is kind of like a struct or union type, in that field
errors are happening during type resolution. The only difference is that
type resolution happens at the time the type is created. So, errors in
fields should not cause the type to be deleted: we've already added a
reference entry, and incremenetal dependencies which must be invalidated
if the compile error is fixed. Once we call `WipEnumType.prepare`, we
should never call `WipEnumType.cancel`. This is analagous to logic for
enum declarations in `Sema.zirEnumDecl`.
Two fixes here.
* Prevent a crash when sorting the list of analysis errors when some
errors refer to lost source locations. These errors can be sorted
anywhere in the list, because they are (in theory) guaranteed to never
be emitted by the `resolveReferences` logic. This case occurs, for
instance, when a declaration has compile errors in the initial update
and is deleted in the second update.
* Prevent a crash when resolving the source location for `entire_file`
errors for a non-existent file. This is the bug underlying #20954.
Resolves: #20954.
This commit updates `Zcu.resolveReferences` to traverse the graph of
`AnalUnit` references (starting from the 1-3 roots of analysis) in order
to determine which `AnalUnit`s are referenced in an update. Errors for
unreferenced entities are omitted from the error bundle. However, note
that unreferenced `Nav`s are not removed from the binary.
This commit makes more progress towards incremental compilation, fixing
some crashes in the frontend. Notably, it fixes the regressions introduced
by #20964. It also cleans up the "outdated file root" mechanism, by
virtue of deleting it: we now detect outdated file roots just after
updating ZIR refs, and re-scan their namespaces.
This fixes the failure to find CLANG_LIBRARIES on debian, which packages
the relevant .so file at these paths:
libclang-cpp18: /usr/lib/llvm-18/lib/libclang-cpp.so.18.1
libclang-cpp18: /usr/lib/x86_64-linux-gnu/libclang-cpp.so.18.1
libclang-cpp18: /usr/lib/x86_64-linux-gnu/libclang-cpp.so.18
(The latter two paths are symlinks to the first.)
Simplifies code in docs creation where we used `std.tar.output.Header`.
Writer uses that Header internally and provides higher level interface.
Updates checksum on write, handles long file names, allows setting mtime and file permission mode. Provides handy interface for passing `Dir.WalkerEntry`.
For csky, we can just always do the gb initialization. For riscv, the gp code is
temporarily pulled above the main switch until the blocking issue is resolved.
It's entirely unclear whether this should map to POWERPC or POWERPCFP, and as I
can find no evidence of people producing PE files for PowerPC since Windows NT,
let's just not make a likely-wrong guess. We can revisit this if the need ever
actually arises.
All of these were mapping to types that are little endian. In fact, I can find
no evidence that either Windows or UEFI have ever been used on big endian
systems.
Using --watch I noticed a couple of issues with my initial attempt. 1) The index I used as 'completion key' was not stable over time, when directories are being added/removed the key no longer corresponds with the intended dir. 2) There exists a race condition in which we receive a completion notification for a directory that was removed. My solution is to generate a key value and associate it with each Directory.
This is the initial implementation of Jacob Young's idea of
re-using old function slots as trampolines for new function's
location. This way the trampoline is guaranteed to be aligned
to the function's alignment.
The only edge case is if an incremental update further overaligns
the function in which case we skip/delete the trampoline and
re-evaluate all references.
This is useful during porting work where you're mainly concerned with tests that
e.g. run under QEMU. Combine with the new -Dtest-target-filter for an even more
streamlined workflow.
Two fixes here:
Sort by addresses after generating the line table. Debug information in
the wild is not sorted and the rest of the implementation requires this
data to be sorted.
Handle DW.LNE.end_sequence correctly. When I originally wrote this code,
I misunderstood what this opcode was supposed to do. Now I understand
that it marks the *end* of an address range, meaning the current address
does *not* map to the current line information.
This fixes source location information for a big chunk of ReleaseSafe
code.
matching the default of clang's behavior. I originally put them in
registerOptimizerEarlyEPCallback because I thought clang was doing that,
but I see now it is behind the flag `--sanitizer-early-opt-ep` which is
disabled by default.
The implementation assumed that compilation units did not overlap, which
is not the case. The new implementation uses .debug_ranges to iterate
over the requested PCs.
This partially resolves#20990. The dump-cov tool is fixed but the same
fix needs to be applied to `std.Build.Fuzz.WebServer` (sorting the PC
list before passing it to be resolved by debug info).
I am observing LLVM emit multiple 8-bit counters for the same PC
addresses when enabling `-fsanitize-coverage=inline-8bit-counters`. This
seems like a bug in LLVM. I can't fathom why that would be desireable.
Some of this is arbitrary since spirv (as opposed to spirv32/spirv64) refers to
the version with logical memory layout, i.e. no 'real' pointers. This change at
least matches what clang does.
Versions can simply use the normal version range mechanism, or alternatively an
Abi tag if that makes more sense. For now, we only care about 4.5 anyway.
Eliding the namespace when a container type has no decls was an
experiment in saving memory, but it ended up causing more trouble than
it was worth in various places. So, take the small memory hit for
reified types, and just give every container type a namespace.
The type `Zcu.Decl` in the compiler is problematic: over time it has
gained many responsibilities. Every source declaration, container type,
generic instantiation, and `@extern` has a `Decl`. The functions of
these `Decl`s are in some cases entirely disjoint.
After careful analysis, I determined that the two main responsibilities
of `Decl` are as follows:
* A `Decl` acts as the "subject" of semantic analysis at comptime. A
single unit of analysis is either a runtime function body, or a
`Decl`. It registers incremental dependencies, tracks analysis errors,
etc.
* A `Decl` acts as a "global variable": a pointer to it is consistent,
and it may be lowered to a specific symbol by the codegen backend.
This commit eliminates `Decl` and introduces new types to model these
responsibilities: `Cau` (Comptime Analysis Unit) and `Nav` (Named
Addressable Value).
Every source declaration, and every container type requiring resolution
(so *not* including `opaque`), has a `Cau`. For a source declaration,
this `Cau` performs the resolution of its value. (When #131 is
implemented, it is unsolved whether type and value resolution will share
a `Cau` or have two distinct `Cau`s.) For a type, this `Cau` is the
context in which type resolution occurs.
Every non-`comptime` source declaration, every generic instantiation,
and every distinct `extern` has a `Nav`. These are sent to codegen/link:
the backends by definition do not care about `Cau`s.
This commit has some minor technically-breaking changes surrounding
`usingnamespace`. I don't think they'll impact anyone, since the changes
are fixes around semantics which were previously inconsistent (the
behavior changed depending on hashmap iteration order!).
Aside from that, this changeset has no significant user-facing changes.
Instead, it is an internal refactor which makes it easier to correctly
model the responsibilities of different objects, particularly regarding
incremental compilation. The performance impact should be negligible,
but I will take measurements before merging this work into `master`.
Co-authored-by: Jacob Young <jacobly0@users.noreply.github.com>
Co-authored-by: Jakub Konka <kubkon@jakubkonka.com>
Prior to this change, we would unconditionally create a symbol and atom
pair for a Decl that could point to an extern, in which case no atom
can be created.
std.crypto has quite a few instances of breaking naming conventions.
This is the beginning of an effort to address that.
Deprecates `std.crypto.utils`.
* Upgrade from u8 to usize element types.
- WebAssembly assumes u64. It should probably try to be target-aware
instead.
* Move the covered PC bits to after the header so it goes on the same
page with the other rapidly changing memory (the header stats).
depends on the semantics of accepted proposal #19755closes#20994
Some projects, such as Cap'n Proto, use .c++ as their filenames. Without
this, compiling them fails because zig c++ will fall back to using the
linker.
The previous mecanism for linux distributions to delivers debug info into `/usr/lib/debug` no longer seems in use.
the current mecanism often is using `debuginfod` (https://sourceware.org/elfutils/Debuginfod.html)
This commit only tries to load already available debuginfo but does not try to make any download requests.
the user can manually run `debuginfod-find debuginfo PATH` to populate the cache.
- look up the debuglink file in the directory of the executable file (instead of the cwd)
- fix parsing of debuglink section (the 4-byte alignement is within the file, unrelated to the in-memory address)
The signature is documented as:
int link(const char *, const char *);
(see https://man7.org/linux/man-pages/man2/link.2.html or https://man.netbsd.org/link.2)
And its not some Linux extension, the [syscall
implementation](21b136cc63/fs/namei.c (L4794-L4797))
only expects two arguments too.
It probably *should* have a flags parameter, but its too late now.
I am a bit surprised that linking glibc or musl against code that invokes
a 'link' with three parameters doesn't fail (at least, I couldn't get any
local test cases to trigger a compile or link error).
The test case in std/posix/test.zig is currently disabled, but if I
manually enable it, it works with this change.
Before this commit, the name `v9.5a` was being used for two different features, and one was overwriting the other in the `all_features` array.
`arrowlake_s` is an alias for `arrowlake-s`
Due to the `std.crypto.ecdsa.KeyPair.create` taking and optional of seed, even if the seed is generated, cross-compiling to the environments without standard random source (eg. wasm) (`std.crypto.random.bytes`) will fail to compile.
This commit changes the API of the problematic function and moves the random seed generation to a new utility function.
During the LLVM 18 upgrade, two changes were made that changed `@alignOf(u64)` to 4 for the x86-windows target:
- `Type.maxIntAlignment` was made to return 16 for x86 (200e06b). Before that commit, `maxIntAlignment` was 8 for windows/uefi and 4 for everything else
- `Type.intAbiAlignment` was made to return 4 for 33...64 (7e1cba7 + e89d6fc). Before those commits, `intAbiAlignment` would return 8, since the maxIntAlignment for x86-windows was 8 (and for other targets, the `maxIntAlignment` of 4 would clamp the `intAbiAlignment` to 4)
`src/codegen/llvm.zig` has its own alignment calculations that no longer match the values returned from the `Type` functions. For the x86-windows target, this loop:
ddcb7b1c11/src/codegen/llvm.zig (L558-L567)
when the `size` is 64 will set `abi` and `pref` to 64 (meaning an align of 8 bytes), which doesn't match the `Type` alignment of 4.
This commit makes `Type.intAbiAlignment` match the alignment calculated in `codegen/llvm.zig`.
Fixes#20047Fixes#20466Fixes#20469
this fix bypasses the slice bounds, reading garbage data for up to the
last 7 bits (which are technically supposed to be ignored). that's going
to need to be fixed, let's fix that along with switching from byte elems
to usize elems.
* libfuzzer: track unique runs instead of deduplicated runs
- easier for consumers to notice when to recheck the covered bits.
* move common definitions to `std.Build.Fuzz.abi`.
build runner sends all the information needed to fuzzer web interface
client needed in order to display inline coverage information along with
source code.
* libfuzzer: close file after mmap
* fuzzer/main.js: connect with EventSource and debug dump the messages.
currently this prints how many fuzzer runs have been attempted to
console.log.
* extract some `std.debug.Info` logic into `std.debug.Coverage`.
Prepares for consolidation across multiple different executables which
share source files, and makes it possible to send all the
PC/SourceLocation mapping data with 4 memcpy'd arrays.
* std.Build.Fuzz:
- spawn a thread to watch the message queue and signal event
subscribers.
- track coverage map data
- respond to /events URL with EventSource messages on a timer
* std.debug.Dwarf: add `sortCompileUnits` along with a field to track
the state for the purpose of assertions and correct API usage.
This makes batch lookups faster.
- in the future, findCompileUnit should be enhanced to rely on sorted
compile units as well.
* implement `std.debug.Dwarf.resolveSourceLocations` as well as
`std.debug.Info.resolveSourceLocations`. It's still pretty slow, since
it calls getLineNumberInfo for each array element, repeating a lot of
work unnecessarily.
* integrate these APIs with `std.Progress` to understand what is taking
so long.
The output I'm seeing from this tool shows a lot of missing source
locations. In particular, the main area of interest is missing for my
tokenizer fuzzing example.
with debug info resolved.
begin efforts of providing `std.debug.Info`, a cross-platform
abstraction for loading debug information into an in-memory format that
supports queries such as "what is the source location of this virtual
memory address?"
Unlike `std.debug.SelfInfo`, this API does not assume the debug
information in question happens to match the host CPU architecture, OS,
or other target properties.
* new .zig-cache subdirectory: 'v'
- stores coverage information with filename of hash of PCs that want
coverage. This hash is a hex encoding of the 64-bit coverage ID.
* build runner
* fixed bug in file system inputs when a compile step has an
overridden zig_lib_dir field set.
* set some std lib options optimized for the build runner
- no side channel mitigations
- no Transport Layer Security
- no crypto fork safety
* add a --port CLI arg for choosing the port the fuzzing web interface
listens on. it defaults to choosing a random open port.
* introduce a web server, and serve a basic single page application
- shares wasm code with autodocs
- assets are created live on request, for convenient development
experience. main.wasm is properly cached if nothing changes.
- sources.tar comes from file system inputs (introduced with the
`--watch` feature)
* receives coverage ID from test runner and sends it on a thread-safe
queue to the WebServer.
* test runner
- takes a zig cache directory argument now, for where to put coverage
information.
- sends coverage ID to parent process
* fuzzer
- puts its logs (in debug mode) in .zig-cache/tmp/libfuzzer.log
- computes coverage_id and makes it available with
`fuzzer_coverage_id` exported function.
- the memory-mapped coverage file is now namespaced by the coverage id
in hex encoding, in `.zig-cache/v`
* tokenizer
- add a fuzz test to check that several properties are upheld
When a unique run is encountered, track it in a bit set memory-mapped
into the fuzz directory so it can be observed by other processes, even
while the fuzzer is running.
This is what upstream's configure does.
Previously, we only disabled warnings in some musl compilations, with `rcrt1.o`
notably being one for which we didn't. This resulted in a warning in `dlstart.c`
which is included in `rcrt1.c`. So let's just be consistent and disable warnings
for all musl code.
Closes#13385.
This is arbitrary since spirv (as opposed to spirv32/spirv64) refers to the
version with logical memory layout, i.e. no 'real' pointers. This change at
least matches what clang does.
Prints _Static_asserts for the size and alignment of all the basic built-in C
types. The output can be run through a compiler for the specified target to
verify that Zig's values are the same as those used by a C compiler for the
target.
It is impossible to even build projects like glibc when targeting a generic
SPARC v8 CPU; LEON3 is effectively considered the baseline for `sparc-linux-gnu`
now, particularly due to it supporting a CASA instruction similar to the one in
SPARC v9. However, it's slightly incompatible with SPARC v9 due to having a
different ASI tag, so resulting binaries would not be portable to regular SPARC
CPUs. So, as the least bad option, make v9 the baseline for sparc32.
`__xl_a` is just a global variable containing a null function pointer. There's
nothing magical about it or its name at all.
The section names used on `__xl_a` and `__xl_b` (`.CRT$XLA` and `.CRT$XLZ`) are
the real magic here. The compiler emits TLS variables into `.CRT$XL<x>`
sections, where `x` is an uppercase letter between A and Z (exclusive). The
linker then sorts those sections alphabetically (due to the `$`), and the result
is a neat array of TLS initialization callbacks between `__xl_a` and `__xl_z`.
That array is null-terminated, though! Normally, `__xl_z` serves as the null
terminator; however, by pointing `AddressesOfCallBacks` to `__xl_a`, which just
contains a null function pointer, we've effectively made it so that the PE
loader will just immediately stop invoking TLS callbacks. Fix that by pointing
to the first actual TLS callback instead (or `__xl_z` if there are none).
If there is a VDSO, it will have clock_gettime(). The main thing we're concerned
about is architectures that don't have a VDSO at all, of which there are a few.
There are two concepts here: one for whether dwarf supports unwinding on
that target, and another for whether the Zig standard library
implements it yet.
...which have a ucontext_t but not a PC register. The current stack
unwinding implementation does not yet support this architecture.
Also fix name of `std.debug.SelfInfo.openSelf` to remove redundancy.
Also removed this hook into root providing an "openSelfDebugInfo"
function. Sorry, this debugging code is not of sufficient quality to
offer a plugin API right now.
This target triple was weird on multiple levels:
* The `ilp32` ABI is the soft float ABI. This is not the main ABI we want to
support on RISC-V; rather, we want `ilp32d`.
* `gnuilp32` is a bespoke tag that was introduced in Zig. The rest of the world
just uses `gnu` for RISC-V target triples.
* `gnu_ilp32` is already the name of an ILP32 ABI used on AArch64. `gnuilp32` is
too easy to confuse with this.
* We don't use this convention for `riscv64-linux-gnu`.
* Supporting all RISC-V ABIs with this convention will result in combinatorial
explosion; see #20690.
After this commit:
`std.debug.SelfInfo` is a cross-platform abstraction for the current
executable's own debug information, with a goal of minimal code bloat
and compilation speed penalty.
`std.debug.Dwarf` does not assume the current executable is itself the
thing being debugged, however, it does assume the debug info has the
same CPU architecture and OS as the current executable. It is planned to
remove this limitation.
This code has the hard-coded goal of supporting the executable's own
debug information and makes design choices along that goal, such as
memory-mapping the inputs, using dl_iterate_phdr, and doing conditional
compilation on the host target.
A more general-purpose implementation of debug information may be able
to share code with this, but there are some fundamental
incompatibilities. For example, the "SelfInfo" implementation wants to
avoid bloating the binary with PDB on POSIX systems, and likewise DWARF
on Windows systems, while a general-purpose implementation needs to
support both PDB and DWARF from the same binary. It might, for example,
inspect the debug information from a cross-compiled binary.
`SourceLocation` now lives at `std.debug.SourceLocation` and is
documented.
Deprecate `std.debug.runtime_safety` because it returns the optimization
mode of the standard library, when the caller probably wants to use the
optimization mode of their own module.
`std.pdb.Pdb` is moved to `std.debug.Pdb`, mirroring the recent
extraction of `std.debug.Dwarf` from `std.dwarf`.
I have no idea why we have both Module (with a Windows-specific
definition) and WindowsModule. I left some passive aggressive doc
comments to express my frustration.
std.debug.Dwarf is the parsing/decoding logic. std.dwarf remains the
unopinionated types and bits alone.
If you look at this diff you can see a lot less redundancy in
namespaces.
* Rename isPPC() -> isPowerPC32().
* Rename isPPC64() -> isPowerPC64().
* Add new isPowerPC() function which covers both.
There was confusion even in the standard library about what isPPC() meant. This
change makes these functions work how I think most people actually expect them
to work, and makes them consistent with isMIPS(), isSPARC(), etc.
I chose to rename from PPC to PowerPC because 1) it's more consistent with the
other functions, and 2) it'll cause loud rather than silent breakage for anyone
who might have been depending on isPPC() while misunderstanding it.
I pointed a fuzzer at the tokenizer and it crashed immediately. Upon
inspection, I was dissatisfied with the implementation. This commit
removes several mechanisms:
* Removes the "invalid byte" compile error note.
* Dramatically simplifies tokenizer recovery by making recovery always
occur at newlines, and never otherwise.
* Removes UTF-8 validation.
* Moves some character validation logic to `std.zig.parseCharLiteral`.
Removing UTF-8 validation is a regression of #663, however, the existing
implementation was already buggy. When adding this functionality back,
it must be fuzz-tested while checking the property that it matches an
independent Unicode validation implementation on the same file. While
we're at it, fuzzing should check the other properties of that proposal,
such as no ASCII control characters existing inside the source code.
Other changes included in this commit:
* Deprecate `std.unicode.utf8Decode` and its WTF-8 counterpart. This
function has an awkward API that is too easy to misuse.
* Make `utf8Decode2` and friends use arrays as parameters, eliminating a
runtime assertion in favor of using the type system.
After this commit, the crash found by fuzzing, which was
"\x07\xd5\x80\xc3=o\xda|a\xfc{\x9a\xec\x91\xdf\x0f\\\x1a^\xbe;\x8c\xbf\xee\xea"
no longer causes a crash. However, I did not feel the need to add this
test case because the simplified logic eradicates most crashes of this
nature.
This is a fairly small hobby OS that has not seen development in 2 years. Our
current policy is that hobby OSs should use the `other` tag.
https://github.com/zhmu/ananas
What is `sparcel`, you might ask? Good question!
If you take a peek in the SPARC v8 manual, §2.2, it is quite explicit that SPARC
v8 is a big-endian architecture. No little-endian or mixed-endian support to be
found here.
On the other hand, the SPARC v9 manual, in §3.2.1.2, states that it has support
for mixed-endian operation, with big-endian mode being the default.
Ok, so `sparcel` must just be referring to SPARC v9 running in little-endian
mode, surely?
Nope:
* 40b4fd7a3e/llvm/lib/Target/Sparc/SparcTargetMachine.cpp (L226)
* 40b4fd7a3e/llvm/lib/Target/Sparc/SparcTargetMachine.cpp (L104)
So, `sparcel` in LLVM is referring to some sort of fantastical little-endian
SPARC v8 architecture. I've scoured the internet and I can find absolutely no
evidence that such a thing exists or has ever existed. In fact, I can find no
evidence that a little-endian implementation of SPARC v9 ever existed, either.
Or any SPARC version, actually!
The support was added here: https://reviews.llvm.org/D8741
Notably, there is no mention whatsoever of what CPU this might be referring to,
and no justification given for the "but some are little" comment added in the
patch.
My best guess is that this might have been some private exercise in creating a
little-endian version of SPARC that never saw the light of day. Given that SPARC
v8 explicitly doesn't support little-endian operation (let alone little-endian
instruction encoding!), and no CPU is known to be implemented as such, I think
it's very reasonable for us to just remove this support.
* Elaborate on the sub-variants of Variant I.
* Clarify the use of the TCB term.
* Rename a bunch of stuff to be more accurate/descriptive.
* Follow Zig's style around namespacing more.
* Use a structure for the ABI TCB.
No functional change intended.
The code would cause LLVM to emit a jump table for the switch in the loop over
the dynamic tags. That jump table was far enough away that the compiler decided
to go through the GOT, which would of course break at this early stage as we
haven't applied MIPS's local GOT relocations yet, nor can we until we've walked
through the _DYNAMIC array.
The first attempt at rewriting this used code like this:
var sorted_dynv = [_]elf.Addr{0} ** elf.DT_NUM;
But this is also problematic as it results in a memcpy() call. Instead, we
explicitly initialize it to undefined and use a loop of volatile stores to
clear it.
This test was originally introduced in 5f38d6e2e9, where it looked like this:
test "cast *[1][*]const u8 to [*]const ?[*]const u8" {
const window_name = [1][*]const u8{c"window name"};
const x: [*]const ?[*]const u8 = &window_name;
assert(mem.eql(u8, std.cstr.toSliceConst(x[0].?), "window name"));
}
Over the years, this has become more and more obfuscated, to the point that the verbosity of the `expect` call overshadows the point of the example. This commit intends to update this test to match the spirit of the original version of the test, while shedding the obfuscation.
In https://github.com/ziglang/zig/pull/19641, this binding changed from `[*]u16` to `LPWSTR` which made it a sentinel-terminated pointer. This introduced a compiler error in the `std.os.windows.GetModuleFileNameW` wrapper since it takes a `[*]u16` pointer. This commit changes the binding back to what it was before instead of introducing a breaking change to `std.os.windows.GetModuleFileNameW`
Related: https://github.com/ziglang/zig/issues/20858
We advertise reproducible builds for release modes, so let's help users achieve
that in C/C++ code. Users can still override this manually if they really want.
This does not completely ignore static asserts - they are validated by aro
during parsing; any failures will render an error and non-zero exit code.
Emit a warning comment that _Static_asserts are not translated - this
matches the behavior of the existing clang-based translate-c.
Aro currently does not store source locations for _Static_assert
declarations so I've hard-coded token index 0 for now.
Accesses to this global variable can require relocations on some platforms (e.g.
MIPS). If we do it before PIE relocations have been applied, we'll crash.
It's actually important for the ABI that r25 (t9) contains the address of the
called function, so that this standard prologue sequence works:
lui $2, %hi(_gp_disp)
addiu $2, $2, %lo(_gp_disp)
addu $gp, $2, $t9
(This is a bit similar to the ToC situation on powerpc that was fixed in
7bc78967b400322a0fc5651f37a1b0428c37fb9d.)
statx() is strictly superior to stat() and friends. We can do this because the
standard library declares Linux 4.19 to be the minimum version supported in
std.Target. This is also necessary on riscv32 where there is only statx().
While here, I improved std.fs.File.metadata() to gather as much information as
possible when calling statx() since that is the expectation from this particular
API.
This is kind of a hack because the timespec in UAPI headers is actually still
32-bit while __kernel_timespec is 64-bit. But, importantly, all the syscalls
take __kernel_timespec from the get-go (because riscv32 support is so recent).
Defining our timespec this way will allow all the syscall wrappers in
std.os.linux to do the right thing for riscv32. For other 32-bit architectures,
we have to use the 64-bit time syscalls explicitly to solve the Y2038 problem.
loongarch64 syscalls not updated because it seems like that kernel port hasn't
been working for a year or so:
In file included from arch/loongarch/include/uapi/asm/unistd.h:5:
include/uapi/asm-generic/unistd.h:2:10: fatal error: 'asm/bitsperlong.h' file not found
That file is just missing from the tree. 🤷
If we're going to abuse the preprocessor, we may as well go all the way and have
it generate a convenient format for us. This achieves two things:
1. We no longer need hacks for the arch-specific syscalls.
2. We now generate the correct syscall names for 32-bit platforms.
The latter is because we now resolve __SC_3264, etc.
Deprecates std.fs.atomicSymLink and removes the allocator requirement
from the new std.fs.Dir.atomicSymLink. Replaces the two usages of this
within std.
I did not include the TODOs from the original code that were based
off of `switch (err) { ..., else => return err }` not having correct
inference that cases handled in `...` are impossible in the error
union return type because these are not specified in many places but
I can add them back if wanted.
Thank you @squeek502 for help with fixing buffer overflows!
The core functionalities are now in two general functions
`extremeInSubtreeOnDirection()` and `nextOnDirection()` so all the other
traversing functions (`getMin()`, `getMax()`, and `InorderIterator`) are
all just trivial calls to these core functions.
The added two functions `Node.next()` and `Node.prev()` are also just
trivial calls to these.
* std.Treap traversal direction: use u1 instead of usize.
* Treap: fix getMin() and getMax(), and add tests for them.
This is a misfeature that we inherited from LLVM:
* https://reviews.llvm.org/D61259
* https://reviews.llvm.org/D61939
(`aarch64_32` and `arm64_32` are equivalent.)
I truly have no idea why this triple passed review in LLVM. It is, to date, the
*only* tag in the architecture component that is not, in fact, an architecture.
In reality, it is just an ILP32 ABI for AArch64 (*not* AArch32).
The triples that use `aarch64_32` look like `aarch64_32-apple-watchos`. Yes,
that triple is exactly what you think; it has no ABI component. They really,
seriously did this.
Since only Apple could come up with silliness like this, it should come as no
surprise that no one else uses `aarch64_32`. Later on, a GNU ILP32 ABI for
AArch64 was developed, and support was added to LLVM:
* https://reviews.llvm.org/D94143
* https://reviews.llvm.org/D104931
Here, sanity seems to have prevailed, and a triple using this ABI looks like
`aarch64-linux-gnu_ilp32` as you would expect.
As can be seen from the diffs in this commit, there was plenty of confusion
throughout the Zig codebase about what exactly `aarch64_32` was. So let's just
remove it. In its place, we'll use `aarch64-watchos-ilp32`,
`aarch64-linux-gnuilp32`, and so on. We'll then translate these appropriately
when talking to LLVM. Hence, this commit adds the `ilp32` ABI tag (we already
have `gnuilp32`).
Contributes to #15607
Although the case is not handled in `openatWasi` (as I could not get a
working wasi environment to test the change) I have added a FIXME
addressing it and linking to the issue.
with this rewrite we can call functions inside of
inline assembly, enabling us to use the default start.zig logic
all that's left is to implement lr/sc loops for atomically manipulating
1 and 2 byte values, after which we can use the segfault handler logic.
I was doing duplicate work with `elemOffset` multiplying by the abi size and then the `ptr_add` `genBinOp` also multiplying.
This led to having writes happening in the wrong place.
the risc-v backend doesn't have `@cmpxchg*` implemented and so it can't use the hint that the current page-allocator uses.
this work-around branch can be removed when I implement the atomic built-in.
the csrs `avl` and `vtype` are considered caller-saved so it could have changed while inside of the function.
the easiest way to handle this is to just set the cached `vtype` and `avl` to null, so that the next time something
needs to set it, it'll emit an instruction instead of relying on a potentially invalid setting.
The flag makes compiler_rt and libfuzzer be in debug mode.
Also:
* fuzzer: override debug logs and disable debug logs for frequently
called functions
* std.Build.Fuzz: fix bug of rerunning the old unit test binary
* report errors from rebuilding the unit tests better
* link.Elf: additionally add tsan lib and fuzzer lib to the hash
This flag makes the build runner rebuild unit tests after the pipeline
finishes, if it finds any unit tests.
I did not make this integrate with file system watching yet.
The test runner is updated to detect which tests are fuzz tests.
Run step is updated to track which test indexes are fuzz tests.
Before, this code:
@setRuntimeSafety(false);
var arr: [38]elf.Addr = undefined;
would emit a call to memset() in the output code in Debug mode, while in all the
release modes, LLVM optimized the memset() out as expected. Emitting the call in
Debug mode is problematic in some contexts, e.g. in std.os.linux.start_pie where
we are not yet ready to correctly perform calls because relocations haven't been
applied yet, or in the early stages of a dynamic linker, etc.
Switches from using r1 as a temporary to r2. That way, we don't have to set the
`noat` assembler option. (r1 is the scratch register used by the assembler's
pseudoinstructions; the assembler warns when code uses that register explicitly
without `noat` set.)
The `TargetOptions` default constructor initializes all `bool`s to
`false`, yet clang defaults to setting this option to `true`. Since
recent glibc versions on linux do not appear to support this being set
to `false`, just changing the default for now unless a use case for
making it configurable is found.
PR [19271](https://github.com/ziglang/zig/pull/19271) added some static function implementations from kernel32, but some parts of the library still used the dynamically loaded versions.
* Add -f(no-)sanitize-coverage-trace-pc-guard CLI flag which defaults to
off. This value lowers to TracePCGuard = true (LLVM backend) and -Xclang
-fsanitize-coverage-trace-pc-guard. These settings are not
automatically included with -ffuzz.
* Add `Build.Step.Compile` flag for sanitize_coverage_trace_pc_guard
with appropriate documentation.
* Add `zig cc` integration for the respective flags.
* Avoid crashing in ELF linker code when -ffuzz -femit-llvm-ir used
together.
Exposes sanitizer coverage flags to the target machine emit function.
Makes it easier to change sancov options without rebuilding the C++
files.
This also enables PCTable = true for sancov which is needed by AFL, and
adds the corresponding Clang flag.
PR https://github.com/ziglang/zig/pull/20679 ("std.c reorganization")
switched feature-detection code to use "T != void" checks in place of
"@hasDecl". However, the std.posix.system struct is empty, so
compile-time feature detection against symbols in there (specifically
`std.posix.system.ucontext_t` in this case), fail at compile time on
freestanding targets.
This PR adds a void ucontext_t into the std.posix.system default.
This PR also adds pseudo-"freestanding" variation of the StackIterator
"unwind" test. It is sort of hacky (its freestanding, but assumes it can
invoke a Linux exit syscall), but it does detect this problem.
Fixes#20710
This prevents it from trying to access thread local storage before it
has set up thread local storage, particularly when code coverage
instrumentation is enabled.
* Add the `-ffuzz` and `-fno-fuzz` CLI arguments.
* Detect fuzz testing flags from zig cc.
* Set the correct clang flags when fuzz testing is requested. It can be
combined with TSAN and UBSAN.
* Compilation: build fuzzer library when needed which is currently an
empty zig file.
* Add optforfuzzing to every function in the llvm backend for modules
that have requested fuzzing.
* In ZigLLVMTargetMachineEmitToFile, add the optimization passes for
sanitizer coverage.
* std.mem.eql uses a naive implementation optimized for fuzzing when
builtin.fuzz is true.
Tracked by #20702
In the case that the allocator is unavailable (OOM, etc.), we can
possibly still output the panic message - so now we stack allocate the
message and copy it to the exit data for passing to boot services.
The previous version of this function referenced the argc_argv_ptr global
variable as an inline asm operand. This caused LLVM to generate prologue code to
initialize the ToC so that the global variable can actually be accessed.
Ordinarily, there's nothing wrong with that. But _start() is a naked function!
This makes it actually super surprising that LLVM did this. It also means that
the old version only really worked by accident.
Once the reference to the global variable was removed, no ToC was set up, thus
violating the calling convention once we got to posixCallMainAndExit(). This
then caused any attempt to access global variables here to crash - namely when
setting std.os.linux.elf_aux_maybe.
The fix is to just initialize the ToC manually in _start().
This was added as an architecture to LLVM's target triple parser and the Clang
driver in 2015. No backend ever materialized as far as I can see (same for GCC).
In 2016, other code referring to it started using "Myriad" instead. Ultimately,
all code related to it that isn't in the target triple parser was removed. It
seems to be a real product, just... literally no one seems to know anything
about the ISA. I figure after almost a decade with no public ISA documentation
to speak of, and no LLVM backend to reference, it's probably safe to assume that
we're not going to learn much about this ISA, making it useless for Zig.
See: 1b5767f72b
See: 84a7564b28
See: 8cfe9d8f2a
This is problematic for PIE. There's nothing but luck preventing the accesses to
this global variable from requiring relocations. I've observed this being an
issue on MIPS and PowerPC personally, but others may be affected.
Besides, we're really just passing the initial stack pointer value to
posixCallMainAndExit(), so... just do that.
The set of signals that cannot have their action changed is documented in POSIX,
and any additional, non-standard signals are documented by the specific OS. I
see no valid reason why EINVAL should be considered an unpredictable error here.
`PKG_CONFIG` environment variable is used to override path to
pkg-config executable, for example when it's name is prepended by
target triple for cross-compilation purposes:
```
PKG_CONFIG=/usr/bin/aarch64-unknown-linux-gnu-pkgconf zig build
```
Signed-off-by: Eric Joldasov <bratishkaerik@landless-city.net>
On decryption tls client should remove zero byte padding after the
content type field. This padding is rarely used, the only site (from the
list of top domains) that I found using it is `tutanota.com`.
From [RFC](https://datatracker.ietf.org/doc/html/rfc8446#section-5.4):
> All encrypted TLS records can be padded.
> Padding is a string of zero-valued bytes appended to the ContentType
field before encryption.
> the receiving implementation scans the field from the end toward the
beginning until it finds a non-zero octet. This non-zero octet is the
content type of the message.
Currently we can't connect to that site:
```
$ zig run main.zig -- tutanota.com
error: TlsInitializationFailed
/usr/local/zig/zig-linux-x86_64-0.14.0-dev.208+854e86c56/lib/std/crypto/tls/Client.zig:476:45: 0x121fbed in init__anon_10331 (http_get_std)
if (inner_ct != .handshake) return error.TlsUnexpectedMessage;
^
/usr/local/zig/zig-linux-x86_64-0.14.0-dev.208+854e86c56/lib/std/http/Client.zig:1357:99: 0x1161f0b in connectTcp (http_get_std)
conn.data.tls_client.* = std.crypto.tls.Client.init(stream, client.ca_bundle, host) catch return error.TlsInitializationFailed;
^
/usr/local/zig/zig-linux-x86_64-0.14.0-dev.208+854e86c56/lib/std/http/Client.zig:1492:14: 0x11271e1 in connect (http_get_std)
} orelse return client.connectTcp(host, port, protocol);
^
/usr/local/zig/zig-linux-x86_64-0.14.0-dev.208+854e86c56/lib/std/http/Client.zig:1640:9: 0x111a24e in open (http_get_std)
try client.connect(valid_uri.host.?.raw, uriPort(valid_uri, protocol), protocol);
^
/home/ianic/Code/tls.zig/example/http_get_std.zig:28:19: 0x1118f8c in main (http_get_std)
var req = try client.open(.GET, uri, .{ .server_header_buffer = &server_header_buffer });
^
```
using this example:
```zig
const std = @import("std");
pub fn main() !void {
var gpa = std.heap.GeneralPurposeAllocator(.{}){};
const allocator = gpa.allocator();
const args = try std.process.argsAlloc(allocator);
defer std.process.argsFree(allocator, args);
if (args.len > 1) {
const domain = args[1];
var client: std.http.Client = .{ .allocator = allocator };
defer client.deinit();
// Add https:// prefix if needed
const url = brk: {
const scheme = "https://";
if (domain.len >= scheme.len and std.mem.eql(u8, domain[0..scheme.len], scheme))
break :brk domain;
var url_buf: [128]u8 = undefined;
break :brk try std.fmt.bufPrint(&url_buf, "https://{s}", .{domain});
};
const uri = try std.Uri.parse(url);
var server_header_buffer: [16 * 1024]u8 = undefined;
var req = try client.open(.GET, uri, .{ .server_header_buffer = &server_header_buffer });
defer req.deinit();
try req.send();
try req.wait();
}
}
```
This was used for LoongArch64, where:
* `gnuf64` -> `ilp32d` / `lp64d` (full hard float)
* `gnuf32` -> `ilp32f` / `lp64f` (hard float for `f32` only)
* `gnusf` -> `ilp32` / `lp64` (soft float)
But Loongson eventually settled on just `gnu` for the first case since that's
what most people will actually be targeting outside embedded scenarios. The
`gnuf32` and `gnusf` specifiers remain in use.
* common symbols are now public from std.c even if they live in
std.posix
* LOCK is now one of the common symbols since it is the same on 100% of
operating systems.
* flock is now void value on wasi and windows
* std.fs.Dir now uses flock being void as feature detection, avoiding
trying to call it on wasi and windows
Without this data, debugger expressions try to pass structs by-value,
which mostly just crashes.
Also: mark enums as enum classes to prevent the enumerators from
shadowing other identifiers.
Now we get working global variable lookup in GDB! LLDB still re-mangles,
and it looks like we can't do much about that for now.
Also: translate non-owning type declarations into typedefs.
This will allow accessing non-local declarations from debuggers, which,
AFAICT, was impossible before.
Getting scopes right already works for type declarations and functions,
but will need some fiddling for variables:
For those, I tried imitating what Clang does for static member
variables, but LLDB tries to re-mangle those and then fails at lookup,
while GDB outright crashes. Hopefully I can find some other dwarven
incantation to do the right thing.
It is now composed of these main sections:
* Declarations that are shared among all operating systems.
* Declarations that have the same name, but different type signatures
depending on the operating system. Often multiple operating systems
share the same type signatures however.
* Declarations that are specific to a single operating system.
- These are imported one per line so you can see where they come from,
protected by a comptime block to prevent accessing the wrong one.
Closes#19352 by changing the convention to making types `void` and
functions `{}`, so that it becomes possible to update `@hasDecl` sites
to use `@TypeOf(f) != void` or `T != void`. Happily, this ended up
removing some duplicate logic and update some bitrotted feature
detection checks.
A handful of types have been modified to gain namespacing and type
safety. This is a breaking change.
Oh, and the last usage of `usingnamespace` site is eliminated.
By allowing finishAir to handle .stack results, we simplify a lot of code in
air*** functions, which try to handle this case. Also this changes will result in optimization, if one of operands is dead after instruction its place could be reused by result.
The only downside to this change is that finishAir now can return error, though it handled by returning finishAir result, because it always needs to be final in air*** functions.
Additionally I migrated WValue{ to .{ inside CodeGen.zig.
* format: fix default character when no alignment
When no alignment is specified, the character that should be used is the
fill character that is otherwise provided, not space.
This is closer to the default that C programmers (and other languages)
use: "04x" fills with zeroes (in zig as of today x:04 fills with spaces)
Test:
const std = @import("std");
const expectFmt = std.testing.expectFmt;
test "fmt.defaultchar.no-alignment" {
// as of today the following test passes:
try expectFmt("0x00ff", "0x{x:0>4}", .{255});
// as of today the following test fails (returns "0x ff" instead)
try expectFmt("0x00ff", "0x{x:04}", .{255});
}
* non breaking improvement of string formatting
* improved comment
* simplify the code a little
* small improvement around how characters identified as valid are consumed
To facilitate #1840, this commit slims `std.windows.kernel32` to only
have the functions needed by the standard library. Since this will break
projects that relied on these, I offer two solutions:
- Make an argument as to why certain functions should be added back in.
Note that they may just be wrappers around `ntdll` APIs, which would
go against #1840.
If necessary I'll add them back in *and* make wrappers in
`std.windows` for it.
- Maintain your own list of APIs. This is the option taken by bun[1],
where they wrap functions with tracing.
- Use `zigwin32`.
I've also added TODO comments that specify which functions can be
reimplemented using `ntdll` APIs in the future.
Other changes:
- Group functions into groups (I/O, process management etc.).
- Synchronize definitions against Microsoft documentation to use the
proper parameter types/names.
- Break all functions with parameters over multiple lines.
This eliminates the statically-reachable recursion loop between code
generation backends and Sema. This is beneficial for optimizers
(although I do not measure any performance improvement for this change),
and for profilers.
They are implementation-defined and can have values other than
hard-coded here. Also, standard permits other values not mentioned
there:
> Additional macro definitions, beginning with the characters LC_
> and an uppercase letter, may also be specified by the implementation.
Signed-off-by: Eric Joldasov <bratishkaerik@landless-city.net>
According to https://en.cppreference.com/mwiki/index.php?title=c/locale/setlocale&oldid=171500 ,
`setlocale` "returns null value on failure":
> Return value
> pointer to a narrow null-terminated string identifying the C locale
> after applying the changes, if any, or null pointer on failure.
Example program:
```zig
const std = @import("std");
pub fn main() void {
const ptr = std.c.setlocale(.ALL, "non_existent");
std.debug.print("ptr = {d}\n", .{@intFromPtr(ptr)});
}
```
Output:
```console
ptr = 0
```
Signed-off-by: Eric Joldasov <bratishkaerik@landless-city.net>
Instead of calling the dynamically loaded kernel32.GetLastError, we can extract it from the TEB.
As shown by [Wine](34b1606019/include/winternl.h (L439)), the last error lives at offset 0x34 of the TEB in 32-bit Windows and at offset 0x68 in 64-bit Windows.
* the file's doc-comment was misleading and did not focus on the correct aspect of SIMD
* added cpu flag awareness to `suggestVectorLengthForCpu` in order to provide a more accurate vector length
Now we generate debug undefined constants when the user asks for them to dedup across the function decl. This takes 2 instructions instead of 7 in the RISC-V backend.
TODO, we need to dedupe across function decl boundaries.
Compilation errors now report a failure on rebuilds triggered by file
system watches.
Compiler crashes now report failure correctly on rebuilds triggered by
file system watches.
The compiler subprocess is restarted if a broken pipe is encountered on
a rebuild.
Remove --debug-incremental
This flag is also added to the build system. Importantly, this tells
Compile step whether or not to keep the compiler running between
rebuilds. It defaults off because it is currently crashing
zirUpdateRefs.
Changes the `make` function signature to take an options struct, which
additionally includes `watch: bool`. I intentionally am not exposing
this information to configure phase logic.
Also adds global zig cache to the compiler cache prefixes.
Closes#20600
Now that we use the PEB to get the precise length of the command line string, there's no need for a multi-item pointer/sliceTo call. This provides a minor speedup:
Benchmark 1 (153 runs): benchargv-before.exe
measurement mean ± σ min … max outliers delta
wall_time 32.7ms ± 429us 32.1ms … 36.9ms 1 ( 1%) 0%
peak_rss 6.49MB ± 5.62KB 6.46MB … 6.49MB 14 ( 9%) 0%
Benchmark 2 (157 runs): benchargv-after.exe
measurement mean ± σ min … max outliers delta
wall_time 31.9ms ± 236us 31.4ms … 32.7ms 4 ( 3%) ⚡- 2.4% ± 0.2%
peak_rss 6.49MB ± 4.77KB 6.46MB … 6.49MB 14 ( 9%) + 0.0% ± 0.0%
Previously, to ensure args were encoded as well-formed WTF-8 (i.e. no encoded surrogate pairs), the code unit would be encoded and then the last 6 emitted bytes would be checked to see if they were a surrogate pair, and this was done for any emitted code unit (although this was not necessary, it should have only been done when emitting a low surrogate).
After this commit, we still want to ensure well-formed WTF-8, but, to do so, the last emitted code point is stored, meaning we can just directly check that the last code unit is a high surrogate and the current code unit is a low surrogate to determine if we have a surrogate pair.
This provides some performance benefit over and above a "use the same strategy as before but only check when we're emitting a low surrogate" implementation:
Benchmark 1 (111 runs): benchargv-master.exe
measurement mean ± σ min … max outliers delta
wall_time 45.2ms ± 532us 44.5ms … 49.4ms 2 ( 2%) 0%
peak_rss 6.49MB ± 3.94KB 6.46MB … 6.49MB 10 ( 9%) 0%
Benchmark 2 (154 runs): benchargv-storelast.exe
measurement mean ± σ min … max outliers delta
wall_time 32.6ms ± 293us 32.2ms … 34.2ms 8 ( 5%) ⚡- 27.8% ± 0.2%
peak_rss 6.49MB ± 5.15KB 6.46MB … 6.49MB 15 (10%) - 0.0% ± 0.0%
Benchmark 3 (131 runs): benchargv-onlylow.exe
measurement mean ± σ min … max outliers delta
wall_time 38.4ms ± 257us 37.9ms … 39.6ms 5 ( 4%) ⚡- 15.1% ± 0.2%
peak_rss 6.49MB ± 5.70KB 6.46MB … 6.49MB 9 ( 7%) - 0.0% ± 0.0%
Before this commit, the WTF-16 command line string would be converted to WTF-8 in `init`, and then a second buffer of the WTF-8 size + 1 would be allocated to store the parsed arguments. The converted WTF-8 command line would then be parsed and the relevant bytes would be copied into the argument buffer before being returned.
After this commit, only the WTF-8 size of the WTF-16 string is calculated (without conversion) which is then used to allocate the buffer for the parsed arguments. Parsing is then done on the WTF-16 slice directly, with the arguments being converted to WTF-8 on-the-fly.
This has a few (minor) benefits:
- Cuts the amount of memory allocated by ArgIteratorWindows in half (or better)
- Makes the total amount of memory allocated by ArgIteratorWindows predictable, since, before, the upfront `wtf16LeToWtf8Alloc` call could end up allocating more-memory-than-necessary temporarily due to its internal use of an ArrayList. Now, the amount of memory allocated is always exactly `calcWtf8Len(cmd_line) + 1`.
This was causing zig2.exe to crash during bootstrap, because there was an atomic
load of read-only memory, and the attempt to write to it as part of the (idempotent)
atomic exchange was invalid.
Aligned reads (of u32 / u64) are atomic on x86 / x64, so this is replaced with an
optimization-proof load (`__iso_volatile_load8*`) and a reordering barrier.
This allows the mutate mutex to only be locked during actual grows,
which are rare. For the lists that didn't previously have a mutex, this
change has little effect since grows are rare and there is zero
contention on a mutex that is only ever locked by one thread. This
change allows `extra` to be mutated without racing with a grow.
Makes the build runner compile successfully for non-linux targets;
printing an error if you ask for --watch rather than making build
scripts fail to compile.
it's not advertised in the usage and only available in debug builds of
the compiler. Makes it easier to test changes to the build runner that
might affect targets differently.
Updates the build runner to unconditionally require a zig lib directory
parameter. This parameter is needed in order to correctly understand
file system inputs from zig compiler subprocesses, since they will refer
to "the zig lib directory", and the build runner needs to place file
system watches on directories in there.
The build runner's fanotify file watching implementation now accounts
for when two or more Cache.Path instances compare unequal but ultimately
refer to the same directory in the file system.
Breaking change: std.Build no longer has a zig_lib_dir field. Instead,
there is the Graph zig_lib_directory field, and individual Compile steps
can still have their zig lib directories overridden. I think this is
unlikely to break anyone's build in practice.
The compiler now sends a "file_system_inputs" message to the build
runner which shares the full set of files that were added to the cache
system with the build system, so that the build runner can watch
properly and redo the Compile step. This is implemented for whole cache
mode but not yet for incremental cache mode.
and add file system watching integration.
`addDirectoryWatchInput` now returns a `bool` which helps remind the
caller to
1. call addDirectoryWatchInputFromPath on any derived paths
2. but only if the dependency is not already captured by a step
dependency edge.
The make function now recursively walks all directories and adds the
found files to the cache hash rather than incorrectly only adding the
directory name to the cache hash.
closes#20571
and deprecate `addFile`. Part of an effort to move towards using
`std.Build.Cache.Path` abstraction in more places, which makes it easier
to avoid absolute paths and path resolution.
And use it to implement InstallDir Step watch integration.
I'm not seeing any events triggered when I run `mkdir` in the watched
directory, however, and I have not yet figured out why.
The goal is to move towards using `std.Build.Cache.Path` instead of
absolute path names.
This was helpful for implementing file watching integration to
the InstallDir Step
This has been planned for quite some time; this commit finally does it.
Also implements file system watching integration in the make()
implementation for UpdateSourceFiles and fixes the reporting of step
caching for both.
WriteFile does not yet have file system watching integration.
So far, only implemented for InstallFile steps.
Default debounce interval bumped to 50ms. I think it should be
configurable.
Next I have an idea to simplify the fanotify implementation, but other
OS implementations might want to refer back to this commit before I make
those changes.
I'm still learning how the fanotify API works but I think after playing
with it in this commit, I finally know how to implement it, at least on
Linux. This commit does not accomplish the goal but I want to take the
code in a different direction and still be able to reference this point
in time by viewing a source control diff.
I think the move is going to be saving the file_handle for the parent
directory, which combined with the dirent names is how we can correlate
the events back to the Step instances that have registered file system
inputs. I predict this to be similar to implementations on other
operating systems.
When calculating how much ciphertext from the stream can fit into
user and internal buffers we should also take into account ciphertext
data which are already in internal buffer.
Fixes: 15226
Tested with
[this](https://github.com/ziglang/zig/issues/15226#issuecomment-2218809140).
Using client with different read buffers until I, hopefully, understood
what is happening.
Not relevant to this fix, but this
[part](95d9292a7a/lib/std/crypto/tls/Client.zig (L988-L991))
is still mystery to me. Why we don't use free_size in buf_cap
calculation. Seems like rudiment from previous implementation without iovec.
* Update `__chkstk_ms` to have weak linkage
`__chkstk_ms` was causing conflicts during linking in some circumstances (specifically with linking object files from Rust sources). This PR switches `__chkstk_ms` to have weak linkage.
#15107
* Update stack_probe.zig to weak linkage for all symbols
Note that the original `cgroup_storage` MapType has been deprecated,
so renamed to `cgroup_storage_deprecated`.
Signed-off-by: Tw <tw19881113@gmail.com>
This makes comparing host name with dns name from certificate case
insensitive.
I found a few domains (from the
[cloudflare](https://radar.cloudflare.com/domains) list of top domains)
for which tls.Client fails to connect. Error is:
```zig
error: TlsInitializationFailed
Code/zig/lib/std/crypto/Certificate.zig:336:9: 0x1177b1f in verifyHostName (http_get_std)
return error.CertificateHostMismatch;
Code/zig/lib/std/crypto/tls23/handshake_client.zig:461:25: 0x11752bd in parseServerCertificate (http_get_std)
try subject.verifyHostName(opt.host);
```
In its certificate this domains have host names which are not strictly
lower case. This is what checkHostName is comparing:
|host_name | dns_name |
|------------------------------------------------|
|ey.com | EY.COM |
|truist.com | Truist.com |
|wscampanhas.bradesco | WSCAMPANHAS.BRADESCO |
|dell.com | Dell.com |
From
[RFC2818](https://datatracker.ietf.org/doc/html/rfc2818#section-2.4):
> Matching is performed using the matching rules specified by
[RFC2459].
From [RFC2459](https://datatracker.ietf.org/doc/html/rfc2459#section-4.2.1.7):
> When comparing URIs, conforming implementations
> MUST compare the scheme and host without regard to case, but assume
> the remainder of the scheme-specific-part is case sensitive.
Testing with:
```
const std = @import("std");
pub fn main() !void {
var gpa = std.heap.GeneralPurposeAllocator(.{}){};
const allocator = gpa.allocator();
const args = try std.process.argsAlloc(allocator);
defer std.process.argsFree(allocator, args);
if (args.len > 1) {
const domain = args[1];
var client: std.http.Client = .{ .allocator = allocator };
defer client.deinit();
// Add https:// prefix if needed
const url = brk: {
const scheme = "https://";
if (domain.len >= scheme.len and std.mem.eql(u8, domain[0..scheme.len], scheme))
break :brk domain;
var url_buf: [128]u8 = undefined;
break :brk try std.fmt.bufPrint(&url_buf, "https://{s}", .{domain});
};
const uri = try std.Uri.parse(url);
var server_header_buffer: [16 * 1024]u8 = undefined;
var req = try client.open(.GET, uri, .{ .server_header_buffer = &server_header_buffer });
defer req.deinit();
try req.send();
try req.wait();
}
}
```
`$ zig run example/main.zig -- truist.com `
The old heuristic of checking only for the number of fields has the
downside of classifying all opaque types, such as `std.c.FILE`, as
"namespaces" rather than "types".
Line `link_directories("${CMAKE_PREFIX_PATH}/lib")` was evaluated as
`link_directories("/lib")` in the default case of `CMAKE_PREFIX_PATH`
being empty.
This caused cmake to add `-L/lib -Wl,-rpath,/lib` to the zig2
build flags.
This could result in errors on systems where libraries set via
`CMAKE_LIBRARY_PATH` had conflicting versions in `/lib`:
- `-L/lib` could cause linking zig2 to fail
- `-Wl,-rpath,/lib` adds `/lib` as the first entry of the zig2 `RPATH`.
This could cause running zig2 (to build zig3) to fail.
In case of conflicting lib dirs, cmake emitted this warning, which is
now fixed:
```
Cannot generate a safe runtime search path for target zig2 because files in
some directories may conflict with libraries in implicit directories:
runtime library [libclang-cpp.so.18.1] in /nix/store/...-clang-18.1.5-lib/lib may be hidden by files in:
/lib
```
The purpose of using path digest was to reference a file in a
serializable manner. Now that there is a stable index associated with
files, it is a superior way to accomplish that goal, since removes one
layer of indirection, and makes TrackedInst 8 bytes instead of 20.
The saved Zig Compiler State file for "hello world" goes from 1.3M to
1.2M with this change.
Primarily, this commit removes 2 fields from File, relying on the data
being stored in the `files` field, with the key as the path digest, and
the value as the struct decl corresponding to the File. This table is
serialized into the compiler state that survives between incremental
updates.
Meanwhile, the File struct remains ephemeral data that can be
reconstructed the first time it is needed by the compiler process, as
well as operated on by independent worker threads.
A key outcome of this commit is that there is now a stable index that
can be used to refer to a File. This will be needed when serializing
error messages to survive incremental compilation updates.
Note that the `_ = Address` statements in tests previously were a nop,
and now actually check that the type is valid. However, on WASI, the
type is *not* valid.
I'm so sorry.
This commit was just meant to be making all types fully resolve by
queueing resolution at the moment of their creation. Unfortunately, a
lot of dominoes ended up falling. Here's what happened:
* I added a work queue job to fully resolve a type.
* I realised that from here we could eliminate `Sema.types_to_resolve`
if we made function codegen a separate job. This is desirable for
simplicity of both spec and implementation.
* This led to a new AIR traversal to detect whether any required type is
unresolved. If a type in the AIR failed to resolve, then we can't run
codegen.
* Because full type resolution now occurs by the work queue job, a bug
was exposed whereby error messages for type resolution were associated
with the wrong `Decl`, resulting in duplicate error messages when the
type was also resolved "by" its owner `Decl` (which really *all*
resolution should be done on).
* A correct fix for this requires using a different `Sema` when
performing type resolution: we need a `Sema` owned by the type. Also
note that this fix is necessary for incremental compilation.
* This means a whole bunch of functions no longer need to take `Sema`s.
* First-order effects: `resolveTypeFields`, `resolveTypeLayout`, etc
* Second-order effects: `Type.abiAlignmentAdvanced`, `Value.orderAgainstZeroAdvanced`, etc
The end result of this is, in short, a more correct compiler and a
simpler language specification. This regressed a few error notes in the
test cases, but nothing that seems worth blocking this change.
Oh, also, I ripped out the old code in `test/src/Cases.zig` which
introduced a dependency on `Compilation`. This dependency was
problematic at best, and this code has been unused for a while. When we
re-enable incremental test cases, we must rewrite their executor to use
the compiler server protocol.
This change modifies `Zcu.ErrorMsg` to store a `Zcu.LazySrcLoc` rather
than a `Zcu.SrcLoc`. Everything else is dominoes.
The reason for this change is incremental compilation. If a failed
`AnalUnit` is up-to-date on an update, we want to re-use the old error
messages. However, the file containing the error location may have been
modified, and `SrcLoc` cannot survive such a modification. `LazySrcLoc`
is designed to be correct across incremental updates. Therefore, we
defer source location resolution until `Compilation` gathers the compile
errors into the `ErrorBundle`.
Previously, `reference_table` mapped from a `Decl` being referenced to
the `Decl` that performed the reference. This is convenient for
constructing error messages, but problematic for incremental
compilation. This is because on an incremental update, we want to
efficiently remove all references triggered by an `AnalUnit` which is
being re-analyzed.
For this reason, `reference_table` now maps the other way: from the
`AnalUnit` *performing* the reference, to the `AnalUnit` whose analysis
was triggered. As a general rule, any call to any of the following
functions should be preceded by a call to `Sema.addReferenceEntry`:
* `Zcu.ensureDeclAnalyzed`
* `Sema.ensureDeclAnalyzed`
* `Zcu.ensureFuncBodyAnalyzed`
* `Zcu.ensureFuncBodyAnalysisQueued`
This is not just important for error messages, but also more
fundamentally for incremental compilation. When an incremental update
occurs, we must determine whether any `AnalUnit` has become
unreferenced: in this case, we should ignore its associated error
messages, and perhaps even remove it from the binary. For this reason,
we no longer store only one reference to every `AnalUnit`, but every
reference. At the end of an update, `Zcu.resolveReferences` will
construct the reverse mapping, and as such identify which `AnalUnit`s
are still referenced. The current implementation doesn't quite do what
we need for incremental compilation here, but the framework is in place.
Note that `Zcu.resolveReferences` does constitute a non-trivial amount
of work on every incremental update. However, for incremental
compilation, this work -- which will effectively be a graph traversal
over all `AnalUnit` references -- seems strictly necessary. At the
moment, this work is only done if the `Zcu` has any errors, when
collecting them into the final `ErrorBundle`.
An unsolved problem here is how to represent inline function calls in
the reference trace. If `foo` performs an inline call to `bar` which
references `qux`, then ideally, `bar` would be shown on the reference
trace between `foo` and `qux`, but this is not currently the case. The
solution here is probably for `Zcu.Reference` to store information about
the source locations of active inline calls betweeen the referencer and
its reference.
This change seeks to more appropriately model the way semantic analysis
works by drawing a more clear line between errors emitted by analyzing a
`Decl` (in future a `Cau`) and errors emitted by analyzing a runtime
function.
This does change a few compile errors surrounding compile logs by adding
more "also here" notes. The new notes are more technically correct, but
perhaps not so helpful. They're not doing enough harm for me to put
extensive thought into this for now.
This commit reworks our representation of exported Decls and values in
Zcu to be memory-optimized and trivially serialized.
All exports are now stored in the `all_exports` array on `Zcu`. An
`AnalUnit` which performs an export (either through an `export`
annotation or by containing an analyzed `@export`) gains an entry into
`single_exports` if it performs only one export, or `multi_exports` if
it performs multiple.
We no longer store a persistent mapping from a `Decl`/value to all
exports of that entity; this state is not necessary for the majority of
the pipeline. Instead, we construct it in `Zcu.processExports`, just
before flush. This does not affect the algorithmic complexity of
`processExports`, since this function already iterates all exports in
the `Zcu`.
The elimination of `decl_exports` and `value_exports` led to a few
non-trivial backend changes. The LLVM backend has been wrangled into a
more reasonable state in general regarding exports and externs. The C
backend is currently disabled in this commit, because its support for
`export` was quite broken, and that was exposed by this work -- I'm
hoping @jacobly0 will be able to pick this up!
Adds a missing call to addLazyPathDependenciesOnly in
std.Build.Module.addCSourceFiles. Also fixes an issue in
std.Build.Step.WriteFile where it wasn't updating all the GeneratedFile
instances for every directory. To fix the second issue, I removed
all the GeneratedFile instances and now all files/directories reference
the steps main GeneratedFile via sub paths.
This was the only kind of error which was raised in pipeToFileSystem and
not added to Diagnostics.
Shell tar silently ignores paths which are stripped out when used with
`--strip-components` switch. This enables that same behavior, errors
will be collected in diagnostics but caller is free to ignore that type
of diagnostics errors.
Enables use case where caller knows structure of the tar file and want
to extract only some deeply nested folders ignoring upper files/folders.
Fixes: #17620 by giving caller options:
- not provide diagnostic and get errors
- provide diagnostics and analyze errors
- provide diagnostics and ignore errors
The eventfd system call and dup3 library call have been available
since FreeBSD 13 and 10 respectively, and are thus available in
all [FreeBSD releases not deemed EOL](<https://endoflife.date/freebsd>)
The lack of these were discovered when porting a terminal emulator
to FreeBSD. It would be nice to have them included in Zig's stdlib.
Apple has already dropped support for macOS 11.
GitHub Actions is dropping macOS 11 support now.
The Zig project is also dropping macOS 11 support now.
This is essentially just a rename. I also changed the representation of
`AnalSubject` to use a `packed struct` rather than a non-exhaustive
enum, but that change is relatively trivial.
This patch is a pure rename plus only changing the file path in
`@import` sites, so it is expected to not create version control
conflicts, even when rebasing.
I renamed std.process.Child.CreateProcessSupportedExtension to WindowsExtension
and made it public to avoid duplicating the list of extensions.
While here, I also improved it to not misreport OOM from std.fs.realpathAlloc()
as a generic failure to find the program, but instead panic like the rest of the
build system does for OOM.
Closes#20314.
In my first [try](https://github.com/ziglang/zig/pull/20224) to fix
20212 I didn't reproduce bug on required kernel (6.9.2) and wrongly
concluded that first two completions have different order on newer
kernel.
On my current kernel (6.5.0) order of completions is: send1, recv,
send2. On 6.9.2 order is send1, send2, recv. This fix allows second two
completions to arrive in any order.
Tested on both kernels.
Fixes: #20212
Stability of std sort was undertested before this change. Add a fuzz
test for more confidence.
Specifically, we used to have a single example test that used an array
of eight elements. That ends up exercising only a tiny fraction of
sorting logic, as it hits a hard-coded sorting network due to small
size.
LLVM fails to notice that in release builds, `logFn` ignores its
arguments, so their computation can be elided. So, LLVM fails to elide
this hashmap lookup. Its cost isn't too significant, but doing it in the
hottest loop in Sema adds up!
Technically, we could do the lookup a single time, before the loop, but
it was cleanest (and a little faster) to just disable this log call at
comptime when debug logging is disabled.
Using std.os.linux directly in e.g. std.posix.timerfd_create() causes
the function to compile but silently fail at runtime when targeting any
OS other than Linux.
To catch errors like this at compile time, std.os.linux must only be
directly accessed within std.posix where there has been a comptime check
that the target os is in fact Linux.
switch from `inline for` with `std.mem.eql`
to `inline else` and tag comparison;
expectEqualDeep(Inner) was already doing this.
add a previously-failing test case.
Since we track `reify` instructions across incremental updates, it is
acceptable to treat it as the baseline for a relative source location.
This turns out to be a good idea, since it makes it easy to define the
source location for a reified type.
I believe this was accidentally broken when the E enum for errno values
was introduces. These functions are quite the special case in that they
return the error value directly rather than returning -1 and passing the
error value through the errno variable.
In any case, using a u16 as the return type at the ABI boundary where a
c_int is expected is asking for trouble.
When analyzing `zirFunc` to instantiate a generic function,
`sema.owner_decl` is not the owner Decl of the generic instance, but
instead of the call site, so that dependencies are propagated correctly.
(This aligns with the fact that in future, generic instantiations will
not have a corresponding `Cau`.) So, when deciding the callconv in this
case, we must check `sema.generic_owner` to determine whether the
function is exported.
Using zig cc to compile and run wasm2c on zig.wasm on Windows triggers
what appears to be a sanitizer crash. The FuncGen reuse array pointer is
initialized to null and at some point it's resized to a length of zero,
which triggers this code to execute:
memcpy(&self->reuse[self->reuse_i], &self->reuse[reuse_top], sizeof(uint32_t) * reuse_n);
Given the current values, this equates to:
memcpy(&(NULL)[0], &(NULL)[0], 0);
Taking the address of the first element of a null pointer doesn't trigger
any actual runtime problem, since the pointer won't be dereferenced because
were passing 0 as the length to memcpy, however, it seems that the C spec
considers indexing a null pointer to be undefined behavior even if you
don't use the resulting value (or are just taking the address of an
indexed pointer).
tidy gives a false positive:
line 304 column 9 - Warning: moved <style> tag to <head>! fix-style-tags: no to avoid.
I noticed that `--show-warnings no` still incorrectly causes exit code 1.
I was unable to find an alternative to tidy.
There was already `zig build test-fmt` but now `zig build test` depends
on that one.
The CI scripts no longer need explicit logic since they already do
`zig build test`.
Now there is `captureChildProcess` which gives access to the
`std.process.Child.RunResult`, which is useful for accessing the stdout.
It also accepts and passes an optional `std.Progress.Node` to the child.
There were two primary issues at play here:
1. The hex float prefix was not handled correctly when the stream was
reset for the fallback parsing path, which occured when the mantissa was
longer max mantissa digits.
2. The implied exponent was not adjusted for hex-floats in this branch.
Additionally, some of the float parsing routines have been condensed, making
use of comptime.
closes#20275
🦀 src_decl is gone 🦀
This commit eliminates the `src_decl` field from `Sema.Block`. This
change goes further to eliminating unnecessary responsibilities of
`Decl` in preparation for its major upcoming refactor.
The two main remaining reponsibilities had to do with namespace types:
`src_decl` was used to determine their line number and their name. The
former use case is solved by storing the line number alongside type
declarations (and reifications) in ZIR; this is actually more correct,
since previously the line number assigned to the type was really the
line number of the source declaration it was syntactically contained
within, which does not necessarily line up. Consequently, this change
makes debug info for namespace types more correct, although I am not
sure how debuggers actually utilize this line number, if at all. Naming
types was solved by a new field on `Block`, called `type_name_ctx`. In a
sense, it represents the "namespace" we are currently within, including
comptime function calls etc. We might want to revisit this in future,
since the type naming rules seem to be a bit hand-wavey right now.
As far as I can tell, there isn't any more preliminary work needed for
me to start work on the behemoth task of splitting `Zcu.Decl` into the
new `Nav` (Named Addressable Value) and `Cau` (Comptime Analysis Unit)
types. This will be a sweeping change, impacting essentially every part
of the pipeline after `AstGen`.
`LazySrcLoc` now stores a reference to the "base AST node" to which it
is relative. The previous tagged union is `LazySrcLoc.Offset`. To make
working with this structure convenient, `Sema.Block` contains a
convenience `src` method which takes an `Offset` and returns a
`LazySrcLoc`.
The "base node" of a source location is no longer given by a `Decl`, but
rather a `TrackedInst` representing either a `declaration`,
`struct_decl`, `union_decl`, `enum_decl`, or `opaque_decl`. This is a
more appropriate model, and removes an unnecessary responsibility from
`Decl` in preparation for the upcoming refactor which will split it into
`Nav` and `Cau`.
As a part of these `Decl` reworks, the `src_node` field is eliminated.
This change aids incremental compilation, and simplifies `Decl`. In some
cases -- particularly in backends -- the source location of a
declaration is desired. This was previously `Decl.srcLoc` and worked for
any `Decl`. Now, it is `Decl.navSrcLoc` in reference to the upcoming
refactor, since the set of `Decl`s this works for precisely corresponds
to what will in future become a `Nav` -- that is, source-level
declarations and generic function instantiations, but *not* type owner
Decls.
This commit introduces more tags to `LazySrcLoc.Offset` so as to
eliminate the concept of `error.NeededSourceLocation`. Now, `.unneeded`
should only be used to assert that an error path is unreachable. In the
future, uses of `.unneeded` can probably be replaced with `undefined`.
The `src_decl` field of `Sema.Block` no longer has a role in type
resolution. Its main remaining purpose is to handle namespacing of type
names. It will be eliminated entirely in a future commit to remove
another undue responsibility from `Decl`.
It is worth noting that in future, the `Zcu.SrcLoc` type should probably
be eliminated entirely in favour of storing `Zcu.LazySrcLoc` values.
This is because `Zcu.SrcLoc` is not valid across incremental updates,
and we want to be able to reuse error messages from previous updates
even if the source file in question changed. The error reporting logic
should instead simply resolve the location from the `LazySrcLoc` on the
fly.
This is in preparation for some upcoming changes to how we represent
source locations in the compiler. The bulk of the change here is dealing
with the removal of `src()` methods from `Zir` types.
The justification for using relative source nodes in ZIR is that it
allows source locations -- which may be serialized across incremental
updates -- to be relative to the source location of their containing
declaration. However, having those "baseline" instructions themselves be
relative to their own parent is counterproductive, since the source
location updating problem is only being moved to `Decl`. Storing the
absolute node here instead makes more sense, since it allows for this
source location update logic to be elided entirely in the future by
storing a `TrackedInst.Index` to resolve a source location relative to
rather than a `Decl.Index`.
Before this commit, the DIRECTORY_NOT_EMPTY/FILE_IS_A_DIRECTORY/NOT_A_DIRECTORY statuses were assumed only to be possible when using `FILE_RENAME_INFORMATION_EX` and `FILE_RENAME_POSIX_SEMANTICS`, but that has empirically been shown to be false; a networked samba share can return the DIRECTORY_NOT_EMPTY status from `FILE_RENAME_INFORMATION` (which doesn't support `FILE_RENAME_POSIX_SEMANTICS`).
`FILE_IS_A_DIRECTORY` and `NOT_A_DIRECTORY` were not proven to be possible, but they were also moved to the outer switch just in case.
Fixes#19785
Deprecated aliases that are now compile errors:
- `std.fs.MAX_PATH_BYTES` (renamed to `std.fs.max_path_bytes`)
- `std.mem.tokenize` (split into `tokenizeAny`, `tokenizeSequence`, `tokenizeScalar`)
- `std.mem.split` (split into `splitSequence`, `splitAny`, `splitScalar`)
- `std.mem.splitBackwards` (split into `splitBackwardsSequence`, `splitBackwardsAny`, `splitBackwardsScalar`)
- `std.unicode`
+ `utf16leToUtf8Alloc`, `utf16leToUtf8AllocZ`, `utf16leToUtf8`, `fmtUtf16le` (all renamed to have capitalized `Le`)
+ `utf8ToUtf16LeWithNull` (renamed to `utf8ToUtf16LeAllocZ`)
- `std.zig.CrossTarget` (moved to `std.Target.Query`)
Deprecated `lib/std/std.zig` decls were deleted instead of made a `@compileError` because the `refAllDecls` in the test block would trigger the `@compileError`. The deleted top-level `std` namespaces are:
- `std.rand` (renamed to `std.Random`)
- `std.TailQueue` (renamed to `std.DoublyLinkedList`)
- `std.ChildProcess` (renamed/moved to `std.process.Child`)
This is not exhaustive. Deprecated aliases that I didn't touch:
+ `std.io.*`
+ `std.Build.*`
+ `std.builtin.Mode`
+ `std.zig.c_translation.CIntLiteralRadix`
+ anything in `src/`
Reorganize how the binOp and genBinOp functions work.
I've spent quite a while here reading exactly through the spec and so many
tests are enabled because of several critical issues the old design had.
There are some regressions that will take a long time to figure out individually
so I will ignore them for now, and pray they get fixed by themselves. When
we're closer to 100% passing is when I will start diving into them one-by-one.
Very similar reasoning to the Wasm backend. I believe that "Self" is
not the most descriptive possible name here and "Func" better explains it.
The generation is happening for a Function, and accessing "Func" is like accessing
the context of that current function.
what was happening is that instructions like `lb` were only affecting the lower bytes of the register and leaving the top dirty. this would lead to situtations were `cmp_eq` for example was using `xor`, which was failing because of the left-over stuff in the top of the register.
with this commit, we now zero out or truncate depending on the context, to ensure instructions like xor will provide proper results.
- implements `airSlice`, `airBitAnd`, `airBitOr`, `airShr`.
- got a basic design going for the `airErrorName` but for some reason it simply returns
empty bytes. will investigate further.
- only generating `.got.zig` entries when not compiling an object or shared library
- reduced the total amount of ops a mnemonic can have to 3, simplifying the logic
The doc comments for this global said:
"Locked to avoid interleaving panic messages from multiple threads."
Huh? There's already a mutex for that, it's the stderr mutex. Lock that
one instead.
The surrogate code points U+D800 to U+DFFF are valid code points but are not Unicode scalar values. This commit makes the error message more accurately reflect what is actually allowed in `\u` escape sequences.
From https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf:
> D71 High-surrogate code point: A Unicode code point in the range U+D800 to U+DBFF.
> D73 Low-surrogate code point: A Unicode code point in the range U+DC00 to U+DFFF.
>
> 3.9 Unicode Encoding Forms
> D76 Unicode scalar value: Any Unicode code point except high-surrogate and low-surrogate code points.
Related: #20270
A node may be freed during the execution of this loop, causing there to
be a parent reference to a nonexistent node. Without this assignment,
this would lead to the map entry containing stale data. By assigning
none, the child node with the bad parent pointer will be harmlessly
omitted from the tree.
Closes#20262
The old vectorization helper (WipElementWise) was clunky and a bit
annoying to use, and it wasn't really flexible enough.
This introduces a new vectorization helper, which uses Temporary and
Operation types to deduce a Vectorization to perform the operation
in a reasonably efficient manner. It removes the outer loop
required by WipElementWise so that implementations of AIR instructions
are cleaner. This helps with sanity when we start to introduce support
for composite integers.
airShift, convertToDirect, convertToIndirect, and normalize are initially
implemented using this new method.
Previously the child type of a vector was always in indirect representation.
Concretely, this meant that vectors of bools are represented by vectors
of u8.
This was undesirable because it introduced a difference between vectorizable
operations with a scalar bool and a vector of bool. This commit changes the
representation to be the same for vectors and scalars everywhere.
Some issues arised with constructing vectors: it seems the previous temporary-
and-pointer approach does not work properly with vectors of bool. To work around
this, simply use OpCompositeConstruct. This is the proper instruction for this,
but it was previously not used because of a now-solved limitation in the
SPIRV-LLVM-Translator. It was not yet applied to Zig because the Intel OpenCL
CPU runtime does not have a recent enough version of the translator yet, but
to solve that we just switch to testing with POCL instead.
Besides the Intel OpenCL CPU runtime, we can now run the
behavior tests using the Portable Computing Language. This
implementation is open-source, so it will be easier for us
to patch in updated versions of spirv-llvm-translator that
have bug fixes etc.
These instructions are not emitted by AstGen. They also would have no
effect even if they did appear in ZIR: the Sema handling for these
instructions creates a Decl which the name strategy is applied to, and
proceeds to never use it. This pointless CPU heater is now gone, saving
2 ZIR tags in the process.
`Elf*_Rela` relocations store their argument in `r_addend`, including for `R_*_RELATIVE` relocations. Unlike `Elf*_Rel` relocations, they are not applied as a delta to the destination virtual address. Instead, they are computed from `base_address + r_addend` directly.
We are posting two submission (zero copy send and receive) and then
reading two completions. There is no guarantee that those completions
will be in the order of submissions.
This test was expecting fist send completion then receive.
Fix is allowing them to come other way too.
* Skip building libcxx mt-only source files when single-threaded.
* This change is required for llvm18 libcxx.
* Add standalone test to link a trivial:
- mt-executable with libcxx
- st-executable with libcxx
This now defaults to false already since the autodocs rework.
The langref still cannot be enabled by default because the langref
contains doctests that exercise the `@cImport` feature which is disabled
in zig2 builds.
Had constrained the `aarch64_be` target, but not `aarch64`. This
constraint is necessary because earlier versions of glibc do not support
the aarch64 architecture.
Also, skip unsupported test cases.
glibc_runtime_check.c is a simple test case that exercises glibc functions
that might smoke out linking problems with Zig's C compiler. The
build.zig compiles it against a variety of glibc versions.
Also document and test glibc v2.2.5 (from 2002) as the oldest working
glibc target for C binaries.
The fstat,lstat,stat,mknod stubs used to build older (before v2.33)
glibc versions depend on the weak_hidden_alias macro. It was removed
from the glibc libc-symbols header, so patch it back in for the older
builds.
The scope of libc_nonshared.a was greatly changed in glibc 2.33 and
2.34, but only the change from 2.34 was reflected so far. Glibc 2.33
finally switched to versioned symbols for stat functions, meaning that
libc_nonshared.a no longer contains them since 2.33. Relevant files were
therefore reverted to 2.32 versions and renamed accordingly.
This commit also removes errno.c, which was probably added to
libc_nonshared.a based on a wrong assumption that glibc/include/errno.h
requires glibc/csu/errno.c. In reality errno.h should refer to
__libc_errno (not to be confused with the public __errno_location),
which should be imported from libc.so. The inclusion of errno.c resulted
in wrong compile options as well; this commit fixes them as well.
These are tripping on 32-bit x86 but are intended to prevent glibc
itself from being built with a bad configuration. Zig is only using this
file to create libc_nonshared.a, so it's not relevant.
This is the only place in all of glibc that this macro is referenced.
What is it doing? Only preventing fstatat.c from knowing the type
definition of `__time64_t`, apparently.
Fixes compilation of fstatat.c on 32-bit x86.
I could have just included the file from upstream glibc, but it was too
silly so I just inlined it. This patch could be dropped in a future
glibc update if desired. If omitted it will cause easily solvable
C compilation failures building glibc nonshared.
- `fcntl` was renamed to `fcntl64` in glibc 2.28 (see #9485)
- `res_{,n}{search,query,querydomain}` became "their own" symbols since
glibc 2.34: they were prefixed with `__` before.
This PR makes it possible to use `fcntl` with glibc 2.27 or older and
the `res_*` functions with glibc 2.33 or older.
These patches will become redundant with universal-headers and can be
dropped. But we have to do with what we have now.
- `fcntl` was renamed to `fcntl64` in glibc 2.28 (see #9485)
- `res_{,n}{search,query,querydomain}` became "their own" symbols since
glibc 2.34: they were prefixed with `__` before.
This PR makes it possible to use `fcntl` with glibc 2.27 or older and
the `res_*` functions with glibc 2.33 or older.
These patches will become redundant with universal-headers and can be
dropped. But we have to do with what we have now.
This is a patch to glibc features.h which makes
_DYNAMIC_STACK_SIZE_SOURCE undefined unless the version is >= 2.34.
This feature was introduced with glibc 2.34 and without this patch, code
built against these headers but then run on an older glibc will end up
making a call to sysconf() that returns -1 for the value of SIGSTKSZ
and MINSIGSTKSZ.
The actual `zig objcopy` does not accept keeping multiple sections.
If you pass multiple `-j .section` arguments to `zig objcopy`, it will
only respect the last one passed.
Originally I changed `zig objcopy` to accept multiple sections
and then concatenate them instead of returning after outputting the
first section (see emitElf) but I realized concatenating probably doesn't make sense.
Fixes a regression introduced in 67455c5e70. The `errdefer` cannot run since its not possible for an error to occur, and we don't want it to run on the last handle, so we move the closing back down to where it was before 67455c5e70.
Most of the functions related to points on the Edwards25519 curve
check that input points are not in a small-order subgroup.
They don't check that points are on the prime-order subgroup,
because this is expensive, and not always necessary.
However, applications may require such a check in order to
ensure that a public key is valid, and that a secret key counterpart
exists.
Many functions in the public API of libsodium related to arithmetic
over Edwards25519 also do that check unconditionally. This is
expensive, but a good way to catch bugs in protocols and
implementations.
So, add a `rejectUnexpectedSubgroup()` function to achieve this.
The documentation on the edwards25519->curve25519 conversion
function was also updated, in order to explain how to match
libsodium's behavior if necessary.
We use an addition chain to multiply the point by the order of
the prime group.
An alternative we may implement later is Pornin's point halving
technique: https://eprint.iacr.org/2022/1164.pdf
On Windows, the console mode flag `ENABLE_VIRTUAL_TERMINAL_PROCESSING` determines whether or not ANSI escape codes are parsed/acted on. On the newer Windows Terminal, this flag is set by default, but on the older Windows Console, it is not set by default, but *can* be enabled (since Windows 10 RS1 from June 2016).
The new `File.getOrEnableAnsiEscapeSupport` function will get the current status of ANSI escape code support, but will also attempt to enable `ENABLE_VIRTUAL_TERMINAL_PROCESSING` on Windows if necessary which will provide better/more consistent results for things like `std.Progress` and `std.io.tty`.
This type of change was not done previously due to a mistaken assumption (on my part) that the console mode would persist after the run of a program. However, it turns out that the console mode is always reset to the default for each program run in a console session.
* Newer versions of Windows added VT seq support not only in Windows Terminal, but also in the old-fashioned Windows Console (standalone conhost.exe), though not enabled by default.
* Try setting the newer console mode flags provides better experience for Windows Console users.
Co-authored-by: Kexy Biscuit <kexybiscuit@biscuitt.in>
Instead of introducing YES_COLOR, a completely new standard, into the mix
it might make more sense to instead tag along with the CLICOLOR_FORCE env var,
which dates back to at least 2000 with FreeBSD 4.1.1 and which is
supported by tools like CMake.
<https://bixense.com/clicolors/>
The \r\n is necessary to get the progress tree to work properly in the old console when ENABLE_VIRTUAL_TERMINAL_PROCESSING and DISABLE_NEWLINE_AUTO_RETURN are set.
The line_upper_bound_len fix addresses part of #20161
This changes the terminal display to keep the cursor at the top left of
the progress display, so that unlocked stderr writes, perhaps by child
processes, don't get eaten by the clear.
The docs for setting stdio to "inherit" say:
It also means that this step will obtain a global lock to prevent other
steps from running in the meantime.
The implementation of this lock was missing but is now provided by this
commit.
closes#20119
Reduce node_storage_buffer_len from 200 to 83. This makes messages over
the pipe fit in a single packet (4096 bytes). There is now a comptime
assert to ensure this. In practice this is plenty of storage because
typical terminal heights are significantly less than 83 rows.
Handling of split reads is fixed; instead of using a global
`remaining_read_trash_bytes`, the value is stored in the "saved
metadata" for the IPC node.
Saved metadata is split into two arrays so that the "find" operation can
quickly scan over fds for a match, looking at 332 bytes maximum, and
only reading the memory for the other data upon match. More typical
number of bytes read for this operation would be 0 (no child processes),
4 (1 child process), or 64 (16 child processes reporting progress).
Removed an align(4) that was leftover from an older design.
This also includes part of Jacob Young's not-yet-landed patch that
implements `writevNonblock`.
rather than ignoring specifically "zig-cache" and "zig-out". The latter
is not necessarily the install prefix and should not be special.
The former will be handled by renaming zig-cache to .zig-cache.
3a3d2187f9 unintentionally broke some of the Windows console API implementation.
- The 'marker' character was no longer being written at all
- The ANSI escape codes for syncing were being written unconditionally
Clarify the usage of .paths in build.zig.zon. Follow the recommendation
of the comments to explicitly list paths by explicitly listing the paths
in the init project.
* Merge a bunch of related state together into TerminalMode. Windows
sometimes follows the same path as posix via ansi_escape_codes,
sometimes not.
* Use a different thread entry point for Windows API but share the same
entry point on Windows when the terminal is in ansi_escape_codes mode.
* Only clear the terminal when the stderr lock is held.
* Don't try to clear the terminal when nothing has been written yet.
* Don't try to clear the terminal in IPC mode.
* Fix size detection logic bug under error conditions.
7281cc1d839da6e84bb76fadb2c1eafc22a82df7 did not solve the problem
because even when Node.index is none, it still counts as initializing
the global Progress object. Just use a normal zig optional, and all is
good.
The update thread was sometimes reading the special state and then
incorrectly getting 0 for the file descriptor, making it hang since it
tried to read from stdin.
Slightly slower refresh rate. It's still updating very quickly.
Lower the initial delay so that CLI applications feel more responsive.
Even though the application is doing work for the full 500ms until
something is displayed on the screen, it feels nicer to get the progress
earlier.
Introduces `disable_zig_progress` which prevents the build runner from
assigning the child process a progress node.
This is needed for the empty_env test which requires the environment to
be completely empty.
This makes it so that any other threads which are writing to stderr have
a chance to finish before the process terminates. It also clears the
terminal in case any progress has been written to stderr, while still
accomplishing the goal of not waiting until the update thread exits.
when the root progress node has a zero length name, the sub-tree is
flattened one layer, reducing visual noise, as well as bytes written to
the terminal.
Split newline_count into written_newline_count and
accumulated_newline_count. This handle the case when the tryLock() fails
to obtain the lock, because in such case there would not be any newlines
written to the terminal but the system would incorrectly think there
were. Now, written_newline_count is only adjusted when the write() call
succeeds.
Furthermore, write() call failure is handled by exiting the update
thread.
Don't truncate trailing newline. This better handles stray writes to
stderr that are not std.Progress-aware, such as from non-zig child
processes.
This commit also makes `Node.start` and `Node.end` bail out early with a
comptime branch when it is known the target will not be spawning an
update thread.
Switch Node.Parent, Node.Index, and Node.OptionalIndex to be backed by
u8 rather than u16. This works fine since we use 200 as the preallocated
node buffer. This has the nice property that scanning the entire parents
array for allocated nodes fits in 4 cache lines, even if we bumped the
200 up to 254 (leaving room for the two special states).
The thread that reads progress updates from the pipe now handles short
reads by ignoring messages that are sent in multiple reads.
When checking the terminal size, if there is a failure, fall back to a
conservative guess of 80x25 rather than panicking. A debug message is
also emitted which would be displayed only in a debug build.
This accomplishes 2 things simultaneously:
1. Don't trust child process data; if the data is outside the expected
range, ignore the data.
2. If there is too much data to fit in the preallocated buffers, drop
the data.
Instead of making static buffers configurable, let's pick strong
defaults and then use the update thread's stack memory to store the
preallocations. The thread uses a fairly shallow stack so this memory is
otherwise unused. This also makes the data section of the executable
smaller since it runtime allocates the memory when a `std.Progress`
instance is allocated, and in the case that the process is not connected
to a terminal, it never allocates the memory.
* correctly report time spent analyzing function bodies
* print fully qualified decl names
* also have a progress node for codegen
The downside of these changes is that it's a bit flickerey, but the
upside is that it's accurate; you can see what the compiler's doing!
This fix is already in master branch for stdin, stdout, and stderr; this
commit solves the same problem but for the progress pipe.
Both fixes were originally included in one commit on this branch,
however it was split it into two so that master branch could receive the
fix before the progress branch is merged.
It stored some metadata into the canonical node storage data but that is
a race condition because another thread recycles those nodes.
Also, keep the parent name for empty child root node names.
* bump default statically allocated resources
* debug help when multiple instances of std.Progress are initialized
* only handle sigwinch on supported operating systems
* handle when reading from the pipe returns 0 bytes
* avoid printing more lines than rows
This time, we preallocate a fixed set of nodes and have the user-visible
Node only be an index into them. This allows for lock-free management of
the node storage.
Only the parent indexes are stored, and the update thread makes a
serialized copy of the state before trying to compute children lists.
The update thread then walks the tree and outputs an entire tree of
progress rather than only one line.
There is a problem with clearing from the cursor to the end of the
screen when the cursor is at the bottom of the terminal.
New design ideas:
* One global instance, don't try to play nicely with other instances
except via IPC.
* One process owns the terminal and the other processes communicate via
IPC.
* Clear the whole terminal and use multiple lines.
What's implemented so far:
* Query the terminal for size.
* Register a SIGWINCH handler.
* Use a thread for redraws.
To be done:
* IPC
* Handling single threaded targets
* Porting to Windows
* More intelligent display of the progress tree rather than only using
one line.
You don't know if it's possible to run a binary until you try. The build
system already integrates with executors and has the
`skip_foreign_checks` for exactly this use case.
- Used `Self` instead of `*const Self` where appropriate (orignally proposed in #19770)
- Replaced `@intFromPtr` and `@ptrFromInt` with `@ptrCast`, `@alignCast`, and pointer arithmetic where appropriate
With this, the only remaining instance on pointer-int conversion in hash_map.zig is in `HashMapUnmanaged.removeByPtr`, which easily be able to be eliminated once pointer subtraction is supported.
The added comment explains the issue here relatively well. The new
progress API made this bug obvious because it became visibly clear that
certain Compile steps were seemingly "hanging" until other steps
completed. As it turned out, these child processes had raced to spawn,
and hence one had inherited the other's stdio pipes, meaning the `poll`
call in `std.Build.Step.evalZigProcess` was not identifying the child
stdout as closed until an unrelated process terminated.
This reverts commit a7de02e052.
This did not implement the accepted proposal, and I did not sign off
on the changes. I would like a chance to review this, please.
Fixes regression introduced by 5d5e89aa8d
Turns out since landing that PR we haven't run any tests requiring
symlinks or any Apple SDK on a macOS host. Not great.
ArrayList uses `items` slice to store len initialized items, while
PriorityQueue stores `capacity` potentially uninitialized items.
This is a surprising difference in the API that leads to bugs!
https://github.com/tigerbeetle/tigerbeetle/pull/1948
The wrong `size_class` was used when fetching stack traces from empty
buckets. The `size_class` would always be the maximum value after
exhausting the search of active buckets rather than the actual
`size_class` of the allocation.
Empty buckets have their `alloc_cursor` set to `slot_count` to allow the
size class to be calculated later. This happens deep within the free
function.
This adds a helper and a test to verify that the size class of empty
buckets is indeed recoverable.
Add module for mapping ASN1 types to Zig types. See
`asn1.Tag.fromZig` for the mapping. Add DER encoder and decoder.
See `asn1/test.zig` for example usage of every ASN1 type.
This implementation allows ASN1 tags to be overriden with `asn1_tag`
and `asn1_tags`:
```zig
const MyContainer = (enum | union | struct) {
field: u32,
pub const asn1_tag = asn1.Tag.init(...);
// This specifies a tag's class, and if explicit, additional encoding
// rules.
pub const asn1_tags = .{
.field = asn1.FieldTag.explicit(0, .context_specific),
};
};
```
Despite having an enum tag type, ASN1 frequently uses OIDs as enum
values. This is supported via an `pub const oids` field.
```zig
const MyEnum = enum {
a,
pub const oids = asn1.Oid.StaticMap(MyEnum).initComptime(.{
.a = "1.2.3.4",
});
};
```
Futhermore, a container may choose to implement encoding and decoding
however it deems fit. This allows for derived fields since Zig has a far
more powerful type system than ASN1.
```zig
// ASN1 has no standard way of tagging unions.
const MyContainer = union(enum) {
derived: PowerfulZigType,
const WeakAsn1Type = ...;
pub fn encodeDer(self: MyContainer, encoder: *der.Encoder) !void {
try encoder.any(WeakAsn1Type{...});
}
pub fn decodeDer(decoder: *der.Decoder) !MyContainer {
const weak_asn1_type = try decoder.any(WeakAsn1Type);
return .{ .derived = PowerfulZigType{...} };
}
};
```
An unfortunate side-effect is that decoding and encoding cannot have
complete complete error sets unless we limit what errors users may
return. Luckily, PKI ASN1 types are NOT recursive so the inferred
error set should be sufficient.
Finally, other encodings are possible, but this patch only implements
a buffered DER encoder and decoder.
In an effort to keep the changeset minimal this PR does not actually
use the DER parser for stdlib PKI, but a tested example of how it may
be used for Certificate is available
[here.](https://github.com/clickingbuttons/asn1/blob/69c5709d/src/Certificate.zig)
Closes#19775.
Not really useful after old C++ compiler removal, and
zig build has own cache system. If someone still wants it,
`CMAKE_C_COMPILER_LAUNCHER` and `CMAKE_CXX_COMPILER_LAUNCHER` exist.
From CMake docs:
> RULE_LAUNCH_COMPILE
> Note: This property is intended for internal use by ctest(1).
> Projects and developers should use the <LANG>_COMPILER_LAUNCHER
> target properties or the associated CMAKE_<LANG>_COMPILER_LAUNCHER
> variables instead.
Signed-off-by: Eric Joldasov <bratishkaerik@landless-city.net>
Set `ZIG_PIE` default to be same as `CMAKE_POSITION_INDEPENDENT_CODE`, and
add check for situation when `ZIG_PIE` is set to True but CMake does not
support compiling position independent code. CMake's support is needed
for "zigcpp" target.
Also remove temporary variables for constructing `ZIG_BUILD_ARGS`,
instead use `list(APPEND ...)` functions.
Also remove long unused `ZIG_NO_LANGREF` variable.
Signed-off-by: Eric Joldasov <bratishkaerik@landless-city.net>
Replace `CMAKE_SOURCE_DIR` and `CMAKE_BUILD_DIR` with different variables,
or in some cases just remove them.
For some function arguments, prepended `CMAKE_SOURCE_DIR` was removed without
replacement. This includes:
* Sources for `add_library` and `add_executable` (`ZIG_CPP_SOURCES` and `ZIG_WASM2C_SOURCES`)
* Inputs for `configure_file` and `target_include_directory`
* For arguments above, CMake already prepends
`CMAKE_CURRENT_SOURCE_DIR` to them by default, if they are relative paths.
Additionaly, it was removed from arguments of commands that have `WORKING_DIRECTORY` set to
`PROJECT_SOURCE_DIR`, they will be similarly converted by CMake for us.
Also:
* Move project declaration to the top so that these variables are
available earlier.
* Avoid calling "git" executable if ".git" directory does not exist.
* Swap "--prefix" and `ZIG_BUILD_ARGS` arguments in cmake/install.cmake
to match same "zig2 build" command in CMakeLists.txt and allow
overriding "--prefix" argument
Signed-off-by: Eric Joldasov <bratishkaerik@landless-city.net>
* Localize most of the global properties and functions, for some time
they are only needed for "zigcpp" static library (sometimes with PUBLIC
keyword, so that it will propagate to zig2): `CMAKE_*_OUTPUT_DIRECTORY`
and two calls to `include_directories`. This removes useless flags when
building other targets and cleans build log a bit.
* Remove `EXE_CXX_FLAGS` variable, instead use more appropriate specific
properties and functions for this target. This gives better errors if
compiler does not support some of them, and CMake also handles for us
duplicate flags. It's also easier to read side-by-side with same
flags from build.zig .
* Add some comments.
Signed-off-by: Eric Joldasov <bratishkaerik@landless-city.net>
Previously the error had a note suggesting to use `try`, `catch`, or
`if`, even for error sets where none of those work.
Instead, in case of an error set the way you can handle the error
depends very much on the specific case. For example you might be in a
`catch` where you are discarding or ignoring the error set capture
value, in which case one way to handle the error might be to `return`
the error.
So, in that case, we do not attach that error note.
Additionally, this makes the error tell you what kind of an error it is:
is it an error set or an error union? This distinction is very relevant
in how to handle the error.
Maybe I'm just being pedantic here (most likely) but I don't like how we're
just telling the user here how to "suppress this error" by "assigning the value to '_'".
I think it's better if we use the word "discard" here which I think is
the official terminology and also tells the user what it actually means
to "assign the value to '_'".
Also, using the value would also be a way to "suppress the error".
It is just one of the two options: discard or use.
* Revert "Revert "Merge pull request #19349 from nolanderc/save-commit""
This reverts commit 6ca4ed5948.
* update to new URI changes, rework `--save` type
* initialize `latest_commit` to null everywhere
this one is even harder to document then the last large overhaul.
TLDR;
- split apart Emit.zig into an Emit.zig and a Lower.zig
- created seperate files for the encoding, and now adding a new instruction
is as simple as just adding it to a couple of switch statements and providing the encoding.
- relocs are handled in a more sane maner, and we have a clear defining boundary between
lea_symbol and load_symbol now.
- a lot of different abstractions for things like the stack, memory, registers, and others.
- we're using x86_64's FrameIndex now, which simplifies a lot of the tougher design process.
- a lot more that I don't have the energy to document. at this point, just read the commit itself :p
this commit is a little too large to document fully, however the main gist of it this
- finish the `genInlineMemcpy` implement
- rename `setValue` to `genCopy` as I agree with jacob that it's a better name
- add in `genVarDbgInfo` for a better gdb experience
- follow the x86_64's method for genCall, as the procedure is very similar for us
- add `airSliceLen` as it's trivial
- change up the `airAddWithOverflow implementation a bit
- make sure to not spill of the elem_ty is 0 size
- correctly follow the RISC-V calling convention and spill the used calle saved registers in the prologue
and restore them in the epilogue
- add `address`, `deref`, and `offset` helper functions for MCValue. I must say I love these,
they make the code very readable and super verbose :)
- fix a `register_manager.zig` issue where when using the last register in the set, the value would overflow at comptime.
was happening because we were adding to `max_id` before subtracting from it.
the truncation panic logic is generated in Sema, so I don't need to roll anything
of my own. I add all of the boilerplate for that detecting the truncation and it works
in basic test cases!
this provides a much better indication of when we are having a controlled panic with an error message
or when we are actually segfaulting, as before the `trap` as causing a segfault.
when the struct is in stack memory, we access it using a byte-offset,
because that's how the stack works. on the other hand when the struct
is in a register, we are working with bits and the field offset should
be a bit offset.
- Added the basic framework for panicing with an overflow in `airAddWithOverflow`, but there is no check done yet.
- added the `cmp_lt`, `cmp_gte`, and `cmp_imm_eq` MIR instructions, and their respective functionality.
lots of thinking later, ive begun to grasp my head around how the pointers should work. this commit allows basic pointer loading and storing to happen.
- before we were storing each arg in it's own function arg register. with this commit now we store the args in the fa register before calling as per the RISC-V calling convention, however as soon as we enter the callee, aka in airArg, we spill the argument to the stack. this allows us to spend less effort worrying about whether we're going to clobber the function arguments when another function is called inside of the callee.
- we were actually clobbering the fa regs inside of resolveCallingConvetion, because of the null argument to allocReg. now each lock is stored in an array which is then iterated over and unlocked, which actually aids in the first point of this commit.
this was an annoying one to do, as there is no (to my knowledge) myriad sequence
that will allow us to do `gte` compares with an immediate without allocating a register.
RISC-V provides a single instruction to do compares, that being `lt`, and so you need to
use more than one for other variants, but in this case, i believe you need to allocate a register.
- implement `airArrayElemVal` for arrays on the stack. This is really easy
as we can just move the offset by the bytes into the array. This only works
when the index access is comptime-known though, this won't work for runtime access.
the current implementation only works when the struct is in a register. we use some shifting magic
to get the field into the LSB, and from there, given the type provenance, the generated code should
never reach into the bits beyond the bit size of the type and interact with the rest of the struct.
we use a code offset map in Emit.zig to pre-compute what byte offset each MIR instruction is at. this is important because they can be
of different size
- rename setRegOrMem -> setValue
- a naive method of passing arguments by register
- gather the prologue and epilogue and generate them in Emit.zig. this is cleaner because we have the final stack size in the emit step.
- define the "fa" register set, which contains the RISC-V calling convention defined function argument registers
Information about installed MSVC instances are stored in `state.json` files within a `Packages/_Instances` directory. The default location for this is `%PROGRAMDATA%\Microsoft\VisualStudio\Packages\_Instances`. However, it is possible for the Packages directory to be put somewhere else. In that case, the registry value `HKLM\SOFTWARE\Microsoft\VisualStudio\Setup\CachePath` is set and contains the path to the Packages directory.
Previously, WindowsSdk did not check that registry value. After this commit, the registry value `HKLM\SOFTWARE\Microsoft\VisualStudio\Setup\CachePath` is checked first, which matches what ISetupEnumInstances does (according to a Procmon log).
Clang 17 passed struct{f128} parameters using rdi and rax, while Clang
18 matches GCC 13.2 behavior, passing them using xmm0.
This commit makes Zig's LLVM backend match Clang 18 and GCC 13.2. The
commit deletes a hack in x86_64/abi.zig which miscategorized f128 as
"memory" which obviously disagreed with the spec.
LLVMABIAlignmentOfType(i128) reports 16 on this target, however the C
ABI uses align(4).
Clang in LLVM 17 does this:
%struct.foo = type { i32, i128 }
Clang in LLVM 18 does this:
%struct.foo = type <{ i32, i128 }>
Clang is working around the 16-byte alignment to use align(4) for the C
ABI by making the LLVM struct packed.
release/18.x branch, commit 78b99c73ee4b96fe9ce0e294d4632326afb2db42
This adds the flag `-D_LIBCPP_HARDENING_MODE` which is determined based
on the Zig optimization mode.
This commit also fixes libunwind, libcxx, and libcxxabi to properly
report sub compilation errors.
LLVM now refuses to lower arguments and return values on x86 targets
when the total vector bit size is >= 512.
This code detects such a situation and uses byref instead of byval.
* some manual fixes to generated CPU features code. In the future it
would be nice to make the script do those automatically.
* add to various target OS switches. Some of the values I was unsure of
and added TODO panics, for example in the case of spirv CPU arch.
New OSs:
* XROS
* Serenity
* Vulkan
Removed OSs:
* Ananas
* CloudABI
* Minix
* Contiki
New CPUs:
* spirv
The removed stuff is removed from LLVM but not Zig.
This allows running commands that take an output directory argument. The
main thing that was needed for this feature was generated file subpaths,
to allow access to the files in a generated directory. Additionally, a
minor change was required to so that the correct directory is created
for output directory args.
* `doc/langref` formatting
* upgrade `.{ .path = "..." }` to `b.path("...")`
* avoid using arguments named `self`
* make `Build.Step.Id` usage more consistent
* add `Build.pathResolve`
* use `pathJoin` and `pathResolve` everywhere
* make sure `Build.LazyPath.getPath2` returns an absolute path
This replaces `extra_file_dependencies` with support for lazy paths.
Also assert output file basenames are not empty, avoid improper use of
field default values, ensure stored strings are duplicated, and
prefer `ArrayListUnmanaged` to discourage misuse of direct field access
which wouldn't add step dependencies.
This was a "fake" type used to handle C varargs parameters, much like
generic poison. In fact, it is treated identically to generic poison in
all cases other than one (the final coercion of a call argument), which
is trivially special-cased. Thus, it makes sense to remove this special
tag and instead use `generic_poison_type` in its place. This fixes
several bugs in Sema related to missing handling of this tag.
Resolves: #19781
This function accepts a WaitGroup parameter and manages the reference
counting therein. It also is infallible.
The existing `spawn` function is still handy when the job wants to
further schedule more tasks.
closes#19803 by changing quota from (30 * N) to (10 * N * log2(N)) where
N = kvs_list.len
* adds reported adversarial test case
* update doc comment of getLongestPrefix()
Adds an `include_paths` field to RcSourceFile that takes a slice of LazyPaths. The paths are resolved and subsequently appended to the -rcflags as `/I <resolved path>`.
This fixes an accidental regression from https://github.com/ziglang/zig/pull/19174. Before that PR, all Win32 resource compilation would inherit the CC flags (via `addCCArgs`), which included things like include directories. After that PR, though, that is no longer the case.
However, this commit intentionally does not restore the previous behavior (inheriting the C include paths). Instead, each .rc file will need to have its include paths specified directly and the include paths only apply to one particular resource script. This allows more fine-grained control and has less potentially surprising behavior (at the cost of some convenience).
Closes#19605
This function incorrectly assumed that module name subsections, function
name subsections, and local name subsections are encoded the same,
however according to
[the specification](https://webassembly.github.io/spec/core/appendix/custom.html)
they are encoded differently.
This commit adds support for parsing module name subsections correctly,
which started appearing after upgrading to LLVM 18.
As of Clang 18, calling memcpy() with a misaligned pointer trips UBSAN,
even if the length is zero. This unfortunately includes any call to
`@memcpy` when source or destination are undefined and the length is
zero.
This patch makes the C backend avoid calling memcpy when the length is
zero, thereby avoiding undefined behavior.
A zig1.wasm update will be needed in the llvm18 branch to activate this
code.
wheras on NetBSD, only 2 PT_LOAD are usually produced by other compilers
(tested with host gcc and clang).
$ ldd -v main_4segs
.../main_4segs: wrong number of segments (4 != 2)
.../main_4segs: invalid ELF class 2; expected 1
* std.crypto.hash.sha2: generalize sha512 truncation
Replace `Sha512224`, `Sha512256`, and `Sha512T224` with
`fn Sha512Truncated(digest_bits: comptime_int)`.
This required refactoring `Sha2x64(comptime params)` to
`Sha2x64(comptime iv: [8]u64, digest_bits: comptime_int)`
for user-specified `digest_bits`.
I left #19697 alone but added a compile-time check that digest_bits is
divisible by 8.
Remove docs which restate type name. Add module docs and reference where
IVs come from.
* std.crypto.sha2: make Sha512_224 and Sha512_256 pub
* make generic type implementation detail, add comments
* fix iv
* address @jedisct1 feedback
* fix typo
* renaming
* add truncation clarifying comment and Sha259T192 tests
* Fix the ELF binaries for freestanding target created with the self-hosted linker.
The ELF specification (generic ABI) states that ``loadable process segments must have congruent
values for p_vaddr and p_offset, modulo the page size''. Linux refuses to load binaries that
don't meet this requirement (execve() fails with EINVAL).
This removes the two original implementations in favour of the single
generic one based on the Algorithm type. Previously we had three, very
similar implementations which was somewhat confusing when knowing what
one should actually be used.
The previous polynomials all have equivalent variants available when
using the Algorithm type.
A volume can be mounted as a NTFS path, e.g. as C:\Mnt\Foo. In that case, IOCTL_MOUNTMGR_QUERY_POINTS gives us a mount point with a symlink value something like `\??\Volume{383da0b0-717f-41b6-8c36-00500992b58d}`. In order to get the `C:\Mnt\Foo` path, we can query the mountmgr again using IOCTL_MOUNTMGR_QUERY_DOS_VOLUME_PATH.
Fixes#19731
* Adjust buffer length a bit.
* Fix detecting if file is a script. Logic below was unreachable,
because 99% of scripts failed "At least 255 bytes long" check and were detected as ELF files.
It should be "At least 4" instead (minimum value of "ELF magic length" and "smallest possible interpreter path length").
* Fix parsing interpreter path, when text after shebang:
1. does not have newline,
2. has leading spaces and tabs,
3. separates interpreter and arguments by tab or NUL.
* Remove empty error set from `defaultAbiAndDynamicLinker`.
Signed-off-by: Eric Joldasov <bratishkaerik@landless-city.net>
* std.crypto: make ff.ct_unprotected.limbsCmpLt compile
* std.crypto: add ff.ct test
* fix testCt to work on x86
* disable test on stage2-c
---------
Co-authored-by: Frank Denis <124872+jedisct1@users.noreply.github.com>
> Note: This first part is mostly a rephrasing of https://flatt.tech/research/posts/batbadbut-you-cant-securely-execute-commands-on-windows/
> See that article for more details
On Windows, it is possible to execute `.bat`/`.cmd` scripts via CreateProcessW. When this happens, `CreateProcessW` will (under-the-hood) spawn a `cmd.exe` process with the path to the script and the args like so:
cmd.exe /c script.bat arg1 arg2
This is a problem because:
- `cmd.exe` has its own, separate, parsing/escaping rules for arguments
- Environment variables in arguments will be expanded before the `cmd.exe` parsing rules are applied
Together, this means that (1) maliciously constructed arguments can lead to arbitrary command execution via the APIs in `std.process.Child` and (2) escaping according to the rules of `cmd.exe` is not enough on its own.
A basic example argv field that reproduces the vulnerability (this will erroneously spawn `calc.exe`):
.argv = &.{ "test.bat", "\"&calc.exe" },
And one that takes advantage of environment variable expansion to still spawn calc.exe even if the args are properly escaped for `cmd.exe`:
.argv = &.{ "test.bat", "%CMDCMDLINE:~-1%&calc.exe" },
(note: if these spawned e.g. `test.exe` instead of `test.bat`, they wouldn't be vulnerable; it's only `.bat`/`.cmd` scripts that are vulnerable since they go through `cmd.exe`)
Zig allows passing `.bat`/`.cmd` scripts as `argv[0]` via `std.process.Child`, so the Zig API is affected by this vulnerability. Note also that Zig will search `PATH` for `.bat`/`.cmd` scripts, so spawning something like `foo` may end up executing `foo.bat` somewhere in the PATH (the PATH searching of Zig matches the behavior of cmd.exe).
> Side note to keep in mind: On Windows, the extension is significant in terms of how Windows will try to execute the command. If the extension is not `.bat`/`.cmd`, we know that it will not attempt to be executed as a `.bat`/`.cmd` script (and vice versa). This means that we can just look at the extension to know if we are trying to execute a `.bat`/`.cmd` script.
---
This general class of problem has been documented before in 2011 here:
https://learn.microsoft.com/en-us/archive/blogs/twistylittlepassagesallalike/everyone-quotes-command-line-arguments-the-wrong-way
and the course of action it suggests for escaping when executing .bat/.cmd files is:
- Escape first using the non-cmd.exe rules
- Then escape all cmd.exe 'metacharacters' (`(`, `)`, `%`, `!`, `^`, `"`, `<`, `>`, `&`, and `|`) with `^`
However, escaping with ^ on its own is insufficient because it does not stop cmd.exe from expanding environment variables. For example:
```
args.bat %PATH%
```
escaped with ^ (and wrapped in quotes that are also escaped), it *will* stop cmd.exe from expanding `%PATH%`:
```
> args.bat ^"^%PATH^%^"
"%PATH%"
```
but it will still try to expand `%PATH^%`:
```
set PATH^^=123
> args.bat ^"^%PATH^%^"
"123"
```
The goal is to stop *all* environment variable expansion, so this won't work.
Another problem with the ^ approach is that it does not seem to allow all possible command lines to round trip through cmd.exe (as far as I can tell at least).
One known example:
```
args.bat ^"\^"key^=value\^"^"
```
where args.bat is:
```
@echo %1 %2 %3 %4 %5 %6 %7 %8 %9
```
will print
```
"\"key value\""
```
(it will turn the `=` into a space for an unknown reason; other minor variations do roundtrip, e.g. `\^"key^=value\^"`, `^"key^=value^"`, so it's unclear what's going on)
It may actually be possible to escape with ^ such that every possible command line round trips correctly, but it's probably not worth the effort to figure it out, since the suggested mitigation for BatBadBut has better roundtripping and leads to less garbled command lines overall.
---
Ultimately, the mitigation used here is the same as the one suggested in:
https://flatt.tech/research/posts/batbadbut-you-cant-securely-execute-commands-on-windows/
The mitigation steps are reproduced here, noted with one deviation that Zig makes (following Rust's lead):
1. Replace percent sign (%) with %%cd:~,%.
2. Replace the backslash (\) in front of the double quote (") with two backslashes (\\).
3. Replace the double quote (") with two double quotes ("").
4. ~~Remove newline characters (\n).~~
- Instead, `\n`, `\r`, and NUL are disallowed and will trigger `error.InvalidBatchScriptArg` if they are found in `argv`. These three characters do not roundtrip through a `.bat` file and therefore are of dubious/no use. It's unclear to me if `\n` in particular is relevant to the BatBadBut vulnerability (I wasn't able to find a reproduction with \n and the post doesn't mention anything about it except in the suggested mitigation steps); it just seems to act as a 'end of arguments' marker and therefore anything after the `\n` is lost (and same with NUL). `\r` seems to be stripped from the command line arguments when passed through a `.bat`/`.cmd`, so that is also disallowed to ensure that `argv` can always fully roundtrip through `.bat`/`.cmd`.
5. Enclose the argument with double quotes (").
The escaped command line is then run as something like:
cmd.exe /d /e:ON /v:OFF /c "foo.bat arg1 arg2"
Note: Previously, we would pass `foo.bat arg1 arg2` as the command line and the path to `foo.bat` as the app name and let CreateProcessW handle the `cmd.exe` spawning for us, but because we need to pass `/e:ON` and `/v:OFF` to cmd.exe to ensure the mitigation is effective, that is no longer tenable. Instead, we now get the full path to `cmd.exe` and use that as the app name when executing `.bat`/`.cmd` files.
---
A standalone test has also been added that tests two things:
1. Known reproductions of the vulnerability are tested to ensure that they do not reproduce the vulnerability
2. Randomly generated command line arguments roundtrip when passed to a `.bat` file and then are passed from the `.bat` file to a `.exe`. This fuzz test is as thorough as possible--it tests that things like arbitrary Unicode codepoints and unpaired surrogates roundtrip successfully.
Note: In order for the `CreateProcessW` -> `.bat` -> `.exe` roundtripping to succeed, the .exe must split the arguments using the post-2008 C runtime argv splitting implementation, see https://github.com/ziglang/zig/pull/19655 for details on when that change was made in Zig.
ensureTotalCapacityPrecise only satisfies the assumptions made in the ArrayListImpl functions (that there's already enough capacity for the entire converted string if it's all ASCII) when the ArrayList has no items, otherwise it would hit illegal behavior.
Windows does not support RPATH and only searches for DLLs in a small
number of predetermined paths by default, with one of them being the
directory from which the application loaded.
Installing both executables and DLLs to `bin/` by default helps ensure
that the executable can find any DLL artifacts it has linked to.
DLL import libraries are still installed to `lib/`.
These defaults match CMake's behavior.
this patch renames ComptimeStringMap to StaticStringMap, makes it
accept only a single type parameter, and return a known struct type
instead of an anonymous struct. initial motivation for these changes
was to reduce the 'very long type names' issue described here
https://github.com/ziglang/zig/pull/19682.
this breaks the previous API. users will now need to write:
`const map = std.StaticStringMap(T).initComptime(kvs_list);`
* move `kvs_list` param from type param to an `initComptime()` param
* new public methods
* `keys()`, `values()` helpers
* `init(allocator)`, `deinit(allocator)` for runtime data
* `getLongestPrefix(str)`, `getLongestPrefixIndex(str)` - i'm not sure
these belong but have left in for now incase they are deemed useful
* performance notes:
* i posted some benchmarking results here:
https://github.com/travisstaloch/comptime-string-map-revised/issues/1
* i noticed a speedup reducing the size of the struct from 48 to 32
bytes and thus use u32s instead of usize for all length fields
* i noticed speedup storing KVs as a struct of arrays
* latest benchmark shows these wall_time improvements for
debug/safe/small/fast builds: -6.6% / -10.2% / -19.1% / -8.9%. full
output in link above.
Stores the original ref as a query parameter in the URL so that it is
possible to automatically check the upstream if there are any newer
commits.
Also adds a flag which opts-out of the new behaivour, restoring the old.
constflat=b.option(bool,"flat","Put files into the installation prefix in a manner suited for upstream distribution rather than a posix file system hierarchy standard")orelsefalse;
consttracy=b.option([]constu8,"tracy","Enable Tracy integration. Supply path to Tracy source");
consttracy_callstack=b.option(bool,"tracy-callstack","Include callstack information with Tracy data. Does nothing if -Dtracy is not provided")orelse(tracy!=null);
consttracy_allocation=b.option(bool,"tracy-allocation","Include allocation information with Tracy data. Does nothing if -Dtracy is not provided")orelse(tracy!=null);
constforce_gpa=b.option(bool,"force-gpa","Force the compiler to use GeneralPurposeAllocator")orelsefalse;
consttracy_callstack_depth:u32=b.option(u32,"tracy-callstack-depth","Declare callstack depth for Tracy data. Does nothing if -Dtracy_callstack is not provided")orelse10;
constdebug_gpa=b.option(bool,"debug-allocator","Force the compiler to use DebugAllocator")orelsefalse;
constlink_libc=b.option(bool,"force-link-libc","Force self-hosted compiler to link libc")orelse(enable_llvmoronly_c);
constpie=b.option(bool,"pie","Produce a Position Independent Executable");
constvalue_interpret_mode=b.option(ValueInterpretMode,"value-interpret-mode","How the compiler translates between 'std.builtin' types and its internal datastructures")orelse.direct;
constvalue_tracing=b.option(bool,"value-tracing","Enable extra state tracking to help troubleshoot bugs in the compiler (using the std.debug.Trace API)")orelsefalse;
constmem_leak_frames:u32=b.option(u32,"mem-leak-frames","How many stack frames to print when a memory leak occurs. Tests get 2x this amount.")orelseblk:{
exe_options.addOption(DevEnv,"dev",b.option(DevEnv,"dev","Build a compiler with a reduced feature set for development of specific features")orelseif(only_c).bootstrapelse.full);
consttest_target_filters=b.option([]const[]constu8,"test-target-filter","Skip tests whose target triple do not match any filter")orelse&[0][]constu8{};
consttest_slow_targets=b.option(bool,"test-slow-targets","Enable running module tests for targets that have a slow compiler backend")orelsefalse;
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
