* don't assert that the child process doesn't crash
* don't give a false negative on warnings printed to stderr
Also fix getSdk from the same file in the same way
Regressed in 2006add849.
References to native_darwin_sdk are no longer kept in the frontend.
Instead the darwin SDK is detected as part of NativePaths and as part of
LibCInstallation.
The `null` value here was missed in
0a4d4eb252a73555a568a532902951a13284d2ef. I hope it is the cause of the
CI failures on Windows.
The fact that libc++ depends on libc is not important for the CLI and
Compilation.create already handles that logic.
This can be null in two cases right now:
1. Windows DLLs that zig ships such as advapi32.
2. extern "foo" fn declarations where we find out about libraries too late
TODO: make this non-optional and resolve those two cases somehow.
chicken out and make -l match the status quo behavior, where it looks
for dynamic libraries and then falls back to static libraries.
library resolution is still done in the CLI now though, and these
options are added:
-search_static_first Search for static libs in all library search
paths, then dynamic libs.
-search_dylibs_only Only search for dynamic libs.
-search_static_only Only search for static libs.
this matches the already existing options below:
-search_paths_first For each library search path, check for dynamic
lib then static lib before proceeding to next path.
-search_dylibs_first Search for dynamic libs in all library search
So, it is still possible to get the strict behavior by passing
`-search_dylibs_only` or `-search_static_only`.
This commit also makes -dynamic and -static affect the preferred link
mode and search strategy.
Putting them in both causes collisions because the same lib ends up
being linked in twice.
Putting them in positionals instead of libs makes their properties about
needed and weak being ignored.
First, system_libs are collected into a list. This is the same as
before.
Next, system_libs are filtered into external_system_libs, which is the
same list but without any libc, compiler_rt, etc.
At this point, if there are any external system libs, native library
directory paths are detected and added to lib_dirs.
Finally, extern_system_libs are filtered into resolved_system_libs,
which has full paths to all of the libraries. This is the list passed
into Compilation.
This makes the required changes noted by @ifreund in the code review for
this branch.
Before, we would end up with the LLVM_LIBRARIES variable containing,
for example, "-lLLVMAnalysis;-lz" and the Zig compiler CLI would be
stuck with ambiguous library names. In this branch, that ambiguity is
not allowed; it will look for a dynamically linked LLVMAnalysis and fail
to find it in the case that the build of LLVM is static.
With this change, LLVM_LIBRARIES will contain, for example,
"/full/path/to/libLLVMAnalysis.a;-lz" and it is unambiguous that the
first library should be linked statically and the second dynamically.
In the case of a dynamically linked system LLVM, it will look like
"/full/path/to/libLLVM.so;-lz" which again is fully unambiguous.
This fixes building Zig from source.
search_strategy is no longer passed to Compilation at all; instead it is
used in the CLI code only.
When using Zig CLI mode, `-l` no longer has the ability to link
statically; use positional arguments for this.
The CLI has a small abstraction around library resolution handling which
is used to remove some code duplication regarding static libraries, as
well as handle the difference between zig cc CLI mode and zig CLI mode.
Thanks to this, system libraries are now included in the cache hash, and
thus changes to them will correctly cause cache misses.
In the future, lib_dirs should no longer be passed to Compilation at
all, because it is a frontend-only concept.
Previously, -search_paths_first and -search_dylibs_first were
Darwin-only arguments; they now work the same for all targets. Same
thing with --sysroot.
Improved the error reporting for failure to find a system library. An
example error now looks like this:
```
$ zig build-exe test.zig -lfoo -L. -L/a -target x86_64-macos --sysroot /home/andy/local
error: unable to find Dynamic system library 'foo' using strategy 'no_fallback'. search paths:
./libfoo.tbd
./libfoo.dylib
./libfoo.so
/home/andy/local/a/libfoo.tbd
/home/andy/local/a/libfoo.dylib
/home/andy/local/a/libfoo.so
/a/libfoo.tbd
/a/libfoo.dylib
/a/libfoo.so
```
closes#14963
This was discussed in #16597. It makes sense for most of the functions
in this file to be marked inline: many are simple helper functions so
inlining is likely a strict win, and having the return values be
comptime-known may improve userspace code in some cases by preventing it
from unintentionally checking properties of the target at runtime.
This changeset is somewhat conservative: the functions marked inline are
generally returning booleans or simple integers, and many are simple
one-line checks.
Without duping, users could get some unexpected behavior if they used a
string with a lifetime that didn't persist throughout the full build,
i.e. if it wasn't heap allocated, or if it was explicitly freed.
* Replaces the exit assembly with the function from std.
* Reads the top-of-stack struct at program startup that can get information
like the pid.
* Changes the read and write functions to use the Pread and Pwrite syscalls
instead of the depreciated _READ and _WRITE
* Changes the openat function to use flags instead of perms.
Plan9 does not support perms when opening, just when creating.
* Adds an errstr function to read the errstr buf created by the kernel
Implements issue #6451.
This was needed to support allocation on Plan 9 and now other operating
systems like DOS can also use it.
It is a modified version of the WasmAllocator since wasm also uses a
sbrk-esque allocation system.
This commit also adds the necessary system bits for sbrk to work on plan 9.
This solves the nuance case of compiling hand-crafted assembly files
which do not feature `MH_SUBSECTIONS_VIA_SYMBOLS` flag resulting in
input `Atom`s encompassing multiple symbols each with unique unwind
information.
The old code was iterating and generating symbols
for fields in their declared order instead of the
memory optimized order while getting offsets in
the memory optimized order.
* Disable runtime calls, since it is not possible to know the proper
stack adjustment to follow the callee abi.
* Disable runtime returns, since it is not possible to know where the
return address is stored in general.
* Allow implicit returns regardless of the return type, which allows
naked functions with a non-void return type to be written.
After fixing some issues with inline assembly in the C backend, the std
cleanups have the side effect of making these functions compatible with
the backend, allowing it to be used on linux without linking libc.
* introduce LazyPath.cwd_relative variant and use it for --zig-lib-dir. closes#12685
* move overrideZigLibDir and setMainPkgPath to options fields set once
and then never mutated.
* avoid introducing Build/util.zig
* use doc comments for deprecation notices so that they show up in
generated documentation.
* introduce InstallArtifact.Options, accept it as a parameter to
addInstallArtifact, and move override_dest_dir into it. Instead of
configuring the installation via Compile step, configure the
installation via the InstallArtifact step. In retrospect this is
obvious.
* remove calls to pushInstalledFile in InstallArtifact. See #14943
* rewrite InstallArtifact to not incorrectly observe whether a Compile
step has any generated outputs. InstallArtifact is meant to trigger
output generation.
* fix child process evaluation code handling of `-fno-emit-bin`.
* don't store out_h_filename, out_ll_filename, etc., pointlessly. these
are all just simple extensions appended to the root name.
* make emit_directory optional. It's possible to have nothing outputted,
for example, if you're just type-checking.
* avoid passing -femit-foo/-fno-emit-foo when it is the default
* rename ConfigHeader.getTemplate to getOutput
* deprecate addOptionArtifact
* update the random number seed of Options step caching.
* avoid using `inline for` pointlessly
* avoid using `override_Dest_dir` pointlessly
* avoid emitting an executable pointlessly in test cases
Removes forceBuild and forceEmit. Let's consider these additions separately.
Nearly all of the usage sites were suspicious.
This change allows the following types to appear in extern structs:
* Zero-bit integers
* void
* zero-sized structs and packed structs
* enums with zero-bit backing integers
* arrays of any length with zero-size elements
When acessing a packed struct member via a byte aligned ptr (from the optimisation in Sema.structFieldPtrByIndex())
the codegen must apply the parent ptr packed_offset in addition to the field offset itself.
resolves https://github.com/ziglang/zig/issues/16609
Before, iteration would stop whenever an installation with vcruntime.lib was found, but that may not be the most recent installed version. Instead, we now iterate all installed instances and choose the one with the newest version.
This mirrors the behavior of the progress indicator for the actual
compilation. Not doing this was causing sporadic CI failures due to the
(non-existent) fetches taking long enough to appear in stderr.
The C++ version of this code used this logic, and it turns out it is able to find some setups that the current registry/Vs7 methods cannot.
For example, if only the "Build Tools for Visual Studio" are installed but not Visual Studio itself, then only the ISetupEnumInstances method seems to find it.
Follow up to #15657, fixes a regression caused by moving from the C++ version to the Zig version
Before this commit, there were three issues with the makePath implementation:
1. The component iteration did not 'collapse' consecutive path separators; instead, it would treat `a/b//c` as `a/b//c`, `a/b/`, `a/b`, and `a`.
2. Trailing path separators led to redundant `makeDir` calls, e.g. with the path `a/b/` (if `a` doesn't exist), it would try to create `a/b/`, then try `a/b`, then try `a`, then try `a/b`, and finally try `a/b/` again.
3. The iteration did not treat the root of a path specially, so on Windows it could attempt to make a directory with a path like `X:` for something like `X:\a\b\c` if the `X:\` drive doesn't exist. This didn't lead to any problems that I could find, but there's no reason to try to make a drive letter as a directory (or any other root path).
This commit fixes all three issues by introducing a ComponentIterator that is root-aware and handles both sequential path separators and trailing path separators and uses it in `Dir.makePath`. This reduces the number of `makeDir` calls for paths where (1) the root of the path doesn't exist, (2) there are consecutive path separators, or (3) there are trailing path separators
As an example, here are the makeDir calls that would have been made before this commit when calling `makePath` for a relative path like `a/b//c//` (where the full path needs to be created):
a/b//c//
a/b//c/
a/b//c
a/b/
a/b
a
a/b
a/b/
a/b//c
a/b//c/
a/b//c//
And after this commit:
a/b//c
a/b
a
a/b
a/b//c
At main startup, if the ELF auxiliary vector contains a stacksize value,
use it as a hint for the minimum stacksize required by the executable.
1. Never lower the hard-limit. Once a hard-limit is lowered, then it can
never be increased (including child processes).
2. If hint exceeds hard-limit then clamp hint to hard-limit.
3. If soft-limit exceeds hint then do nothing.
This commit does two things which seem unrelated at first, but,
together, solve a miscompilation, and potentially slightly speed up
compiler perf, at the expense of making #2765 trickier to implement in
the future.
Sema: avoid returning a false positive for whether an inferred error set
is comptime-known to be empty.
AstGen: mark function calls as not being interested in a result
location. This prevents the test case "ret_ptr doesn't cause own
inferred error set to be resolved" from being regressed. If we want to
accept and implement #2765 in the future, it will require solving this
problem a different way, but the principle of YAGNI tells us to go ahead
with this change.
Old ZIR looks like this:
%97 = ret_ptr()
%101 = store_node(%97, %100)
%102 = load(%97)
%103 = ret_is_non_err(%102)
New ZIR looks like this:
%97 = ret_type()
%101 = as_node(%97, %100)
%102 = ret_is_non_err(%101)
closes#15669
When calling NtCreateFile with a UNC path, if either `\\server` or `\\server\share` are not found, then the statuses `BAD_NETWORK_PATH` or `BAD_NETWORK_NAME` are returned (respectively).
These statuses are not translated into `error.FileNotFound` because they convey more information than the typical FileNotFound error. For example, if you were trying to call `Dir.makePath` with an absolute UNC path like `\\MyServer\MyShare\a\b\c\d`, then knowing that `\\MyServer\MyShare` was not found allows for returning after trying to create the first directory instead of then trying to create `a\b\c`, `a\b`, etc. when it's already known that they will all fail in the same way.
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
Compatibility with CMake < 3.5 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
The special case to take advantage of llvm's intrinsic
was generating invalid llvm ir:
```
Invalid bitcast
%60 = bitcast <2 x i2> %59 to i8, !dbg !3122
thread 145453 panic: LLVM module verification failed
```
fixes#10731
Thanks @nektro for previous work in #14878
This change creates a small breaking change:
It removes the `is_pub` field of a decl in `@typeInfo`
This is another minor change but still makes a visual difference and will reduce the amount you have to scroll in your terminal by a little bit.
Reasoning:
1. The `for (0..src.data.reference_trace_len)` loop will run at least once due to the `src.data.reference_trace_len > 0` check above.
2. In all 3 branches of the `if` in that `for` it will print something.
3. The 3 strings of all of those prints already end in `\n`.
Therefore, the extra `try stderr.writeByte('\n');` is unnecessary.
67d5bfef removed std.ChildProcess tests, suggesting to make them
standalone instead. This commit does exactly that after the
bug creating SIGPIPE in ReleaseFast is no more with LLVM 15.0.5.
Thanks to @x1ddos for the idea with the compile artifacts and PR
improvements.
Improvements for xxHash performance, both on small keys as well as large slices.
* std.hash: make benchmarks consistent for xxhash
There is some odd behaviour in the timings for the XXHash benchmarks
introduced in 879f0b9, specifically the changes to the allocation in
benchmarkHash. The problem is somewhere in the difference between
9628243 and 9362d61 (these are commit that were force-pushed over but
can be found in PR #15917).
* std.hash: correctly calculate throughput in benchmark
* std.hash: add hashes per sec to small key output
* std.hash: add exact and small xxhash routines
* std.hash: add --small-only flag to benchmark
* std.hash.xxhash: extract stateless Accumulator type
* std.hash.xxhash: cleanup hash() and improve small key perf
* std.hash.xxhash: port xxhash64 changes to xxhash32
* std.hash: change benchmark --small-only flag to --include-array
Specifically this is to make sure llvm data layout generation doesn't
regress. The no emit bin is to allow testing targets that can't
currently be linked. The commented out targets are ones that fail in
the linker anyway when no emit bin is passed.
This fixes a regression caused by https://github.com/ziglang/zig/pull/13993
As an optimization, the first call to `NtQueryDirectoryFile` would only ask for a single result and assume that if the result returned did not match the app_name exactly, then the unappended app_name did not exist. However, this relied on the assumption that the unappended app_name would always be returned first, but that only seems to be the case on NTFS. On FAT filesystems, the order of returned files can be different, which meant that it could assume the unappended file doesn't exist when it actually does.
This commit fixes that by fully iterating the wildcard matches via `NtQueryDirectoryFile` and taking note of any unappended/PATHEXT-appended filenames it finds. In practice, this strategy does not introduce a speed regression compared to the previous (buggy) implementation.
Benchmark 1 (10 runs): winpathbench-master.exe
measurement mean ± σ min … max outliers delta
wall_time 508ms ± 4.08ms 502ms … 517ms 1 (10%) 0%
peak_rss 3.62MB ± 2.76KB 3.62MB … 3.63MB 0 ( 0%) 0%
Benchmark 2 (10 runs): winpathbench-fat32-fix.exe
measurement mean ± σ min … max outliers delta
wall_time 500ms ± 21.4ms 480ms … 535ms 0 ( 0%) - 1.5% ± 2.8%
peak_rss 3.62MB ± 2.76KB 3.62MB … 3.63MB 0 ( 0%) - 0.0% ± 0.1%
---
Partially addresses #16374 (it fixes `zig build` on FAT32 when no `zig-cache` is present)
Since the same Key.Func data structure is used for coerced function
bodies as well as uncoerced function bodies, there is danger of them
being hashed and equality-checked as the same. When that happens, the
type of a function body value might be wrong, causing lots of problems.
In this instance, it causes an assertion failure.
This commit fixes it by introducing an `uncoerced_ty` field which
is different than `ty` in the case of `func_coerced` and is used to
differentiate when doing hashing and equality checking.
I have a new behavior test to cover this bug, but it revealed *another*
bug at the same time, so I will fix it in the next commit and and add
the new test therein.
This commit changes the type of the first parameter of parseTimeDigits
to *const [2]u8 for consistency with parseYear4 which uses *const [4]u8
as its first parameter. This is also more ergonomic for the caller since
they don't need to dereference the array.
This commit fixes parsing in parseYear4 and parseTimeDigits by using a
wider vector data type such that the intermediate result cannot overflow
and the error check remains correct.
These two tests can't be disambiguated at comptime, since the filesystem that the test is running on also matters for whether or not POSIX_SEMANTICS / IGNORE_READONLY_ATTRIBUTE can actually be used (since they are only supported on NTFS).
Using FileDispositionInformationEx (and therefore flags like FILE_DISPOSITION_POSIX_SEMANTICS and FILE_DISPOSITION_IGNORE_READONLY_ATTRIBUTE) is only supported on NTFS, so the comptime Windows version range check is not enough to determine whether or not the NtSetInformationFile will succeed.
This commit makes DeleteFile always try using FileDispositionInformationEx first, but if INVALID_PARAMETER is received (which is the status that occurs when the filesystem doesn't support FileDispositionInformationEx), then it will fallback and try calling NtSetInformationFile with FileDispositionInformation.
This keeps NTFS as fast as it was before, since it will do at most 1 NtSetInformationFile call, but on non-NTFS filesystems (e.g. FAT32), DeleteFile may need to do 2 NtSetInformationFile calls.
Closes#16497
There were two missing places.
Regressed in the #16318 branch.
Found from compiling Bun. Unfortunately we do not have a behavior test
reduction for this bug.
The logic incorrectly assumed that adhoc_inferred_error_set_type would
be part of the inferred_error_set InternPool.Key when it actually is
part of `simple_type`.
Regressed in the #16318 branch.
Found from compiling Bun. Unfortunately we do not have a behavior test
reduction for this bug.
Apparently hard things:
* run autodoc on the std lib
* build the langref
If any zig distros want these documentation, they'll need to get them
from a stage3 `zig build` instead of stage2 (LLVM backend instead of C
backend).
* build.zig: introduce `-Dflat` option which makes the installation
match what we want to ship for our download tarballs. This allows
deleting a bunch of shell script logic from the CI.
- for example it puts the executable directly in prefix/zig rather
than prefix/bin/zig and it additionally includes prefix/LICENSE.
* build.zig: by default also install std lib documentation to doc/std/
- this can be disabled by `-Dno-autodocs` similar to how there is
already `-Dno-langref`.
* build.zig: add `std-docs` and `langref` steps which build and install
the std lib autodocs and langref to prefix/doc/std and
prefix/doc/langref.html, respectively.
* std.Build: implement proper handling of `-femit-docs` using the
LazyPath system. This is a breaking change.
- this is a partial implementation of #16351
* frontend: fixed the handling of Autodocs with regards to caching and
putting the artifacts in the proper location to integrate with the
build system.
- closes#15864
* CI: delete the logic for autodocs since it is now handled by build.zig
and is enabled by default.
- in the future we should strive to have nearly all the CI shell
script logic deleted in favor of `zig build` commands.
* CI: pass `-DZIG_NO_LIB=ON`/`-Dno-lib` except for the one command where
we want to actually generate the langref and autodocs. Generating the
langref takes 14 minutes right now (why?!) so we don't want to do that
more times than necessary.
* Autodoc: fixed use of a global variable. It works fine as a local
variable instead.
- note that in the future we will want to make Autodoc run
simultaneously using the job system, but for now the principle of
YAGNI dictates that we don't have an init()/deinit() API and instead
simply call the function that does the things.
* Autodoc: only do it when there are no compile errors
Both ways do the same thing but I think the compiler might have an
easier time optimizing `appendNTimes` because it does less
things/the path is shorter.
I have not done any benchmarking at runtime but have compared the
instruction count of both ways a little here: https://zig.godbolt.org/z/vr193W9oj
`b` (`appendNTimes`) is ~103 instructions while `a`
(`writer().writeByteNTimes`) is ~117 instructions.
And looking at the implementation of `writeByteNTimes`, it only seems to
buffer up 256 bytes before doing another `writeAll` which for
`std.ArrayList` probably means another allocation, whereas when directly
using `appendNTimes`, the entire exact additional capacity required is known from the start.
Either way, this would be more consistent anyway.
When a signed integer's bitsize is not 32 or 64, but the given
bitsize and wanted bitsize are either both represented by Wasm's i32
or i64, we must either sign extend or wrap the integer.
Currently, in a debug build of the compiler, `@embedFile("")` is a crash;
in a release build the compiler, `@embedFile("")` is "error: unable to open '': OutOfMemory".
dwarf: fixup x86 register mapping logic
dwarf: change the register context update to update in-place instead of copying
debug: always print the unwind error type
This change enhances stack trace output to include a note that debug info was missing,
and therefore the stack trace may not be accurate. For example, if the user is using a libc
compiled with -fomit-frame-pointer and doesn't have debug symbols installed, any traces
that begin in a libc function may not unwind correctly. This allows the user to notice this and
potentially install debug symbols to improve the output.
Some DWARF5 subprograms have non-contiguous instruction ranges. An example of such
a function is `puts` in Ubuntu's libc. This change fixes name lookups for functions that use
DW_AT_range in their DIE.
I had accidentally regressed support for -gdwarf in 461fb499f3 when I changed the logic to
use the already-mapped exe/dll image instead of loading it from disk. The string table is mapped as all zeroes by the loader,
so if a section header's name is longer than 8 bytes (like the ones generated by -gdwarf), then the name can't be read.
Now, if any section headers require the string table, the file is mapped from disk.
windows: Add NtCreateSection/NtMapViewOfSection/NtUnmapViewOfSection
- Fix unwindFrame using the previous FDE row instead of the current one
- Handle unwinding through noreturn functions
- Add x86-linux getcontext
- Fixup x86_64-linux getcontext not restoring the fp env
- Fix start_addr filtering on x86-windows
- .eh_frame_hdr contains a binary-searchable data structure for finding an FDE. If present, we can use this
section to avoid having to parse the entire FDE/CIE list in the binary, instead only entries that are actually
required for unwinding are read.
- rework the inputs pc-relative pointer decoding to support both already-mapped sections as well as sections
mapped from a file
- store the VirtualMachine on UnwindContext so the allocations can be reused
Some distributions (ie. Ubuntu) have their libc debug
info in separate files. This change allows the stack walking
code to read that debug info.
- add support for reading compressed ELF sections
- support reading the build-id from the elf headers in order to lookup external debug info
- support reading the .gnu_debuglink section to look up external debug info
This is consistent with what clang and gcc are doing, because
headers such as <cet.h> are specifically designed to be used
in the context of assembly code.
Fixes#16449
At the moment, the LLVM IR we generate for this fn is
define internal fastcc void @AstGen.numberLiteral ... {
Entry:
...
%16 = alloca %"fmt.parse_float.decimal.Decimal(f128)", align 8
...
That `Decimal` is huuuge! It stores
pub const max_digits = 11564;
digits: [max_digits]u8,
on the stack.
It comes from `convertSlow` function, which LLVM happily inlined,
despite it being the cold path. Forbid inlining that to not penalize
callers with excessive stack usage.
Backstory: I was looking for needles memcpys in TigerBeetle, and came up
with this copyhound.zig tool for doing just that:
ee67e2ab95/src/copyhound.zig
Got curious, run it on the Zig's own code base, and looked at some of
the worst offenders.
List of worst offenders:
warning: crypto.kyber_d00.Kyber.SecretKey.decaps: 7776 bytes memcpy
warning: crypto.ff.Modulus.powPublic: 8160 bytes memcpy
warning: AstGen.numberLiteral: 11584 bytes memcpy
warning: crypto.tls.Client.init__anon_133566: 13984 bytes memcpy
warning: http.Client.connectUnproxied: 16896 bytes memcpy
warning: crypto.tls.Client.init__anon_133566: 16904 bytes memcpy
warning: objcopy.ElfFileHelper.tryCompressSection: 32768 bytes memcpy
Note from Andrew: I removed `noinline` from this commit since it should
be enough to set it to be cold.
Instead, we now have a looser helper called `checkContains(...)`
that will match on any occurrence similarly to `std.mem.indexOf()`.
While at it, I have cleaned up other combinators to make the entire
API more consistent, and so:
* `checkStart(phrase)` is now `checkStart()` followed by
`checkExact(phrase)`
* `checkNext(phrase)` if matching exactly is now `checkExact(phrase)`
* `checkNext(phrase)` if matching loosely is now `checkContains(phrase)`
* `checkNext(phrase)` if matching exactly with var extractors is now
`checkExtract(phrase)`
Finally, `ElfDumper` is now dumping contents of `.symtab` and `.dynsym`
symbol tables. I have also removed dumping of symtabs as optional - they
are now always dumped which cleaned up the implementation even more.
It is assumed that generating a collision requires more than 2^156
ciphertext modifications. This is plenty enough for any practical
purposes, but it hasn't been proven to be >= 2^256.
Be consistent and conservative here; just claim the same security
as the other variants.
Previously, they were only created when we had any TLS segment.
This meant that while the symbol existed, the global itself wouldn't.
The result of this was a crash during symbol names writing as it
would attempt to write the symbol name of a global that didn't exist.
Now we always create them, and instead update its `init` value during
`setupMemory`.
In the future, the entire symbol (and global) will be removed by
the garbage collector.
Implements the `start` section which will execute a given function
at startup of the program. After function execution, the _start
function will be called by the runtime. In the case of shared-memory
we set this section to the function `__wasm_init_memory` which will
initialize all memory on startup.
This also fixes the above mentioned function to ensure we correctly
lower the i32 values.
Lastly, this fixes a typo where we would retrieve a global, instead
of setting its value.
Rather than verifying if importing memory is false, we now rely
on the option that was passed to the CLI (where export is defaulted
to `true` unless only import-memory is given).
* getOwnedFunctionIndex no longer checks if the value is actually a
function.
* The callsites to `intern` that I added want to avoid the `getCoerced`
call, so I added `intern2`.
* Adding to inferred error sets should not happen if the destination
error set is not the inferred error set of the current Sema instance.
* adhoc_inferred_error_set_type can be seen by the backend. Treat it
like anyerror.
when other function's inferred error set is the return type of a
function, it should not try to insert the error set into it.
this implies that this branch fixes a bug in master branch.
When instantiating a generic function call, in the case of a function
call generated by the language, the LazySrcLoc provided for `call_src`
may not point to anything interesting.
* Introduce InternPool.Tag.func_coerced to handle the case of a
function body coerced to a new type. `InternPool.getCoerced` is now
implemented for function bodies in this branch.
* implement resolution of ad-hoc inferred error sets in
`Sema.analyzeCall`.
* fix generic_owner being set wrong for child Sema bodies of param
expressions.
* fix `Sema.resolveInferredErrorSetTy` when passed `anyerror`.
Previously, they shared function index with the owner decl, but that
would clobber the data stored for inferred error sets of runtime calls.
Now there is an adhoc_inferred_error_set_type which models the problem
much more correctly.
Before, it incorrectly passed an InternPool.Index where an extra array
index was expected (to the function which is renamed to `extraErrorSet`
in this commit).
There is one case where function types may be inequal but we still want
to find the same function body instance in InternPool.
In the case of the functions having an inferred error set, the key used
to find an existing function body will necessarily have a unique
inferred error set type, because it refers to the function body
InternPool Index. To make this case work we omit the inferred error set
from the equality and hashing functions.
* move inferred error sets into InternPool.
- they are now represented by pointing directly at the corresponding
function body value.
* inferred error set working memory is now in Sema and expires after
the Sema for the function corresponding to the inferred error set is
finished having its body analyzed.
* error sets use a InternPool.Index.Slice rather than an actual slice
to avoid lifetime issues.
The idea here is to move towards a future where anonymous decls are
represented entirely by an `InternPool.Index`. This was needed to start
implementing `InternPool.getFuncDecl` which requires moving creation and
deletion of Decl objects into InternPool.
* remove `Namespace.anon_decls`
* remove the concept of cleaning up resources from anonymous decls,
relying on InternPool instead.
* move namespace and decl object allocation into InternPool
Abridged summary:
* Move `Module.Fn` into `InternPool`.
* Delete a lot of confusing and problematic `Sema` logic related to
generic function calls.
This commit removes `Module.Fn` and replaces it with two new
`InternPool.Tag` values:
* `func_decl` - corresponding to a function declared in the source
code. This one contains line/column numbers, zir_body_inst, etc.
* `func_instance` - one for each monomorphization of a generic
function. Contains a reference to the `func_decl` from whence the
instantiation came, along with the `comptime` parameter values (or
types in the case of `anytype`)
Since `InternPool` provides deduplication on these values, these fields
are now deleted from `Module`:
* `monomorphed_func_keys`
* `monomorphed_funcs`
* `align_stack_fns`
Instead of these, Sema logic for generic function instantiation now
unconditionally evaluates the function prototype expression for every
generic callsite. This is technically required in order for type
coercions to work. The previous code had some dubious, probably wrong
hacks to make things work, such as `hashUncoerced`. I'm not 100% sure
how we were able to eliminate that function and still pass all the
behavior tests, but I'm pretty sure things were still broken without
doing type coercion for every generic function call argument.
After the function prototype is evaluated, it produces a deduplicated
`func_instance` `InternPool.Index` which can then be used for the
generic function call.
Some other nice things made by this simplification are the removal of
`comptime_args_fn_inst` and `preallocated_new_func` from `Sema`, and the
messy logic associated with them.
I have not yet been able to measure the perf of this against master
branch. On one hand, it reduces memory usage and pointer chasing of the
most heavily used `InternPool` Tag - function bodies - but on the other
hand, it does evaluate function prototype expressions more than before.
We will soon find out.
The code removed does unnecessary copying in order to create a null-terminated pointer, just to pass it to libc getenv. It only does this for `small keys`, which are under 64 bytes in size.
Instead of going out of the way to add a null byte to a function that takes normal slices, this should just be handled by the loop below, which scans c.environ to find the value
And when we have the choice, favor little-endian because it's 2023.
Gives a slight performance improvement:
md5: 552 -> 555 MiB/s
sha1: 768 -> 786 MiB/s
sha512: 211 -> 217 MiB/s
* Small documentation fix of ChaCha variants
Previous documentation was seemingly copy-pasted and left
behind some errors where the number of rounds was not
properly updated.
* Suggest `std.crypto.utils.secureZero` on `@memset` docs
* Revert previous change
* autodoc: init work to refactor exprName
* autodoc: Implement more expressions in exprName refactor
* autodoc: more work
* autodoc: More exprName to ex refactoring
* autodoc: Remove whitespace flag from renderer; Add pre tags in
value and variable drawing in renderContainer
* autodoc: add inline styling to pre blocks
* autodoc: move renderer code to main.js
* autodoc: More exprName to ex refactoring; Fn signatures rendered with new code
* autodoc: Fix function rendering. Add more things to ex
* autodoc: nuke exprName
---------
Co-authored-by: Krzysztof Wolicki <der.teufel.mail@gmail.com>
The 'at least' behavior of the Allocator interface was removed in #13666, so anything that used reallocAtLeast or the .at_least Exact behavior could still have doc comments that reference no-longer-true behavior.
Funnily enough, ArrayList is the only place that used this functionality (outside of allocator test cases), so its doc comments are the only things that need to be fixed. This was checked by resetting to deda6b5146 and searching for all instances of `reallocAtLeast` and `.at_least` (one of which would need to be used to get the `.at_least` behavior)
Needed due to the breaking changes to `@splat` which are used by the
self-hosted compiler.
This update also includes the improvement that allows casting builtins
to infer the result type through optionals and error unions.
Don't pass the object files from a static library to the linker invocation.
The lib.a file already contains them.
Avoids "duplicate symbol" errors (and useless work by the linker)
This adds a standalone test case to ensure the runtime does not trap
when performing a memory.copy or memory.fill instruction while the
destination or source address is out-of-bounds and the length is 0.
Fixes#16311
The actual cause of #16311 is the `start_is_zero` special case:
```zig
const range_len = if (end_val == .none or start_is_zero)
end_val
else
try parent_gz.addPlNode(.sub, input, Zir.Inst.Bin{
.lhs = end_val,
.rhs = start_val,
});
```
It only happens if the range start is 0. In that case we would not perform any type checking.
Only in the other cases coincidentally `.sub` performs type checking in Sema, but the errors are still rather poor:
```
$ zig test x.zig
x.zig:9:15: error: invalid operands to binary expression: 'Pointer' and 'Pointer'
for ("abc".."def") |val| {
~~~~~^~~~~~~
```
Note how it's the same as if I use `-`:
```
x.zig:9:11: error: invalid operands to binary expression: 'Pointer' and 'Pointer'
"abc" - "def";
~~~~~~^~~~~~~
```
Now after this PR, the errors are much clearer for both range start and end:
```
x.zig:9:10: error: expected type 'usize', found '*const [3:0]u8'
for ("abc".."def") |val| {
^~~~~
```
This is why I decided to use `.ty` instead of `.coerced_ty` for both range start and end rather than
just perform type checking in that `end_val == .none or start_is_zero` case.
Now you can add new calling conventions that you confirmed to work with
variadic functions simply in a single place and the rest will work
automatically.
I tested this and this definitely compiles and these
changes were done programmatically but if there's still anything wrong
it shouldn't be hard to fix.
With this change it's going to be very easy to make further adjustments
to the calling conventions of all these external UEFI functions.
Closes#16309
When lowering the `memset` instruction, LLVM will lower it to WebAssembly's
`memory.fill` instruction when the bulk-memory feature is enabled. This
instruction will trap when the destination address is out-of-bounds.
By Zig's semantics, it is valid to have an invalid pointer when the length is 0.
To prevent runtimes from trapping, we add a safety-check for slices to only
lower to a memset instruction when the length is larger than 0.
When lowering the `memcpy` instruction, LLVM will lower it to WebAssembly's
`memory.copy` instruction when the bulk-memory feature is enabled. This
instruction will trap when the destination or source pointer is out-of-bounds.
By Zig's semantics, it is valid to have an invalid pointer when the length is 0.
To prevent runtimes from trapping, we add a safety-check for slices to only
lower to a memcpy instruction when the length is larger than 0.
* Move functionality from generic functions that doesn't depend on the type into a function that only depends on comptime alignment.
This reduces comptime code duplication because e.g. `alloc(u32, )` and `alloc(i32, )` now use the same function `allocWithFoo(4, 4, )` under the hood.
* `CMakeLists.txt`: support the weird `uname -m` output.
* `CMakeLists.txt`: detect and use the C compiler's default arm mode.
* cbe: support gcc with both `f128` and `u128` emulated.
* std.os.linux.thumb: fix incorrectly passed asm inputs.
Commit c0b774fbc6 originally added this
logic but it did not properly clear the C command line flags to empty
when a previous positional argument had command line flags, because it
never set the "previous" flag to true.
This fixes C compiler flags not being reset to empty when using the
build system and a second positional argument has no arguments after a
first positional argument has arguments.
Thanks to @squeek502 for finding this.
All of the std except these few functions call it "eql" instead of "eq".
This has previously tripped me up when I expected the equality check function to be called "eql"
(just like all the rest of the std) instead of "eq".
The motivation is consistency.
If search "eq" on Autodoc, these functions stick out and it looks inconsistent.
I just noticed there are also a few functions spelling it out as "equal" (such as std.mem.allEqual).
Maybe those functions should also spell it "eql" but that can be done in a future PR.
* Most of the js lexer ported
* Fixed lexical Ranges
* Mini test framework
* More testing and bug fixing in zig js lexer
* Removed bad import
* Implemented good html generation
Closes#15863
e05412669c inadvertently changed the logic on Windows and made it so that supportsAnsiEscapeCodes was never checked. This fixes that regression while keeping the logic intact for other platforms.
The panic handler expects that this value is represented with the
generic address space, so cast the global to the generic address-
space before caching and returning the value.
The SPIR-V cannot print the helpful error messages from the
std.testing module. This commit overrides the default print
function used here to one that only prints if the target supports
it. For now, its only done for SPIR-V, but this function could
be adapted to more targets that need it.
This implements the semantics as discussed in today's compiler meeting,
where the alignment of pointers to fields of default-layout unions
cannot exceed the field's alignment.
Resolves: #15878
This allows doing Windows-style case insensitive comparisons from any target, but means that it will need to include its own copy of the uppercase data table (5,088 bytes) to do so.
When targeting Windows, the ntdll functions are used instead to avoid including a redundant copy of the uppercase data in the resulting binary.
Previously, interned values were represented as AIR instructions using
the `interned` tag. Now, the AIR ref directly encodes the InternPool
index. The encoding works as follows:
* If the ref matches one of the static values, it corresponds to the same InternPool index.
* Otherwise, if the MSB is 0, the ref corresponds to an InternPool index.
* Otherwise, if the MSB is 1, the ref corresponds to an AIR instruction index (after removing the MSB).
Note that since most static InternPool indices are low values (the
exceptions being `.none` and `.var_args_param_type`), the first rule is
almost a nop.
When targeting WebAssembly, we default to building a single-threaded build
as threads are still experimental. The user however can enable a multi-
threaded build by specifying '-fno-single-threaded'. It's a compile-error
to enable this flag, but not also enable shared-memory.
When a thread is detached from the main thread, we automatically
cleanup any allocated memory. For this we first reset the stack-pointer
to the original stack-pointer of the main-thread so we can safely clear
the memory which also contains the thread's stack.
When `join` detects a thread has completed, it will free the allocated
memory of the thread. For this we must first copy the allocator. This is
required as the allocated memory holds a reference to the original
allocator. If we free the memory, we would end up with UB as the
allocator would free itself.
We now reset the Thread ID to 0 and wake up the main thread listening
for the thread to finish. We use inline assembly as we cannot use
the stack to set the thread ID as it could possibly clobber any
of the memory.
Currently, we leak the memory that was allocated for the thread.
We need to implement a way where we can clean up the memory without
using the stack (as the stack is stored inside this same memory).
We now store the original allocator that was used to allocate the
memory required for the thread. This allocator can then be used
in any cleanup functionality to ensure the memory is freed correctly.
Secondly, we now use a function to set the stack pointer instead of
generating a function using global assembly. This is a lot cleaner
and more readable.
This implements a first version to spawn a WASI-thread. For a new thread
to be created, we calculate the size required to store TLS, the new stack,
and metadata. This size is then allocated using a user-provided allocator.
After a new thread is spawn, the HOST will call into our bootstrap procedure.
This bootstrap procedure will then initialize the TLS segment and set the
newly spawned thread's TID. It will also set the stack pointer to the newly
created stack to ensure we do not clobber the main thread's stack.
When bootstrapping the thread is completed, we will call the user's
function on this new thread.
Implements std's `Futex` for the WebAssembly target using Wasm's
`atomics` instruction set. When the `atomics` cpu feature is disabled
we emit a compile-error.
When the user enabled the linker-feature 'shared-memory' we do not force
a singlethreaded build. The linker already verifies all other CPU features
required for threads are enabled. This is true for both WASI and
freestanding.
This flag allows the user to force export the memory to the host
environment. This is useful when the memory is imported from the
host but must also be exported. This is (currently) required
to pass the memory validation for runtimes when using threads.
In this future this may become an error instead.
This implements the semantics as discussed in today's compiler meeting,
where the alignment of pointers to fields of default-layout unions
cannot exceed the field's alignment.
Resolves: #15878
emmalloc.c does a fair amount of type punning in order to access
the size of memory regions and traverse them.
Unfortunately, that can lead to unwanted optimizations.
This simple test case currently triggers a memory fault:
int main(void) {
char * volatile p = malloc(1);
p = realloc(p, 12);
p = malloc(1);
printf("%p\n", p);
}
Work around this by adding "-fno-strict-aliasing" when compiling
that file.
.res files are compiled Windows resource files that get linked into executables/libraries. The linker knows what to do with them, but previously you had to trick Zig into thinking it was an object file (by renaming it to have the .obj extension, for example).
After this commit, the following works:
zig build-exe main.zig resource.res
or, in build.zig:
exe.addObjectFile("resource.res");
Closes#6488
This actually used to be how it worked in stage1, and there was this
issue to change it: #2649
So this commit is a reversal to that idea. One motivation for that issue
was avoiding emitting the panic handler in compilations that do not have
any calls to panic. This commit only resolves the panic handler in the
event of a safety check function being emitted, so it does not have that
flaw.
The other reason given in that issue was for optimizations that elide
safety checks. It's yet to be determined whether that was a good idea or
not; this can get re-explored when we start adding optimization passes
to AIR.
This commit adds these AIR instructions, which are only emitted if
`backendSupportsFeature(.safety_checked_arithmetic)` is true:
* add_safe
* sub_safe
* mul_safe
It removes these nonsensical AIR instructions:
* addwrap_optimized
* subwrap_optimized
* mulwrap_optimized
The safety-checked arithmetic functions push the burden of invoking the
panic handler into the backend. This makes for a messier compiler
implementation, but it reduces the amount of AIR instructions emitted by
Sema, which reduces time spent in the secondary bottleneck of the
compiler. It also generates more compact LLVM IR, reducing time spent in
the primary bottleneck of the compiler.
Finally, it eliminates 1 stack allocation per safety-check which was
being used to store the resulting tuple. These allocations were going to
be annoying when combined with suspension points.
Needed due to the breaking changes to casting builtins, which are used
by the compiler when building itself.
Note from Andrew: I re-ran update-zig1 on my PC and replaced this
commit.
Signed-off-by: Andrew Kelley <andrew@ziglang.org>
Most of this migration was performed automatically with `zig fmt`. There
were a few exceptions which I had to manually fix:
* `@alignCast` and `@addrSpaceCast` cannot be automatically rewritten
* `@truncate`'s fixup is incorrect for vectors
* Test cases are not formatted, and their error locations change
This reverts commit fa6cea22bf.
Apologies for the merge. I thought this was a bug fix, but I see that it
is implementing a proposal that I intended to reject.
DeclGen/FuncGen methods are for things that pertain to a particular
declaration or function, while Object methods are for things that
pertain to the entire LLVM Module. Many methods were in the wrong
category, such as type and value lowering.
This is a prerequisite commit for a local branch I am working on, which
needs to be able to call lowerValue() without the context of any
particular function or declaration.
The signature is `getOrCreateSubrange(int64_t Lo, int64_t Count)`, so this updates the bindings to match.
This fixes a crash in `lowerDebugTypeImpl` when analyzing slices that have a length of 2^32 or
larger (up to `2^64 >> 3`, which still crashes, because above that the array size in bits overflows u64).
(Firstly, I changed `n` to `b`, as that is less confusing. It's not a length, it's a right boundary.)
The invariant maintained is `cur < b`. In the worst case `2*cur + 1` results in a maximum of `2b`. Since `2b` is not guaranteed to be lower than `maxInt`, we have to add one overflow check to `siftDown` to make sure we avoid undefined behavior.
LLVM also seems to have a nicer time compiling this version of the function. It is about 2x faster in my tests (I think LLVM was stumped by the `child += @intFromBool` line), and adding/removing the overflow check has a negligible performance difference on my machine. Of course, we could check `2b <= maxInt` in the parent function, and dispatch to a version of the function without the overflow check in the common case, but that probably is not worth the code size just to eliminate a single instruction.
This makes it so `first_child_index` will not be accessed when it is equal to `self.len`. (i.e. `self.items[self.len]` will not happen) The access itself was "safe" (as in, `self.len < self.items.len`) because we were only calling `doSiftDown` in the case where there was a stale value at `self.items[self.len]`. However, it is still technically a bug, and can manifest by an unnecessary comparison of a value to a copy of itself.
LLVM has trouble compiling the old implementation, (presumably) because `leading_zeros` is thought to be a `u7` rather than a `u6`, which means `63 - clz` is not equivalent to `63 ^ clz`, which means it can't deduce that the final condition can simply be flipped. (I am assuming `usize` is a `u64` here for ease of understanding, but it's the same for any power of 2)
https://zig.godbolt.org/z/Pbj4P7ob3
The new version is slightly better too because `isMinLayer(maxInt(usize))` is now well-defined behavior.
The 'Content-Length' header was inspected by mistake,
which makes it effectively impossible to use chunked
Transfer-Encoding when using the http client.
Tested locally with a HTTP server - data is properly sent
with POST method and the proper encoding declared, after the fix.
Opaque and `noreturn` makes sense since they don't represent real
values, but `null` and `undefined` are perfectly normal
comptime-only values.
Closes#16088
musl v1.2.4 dropped the "64"-suffixed aliases for legacy "LFS64" ("large
file support") interfaces, so this commit changes the corresponding Zig
logic to call the correct names.
"SHT_NOBITS" sections can be easily moved around in the file, they can be anywhere since there is no data...
and remove logic about the categorization of symbol tables:
I don't understand or remember why it was needed,
and, in my tests, it works fine without...
It previouly failed to strip an exe linked with `mold`
The runCases function incorrectly called the introspect.findZigLibDir
function, causing a possible error, depending on the location of the
local cache directory, since the current executable is check-case.
Use findZigLibDirFromSelfExe, passing the zig_exe_path argument.
The current CI scripts work correctly because ZIG_LOCAL_CACHE_DIR is set
to "$(pwd)/zig-local-cache".
Fixes#15044
PPC targets can also use the functionality-equivalent standard routine, so
unconditionally export the standard routine.
Fixup of #16054 merged in f043071cdf.
The assert is changed from `int != 0` to `int > 0` because negative
integers always return `false`.
Python's `math.log2` does the same and errors for 0 or negative integers.
isatty on Windows is implemented as a isCygwinPty call and a GetConsoleMode call, so calling isatty just duplicates the function calls we already need to do in supportsAnsiEscapeCodes.
Make to avoid releasing request's connection twice.
Change the `Request.connection` field optional. This field is null while the connection is released.
Fixes#15965
Adress review comments from https://github.com/ziglang/zig/pull/13977
by using the same naming convention as zstd.
And by using `finish()` instead of `close()` for the finalisation of the compressed stream.
rationale:
- it is not the same as how close() is usually used, since it must be called to flush and write the final bytes. And as such it may fail.
- it is not the same `flush` in the deflate code, which allows to keep writting more bytes later, and doesn't write the final checksum.
- it is the same name as used in the original zlib library (Z_FINISH)
Also, use a packed struct for the header, which seems a better fit.
* langref: improve the Hello World section
Clarify that a Zig source file must have the .zig extension.
Fix a typo.
* langref: improve the Comments section
- Document how doc comments are used by -femit-docs
- Rename "Doc comments" to "Doc Comments", for consistency
- Clarify that placing a doc comment in an unexpected place is a
compiler error and add two incorrect examples
- Document the current Autodoc behavior, when normal comments are
interleaved with doc comments
- Rewrite the documentation for top-level doc comments
* langref: improve the Zig Test section
Document that the test name can also be an identifier, in addition to a
string literal.
Update the test output for the first test.
Closes#14085
Anecdote 1: The generic version is way more popular than the non-generic
one in Zig codebase:
git grep -w alignForward | wc -l
56
git grep -w alignForwardGeneric | wc -l
149
git grep -w alignBackward | wc -l
6
git grep -w alignBackwardGeneric | wc -l
15
Anecdote 2: In my project (turbonss) that does much arithmetic and
alignment I exclusively use the Generic functions.
Anecdote 3: we used only the Generic versions in the Macho Man's linker
workshop.
Adds conditional exports
- __fixkfti
- __fixunskfti
- __floattikf
- __negkf2
- __mulkc3
- __divkc3
- __powikf2
and adjusts tools/gen_stubs.zig.
From https://gcc.gnu.org/onlinedocs/gcc/Floating-Types.html:
"When long double transitions to __float128 on PowerPC in the future,
__ibm128 will remain for use in conversions between the two types."
Hence `__extendkftf2` and `__trunctfkf2` for conversion are superfluous
and only using f128 for `kf` routines is justified.
Closes#16057.
Missed this originally because I was only able to trigger it when
SA_RESTART was missing from the sigaction handlers. I'm unconvinced this
is actually a sane way for stdlib to behave (see #15664). But this does
duplicate the existing behavior throughout os.zig which IMO should be
prioritized here.
See https://reviews.llvm.org/D81809 for upstream description.
In summary this is ~10x improvement for small divisors and similar
performance for equal divisors.
Closes#13523.
This is needed in order to remove math.{min,max} from std in favour of
the builtins, since the builtins need the behavior fix from the previous
commit.
Note from Andrew: I updated this commit with zig1.wasm built by me.
Signed-off-by: Andrew Kelley <andrew@ziglang.org>
I achieved this through a major refactor of the logic of analyzeMinMax.
This change should be compatible with vectors of comptime_int, which
Andrew said are supposed to work (but which currently do not).
We now resolve undefined symbols during incremental-compilation
where we discard the current symbol if we detect we found
an existing symbol which is not the one currently being updated.
The symbol will always be discarded in favor of the existing symbol
in such a case.
When lowering a decl value we verify whether its owner decl index
equals to the decl index of the decl being lowered. When this is not
the case, we are lowering an alias. So instead, we will now lower
the owner decl instead and call its symbol to ensure its type
is being correctly generated.
When compiling Zig code using the Wasm backend, we would previously
incorrectly resolve exported symbols as it would not correctly remove
existing symbols if they were to be overwritten. This meant that
undefined symbols could cause collisions although they should be
resolved by the exported symbol.
- fix getdents return type usize → c_int
- special-case process.zig to use sysctl instead of sysctlbyname
- use struct/field pattern for sysctl HW_* constants
- fix getdents return type usize → isize
- usize ultimately forced errors to .SUCCESS in std.c.getError
New behavior in freebsd 13.2 is to return ENOENT if the directory being
iterated is deleted during iteration. We now detect this and treat it
consistent with iteration ending.
Calling into coercion logic here is a little opaque, and more to the
point wholly unnecessary. Instead, the (very short) logic is now
implemented directly in Sema.
Resolves: #16033
Previously, this logic was split between Sema.coerceValueInMemory and
InternPool.getCoerced. This led to issues when trying to coerce e.g. an
optional containing an aggregate, because we'd call through to
InternPool's version which only recurses on itself so could not coerce
aggregates. Unifying them is fairly simple, and also simplified a bit of
logic in Sema.
Also fixes a key lifetime bug in aggregate coercion.
This allows tuples whose fields are in-memory coercible to themselves be
coerced in memory. No InMemoryCoercionResult field has been added, so in
future one could be added to improve error messages.
The existing logic for peer type resolution was quite convoluted and
buggy. This rewrite makes it much more resilient, readable, and
extensible. The algorithm works by first iterating over the types to
select a "strategy", then applying that strategy, possibly applying peer
resolution recursively.
Several new tests have been added to cover cases which the old logic did
not correctly handle.
Resolves: #15138Resolves: #15644Resolves: #15693Resolves: #15709Resolves: #15752
Failing to forward free calls to the underlying allocator makes
`ValidationAllocator` unusable for testing allocators while checking for
leaks. This change allows allocators that wrap `std.testing.allocator`
to be tested with `std.heap.testAllocator()` in test decls without
reporting erroneous leaks.
To do this, I expanded SwitchProngSrc a bit. Several of the tags there
aren't actually used by any current errors, but they're there for
consistency and if we ever need them.
Also delete a now-redundant test and fix another.
This is a bit harder than it seems at first glance. Actually resolving
the type is the easy part: the interesting thing is actually getting the
capture value. We split this into three cases:
* If all payload types are the same (as is required in status quo), we
can just do what we already do: get the first field value.
* If all payloads are in-memory coercible to the resolved type, we still
fetch the first field, but we also emit a `bitcast` to convert to the
resolved type.
* Otherwise, we need to handle each case separately. We emit a nested
`switch_br` which, for each possible case, gets the corresponding
union field, and coerces it to the resolved type. As an optimization,
the inner switch's 'else' prong is used for any peer which is
in-memory coercible to the target type, and the bitcast approach
described above is used.
Pointer captures have the additional constraint that all payload types
must be in-memory coercible to the resolved type.
Resolves: #2812
This finishes the process of consolidating switch expressions in ZIR
into as simple and compact a representation as is possible. There are
now just two ZIR tags dedicated to switch expressions: switch_block and
switch_block_ref, with the latter being for an operand passed by
reference.
This is a follow-up to a previous commit which eliminated switch_capture
and switch_capture_ref. All captures are now handled directly by
`switch_block`, which has also eliminated some unnecessary Block data in
Sema.
These tags are unnecessary, as this information can be more efficiently
encoded within the switch_block instruction itself. We also use a neat
little trick to avoid needing a dummy instruction (like is used for
errdefer captures): since the switch_block itself cannot otherwise be
referenced within a prong, we can repurpose its index within prongs to
refer to the captured value.
By indexing from the very first switch case rather than into scalar and
multi cases separately, the instructions for capturing in multi cases
become unnecessary, freeing up 2 ZIR tags.
1. When the arena is already empty, resetting with `retain_capacity` no longer
results in allocating a buffer with zero capacity.
This behavior was previously intended by the `(current_capacity == 0)` check,
but wasn't correctly implemented.
2. Resetting with `.{ .retain_with_limit = 0 }` is now equivalent to
`free_all` and a new buffer with zero capacity is no longer created.
This is a useful side-effect of the above fixes.
Previously, when the last buffer in `buffer_list` was retained after
deleting all other buffers, `buffer_list` wasn't updated and pointed
to a deleted buffer.
Previously, the buffer reserved with `retain_with_limit` was missing
space for the `BufNode`.
When the user-provided a limit that was smaller than `@sizeOf(BufNode)`,
`reset` would store a new `BufNode` in an allocation smaller than
`BufNode`, leading to a buffer overrun.
Long term, linker backends will need to manage their own string tables
for things like this because my mandate is: no long-lived pointers
allowed in any of the codepaths touched by incremental compilation, so
that we can serialize and deserialize trivially.
Short term, I solved this with a couple calls to Allocator.dupe,
incurring some harmless leaks.
By correctly handling comptime-only types appearing in non-comptime
parameters (when the parameter is either anytype or generic), this
avoids an index out of bounds later when later filling out
`monomorphed_args` using what used to be slightly different logic.
These are frequently invalidated whenever a string is interned, so avoid
creating pointers to `string_bytes` wherever possible. This is an
attempt to fix random CI failures.
Previously, these checks worked by performing the arithmetic operation,
then checking whether the result fit in the type in question. Since all
values are now typed, this approach was no longer valid, and was
tripping some assertions due to trying to store too-large values in
smaller types.
Now, `intAdd`, `intSub`, `intMul` and `intDiv` all check for overflow,
and if it happens, re-do the operation with the result being a
`comptime_int`, and reporting the error (and vector index) to the caller
so that the error can be reported.
After this change, all test cases are passing.
All but 2 test cases now pass (tested on x86_64 Linux, native only). The
remaining two signify an issue requiring a larger refactor, which I will
do in a separate commit.
Notable changes:
* Fix uninitialized memory when allocating objects from free lists
* Implement TypedValue printing for pointers
* Fix some TypedValue printing logic
* Work around non-existence of InternPool.remove implementation
Instead of carrying an optimized version of wyhash in the compiler for
small keys, put it into the std lib where it belongs.
...except it does not match the official test cases. This will need to
be fixed before merging into master. This is an extremely
contributor-friendly task.
Related issue: #15916
This is a workaround for InternPool currently not handling
non-null-terminated strings. It avoids using the `bytes` storage for
aggregates if there are any null bytes.
In the future this should be changed so that the `bytes` storage can be
used regardless of whether there are any null bytes. This is important
for use cases such as `@embedFile`.
However, this fixes a bug for now, and after this commit, stage2
self-hosts again.
mlugg: stage5 passes all enabled behavior tests on my system.
Commit message edited by Andrew Kelley <andrew@ziglang.org>
Notably, there was a bug where the fields of reified structs and unions
were allocated into an arena which was leaked. These are now in the
Module.tmp_hack_arena.
In an effort to delete `Value.hashUncoerced`, generic instantiation has
been redesigned. Instead of just storing instantiations in
`monomorphed_funcs`, partially instantiated generic argument types are
also cached. This isn't quite the single `getOrPut` that it used to be,
but one `get` per generic argument plus one get for the instantiation,
with an equal number of `put`s per unique instantiation isn't bad.
InternPool is nice in some ways but it also comes with its own set of
footguns. This commit fixes 5 instances. I see quite a few Valgrind
warnings remaining when running the behavior tests.
Perhaps the solution is to have stringToSlice return a struct with start
and length as indexes, which has a format function?
The main motivation for this commit is eliminating Decl.value_arena.
Everything else is dominoes.
Decl.name used to be stored in the GPA, now it is stored in InternPool.
It ended up being simpler to migrate other strings to be interned as
well, such as struct field names, union field names, and a few others.
This ended up requiring a big diff, sorry about that. But the changes
are pretty nice, we finally start to take advantage of InternPool's
existence.
global_error_set and error_name_list are simplified. Now it is a single
ArrayHashMap(NullTerminatedString, void) and the index is the error tag
value.
Module.tmp_hack_arena is re-introduced (it was removed in
eeff407941560ce8eb5b737b2436dfa93cfd3a0c) in order to deal with
comptime_args, optimized_order, and struct and union fields. After
structs and unions get moved into InternPool properly, tmp_hack_arena
can be deleted again.
This is neither a type nor a value. Simplifies `addStrLit` as well as
the many places that switch on `InternPool.Key`.
This is a partial revert of bec29b9e498e08202679aa29a45dab2a06a69a1e.
This is a continuation of 2f24228c758bc8a35d13379703bc1695008212b0.
This commit comes with smaller gains, but gains nonetheless. memcpy is
showing up as much less interesting in callgrind output for behavior
tests.
Current status: this branch is 1.15 ± 0.02 times slower than merge-base.
This is a bit odd, because this value doesn't actually exist:
see #15909. This gets all the empty enum/union behavior tests passing.
Also adds an assertion to `Sema.analyzeBodyInner` which would have
helped figure out the issue here much more quickly.
Key.PtrType is now an extern struct so that hashing it can be done by
reinterpreting bytes directly. It also uses the same representation for
type_pointer Tag encoding and the Key. Accessing pointer attributes now
requires packed struct access, however, many operations are now a copy
of a u32 rather than several independent fields.
This function moves the top two most used Key variants - pointer types
and pointer values - to use a single-shot hash function that branches
for small keys instead of calling memcpy.
As a result, perf against merge-base went from 1.17x ± 0.04 slower to
1.12x ± 0.04 slower. After the pointer value hashing was changed, total
CPU instructions spent in memcpy went from 4.40% to 4.08%, and after
additionally improving pointer type hashing, it further decreased to
3.72%.
The Zig language allows the compiler to make this optimization
automatically. We should definitely make the compiler do that, and
revert this commit. However, that will not happen in this branch, and I
want to continue to explore achieving performance parity with
merge-base. So, this commit changes all InternPool parameters to be
passed by const pointer rather than by value.
I measured a 1.03x ± 0.03 speedup vs the previous commit compiling the
(set of passing) behavior tests. Against merge-base, this commit is
1.17x ± 0.04 slower, which is an improvement from the previous
measurement of 1.22x ± 0.02.
Related issue: #13510
Related issue: #14129
Related issue: #15688
The old code assumed that `intAddScalar` could return a value outside
of the range of `ty`, which is problematic for many reasons.
The new code (ab)uses the InternPool for speed.
Recursion makes this hot function more difficult to profile and
optimize.
I measured a 1.05x speedup vs the previous commit with the (set of
passing) behavior tests.
This commit was the last in a series, and the main thing it needed to do
was make InternPool.typeOf not call indexToKey(). This required adding a
type field to the runtime_value encoding even though it is technically
redundant. This could have been avoided with a loop inside typeOf, but I
wanted to keep the machine code of that hot function as simple as
possible. The variable encoding is still responsible for a relatively
small slice of the InternPool data size.
I added a function that provides the payload type corresponding to the
InternPool.Tag type, which allows for some handy inline switch prongs.
Let's start moving the structs that are specific to InternPool.Tag into
the corresponding namespace. This will provide type safety if the
encoding of InternPool changes for these types later.
Recursion makes this hot function more difficult to profile and
optimize.
This commit adds the integer tag type to the type_enum_auto encoding
even though the integer tag type can be inferred based on the number of
fields of the enum. This avoids a call to getAssumeExists of the integer
tag type inside indexToKey.
Recursion makes this hot function more difficult to profile and
optimize.
The ptr_slice encoding now additionally includes the slice type. This
makes typeOf() implementable without indexToKey() as well as no longer
using recursion in the ptr_slice prong of indexToKey itself.
Unfortunately some logic had to be duplicated. However, I think that a
future enhancement could eliminate the duplication as well as remove
some other unwanted code, improving performance, by representing a slice
value in `Key.Ptr` without `addr` populated directly, but with an
`Index` pointing to the underlying manyptr value.
This is a hot function, and recursion makes it more difficult to
profile, as well as likely making it more difficult to optimize.
Previously, indexToKey for opt_payload would call getAssumeExists() on
the optional type. This made it possible to omit the optional type in
the encoding of opt_payload. However, getAssumeExists() *must* call
indexToKey because of hashing/equality.
So, this commit adds the optional type to the opt_payload encoding,
which increases its "extra" size from 0 to 8 bytes. As a result,
the opt_payload encoding went from not showing up on the top 25 largest
tags to...still not showing up in the top 25 largest tags.
This also helps make InternPool.typeOf() no longer need to call
indexToKey which is another hot function and another source of
recursion.
This is a particularly hot function, so we operate directly on encodings
rather than the more straightforward implementation of calling
`indexToKey`.
I measured this as 1.05 ± 0.04 times faster than the previous commit
with a ReleaseFast build against hello world (which includes std.debug
and formatted printing).
I also profiled the function and found that zigTypeTag() went from being
a major caller of `indexToKey` to being completely insignificant due to
being so fast.
I'm not sure if this is the right place for this to happen, and
it should become obsolete when comptime mutation is rewritten
and the remaining legacy value tags are remove, so keeping this
as a separate revertable commit.
Previously, there were types and values for inferred allocations and a
lot of special-case handling. Now, instead, the special casing is
limited to AIR instructions for these use cases.
Instead of storing data in Value payloads, the data is now stored in AIR
instruction data as well as the previously `void` value type of the
`unresolved_inferred_allocs` hash map.
Store `InternPool.Index` as the key instead which means that an AIR
instruction no longer needs to be burned to store the type, and also
that we can use AutoArrayHashMap instead of an ArrayList, which avoids
storing duplicates into the set, potentially saving CPU time.
This allows some code (like struct initializers) to use interned types
while other code (such as comptime mutation) continues to use legacy
types.
With these changes, an `zig build-obj empty.zig` gets to a crash on
missing interned error union types.
One change worth noting in this commit is that `module.global_error_set`
is no longer kept strictly up-to-date. The previous code reserved
integer error values when dealing with error set types, but this is no
longer needed because the integer values are not needed for semantic
analysis unless `@errorToInt` or `@intToError` are used and therefore
may be assigned lazily.
Also I moved `anyframe` from being represented by `SimpleType` to being
represented by the `none` tag of `anyframe_type` because most code wants
to handle these two types together.
Anonymous structs and anonymous tuples can be stored via a
only_possible_value tag because their type encodings, by definition,
will have every value specified, which can be used to populate the
fields slice in `Key.Aggregate`.
Also fix `isTupleOrAnonStruct`.
I'm seeing a new assertion trip: the call to `enumTagFieldIndex` in the
implementation of `@Type` is attempting to query the field index of an
union's enum tag, but the type of the enum tag value provided is not the
same as the union's tag type. Most likely this is a problem with type
coercion, since values are now typed.
Another problem is that I added some hacks in std.builtin because I
didn't see any convenient way to access them from Sema. That should
definitely be cleaned up before merging this branch.
Unlike unions and structs, enums are actually *encoded* into the
InternPool directly, rather than using the SegmentedList trick. This
results in them being quite compact, and greatly improved the ergonomics
of using enum types throughout the compiler.
It did however require introducing a new concept to the InternPool which
is an "incomplete" item - something that is added to gain a permanent
Index, but which is then mutated in place. This was necessary because
enum tag values and tag types may reference the namespaces created by
the enum itself, which required constructing the namespace, decl, and
calling analyzeDecl on the decl, which required the decl value, which
required the enum type, which required an InternPool index to be
assigned and for it to be meaningful.
The API for updating enums in place turned out to be quite slick and
efficient - the methods directly populate pre-allocated arrays and
return the information necessary to output the same compilation errors
as before.
This introduces a string table into InternPool as well as a curious new
field called `maps` which is an array list of array hash maps with
void/void key/value.
Some types such as enums, structs, and unions need to store mappings
from field names to field index, or value to field index. In such cases,
they will store the underlying field names and values directly, relying
on one of these maps, stored separately, to provide lookup.
This allows the InternPool to be serialized via simple array copies,
omitting all the maps, which are only used for optimizing lookup based
on field name or field value.
When the InternPool is deserialized it can be loaded via simple array
copies, and then as a post-processing step the field name maps can be
generated as extra metadata that is tacked on.
This commit provides two encodings for enums - one when the integer tag
type is explicitly provided and one when it is not. This is simpler than
the previous setup, which has three encodings.
Previous sizes:
* EnumSimple: 40 bytes + 16 bytes per field
* EnumNumbered: 80 bytes + 24 bytes per field
* EnumFull: 184 bytes + 24 bytes per field
Sizes after this commit:
* type_enum_explicit: 24 bytes + 8 bytes per field
* type_enum_auto: 16 bytes + 4 bytes per field
This is handy if you have a u32 and want a u32 and don't want to take a
detour through many layers of abstraction elsewhere in the std.hash
namespace.
Copied from https://nullprogram.com/blog/2018/07/31/
* make Sema.zirPtrType coerce the sentinel value against the element
type
* fix lazyAbiAlignment wrong result type
* typeHasOnePossibleValue no longer tries to create interned enum tag
value with integer zero, instead uses enum_field_index
* Type.ptr avoids trying to store typed null values into the intern
pool
On a simple input file, this had a total savings of 21% in the
InternPool:
Before:
int_positive: 427 occurrences, 8975 total bytes
After:
int_positive: 258 occurrences, 5426 total bytes
int_u8: 169 occurrences, 845 total bytes
This commit changes a lot of `*const Module` to `*Module` to make it
work, since accessing the integer tag type of an enum might need to
mutate the InternPool by adding a new integer type into it.
An alternate strategy would be to pre-heat the InternPool with the
integer tag type when creating an enum type, which would make it so that
intTagType could accept a const Module instead of a mutable one,
asserting that the InternPool already had the integer tag type.
Temporarily used for some unfortunate allocations made by backends that
need to construct pointer types that can't be represented by the
InternPool. Once all types are migrated to be stored in the InternPool,
this can be removed.
* Add some assertions to make sure instructions are not none. I tested
all these with master branch as well and made sure the behavior tests
still passed with the assertions intact (along with a handful of
callsite updates).
* Fix Sema.resolveMaybeUndefValAllowVariablesMaybeRuntime not noticing
that interned values are comptime-known. This was causing all kinds
of chaos.
* Fix print_air writeType calling tag() without checking for ip_index
The Key struct now has a Storage tagged union which can store a u64,
i64, or big int.
This is needed so that indexToKey can be implemented for integers stored
compactly in the data structure.
Instead of doing everything at once which is a hopelessly large task,
this introduces a piecemeal transition that can be done in small
increments at a time.
This is a minimal changeset that keeps the compiler compiling. It only
uses the InternPool for a small set of types.
Behavior tests are not passing.
Air.Inst.Ref and Zir.Inst.Ref are separated into different enums but
compile-time verified to have the same fields in the same order.
The large set of changes is mainly to deal with the fact that most Type
and Value methods now require a Module to be passed in, so that the
InternPool object can be accessed.
We need to set rbp last because the arguments are stored on the
stack. If we clobber rbp first, then we will get a segfault when
trying to access the function arguments.
I believe I had already done this with the other syscall* functions,
but not with syscall1, so this allows single argument syscalls like
close to work.
To avoid leaking slots, `fetchRemove` must increment `available` (when the "fetch" succeeds).
Without the `available += 1`, the added test `"std.hash_map repeat fetchRemove"` fails with
run test std-x86-linux-none-Debug: error: thread 432734 panic: integer overflow
.../zig/lib/std/hash_map.zig:1365:28: 0x6471d5 in getOrPutAssumeCapacityAdapted__anon_47495 (test)
self.available -= 1;
^
.../zig/lib/std/hash_map.zig:1308:62: 0x616950 in getOrPutAssumeCapacityContext (test)
const result = self.getOrPutAssumeCapacityAdapted(key, ctx);
^
Alternatively, `fetchRemove` could call `removeByIndex`, though that would entail calling `header()` twice instead of once.
Before this commit, if Walker.next errored with e.g. `error.AccessDenied` and the caller did something like `while (true) { walker.next() catch continue; }`, then the directory that errored with AccessDenied would be continually iterated in each `next` call and error every time with AccessDenied.
After this commit, the directory that errored will be popped off the stack before the error is returned, meaning that in the subsequent `next` call, it won't be retried and the Walker will continue with whatever directories remain on its stack.
For a real example, before this commit, walking `/proc/` on my system would infinitely loop due to repeated AccessDenied errors on the same directory. After this commit, I am able to walk `/proc/` on my system fully (skipping over any directories that are unable to be iterated).
Since we are opening each directory for iteration, we know that we don't need to reset the cursor's directory before iterating. Using `iterateAssumeFirstIteration` skips the cursor resetting which eliminates an `lseek` syscall for every directory opened on non-Windows platforms.
This doesn't seem to actually matter much for performance (1.01 ± 0.02 times faster when walking /home/ on my system) but avoiding unnecessary syscalls is always nice anyway.
Fixes https://github.com/ziglang/zig/issues/14465
For aarch64, LLVM was crashing because Zig commands it to generate FPU code
even when there is no FPU present. This commit implements the necessary checks
to avoid this undesired situation and aarch64 can be compiled again with
no FPU.
crypto.bcrypt: allow very large passwords to be pre-hashed
bcrypt has a slightly annoying limitation: passwords are limited
to 72 characters. In the original implementation, additional characters
are silently ignored.
When they care, applications adopt different strategies to work around
this, in incompatible ways.
Ideally, large passwords should be pre-hashed using a hash function that
hinders GPU attackers, and the hashed function should not be deterministic
in order to defeat shucking attacks.
This change improves the developer experience by adding a very explicit
`silently_truncate_password` option, that can be set to `false` in
order to do that automatically, and consistently across Zig applications.
By default, passwords are still truncated, so this is not a breaking
change.
Add some inline documentation for our beloved autodoc by the way.
Windows defines `recv` and `recvfrom` to return a value of type `int`,
rather than a pointer-sized signed integer, and so should use `c_int`
rather than `isize` for their return types.
Fixes bug causing ReleaseSmall to fail on Windows.
Due to the change in default behavior of ReleaseSmall, debug info are
stripped by default. However because `Compile.create` still defaults to
null, `producesPdbFile` will report true for
`lib/std/Build/Step/InstallArtifact.zig` causing it to fail on copying a
file that does not exist. This commit change the default of strip
depending on `optimize`.
Previously we did not write any missing padding bytes after the smallest
field (either tag or payload, depending on alignment). This resulted in
writing too few bytes and not matching the full abisize of the union.
When storing the address after calculating the element's address,
ensure it's stored in a local with the correct type. Previously it
would incorrectly use the element's type, which could be a float for
example and therefore generate invalid WebAssembly code.
This change also introduces a more robust `store` function.
When the paylaod is zero-sized we must ensure a valid pointer
is still returned for the ptr variation of the instruction. This,
because it's valid to have a pointer to a zero-sized value.
In such a case, we simply return the operand.
For packed unions where its abi size is less than or equal to 8 bytes
we store it directly and don't pass it by reference. This means that
when retrieving the field, we will perform shifts and bitcasts to ensure
the correct type is returned. For larger packed unions, we either allocate
a new stack value based on the field type when the field type is also passed
by reference, or load it directly into a local if it's not.
We currently have `isRef` return true for any type of union, including
packed unions. This means we can simply load it from the data section
to the exact type we want. In the future we can optimize it so it works
similarly to packed structs below 64 bits which do not get stored in
the data section and are not passed by ref.
Previously we would only store the payload, but not the actual tag
that was set. This meant miscompilations where it would incorrectly
return the tag value.
This also adds a tiny optimization for payloads which are not `byRef`
by directly storing them based on offset, rather than first calculating
a pointer to an offset.
Previously it was incorrectly assumed that all memcopy's generated by
the `memcpy` AIR instruction had an element size of 1 byte. However,
this would result in miscompilations for pointer's to arrays where
the element size of the array was larger than 1 byte. We now corectly
calculate this size.
Individual max buffer sizes are well known, now that arithmetic doesn't
require allocations any more.
Also bump `main_cert_pub_key_buf`, so that e.g. `nodejs.org` public
keys can fit.
`-l :path/to/lib.so` behavior on gcc/clang is:
- the path is recorded as-is: no paths, exact filename (`libX.so.Y`).
- no rpaths.
The previous version removed the `:` and pretended it's a positional
argument to the linker. That works in almost all cases, except in how
rules_go[1] does things (the Bazel wrapper for Go).
Test case in #15743, output:
gcc rpath:
0x0000000000000001 (NEEDED) Shared library: [libversioned.so.2]
0x000000000000001d (RUNPATH) Library runpath: [$ORIGIN/x]
gcc plain:
0x0000000000000001 (NEEDED) Shared library: [libversioned.so.2]
zig cc rpath:
0x0000000000000001 (NEEDED) Shared library: [libversioned.so.2]
0x000000000000001d (RUNPATH) Library runpath: [$ORIGIN/x]
zig cc plain:
0x0000000000000001 (NEEDED) Shared library: [libversioned.so.2]
Fixes#15743
[1]: https://github.com/bazelbuild/rules_go
These were failing on the prior commit. Unfortunately, the fix would
have been relatively complicated, and ties into underlying issues with
the current incremental compilation logic. After discussing this with
Andrew, we agreed that the best course of action is to completely
disable incremental compilation tests for now until it's more mature, at
which point we can re-enable them.
The idea here is that there are two ways we can reference a function at runtime:
* Through a direct call, i.e. where the function is comptime-known
* Through a function pointer
This means we can easily perform a form of rudimentary escape analysis
on functions. If we ever see a `decl_ref` or `ref` of a function, we
have a function pointer, which could "leak" into runtime code, so we
emit the function; but for a plain `decl_val`, there's no need to.
This change means that `comptime { _ = f; }` no longer forces a function
to be emitted, which was used for some things (mainly tests). These use
sites have been replaced with `_ = &f;`, which still triggers analysis
of the function body, since you're taking a pointer to the function.
Resolves: #6256Resolves: #15353
This data changed quite significantly between DWARF 4 and 5. Some
systems are shipping DWARF 5 libraries (Void Linux on musl libc seems to
use it for crt1 etc), which meant when printing stack traces, a random
compile unit might be incorrectly identified as containing an address,
resulting in incorrect location information.
I was consistently experiencing this issue with compiler stack traces,
and this change fixed it.
There are many different types of Windows paths, and there are a few different possible namespaces on top of that. Before this commit, NT namespaced paths were somewhat supported, and for Win32 paths (those without a namespace prefix), only relative and drive absolute paths were supported. After this commit, all of the following are supported:
- Device namespaced paths (`\\.\`)
- Verbatim paths (`\\?\`)
- NT-namespaced paths (`\??\`)
- Relative paths (`foo`)
- Drive-absolute paths (`C:\foo`)
- Drive-relative paths (`C:foo`)
- Rooted paths (`\foo`)
- UNC absolute paths (`\\server\share\foo`)
- Root local device paths (`\\.` or `\\?` exactly)
Plus:
- Any of the path types and namespace types can be mixed and matched together as appropriate.
- All of the `std.os.windows.*ToPrefixedFileW` functions will accept any path type, prefixed or not, and do the appropriate thing to convert them to an NT-prefixed path if necessary.
This is achieved by making the `std.os.windows.*ToPrefixedFileW` functions behave like `ntdll.RtlDosPathNameToNtPathName_U`, but with a few differences:
- Does not allocate on the heap (this is why we can't use `ntdll.RtlDosPathNameToNtPathName_U` directly, it does internal heap allocation).
- Relative paths are kept as relative unless they contain too many .. components, in which case they are treated as 'drive relative' and resolved against the CWD (this is how it behaved before this commit as well).
- Special case device names like COM1, NUL, etc are not handled specially (TODO)
- `.` and space are not stripped from the end of relative paths (potential TODO)
Most of the non-trivial conversion of non-relative paths is done via `ntdll.RtlGetFullPathName_U`, which AFAIK is used internally by `ntdll.RtlDosPathNameToNtPathName_U`.
Some relevant reading on Windows paths:
- https://googleprojectzero.blogspot.com/2016/02/the-definitive-guide-on-win32-to-nt.html
- https://chrisdenton.github.io/omnipath/Overview.htmlCloses#8205
Might close (untested) #12729
Note:
- This removes checking for illegal characters in `std.os.windows.sliceToPrefixedFileW`, since the previous solution (iterate the whole string and error if any illegal characters were found) was naive and won't work for all path types. This is further complicated by things like file streams (where `:` is used as a delimiter, e.g. `file.ext:stream_name:$DATA`) and things in the device namespace (where a path like `\\.\GLOBALROOT\??\UNC\localhost\C$\foo` is valid despite the `?`s in the path and is effectively equivalent to `C:\foo`). Truly validating paths is complicated and would need to be tailored to each path type. The illegal character checking being removed may open up users to more instances of hitting `OBJECT_NAME_INVALID => unreachable` when using `fs` APIs.
+ This is related to https://github.com/ziglang/zig/issues/15607
- `bytes` -> `buffer`
- Correct naming consistency between
1. fn argument and doc comment of `(read|write)PackedTwosComplement`
2. fn arguments of `(read|write)PackedTwosComplement` and `(read|write)TwosComplement`
* move `ptrBitWidth` from Arch to Target since it needs to know about the abi
* double isn't always 8 bits
* AVR uses 1-byte alignment for everything in GCC
Also get rid of the TTY wrapper struct, which was exlusively used as a
namespace - this is done by the tty.zig root struct now.
detectTTYConfig has been renamed to just detectConfig, which is enough
given the new namespace. Additionally, a doc comment had been added.
* std.crypto: faster ghash and polyval on WebAssembly
Before: 91 MiB/s
After : 243 MiB/s
Some other platforms might benefit from this, but WebAssembly is
the obvious one (simd128 doesn't make a difference).
Also remove all the wasi-libc files we used to ship, but never compile.
The latest wasi-libc HEAD has an extra commit (a6f871343313220b76009827ed0153586361c0d5), which makes preopen initialization lazy.
Unfortunately, that breaks quite a lot of things on our end. Applications now need to explicitly call __wasilibc_populate_preopens() everywhere when the libc is linked. That can wait after 0.11.
Support for 64-bit counters was a hack built upon the version with
a 32-bit counter, that emulated a larger counter by splitting the
input into large blocks.
This is fragile, particularily if the initial counter is set to
a non-default value and if we have parallelism.
Simply add a comptime parameter to check if we have a 32 bit or a
64 bit counter instead.
Also convert a couple while() loops to for(), and change @panic()
to @compileError().
These operations are constant-time on most, if not all currently
supported architectures. However, even if they are not, this is not
a big deal in the case on Poly1305, as the key is added at the end.
The final addition remains protected.
SalsaPoly and ChaChaPoly do encrypt-then-mac, so side channels would
not leak anything about the plaintext anyway.
* Apple Silicon (M1)
Before: 2048 MiB/s
After : 2823 MiB/s
* AMD Ryzen 7
Before: 3165 MiB/s
After : 4774 MiB/s
* std.crypto.chacha: support larger vectors on AVX2 and AVX512 targets
Ryzen 7 7700, ChaCha20/8 stream, long outputs:
Generic: 3268 MiB/s
AVX2 : 6023 MiB/s
AVX512 : 8086 MiB/s
Bump the rand.chacha buffer a tiny bit to take advantage of this.
More than 8 blocks doesn't seem to make any measurable difference.
ChaChaPoly also gets a small performance boost from this, albeit
Poly1305 remains the bottleneck.
Generic: 707 MiB/s
AVX2 : 981 MiB/s
AVX512 : 1202 MiB/s
aarch64 appears to generally benefit from 4-way vectorization.
Verified on Apple Silicon, but also on a Cortex A72.
A minimal set of simple, safe functions for Montgomery arithmetic,
designed for cryptographic primitives.
Also update the current RSA cert validation to use it, getting rid
of the FixedBuffer hack and the previous limitations.
Make the check of the RSA public key a little bit more strict by
the way.
This commit removes the `field_call_bind` and `field_call_bind_named` ZIR
instructions, replacing them with a `field_call` instruction which does the bind
and call in one.
`field_call_bind` is an unfortunate instruction. It's tied into one very
specific usage pattern - its result can only be used as a callee. This means
that it creates a value of a "pseudo-type" of sorts, `bound_fn` - this type used
to exist in Zig, but now we just hide it from the user and have AstGen ensure
it's only used in one way. This is quite silly - `Type` and `Value` should, as
much as possible, reflect real Zig types and values.
It makes sense to instead encode the `a.b()` syntax as its own ZIR instruction,
so that's what we do here. This commit introduces a new instruction,
`field_call`. It's like `call`, but rather than a callee ref, it contains a ref
to the object pointer (`&a` in `a.b()`) and the string field name (`b`). This
eliminates `bound_fn` from the language, and slightly decreases the size of
generated ZIR - stats below.
This commit does remove a few usages which used to be allowed:
- `@field(a, "b")()`
- `@call(.auto, a.b, .{})`
- `@call(.auto, @field(a, "b"), .{})`
These forms used to work just like `a.b()`, but are no longer allowed. I believe
this is the correct choice for a few reasons:
- `a.b()` is a purely *syntactic* form; for instance, `(a.b)()` is not valid.
This means it is *not* inconsistent to not allow it in these cases; the
special case here isn't "a field access as a callee", but rather this exact
syntactic form.
- The second argument to `@call` looks much more visually distinct from the
callee in standard call syntax. To me, this makes it seem strange for that
argument to not work like a normal expression in this context.
- A more practical argument: it's confusing! `@field` and `@call` are used in
very different contexts to standard function calls: the former normally hints
at some comptime machinery, and the latter that you want more precise control
over parts of a function call. In these contexts, you don't want implicit
arguments adding extra confusion: you want to be very explicit about what
you're doing.
Lastly, some stats. I mentioned before that this change slightly reduces the
size of ZIR - this is due to two instructions (`field_call_bind` then `call`)
being replaced with one (`field_call`). Here are some numbers:
+--------------+----------+----------+--------+
| File | Before | After | Change |
+--------------+----------+----------+--------+
| Sema.zig | 4.72M | 4.53M | -4% |
| AstGen.zig | 1.52M | 1.48M | -3% |
| hash_map.zig | 283.9K | 276.2K | -3% |
| math.zig | 312.6K | 305.3K | -2% |
+--------------+----------+----------+--------+
This is in preparation of removing indirect lowering again. Also
modifies constant() to accept a repr so that both direct as well
as indirect representations can be generated. Indirect is not yet
used, but will be used for globals.
Previously the tag type was generated even if it was nonexistant,
triggering an assertion that an integer type should never have
zero bits. Now its only generated when the tag type is actually emitted.
The first dereference of PtrAccessChain returns a pointer of the same type
as the base pointer, in contrast to AccessChain, where the first dereference
returns a pointer of the dereferenced type of the base pointer.
This instruction is not really working well in the LLVM SPIRV translator,
as it is not implemented.
This commit also intruces the constructStruct helper function to initialize
structs at runtime. This is ALSO buggy in the translator, and we must work
around OpCompositeConstruct not working when some of the constituents are
runtime-known only.
Some other improvements are made:
- improved variable() so that it is more useful and no longer requires the
address space. It always puts values in the Function address space,
and returns a pointer to the Generic address space
- adds a boolToInt utility function
This ensures that we can also cast enums and error sets here. In the future
this function will need to be changed to support composite and strange
integers, but that is fine.
It turns out that the Khronos LLVM SPIRV translator does not support OpPtrEqual.
Therefore, this instruction is emitted using a series of conversions.
This commit breaks intToEnum, because enum was removed from the arithmetic type
info. The enum should be converted to an int before this function is called.
When initializing a packed struct, we must ensure the result local
is zero'd. Previously we would do this by ensuring a new local is
allocated. Although a local is always zero by default, it meant that
if such an initialization was being done inside a loop, it would re-
use that very same local that could potentially still hold a different
value. Because this value is `or`'d with the value, it would result
in a miscompilation. By manually setting this result to 0, we guarantee
the correct behavior.
Previously we incorrectly assumed all memset's to have its element
abi-size be 1 byte. This would set the region of memory incorrectly.
We now have a more efficient loop, as well as support any element
type by re-using the `store` function for each element and moving
the pointer by 1 element.
Previously we would use the address of the slice itself, which would
result in miscompilations and accidently setting the memory region
of the slice itself, rather than based on the `ptr` field.
Previously when lowering a value of `elem_ptr` we would multiply the
abisize of the parent type by the index, rather than the element type.
This would result in an invalid pointer way beyond the correct pointer.
We now also pass the current offset to each recursive call to ensure
we do not miss inner offsets.
When we have a `ret_load` instruction with a zero-sized type which was
not an error, we would not emit any instruction. This resulted in
no `return` instruction and also not correctly resetting the global
stack_pointer.
This commit also enables the regular test runner for the WebAssembly
backend.
A copy was being made of a WValue variable, which meant the call
to `free` would insert the local that was being held by said WValue
was appended to the free list twice. This led to the same local being
reused even though it wasn't free and would lead to it being over-
written by a new value.
Rather than adding all values that were generated in the child branch,
we simply discard them as outer branches cannot refer to values
produced from an inner branch.
This new tag is used for freed locals that are not allowed to have any
remaining references pointing to it. This new tag allows us to easily
identify liveness bugs. Previously we would set the entire region to
`undefined` which would incorrectly set the tag to `function_index`,
making codegen think it was a valid `WValue` while it wasn't.
Make sure to increase the reference count for `intcast` when the
operand doesn't require any casting of the respective WebAssembly type.
Function arguments have a reserved slot, and therefore cannot be
re-used arbitrarily
This fix ensures that when we are shifting left or right,
both operands have the same WebAssembly type. e.g. it's not possible
to shift a 64 bit integer and 32 bit integer together and will fail
WebAssembly's validator. By first coercing the values to the same
type, we ensure we satisfy the validator.
This was a special type tag used for hacky stuff in Semantic Analysis.
Move the hacky stuff to use a dedicated `Air.Inst.Ref` instead.
This way, `var_args_param` is not involved in the type system or intern
pool.
Since #14819 this test failed with:
$ ../../../build/stage3/bin/zig test multi_writer.zig
multi_writer.zig:26:57: error: unable to evaluate comptime expression
var batch = std.event.Batch(Error!void, self.streams.len, .auto_async).init();
~~~~^~~~~~~~
referenced by:
Writer: multi_writer.zig:19:52
writer: multi_writer.zig:21:36
remaining reference traces hidden; use '-freference-trace' to see all reference traces
Thanks @jacobly for hints how to fix this on IRC.
* Avoid redundant words ("found")
- All compile errors are found by the compiler
* Avoid unnecessary prepositions ("with")
- There is a grammatically correct alternate word order without the
preposition.
`@trap` is a special function that we know never returns so it should
behave just like `@panic` and `@compileError` do currently and cause the
"unreachable code" + "control flow is diverted here" compile error.
Currently, `@trap(); @trap();` does not cause this error. Now it does.
* When there is buffered cleartext, return it without calling the
underlying read function. This prevents buffer overflow due to space
used up by cleartext.
* Avoid clearing the buffer when the buffered cleartext could not be
completely given to the result read buffer, and there is some
buffered ciphertext left.
* Instead of rounding up the amount of bytes to ask for to the nearest
TLS record size, round down, with a minimum of 1. This prevents the
code path from being taken which requires extra memory copies.
* Avoid calling `@memcpy` with overlapping arguments.
closes#15590
When producing zig2.c, don't waste time emitting C code for subcommands
that won't be used, such as objcopy.
This takes zig2.c down from 111M to 109M, and sidesteps some unfortunate
warnings that are currently emitted by GCC.
* build.zig: the result of b.option() can be assigned directly in many
cases thanks to the return type being an optional
* std.Build: make the build system aware of the
std.Build.Step.Compile.BuildId type when used as an option.
- remove extraneous newlines in error logs
* simplify caching logic
* simplify hexstring parsing tests and use a doc test
* simplify hashing logic. don't use an optional when the `none` tag
already provides this meaning.
* CLI: fix incorrect linker arg parsing
In one of the happy paths, execve() is used to switch to clang in which
case any cleanup logic that exists for this temporary file will not run
and this temp file will be leaked. Oh well. It's a minor punishment for
using `-x c` which nobody should be doing. Therefore, we make no effort
to clean up. Using `-` for stdin as a source file always leaks a temp
file.
Note that the standard `zig build-exe` CLI does not support stdin as an
input file. This is only for `zig cc` C compiler compatibility.
* no need to move `tmpFilePath` around
* no need for calculating max length of `FileExt` tag name
* provide a canonical file extension name for `FileExt` so that, e.g.
the file will be named `stdin.S` instead of
`stdin.assembly_with_cpp`.
* move temp file cleanup to a function to reduce defer bloat in a large
function.
* fix bug caused by mixing relative and absolute paths in the cleanup
logic.
* remove commented out test and dead code
echo 'some C program' | $CC -x c -
Is a common pattern to test for compiler or linker features. This patch
adds support for reading from non-regular files.
This will make at least one more Go test to pass.
Allows the package manager to download gitlab tarballs from urls such as
https://gitlab.com/<namespace>/<project>/-/archive/<sha>/<project>-<sha>.tar.gz
Such http requests have headers Content-Type=application/octet-stream
and Content-Disposition='attachment; filename="<project>-<sha>.tar.gz"'.
The package manager doesn't yet support these headers. This patch
doesn't attempt to properly parse the content-disposition header.
Instead it checks that it starts with 'attachment;' and ends with
'.tar.gz"'.
This allows users to choose which version they need for their particular use case, as the previous default (now the 'any' version) was (1) not always the desired type of delimiter and (2) performed worse than the scalar version if the delimiter was a single item.
This allows users to choose which version they need for their particular use case, as the previous default (now the 'full' version) was (1) not always the desired type of delimiter and (2) performed worse than the scalar version if the delimiter was a single item.
`branch_deaths` was a relic from before I had a full understanding of
AIR's control flow structure, and so was unnecessary. This change
simplifies Liveness, fixes a bug exposed by #15235, and likely improves
performance (due to cloning hashmaps less often).
The same declaration can be added to the dependency set multiple
times, and in this case we still need to emit it once. By making
this list a hash map instead, we can do that quite easily.
This commit also introduces some additional debug logging regarding
decls.
Implements AIR lowering for is_null and is_non_null tags.
Additionally this cleans up and centralizes the logic to convert from 'direct'
representation to 'indirect' representation and vice-versa. The related functions,
as well as the functions that use it, are all moved near eachother so that the
conversion logic remains in a central place. Extracting/inserting fields
and loading/storing pointers should go through these functions.
The result-id and result-type-id of the OpPhi used to merge the
break values was not properly emitted, as some of the operands
were not written out. This caused an invalid spir-v module.
Implements code generation for the try air tag. This commit also adds
a utility `errorUnionLayout` function that helps keeping the layout
of a spir-v error union consistent.
resolve() is now able to emit instructions. If usage of this function is
interleaved with calls to emitRaw() and writeOperand(), then an
instruction may get inserted between operands, causing an invalid module.
The solution here is to just perform a temporary allocation.
The pointer to a slot in a hash map was fetched before a recursive call.
If the hash map's size changed during the recursive call, this would write
to an invalid pointer.
The solution is to use an index instead of a pointer. Note that care must be
taken that resolved types (from the type_cahce) must not be accessed, as they
might be incomplete during this operation.
Define the size of the c types according the OpenCL specification.
Note that OpenCL does not define the size of long double. Clang generates
fp128, even though there is no extension that allows such types. The
llvm-spirv translator simply crashes.
With the old logic, it was possible for a bunch of processes to queue up
to update a cache entry, and then each to do so one at a time. Now, it
rechecks whether there still a cache miss or another process has
completed the work in the interim.
When checking a cache entry with no input files for a hit, if
`createFile` returned `error.WouldBlock` we would forget about the fact
that the file has been created, and all future checks will assume that a
cache hit has happened, even though one never has or does, leading to
rare `FileNotFound` errors trying the access the protected files.
This fix works by writing an extra byte to the manifest file to
distinguish hits and misses when there no input files to write.
This introduces a parallel set of functions to the std.mem.indexOfAny
functions. They simply invert the logic, yielding the first/last index
which is *not* contained in the "values" slice.
Inverting this logic is useful when you are attempting to determine the
initial span which contains only characters in a particular set.
* Document the `indexOfNone` family.
These descriptions are somewhat brief, but the functions themselves are
also simple enough to describe in such a way.
```
$ cat overflow.zig
test {
var a: [1]u65535 = undefined;
_ = a;
}
$ zig-out/bin/zig test overflow.zig
thread 290266 panic: integer overflow
zig/src/type.zig:3604:55: 0xada43d in intAbiAlignment (zig)
std.math.ceilPowerOfTwoPromote(u16, (bits + 7) / 8),
^
zig/src/type.zig:3598:42: 0xadd4ea in intAbiSize (zig)
const alignment = intAbiAlignment(bits, target);
^
zig/src/type.zig:3500:61: 0x92be91 in abiSizeAdvanced (zig)
return AbiSizeAdvanced{ .scalar = intAbiSize(bits, target) };
^
zig/src/type.zig:3385:62: 0x928933 in abiSizeAdvanced (zig)
switch (try payload.elem_type.abiSizeAdvanced(target, strat)) {
^
zig/src/type.zig:3268:32: 0x92c012 in abiSize (zig)
return (abiSizeAdvanced(ty, target, .eager) catch unreachable).scalar;
^
```
This is only noticed in a debug build of zig and silently does the wrong
thing and overflows in release builds.
This happened to `[x]u65529` and above because of the ` + 7` on a `u16`.
This can be used to escape the usual meaning of `_` to indicate a
non-exhaustive enum and create an enum tag that is a literal underscore,
so zig fmt should allow this syntax.
Before, zig fmt changes
const E = enum { @"_" };
to the semantically different
const E = enum { _ };
After, it remains the same.
Allocating an extended tag is much cleaner and easier to reason about
than reusing an existing tag. The previous `.data = undefined` was a
clear indication that we don't have any data to store, and so might as
well store an extended tag in that space almost for free.
`isAlwaysVoid` was being called with the undefined tag added by
`addOne`, causing non-deterministic behavior failures with release
builds of the compiler. Prevents the following random failure:
test/behavior/defer.zig:120:40: error: expected type 'error{One}', found 'void'
The question mark character can appear in identifiers as part of the
`@"syntax"` so we should allow typing it. Now, when the search is
selected, "?" is entered instead. It also shouldn't be that common in
general for the user to want to open the help modal.
Since the owner can either be a `Decl` or a `LazySymbol` we need
to preserve this information at the codegen generate function level
so that we can then correctly work out the corresponding `Atom`
in the linker.
Pass extra pointer param with a linker ref when calling the lazy tagName
function to workaround not being able to lower linker refs during
codegen of a lazy func.
FILE_DISPOSITION_ON_CLOSE is used to set/clear the FILE_DELETE_ON_CLOSE,
but we do not use that anymore and FILE_DISPOSITION_POSIX_SEMANTICS
already implies unmapping of the handle and removal fo it on close.
Justification: When a file is deleted on Windows, it may not be
immediately removed from the directory. This can cause problems
with future scans of that directory, which will see the partially
deleted file. Under some workloads and system configurations,
Windows files may appear to be deleted immediately.
This is the PR with requested fixup. Thanks to @SpexGuy for the
original PR.
The majority of these are in comments, some in doc comments which might
affect the generated documentation, and a few in parameter names -
nothing that should be breaking, however.
This commit installs Zig to "build_dir/stage3" during building
so that distros' can easily find binary and use it for testing/etc.
This commit also splits "add_custom_target(stage3 ALL" and command that it invokes,
so that it won't retry it during installation,
as target will be considered not out-of-date.
See also https://www.github.com/ziglang/zig/issues/14240#issuecomment-1374642063
Fixes a regression introduced in
9295355985 that caused 32-bit builds with
`-Denable-llvm` to fail to build from source due to that common
u64/usize casting issue.
Looks like I might have messed up the wasm kernel in my recent branch causing some sporadic failures on the CI.
This file was built the following way:
1. check out d0311e28b3
2. do the cmake bootstrap
3. check out 57ea6207d3
4. `zig build update-zig1` and stash those modified files
5. check out 440b3df702 (master)
6. do the cmake bootstrap
7. `zig build update-zig1` to produce this commit
Fixes#15489
This also lays the groundwork for exposing the whether or not a function is
noinline in std.builtin.Fn as an `is_noinline: bool` field if we ever want to do that.
This is needed because bug fixes to the C backend are required in order
to actually update the standard library and compiler sources to use the
new `@memcpy` and `@memset` semantics.
This is a small change to help when reading failure logs which makes the
"exited with code 1" and similar message include the test name.
Further enhancements could do the following:
* even if one unit test crashes the process, the parent process
continues running the other unit tests
* ability to test for expected panics (#1356)
* timeouts on individual tests
The previous commit introduced an optimization to the LLVM backend that
makes `@memset` lower more optimally when the element is comptime-known
and has a repeating byte pattern.
By making these functions inline, if the element parameter is
comptime-known at the callsite, it will be comptime-known in the
`@memset` call, causing more use of the LLVM `memset` intrinsic rather
than an inline for loop when using the LLVM backend.
This affects, for example, std.crypto.argon2, which calls
appendNTimesAssumeCapacity with a `[128]u64` as the element. This is now
lowered with a single `memset` call.
When the element is comptime-known, we can check if it has a repeated
byte representation. In this case, `@memset` can be lowered with the
LLVM intrinsic rather than with a loop.
On Windows, a directory that's set as the current working directory is
not allowed to be removed. This can cause error on `deleteTree` if the
CWD is set to the file to be removed and will cause `error.FileBusy`.
However, due to `tmp.cleanup()` ignoring the errors, the folder removal error will
be ignored. The only test violating this is `windows_spawn`. As a
solution, setting the parent directory to be the CWD before deletion
will allow the cleanup to pass.
In semaDecl, it was possible for a new ArenaAllocators state to replace an existing one that
hadn't been freed yet. Instead of the ref_count (which was made redundant by adding
the allocator parameter to `release`), I now store a pointer to the previous arena, if one exists.
This allows a recursive deinit to happen when the last arena created is destroyed.
This fixes a bug where resolveStructLayout to was promoting from stale
value_arena state which was then overwrriten when another ArenaAllocator
higher in the call stack saved its state back. This resulted in the memory
for struct_obj.optmized_order overlapping existing allocations.
My initial fix in c7067ef wasn't sufficient, as it only checked if the struct being
resolved had the same owner as the current sema instance. However, it's
possible for resolveStructLayout to be called when the sema instance
has a different owner, but the struct decl's value_arena is currently in
use higher up in the callstack.
This change introduces ValueArena, which holds the arena state as well as tracks
if an arena has already been promoted from it. This allows callers to use the
value_arena storage without needing to be aware of another user of this same storage
higher up in the call stack.
This allows abstract references to the stack frame which enables:
* deferred frame layout until after the function has been processed
* frame compaction (sorting by alignment)
* being able to overalign the stack
* references that change based on an overaligned stack
* moving callee saved spills to the usual part (smaller code size)
* shared call frame (avoids stack adjustments before and after calls)
New search system is based on a Radix Tree. The Radix Tree contains a shallow list of all decl names (ie no paths), plus some suffixes, split by following the official style guide (eg "HashMapUnmanaged" also produces "MapUnmanaged" and "Unmanaged", same with snake_case and camelCase names).
Additionally, the search system uses the decl graph data to recognize hierarchical relationships between decls, allowing you to zero on a target namespace for search. As an example "fs create" will score highe all things related to the creation of files and directories inside of `std.fs`, while still showing (but with lower score) matches from `std.Bulild`.
As another example "fs windows" will prioritize windows-related results in `std.fs`, while "windows fs" will prioritize fs-related results in `std.windows`.
If the C code had variables that were named the same as the prefixes used
for name mangling, such as "tmp" or "ref", then the codegen would generate
incorrect code in some cases. This was because these aliases were immediately
visible to expressions that actually needed to use the original name.
I introduced the concept of reserving aliases without enabling them. An alias
that isn't enabled isn't visible to expression translation, but is still
reserved so that sub-expressions generate aliases that don't overlap.
Add test cases to cover the cases that would break before this change.
Co-authored-by: Veikka Tuominen <git@vexu.eu>
Uses the `atomic.fence` instruction for multi-thread-enabled builds
where the `atomics` feature is enabled for the wasm32 target.
In all other cases, this lowers to a nop.
Implements the lowering of the `@atomicRmw` builtin. Uses the atomic
opcodes when the cpu feature `atomics` is enabled. Otherwise lowers
it to regular instructions. For the operations that do not lower to
a direct atomic opcode, we use a loop in combiantion with a cmpxchg
to ensure the swapping of values is doing atomically.
When the user passes the cpu feature `atomics` to the target triple,
the backend will lower the AIR instruction using opcodes from the
atomics feature instead of manually lowering it.
Often, a `dbg_stmt` ends up being associated with no real code because
whatever it referred to was eliminated by semantic analysis. In these
cases, Sema can replace the last `dbg_stmt` with the new one to avoid
redundant AIR instructions which at best are nops and at worst cause
backends to emit useless info (e.g. CBE does this).
to avoid getting OOM killed. We need to have a high number of GitHub
Actions runners simultaneously active, otherwise the hardware ends up
underutilized. But with zig build running in parallel now, this
sometimes results in memory usage spikes that have been causing the
runner to get killed.
This number is the total physical memory (251G) divided by the number of
runners we have active (11).
See related commit 2d41dac57d which did
the same but for our x86_64-linux machines.
Pointer comparisons were triggering `-Wcompare-distinct-pointer-types`
before this fix, which adds `(void*)` casts if the lhs type and rhs type
do not match pointer sizeness.
store:
The value to store may be undefined, in which case the destination
memory region has undefined bytes after this instruction is
evaluated. In such case ignoring this instruction is legal
lowering.
store_safe:
Same as `store`, except if the value to store is undefined, the
memory region should be filled with 0xaa bytes, and any other
safety metadata such as Valgrind integrations should be notified of
this memory region being undefined.
Needed due to the compiler depending on standard library APIs such as
ArrayList that contain `@memset` and `@memcpy` calls in them. The number
of parameters changed, so this is necessary for the compiler to build.
Previously, this code casted the array pointer to u8 pointer, but I
removed that in a different commit. This commit restores the cast, but
instead of hard-coding u8, it uses the destination element pointer,
since memset now supports arbitrary element types.
* Sema: upgrade operands to array pointers if possible when emitting
AIR.
* Implement safety checks for length mismatch and aliasing.
* AIR: make ptrtoint support slice operands. Implement in LLVM backend.
* C backend: implement new `@memset` semantics. `@memcpy` is not done
yet.
Now they use slices or array pointers with any element type instead of
requiring byte pointers.
This is a breaking enhancement to the language.
The safety check for overlapping pointers will be implemented in a
future commit.
closes#14040
addition to #15450
createFileW does not account for failure on `LockFile`. This can result
in a file handle not being closed on failure which can be seen on test
such as `fs.test.'open file with exclusive nonblocking lock twice'`.
This reverts commit c75e11bf6a.
Caused this problem on some machines:
"this step declares an upper bound of 9126805504 bytes of memory,
exceeding the available 7515721728 bytes of memory".
Instead the next commit will disable std lib tests with the C backend on
Windows.
I observed clang taking 8G to compile the output from the std lib tests
using the C backend. This commit should make the Windows CI stop failing
due to OOM.
Before this change, if a request errored before getting its `response.headers` initialized, then it would attempt to `deinit` `response.headers` which would still be `undefined`. Since all locations that set `response.headers` use the same code, it can just be done upfront in `request` instead.
Closes#15380
* docs(std.math): elaborate on difference between absCast and absInt
* docs(std.rand.Random.weightedIndex): elaborate on likelihood
I think this makes it easier to understand.
* langref: add small reminder
* docs(std.fs.path.extension): brevity
* docs(std.bit_set.StaticBitSet): mention the specific types
* std.debug.TTY: explain what purpose this struct serves
This should also make it clearer that this struct is not supposed to provide unrelated terminal manipulation functionality such as setting the cursor position or something because terminals are complicated and we should keep this struct simple and focused on debugging.
* langref(package listing): brevity
* langref: explain what exactly `threadlocal` causes to happen
* std.array_list: link between swapRemove and orderedRemove
Maybe this can serve as a TLDR and make it easier to decide.
* PrefetchOptions.locality: clarify docs that this is a range
This confused me previously and I thought I can only use either 0 or 3.
* fix typos and more
* std.builtin.CallingConvention: document some CCs
* langref: explain possibly cryptic names
I think it helps knowing what exactly these acronyms (@clz and @ctz) and
abbreviations (@popCount) mean.
* variadic function error: add missing preposition
* std.fmt.format docs: nicely hyphenate
* help menu: say what to optimize for
I think this is slightly more specific than just calling it
"optimizations". These are speed optimizations. I used the word
"performance" here.
- Add an example of iterating over consecutive integers using the
range syntax
- Add an example of iterating over multiple objects
- Update the "nested break" and "nested continue" tests to use the
range syntax, instead of a temporary array
Rather than using a function call to verify if an error fits within
the global error set's length, we now store the error set' size in
the .rodata segment of the linear memory and load that value onto
the stack to check with the integer value.
This implements the safety check for error casts. The instruction
generates a jump table with 2 possibilities. The operand is used
as an index into the jump table. For cases where the value does
not exist within the error set, it will generate a jump to the
'false' block. For cases where it does exist, it will generate
a jump to the 'true' block. By calculating the highest and lowest
value we can keep the jump table smaller, as it doesn't need to
contain an index into the entire error set.
Creates a global undefined symbol when this instruction is called.
The linker will then resolve it as a lazy symbol, ensuring it is
only generated when the symbol was created. In `flush` it will then
generate the function as only then, all errors are known and we can
generate the function body. This logic allows us to re-use the same
functionality of linker-synthetic-functions.
Tests that only reference decls for the purpose of analyzing more tests
should be unnamed, otherwise trying to filter for just a referenced test
can become impossible depending on the names.
* Disable 128-bit atomics for x86_64 generic (currently also baseline)
because they require heavy abi agreement to correctly lower.
** This is a breaking change **
* Enable 128-bit atomics for aarch64 in Sema since it just works.
Fixes the abi of `f128` on darwin and/or aarch64. Also, when `double`
and `long double` were the same size, we used to incorrectly define
`zig_c_longdouble` as `zig_f128`.
Since the Zig language documentation claims support for `.Min` and
`.Max` in `@atomicRmw` with floats, allow in Sema and implement for both
the llvm and C backends.
When a big.Int.Mutable had more than two limbs, it was possible for
this function to change the `len` field without zeroing limbs in the
active range. These uninitialized limbs would then be used in
`truncate()` and could cause invalid results.
Closes#13571
* CompileStep: Avoid calling producesPdbFile() to determine whether the
option should be respected. If the user asks for it, put it on the
command line and let the Zig CLI deal with it appropriately.
* Make the namespace of `std.dwarf.Format.dwarf32` no longer have a
redundant "dwarf" in it.
* Add `zig cc` integration for `-gdwarf32` and `-gdwarf64`.
* Toss in a bonus bug fix for `-gdwarf-2`, `-gdwarf-3`, etc.
* Avoid using default init values for struct fields unnecessarily.
* Add missing cache hash addition for the new option.
This commit enables producing 64-bit DWARF format for Zig executables
that are produced through the LLVM backend. This is achieved by exposing
both command-line flags and CompileStep flags. The production of the
64-bit format only affects binaries that use the DWARF format and it is
disabled on MacOS due to it being problematic. This commit, despite
generating the interface for the Zig user to be able to tell the compile
which format is wanted, is just implemented for the LLVM backend, so
clang and the self-hosted backends will need this to be implemented in a
future commit.
This is an effort to work around #7962, since the emission of the 64-bit
format automatically produces 64-bit relocations. Further investigation
will be needed to make DWARF 32-bit format to emit bigger relocations
when needed and not make the linker angry.
The self-hosted aarch64 backend is not currently functional due to the
Liveness changes. A previous commit disabled aarch64 on the behavior
tests; this commit disables it and arm for the test cases. Moreover, all
incremental test cases have been unified into shared cross-platform
cases, which can be gradually enabled as the backends improve.
Uses the new liveness behaviour. This also removes useless calls
to `processDeath` on branches that were just initialized. Branch
consolidation and processing deaths on branches inside `condbr`
is still a TODO, just like before.
This also skips var_args on other native backends as they do not
support this feature yet.
Backends want to avoid emitting unused instructions which do not have
side effects: to that end, they all have `Liveness.isUnused` checks for
many instructions. However, checking this in the backends avoids a lot
of potential optimizations. For instance, if a nested field is loaded,
then the first field access would still be emitted, since its result is
used by the next access (which is then unreferenced).
To elide more instructions, Liveness can track this data instead. For
operands which do not have to be lowered (i.e. are not side effecting
and are not something special like `arg), Liveness can ignore their
operand usages, and push the unused information further up, potentially
marking many more instructions as unreferenced.
In doing this, I also uncovered a bug in the LLVM backend relating to
discarding the result of `@cVaArg`, which this change fixes. A behaviour
test has been added to cover it.
This is a partial rewrite of Liveness, so has some other notable changes:
- A proper multi-pass system to prevent code duplication
- Better logging
- Minor bugfixes
Also remove an incorrect piece of logic which allowed fetching the 'len'
property on non-single-ptrs (e.g. many-ptrs) and add a corresponding
compile error test case.
Resolves: #4765
The issue with just passing `-gcodeview` is that it's not part of
the `OPT_g_Group` in `clang/Driver/Options.td`, so it doesn't trigger
debug info to be generated.
Passing only `-g` is sufficient on MSVC, as there is logic in `Clang.cpp`
which enables codeview if that is the default for the toolchain. Since
the default for -gnu is not codeview, we do pass `-gcodeview` in that case.
ccf670c made using `return` from within a comptime block in a non-inline
function illegal, since it is a use of runtime control flow in a
comptime block. It is allowed if the function in question is `inline`,
since no actual control flow occurs in this case. A few functions from
std (notably `std.fmt.comptimePrint`) needed to be marked `inline` to
support this change.
This pull request removes the optional allocator argument from functions
`divFloor` and `divTrunc`. As a result, the comments related to accepting an
optional `allocator` are no longer applicable. The support for accepting
an optional allocator was removed in #10017.
- Hold the lock for a shorter amount of time
- Previously, when holding the lock while signaling, the other, resumed
thread could potentially get suspended again immediately because
the mutex was still locked.
- Fix comment
- walk the stack via the method that is aware of unwind info (fixes x86_64 / aarch64 traces)
- enhance the output for frames where the debug info isn't available by printing the module name
DELETE_PENDING can happen when the file is yet to be closed for deletion
or if it never get closed. In that case, DeleteFile should assume the
file deletion is succeeding (no CloseHandle is required as it's a "failure"). In case of `DELETE_PENDING` failure, the file
may still exist. In which case if it's part of `deleteTree`, it will
eventually fail on `error.DirNotEmpty`.
This fixes `.INVAL => unreachable` being triggered by the cache system
on macOS when multiple processes race to create the same compilation.
The problem is that when two processes race to create a file, it
sometimes returns ENOENT even though that error code is nonsensical for
this situation.
Commit 2b0929929d purportedly solved this,
but it did not open the file with write permissions, leading to the
EINVAL panic later on. This commit remedies the situation by introducing
a loop and simply retrying when the ENOENT occurs.
Deleting a read-only file should result in `AccessDenied` (`CANNOT_DELETE`).
Note: This test was observed to fail when the file is closed then reopened
before the change in permission due to the absence of
`FILE_WRITE_ATTRIBUTES` when re-opened. (see #15316).
to avoid getting OOM killed. It's been happening regularly recently. We
need to have a high number of GitHub Actions runners simultaneously
active, otherwise the hardware ends up underutilized. But with zig build
running in parallel now, this sometimes results in memory usage spikes
that have been causing the runner to get killed.
This number is the total physical memory (126G) divided by the number of
runners we have active (6).
This reverts commit a34752c941.
With #15277 merged, the macOS CI will still run the self-hosted x86_64
tests even with -Dskip-non-native. This was the main motivation for the
commit.
It would be nice to keep this flag enabled, however, we don't have
enough CI computing power, and the macOS CI lags behind with these extra
tests enabled.
Following @Luukdegram's comment[1]:
> The default should remain Debug when unspecified. In ReleaseSmall the
> user would lose all DWARF support when testing their WebAssembly code.
> Building a release version should be opt-in, not the default.
[1]: https://github.com/ziglang/zig/pull/15192#discussion_r1160824726
I find myself quite often creating ReleaseSafe builds and putting them
to production for certain experiments:
- Debug info are for stack traces. An ongoing example where those would
help is #14815.
- Safety checks would have saved a couple of mine and @kubkon's hours in
#15098.
This is a breaking change for scripts that make Zig releases -- I will
submit another PR to zig-bootstrap and release-cutter after this is
merged.
Using `FILE_DELETE_ON_CLOSE` can silently succeed without reporting any error
on non-empty directory. This commit adds usage of NtSetInformationFile
which will report `DIRECTORY_NOT_EMPTY`.
The CI now runs C backend tests in addition to compiling them. It uses
-std=c99 -pedantic -Werror in order to catch non-conformant C code.
This necessitated disabling a test case that caused a C compile error,
in addition to disabling a handful of warnings that are already being
triggered by Zig's C backend output for the behavior tests.
The upshot is that I was able to, very cleanly, integrate the C backend
tests into the build system, so that it communicates via the test runner
protocol along with all the other behavior tests.
Make the test targets use options that match the actual options of
CompileStep. This makes the code more straightforward, and ends up
making fewer tests incorrectly skipped. For example, now the CI runner
on Windows will no longer skip self-hosted x86_64 backend tests.
reverts regression introduced in
d2ad3f5074.
The commit correctly removed dest_builder from InstallArtifactStep, but
the change to installLibraryHeaders was incorrect since it affected
different steps than that one.
`GetPhysicallyInstalledSystemMemory` uses SMBios to grab the physical
memory size which can lead to unecessary allocation and inacurate
representation of the total memory. Using `System_Basic_Information`
help to retrieve the physical memory which is not reserved for the
kernel/tables. This aligns better with the linux side as `/proc/meminfo`
does the same thing.
The Wasm backend now supports all features required to use
the full `start.zig` logic, rather than the simplified version.
With this commit we remove all references to the simplified logic
for Wasm specifically.
When we lower the instruction for `@tagName` we generate a new
function if it doesn't exist yet for that decl. This function creates
an if-else chain to determine which value was provided. Each tag
generates a constant that contains its name as the value. For each
tag we generate a case where the pointer of the string is stored in
the result slice. The length of the tagname is comptime-known, therefore
will be stored in the slice directly without having it being part of
the tagname symbol. In the future this can use a jump table instead
of an if-else chain, similar to the `switch` instruction.
OpenBSD 7.3 changed its implementation of
pthread_get_name_np/pthread_set_name_np to wrap new libc functions
getthrname/setthrname and lowered the max buffer size from 32 to 24.
This is not a backwards-compatible change because if we were to put in
comptime version logic to use size 32 when target < 7.3 the binaries
would be undefined when running on >= 7.3. It also could simply be that
OpenBSD has a policy to not support older binaries running on newer
releases? Regardless, the safest course is to simply use the smallest
known buffer size.
As an aside, this bug manifested as a "hung" std.Thread test because 7.3
pthread API never checks for error result when wrapping getthrname/setthrname.
This is not a problem in std.Thread when we use the correct max buffer
size because ERANGE/EINVAL become unreachable.
* autodoc: init support for usingnamespace decls
* autodoc: don't build autodoc when building zig2.c
* autodoc: usingnamespace decls support in frontend (#15203)
* autodoc: init support for usingnamespace decls
* autodoc: usingnamespace decls support in frontend
---------
Co-authored-by: Krzysztof Wolicki <46651553+der-teufel-programming@users.noreply.github.com>
* Add `omit_pkg_fetching_code` option to test_cases
* Change `omit_pkg_fetching_code` to always `true` for test_cases
Co-authored-by: Andrew Kelley <andrew@ziglang.org>
---------
Co-authored-by: Andrew Kelley <andrew@ziglang.org>
* remove setName, setFilter, and setTestRunner. Please set these
options directly when creating the CompileStep.
* removed unused field
* remove computeOutFileNames and inline the logic, making clear the
goal of avoiding state mutations after the build step is created.
These functions are problematic in light of dependencies because they
run and install, respectively, for the *owner* package rather than for
the *user* package. By removing these functions, the build script is
forced to provide the *Build object to associate the new step with,
making everything less surprising.
Unfortunately, this is a widely breaking change.
see #15079
This was used to ensure that an artifact would only be installed once,
but this is not only unnecessary, but actively harmful, in the face of
dependencies.
see #15079
Support for the built-in libcrypt was removed in commit 6b7ddfba,
but the -lcrypt flag remained ignored, preventing linking against
external libcrypt.
Fixes#5990.
Signed-off-by: Piotr Sikora <piotr@aviatrix.com>
In order to be able to report nice errors in the test runner, and in order
to check SkipZigTest errors, we need to pass the error names to the consumer.
This kind of information can be passed via nonsemantic instructions - using
OpSourceExtension here.
All errors are concatenated into a single string, starting with 'zig_errors:'
for identification, separated by a colon (:). To ensure that we can represent
all error codes, even those which contain a colon, the error names are URI-
escaped. URI-escaping, rather than base64, allows us to see the error names
when viewing disassembled SPIR-V code.
It seems that some implementations may have problems with these right now,
like Intel and Rusticl. In theory, these attributes should be superficial
on the pointer type, as alignment guarantees are also added via the
alignment option of the OpLoad and OpStore instructions. Therefore, get rid
of them for now.
For SPIR-V, only export the main function if it is actually declared. Kernel
entry points will often have parameters and more than one kernel declared.
In general, SPIR-V binaries should mostly be compiled as libraries and not as
executables. However, this start code is required so that we can build test
executables.
Note that a call to isSpirV() would emit the code for that function, even though
the call is at comptime. To save that function from being emitted the checks
are just inlined manually.
AmdgpuKernel and NvptxKernel are unified into a Kernel calling convention.
There is really no reason for these to be separate; no backend is allowed to
emit the calling convention of the other. This is in the same spirit as the
.Interrupt calling convention lowering to different LLVM calling conventions,
and opens the way for SPIR-V kernels to be exported using the Kernel calling
convention.
Entry points need to be attributed with a complete list of
global variables that they use. To that end, the global dependencies
mechanism is extended to also allow functions - when flushing the
module, the list of dependencies is examined to generate this
list of global variable result-ids.
SPIR-V cannot represent function pointers without extensions
that no vendor implements. For the time being, generate a test
kernel for each error, so that we can at least run SOME tests.
In the future we may be able to emulate function pointers in some
way, but that is not today.
SPIR-V globals must be emitted in order, so that any
declaration precedes usage. Zig, however, generates globals in
random order. To this end we keep for each global a list of
dependencies and perform a topological sort when flushing the
module.
Lowering constants is currently not really compatible with unions. In
this commit, constant lowering is drastically overhauled: instead of
playing nice and generating SPIR-V constant representations for everything
directly, we're just going to treat globals as an untyped bag of bytes (
or rather, SPIR-V 32-bit words), which we cast to the desired type at
usage. This is similar to how Rust generates constants in its LLVm backend.
Similar to function locals, taking the address of a global that does
not have an explicit address space assigned to it should result
in a generic pointer, not a global pointer. Also similar to function
locals, they cannot be generated into the generic storage class, and
so are generated into the global storage class and then cast to a
generic pointer, using OpSpecConstantOp. Note that using
OpSpecConstantOp results is only allowed by a hand full of other
OpSpecConstant instructions - which is why we generate constant
structs using OpSpecConstantComposite: These may use OpVariable
and OpSpecConstantOp results, while OpConstantComposite may not.
This cloneAir/cloneLiveness idea used to ignore Zig's internals
has proven buggy. Instead, just generate the code directly from
updateFunc and updateDecl as the other backends do, but pretend
that Zig is not an incremental compiler. The SPIR-V backend will
for the time being not support this.
Bools have a different immediate representation and memory
representation - which means that they must be converted every time
a bool is loaded from or stored into memory.
Taking the address of a local variable should result in a generic
pointer - too much code breaks if we do not do this. We cannot
lower locals into the generic storage class directly though, so
instead, lower the variables into the Function storage class
implicitly, and convert the pointer to a generic pointer.
Also Correct OpInboundsAccessChain generation (we only need
the one index).
There are two main ways in which a value can be stored: "Direct", as it
will be operated on as an immediate value, and "indirect", as it is stored
in memory. Some types need a different representation here: Bools, for
example, are opaque in SPIR-V, and so these need to have a different
representation in memory. The bool operations are not easily interchangable
with integer operations, though, so they need to be OpTypeBool as
immediate value.
Previously they were strong aliases, but as these types are used quite
intermittendly it resulted in a lot of toRef() calls. Removing them
improves readability a bit.
- Adds the Int8. Int16, Int64 and GenericPointer capabilities.
TODO: This should integrate with the feature system.
- Default some struct fields of SPIR-V types so that we dont
need to type them all the time.
- Store struct field name's in SPIR-V types, and generate the
OpMemberName decoration if they are non-null.
- Also add the field names to the actual SPIR-V types.
- Generate OpName for functions.
Implements the div-family and intcast AIR instructions, and starts
implementing a mechanism for masking the value of 'strange' integers
before they are used in an operation that does not hold under modulo.
Implements lowering for the add_with_overflow AIR instructions. Also implements
a helper function, simpleStructType, to quickly generate a SPIR-V structure type
without having to do the whole allocation dance.
When a result of a pure instruction is not used, it also does not need to
be generated. The other backends already implement these checks, they were
ignored in SPIR-V up until now. New instructions added in the future should
have these be implemented from the start.
Implements type lowering for arrays and structs, and implements instruction
lowering for bitcast and call. Bitcast currently naively maps to the OpBitcast
instruction - this is only valid for some primitive types, and should be
improved to work with composites.
OpName instructions assign a debug name to a type. Some basic
types - bool, void, ints, and floats are given a debug name this way.
TODO is to extend this to the other types.
In practice there are only a few variations of these types allowed, so it
kind-of makes sense to write them all out. Because the types are hashed this
does not actually save all that many bytes in the long run, though. Perhaps
some of these types should be pre-registered?
This allows the Zig calling convention and makes way for a Kernel
calling convention in the future. Any future checks on calling
conventions should be placed in Sema.zig.
This adds a general target for SPIR-V compilation. Previously there was not
any target machine defined for SPIR-V.
TODO is to reword the features for this target. We don't really need the full
list of capabilities in the features, we should only put a few features here
which we can actually use during code generation.
This makes -r treated the same as -c which is to output an object file.
Zig's ELF linker code already handles multiple object files into an
object file with the -r flag to LLD.
closes#11683
fix for 32bit arches
curate error sets for api facing functions, expose raw errors in client.last_error
fix bugged dependency loop, disable protocol tests (needs mocking)
add separate mutex for bundle rescan
* extract http protocol into protocol.zig, as it is shared between client and server
* coalesce Request and Response back into Client.zig, they don't contain
any large chunks of code anymore
* http.Server is implemented as basic as possible, a simple example below:
```zig
fn handler(res: *Server.Response) !void {
while (true) {
defer res.reset();
try res.waitForCompleteHead();
res.headers.transfer_encoding = .{ .content_length = 14 };
res.headers.connection = res.request.headers.connection;
try res.sendResponseHead();
_ = try res.write("Hello, World!\n");
if (res.connection.closing) break;
}
}
pub fn main() !void {
var server = Server.init(std.heap.page_allocator, .{ .reuse_address = true });
defer server.deinit();
try server.listen(try net.Address.parseIp("127.0.0.1", 8080));
while (true) {
const res = try server.accept(.{ .dynamic = 8192 });
const thread = try std.Thread.spawn(.{}, handler, .{res});
thread.detach();
}
}
```
* use a set instead of a list
* use of this flag currently requires LLD
* add documentation
* make it only a zig cc compatibility flag for now because I personally
think this is an anti-feature.
* GPA: Catch invalid frees
Fix#14791: Catch cases where an invalid slice is passed to free().
This was silently ignored before but now logs an error. This change
uses a AutoHashMap to keep track of the sizes which seems to be an
overkill but seems like the easiest way to catch these errors.
* GPA: Add wrong alignment checks to free/resize
Implement @Inkryption's suggestion to catch free/resize with the wrong
alignment. I also changed the naming to match large allocations.
Since one of the program header entries is now the program header table
itself, we can avoid tracking it explicitly and just track it as yet
another program segment.
This lets us generate the store with knowledge of the type to be stored.
Therefore, we can avoid generating garbage Air with stores through
pointers to comptime-only types which backends cannot lower.
Closes#13410Closes#15122
Latest macOS 13.3 rolled out LLVM 15 and thus the way `nullptr_t` is
defined within the `libc++`:
157bbe6aea
This seems to require including `/usr/include` with `-isystem` directive
rather than `-I`. Otherwise we get clang miscompilation issues due to
missing `nullptr_t` declaration.
Also clean up parsing of linker args - reuse `ArgsIterator`.
In MachO, ensure we add every symbol marked with `-u` as undefined
before proceeding with symbol resolution. Additionally, ensure those
symbols are never garbage collected.
MachO entry_in_dylib test: pass `-u _my_main` when linking executable
so that it is not incorrectly garbage collected by the linker.
- Fix usage string `--skip-code-test` to `--skip-code-tests`.
- Added a token `{#additonal_option|-rdynamic#}` which introduce arbitrary flag to `build-exe/obj/lib` example.
- Fix wasm freestanding example, it now needs explicit export symbols to work.
In addition, we need to be careful not to mark the relocations as
resolved prematurely as then we are risking malforming the binary
as we need to resolve the relocs twice: once for in-memory writes,
and once for in-file updates.
Ideally, we would just do an atomic rename, but so far I had no
luck. I have also tried marking the file to delete-on-close but
then we cannot use it to spawn the process. So for now, let's just
put a copy in `zig-cache` and let the user decide when to recycle
the cache dir.
* @workItemId returns the index of the work item in a work group for a
dimension.
* @workGroupId returns the index of the work group in the kernel dispatch for a
dimension.
* @workGroupSize returns the size of the work group for a dimension.
These builtins are mainly useful for GPU backends. They are currently only
implemented for the AMDGCN LLVM backend.
As bpf program has no global section for constant values (especially strings),
so use llvm's builtins (like memcpy, memset, etc) will lead to compilation failure
(something like this: A call to built-in function 'memcpy' is not supported.)
Signed-off-by: Tw <tw19881113@gmail.com>
- `fcntl` was renamed to `fcntl64` in glibc 2.28 (see #9485)
- `res_{,n}{search,query,querydomain}` became "their own" symbols since
glibc 2.34: they were prefixed with `__` before.
This PR makes it possible to use `fcntl` with glibc 2.27 or older and
the `res_*` functions with glibc 2.33 or older.
These patches will become redundant with universal-headers and can be
dropped. But we have to do with what we have now.
Closes#9485
Instead, introduce a custom ImportTable structure which will
act as a thunk in the MachO linker, and we will use that to
calculate the address of a pointer on-the-fly.
Additionally, fix logic in writeImportTables to allow for multiple
DLLs.
Make the termColor function more robust, ensuring that the generated
HTML classes are more consistent and that they are supported in the CSS.
Add documentation about the ANSI codes generated by Zig, and remove the
previous comment with the supported colors.
Improve test coverage for the termColor function, and move the tests at
the end of the file before the printShell tests.
Rename the "term color" test to "term supported colors" and add an
additional "term output from zig" test, using test data generated from
the Zig compiler.
Update the langref.html.in CSS to use the new names and remove incorrect
or obsolete colors like .t0_1, .t37 and .t37_1.
Fix support for the 1m color. Change font-weight to normal for the kbd
element, to avoid both the command (like `zig build-exe`) and the first
line of the error message having a bold font.
Rename the "shell parsed" test to "printShell".
Line support in printShell breaks the support to terminal colors, when a
color is active over multiple lines. An example is when printing
reference traces.
Remove line support in printShell, since they are not really necessary
like in printSourceBlock.
In genHtml, remove an empty line when printing the execution of the
executable generated by `zig build-exe`.
Update the "shell parsed" tests.
Remove the call to log.emerg, since it is no longer supported.
Add an extra space after "--build-option1 \", as documented.
Fixes#13280
Notably the Darwin (XNU) kernel the maxrss field is number of bytes
and not kilobytes (kibibytes) like other platforms (e.g. Linux, BSD).
watchOS and tvOS are not supported because they do not have the ability
to spawn a child process. iOS is enabled but due to OS sandboxing it
should fail with a permission error.
Common headers in a response are:
Content-Encoding: gzip
Transfer-Encoding: chunked
We used to return `HttpHeadersInvalid` if a `Transfer-Encoding` header
was received while the compression was already set.
However, Transfer-Encoding may not include compression. We should
only return an error if we are setting a value that was already set.
Fixes compatibility with a bunch of websites.
On CPUs without AES support, ChaCha is always faster and safer than
software AES.
Add `crypto.core.aes.has_hardware_support` to represent whether
AES acceleration is available or not, and in `tls.Client`, favor
AES-based ciphers only if hardware support is available.
This matches what BoringSSL is doing.
`GetProcessMemoryInfo` is implemented using `NtQueryInformationProcess`
with `ProcessVmCounters` to obtain `VM_COUNTERS`. The structs, enum
definitions are found in `winternl.h` or `ntddk.h` in the latest WDK.
This should give the same results as using `K32GetProcessMemoryInfo`
In Windows, the equivalent to maxrss is PeakWorkingSetSize which is
found in PROCESS_MEMORY_COUNTERS in bytes.
Currently, this is done by calling `GetProcessMemoryInfo` in kernel32.
HMAC supports arbitrary key sizes, and there are no practical reasons
to use more than 256 bit keys.
It still makes sense to match the security level, though, especially
since a distinction between the block size and the key size can be
confusing.
Using HMAC.key_size instead of HMAC.mac_size caused our TLS
implementation to compute wrong shared secrets when SHA-384 was
used. So, fix it directly in `crypto.hmac` in order to prevent
other misuses.
Previously, if we were looking for the very last symbol by address in some
section, and the next symbol happened to also have the same address value
but would reside in a different section, we would keep going finding the
wrong symbol in the wrong section.
This mechanism turns out vital for correct linking of Go binaries
where the runtime looks for specially crafted synthetic symbols
which mark the beginning and end of each section. In this case,
we had an unfortunate clash between the end of PC marked machine code
section (`_runtime.etext`) and beginning of read-only data (`_runtime.rodata`).
This should add all remaining missing return types to all builtin
functions.
For @clz, @ctz, and @popCount it uses anytype for the lack of a better
alternative. We already use this return type for other builtin functions in the langref
to indicate that the type is not always the same.
It is not possible to use anytype as the return type for regular
functions but builtin functions are special.
These are great permutations, and there's nothing wrong with them
from a practical security perspective.
However, both were competing in the NIST lightweight crypto
competition.
Gimli didn't pass the 3rd selection round, and is not much used
in the wild besides Zig and libhydrogen. It will never be
standardized and is unlikely to get more traction in the future.
Xoodyak, that Xoodoo is the permutation of, was a finalist.
It has a lot of advantages and *might* be standardized without NIST.
But this is too early to tell, and too risky to commit to it
in a standard library.
For lightweight crypto, Ascon is the one that we know NIST will
standardize and that we can safely rely on from a usage perspective.
Switch to a traditional ChaCha-based CSPRNG, with an Ascon-based one
as an option for constrained systems.
Add a RNG benchmark by the way.
Gimli and Xoodoo served us well. Their code will be maintained,
but outside the standard library.
It is reasonable to pass -Dskip-non-native when unable to run foreign
binaries, however there is no option for being able to run foreign
static binaries but unable to run foreign dynamic binaries. This can
occur when qemu is installed but not cross glibc.
There are now very few stage1 cases remaining:
* `cases/compile_errors/stage1/obj/*` currently don't work correctly on
stage2. There are 6 of these, and most of them are probably fairly
simple to fix.
* `cases/compile_errors/async/*` and all remaining `safety/*` depend on
async; see #6025.
Resolves: #14849
In case the compiler outputted an object file that is not slicable
into subsections, entry point may overlap with a section atom which
is perfectly fine, so don't panic in that case.
as documented at
https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html
It is now equivalent to do
```
zig objcopy --only-keep-debug bar foo.debug
zig objcopy -g bar foo.tmp
zig objcopy --add-gnu-debuglink=foo.debug foo.tmp foo
rm foo.tmp
```
or
```
zig objcopy --only-keep-debug bar foo foo.debug
zig objcopy -g --add-gnu-debuglink=foo.debug bar foo
```
or
```
zig objcopy -g --extract-to=foo.debug bar foo
```
Support for the use case:
`zig objcopy --strip-all program stripped --extract-to stripped.dbg`
to separate the debug & symbols sections out to a companion file, with a corresponding a .gnu_debuglink.
note: this is "a minimal effort implementation"
It doesn't support all possibile elf files: there may be some sections type that need fixups, the program header may need fix up, ...
It was written for a specific use case (strip debug info to a sperate file, for linux 64-bits executables built with `zig` or `zig c++` )
It doesn't support 32-bit files, or file with non-native endianess.
* Fix GetFileInformationByHandle compile error
The wrapper function was mistakenly referencing ntdll.zig when the actual function is declared in kernel32.zig.
* delete GetFileInformationByHandle since it's not used by the stdlib
If the shared library is a relative path, dirname will return null causing a segfault. In the case I debugged, the current directory was already in RPATH so just ignoring this case seems a reasonable fix. After this fix "make" and "make test" pass for mimalloc.
Closes#13766
This function is unused, and the current implementation contains a few footguns:
- The current wrapper treats all possible errors as unexpected, even likely ones like BUFFER_OVERFLOW (which is returned if the size of the out_buffer is too small to contain all the variable-length members of the requested info, which the user may not actually care about)
- Each caller may need to handle errors differently, different errors might be possible depending on the FILE_INFORMATION_CLASS, etc, and making a wrapper that handles all of those different use-cases nicely seems like it'd be more trouble than it's worth (FILE_INFORMATION_CLASS has 76 different possible values)
If a wrapper for NtQueryInformationFile is wanted, then it should probably have wrapper functions per-use-case, like how QueryObjectName wraps NtQueryObject for the `ObjectNameInformation` class
* use the same hash function as the rest of the steps
* fix race condition due to a macOS oddity.
* fix race condition due to file truncation (rename into place instead)
* integrate with marking Step.result_cached. check if the file already
exists with fs.access before doing anything else.
* use a directory so that the file basename can be "options.zig"
instead of a hash digest.
* better error reporting in case of file system failures.
Implements the __wasm_init_memory and __wasm_init_memory_flag synthetic
function and symbol.
The former will initialize all passive segments during runtime. For the
bss section we will fill it with zeroes, whereas the other segments
will simply be initialized only.
The latter stores the offset into the linear data section, after all
heap memory that is part of the Wasm module. Any memory initialized
at runtime starts from this offset.
This adds the atomic opcodes for the Threads proposal to the
WebAssembly specification: https://github.com/WebAssembly/threads
PrefixedOpcode has been renamed to MiscOpcode as there's multiple
types of prefixed opcodes. This naming is similar to other tools
such as LLVM. As we now use the 0xFE prefix, we moved the
function_index MIR instruction as it was occupying the same value.
This commit includes renaming all related opcodes.
Implements the TLS initialization function. This is a synthetic function
created by the linker. This will only be created when shared-memory is
enabled. This function will be called during thread creation, if there's
any TLS symbols, which will initialize the TLS segment using the
bulk-memory feature.
When linking with shared-memory enabled, we must ensure to emit
the "data count" section as well as emit the correct segment flags
to tell the runtime/loader that each segment is passive. This is
required as we don't emit the offsets for such segments but instead
initialize each segment (for each thread) during runtime.
Rather than adding the flags "on-demand" during limits writing,
we now properly parse them and store the flags within the limits
itself. This also allows us to store whether we're using shared-
memory or not. Only when the correct flag is set will we set the
max within `Limits` or else we will leave it `undefined`.
Linker now parses segments with regards to TLS segments. If the name
represents a TLS segment but does not contain the TLS flag, we set it
manually as the object file is created using an older compiler (LLVM).
For now we panic when we find a TLS relocation and implement those
later.
When a decl is `undefined` is must be stored in the data segment when
the build mode is safe. For unsafe optimize modes, it must be stored
in the bss segment instead.
For mutable decls where the atom contains all zeroes, it must always
be stored in the bss segment. All other values will result in the
atom being stored in the data segment.
--sysroot was copy-pasted instead of --maxrss from above. Also, make it less likely in other places for this to be repeated.
I found this by accident when reviewing f51413d2cf
It seems like the original code of setsockopt is not effective because
i catch the EINVAL branch when uncomment this code, it should call
setsockopt before the bind call.
This should fix issue #14900.
Co-authored-by: Qun He <hawkbee@qq.com>
This is useful for tests that want to `execve` zig directly. The string
is already null-terminated, so this will just expose it as such,
removing an extra allocation from the test.
Will be used in #14462
This is useful for creating byte buffers of actually-different-things.
Copied the argument order from `Allocator.alignedAlloc`
I noted that `ArrayListAligned` is going out of it's way to not set the
alignment at comptime when it is not specified. However, I was not able
to do that the same way here, and good people on IRC, @ifreund in
particular (thanks!) assured me that
[N]T align(@alignOf(T))
is equivalent to
[N]T
It doesn't matter if a pointer to a zero-bit (i.e. OPV) type is
undefined or runtime-known; we still know the result of the dereference
at comptime. Code may use this, for instance, when allocating zero-bit
types: `@as(*void, undefined)` is entirely reasonable to use at runtime,
since we know the pointer will never be accessed, thus it should be
valid at comptime too.
This test has a few problems:
* I don't want to vendor third party projects into the main compiler
repository such as kuba-zip just for test cases. If we want to test
third party projects, that should be a separate repository dedicated
to that purpose.
* Ideally tests would be isolated to test a particular thing, rather
than have a lot of unrelated logic that is not what is primarily
being tested.
* Ideally tests will not be named after GitHub issues, but named after
the thing that is being tested. And not testing for the absence of a
bug, but for the existence of correct behavior.
Aside from these issues, it's also failing in the LLVM 16 branch:
```
kuba-zip/zip.c:276:16: error: call to undeclared function 'fileno'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
kuba-zip/zip.c:277:14: error: call to undeclared function 'ftruncate'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
kuba-zip/zip.c:364:11: error: call to undeclared function 'symlink'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
```
These are not interesting failures related to the thing actually being
tested; this is busywork related to the fact that we vendor third party
code. So, that is why I chose to delete this test case instead of repair
it.
Move to c/darwin.zig as they really are libSystem/libc imports/wrappers.
As an added bonus, get rid of the nasty `usingnamespace`s which are now
unneeded.
Finally, add `os.ptrace` but currently only implemented on darwin.
commit 3204d00a5e intended to move this
passing test case from stage1 folder but it was accidentally changed to
have identical contents as a different test case instead.
Fortunately, the test case has not regressed, so I simply replaced it
with the intended test from before.
In master branch this test tests native Windows. In this branch, I
accidentally made aarch64-windows test x86_64-windows which caused some
subtle behavior that we aren't ready to add test coverage for yet.
Previously this code asserted that a fifo's readable length was greater
than or equal to the length of its readable slice, which was an invalid
assertion.
This code avoids making that assumption.
This error means that there *was* a file in this location on the file
system, but it was deleted. However, the OS is not finished with the
deletion operation, and so this CreateFile call has failed. There is not
really a sane way to handle this other than retrying the creation after
the OS finishes the deletion.
The error was caught and created a Step failure rather than bubbling up
so that the interpreter logic could handle it. Fixes hundreds of test
failures on Windows.
There are no dir components, so you would think that this was
unreachable, however we have observed on macOS two processes racing
to do openat() with O_CREAT manifest in ENOENT.
The OS layer expects pointer addresses to be inside the application's
address space even if the length is zero. Meanwhile, in Zig, slices may
have undefined pointer addresses when the length is zero. So this
function now modifies the iov_base fields when the length is zero.
This is a companion commit to b4893eb05565b2cb033c6ed88617d73faf878455.
This was from master branch commit
c93e0d8618. Since standalone test are
completely reworked, I had to resolve the merge conflict later, in this
commit.
* Remove some functions that are no longer needed since
EmulateableRunStep is gone.
* Add removeEnvironmentVariable function.
* Support printing environment variables in --verbose mode.
This unit test tested the command line arguments which were passed to
the test runner, which is not really something that unit tests are
supposed to observe.
The proper way to test command line argument parsing is with a
standalone test, where the set of command line arguments being tested
for are also being controlled by the test itself.
std.Build.addTest creates a CompileStep as before, however, this kind of
step no longer actually runs the unit tests. Instead it only compiles
it, and one must additionally create a RunStep from the CompileStep in
order to actually run the tests.
RunStep gains integration with the default test runner, which now
supports the standard --listen=- argument in order to communicate over
stdin and stdout. It also reports test statistics; how many passed,
failed, and leaked, as well as directly associating the relevant stderr
with the particular test name that failed.
This separation of CompileStep and RunStep means that
`CompileStep.Kind.test_exe` is no longer needed, and therefore has been
removed in this commit.
* build runner: show unit test statistics in build summary
* added Step.writeManifest since many steps want to treat it as a
warning and emit the same message if it fails.
* RunStep: fixed error message that prints the failed command printing
the original argv and not the adjusted argv in case an interpreter
was used.
* RunStep: fixed not passing the command line arguments to the
interpreter.
* move src/Server.zig to std.zig.Server so that the default test runner
can use it.
* the simpler test runner function which is used by work-in-progress
backends now no longer prints to stderr, which is necessary in order
for the build runner to not print the stderr as a warning message.
std.zig.system.NativeTargetInfo.getExternalExecutor previously would
incorrectly communicate that wine could be used to run aarch64-windows
binaries on x86_64-linux, and x86_64-windows binaries on aarch64-linux.
Neither of these things are true.
This reverts commit e6f759e1c64668c50d3ff2d02c64a66c871da0ac.
I changed my mind. I don't like the output because it makes it harder to
find the actual errors in CI logs.
Terminal progress is suppressed and instead there is an explicit
handling of printing to stderr, one line per step make() function call.
The output looks very similar to Ninja.
A future commit should add a -q to quiet the output.
it still writes the output to zig-cache/langref.html but now it does
that explicitly as a legacy step with the intention of having that
removed in the future. It also outputs the langref to the install
prefix.
This commit extracts out server code into src/Server.zig and uses it
both in the main CLI as well as `zig objcopy`.
std.Build.ObjCopyStep now adds `--listen=-` to the CLI for `zig objcopy`
and observes the protocol for progress and other kinds of integrations.
This fixes the last two test failures of this branch when I run
`zig build test` locally.
Previously this code used SipHash(1, 3) directly; now that we have the
cache system available in the build system, borrow the same
implementation as is being used everywhere else.
Instead of using `zig test` to build a special version of the compiler
that runs all the test-cases, the zig build system is now used as much
as possible - all with the basic steps found in the standard library.
For incremental compilation tests (the ones that look like foo.0.zig,
foo.1.zig, foo.2.zig, etc.), a special version of the compiler is
compiled into a utility executable called "check-case" which checks
exactly one sequence of incremental updates in an independent
subprocess. Previously, all incremental and non-incremental test cases
were done in the same test runner process.
The compile error checking code is now simpler, but also a bit
rudimentary, and so it additionally makes sure that the actual compile
errors do not include *extra* messages, and it makes sure that the
actual compile errors output in the same order as expected. It is also
based on the "ends-with" property of each line rather than the previous
logic, which frankly I didn't want to touch with a ten-meter pole. The
compile error test cases have been updated to pass in light of these
differences.
Previously, 'error' mode with 0 compile errors was used to shoehorn in a
different kind of test-case - one that only checks if a piece of code
compiles without errors. Now there is a 'compile' mode of test-cases,
and 'error' must be only used when there are greater than 0 errors.
link test cases are updated to omit the target object format argument
when calling checkObject since that is no longer needed.
The test/stage2 directory is removed; the 2 files within are moved to be
directly in the test/ directory.
* std.zig.ErrorBundle: support rendering options for whether to include
the reference trace, whether to include the source line, and TTY
configuration.
* build runner: don't print progress in dumb terminals
* std.Build.CompileStep:
- add a way to expect compilation errors via the new `expect_errors`
field. This is an advanced setting that can change the intent of
the CompileStep. If this slice has nonzero length, it means that
the CompileStep exists to check for compile errors and return
*success* if they match, and failure otherwise.
- remove the object format parameter from `checkObject`. The object
format is known based on the CompileStep's target.
- Avoid passing -L and -I flags for nonexistent directories within
search_prefixes. This prevents a warning, that should probably be
upgraded to an error in Zig's CLI parsing code, when the linker
sees an -L directory that does not exist.
* std.Build.Step:
- When spawning the zig compiler process, takes advantage of the new
`std.Progress.Node.setName` API to avoid ticking up a meaningless
number at every progress update.
The OS layer expects pointer addresses to be inside the application's
address space even if the length is zero. Meanwhile, in Zig, slices may
have undefined pointer addresses when the length is zero. So this
function now modifies the iov_base fields when the length is zero.
I disagree with this behavior and will be reverting the changes
corresponding to this test case.
Also this test case unnecessarily uses a .c file when a .zig file would
be preferred, and has a problematic dependency on the install step,
preventing this test case from playing nicely with the cache.
It's simpler and it takes advantage of
`std.Build.addAnonymousDependency`, which has a number of benefits,
including concurrenc and preventing extra zig-cache and zig-out
directories being created.
4 tests are ported over as an example.
This is for bypassing the package manager and directly depending on
another package via the build system. For this to work the anonymous
package must be found on the file system relative to the current
package's build.zig.
These were mostly already using the correct build API. I cleaned up the
code a bit and unconditionally disabled LTO for these tests since that
actually tests the intended behavior better.
CLI tests are now ported over to the new std.Build API and thus work
properly with concurrency.
* add `std.Build.addCheckFile` for creating a
`std.Build.CheckFileStep`.
* add `std.Build.makeTempPath`. This function is intended to be called
in the `configure` phase only. It returns an absolute directory path,
which is potentially going to be a source of API breakage in the
future, so keep that in mind when using this function.
* add `std.Build.CheckFileStep.setName`.
* `std.Build.CheckFileStep`: better error message when reading the
input file fails.
* `std.Build.RunStep`: add a `has_side_effects` flag for when you need
to override the autodetection.
* `std.Build.RunStep`: add the ability to obtain a FileSource for the
directory that contains the written files.
* `std.Build.WriteFileStep`: add a way to write bytes to an arbitrary
path - absolute or relative to the package root. Be careful with this
because it updates source files. This should not be used as part of
the normal build process, but as a utility occasionally run by a
developer with intent to modify source files and then commit those
changes to version control. A file added this way is not available
with `getFileSource`.
* RunStep: ability to set stdin
* RunStep: ability to capture stdout and stderr as a FileSource
* RunStep: add setName method
* RunStep: hash the stdio checks
The problem is that one may execute too many subprocesses concurrently
that, together, exceed an RSS value that causes the OOM killer to kill
something problematic such as the window manager. Or worse, nothing, and
the system freezes.
This is a real world problem. For example when building LLVM a simple
`ninja install` will bring your system to its knees if you don't know
that you should add `-DLLVM_PARALLEL_LINK_JOBS=1`.
In particular: compiling the zig std lib tests takes about 2G each,
which at 16x at once (8 cores + hyperthreading) is using all 32GB of my
RAM, causing the OOM killer to kill my window manager
The idea here is that you can annotate steps that might use a high
amount of system resources with an upper bound. So for example I could
mark the std lib tests as having an upper bound peak RSS of 3 GiB.
Then the build system will do 2 things:
1. ulimit the child process, so that it will fail if it would exceed
that memory limit.
2. Notice how much system RAM is available and avoid running too many
concurrent jobs at once that would total more than that.
This implements (1) not with an operating system enforced limit, but by
checking the maxrss after a child process exits.
However it does implement (2) correctly.
The available memory used by the build system defaults to the total
system memory, regardless of whether it is used by other processes at
the time of spawning the build runner. This value can be overridden with
the new --maxrss flag to `zig build`. This mechanism will ensure that
the sum total of upper bound RSS memory of concurrent tasks will not
exceed this value.
This system makes it so that project maintainers can annotate
problematic subprocesses, avoiding bug reports from users, who can
blissfully execute `zig build` without worrying about the project's
internals.
Nobody's computer crashes, and the build system uses as much parallelism
as possible without risking OOM. Users do not need to unnecessarily
resort to -j1 when the build system can figure this out for them.
* remove std.Build.updateFile. I noticed some people use it from
build.zig (declare phase) when it is intended only for use in the
make phase.
- This also was incorrectly reporting errors with std.log.
* std.Build.InstallArtifactStep
- report better errors on failure
- report whether the step was cached or not
* std.Build.InstallDirStep: report better error on failure
* std.Build.InstallFileStep: report better error on failure
* Eliminate all uses of `std.debug.print` in make() functions, instead
properly using the step failure reporting mechanism.
* Introduce the concept of skipped build steps. These do not cause the
build to fail, and they do allow their dependants to run.
* RunStep gains a new flag, `skip_foreign_checks` which causes the
RunStep to be skipped if stdio mode is `check` and the binary cannot
be executed due to it being a foreign executable.
- RunStep is improved to automatically use known interpreters to
execute binaries if possible (integrating with flags such as
-fqemu and -fwasmtime). It only does this after attempting a native
execution and receiving a "exec file format" error.
- Update RunStep to use an ArrayList for the checks rather than this
ad-hoc reallocation/copying mechanism.
- `expectStdOutEqual` now also implicitly adds an exit_code==0 check
if there is not already an expected termination. This matches
previously expected behavior from older API and can be overridden by
directly setting the checks array.
* Add `dest_sub_path` to `InstallArtifactStep` which allows choosing an
arbitrary subdirectory relative to the prefix, as well as overriding
the basename.
- Delete the custom InstallWithRename step that I found deep in the
test/ directory.
* WriteFileStep will now update its step display name after the first
file is added.
* Add missing stdout checks to various standalone test case build
scripts.
Rework std.Build.Step to have an `owner: *Build` field. This
simplified the implementation of installation steps, as well as provided
some much-needed common API for the new parallelized build system.
--verbose is now defined very concretely: it prints to stderr just
before spawning a child process.
Child process execution is updated to conform to the new
parallel-friendly make() function semantics.
DRY up the failWithCacheError handling code. It now integrates properly
with the step graph instead of incorrectly dumping to stderr and calling
process exit.
In the main CLI, fix `zig fmt` crash when there are no errors and stdin
is used.
Deleted steps:
* EmulatableRunStep - this entire thing can be removed in favor of a
flag added to std.Build.RunStep called `skip_foreign_checks`.
* LogStep - this doesn't really fit with a multi-threaded build runner
and is effectively superseded by the new build summary output.
build runner:
* add -fsummary and -fno-summary to override the default behavior,
which is to print a summary if any of the build steps fail.
* print the dep prefix when emitting error messages for steps.
std.Build.FmtStep:
* This step now supports exclude paths as well as a check flag.
* The check flag decides between two modes, modify mode, and check
mode. These can be used to update source files in place, or to fail
the build, respectively.
Zig's own build.zig:
* The `test-fmt` step will do all the `zig fmt` checking that we expect
to be done. Since the `test` step depends on this one, we can simply
remove the explicit call to `zig fmt` in the CI.
* The new `fmt` step will actually perform `zig fmt` and update source
files in place.
std.Build.RunStep:
* expose max_stdio_size is a field (previously an unchangeable
hard-coded value).
* rework the API. Instead of configuring each stream independently,
there is a `stdio` field where you can choose between
`infer_from_args`, `inherit`, or `check`. These determine whether the
RunStep is considered to have side-effects or not. The previous
field, `condition` is gone.
* when stdio mode is set to `check` there is a slice of any number of
checks to make, which include things like exit code, stderr matching,
or stdout matching.
* remove the ill-defined `print` field.
* when adding an output arg, it takes the opportunity to give itself a
better name.
* The flag `skip_foreign_checks` is added. If this is true, a RunStep
which is configured to check the output of the executed binary will
not fail the build if the binary cannot be executed due to being for
a foreign binary to the host system which is running the build graph.
Command-line arguments such as -fqemu and -fwasmtime may affect
whether a binary is detected as foreign, as well as system
configuration such as Rosetta (macOS) and binfmt_misc (Linux).
- This makes EmulatableRunStep no longer needed.
* Fix the child process handling to properly integrate with the new
bulid API and to avoid deadlocks in stdout/stderr streams by polling
if necessary.
std.Build.RemoveDirStep now uses the open build_root directory handle
instead of an absolute path.
This makes progress be exposed to the top-level caller of update().
I tossed in a bonus change: when the `zig build` subcommand sees exit
code 2, it omits the "following command failed" line, and the build
runner uses exit code 2 when there are compile errors. This tidies up
the output on build failure by a little bit.
The compiler now provides a server protocol for an interactive session
with another process. The build runner uses this protocol to communicate
compilation errors semantically from zig compiler subprocesses to the
build runner.
The protocol is exposed via stdin/stdout, or on a network socket,
depending on whether the CLI flag `--listen=-` or e.g.
`--listen=127.0.0.1:1337` is used.
Additionally:
* add the zig version string to the build runner cache prefix
* remove --prominent-compile-errors CLI flag because it no longer does
anything. Compilation errors are now unconditionally displayed at the
bottom of the build summary output when using the terminal-based
build runner.
* Remove the color field from std.Build. The build steps are no longer
supposed to interact with stderr directly. Instead they communicate
semantically back to the build runner, which has its own logic about
TTY configuration.
* Use the cleanExit() pattern in the build runner.
* Build steps can now use error.MakeFailed when they have already
properly reported an error, or they can fail with any other error
code in which case the build runner will create a simple message
based on this error code.
Introduces std.zig.ErrorBundle which is a trivially serializeable set
of compilation errors. This is in the standard library so that both
the compiler and the build runner can use it. The idea is they will
use it to communicate compilation errors over a binary protocol.
The binary encoding of ErrorBundle is a bit problematic - I got a little
too aggressive with compaction. I need to change it in a follow-up
commit to use some indirection in the error message list, otherwise
iteration is too unergonomic. In fact it's so problematic right now that
the logic getAllErrorsAlloc() actually fails to produce a viable
ErrorBundle because it puts SourceLocation data in between the root
level ErrorMessage data.
This commit has a simplification - redundant logic for rendering AST
errors to stderr has been removed in favor of moving the logic for
lowering AST errors into AstGen. So even if we get parse errors, the
errors will get lowered into ZIR before being reported. I believe this
will be useful when working on --autofix. Either way, some redundant
brittle logic was happily deleted.
In Compilation, updateSubCompilation() is improved to properly perform
error reporting when a sub-compilation object fails. It no longer dumps
directly to stderr; instead it populates an ErrorBundle object, which
gets added to the parent one during getAllErrorsAlloc().
In package fetching code, instead of dumping directly to stderr, it now
populates an ErrorBundle object, and gets properly reported at the CLI
layer of abstraction.
- improve fn prototypes of process_vm_writev
- make the memory writable in the ELF file
- force the linker to always append the function
- write updates with process_vm_writev
Zig's build script has several race conditions preventing proper
concurrent builds from working. By using -j1 for now, finishing this
branch (concurrent zig builds) is untangled from the separate problem of
correcting concurrency issues with zig's own build script.
In other words, let's solve one problem at a time.
With this commit, the build runner now communicates progress towards
completion of the step graph to the terminal, while also printing the
stderr of child processes as soon as possible, without clobbering each
other, and without clobbering the CLI progress output.
API users can take advantage of these to freely write to the terminal
which has an ongoing progress display, similar to what Ninja does when
compiling C/C++ objects and a warning or error message is printed.
* Step.init() now takes an options struct
* Step.init() now captures a small stack trace and stores it in the
Step so that it can be accessed when printing user-friendly debugging
information, including the lines of code that created the step in
question.
Instead of dumping directly to stderr. This prevents processes running
simultaneously from racing their stderr against each other.
For now it only reports at the end, but an improvement would be to
report as soon as a failed step occurs.
After sorting the step stack so that dependencies can be popped before
their dependants are popped, there is still a situation left to handle
correctly:
Example:
A depends on:
B
C
D depends on:
E
F
They will be ordered like this:
A B C D E F
If there are 6+ cores, then all of them will be evaluated at once,
incorrectly evaluating A and D before their dependencies.
Starting evaluation of F and then E is correct, but waiting until they
are done is not correct because it should start working on B and C as
well.
This commit solves the problem by computing dependants in the dependency
loop checking logic, and then having workers queue up their dependants
when they finish their own work.
Implementation of the IND-CCA2 post-quantum secure key encapsulation
mechanism (KEM) CRYSTALS-Kyber, as submitted to the third round of the NIST
Post-Quantum Cryptography (v3.02/"draft00"), and selected for standardisation.
Co-authored-by: Frank Denis <124872+jedisct1@users.noreply.github.com>
Previously, if you had a pointer to multiple array elements and tried to
write to it at comptime, it was incorrectly treated as a pointer to one
specific array value, leading to an assertion down the line. If we try
to mutate a value at an elem_ptr larger than the element type, we need
to perform a modification to multiple array elements.
This solution isn't ideal, since it will result in storePtrVal
serializing the whole array, modifying the relevant parts, and storing
it back. Ideally, it would only take the required elements. However,
this change would have been more complex, and this is a fairly rare
operation (nobody ever ran into the bug before after all), so it doesn't
matter all that much.
* Add configurable side channels mitigations; enable them on soft AES
Our software AES implementation doesn't have any mitigations against
side channels.
Go's generic implementation is not protected at all either, and even
OpenSSL only has minimal mitigations.
Full mitigations against cache-based attacks (bitslicing, fixslicing)
come at a huge performance cost, making AES-based primitives pretty
much useless for many applications. They also don't offer any
protection against other classes of side channel attacks.
In practice, partially protected, or even unprotected implementations
are not as bad as it sounds. Exploiting these side channels requires
an attacker that is able to submit many plaintexts/ciphertexts and
perform accurate measurements. Noisy measurements can still be
exploited, but require a significant amount of attempts. Wether this
is exploitable or not depends on the platform, application and the
attacker's proximity.
So, some libraries made the choice of minimal mitigations and some
use better mitigations in spite of the performance hit. It's a
tradeoff (security vs performance), and there's no one-size-fits all
implementation.
What applies to AES applies to other cryptographic primitives.
For example, RSA signatures are very sensible to fault attacks,
regardless of them using the CRT or not. A mitigation is to verify
every produced signature. That also comes with a performance cost.
Wether to do it or not depends on wether fault attacks are part of
the threat model or not.
Thanks to Zig's comptime, we can try to address these different
requirements.
This PR adds a `side_channels_protection` global, that can later
be complemented with `fault_attacks_protection` and possibly other
knobs.
It can have 4 different values:
- `none`: which doesn't enable additional mitigations.
"Additional", because it only disables mitigations that don't have
a big performance cost. For example, checking authentication tags
will still be done in constant time.
- `basic`: which enables mitigations protecting against attacks in
a common scenario, where an attacker doesn't have physical access to
the device, cannot run arbitrary code on the same thread, and cannot
conduct brute-force attacks without being throttled.
- `medium`: which enables additional mitigations, offering practical
protection in a shared environement.
- `full`: which enables all the mitigations we have.
The tradeoff is that the more mitigations we enable, the bigger the
performance hit will be. But this let applications choose what's
best for their use case.
`medium` is the default.
Currently, this only affects software AES, but that setting can
later be used by other primitives.
For AES, our implementation is a traditional table-based, with 4
32-bit tables and a sbox.
Lookups in that table have been replaced by function calls. These
functions can add a configurable noise level, making cache-based
attacks more difficult to conduct.
In the `none` mitigation level, the behavior is exactly the same
as before. Performance also remains the same.
In other levels, we compress the T tables into a single one, and
read data from multiple cache lines (all of them in `full` mode),
for all bytes in parallel. More precise measurements and way more
attempts become necessary in order to find correlations.
In addition, we use distinct copies of the sbox for key expansion
and encryption, so that they don't share the same L1 cache entries.
The best known attacks target the first two AES round, or the last
one.
While future attacks may improve on this, AES achieves full
diffusion after 4 rounds. So, we can relax the mitigations after
that. This is what this implementation does, enabling mitigations
again for the last two rounds.
In `full` mode, all the rounds are protected.
The protection assumes that lookups within a cache line are secret.
The cachebleed attack showed that it can be circumvented, but
that requires an attacker to be able to abuse hyperthreading and
run code on the same core as the encryption, which is rarely a
practical scenario.
Still, the current AES API allows us to transparently switch to
using fixslicing/bitslicing later when the `full` mitigation level
is enabled.
* Software AES: use little-endian representation.
Virtually all platforms are little-endian these days, so optimizing
for big-endian CPUs doesn't make sense any more.
In 41 (Undefined Behavior) . 5 (Integer Overflow) . 3 (Builtin Overflow Functions), it is stated that
> These builtins return a bool of whether or not overflow occurred, as well as returning the overflowed bits:
> * @addWithOverflow
> * @subWithOverflow
> * @mulWithOverflow
> * @shlWithOverflow
but in their definition says that it returns a `tuple`/`struct`.
Example;
`@addWithOverflow(a: anytype, b: anytype) struct { @TypeOf(a, b), u1 }`
Co-authored-by: zooster <r00ster91@proton.me>
Apple M1/M2 have an EOR3 instruction that can XOR 2 operands with
another one, and LLVM knows how to take advantage of it.
However, two EOR can't be automatically combined into an EOR3 if
one of them is in an assembly block.
That simple change speeds up ciphers doing an AES round immediately
followed by a XOR operation on Apple Silicon.
Before:
aegis-128l mac: 12534 MiB/s
aegis-256 mac: 6722 MiB/s
aegis-128l: 10634 MiB/s
aegis-256: 6133 MiB/s
aes128-gcm: 3890 MiB/s
aes256-gcm: 3122 MiB/s
aes128-ocb: 2832 MiB/s
aes256-ocb: 2057 MiB/s
After:
aegis-128l mac: 15667 MiB/s
aegis-256 mac: 8240 MiB/s
aegis-128l: 12656 MiB/s
aegis-256: 7214 MiB/s
aes128-gcm: 3976 MiB/s
aes256-gcm: 3202 MiB/s
aes128-ocb: 2835 MiB/s
aes256-ocb: 2118 MiB/s
* There was an edge case where the arena could be destroyed twice on
error: once from the arena itself and once from the decl destruction.
* The type of the created decl was incorrect (it should have been the
pointer child type), but it's not required anyway, so it's now just
initialized to anyopaque (which more accurately reflects what's
actually at that memory, since e.g. [*]T may correspond to nothing).
* A runtime bitcast of the pointer was performed, meaning @extern didn't
work at comptime. This is unnecessary: the decl_ref can just be
initialized with the correct pointer type.
There are no dir components, so you would think that this was
unreachable, however we have observed on macOS two processes racing to
do openat() with O_CREAT manifest in ENOENT.
closes#12138
Today I found out that posix_spawn is trash. It's actually implemented
on top of fork/exec inside of libc (or libSystem in the case of macOS).
So, anything posix_spawn can do, we can do better. In particular, what
we can do better is handle spawning of child processes that are
potentially foreign binaries. If you try to spawn a wasm binary, for
example, posix spawn does the following:
* Goes ahead and creates a child process.
* The child process writes "foo.wasm: foo.wasm: cannot execute binary file"
to stderr (yes, it prints the filename twice).
* The child process then exits with code 126.
This behavior is indistinguishable from the binary being successfully
spawned, and then printing to stderr, and exiting with a failure -
something that is an extremely common occurrence.
Meanwhile, using the lower level fork/exec will simply return ENOEXEC
code from the execve syscall (which is mapped to zig error.InvalidExe).
The posix_spawn behavior means the zig build runner can't tell the
difference between a failure to run a foreign binary, and a binary that
did run, but failed in some other fashion. This is unacceptable, because
attempting to excecve is the proper way to support things like Rosetta.
The TurboSHAKE paper just got published:
https://eprint.iacr.org/2023/342.pdf
and unlike the previous K12 paper, suggests 0x1F instead of 0x01
as the default value for "D".
For data symbols we will now store its virtual address. This means
we do no longer have to calculate it each time a relocation asks
for the address. This is now done for all data symbols only once
rather than every single relocation for that symbol.
This now also allows us directly store the virtual address of synthetic
symbols without having to create an atom for them. This means we also
don't need to have a "synthetic" segment any longer and do not emit
the synthetic symbols such as __heap_end and __heap_base into the final
binary.
* Fix SHA3 with streaming
Leftover bytes should be added to the buffer, not to the state.
(or, always to the state; we can and probably should eventually get
rid of the buffer)
Fixes#14851
* Add a test for SHA-3 with streaming
Previously this worked for array to vector where the element type
matched exactly (e.g `[4]u8` to `@Vector(4, u8)`) since that is
performed with a simple `.bitcast` operation, but now it also works for
types where the array is coercible to the vector type (e.g `[4]u8` to
`@Vector(4, u16)`).
This is consistent with how coercion for vectors work. So now you can do
this:
```
var a: @Vector(2, u16) = .{1, 2};
var b: @Vector(2, u8) = .{2, 1};
const c = @min(a, b);
```
where previously you had to cast explicitly:
```
var a: @Vector(2, u16) = .{1, 2};
var b: @Vector(2, u8) = .{2, 1};
var c: @Vector(2, u16) = b;
const c = @min(a, c);
```
It is becoming increasingly common for distributions to want to enable
PIE for all binaries and zig currently does not provide any way to do
so aside from patching the build.zig.
Man page for posix lists EMFILE, man page for linux ENFILE.
Also posix says "The mmap() function adds an extra reference to the file
associated with the file descriptor fildes which is not removed by a
subsequent close() on that file descriptor. This reference is removed
when there are no more mappings to the file."
It sounds counter-intuitive, that a process limit but no system limit can
be exceeeded.
As far as I understand, fildes is only used for file descriptor backed mmaps.
With this change, `break` and `break :blk` will fill the result location
with `.void_value`, ensuring that the value will be type checked.
The same will happen for a for loop that contains no `break`s in it's body.
Closes https://github.com/ziglang/zig/issues/14686.
- Fixes the first few code units of the name being omitted (it was using `@sizeOf(FILE_NAME_INFO)` as the start of the name bytes, but that includes the length of the dummy [1]u16 field and padding; instead the start should be the offset of the dummy [1]u16 field)
- Replaces kernel32.GetFileInformationByHandleEx call with ntdll.NtQueryInformationFile
+ Contributes towards #1840
- Checks that the handle is a named pipe first before querying and checking the name, which is a much faster call than NtQueryInformationFile (this was about a 10x speedup in my probably-not-so-good/take-it-with-a-grain-of-salt benchmarking)
Multiple processes can sit waiting for the exclusive lock at the same
time, so we want to recheck whether it needs to be updated whenever
we get an exclusive lock.
This also fixes a race condition between one process truncating the
cache file and another process reading it without atomic locking.
In Linux when interacting with the virtual file system when writing
in invalid value to a file the OS will return errno 22 (INVAL).
Instead of triggering an unreachable, this change now returns a
newly introduced error.InvalidArgument.
In Linux when writing to various files in the virtual file system,
for example /sys/fs/cgroup, if you write an invalid value to a file
you'll get errno 16.
This change allows for these specific cases to be caught instead of
being lumped together in UnexpectedError.
This introduces a new builtin function that compiles down to something that results in an illegal instruction exception/interrupt.
It can be used to exit a program abnormally.
This implements the builtin for all backends.
If I could mark a builtin function as cold, I would mark @setCold as cold.
We have run out of `Zir.Inst.Tag`s so I had to move a tag from Zir.Inst.Tag to
Zir.Inst.Extended. This is because a new noreturn builtin will be added and
noreturn builtins cannot be part of Inst.Tag:
```
/// `noreturn` instructions may not go here; they must be part of the main `Tag` enum.
pub const Extended = enum(u16) {
```
Here's another reason I went for @setCold:
```
$ git grep setRuntimeSafety | wc -l
322
$ git grep setCold | wc -l
79
$ git grep setEvalBranchQuota | wc -l
82
```
This also simply removes @setCold from Autodoc and the docs frontend because
as far as I could understand it, builtins represented using Zir extended
instructions are not yet supported because I couldn't find
@setStackAlign or @setFloatMode there, either.
Give +x to the .wasm file if it is an executable and the OS is WASI.
Some systems may be configured to execute such binaries directly. Even
if that is not the case, it means we will get "exec format error" when
trying to run it rather than "access denied", and then can react to that
in the same way as trying to run an ELF file from a foreign CPU
architecture.
This is part of the strategy to unify RunStep and EmulatableRunStep.
- Use .flash as the default address space for functions on AVR
- Return .flash as the address space for function pointers on AVR
without explicit address space
Previously, this API had pid, to be used on POSIX systems, and handle,
to be used on Windows.
This commit unifies the API, defining an Id type that is either the pid
or the HANDLE depending on the target OS.
This commit also prepares for the future by allowing one to import via
`std.process.Child` which is the fully qualified namespace that I intend
to migrate to in the future.
Also add `std.fs.has_executable_bit` for doing conditional compilation.
This adds the linux syscalls for chmod and fchmodat, as well as the
extern libc function declarations.
Only `fchmodat` is added to `std.os`, and it is not yet added to std.fs.
Make the Keccak permutation public, as it's useful for more than
SHA-3 (kMAC, SHAKE, TurboSHAKE, TupleHash, etc).
Our Keccak implementation was accepting f as a comptime parameter,
but always used 64-bit words and 200 byte states, so it actually
didn't work with anything besides f=1600.
That has been fixed. The ability to use reduced-round versions
was also added in order to support M14 and K12.
The state was constantly converted back and forth between bytes
and words, even though only a part of the state is actually used
for absorbing and squeezing bytes. It was changed to something
similar to the other permutations we have, so we can avoid extra
copies, and eventually add vectorized implementations.
In addition, the SHAKE extendable output function (XOF) was
added (SHAKE128, SHAKE256). It is required by newer schemes,
such as the Kyber post-quantum key exchange mechanism, whose
implementation is currently blocked by SHAKE missing from our
standard library.
Breaking change: `Keccak_256` and `Keccak_512` were renamed to
`Keccak256` and `Keccak512` for consistency with all other
hash functions.
When there is a diamond dependency, reuse a *Module instead of creating
a redundant one using the same build.zig file. Otherwise, the compile
error "file exists in multiple modules" would occur.
Partial revert of 1e963053d0. Now that
lldb has a new method for accessing this data that does not require
manual updating, there is no longer any reason to share this data.
And make it not do any installation, only objcopying. We already have
install steps for doing installation.
This commit also makes ObjCopyStep properly integrate with caching.
Many `Type`s can correspond to the same `CType`, so this reduces the
number of used locals by 27760 when compiling only-c.
Also, disabled some tests that were only passing by accident and
shouldn't really be considered working.
I believe the reason we had that check in the first place was because for both `pthread_setname_np` and `pthread_getname_np` man says:
```
CONFORMING TO
These functions are nonstandard GNU extensions; hence the suffix "_np" (nonportable) in the names.
```
However, this `is_gnu` check was never consistently applied; it was
missing in `setName` in the Linux case. It is also missing on all other call sites for other platforms (macOS, iOS, et al.).
Though, that could be because it may only apply to Linux.
I think for a best-effort approach it is okay to drop this. It's probably less non-standard than man makes it out to be.
I noticed a comment saying that the intent of a code's author was unclear.
What happened is that the author forgot to put the check for whether the
thread is the calling thread (`self.getHandle() ==
std.c.pthread_self()`) in the `if (use_pthreads)`.
If the thread is the calling thread, we use `prctl` to set or get the
thread's name and it does not take a thread id because it knows the id
of the thread we're calling `getName` or `setName` from.
I have found a source saying that using `pthread_setname_np` on either the calling thread
or any other thread by thread id would work too (so we don't need to
call `prctl`) but I was not sure if that is the case on all systems
so we keep using `pthread_setname_np` if we have a
specific thread that is not the thread we're calling from, and `prctl`
otherwise.
- Combine mulXi3 routines for follow-up cleanup.
- DRY up Dwords and Twords
- rename both to HalveInt and use instance
* Justification: Not all processors have word size 32 bit.
* remove test file from CMakeLists
* DRY things.
After this, the last MSVC warnings are in behavior/bugs/529.zig:
behavior.c(37971): warning C4133: 'function': incompatible types - from 'A__8479 *' to 'A__8474 *'
behavior.c(37974): warning C4133: 'function': incompatible types - from 'A__8480 *' to 'A__8474 *'
* Support always_tail and never_tail/never_inline with a comptime callee using clang
* Support never_inline using gcc
* Support never_inline using msvc
Unfortunately, can't enable behavior tests because of the conditional support.
And additionally support writing files to source files. This means a
custom build step in zig's own build.zig is no longer needed for copying
zig.h because it is handled by WriteFileStep.
* Implement @fieldParentPtr on a union
* Refactor field access to ensure that it is handled consistently
* Remove `renderTypecast` as it is now behaves the same as `renderType`
Forcing the key to be of the same type as the sorted items used during
the search is a valid use case.
There, however, exists some cases where the key and the items are of
heterogeneous types, like searching for a code point in ordered ranges
of code points:
```zig
const CodePoint = u21;
const CodePointRange = [2]CodePoint;
const valid_ranges = &[_]CodePointRange{
// an ordered array of ranges
};
fn orderCodePointAndRange(
context: void,
code_point: CodePoint,
range: CodePointRange
) std.math.Order {
_ = context;
if (code_point < range[0]) {
return .lt;
}
if (code_point > range[1]) {
return .gt;
}
return .eq;
}
fn isValidCodePoint(code_point: CodePoint) bool {
return std.sort.binarySearch(
CodePointRange,
code_point,
valid_ranges,
void,
orderCodePointAndRange
) != null;
}
```
It is so expected that `std.sort.binarySearch` should therefore support
both homogeneous and heterogeneous keys.
This requires manual defines before C99 which may not have stdint.h.
Also have update-zig1 leave a copy of lib/zig.h in stage1/zig.h, which
allows lib/zig.h to be updated without needing to update zig1.wasm.
Note that since the object already existed with the exact same contents,
this completely avoids repo bloat due to zig.h changes.
- Fix assertion failure if AstGen failed on a multi-module file
- Cap number of per-error reference notes and total multi-module errors each at 5
- Always put "root of package" reference notes first
Resolves: #14499
lld accepts both syntaxes, but we were rejecting (and, before
3f7e9ff597, ignoring) the former.
In particular, "cargo-zigbuild" was broken since Rust
unconditionally adds "-znoexecstack" (not "-z noexecstack")
on non-Windows platforms.
Co-authored-by: Andrew Kelley <andrew@ziglang.org>
Previously `executeSequenceRingBuffer()` would not verify the offset
against the number of bytes already decoded, so it would happily copy
garbage bytes rather than return an error before the window was filled.
To fix this a new `written_count` is added to the decode state that
tracks the total number of bytes decoded.
Justifications
- compiler_rt base routines are almost finished, so make 1:1
mapping of code and documentation.
- Make adjustments to code + documentation simpler to prevent technical or
documentation debt.
This reverts commit 772a0eb68a, reversing
changes made to 0bb178bbb2.
This needs a rebase against master branch - it has build-breaking merge
conflicts. I also added a "changes requested" review on the original
pull request.
* Scan from line start when finding tag in tokenizer
This resolves a crash that can occur for invalid bytes like carriage
returns that are valid characters when not parsed from within literals.
There are potentially other edge cases this could resolve as well, as
the calling code for this function didn't account for any potential
'pending_invalid_tokens' that could be queued up by the tokenizer from
within another state.
* Fix carriage return crash in multiline string
Follow the guidance of #38:
> However CR directly before NL is interpreted as only a newline and not part of the multiline string. zig fmt will delete the CR.
Zig fmt already had code for deleting carriage returns, but would still
crash - now it no longer does so. Carriage returns encountered before
line-feeds are now appropriately removed on program compilation as well.
* Only accept carriage returns before line feeds
Previous commit was much less strict about this, this more closely
matches the desired spec of only allow CR characters in a CRLF pair, but
not otherwise.
* Fix CR being rejected when used as whitespace
Missed this comment from ziglang/zig-spec#83:
> CR used as whitespace, whether directly preceding NL or stray, is still unambiguously whitespace. It is accepted by the grammar and replaced by the canonical whitespace by zig fmt.
* Add tests for carriage return handling
One of the main points of for loops is that you can safety check the
length once, before entering the loop, and then safely assume that every
element inside the loop is in bounds.
In master branch, the safety checks are incorrectly intact even inside
for loops. This commit fixes it. It's especially nice with multi-object
loops because the number of elided checks is N * M where N is how many
iterations and M is how many objects.
This also makes another breaking change to for loops: in order to
capture a pointer of an element, one must take the address of array
values. This simplifies a lot of things, and makes more sense than how
it was before semantically.
It is still legal to use a for loop on an array value if the
corresponding element capture is byval instead of byref.
This strategy uses pointer arithmetic to iterate through the loop. This
has a problem, however, which is tuples. AstGen does not know whether a
given indexable is a tuple or can be iterated based on contiguous
memory. Tuples unlike other indexables cannot be represented as a
many-item pointer that is incremented as the loop counter.
So, after this commit, I will modify AstGen back closer to how @vexu had
it before, using a counter and array element access.
* Allow unbounded looping.
* Lower by incrementing raw pointers for each iterable rather than
incrementing a single index variable. This elides safety checks
without any analysis required thanks to the length assertion and
lowers to decent machine code even in debug builds.
- An "end" value is selected, prioritizing a counter if possible,
falling back to a runtime calculation of ptr+len on a slice input.
* Specialize on the pattern `0..`, avoiding an unnecessary subtraction
instruction being emitted.
* Add the `for_check_lens` ZIR instruction.
RunStep is supposed to auto-detect whether the intend is for
side-effects or for producing an output file. The auto-detection logic
was incorrect, and this commit fixes it.
I tested this manually locally. Automated testing will require a more
significant investment in the test harness, which I will work on in a
future enhancement.
closes#14666
As of Clang 16.x, it will by default read extra flags from /etc/clang.
I'm sure the person who implemented this means well, but they have a lot
to learn about abstractions and where the appropriate boundaries between
them are. The road to hell is paved with good intentions. Fortunately it
can be disabled.
Currently `zig cc`, when confronted with a linker argument it does
not understand, skips the flag and emits a warning.
This has been causing headaches for people that build third-party
software (including me). Zig seemingly builds and links the final
executable, only to segfault when running it.
If there are linker warnings when compiling software, the first thing we
have to do is add support for ones linker is complaining, and only then
go file issues. If zig "successfully" (i.e. status code = 0) compiles a
binary, there is instead a tendency to blaim "zig doing something
weird". (I am guilty of this.) In my experience, adding the unsupported
arguments has been quite easy; see #11679, #11875, #11874 for recent
examples.
With the current ones (+ prerequisites below) I was able to build all of
the CGo programs that I am encountering at $dayjob. CGo is a reasonable
example, because it is exercising the unusual linker args quite a bit.
Prerequisites: #11614 and #11863.
Tested by: created a "hello world" C file and compiled with:
zig cc -v main.c -Wl,--sort-section=name -o main
... and verified the `--sort-section=name` is passed to ld.lld.
Refs https://github.com/zigchroot/zig-chroot/issues/1
Both original Markdown and CommonMark require at least one space or tab
between the list marker and any following content. Following this allows
starting a line with a negative number or one with a decimal point.
When the following is specified
```
$ zig build-obj -fcompiler-rt example.zig
```
the resulting relocatable object file will have the compiler-rt
unconditionally embedded inside.
```
$ nm example.o
...
0000000012345678 W __truncsfhf2
...
```
This is just a simple/hacky feature to report the source of a linking
error. I found this helpful in fixing-up some of my libs when recently
switching from an x86_64 to aarch64 device, so thought it might be
useful to others a well before zld has a fully featured error reporting
system.
Some examples using {#syntax_block|zig|...#} either have a valid syntax
or it is easy to make the syntax valid. Update these example to use
{#code_begin|syntax|...#}.
Remove extra whitespace in the error_union_parsing_u64.zig example.
Replace size_t with usize in the call_malloc_from_zig.zig example.
This makes it so that when there is a tree of std.Build objects, only
one zig-cache is used (the top-level application) instead of polluting
package directories with zig-cache folders.
* Use std.Build.Cache.Directory instead of a string for storing the
cache roots and build roots.
* Set up a std.Build.Cache in build_runner.zig and use it in
std.Build.RunStep for avoiding redundant work.
Now that this API is used by the build system, these debug logs are
problematic because build scripts run in debug mode, making these logs
noisy output.
This provides file path as a command line argument to the command being
run, and returns a FileSource which can be used as inputs to other APIs
throughout the build system.
Unfortunately, it is implemented by pooping a ton of temporary files
into zig-cache/tmp for the time being. I think one of the very next
improvements to the build system should be moving the compiler's cache
system to the standard library and using it in the build system. I had a
look at the dependencies and it is already pretty untangled.
Deprecate CompileStep.run. The problem with this function is that it
does the RunStep with the same build.zig context as the CompileStep, but
this is not desirable when running an executable that is provided by a
dependency package. Instead, users should use `b.addRunArtifact`.
This has the additional benefit of conforming to the existing naming
conventions.
Additionally, support enum literals in config header options values.
Ascon has been selected as new standard for lightweight cryptography
in the NIST Lightweight Cryptography competition.
Ascon won over Gimli and Xoodoo.
The permutation is unlikely to change. However, NIST may tweak
the constructions (XOF, hash, authenticated encryption) before
standardizing them. For that reason, implementations of those
are better maintained outside the standard library for now.
In fact, we already had an Ascon implementation in Zig:
`std.crypto.aead.isap` is based on it. While the implementation was
here, there was no public API to access it directly.
So:
- The Ascon permutation is now available as `std.crypto.core.Ascon`,
with everything needed to use it in AEADs and other Ascon-based
constructions
- The ISAP implementation now uses std.crypto.core.Ascon instead of
keeping a private copy
- The default CSPRNG replaces Xoodoo with Ascon. And instead of an
ad-hoc construction, it's using the XOFa mode of the NIST submission.
This reverts commit abc9530a88.
This patch implies that the idiomatic Zig way of handling anytype
parameter is to write a bunch of boilerplate instead of directly
accessing type information and relying on the compiler to be useful.
I don't want it to be this way.
It is the compiler's job to make useful error messages when the wrong
field of a type info result is accessed, and it is the zig programmer's
job to understand what it means when a compile error points at the field
access of `@typeInfo` (along with the relevant callsites).
One thing that might be useful would be having the compiler be aware of
module boundaries and highlighting the boundaries of them. The first
reference note after crossing a module boundary is likely the most
interesting one.
- Copy and adjust __divmodsi4 tests for __divmoddi4 and __divmodti4.
- Assuming d = a/b does not overflow (MIN/-1) or uses div by 0,
then tmp = (d * b) = (a/b * b) = a does not overflow.
=> Remove wraparound for remainder in applicable routines.
These are unused in LLVM, but in gcc used for example by rl78.
Move test files into common file, add tests for aforementioned routines
and missing ones for __lshrti3.
There can be a maximum of 256 compact encodings per page in compact
unwind info, and we were using `u8` to represent the count which is insufficient.
This commit bumps it to `u9`.
Fixes#13970.
This fix makes test runners resolve package paths relative to the
directory the test runner is in. This means it is not possible to import
a file from outside the file tree root at the directory containing the
test runner.
Breaking API change to std.Build.addConfigHeader. It now uses an options
struct.
Introduce std.Build.CompileStep.installConfigHeader which also accepts
an options struct. This is used to add a generated config file into the
set of installed header files for a particular compilation artifact.
std.Build.ConfigHeaderStep now additionally supports a "blank" style
where a header is generated from scratch. It no longer exposes
`output_dir`. Instead it exposes a FileSource via `output_file`.
It now additionally accepts an `include_path` option which affects the
include path of CompileStep when using the `#include` directive, as well
as affecting the default installation subdirectory for header
installation purposes.
The hash used for the directory to store the generated config file now
includes the contents of the generated file. This fixes possible race
conditions when generating multiple header files simultaneously.
The values hash table is now an array hash map, to preserve order for
the "blank" use case.
I also took the opportunity to remove output_dir from TranslateCStep and
WriteFileStep. This is technically a breaking change, but it was always
naughty to access these fields.
Add ability to generate a c header file from scratch, and then both
compile with it and install it if needed.
Example:
```zig
const avconfig_h = b.addConfigHeader(.{ .path = "libavutil/avconfig.h" }, .generated, .{
.AV_HAVE_BIGENDIAN = 0, // TODO: detect based on target
.AV_HAVE_FAST_UNALIGNED = 1, // TODO: detect based on target
});
lib.addConfigHeader(avconfig_h);
lib.installConfigHeader(avconfig_h);
```
Commit 40f5e5dfc6 ("CLI: introduce -fsingle-threaded/
-fno-single-threaded", 2021-12-01) removed the `--single-threaded`
option, but didn't update all mentions of it.
$ zig version
0.11.0-dev.1568+c9b957c93
$ zig build-exe --help | grep single
-fsingle-threaded Code assumes there is only one thread
-fno-single-threaded Code may not assume there is only one thread
$ zig build-exe --single-threaded foo.zig
error: unrecognized parameter: '--single-threaded'
assumeSentinel was removed and replaced with a compileError, but it's
not pub, so the error message indicates it's private rather than
providing the compileError message.
- In Integer Bit Operations, use zeros consistently and fix the
incorrect function descriptions
- In ucmp, fix the incorrect signedness of the operands
- In Integer Arithmetic, document the signedness of the operands in
div, udiv, mod, umod, udivmod and divmod as it is done in LLVM
documentation
- In Float Conversion, replace "int_to_float" to "int to float" and
"uint_to_float" to "uint to float"
New API introduced: std.Build.addModule
This function exposes a zig module with the given name, which can be
used by packages that depend on this one via std.Build.Dependency.module.
std.Build.Pkg and related functionality is deleted. Every use case has a
straightforward upgrade path using the new Module struct.
std.Build.OptionsStep.getPackage is replaced by
std.Build.OptionsStep.createModule.
std.Build.CompileStep.addPackagePath is replaced by
std.Build.CompileStep.addAnonymousModule.
This partially addresses #14307 by renaming some of the instances of
"package" to "module".
Closes#14278
Underlying fix should have been d7b029995c.
u128 limb sizes are still not fully tested as we are missing compiler-rt
support (__divei4, __modei4 on x86_64). Should be no zig blockers so the
assertion has been removed.
This finishes the work started in #14502 where atoms are owned by the
linker themselves. This now makes debug atoms fully owned by dwarf,
and no information is left stored on the decl.
* improve error message when build manifest file is missing
* update std.zig.Ast to support ZON
* Compilation.AllErrors.Message: make the notes field a const slice
* move build manifest parsing logic into src/Manifest.zig and add more
checks, and make the checks integrate into the standard error
reporting code so that reported errors look sexy
closes#14290
* std.zig.parse is moved to std.zig.Ast.parse
* the new function has an additional parameter that requires passing
Mode.zig or Mode.zon
* moved parser.zig code to Parse.zig
* added parseZon function next to parseRoot function
This commit makes the text more dense with sources, noes the outstanding
audit and adds an explanation of the table before converting everything
to markdown tables.
For the status a checkmark or cross in utf8 encoding are used and the
input and output sizes of all operations are given inclusive comments for
understanding of the routines.
This should document all compiler_rt integer and float routines, but
does not include a documentation of existing ieee floating and math
routines.
this patch is from upstream, to fix -Wdeprecated-non-prototypes issues.
K&R-style has apparently been deprecated since even C89, and C2x will be
repurposing the syntax space. this warning triggers when the change would
affect the meaning of the code.
Unfortunately, due to the Windows equivalent of executable permissions
being a bit tricky, there is follow-up work to be done.
What is done in this commit is the hash modifications. At the fetch
layer, executable bits inside packages are ignored. In the hash
computation layer, executable bit is implemented for POSIX but not yet
for Windows. This means that the hash will not break again in the future
for packages that do not have any executable files, but it will break
for packages that do.
This is a hash-breaking change.
Closes#14308
PR #10610 addressed most of the points from #1854. This
additional paragraph and examples covers the OMISSIONS section
clarifying issues about indexing into non-ASCII strings (whether valid
UTF-8 or not). I think this finally closes#1854.
Several enhancements to the build system. Many breaking changes to the API.
* combine `std.build` and `std.build.Builder` into `std.Build`
* eliminate `setTarget` and `setBuildMode`; use an options struct for `b.addExecutable` and friends
* implement passing options to dependency packages. closes#14285
* rename `LibExeObjStep` to `CompileStep`
* move src.type.CType to std lib, use it from std.Build, this helps with populating config.h files.
Usage of `catch unreachable` in build scripts is completely harmless
because build scripts are always run in Debug mode, however, it sets a
poor example for beginners to learn from.
These declarations are now aliases of their new APIs and marked as
deprecated via doc comments:
* std.build.Builder
* std.build
* std.Build.LibExeObjStep
This is a breaking change that makes the API for creating build
artifacts no longer have any period of time where the target and
optimization mode are not set.
* introduce the concept of maps to user input options, but don't
implement it for command line arg parsing yet.
* remove setPreferredReleaseMode and standardReleaseOptions in favor of
standardOptimizeOption which has a future-proof options parameter.
These functions are currently footgunny when working with pointers to
arrays and slices. They just return the stated length of the array/slice
without iterating and looking for the first sentinel, even if the
array/slice is a sentinel terminated type.
From looking at the quite small list of places in the standard
library/compiler that this change breaks existing code, the new code
looks to be more readable in all cases.
The usage of std.mem.span/len was totally unneeded in most of the cases
affected by this breaking change.
We could remove these functions entirely in favor of other existing
functions in std.mem such as std.mem.sliceTo(), but that would be a
somewhat nasty breaking change as std.mem.span() is very widely used for
converting sentinel terminated pointers to slices. It is however not at
all widely used for anything else.
Therefore I think it is better to break these few non-standard and
potentially incorrect usages of these functions now and at some later
time, if deemed worthwhile, finally remove these functions.
If we wait for at least a full release cycle so that everyone adapts to
this change first, updating for the removal could be a simple find and
replace without needing to worry about the semantics.
- cbe: Implement linksection support, to support TLS when not linking libc
- cbe: Support under-aligned variables / struct fields
- cbe: Support packed structs (in the C definition of packed)
- windows: Fix regression with x86 _tls_array
- compiler_rt: Add 128-bit atomics to compiler_rt
- tests: Re-enable threadlocal tests on cbe+windows, and llvm+x86
- tests: Re-enable f80 tests that now pass
- ci: change windows ci to run the CBE behaviour tests with -lc, to match how the compiler is bootstrapped
- update zig1.wasm
This enhances the debugging experience as upon encountering a
breakpoint in a function, all arguments passed as registers have
already been moved to the stack, ready to be inspected by the
debugger.
The system linker shows warning when `msync` is linked instead of `__msync13`:
"warning: reference to compatibility msync(); include <sys/mman.h> for correct reference"
closes#14422
In the "Type Coercion" section, rename:
- Coercion Float to Int => Float to Int
- unions and enums => Unions and Enums
- tuples to arrays => Tuples to Arrays
In the "C Translation CLI" section, move the paragraph inside the p
element.
The current HTML is valid, but, as an example, a paragraph outside a p
element is not handled correctly by the browser Inspect tool.
This function is needed when a library exposes one of its own library
dependency's headers as part of its own public API.
Also, improve error message when a file system error occurs during
install file step.
This includes a breaking change:
std.compress.gzip.GzipStream renamed to
std.compress.gzip.Decompress
This follows the same naming convention as std.compress.xz so that the
stream type can be passed as a comptime parameter.
* add xz to std.compress
* prefer importing std.zig by file name, to reduce reliance on the
standard library being a special case.
* extract some types from inside generic functions. These types are the
same regardless of the generic parameters.
* expose some more types in the std.compress.xz namespace.
* rename xz.stream to xz.decompress
* rename check.Kind to Check
* use std.leb for LEB instead of a redundant implementation
All but 3 callsites of this function in the standard library and
compiler were unnecessary and were removed in faf2fd18.
In this commit, the remaining 3 callsites are removed. One of them
turned out to also be unnecessary and has been replaced by slicing
directly with the length..
The 2 remaining callsites were in the very pointer-math heavy
std/os/linux/vdso.zig code which should perhaps be refactored to better
utilize slices. These 2 callsites are replaced with a plain
@ptrCast([*:0]u8, ptr) though could likely use std.mem.sliceTo() if the
surrounding code was refactored.
In commit 3542dbf0ea (langref: add section numbers) I accidentally
added an extra closing curly bracket at the end of the style element.
Sorry for not validating the HTML file before creating the PR.
Update the genToc funtion to make the name required in the Code node,
and add an additional optional field for the expected error, to use with
test_err, test_safety and obj_err.
Update langref.html.in to ensure all code blocks have a name that
- is unique, so that a doctest can be identified by it
- is descriptive
For test, test_err and test_safefy, ensure that the doctest name starts
with "test_", excluding doctests in the "Zig Test" section and doctests
that are imported by other doctests.
Ensure that the indentation of code_begin and code_end blocks are
consistent.
Fix a typo in pointer_arthemtic.
This fixes a bug in std.net caused during the introduction of
meta.assumeSentinel due to the unfortunate semantics of mem.span()
This leaves only 3 remaining uses of meta.assumeSentinel() in the
standard library, each of which could be a simple @ptrCast([*:0]T, foo)
instead. I think this function should likely be removed.
This avoids a warning that sometimes occurs when providing both a
-target argument that contains a version as well as the
-mmacosx-version-min argument. Zig provides the correct value in both
places, so it doesn't matter which one gets overridden.
Previously, if a source file was referenced from multiple packages, it
just became owned by the first one AstGen happened to reach; this was a
problem, because it could lead to inconsistent behaviour in the compiler
based on a race condition. This could be fixed by just analyzing such
files multiple times - however, it was pointed out by Andrew that it
might make more sense to enforce files being part of at most a single
package. Having a file in multiple packages would not only impact
compile times (due to Sema having to run multiple times on potentially a
lot of code) but is also a confusing anti-pattern which more often than
not is a mistake on the part of the user.
Resolves: #13662
By @Vexu's suggestion, since fetching the name from the parent package
is error-prone and complex, and optimising Package for size isn't really
a priority.
From RFC 1952:
> If FHCRC is set, a CRC16 for the gzip header is present,
> immediately before the compressed data. The CRC16 consists
> of the two least significant bytes of the CRC32 for all
> bytes of the gzip header up to and not including the CRC16.
* no longer repeat -lc on the linker line redundantly
* when using linkLibrary() with a static library, it will now also put
the static library's static library dependencies on the linker line,
recursively.
* refactor out a common pattern to an addFlag function
Changes only pertain arm64 as for x86_64 turns out we are
missing one additional feature which is synthesising unwind info
from only eh frame records, or at least this is what it looks like
when I analysed the output of Apple's ld.
In a syntax_block the code always start on a separate code, expect for
C, JavaScript, Peg and with Zig inline assembly.
Ensure that the code starts on a separate line, even in cases where
there is only one line.
Ensure that the end_syntax_block is always on a separate line and that
the indentation is consistent.
I was testing this with wazero, which defaults to not propagate any env
variables. This ensures we don't try to allocate zero length buffers
when there are no results from either function.
Signed-off-by: Adrian Cole <adrian@tetrate.io>
We do not need more for the purpose of parsing and synthesising
unwind info by the linker. If we ever decide to generate unwind
info for Zig by the compiler, we can re-add packed struct defs
again.
The current compiler ignores the UTF-8 BOM if it is at the start of the
file, and disallows it anywhere else. Document it in the Source Encoding
section.
Add the new addrspace keyword in the Keyword Reference section, without
documentation.
Move the linksection keyword in order to keep the keywords list sorted.
After this change, the system will be inspected for root certificates
only upon the first https request that actually occurs. This makes the
compiler no longer do SSL certificate scanning when running `zig build`
if no network requests are made.
I originally started monkeying with this because std.mem.zeroes doesn't support sentinel-terminated const slices even with defaults in 0.10.x.
I see that std.mem.zeroes was modified in #13256 to allow setting these slices to "".
That got me partway to where I wanted, but there was still an issue fields whose types are structs, they wouldn't get their defaults.
So when iterating struct fields looking for default values, when there is no default value and the type is .Struct, it will delegate to a call to zeroInit.
* Initialize struct fields in zeroInit exactly once
In my changes, similar to the previous implementation, the priority order for fields being initialized is:
1. If the `init` argument is a tuple, the nth element corresponding to the nth field of the struct.
2. Otherwise, if the `init` argument is not a tuple, try to find a matching field name on `init` and use that field.
3. Is the field has a default value, initalize with that value.
4. Fall back to what the field would have been initialized to via a recursive call to `std.mem.zeroInit`.
But instead of initializing a default instance of the struct and then running multiple passes over it, the init method is chosen per-field and each field is initialized exactly once.
This commit adds support for "-x language" for a couple of hand-picked
supported languages. There is no reason the list of supported languages
to not grow (e.g. add "c-header"), but I'd like to keep it small at the
start.
Alternative 1
-------------
I first tried to add a new type "Language", and then add that to the
`CSourceFile`. But oh boy what a change it turns out to be. So I am
keeping myself tied to FileExt and see what you folks think.
Alternative 2
-------------
I tried adding `Language: ?[]const u8` to `CSourceFile`. However, the
language/ext, whatever we want to call it, still needs to be interpreted
in the main loop: one kind of handling for source files, other kind of
handling for everything else.
Test case
---------
*standalone.c*
#include <iostream>
int main() {
std::cout << "elho\n";
}
Compile and run:
$ ./zig run -x c++ -lc++ standalone.c
elho
$ ./zig c++ -x c++ standalone.c -o standalone && ./standalone
elho
Fixes#10915
When any of the object files reference the __heap_end symbol, we will
create it as a synthetic symbol. The symbol only exists within the
linker and will not be emit within the binary as it's solely used for
relocations. The symbol represents where the heap ends, so allocators
can determine whether to allocate a new page or not.
The deallocations of the process arguments are unnecessary, since the
memory will be deallocated by arena.deinit().
The deallocations are incorrect, since ArgIterator.next() returns a
slice pointing to the iterator's internal buffer, that should be
deallocated with args_it.deinit().
When any object files provides an undefined reference to the __heap_base
symbol, we create a new defined symbol for it. During setupMemory we
set the virtual address of this symbol so it can be used for relocations.
This symbol represents where the heap starts and allocators can use
this value for its allocations when it needs to determine where the heap
lives.
This implements the `__wasm_call_ctors` symbol. This symbol is
automatically referenced by libc to initialize its constructors.
We first retrieve all constructors from each object file, and then
create a function body that calls each constructor based on its
priority. Constructors are not allowed to have any parameters, but are
allowed to have a return type. When a return type does exist, we simply
drop its value from the stack after calling the constructor to ensure
we pass the stack validator.
When emitting errors for undefined symbols, rather than
unconditionally always using the name from an import, we must verify
it's a symbol type that could have such an import. e.g. undefined
data symbols do not have a corresponding import. For this reason we
must use the regular name.
During symbol resolution when both symbols are undefined, we must
discard the new symbol with a reference to the existing symbol. This
ensures the original symbol remains undefined. This fixes symbol
resolution when linking with WASI-libC.
* Changed the interface to align with the new allocator interface.
* Fixed bug where not enough memory was allocated for the header or to
align the pointer.
The `zig build` command now makes `@import("@dependencies")` available
to the build runner package. It contains all the dependencies in a
generated file that looks something like this:
```zig
pub const imports = struct {
pub const foo = @import("foo");
pub const @"bar.baz" = @import("bar.baz");
};
pub const build_root = struct {
pub const foo = "<path>";
pub const @"bar.baz" = "<path>";
};
```
The build runner exports this import so that `std.build.Builder` can
access it. `std.build.Builder` uses it to implement the new `dependency`
function which can be used like so:
```zig
const libz_dep = b.dependency("libz", .{});
const libmp3lame_dep = b.dependency("libmp3lame", .{});
// ...
lib.linkLibrary(libz_dep.artifact("z"));
lib.linkLibrary(libmp3lame_dep.artifact("mp3lame"));
```
The `dependency` function calls the build.zig file of the dependency as
a child Builder, and then can be ransacked for its build steps via the
`artifact` function.
This commit also renames `dependency.id` to `dependency.name` in the
`build.zig.ini` file.
This allows setting a custom buffer size. In this case I wanted it
because using a buffer size large enough to fit a TLS ciphertext record
elides a memcpy().
This commit also adds `readAtLeast` to the Reader interface.
Some notable changes:
- `ENOENT` is returned instead of `ENOTCAPABLE` when a path has not
be pre-opened (https://github.com/WebAssembly/wasi-libc/pull/370)
- `fd_readdir()`: some implementations may not set the inode number,
so an additional call to `fstatat()` is now done in order to get it
when that happens.
This API converts a config.h.in file into config.h. This is useful when
introducing a build.zig file to an existing C/C++ project that is
configured with autotools or cmake.
The cmake syntax is not implemented yet.
Currently llvm-linkage mode (static vs dynamic) decides linkage mode
for system libstdc++/libc++ .
A previous commit only tested static mode for *BSD and netbsd was
reported to not work in dynamic mode.
We now special-case freebsd, openbsd, netbsd and dragonfly for dynamic
linking too.
closes#14204
In order to add tests for this I need to implement an HTTP server in the
standard library (#910) so that's probably the next thing I'll do.
A followup to 5bd69c655d .
For compatibility purposes clang defines __GNUC__. This broke logic
where clang14 __has_builtin returns false but `|| defined(__GNUC__)`
made it true.
CI did not catch this because it uses clang15. However, bootstrapping
with latest apple compiler is clang14. We also want to support older
clang.
RFC 9110 section 15:
Values outside the range 100..599 are invalid. Implementations often use
three-digit integer values outside of that range (i.e., 600..999) for
internal communication of non-HTTP status (e.g., library errors). A
client that receives a response with an invalid status code SHOULD
process the response as if it had a 5xx (Server Error) status code.
The resolvePosix and resolveWindows routines changed behaviour in an
earlier commit so that the return value is not always an absolute path.
That caused the relativePosix and relativeWindows to return a relative
path that is not correct.
The change in behaviour mentioned above would cause a local cache-dir to
be created in the wrong directory when --cache-dir was specified for a
build.
@bitCast from integer NaN representation to float NaN resulted in
changed bits in float. This only happened with signaled NaN.
- added test for signaled NaN
- added tests for quiet NaN (for completeness)
closes#14198
- Revert the addition of CLANG_SYSTEM_LIBARIES and LLVM_SYSTEM_LIBRARIES
- Change addCMakeLibraryList to parse non-absolute path .lib dependencies as system libraries
- Remove ZIG_ENABLE_ZSTD in favour of allowing ZIG_STATIC_ZSTD to be toggled off explicitly when ZIG_STATIC is on
- Remove ZIG_ENABLE_LIBCPP (now unused)
Using zig cc with CMake on Windows was failing during compiler
detection. -nostdinc was causing the crt not to be linked, and Coff/lld.zig
assumed that wWinMainCRTStartup would be present in this case.
-nostdlib did not prevent the default behaviour of linking libc++ when
zig c++ was used. This caused libc++ to be built when CMake ran
ABI detection using zig c++, which fails as libcxxabi cannot compile
under MSVC.
- Change the behaviour of COFF -nostdinc to set /entry to the function that the
default CRT method for the specified subsystem would have called.
- Fix -ENTRY being passed twice if it was specified explicitly and -nostdlib was present.
- Add support for /pdb, /version, /implib, and /subsystem as linker args (passed by CMake)
- Remove -Ddisable-zstd, no longer needed
- Add -Ddisable-libcpp for use when bootstrapping on msvc
- add support for passing through .def files to the linker,
required for building libLTO.dll in LLVM
- fixup libcpp linking conditionals
- add option to skip linking zstd for use in bootstrapping (when
building against an LLVM with LLVM_ENABLE_ZSTD=OFF)
* std.http.Status.Class: add a "nonstandard" enum tag. Instead of
having `class` return an optional value, it can potentially return
nonstandard.
* extract out std.http.Client.Connection from std.http.Client.Request
- this code abstracts over plain/TLS only
- this is the type that will potentially be stored in a client's LRU
connection map
* introduce two-staged HTTP header parsing
- API users can rely on a heap-allocated buffer with a maximum limit,
which defaults to 16 KB, or they can provide a static buffer that
is borrowed by the Request instance.
- The entire HTTP header is buffered because there are strings in
there and they must be accessed later, such as with the case of
HTTP redirects.
- When buffering the HTTP header, the parser only looks for the
\r\n\r\n pattern. Further validation is done later.
- After the full HTTP header is buffered, it is parsed into
components such as Content-Length and Location.
* HTTP redirects are handled, with a maximum redirect count option that
defaults to 3.
- Connection: close is always used for now; implementing keep-alive
connections and an LRU connection pool in std.http.Client is a task
for another day.
see #2007
windows: add RtlCaptureContext, RtlLookupFunctionEntry, RtlVirtualUnwind and supporting types
windows: fix alignment of CONTEXT structs to match winnt.h as required by RtlCaptureContext (fxsave instr)
windows aarch64: fix __chkstk being defined twice if libc is not linked on msvc
Co-authored-by: Jakub Konka <kubkon@jakubkonka.com>
- Add an assert that an exclusive lock is help to writeManifest
- Only call writeManifest in updateCObject if an exclusive lock is held
- cache: fixup test to verify hits don't take an exclusive lock, instead of writing the manifest
Before, --color on would affect colored compile error printing but not
affect terminal progress bar printing. It was intended for this option
to affect both; now it does.
This causes a failure when building the language reference, which
contains code for parsing terminal output and rendering HTML. Now it
must be expanded to handle 'K' and 'D' codes to simulate a terminal
cursor moving, and the CI will fail until that capability is added in a
later commit of this branch.
I extracted this change from #13560 so that the idea is not lost but we
can solve this issue separately.
This is simply a small convenience wrapper around
'meta.fieldInfo(T, .field).type'. However, this operation is common
enough that it makes sense to have its own function for.
When outputting the names section, we should output the actual symbol
name rather than the import name. This makes sure that symbols with
an explicit name set have the correct name but retain the import name
too.
We also now correctly mangle the name of an extern function with an
explicit library name. This ensures that functions that have a
different library name, but the same import/function name, can be
resolved correctly with other modules and don't resolve to the
same symbol.
Although RFC 8446 states:
> Each party MUST send a "close_notify" alert before closing its write
> side of the connection
In practice many servers do not do this. Also in practice, truncation
attacks are thwarted at the application layer by comparing the amount of
bytes received with the amount expected via the HTTP headers.
This allows for a more optimal std.crypto.tlcsprng codepath.
Without it a an "incorrect alignment" panic is triggered from
crypto.tlcsprng which aligns a threadlocal but it's actually
not aligned, thus detected by the safety check.
It appears that LLVM-IR does attribute the storage with alignment
but it is ultimately not respected in the final binary for netbsd
and dragonfly.
- per darwin-xnu source, fcntl F_GETPATH will return ENOSPC when path
exceeds either user-supplied buffer or system MAXPATHLEN
- macOS does not document this (and other) possible errno values
- first convert to lower-case then perform mappings
- effect: freebsd, netbsd and openbsd convert arch `amd64` to `x86_64`
- effect: convert `arm64` to `aarch64`
Previously, the code only checked Common Name, leading to unable to
validate valid certificates which relied on the subject_alt_name
extension for host name verification.
This commit also adds rsa_pss_rsae_* back to the signature algorithms
list in the ClientHello.
It appears to be implemented incorrectly in the wild and causes the read
connection to be closed even though that is a direct violation of RFC
8446 Section 6.1.
The writeEnd function variants are still there, ready to be used.
This commit adds `writeEnd` and `writeAllEnd` in order to send data and
also notify the server that there will be no more data written.
Unfortunately, it seems most TLS implementations in the wild get this
wrong and immediately close the socket when they see a close_notify,
rather than only ending the data stream on the application layer.
This commit introduces tls.Decoder and then uses it in tls.Client. The
purpose is to make it difficult to introduce vulnerabilities in the
parsing code. With this abstraction in place, bugs in the TLS
implementation will trip checks in the decoder, regardless of the actual
length of packets sent by the other party, so that we can have
confidence when using ReleaseFast builds.
The code we are borrowing from https://github.com/shiguredo/tls13-zig
requires an Allocator for doing RSA certificate verification. As a
stopgap measure, this commit uses a FixedBufferAllocator to avoid heap
allocation for these functions.
Thank you to @naoki9911 for providing this great resource which has been
extremely helpful for me when working on this standard library TLS
implementation. Until Zig has std.crypto.rsa officially, we will borrow
this implementation of RSA. 🙏
This is a streaming HTTP header parser. All it currently does is detect
the end of headers. This will be a non-allocating parser where one can
bring supply their own buffer if they want to handle custom headers.
This commit also improves std.http.Client to not return the HTTP headers
with the read functions.
Here's what I landed on for the TLS client. It's 16896 bytes
(max_ciphertext_record_len is 16640). I believe this is the theoretical
minimum size, give or take a few bytes.
These constraints are satisfied:
* a call to the readvAdvanced() function makes at most one call to the
underlying readv function
* iovecs are provided by the API, and used by the implementation for
underlying readv() calls to the socket
* the theoretical minimum number of memcpy() calls are issued in all
circumstances
* decryption is only performed once for any given TLS record
* large read buffers are fully exploited
This is accomplished by using the partial read buffer to storing both
cleartext and ciphertext.
The read function has been renamed to readAdvanced since it has slightly
different semantics than typical read functions, specifically regarding
the end-of-file. A higher level read function is implemented on top.
Now, API users may pass small buffers to the read function and
everything will work fine. This is done by re-decrypting the same
ciphertext record with each call to read() until the record is finished
being transmitted.
If the buffer supplied to read() is large enough, then any given
ciphertext record will only be decrypted once, since it decrypts
directly to the read() buffer and therefore does not need any memcpy. On
the other hand, if the buffer supplied to read() is small, then the
ciphertext is decrypted into a stack buffer, a subset is copied to the
read() buffer, and then the entire ciphertext record is saved for the
next call to read().
Only a little bit of generalized logic for DER encoding is needed and so
it can live inside the Certificate namespace.
This commit removes the generic "parse object id" function which is no
longer used in favor of more specific, smaller sets of object ids used
with ComptimeStringMap.
When scanning the file system for root certificates, expired
certificates are skipped and therefore not used for verification in TLS
sessions. There is only this one check, however, so a long-running
server will need to periodically rescan for a new Certificate.Bundle
and strategically start using it for new sessions. In this commit I made
the judgement call that applications would like to opt-in to root
certificate rescanning at a point in time that makes sense for that
application, as opposed to having the system clock potentially start
causing connections to fail.
Certificate verification checks the subject only, as opposed to both the
subject and the issuer. The idea is that the trust chain analysis will
always check the subject, leading to every certificate in the chain's
validity being checked exactly once, with the root certificate's
validity checked upon scanning.
Furthermore, this commit adjusts the scanning logic to fully parse
certificates, even though only the subject is technically needed. This
allows relying on parsing to succeed later on.
- fixup 128-bit atomics test to only run on x86_64
- add truncation test for 128-bit types, including non power of two targets (there was a bug with broken non-power-of-two truncation in the cbe)
- add 128-bit binary not test (covers another bug fixed in the cbe)
Rather than checking for function pointers during the writing phase,
we now create a synethtic symbol when a new link job has started.
This means the symbol can correctly be resolved during link time
with the indirect function table from other object files, ensuring
we are properly performing relocations and our binary writer is now
unaware of any of its logic and simply emits the table according to
the symbol such as any other symbols.
Adds a linker test case for each possible export case. This means
one where no exports are done (i.e. no flags set), when the -dynamic
flag is set, and finally when --export=<value> flag(s) are set.
Adds support for both the `-rdynamic` and the `--export=<value>`
flags. Support is added to both the incremental linker as well as
the traditional linker (zld).
- Add cpuid / getXCR0 functions for the cbe to use instead of asm blocks
- Don't cast between 128 bit types during truncation
- Fixup truncation to use functions for shifts / adds
- Fixup float casts for undefined values
- Add test for 128 bit integer truncation
- Fix zig_clz_u128 not respecting the bits argument. This was crashing the compile-rt addxf3 tests with the cbe
- Instead of redering a negation for negative 128 bit int literals, render the literal as twos complement. This allows
rendering int representations of floats correctly (specifically f80).
- Add .StaticInitializer to ValueRenderLocation to indicate that the emitted values
must be constant expressions (no function calls, struct casting).
- Add new path for special float types (nan, inf) that works in constant expressions
- Implement windows.teb() using a syscall for .stage2_c because x64 MSVC
doesn't support any kind of inline asm
- Map the __builtin_nan(f|l)? functions to nan(f|l)? on msvc
- MSVC throws C2099 when initializing a struct with cast syntax
in a global initializer. Added zig_as_init_ to handle this case,
and generate it only in static initializers for > 64 bit ints.
- Change float initialization to emit the integer representation
in global initializers to avoid C2099 caused by calling nan.
- Uses zig_as/zig_lo as necessary when int casting to support !zig_has_int128
- Remove redundant cast if the type is the same
- Use zig_lo when truncating > 64 bits
MSVC can't explicitly cast a struct to a typedef of itself (ie. f128 to i128). Added a
set of macros to handle float casting, and to not produce a cast for this specific
case on MSVC. A better approach would probably be to know if the cast is redundant
and not do it.
- Forward declare int builtins, so the definitions aren't assumed incorrectly
- Add define to handle MSVC not support static const in function parameter array lengths
- Fixup several spots where int128 support was assumed.
- Support zig_align
- Support zig_export
- Stub out some missing non-builtin functions
- Added StringLiteral to automatically split string literals when they get to 16380 in size,
which is the maxmimum pre-concatenation string literal size on MSVC.
Previously, the implementation would essentially check `startsWith` instead of `eql` (e.g. it would return true for `.exec` because it erroneously 'matched' `.exe`).
Follow up to #13993
In sponge-based constructions, the block size is not the same as
the state size. For practical purposes, it's the same as the rate.
Size this is a constant for a given type, we don't need to keep
a copy of that value in the state itself. Just use the constant
directly. This saves some bytes and may even be slightly faster.
More importantly:
Fixes#14128
Move the parsing of Root from the parse function to the Parser.parseRoot
method.
Add an empty line between each rule, for consistency with grammar.y.
Ensure that normal doc-comments are always on the top, for consistency.
Add extract-grammar.zig to the tools directory; it is used to extract
the PEG grammar from parse.zig, so that it can be compared with
grammar.y.
The HKDF extract function uses HMAC under the hood, but requiring
applications to directly use HMAC functions reduces clarity and
feels like the wrong abstraction.
So, in order to get the PRK length, add a `prk_length` constant
that applications can use directly.
Also add an `extractInit()` function for cases where the keying
material has to be provided as multiple chunks.
See also commit 7287c7482a where
LinearFifo is removed from BufferedWriter
Basically, LinearFifo does extra work via copying bytes and increasing
an offset.
This hack was initially introduced as we would export all symbols
unconditionally, including non-function definitions. This would cause
an error from the Wasmtime runtime engine, which this flag would
suppress. As we now properly export symbols, this flag is no longer
needed and any user running into this error can manually include it.
This commit also adds the `--import-symbols` ability to build.zig
When we're compiling compiler_rt for any WebAssembly target, we do
not want to expose all the compiler-rt functions to the host runtime.
By setting the visibility of all exports to `hidden`, we allow the
linker to resolve the symbols during linktime, while not expose the
functions to the host runtime. This also means the linker can
properly garbage collect any compiler-rt function that does not get
resolved. The symbol visibility for all target remains the same as
before: `default`.
No longer automatically append the `--export` flag for each exported
function unconditionally. This was essentially a hack to prevent
binary bloat caused by compiler-rt symbols being always included in
the final binary as they were exported and therefore not garbage-
collected. This is no longer needed as we now support the ability to
set the visibility of exports.
This essentially reverts 6d951aff7e
This reverts commit 3370d58956.
This commit was done with an LLVM build that did not have assertions
enabled. There are LLVM assertions being triggered due to this commit.
Reopens#10627Reopens#12013Reopens#12027
This reverts commit 55c3efcb58.
This caused a regression when building Zig with Homebrew.
Also I don't like that it prints a message on success. Only when there
is a problem should an error message be printed.
closes#14093
`-undefined dynamic_lookup` was added in #13991. `-undefined error` is the
opposite, and can be used to revert an `-undefined dynamic_lookup` flag
specified previously on the command line.
When using llvm opaque pointers, typed pointers and pointer bitcasts are
no longer needed. This also avoids needing packed struct layouts that
are nested inside pointers, letting us avoid computing struct layouts
in Sema that could cause unnecessary dependency loops.
In #1622, when targeting WebAsembly, the --allow-undefined flag
became unconditionally added to the linker.
This is not always desirable.
First, this is error prone. Code with references to unkown symbols
will link just fine, but then fail at run-time.
This behavior is inconsistent with all other targets.
For freestanding wasm applications, and applications that only use
WASI, undefined references are better reported at compile-time.
This behavior is also inconsistent with clang itself. Autoconf and
cmake scripts checking for function presence think that all tested
functions exist, but then resulting application cannot run.
For example, this is one of the reasons compilation of Ruby 3.2.0
to WASI fails with zig cc, while it works out of the box with clang.
But all applications checking for symbol existence before compilation
are affected.
This reverts the behavior to the one Zig had before #1622, and
introduces an `import_symbols` flag to ignore undefined symbols,
assuming that the webassembly runtime will define them.
* allow file level `union {}` to parse as tuple field
this was found while fuzzing zls.
* before this patch the input `union {}` crashed the parser. after
this, it parses correctly just like `struct {}`.
* adds behavior tests for both inputs `struct {}` and `union {}`,
checking that each becomes a file level tuple field.
Otherwise, we were prematurely committing `__LINKEDIT` segment LC
with outdated size (i.e., without code signature being taken into account).
This would scaffold into strict validation failures by Apple tooling.
Can occur when trying to open a directory for iteration but the 'List folder contents' permission of the directory is set to 'Deny'.
This was found because it was being triggered during PATH searching in ChildProcess.spawnWindows if a PATH entry did not have 'List folder contents' permission, so this fixes that as well (note: the behavior on hitting this during PATH searching is to treat it as the directory not existing and therefore will fail to find any executables in a directory in the PATH without 'List folder contents' permission; this matches Windows behavior which also fails to find commands in directories that do not have 'List folder contents' permission).
* pthread_key_t should also be available for non-android platforms
* Also change the type to c_uint because Linux pthreadtypes.h typedefs it as "unsigned int"
Partially addresses #13950
The extended instructions starting with opcode `0xFC` are refactored
to make the work the same as the SIMD instructions. This means a
`Mir` instruction no longer contains a field 'secondary'. Instead,
we use the `payload` field to store the index into the extra list
which contains the extended opcode value. In case of instructions
such as 'memory.fill' which also have an immediate value, such
values will also be stored in the extra list right after the
instruction itself. This makes each `Mir` instruction smaller.
These errdefer where never executed, while this didn't bother the stage1
compiler, it caused an error in stage2.
The fix is just removing those errdefer which doesn't change any
behaviour because they were never executed in the first place.
This reverts commit aa3964477f.
This declaration is already provided by operating-system-specific files.
This is not the correct solution to the problem.
See #14013Reopens#13950
these were found while fuzzing zls.
this patch prevents overflow for the following file contents and adds
tests for them.
* `enum(u32)` - causes overflow in std.zig.Ast.fullContainerDecl()
* `*x` - causes overflow in std.zig.Ast.fullPtrType()
* `**x` - causes overflow in std.zig.Ast.firstToken()
When reading a document with nested sections, it is not easy to discover
the depth of the current section.
Add support for nested section numbers, from the h2 to the h4 heading,
in the format "lv1. ", "lv1.lv2. ", "lv1.lv2.lv3. ". The "Zig Version"
and "Table of Content" sections are excluded.
The section numbers are implemented in CSS, with the CSS rules declared
inside a @media rule. Currently the @media rule targets all media.
This merges the paths from flushModule and linkWithZld to a single
function that will write the entire WebAssembly module to the file.
This reduces the chance of mistakes as we do not have to duplicate
the logic. A similar action may be needed later for linkWithLLD.
Fixes a regression caused by https://github.com/ziglang/zig/pull/13983
From the added comment:
We still search the path if the cwd is absolute because of the
"cwd set in ChildProcess is in effect when choosing the executable path
to match posix semantics" behavior--we don't want to skip searching
the PATH just because we were trying to set the cwd of the child process.
Normally when we want a pointer to the end of a struct we just add 1 to
the struct pointer. However, when it is a zero-bit struct, the pointer
type being used during lowering is often a dummy pointer type that
actually points to a non-zero-bit type, so we actually want to add 0
instead, since a zero-bit struct begins and ends at the same address.
* Initialize `big_align` with 1 as 0 is not a valid alignment.
* Add an assert to `alignForwardGeneric` to catch this issue earlier.
* Refactor valid alignment checks to call a more descriptive function.
* Handle a `null` return from `llvmFieldIndex`.
* Add a behavior test to test this code path.
* Reword this test name, which incorrectly described how pointers to
zero-bit fields behave, and instead describe the actual test.
When an atom has one or multiple aliasses, we we could not find the
target atom from the alias'd symbol. This is solved by ensuring that
we also insert each alias symbol in the symbol-atom map.
The name of the game here is to avoid CreateProcessW calls at all costs,
and only ever try calling it when we have a real candidate for execution.
Secondarily, we want to minimize the number of syscalls used when checking
for each PATHEXT-appended version of the app name.
An overview of the technique used:
- Open the search directory for iteration (either cwd or a path from PATH)
- Use NtQueryDirectoryFile with a wildcard filename of `<app name>*` to
check if anything that could possibly match either the unappended version
of the app name or any of the versions with a PATHEXT value appended exists.
- If the wildcard NtQueryDirectoryFile call found nothing, we can exit early
without needing to use PATHEXT at all.
This allows us to use a <open dir, NtQueryDirectoryFile, close dir> sequence
for any directory that doesn't contain any possible matches, instead of having
to use a separate look up for each individual filename combination (unappended +
each PATHEXT appended). For directories where the wildcard *does* match something,
we only need to do a maximum of <number of supported PATHEXT extensions> more
NtQueryDirectoryFile calls.
---
In addition, we now only evaluate the extensions in PATHEXT that we know we can handle (.COM, .EXE, .BAT, .CMD) and ignore the rest.
---
This commit also makes two edge cases match Windows behavior:
- If an app name has the extension .exe and it is attempted to be executed, that is now treated as unrecoverable and InvalidExe is immediately returned no matter where the .exe is (cwd or in the PATH). This matches the behavior of the Windows cmd.exe.
- If the app name contains more than just a filename (e.g. it has path separators), then it is excluded from PATH searching and only does a cwd search. This matches the behavior of Windows cmd.exe.
* Handle a `null` return from `llvmFieldIndex`.
* Add a behavior test to test this code path.
* Reword this test name, which incorrectly described how pointers to
zero-bit fields behave, and instead describe the actual test.
Previously we used the relocation index to find the corresponding
symbol that represents the type. However, the index actually
represents the index into the list of types. We solved this by
first retrieving the original type, and then finding its location
in the new list of types. When the atom file is 'null', it means
the type originates from a Zig function pointer or a synthetic
function. In both cases, the final type index was already resolved
and therefore equals to relocation's index value.
This was a poor naming choice; these are parameters, not arguments.
Parameters specify what kind of arguments are expected, whereas the arguments are the actual values passed.
This matches `cmd.exe` behavior. For example, if there is only a file named `mycommand` in the cwd but it is a Linux executable, then running the command `mycommand` will result in:
'mycommand' is not recognized as an internal or external command, operable program or batch file.
However, if there is *both* a `mycommand` (that is a Linux executable) and a `mycommand.exe` that is a valid Windows exe, then running the command `mycommand` will successfully run `mycommand.exe`.
Avoid a lot of unnecessary utf8 -> utf16 conversion and use a single ArrayList buffer for all the joined paths instead of a separate allocation for each join
For example, if the command is specified as `something.exe`, the retry will now try:
```
C:\some\path\something.exe
C:\some\path\something.exe.COM
C:\some\path\something.exe.EXE
C:\some\path\something.exe.BAT
... etc ...
```
whereas before it would only try the versions with an added extension from `PATHEXT`, which would cause the retry to fail on things that it should find.
When parsing the table of contents containing the symbols and their
positions we initially used the index within the map to retrieve
the offset. However, during resizing of the underlaying array this
would invalidate those indexes which meant incorrect offsets were
being stored for symbols. We now use the current symbol index
to also get the index into the symbol position instead.
Given `main.go`:
package main
import _ "os/user"
func main() {}
Compiling it to linux/arm64:
$ CGO_CFLAGS='-O0' GOOS=linux GOARCH=arm64 CGO_ENABLED=1 CC="zig cc -target aarch64-linux-gnu.2.28" go build main.go
Results in this error:
runtime/cgo(.text): unknown symbol memset in callarm64
runtime/cgo(.text): unknown symbol memset in callarm64
runtime/cgo(.text): relocation target memset not defined
In the midst of intermediate compilations files we can see this commmand:
ld.lld -o _cgo_.o <...> /tmp/go-build206961058/b043/_x009.o <...> ~/.cache/zig/.../libcompiler_rt.a <...> ~/.cache/.../libc.so.6
`_x009.o` needs memset:
$ readelf -Ws ./b043/_x009.o | grep memset
22: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND memset
Both `libcompiler_rt.a` and `libc.so.6` provide it:
$ readelf -Ws ~/.cache/zig/.../libcompiler_rt.a | grep memset
870: 0000000000000000 318 FUNC WEAK DEFAULT 519 memset
$ readelf -Ws ~/.cache/zig/.../libc.so.6 | grep -w memset
476: 000000000001d34c 0 FUNC GLOBAL DEFAULT 7 memset@@GLIBC_2.2.5
Since `libcompiler_rt.a` comes before libc in the linker line, the
resulting `_cgo_.o` still links to a weak, unversioned memset:
$ readelf -Ws ./b043/_cgo_.o | grep -w memset
40: 000000000022c07c 160 FUNC WEAK DEFAULT 14 memset
719: 000000000022c07c 160 FUNC WEAK DEFAULT 14 memset
Since the final linking step is done by Golang's linker, it does not
know of `libcompiler_rt.a`, and fails to link with the error message
above. However, Go linker does recognize memset from glibc. If we
specify an `-lc` equivalent before the `libcompiler_rt.a`, it will link
to memset from libc:
$ readelf -Wa ./b043/_x009.o |grep memset
14: 0000000000000000 0 FUNC GLOBAL DEFAULT UND memset@GLIBC_2.17 (2)
157: 0000000000000000 0 FUNC GLOBAL DEFAULT UND memset@GLIBC_2.17
... and then `main.go` will compile+link successfully.
Why doesn't Go linker take memset from glibc? An educated guess: Go
determines whether to link with glibc from what the program asks (I
presume `.dynsym`). Since `memset` is no longer attributed to glibc, Go
skips linking to glibc altogether.
Bonus question: curious why `-O0` is necessary? Because when
optimizations are enabled (the default), the C compiler replaces
`memset` function call with plain `stp` instructions (on aarch64).
By pulling out the parallel hashing setup from `CodeSignature.zig`,
we can now reuse it different places across MachO linker (for now;
I can totally see its usefulness beyond MachO, eg. in COFF or ELF too).
The parallel hasher is generic over actual hasher such as Sha256 or MD5.
The implementation is kept as it was.
For UUID calculation, depending on the linking mode:
* incremental - since it only supports debug mode, we don't bother with MD5
hashing of the contents, and populate it with random data but only once
per a sequence of in-place binary patches
* traditional - in debug, we use random string (for speed); in release,
we calculate the hash, however we use LLVM/LLD's trick in that we
calculate a series of MD5 hashes in parallel and then one an MD5 of MD5
final hash to generate digest.
- Previously, some of the compress tests used `@src()` in combination with `dirname` and `openDirAbsolute` to read test files at runtime, which both excludes platforms that `openDirAbsolute` is not implemented for (WASI) and platforms that `SourceLocation.file` is not absolute (this was true for me locally on Windows). Instead of converting the tests to use `fs.cwd().openDir`, they have been converted to use `@embedFile` to avoid any potential problems with the runtime cwd.
- In order to use `@embedFile`, some of the `[]u8` parameters needed to be changed to `[]const u8`; none of them needed to be non-const anyway
- The tests now use `expectEqual` and `expectEqualSlices` where appropriate for better diagnostics
This fixes a regression introduced in #12298 where colors would never reset in a Windows console because the attributes would be queried on every `setColor` call, and then try to 'reset' the attributes to what it just queried (i.e. it was essentially doing a complicated no-op on .Reset).
This fixes the problem while (I think) keeping with the spirit of the changes in #12298--that is, `TTY.Config` is not specifically tied to stderr like it was before #12298. To that end, detectTTYConfig now takes a `File` and that's what gets used to query the initial attributes to reset to.
(for context, before #12298, the first `setColor` call is where the reset attributes would get queried and it would always use stderr to do it)
#13666 introduced a footgun when using `toOwnedSlice` with `errdefer array_list.deinit()`, since `toOwnedSlice` could retain capacity if `resize` failed, meaning it would leak without `deinit` being called. This meant that the only correct way to use `toOwnedSlice` was with `defer` instead of `errdefer` to ensure that the ArrayList would get cleaned up.
Now, toOwnedSlice will now behave similarly to how it did before #13666, in that it will always clear the ArrayList's capacity if the resize/realloc succeeds.
This also reverts commit 05890a12f5, which was contingent on the modified toOwnedSlice behavior.
Closes#13946
In glibc.zig, there were a few instances where the zig lib dir path name
incorrectly made its way into the cache namespace for various build
artifacts, resulting in unnecessary rebuilds of glibc.
Closes#13619
* revert changes to Module because the error set is consistent across
operating systems.
* remove duplicated Stat.fromSystem code and use a less redundant name.
* make fs.Dir.statFile follow symlinks, and avoid pointless control
flow through the posix layer.
When a data symbol is required to be exported, we instead generate
a global that will be exported. This global is immutable and contains
the address of the data symbol.
This commit moves the logic from `std.build.InstallRawStep` into `zig
objcopy`. The options here are limited, but we can add features as
needed.
closes#9261
New issues can be opened for specific objcopy flag support.
Add a missing ReleaseSmall when describing unreachable in the try
section and the unreachable entry in the Keyword Reference section.
Additionally, transform Debug, ReleaseSafe, ReleaseFast and ReleaseSmall
into links in the try section.
Co-authored-by: Andrew Kelley <andrew@ziglang.org>
Currently, the test_fn_reflection.zig doctest has an example of
.Fn.is_var_args. This example can confuse the reader, since there is no
documentation about variadic functions and is_var_args is mainly used in
the compiler.
Remove the example with .Fn.is_var_args and add instead examples with
.Fn.return_type and .Fn.is_generic.
In the slice_bounds.zig doctest, the code "const slice = array[2..4]" is
incorrect, since the actual type is a pointer to an array, instead of a
slice.
Use a runtime know value to slice the array.
In the slices.zig doctest, the code
`const all_together_slice = all_together[0..]` is incorrect, since the
actual type is a pointer to an array, instead of a slice.
Use a runtime-known value to slice the array.
In the next "slice pointer" test, clarify that slicing a slice to
produce a pointer to an array, requires comptime-known indexes, not just
constant indexes.
When invoking the self-hosted linker using `-fno-LLD` while using the
LLVM backend or invoking it as a linker, we create a seperate path.
This path will link the object file generated by LLVM and the
supplied object files just once, allowing to simplify the
implementation between incremental and regular linking.
Originally I just wanted to move these down because they seemed to be in some random position of the file,
but these constants look very old with their CASING and pretty unnecessary anyway so I just inlined them.
This change adds to Liveness a simple pattern match for the
try-like `.condbr` blocks emitted by Sema's safety checks. This
allows us to determine that these do not modify memory, which
permits us to elide additional loads in the backend.
As @Vexu points out in the main issue, this is probably not a
complete solution on its own. We'll still want a way to reliably
narrow the load/copy when performing several consecutive accesses,
such as `foo.arr[x][y].z`
Resolves https://github.com/ziglang/zig/issues/12215
This helps prevent errors related to undefined pointers being passed
through to some OS apis when slices have 0 length.
Tests have also been added to catch these cases.
Split the original `defer.zig` doctest into 3 doctest:
1. Document the defer keyword
2. Document that the return statement is not allowed inside a defer
expression
3. Document the errdefer keyword
Replace "method" with "expression" in the text `defer method`.
The PCG32 fill function seems to have been copy-pasted from code using u64, so requesting `n` bytes where `(n & 7) > 4` bytes would cause the last few bytes to be all 0.
This implements `@splat` for vectors where the element type is
divisible by 8 and a power of two. This is fairly simple as we can
store the values directly within the virtual stack. But for
all other sizes, we must first shift and bitwise-or the values
before we can store them to fit them like a packed-struct, rather
than an array.
This implements `airSplat` for the native WebAssembly backend when
the features 'simd128' or 'relaxed-simd' are enabled. The commit
supports splat where the value lives in the linear memory segment,
as well as on the stack. This saves a lot of instruction cost.
When it detects the element type is not 8, 16, 32 or 64 bits,
the backend will instead use the same strategy as if the features
where disabled.
This implements loading and storing immediate values representing
a vector with exactly 128 bits. When the vector does not equal to
128 bits, or when the simd128 or relaxed-simd features are disabled
the value will be treated as an array instead.
This adds the opcodes for both the simd128 and relaxed-simd features.
Those instructions are required by the self-hosted WebAssembly
backend. Additionally, this also adds the new `v128` Valtype which
is required to represent a 128bit simd value. SIMD values that do
not have exactly 128 bits will be represented differently.
This fixes a class of bugs on macOS where a segfault happening in
a loaded dylib with no debug info would cause a panic in the panic
handler instead of simply noting that the dylib has no valid debug
info via `error.MissingDebugInfo`. An example could be code linking
some system dylib and causing some routine to segfault on say invalid
pointer value, which should normally cause Zig to print an incomplete
stack trace anchored at the currently loaded image and backtrace all
the way back to the Zig binary with valid debug info. Currently, in
a situation like this we would trigger a panic within a panic.
This includes the latest changes from master branch with fixes to the C
backend that affect aarch64-windows which are necessary to build from
source on this target.
This allows using `zig ar` for `CMAKE_AR`. Unfortunately, it requires a
patch to CMakeLists.txt and cannot be done merely with flags to the
cmake line.
We definitely want a shared lock on a cache hit. Without this, we get a
deadlock when Zig is asked to compile the same C source file multiple
times as part of the same compilation.
This is a partial revert of 8ccb9a6ad3.
cc @kcbanner
These bugs were triggered in the C backend by aarch64-specific code in
os/windows.zig. Intentionally not updating zig1.wasm yet because of
upcoming changes and since aarch64-windows is not tested on master yet.
Several <em> elements where added inside a shell node in the
Using--target-and--cflags section. Remove them, since they are not
supposed to be handled by codegen.
For the original commit, see
0c091feb5 (Improve HTML semantics and a11y of language reference).
Fixes#13846
We were missing some math functions. After this enhancement I verified
that I was able to cross-compile ninja.exe for aarch64-windows and
produce a viable binary.
This also no longer uses zstd compression on this file. The reasoning
for this is:
* It has been demonstrated that the release tarballs are actually
smaller if zig1.wasm gets compressed along with the other files
rather than separately compressed.
* More importantly, leaving zig1.wasm uncompressed may result in a
smaller git repository size, since the repository as a whole could
have savings across the multiple versions of zig1.wasm, which would
not be possible if each one was independently compressed.
* When in doubt, do what is simpler, which is to not have this extra
zstd mechanism. This will remove the only "vendored" code from our
build process, which is a nice property to have.
Co-authored-by: Andrew Kelley <andrew@ziglang.org>
On x86 interestingly I can see a reduction in codesize by 1 instruction with this.
While not necessarily faster, it might still reduce codesize a bit and this ordering is also more logical
because it follows ASCII table order. Rust's std uses this ordering too.
See https://zig.godbolt.org/z/PqodY8YqY for the difference.
This makes it so that we no longer use a LUT (Look-Up Table):
* The code is much simpler and easier to understand now.
* Using a LUT means we rely on a warm cache.
Relying on the cache like this results in inconsistent performance and in many cases codegen will be worse.
Also as @topolarity once pointed out, in some cases while it seems like the code may branch, it actually doesn't:
https://github.com/ziglang/zig/pull/11629#issuecomment-1213641429
* Other languages' standard libraries don't do this either.
JFF I wanted to see what other languages codegen compared to us now:
https://rust.godbolt.org/z/Te4ax9Edf, https://zig.godbolt.org/z/nTbYedWKv
So we are pretty much on par or better than other languages now.
* Improve and remove duplicate doNotOptimizeAway() implementations
We currently have two doNotOptimizeAway() implementations, one in
std.math and the other one in std.mem.
Maybe we should deprecate one. In the meantime, the std.math one
now just calls the std.mem one.
In a comptime environment, just ignore the value. Previously,
std.mem.doNotOptimizeAway() did not work at comptime.
If the value fits in a CPU register, just tell the compiler we
need that value to be computed, without clobbering anything else.
Only clobber all possibly escaped memory on pointers or large arrays.
Add tests by the way since we didn't had any (we had, but only
indirect ones).
This will make stack traces and debugging experience more consistent
in the sense that the presence of source lines in stack traces will
not be dependent on the current working directory of the running process.
- In #13720, expectEqualBytes was added as a standalone function
- In #13723, expectEqualSlices was made to use expectEqualBytes when the type was u8
- In this commit, expectEqualSlices has fully absorbed expectEqualBytes, and expectEqualBytes itself has been removed
For non-`u8` types, expectEqualSlices will now work similarly to expectEqualBytes (highlighting diffs in red), but will use a full line for each index and therefore will only print a maximum of 16 indexes.
When building with LLVM, always do -lole32 -lversion -luuid even when
using the cmake-provided build stuff. Otherwise we get undefined symbols
when linking.
- C compilation flows didn't hold an exclusive lock on the cache manifest file when writing to it in all cases
- On windows, explicitly unlock the file lock before closing it
There are still a few occurrences of "stage1" in the standard library
and self-hosted compiler source, however, these instances need a bit
more careful inspection to ensure no breakage.
When testing with
```
stage4/bin/zig test ../test/behavior.zig -I../test -ofmt=c -target x86_64-windows
```
previously, one would see this message:
```
error: lld-link: subsystem must be defined
```
Now, `--subsystem console` as well as `-lntdll -lkernel32` are on the
`zig run` line, producing a binary.
This commit adds a 637 KB binary file to the source repository. This
commit does nothing else, so it should be replaced with a different
commit before this branch is merged to avoid bloating the git
repository.
On windows we get:
lld-link: error: undefined symbol: __stack_chk_fail
>>> referenced by CMakeFiles/zig2.dir/zig2.c.obj:(main)
>>> referenced by CMakeFiles/zig2.dir/zig2.c.obj:(main_main)
>>> referenced by CMakeFiles/zig2.dir/zig2.c.obj:(log_scoped_28_default_29_err__anon_2764)
>>> referenced 36192 more times
lld-link: error: undefined symbol: __stack_chk_guard
>>> referenced by CMakeFiles/zig2.dir/zig2.c.obj:(.refptr.__stack_chk_guard)
>>> referenced by CMakeFiles/zig2.dir/compiler_rt.c.obj
I messed up the spelling of '-stack_size' making it '-stack' instead.
Will need to fix on master branch. But let's test this here before
making another master branch commit.
In particular, these two changes are relevant:
* zig cc: support -stack in addition to --stack for linker arg
- Fixes stack overflow when running zig2 on aarch64-macos.
* compiler_rt: avoid using weak aliases
- Fixes duplicate symbol when linking zig2 on aarch64-linux.
In the CI system, I copied the old tarball and then applied
05c21a26cb2d5daf06191bd996d0770192704b66 to its compiler_rt
implementation.
After this is verified we can drop this commit and regenerate the
tarballs from a master branch commit.
Before it was checking if the compiler was AppleClang, however, this did
not handle the case when using a compiled-from-source Clang on a macOS
computer, in which case the linker is still Apple ld64, and it is in
fact the linker that needs to have the different flag to communicate
stack size.
This branch largely reverts 58f961f4cb. I
would like to revisit the proposal to modify the standard library in
this way and think more carefully about it before adding isAbsolute()
checks everywhere.
In ea9ad1e85dd5e2ba18e7d55f7a7f9694282159f1, I incorrectly applied
boolean logic to one of the pieces of logic, resulting in a regression
in translate-c.
This flag makes CI servers print garbage to the terminal. I started
implementing detection in wasi.c, but it would have destroyed the
beautiful code that only touches the C stdlib, does not do any
conditional compilation based on the operating system, or rely on any
POSIX functions.
So instead, let's just do without this flag to retain simplicity in this
step of the build process.
This takes a bit longer since the interpreted part has to do more work
but it saves a round trip through the compiler by allowing `zig2 build`
to be the final step. 1-2-3, done.
For me this is currently failing due to compilation errors generated by
GCC when compiling zig2.c but in theory if those are fixed, it should
work!
* synchronize zig1.c from zig-wasi external project
* change the way argv works to avoid absolute paths
* autodetect isatty
* compiler_rt: disable some functions when object format is C
* add missing flag from config.zig.in
The next problem is that compiling compiler_rt.c with gcc gives
"conflicting types" errors for `__eqhf2` and friends.
Instead of checking for absolute paths and current working directories
in various file system operations, there is one simple solution: allow
overriding `std.fs.cwd` on WASI.
os.realpath is back to causing a compile error when used on WASI. This
caused a compile error in the Sema handling of `@src()`. The compiler
should never call realpath, so the commit that made this change is
reverted (95ab942184). If this breaks
debug info, a different strategy is needed to solve it other than using
realpath.
I also removed the preopens code and replaced it with something much
simpler. There is no longer any global state in the standard library.
Additionally-
* os.openat no longer does an unnecessary fstat on WASI when O.WRONLY
is not provided.
* os.chdir is back to causing a compile error on WASI.
Before this commit, I observe a crash from this abiSize call because the
element type is a struct that is not yet resolved. This is triggered by
running the behavior tests with -ofmt=c -target x86_64-windows.
When the bulk_memory feature is enabled, wasi-libc will only use
it if the number of bytes is >= BULK_MEMORY_THRESHOLD
Set it to 32 as in the original wasi-libc Makefile.
This is a partial revert of 0d533433e2,
which regressed this behavior. The idea here is to avoid aliases, which
happens when the same function is exported with multiple names. The
problem with aliases is that weak aliases don't seem to work, causing
symbol collisions when multiple of the same symbol are provided, despite
the desired behavior that weak symbols are overridden.
In this case we export redundant functions with different names. Thanks
to -ffunction-sections, the unused functions will be garbage-collected
at link time. This leaves us with the best of both worlds: Zig's
compiler-rt will provide both sets of symbols, and it will be
binary-compatible with different compilers that expect different names,
while still resulting in binaries without garbage.
When a local defined outside a loop dies inside the loop, it can still
be needed on subsequent loop iterations, so reuse of the local must be
deferred until after the loop ends. This causes behavior tests to pass.
In both backends they did not observe the Liveness information for try
instructions. Now they do. For the C backend this is necessary for
correctness; for the LLVM backend, it improves code generation.
I did not do a full audit, but I did notice a few issues which are
resolved in this commit. Probably it would be worth adding debug
infrastructure to assert that the number of reaps equals the number of
calls to resolveInst() per air lowering function.
Override the cache directories because they won't actually help other CI
runs which will be testing alternate versions of zig, and ultimately
would just fill up space on the hard drive for no reason.
In practice we did see one of the CI servers fill up too many files
inside ~/.cache/zig, which caused certain file system operations to
start returning ENOSPC, despite the hard drive having plenty of space
left.
In general the C backend should lower to human-maintainable C code
whenever possible. Directly using C types that one would use when
writing C code is one part of the strategy.
The concern with including stdint.h is C89 compatibility. Well, we can
just check the C std lib version before deciding to include that header.
These operations are allowed because the string literals are just
pointers but they produce unexpected results. These errors prevent
beginners from shooting themselves in the foot while still allowing
advanced users to circumvent them if they desire to do so.
Closes#8290
This commit removes async/await/suspend/resume from the language
reference, as that feature does not yet work in the self-hosted
compiler.
We will be regressing this feature temporarily. Users of these language
features should stick with 0.10.x with the `-fstage1` flag until they
are restored.
See tracking issue #6025.
`expectEqualBytes` will now truncate the hexdump of each input to a maximum window of 256 bytes, which makes it safe to use for arbitrarily large inputs. Therefore, it can be used in `expectEqualSlices` when the type is u8.
`@llvm.dbg.value` is absolutely useless, adding a temporary alloca
to store the constant in will make it actually show up in debuggers.
The effect on performance should be minimal since there is only one
store and it the change is not applied to ReleaseSafe builds.
```zig
fn foo(a: u32, b: []const u8, c: bool, d: enum { yes, no }) void {
_ = a; _ = b; _ = c; _ = d;
}
```
before:
```
Breakpoint 1, a.foo (a=<optimized out>, b=..., c=<optimized out>, d=<optimized out>) at a.zig:18
18 _ = d;
```
after:
```
Breakpoint 1, a.foo (a=1, b=..., c=false, d=yes) at a.zig:15
15 _ = a; _ = b; _ = c; _ = d;
(gdb) p b
$1 = {ptr = 0x20854f <a.main.anon_3888> "bar", len = 3}
```
This implements support for loading and storing where the lhs is
of pointer type with host_size != 0. e.g. when loading a specific
field from a packed struct with a non-byte alignment such as (0:1:3).
This allows the Wasm backend to construct an instance of a packed
struct during runtime. We first allocate a local, and then
shift+or each field's value into the result local. We then finally
return this result local as value.
The commit also fixes a type-issue in `airElemVal` where we used
the element type instead of a pointer type to store the value's
address into.
This also adds support loading a runtime pointer from a packed struct.
Also, this commit improves many utility functions such as `trunc` and
`intcast` to also support non-integer types such as booleans.
Simplifies the airStructFieldPtr(index) functions to only obtain the
correct struct type and field index, which is then passed into the
structFieldPtr function. This function now calculates the byte-offset
of the field's address and returns a new `WValue` with this offset.
This means we only have to do this calculation in a single function,
and no longer have to duplicate any logic. This also handles both
regular (tagged) unions and packed unions.
This implements loading a field from a packed struct, regardless of
its field's type. This means it supports pointers, floats and
integers. The commit also extracts the logic from airTrunc into its
own `trunc` function so it can be re-used.
When lowering constants of packed structs, which are smaller than 65
bits, we lower the value to an integer rather than store it in the
constant data section. This allows us to use an immediate value,
for quick loads and stores.
Different (simple) tuple types do not necessarily print out as different strings.
This is issue would be caused by passing std.fmt.Formatter to std.fmt.format.
Now it can refuse to resize when it would disturb the metadata tracking
strategy, resulting in smaller code size, a simpler implementation, and
less fragmentation.
The previous version had a fatal flaw: it did ensureCapacity(1) on the
freelist when allocating, but I neglected to consider that you could
free() twice in a row. Silly!
This strategy allocates an intrusive freelist node with every
allocation, big or small. It also does not have the problems with resize
because in this case we can push the upper areas of freed stuff into the
corresponding freelist.
* Update wasi-libc to a00bf321eeeca836ee2a0d2d25aeb8524107b8cc
It includes a port of emscripten's allocator that performs
performs much better than the old one.
Most importantly, it includes the prerequisites to later add
support for POSIX threads.
Ran into this when using a program that uses CreateFileMapping and then trying to call `std.fs.createFile` on the mapped file. More info can be found here:
https://stackoverflow.com/questions/41844842/when-error-1224-error-user-mapped-file-occurs
Before:
```
error.Unexpected NTSTATUS=0xc0000243
C:\Users\Ryan\Programming\Zig\zig\lib\std\os\windows.zig:138:40: 0x7ff74e957466 in OpenFile (test.exe.obj)
else => return unexpectedStatus(rc),
^
```
After:
```
FAIL (AccessDenied)
C:\Users\Ryan\Programming\Zig\zig\lib\std\os\windows.zig:137:30: 0x7ff7f5b776ea in OpenFile (test.exe.obj)
.USER_MAPPED_FILE => return error.AccessDenied,
^
```
Now we can reuse the table between CPU model parsers on Linux and
Windows.
Use similar parsing structure for Windows as we do for Linux. On
Windows, we rely on two entries in the registry per CPU core:
`CP 4000` and `Identifier`. Collating the data from the two allows
us recreating most of the `/proc/cpuinfo` data natively on Windows.
Additionally, we still allow for overwriting any CPU features as flagged
by pulling the feature data embedded in `SharedUserData`.
In general, we prefer compiler code to use relative paths based on open
directory handles because this is the most portable. However, sometimes
absolute paths are used, and sometimes relative paths are used that go
up a directory.
The recent improvements in 81d2135ca6
regressed the use case when an absolute path is used for the zig lib
directory mixed with a relative path used for the root source file. This
could happen when, for example, running the standard library tests, like
this:
stage3/bin/zig test ../lib/std/std.zig
This happened because the zig lib dir was inferred to be an absolute
directory based on the zig executable directory, while the root source
file was detected as a relative path. There was no common prefix and so
it was not determined that the std.zig file was inside the lib
directory.
This commit adds a function for resolving paths that preserves relative
path names while allowing absolute paths, and converting relative
upwards paths (e.g. "../foo") to absolute paths. This restores the
previous functionality while remaining compatible with systems such as
WASI that cannot deal with absolute paths.
Does what the name says: rejects generators of low-order groups.
`clearCofactor()` was previously used to do it, but for e.g.
cofactored signature verification, we don't need the result of an
actual multiplication. Only check that we didn't end up with a
low-order point, which is a faster operation.
The `sema.inst_map` datastructure is very often accessed. All
instructions that reference the result of other instructions does a
lookup into this field. Because of this, a significant amount of time,
is spent in `std.HashMap.get`.
This commit replaces the `HashMap` with a simpler data structure that
uses the zir indexes to index into a slice for the result. See the data
structure doc comment for more info.
Returning a bool allows to conveniently use it as the condition
of a while loop.
Also remove restriction that ST cannot be double-word.
While imm is only 32-bit, this value is extended into a 64-bit
memory location.
Test coverage was lacking for chdir() on WASI, allowing this to
regress.
This change makes os.chdir() compile again, and improves the test
logic to use our standard CWD support for WASI.
If a parse node is reserved but never set the node remains
uninitialized and can crash tools doing a linear scan of the nodes
(like ZLS) when switching on the tag.
This is the main way this feature is supposed to work:
1. you discover that a test started failing at some point
but a misconfiguration in the CI script didn't catch that
causing a bunch of false negative (ie false green) CIs
over time
2. you fix the script and now want to figure out when the
test started failing
3. you want to rebuild older commits, potentially only on specific
targets, but with the new fixed CI script that correctly
catches failures
This is what this commit enables
If the noise parameter was null, we didn't use any noise at all.
We unconditionally generated random noise (`noise2`) but didn't use it.
Spotted by @cryptocode, thanks!
The TODO comment in safetyPanic mentions introducing the concept of
reference-counted decls. That sounds like Zig current semantics for
normal declarations. By placing the panic messages in builtin there is
no need for another concept in the compiler.
spirv: introduce SpvModule.Fn to generate function code into
spirv: assembler error message setup
spirv: runtime spec info
spirv: inline assembly tokenizer
spirv: inline assembly lhs result/opcode parsing
spirv: forgot to fmt
spirv: tokenize opcodes and assigned result-ids
spirv: operand parsing setup
spirv: assembler string literals
spirv: assembler integer literals
spirv: assembler value enums
spirv: assembler bit masks
spirv: update assembler to new asm air format
spirv: target 1.5 for now
Current vulkan sdk version (1.3.204) ships spirv tools targetting 1.5,
and so these do not work with binaries targetting 1.6 yet. In the
future, this version number should be decided by the target.
spirv: store operands in flat arraylist.
Instead of having dedicated Operand variants for variadic operands,
just flatten them and store them in the normal inst.operands list.
This is a little simpler, but is not easily decodable in the operand
data representation.
spirv: parse variadic assembly operands
spirv: improve assembler result-id tokenization
spirv: begin instruction processing
spirv: only remove decl if it was actually allocated
spirv: work around weird miscompilation
Seems like there are problems with switch in anonymous struct literals.
spirv: begin resolving some types in assembler
spirv: improve instruction processing
spirv: rename some types + process OpTypeInt
spirv: process OpTypeVector
spirv: process OpTypeMatrix and OpTypeSampler
spirv: add opcode class to spec, remove @exclude'd instructions
spirv: process more type instructions
spirv: OpTypeFunction
spirv: OpTypeOpaque
spirv: parse LiteralContextDependentNumber operands
spirv: emit assembly instruction into right section
spirv: parse OpPhi parameters
spirv: inline assembly inputs
spirv: also copy air types
spirv: inline assembly outputs
spirv: spir-v address spaces
spirv: basic vector constants/types and shuffle
spirv: assembler OpTypeImage
spirv: some stuff
spirv: remove spirv address spaces for now
Standard library tests require the root source file to be the
corresponding file inside the Zig lib directory. In other words, there
may not be two copies of the standard library. After the changes in this
branch, Zig no longer notices that `../lib/std.zig` and
`$(pwd)/../lib/std.zig` are the same file because one is relative and
one is absolute.
* Update for the breaking changes to std.fs.path.resolve. This had a
happy side effect of deleting some error handling code which is no
longer needed.
* Introduce cache_exempt_flags field to CSourceFile. This is used only
for include directories when building libc++ and libc++abi which
depend only on the zig lib path.
* libc_include_dir_list is only added to the cache hash when it
contains directories which have been obtained from system probing. It
is exempt when the directories depend only on the zig lib path.
This is a breaking change to the API. Instead of the first path
implicitly being the current working directory, it now asserts that the
number of paths passed is greater than zero.
Importantly, it never calls getcwd(); instead, it can possibly return
".", or a series of "../". This changes the error set to only be
`error{OutOfMemory}`.
closes#13613
Before, cache manifest files would have absolute file paths. This is
problematic for two reasons:
* Absolute file paths are not portable. Some operating systems such as
WASI have trouble with them. The files themselves are less portable;
they cannot be migrated from one user's home directory to another's.
And finally they can break due to file paths exceeding maximum path
component size.
* They would prevent some advanced use cases of Zig, where the lib dir
has a different path in a different invocation but is ultimately the
same Zig version and lib directory as before.
This commit adds a new column that specifies the prefix directory for
each file. 0 is an escape hatch and has the previous behavior. The other
two prefixes introduced are zig lib directory, and the cache directory.
This means files in zig-cache manifests can reference files local to
these directories.
In practice, this means it is possible to use a different file path for
the zig lib directory in a subsequent run of zig and have it still take
advantage of the global cache, provided that the files inside remain
unchanged.
closes#13050
* Update the AEGIS specification URL to the current draft
* std.crypto.auth: add AEGIS MAC
The Pelican-based authentication function of the AEGIS construction
can be used independently from authenticated encryption, as a faster
and more secure alternative to GHASH/POLYVAL/Poly1305.
We already expose GHASH, POLYVAL and Poly1305 for use outside AES-GCM
and ChaChaPoly, so there are no reasons not to expose the MAC from AEGIS
as well.
Like other 128-bit hash functions, finding a collision only requires
~2^64 attempts or inputs, which may still be acceptable for many
practical applications.
Benchmark (Apple M1):
siphash128-1-3: 3222 MiB/s
ghash: 8682 MiB/s
aegis-128l mac: 12544 MiB/s
Benchmark (Zen 2):
siphash128-1-3: 4732 MiB/s
ghash: 5563 MiB/s
aegis-128l mac: 19270 MiB/s
PR #12837 handled control flow for break and return, but I forgot
about `continue`. This is effectively another break, so we just
need another `.restore_err_ret_index` ZIR instruction.
Resolves#13618.
* Rely on libSystem when targeting macOS.
* Make tools/gen_outline_atomics.zig more idiomatic.
* Remove the CPU detection / auxval checking from compiler_rt. This
functionality belongs in a different component. Zig's compiler_rt
must not rely on constructors. Instead it will export a symbol for
setting the value, and start code can detect and activate it.
* Remove the separate logic for inline assembly when the target does or
does not have lse support. `.inst` works in both cases.
Previous implementation didn't check whether there are pending signals
after return from futex.wait. While it is ok for broadcast case it can
result in multiple wakeups when only one thread is signaled.
This implementation checks that there are pending signals before
returning from wait.
It is similar to the original implementation but the without initial
signal check, here we first go to the futex and then check for pending
signal.
POLYVAL is GHASH's little brother, required by the AES-GCM-SIV
construction. It's defined in RFC8452.
The irreducible polynomial is a mirror of GHASH's (which doesn't
change anything in our implementation that didn't reverse the raw
bits to start with).
But most importantly, POLYVAL encodes byte strings as little-endian
instead of big-endian, which makes it a little bit faster on the
vast majority of modern CPUs.
So, both share the same code, just with comptime magic to use the
correct endianness and only double the key for GHASH.
Closes#7484. Right now for UEFI targets an alignment
of 32 is being used for no reason other than support
a rare bytecode. As this is far from the standard case,
removing this alignment and using the default one,
as most toolchains do, should be the desired behavior.
I wasn't able to create a reduced test case for this but the reasoning
can be seen in `abiAlignmentAdvancedUnion` where if `strat` was lazy
`hasRuntimeBitsAdvanced` would be given `null` instead of `sema`
which would cause eager evaluation when it is not valid or desired.
This also modifies the inline assembly to be more optimizable - instead of
doing explicit movs, we instead communicate to LLVM which registers we
would like to, somehow, have the correct values. This is how the x86_64
code already worked and thus allows the code to be unified across the
two architectures.
As a bonus, I threw in x86 support.
There was no check for linker errors after flushing,
which meant that if the link failed the build would
continue and try to copy the non-existant exe, and
also write the manifest as if it had succeeded.
Also adds parsing of lld output, which is surfaced at the
end of the compilation with the other errors instead
of via stderr
While it is already mentioned on the `items` attributes of the structs, it is
interesting to comment in every method potentially invalidating pointers to items
that they may do so.
fixes#12877
Current implementation (before this fix) observes number of waiters when
broadcast occurs and then makes that number of wakeups.
If we have multiple threads waiting for wakeup which immediately go into
wait if wakeup is not for that thread (as described in the issue). The
same thread can get multiple wakeups while some got none.
That is not consistent with documented behavior for condition variable
broadcast: `Unblocks all threads currently blocked in a call to wait()
or timedWait() with a given Mutex.`.
This fix ensures that the thread waiting on futext is woken up on futex wake.
* std.crypto.onetimeauth.ghash: faster GHASH on modern CPUs
Carryless multiplication was slow on older Intel CPUs, justifying
the need for using Karatsuba multiplication.
This is not the case any more; using 4 multiplications to multiply
two 128-bit numbers is actually faster than 3 multiplications +
shifts and additions.
This is also true on aarch64.
Keep using Karatsuba only when targeting x86 (granted, this is a bit
of a brutal shortcut, we should really list all the CPU models that
had a slow clmul instruction).
Also remove useless agg_2 treshold and restore the ability to
precompute only H and H^2 in ReleaseSmall.
Finally, avoid using u256. Using 128-bit registers is actually faster.
* Use a switch, add some comments
The packed struct example was mistakenly applying endianness where it
shouldn't have been. This wasn't being caught because we don't currently
test the examples on Big-endian systems.
I updated the test to remove the endianness where it didn't apply, and
added a new part of the test to demonstrate when it would apply.
perf_event_attr.type needs to take a runtime defined value to enable
dynamic PMU:s, such as kprobe and uprobe. This value can exceed
predefined values defined in the linux headers.
reference: perf_event_open(2) man page
Previously the compiler would crash on branching on undefined values
if you tried using `zig test` with a freestanding target since there
was no start code referencing `builtin.test_functions`.
Closes#12554
* point to init part of field delc when that's where the error occurs
* update test to reflect fixed error message
* only lookup source location in case of error
In the process of 'remaining blocks',
the length of processed message can be from 1 to 79.
The value of 'n-1' is ranged from 0 to 3.
So, st.hx[i] must be initialized at least from st.hx[0] to st.hx[3]
These parameters are only ever needed when `std.builtin` is out of sync
with the compiler in which case panicking is the only valid operation
anyways. Removing them causes a domino effect of functions no longer
needing a `src` and/or a `block` parameter resulting in handling
compilation errors where they are actually meaningful becoming simpler.
* Export invalidFmtErr
To allow consistent use of "invalid format string" compile error
response for badly formatted format strings.
See https://github.com/ziglang/zig/pull/13489#issuecomment-1311759340.
* Replace format compile errors with invalidFmtErr
- Provides more consistent compile errors.
- Gives user info about the type of the badly formated value.
* Rename invalidFmtErr as invalidFmtError
For consistency. Zig seems to use “Error” more often than “Err”.
* std: add invalid format string checks to remaining custom formatters
* pass reference-trace to comp when building build file; fix checkobjectstep
Previously sema only checked that the private declaration was in the same
file as the lookup but now it also checks that the namespace where
the decl was included from was also in the same file.
Closes#13077
All implicit casts can also potentially lead to a panic being emitted
but adding a dbg_stmt before every instruction is not feasible.
This adds 24k new instructions to the ZIR for Sema.zig increasing its
size from 3.8MiB to 4.0MiB.
Closes#13488
This ensures that the Zig version will be re-computed when jumping
through the source tree, which is especially important if bisecting
across AstGen- or other changes that must not use the old cache.
* Add tagName to Value which behaves like @tagName.
* Add hashUncoerced to Value as an alternative to hash when we want to
produce the same hash for value that can coerce to each other.
* Hash owner_decl instead of module_fn in Sema.instantiateGenericCall
since Module.Decl.Index is not affected by ASLR like *Module.Fn was,
and also because GenericCallAdapter.eql was already doing this.
* Use Value.hashUncoerced in Sema.instantiateGenericCall because
GenericCallAdapter.eql uses Value.eqlAdvanced to compare args, which
ignores coersions.
* Add revealed missing cases to Value.eqlAdvanced.
Without these changes, we were breaking the hash contract for
monomorphed_funcs, and were generating different hashes for values that
compared equal. This resulted in a 0.2% chance when compiling
self-hosted of producing a different output, which depended on
fingerprint collisions of hashes that were affected by ASLR. Normally,
the different hashes would have resulted in equal checks being skipped,
but in the case of a fingerprint collision, the truth would be revealed
and the compiler's behavior would diverge.
These functions have a very error-prone API. They are essentially
`all(cmp(op, ...))` but that's not reflected in the name.
This renames these functions to `compareAllAgainstZero...` etc.
for clarity and fixes >20 locations where the predicate was
incorrect.
In the future, the scalar `compare` should probably be split off
from the vector comparison. Rank-polymorphic programming is great,
but a proper implementation in Zig would decouple comparison and
reduction, which then needs a way to fuse ops at comptime.
Adds optimizations for by-ref types to:
- .struct_field_val
- .slice_elem_val
- .ptr_elem_val
I would have expected LLVM to be able to optimize away these
temporaries since we don't leak pointers to them and they are fed
straight from def to use, but empirically it does not.
Resolves https://github.com/ziglang/zig/issues/12713
Resolves https://github.com/ziglang/zig/issues/12638
This change makes any of the `*_val` instructions check whether it's
safe to elide copies for by-ref types rather than performing this
elision blindly.
AIR instructions fixed:
- .array_elem_val
- .struct_field_val
- .unwrap_errunion_payload
- .try
- .optional_payload
These now all respect value semantics, as expected.
P.S. Thanks to Andrew for the new way to approach this. Many of the
lines here are from his recommended change, which comes with the
significant advantage that loads are now as small as the intervening
memory access allows.
Co-authored by: Andrew Kelley <andrew@ziglang.org>
This is a follow-up to 9dc98fba, which made comptime initialization
patterns for union/struct more robust, especially when storing to
comptime-known pointers (and globals).
Resolves#13063.
These constants were read as a block count in initForBlockCount()
but at the same time, as a size in update().
The unit could be blocks or bytes, but we should use the same one
everywhere.
So, use blocks as intended.
Fixes#13506
On Windows, lld-link resolves PDB output paths using `/` and embeds the
result in the final executable, which breaks some native tooling like
WPR/WPA. This commit overrides the default behavior of lld-link by
explicitly setting the output PDB filename and binary-embedded path.
We postpone emitting debug info until *after* we generate the function
so that we have an idea of the consumed stack space. The stack offsets
encoded within DWARF are with respect to the frame pointer `.fp`.
- the meaning of packed structs changed in zig 0.10. adjust accordingly.
Use "extern struct" for the cases that directly map to C structs.
- Add new type info kinds, like enum64 and DeclTag
- change the Type enum to use the canonical names from libbpf.
This is more predictable when comparing with external BPF
documentation (than invented synonyms that need to be guessed)
This service stopped working two days ago for unknown reasons. Until it
is determined how to get it working again, or we switch to a different
CI provider for aarch64, this CI test coverage is disabled so that
we can continue to use the CI for other targets.
* std.crypto: make ghash faster, esp. for small messages
Aggregated reduction requires 5 additional multiplications (to
precompute the powers of H), in order to save 2 multiplications
per batch.
So, only use large batches when it's actually interesting to do so.
For the last blocks, reuse the precomputations in order to perform
a single reduction.
Also, even in .ReleaseSmall, allow 2-block aggregation.
The speedup is worth it, and the code increase is reasonable.
And in .ReleaseFast, bump the upper batch size up to 16.
Leverage comptime by the way instead of duplicating code.
std/crypto/benchmark.zig on Apple M1:
Zig 0.10.0: 2769 MiB/s
Before: 6014 MiB/s
After: 7334 MiB/s
Normalize function names by the way.
* Change clmul() to accept the half to be processed
This avoids a bunch of truncate() calls.
* Add more ghash tests to check all code paths
* crypto.core.aes: process 6 block in parallel instead of 8 on aarch64
At least on Apple Silicon, this is slightly faster than 8 blocks.
* AES: add parallel blocks for tigerlake, rocketlake, alderlake, zen3
and reimplement shuffle in terms of it. This allows the caller to specify an index type of a fixed bit width, allowing results to be independent usize.
...instead of hard-coding it to 20.
- This is consistent with the ChaCha implementation
- NaCl and libsodium, that this API is designed to interop with,
also support 8 and 12 round variants. The 12 round variant, in
particular, provides the same security level as the 20 round variant,
but is obviously faster.
- scrypt currently uses its own non optimized version of Salsa, just
because it use 8 rounds instead of 20. This will help remove code
duplication.
No behavior nor public API changes. The Salsa20 and XSalsa20 still
represent the 20-round variant.
PR #13101 recently renamed the "i386" architecture to "x86", and it
seems the specific CPU model got swept up in that. "x86" is an umbrella
term that describes a family of CPUs, and the "i386" is the oldest
supported model under that umbrella.
Empirically, this `AutoHashMapUnmanaged` -> `AutoArrayHashMapUnmanaged`
change fixes all non-determinism in `ReleaseFast` build artifacts.
Closes#12183
On platforms where c_longdouble is 128-bits, interop with C code
is simplified by making f128 match the alignment of c_longdouble.
I intended to make this change as part of #13257, but I missed this
part.
Can be changed to `!inst_ty.hisRuntimeBitsIgnoreComptime()` when the
"result location with inferred type ends up being pointer to comptime_int"
test stops producing Air containing a `bitcast(*comptime_int, ...)`.
See #13410
Rewrite GHASH to use 128-bit multiplication over non-reversed
integers, and up to 8 blocks aggregated reduction.
lib/std/crypto/benchmark.zig results:
Xeon E5:
Before: 1604 MiB/s
After: 4005 MiB/s
Apple M1:
Before: 2769 MiB/s
After: 6014 MiB/s
This also makes AES-GCM faster by the way.
When Wasm extern functions contain the same name, but have a
different module name such as `extern "a"` vs `extern "b"` LLVM will
currently resolve the two functions to the same symbol. By mangling
the name of the symbol, we ensure the functions are resolved
seperately. We mangle the name by applying <name>|<module> where
module is also known as the library name.
exists to help you find issues that are **limited in scope and/or
knowledge of Zig internals.**
### Editing Source Code
First, build the Stage 1 compiler as described in
[Building Zig From Source](https://github.com/ziglang/zig/wiki/Building-Zig-From-Source).
Zig locates lib files relative to executable path by searching up the
filesystem tree for a sub-path of `lib/zig/std/std.zig` or `lib/std/std.zig`.
Typically the former is an install and the latter a git working tree which
contains the build directory.
During development it is not necessary to perform installs when modifying
stage1 or userland sources and in fact it is faster and simpler to run,
test and debug from a git working tree.
-`make` is typically sufficient to build zig during development iterations.
-`make install` performs a build __and__ install.
-`msbuild -p:Configuration=Release INSTALL.vcxproj` on Windows performs a
build and install. To avoid install, pass cmake option `-DZIG_NO_LIB=ON`.
To test changes, do the following from the build directory:
1. Run `make` (on POSIX) or
`msbuild -p:Configuration=Release INSTALL.vcxproj` (on Windows).
2.`$BUILD_DIR/zig build test` (on POSIX) or
`$BUILD_DIR/Release\zig.exe build test` (on Windows).
That runs the whole test suite, which does a lot of extra testing that you
likely won't always need, and can take upwards of 1 hour. This is what the
CI server runs when you make a pull request. (Note: actually it runs a few
more tests; keep reading.)
To save time, you can add the `--help` option to the `zig build` command and
see what options are available. One of the most helpful ones is
`-Dskip-release`. Adding this option to the command in step 2 above will take
the time down from around 2 hours to about 6 minutes, and this is a good
enough amount of testing before making a pull request.
Another example is choosing a different set of things to test. For example,
`test-std` instead of `test` will only run the standard library tests, and
not the other ones. Combining this suggestion with the previous one, you could
do this:
`$BUILD_DIR/bin/zig build test-std -Dskip-release` (on POSIX) or
`$BUILD_DIR/Release\zig.exe build test-std -Dskip-release` (on Windows).
This will run only the standard library tests, in debug mode only, for all
targets (it will cross-compile the tests for non-native targets but not run
them).
When making changes to the compiler source code, the most helpful test step to
run is `test-behavior`. When editing documentation it is `docs`. You can find
this information and more in the `--help` menu.
#### Testing Changes to std lib
To quickly test a change to a file in the standard library, you can run zig test and specify a custom lib directory with the follow command-line argument.
```bash
./build/zig test lib/std/fmt.zig --zig-lib-dir lib --main-pkg-path lib/std
```
#### Testing Non-Native Architectures with QEMU
The Linux CI server additionally has qemu installed and sets `-fqemu`.
This provides test coverage for, e.g. aarch64 even on x86_64 machines. It's
recommended for Linux users to install qemu and enable this testing option
when editing the standard library or anything related to a non-native
architecture.
##### glibc
Testing foreign architectures with dynamically linked glibc is one step trickier.
This requires enabling `--glibc-runtimes /path/to/glibc/multi/install/glibcs`.
This path is obtained by building glibc for multiple architectures. This
process for me took an entire day to complete and takes up 65 GiB on my hard
drive. The CI server does not provide this test coverage. Instructions for
producing this path can be found
[on the wiki](https://github.com/ziglang/zig/wiki/Updating-libc#glibc).
Just the part with `build-many-glibcs.py`.
It's understood that most contributors will not have these tests enabled.
#### Testing Windows from a Linux Machine with Wine
When developing on Linux, another option is available to you: `-fwine`.
This will enable running behavior tests and std lib tests with Wine. It's
recommended for Linux users to install Wine and enable this testing option
when editing the standard library or anything Windows-related.
#### Testing WebAssembly using wasmtime
If you have [wasmtime](https://wasmtime.dev/) installed, take advantage of the
`-fwasmtime` flag which will enable running WASI behavior tests and std
lib tests. It's recommended for all users to install wasmtime and enable this
testing option when editing the standard library and especially anything
WebAssembly-related.
#### Improving Translate-C
Please read the [Editing Source Code](#editing-source-code) section as a
prerequisite to this one.
`translate-c` is a feature provided by Zig that converts C source code into
Zig source code. It powers the `zig translate-c` command as well as
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
