The goals of this branch are to:
* compile faster when using the wasm linker and backend
* enable saving compiler state by directly copying in-memory linker
state to disk.
* more efficient compiler memory utilization
* introduce integer type safety to wasm linker code
* generate better WebAssembly code
* fully participate in incremental compilation
* do as much work as possible outside of flush(), while continuing to do
linker garbage collection.
* avoid unnecessary heap allocations
* avoid unnecessary indirect function calls
In order to accomplish this goals, this removes the ZigObject
abstraction, as well as Symbol and Atom. These abstractions resulted
in overly generic code, doing unnecessary work, and needless
complications that simply go away by creating a better in-memory data
model and emitting more things lazily.
For example, this makes wasm codegen emit MIR which is then lowered to
wasm code during linking, with optimal function indexes etc, or
relocations are emitted if outputting an object. Previously, this would
always emit relocations, which are fully unnecessary when emitting an
executable, and required all function calls to use the maximum size LEB
encoding.
This branch introduces the concept of the "prelink" phase which occurs
after all object files have been parsed, but before any Zcu updates are
sent to the linker. This allows the linker to fully parse all objects
into a compact memory model, which is guaranteed to be complete when Zcu
code is generated.
This commit is not a complete implementation of all these goals; it is
not even passing semantic analysis.
Before, the wasm struct had a string table, the ZigObject had a string
table, and each Object had a string table.
Now there is just the one. This makes for more efficient use of memory
and simplifies logic, particularly with regards to linker state
serialization.
This commit additionally adds significantly more integer type safety.
Primarily, this moves linker input parsing from flush() into the linker
task queue, which is executed simultaneously with the frontend.
I also made it avoid redundantly opening the same archive file N times
for each object file inside. Furthermore, hard code fixed buffer stream
rather than using a generic stream type.
Finally, I fixed the error handling of the Wasm.Archive.parse function.
Please pay attention to this pattern of returning a struct rather than
accepting a mutable struct as an argument. This ensures function-level
atomicity and makes resource management straightforward.
Deletes the file and path fields from Archive and Object.
Removed a well-meaning but ultimately misguided suggestion about how to
think about ZigObject since thinking about it that way has led to
problematic anti-DOD patterns.
Removes the `files` field from the Wasm linker, storing the ZigObject
as its own field instead using a tagged union.
This removes a layer of indirection when accessing the ZigObject, and
untangles logic so that we can introduce a "pre-link" phase that
prepares the linker state to handle only incremental updates to the
ZigObject and then minimize logic inside flush().
Furthermore, don't make array elements store their own indexes, that's
always a waste.
Flattens some of the file system hierarchy and unifies variable names
for easier refactoring.
Introduces type safety for optional object indexes.
By organizing linker diagnostics into this struct, it becomes possible
to share more code between linker backends, and more importantly it
becomes possible to pass only the Diag struct to some functions, rather
than passing the entire linker state object in. This makes data
dependencies more obvious, making it easier to rearrange code and to
multithread.
Also fix MachO code abusing an atomic variable. Not only was it using
the wrong atomic operation, it is unnecessary additional state since
the state is already being protected by a mutex.
The compiler actually doesn't need any functional changes for this: Sema
does reification based on the tag indices of `std.builtin.Type` already!
So, no zig1.wasm update is necessary.
This change is necessary to disallow name clashes between fields and
decls on a type, which is a prerequisite of #9938.
Certain symbols were left unmarked, meaning they would not be emit into
the final binary incorrectly. We now mark the synthetic symbols to ensure
they are emit as they are already created under the circumstance they're
needed for. This also re-enables disabled tests that were left disabled
in a previous merge conflict.
Lastly, this adds the shared-memory test to the test harnass as it was
previously forgotten and therefore regressed.
Rather than using the logger, we now emit proper 'compiler'-errors just
like the ELF and MachO linkers with notes. We now also support emitting
multiple errors before quiting the linking process in certain phases,
such as symbol resolution. This means we will print all symbols which
were resolved incorrectly, rather than the first one we encounter.
This introduces some type safety so we cannot accidently give an atom
index as a symbol index. This also means we do not have to store any
optionals and therefore allow for memory optimizations. Lastly, we can
now always simply access the symbol index of an atom, rather than having
to call `getSymbolIndex` as it is easy to forget.
Rather than using the optional, we now directly use `File.Index` which
can already represent an unknown file due to its `.null` value. This
means we do not pay for the memory cost.
This type of index is now used for:
- SymbolLoc
- Key of the functions map
- InitFunc
Now we can simply pass things like atom.file, object.file, loc.file etc
whenever we need to access its representing object file which makes it
a lot easier.
When merging sections we now make use of the `File` abstraction so all
objects such as globals, functions, imports, etc are also merged from
the `ZigObject` module. This allows us to use a singular way to perform
each link action without having to check the kind of the file.
The logic is mostly handled in the abstract file module, unless its
complexity warrants the handling within the corresponding module itself.
Not all custom sections are represented by a symbol, which means the
section will not be parsed by the lazy parsing and therefore get garbage-
collected. This is problematic as it may contain debug information that
should not be garbage-collected. To resolve this, we manually create
local symbols for those sections and also ensure they do not get garbage-
collected.
By default we garbage-collect sections for Wasm to reduce size, as well
as finish linking quicker (as we have fewer things to do). However,
when the user specifies `--no-gc-sections` we ensure all resolved symbols
get marked and therefore do not get garbage collected.
This is supported in both incremental-mode and traditional linking.
Rather than parsing every symbol into an atom, we now only parse them
into an atom when such atom is marked. This means garbage-collected
symbols will also not be parsed into atoms, and neither are discarded
symbols which have been resolved by other symbols. (Such as multiple
weak symbols).
This also introduces a binary search for finding the start index into
the list of relocations. This speeds up finding the corresponding
relocations tremendously as they're ordered ascended by address.
Lastly, we re-use the memory of atom's data as well as relocations
instead of duplicating it. This means we half the memory usage of
atom's data and relocations for linked object files. As we are
aware of decls and synthetic atoms, we free the memory of those
atoms indepedently of the atoms of object files to prevent double-frees.
In general, I don't like the idea of std.meta.trait, and so I am
providing some guidance by deleting the entire namespace from the
standard library and compiler codebase.
My main criticism is that it's overcomplicated machinery that bloats
compile times and is ultimately unnecessary given the existence of Zig's
strong type system and reference traces.
Users who want this can create a third party package that provides this
functionality.
closes#18051
Use inline to vastly simplify the exposed API. This allows a
comptime-known endian parameter to be propogated, making extra functions
for a specific endianness completely unnecessary.
Structs were previously using `SegmentedList` to be given indexes, but
were not actually backed by the InternPool arrays.
After this, the only remaining uses of `SegmentedList` in the compiler
are `Module.Decl` and `Module.Namespace`. Once those last two are
migrated to become backed by InternPool arrays as well, we can introduce
state serialization via writing these arrays to disk all at once.
Unfortunately there are a lot of source code locations that touch the
struct type API, so this commit is still work-in-progress. Once I get it
compiling and passing the test suite, I can provide some interesting
data points such as how it affected the InternPool memory size and
performance comparison against master branch.
I also couldn't resist migrating over a bunch of alignment API over to
use the log2 Alignment type rather than a mismash of u32 and u64 byte
units with 0 meaning something implicitly different and special at every
location. Turns out you can do all the math you need directly on the
log2 representation of alignments.
Most of this migration was performed automatically with `zig fmt`. There
were a few exceptions which I had to manually fix:
* `@alignCast` and `@addrSpaceCast` cannot be automatically rewritten
* `@truncate`'s fixup is incorrect for vectors
* Test cases are not formatted, and their error locations change
Rather than adding the flags "on-demand" during limits writing,
we now properly parse them and store the flags within the limits
itself. This also allows us to store whether we're using shared-
memory or not. Only when the correct flag is set will we set the
max within `Limits` or else we will leave it `undefined`.
Linker now parses segments with regards to TLS segments. If the name
represents a TLS segment but does not contain the TLS flag, we set it
manually as the object file is created using an older compiler (LLVM).
For now we panic when we find a TLS relocation and implement those
later.
For data symbols we will now store its virtual address. This means
we do no longer have to calculate it each time a relocation asks
for the address. This is now done for all data symbols only once
rather than every single relocation for that symbol.
This now also allows us directly store the virtual address of synthetic
symbols without having to create an atom for them. This means we also
don't need to have a "synthetic" segment any longer and do not emit
the synthetic symbols such as __heap_end and __heap_base into the final
binary.
When an atom has one or multiple aliasses, we we could not find the
target atom from the alias'd symbol. This is solved by ensuring that
we also insert each alias symbol in the symbol-atom map.
Addends in relocations are signed integers as theoretically it could
be a negative number. As Atom's offsets are relative to their parent
section, the relocation value should still result in a postive number.
For this reason, the final result is stored as an unsigned integer.
Also, rather than using `null` for relocations that do not support
addends. We set the value to 0 for those that do not support addends,
and have to call `addendIsPresent` to determine if an addend exists
or not. This means each Relocation costs 4 bytes less than before,
saving memory while linking.
Although the wasm-linker previously already supported
debug information in incremental-mode, this was no longer
working as-is with the addition of supporting object-file-parsed
debug information. This commit implements the Zig-created debug information
structure from scratch which is a lot more robust and also allows
being linked with debug information from other object files.
This correctly performs a relocation for debug sections.
The result is that the wasm-linker can now correctly create
a binary from object files while preserving all debug information.
We now link relocatable debug sections with the correct
section symbol and then allocate and resolve the debug atoms
before writing them into the final binary.
Although this does perform the relocation, the actual relocations
are not done correctly yet.
Rather than storing the name of a debug section into the structure
`RelocatableData`, we use the `index` field as an offset into the
debug names table. This means we do not have to store an extra 16 bytes
for non-debug sections which can be massive for object files where each
data symbol has its own data section. The name of a debug section
can then be retrieved again when needed by using the offset and
then reading until the 0-delimiter.
When an object file is being parsed from within an archive
file, we provide the object file size to ensure we do not
read past the object file. This is because follow up object
files can exist there, as well as an LF character to notate
the end of the file was reached. Such a character is invalid
within the object file.
This also fixes a bug in getting the function/global type
for defined globals/functions from object files as it was missing
the substraction with the import count of the respective type.
This reverts commit 7cbd586ace.
This is causing a fail to build from source:
```
./lib/std/fmt.zig:492:17: error: cannot format optional without a specifier (i.e. {?} or {any})
@compileError("cannot format optional without a specifier (i.e. {?} or {any})");
^
./src/link/MachO/Atom.zig:544:26: note: called from here
log.debug(" RELA({s}) @ {x} => %{d} in object({d})", .{
^
```
I looked at the code to fix it but none of those args are optionals.
