`zig build` CLI kicks off async task to compile optimized make runner
executable, does fetch, compiles configure process in debug mode, then
checks cache for the CLI options that affect configuration only. On hit,
skips building/running the configure script. On miss, runs it, saves
result in cache.
The cached artifact is a "configuration" file - a serialized build step
graph, which also includes unlazy package dependencies and additional
file system dependencies.
Next, awaits task for compiling optimized make runner executable, passes
configuration file to it. Make runner is responsible for the CLI after
that point.
For the use case of detecting when `git describe` needs to be rerun, we
can allow the configure process to manually add a file system mtime
dependencies, in this case it would be on `.git/index` and `.git/HEAD`.
This will enable two optimizations:
1. The bulk of the build system will not be rebuilt when user changes
their configure script.
2. The user logic can be completely bypassed when the CLI options
provided do not affect the configure phase - even if they affect the
make phase.
Remaining tasks in the branch:
* some stuff in `zig build` CLI is `@panic("TODO")`.
* configure runner needs to implement serialization of build graph using
std.zig.Configuration
* build runner needs to be transformed into make runner, consuming
configuration file as input and deserializing the step graph.
* introduce depending only on a file's metadata and *not* its contents
into the cache system, and add a std.Build API for using it.
Previously, when using `zig build -fincremental --watch some-run-step`,
if the binary initially builds fine but a future update introduces a
compile error, the old file system inputs of the `Step.Run`---containing
the executable file itself---would remain. This is a bug, because that
file *has* changed since the input was registered (due to incremental
compilation), and yet the `Run` step is not out-of-date because it was
skipped due to a transitive failure.
A simple reproduction for this issue was:
$ zig init
$ zig build test --watch -fincremental
Then, in another terminal, introduce a compile error:
$ sed -i 's/const/onst/' src/main.zig
Before this commit, this would cause the `zig build` command to repeat
updates forever, separated only by the 50ms debounce interval.
To fix this, when we reset a step with intent to re-run it, we should
clear its inputs immediately, so that if the step is skipped, it is
correctly marked as having no inputs.
This relates to a more general problem with the file system watching
logic which is that the set of inputs includes files in the cache which
are actually artifacts of other steps. Eventually, the build system
should learn to identify such files and exclude them from the set of
file system inputs. In other words, the fact that a `Run` step depends
on the executable generated by a `Compile` step should be modeled by a
dependency in the build step graph, *not* by a dependency on the path
where the `Compile` step happens to have emitted its executable file.
The active contributors and maintainers of Zig's linker code have
generally found the current linker test harness to be cumbersome. The
tests require a lot of maintenance, but do not provide a lot of
coverage, and when they fail it is painful to troubleshoot.
Furthermore, as part of working on #31691, I don't want to port over the
CheckObject step, because I don't like the code anyway.
The plan forward is to start enhancing `zig objdump` to assist in
linker development, as well as using it as the basis for snapshot
testing.
We absolutely need linker test coverage, but we need to try to improve
these things about the next attempt:
* less effort to create and maintain tests
* less CPU overhead - we should be able to add a lot of tests without
adding a lot of CI time.
* more helpful failures. A failed linker test should provide the next
steps a developer can take to understand why the test failed.
* a goal of porting over all of LLD's test suite, or at least the good
ones.
I'm not going to open an issue to track the lost linker test coverage,
because there was already so much lack of coverage for linker stuff.
However I will open issues to track this lost coverage:
* the deleted checks from test/standalone/glibc_compat/build.zig
* the deleted checks from test/standalone/compiler_rt_panic/build.zig
* the deleted checks from test/standalone/ios/build.zig
Followup to #30769
I grepped for `try .*toOwnedSlice` and checked all of them by hand.
Fixes a bunch of memory leaks removes usages or `errdefer` and `vars` in some places. I also switched array_list.Managed to ArrayList where it was convenient.
Reviewed-on: https://codeberg.org/ziglang/zig/pulls/32001
Reviewed-by: Andrew Kelley <andrew@ziglang.org>
- New Features
-- Multiprocess Fuzzing
The fuzzer now is able to utilize multiple cores. This is controllable
with the `-j` build option. Limited fuzzing still uses one core.
-- Fuzzing Infinite Mode
When provided multiple tests, the fuzzer now switches between them and
prioritizes the most effective and interesting ones. Over time already
explored tests will become barely run compared to tests yielding new
inputs.
-- Crash Dumps
Crashing inputs are now saved to a file indicated by the crash message.
It is recommended to use these files to reproduce the crash using
`std.testing.FuzzInputOptions.corpus` and @embedFile.
- Design
Each fuzzing process is assigned an instance id which has the following
uses:
* In conjunction with the pc hash and running test index, they uniquely
identify input files in the case of a crash.
* It is combined with the test seed for a unique rng seed.
* Instance 0 is solely responsible for syncing the filesystem corpus.
When new inputs are found, they are sent to the build server. It then
distributes the new input to the other instances. Each instance has a
concurrent poller managed by the test runner which sends received
inputs to libfuzzer. (note that this is affected by #31718 and so can
(rarely) deadlock)
For fuzzing infinite mode, the test runner now receives a list of tests
from the build server. The fuzzer runs tests in batches of one second,
approximated in cycles by the previous batch's run speed. Tests finding
new inputs or with few runs are given a higher run chance. The baseline
run chance is based off the recency of the last find and the number of
pcs the test has hit.
Previously, each message requires an unseekable error to be returned
from a syscall before proceeding. Ideally, the code would just pass
around `*std.Io.Writer` instead of `std.Io.File` in the first place, but
even then, you could argue for saving a syscall with `writerStreaming`.
This generates zig ASTs from `testing.Smith` and is based off the
langref's PEG.
The choice to not build the Ast while generating and instead parsing it
afterwards makes the smith more versatile by not being tied to a single
implementation at a cost of efficiency.
Additionally, a new function `boolWeighted` was added to `Smith` due to
its frequent use in `AstSmith`.
The indexes can change between recompilation due to conditional
compilation and compiler quirks. While unit test names are still not a
perfect solution, they are better than indexes.
Andrew Kelley