- New Features
-- Multiprocess Fuzzing
The fuzzer now is able to utilize multiple cores. This is controllable
with the `-j` build option. Limited fuzzing still uses one core.
-- Fuzzing Infinite Mode
When provided multiple tests, the fuzzer now switches between them and
prioritizes the most effective and interesting ones. Over time already
explored tests will become barely run compared to tests yielding new
inputs.
-- Crash Dumps
Crashing inputs are now saved to a file indicated by the crash message.
It is recommended to use these files to reproduce the crash using
`std.testing.FuzzInputOptions.corpus` and @embedFile.
- Design
Each fuzzing process is assigned an instance id which has the following
uses:
* In conjunction with the pc hash and running test index, they uniquely
identify input files in the case of a crash.
* It is combined with the test seed for a unique rng seed.
* Instance 0 is solely responsible for syncing the filesystem corpus.
When new inputs are found, they are sent to the build server. It then
distributes the new input to the other instances. Each instance has a
concurrent poller managed by the test runner which sends received
inputs to libfuzzer. (note that this is affected by #31718 and so can
(rarely) deadlock)
For fuzzing infinite mode, the test runner now receives a list of tests
from the build server. The fuzzer runs tests in batches of one second,
approximated in cycles by the previous batch's run speed. Tests finding
new inputs or with few runs are given a higher run chance. The baseline
run chance is based off the recency of the last find and the number of
pcs the test has hit.
The cmpxchg is there to recover alignment padding that isn't needed (which
can only be determined after the fetch-and-add that reserves it as allocated
memory). As cmpxchg tends to be a very expensive operation, it is actually
faster to introduce an additional branch here that checks if the cmpxchg
would be a noop (because all of the reserved alignment padding was in fact
necessary) and skips it if that's the case.
This does not measurably regress performance if the arena is only accessed
by a single thread and yields slight performance benefits for multi-threaded
usage. If the arena is commonly used for unaligned allocations, the perf
benefits are quite significant.
Co-authored-by: Jacob Young <amazingjacob@gmail.com>
The implementations of `std.ArrayList.replaceRange` and `std.ArrayList.replaceRangeAssumeCapacity` were needlessly complicated, called `@memcpy` too many times, and left the structure in a semi-modified state in OOM conditions. This commit fixes that.
Reviewed-on: https://codeberg.org/ziglang/zig/pulls/31710
Reviewed-by: Andrew Kelley <andrew@ziglang.org>
Co-authored-by: Hila Friedman <red.black.liquorice@gmail.com>
Co-committed-by: Hila Friedman <red.black.liquorice@gmail.com>
I hit an unexpected errno 30 on macos attempting to call createFile on a
directory in PATH. errno 30 is EROFS, this change propagates that error
as error.ReadOnlyFileSystem.
This isn't causing any functional problem today, but technically
`mapped[tls_offset..]` runs past the tls part of `mapped` and into
the `Instance` storage, and currently `prepareArea()` memsets its
entire argument to zero. It is only the current layout and
initialization order of `mapped` that prevents this from being a
problem. Being more precise here avoids future footguns if any of
that changes.
The main goal here is to make incremental compilation work a bit better.
I also slightly expanded some `std.zig.llvm.Builder` APIs so that we
don't need to pointlessly create new `Global`s whenever e.g. a function
turns into a variable or vice versa.
Also, lean into aliases for exports! If we just use aliases for every
export, everything becomes simpler. Besides, we can't just go around
renaming the globals of `Nav`s: the export could disappear on a future
update, in which case we'd have to somehow revert that change, which is
easier said than done.
RFC 9110 says
> It is RECOMMENDED that all senders and recipients support, at a minimum,
> URIs with lengths of 8000 octets in protocol elements.
closes#30207
Previously, each message requires an unseekable error to be returned
from a syscall before proceeding. Ideally, the code would just pass
around `*std.Io.Writer` instead of `std.Io.File` in the first place, but
even then, you could argue for saving a syscall with `writerStreaming`.
This problem also affected determining if an expression became multiline
as that depends on if the line is overindented. As such,
`becomesMultilineExpr` has been replaced by `rendersMultiline` which
constructs a temporary writer which returns `error.WriteFailed` when
newlines are written. This new approach also has the advantage of being
more maintainable.
This was especially bad as it could crash or duplicate them due to
them being moved around "zig fmt: on/off".
As a bonus this fixes a crash from reordering at the start of the file.
Previously, the comments would be lost with the colons.
This required a substantial rewrite of renderAsm to determine how many
colons should be rendered.
- PEG / Parser Changes
All the changes made here are to places where the PEG was more
permissive than the parser. Changes to the parser make it more
permissive and changes to the PEG make it more strict. When choosing
between these two options for discrepancies, I opted for the choice
that was more natural and increased code readability.
Changes to the Parser
* Tuple types can now be `inline` and `extern` (e.g. `extern struct`).
* Break labels are now only consumed if both the colon and identifier
are present instead of failing if there is only a colon.
* Labeled blocks are no longer parsed in PrimaryExpr (so they are now
allowed to have CurlySuffixExpr) as in the PEG.
* While expressions can now be grouped on the same line.
* Added distinction in error messages for "a multiline string literal"
so places where only single string literals are allowed do not give
"expected 'a string literal', found 'a string literal'".
Changes to the PEG
* Made it so extern functions cannot have a body
* Made it so ... can be only the last function argument
* Made it so many item pointers can't have bit alignment
* Made it so asm inputs / outputs can not be multiline string literals
* Added distinction between block-level statements and regular
statements
-- Pointer Qualifier Order
The PEG allowed for duplicated qualifiers, which the parser did not.
The simplest fix for this was to make each be allowed zero or one times
which required giving them a order similar to how FnProto already
works. The chosen order is the same as used by zig fmt. The parser
still accepts them in any order similar to functions.
-- Backtracking
Made it so several places could not backtrack in the PEG. A common
pattern for this was (A / !A).
--- !ExprSuffix
Expressions ending with expressions now have !ExprSuffix after.
This change prevents expressions such as `if (a) T else U{}` being be
parsable as `(if (a) T else U){}`. It also stops some backtracking,
take for example:
`if (a) for (b) |c| d else |e| f`
It may seem at first that the else clause belongs to the `for`, however
it actually belongs to the `if` because for else-clauses cannot have a
payload. This is fixed by a new `KEYWORD_else / !KEYWORD_else`, however
this alone does not fix more complex cases such as:
`if (a) for (b) |c| d() else |e| f`
The PEG would first attempt to parse it as expected but fail due to the
new guard. It will then backtrack to
`if (a) (for (b) |c| d)() else |e| f`
which is surprising but avoids the new gaurd. So, !ExprSuffix is
required to disallow this type of backtracking.
--- !LabelableExpr
For identifiers, excluding labels is necessary despite ordered choice
due to pointer bit alignment. For example `*align(a : b: for (c) e) T`
could backtrack to `*align(a : b : (for (c) e)) T`.
--- !SinglePtrTypeStart
Prevents expressions like `break * break` which is parsed as
`break (*break)` backtracking to `(break) * (break)`
--- !BlockExpr
Prevents expressions like `test { {} = a; }` being backtracked to and
parsed as `test { ({} = a); }` (the parenthesis are just for
demonstration, that expression is not legal either)
--- !ExprStatement
In addition to splitting up block level statements, statements that are
also parsable as expressions are now part of ExprStatement to disallow
backtracking.
This generates zig ASTs from `testing.Smith` and is based off the
langref's PEG.
The choice to not build the Ast while generating and instead parsing it
afterwards makes the smith more versatile by not being tied to a single
implementation at a cost of efficiency.
Additionally, a new function `boolWeighted` was added to `Smith` due to
its frequent use in `AstSmith`.
murtaza