On 32-bit systems usize is not u64 but it was assumed in multiple places
of the build script.
Sometimes using u64 instead of usize since available and used memory can
exceed usize on 32-bit systems (just like totalSystemMemory)
Example errors:
lib/compiler/build_runner.zig:508:26: error: expected type 'usize', found 'u64'
.available_rss = max_rss,
^~~~~~~
lib/compiler/build_runner.zig:508:26: note: unsigned 32-bit int cannot represent all possible unsigned 64-bit values
referenced by:
callMain [inlined]: lib/std/start.zig:699:88
callMainWithArgs [inlined]: lib/std/start.zig:638:20
posixCallMainAndExit: lib/std/start.zig:590:38
2 reference(s) hidden; use '-freference-trace=5' to see all references
lib/std/Build/WebServer.zig:849:38: error: expected type 'usize', found 'u64'
const buf = gpa.realloc(old_buf, new_len) catch @panic("out of memory");
^~~~~~~
lib/std/Build/WebServer.zig:849:38: note: unsigned 32-bit int cannot represent all possible unsigned 64-bit values
lib/std/mem/Allocator.zig:399:58: note: parameter type declared here
pub fn realloc(self: Allocator, old_mem: anytype, new_n: usize) Error!@TypeOf(old_mem) {
^~~~~
build.zig:548:10: error: type 'usize' cannot represent integer value '9300000000'
.max_rss = 9_300_000_000,
~^~~~~~~~~~~~~~~~~~~~~~~
build.zig:778:10: error: type 'usize' cannot represent integer value '8000000000'
.max_rss = 8_000_000_000,
~^~~~~~~~~~~~~~~~~~~~~~~
Currently, neither configurer nor Maker is aware of the standard zig
package path, and the root path is stored as a bare string rather than
relative to a known base directory. Without changing that, we must
construct a cwd relative path here rather than using knowledge of the
standard package path plus package hash.
Also fixes a bug that would have been prevented by implementing the
accepted proposal https://github.com/ziglang/zig/issues/25315
provides a way for the build system to append -target and -isystem/-I
flags to a Run step.
needed by translate-c package to avoid doing naughty stuff in the
configure phase.
It is generally best practice to avoid calling this function, instead
relying on the user to provide these paths via the standard build system
interface. However, when integrating with other build systems, the user
may have already provided the information to the other build system, and
thus it is desirable to use that same information without requiring the
user to provide it again.
options which are passed to configurer and therefore observable by the
build script are added to the cache hash. A sorted list is hashed since
they are unordered.
I had this idea to make b.dupe() also intern the strings since they will
be ultimately serialized to Configuration. Unfortunately the idea does
not work, because although a process-lived arena is used for the
string_bytes ArrayList of the Configuration.Wip, when the ArrayList is
resized, Allocator.free() memsets the freed memory to undefined, even
though it still technically lives due to being in a process-scoped
arena. So this commit will need to be partially reverted. However, I
kept it for posterity, and there are some more changes which I will now
note below.
- dupePaths: don't rewrite backslashes to forward slashes. backslashes
are valid in filenames on non-windows systems.
- always compile configurer in single-threaded mode
- use arena allocator for everything, no gpa for anything
- construct the Configuration.Wip instance earlier, so some stuff can be
prepopulated as desired.
- don't forget to flush
Number of generated files is recorded in serialized Configuration. Maker
preallocates array of generated files so that loads and stores can be
synchronization-free (protected by the dependency tree ordering).
More progress on Compile Step Zig CLI lowering.
- New Features
-- Multiprocess Fuzzing
The fuzzer now is able to utilize multiple cores. This is controllable
with the `-j` build option. Limited fuzzing still uses one core.
-- Fuzzing Infinite Mode
When provided multiple tests, the fuzzer now switches between them and
prioritizes the most effective and interesting ones. Over time already
explored tests will become barely run compared to tests yielding new
inputs.
-- Crash Dumps
Crashing inputs are now saved to a file indicated by the crash message.
It is recommended to use these files to reproduce the crash using
`std.testing.FuzzInputOptions.corpus` and @embedFile.
- Design
Each fuzzing process is assigned an instance id which has the following
uses:
* In conjunction with the pc hash and running test index, they uniquely
identify input files in the case of a crash.
* It is combined with the test seed for a unique rng seed.
* Instance 0 is solely responsible for syncing the filesystem corpus.
When new inputs are found, they are sent to the build server. It then
distributes the new input to the other instances. Each instance has a
concurrent poller managed by the test runner which sends received
inputs to libfuzzer. (note that this is affected by #31718 and so can
(rarely) deadlock)
For fuzzing infinite mode, the test runner now receives a list of tests
from the build server. The fuzzer runs tests in batches of one second,
approximated in cycles by the previous batch's run speed. Tests finding
new inputs or with few runs are given a higher run chance. The baseline
run chance is based off the recency of the last find and the number of
pcs the test has hit.
Remove the RemoveDir step with no replacement. This step had no valid
purpose. Mutating source files? That should be done with
UpdateSourceFiles step. Deleting temporary directories? That required
creating the tmp directories in the configure phase which is broken.
Deleting cached artifacts? That's going to cause problems.
Similarly, remove the `Build.makeTempPath` function. This was used to
create a temporary path in the configure place which, again, is the
wrong place to do it.
Instead, the WriteFile step has been updated with more functionality:
tmp mode: In this mode, the directory will be placed inside "tmp" rather
than "o", and caching will be skipped. During the `make` phase, the step
will always do all the file system operations, and on successful build
completion, the dir will be deleted along with all other tmp
directories. The directory is therefore eligible to be used for
mutations by other steps. `Build.addTempFiles` is introduced to
initialize a WriteFile step with this mode.
mutate mode: The operations will not be performed against a freshly
created directory, but instead act against a temporary directory.
`Build.addMutateFiles` is introduced to initialize a WriteFile step with
this mode.
`Build.tmpPath` is introduced, which is a shortcut for
`Build.addTempFiles` followed by `WriteFile.getDirectory`.
* give Cache a gpa rather than arena because that's what it asks for
this gets the build runner compiling again on linux
this work is incomplete; it only moves code around so that environment
variables can be wrangled properly. a future commit will need to audit
the cancelation and error handling of this moved logic.
pentuppup