Having the matrix of test targets for incremental compilation in the
individual test manifests has turned out to be inconvenient for a few
reasons: the tests are almost certain to accidentally get out of sync,
disabling targets entirely is annoying to do, and incr-check needs to
take care to print the target in all error messages (which currently
does not always happen).
If I recall correctly, I originally designed it this way because it
allows targets to be disabled at the granularity of individual tests,
but there's an easier approach to that: just let a test manifest that it
should be *skipped* on a certain target! As skipping is the rare case,
and also the case you want readers to notice, it makes sense for *it* to
be explicitly specified, like how unit tests use `error.SkipZigTest`.
So, `incr-check` no longer runs through a list of targets specified in
the manifest. Instead, it accepts (and, in fact, requires) a single
target on the command line, and runs the test for that specific target.
If the file contains a `#skip_target` directive for that target, then
`incr-check` exits immediately, so we can still disable targets at
individual test granularity, but you can also disable a target for all
tests by just commenting it out of the matrix in `test/tests.zig`.
As a nice bonus, this also allows the build system to run different
incremental test targets in parallel, because the targets are now
different steps.
This definitely seems like a better way to split the work between the
build system and incr-check---sorry for getting this wrong initially!
In Zig standard library, Dir means an open directory handle. path
represents a file system identifier string. This function is better
named after "current path" than "current dir". "get" and "working" are
superfluous.
Change the log implementation to prepend the current target and update
to all logs which happen during an update.
Makes progress on https://github.com/ziglang/zig/issues/22510, but does
not fully resolve it.
Since the child process is spawned with the tmp directory as its CWD, the child process opens it without DELETE access. On error, the child process would still be alive while the tmp directory is attempting to be deleted, so it would fail with `.SHARING_VIOLATION => return error.FileBusy`.
Fixes arguably the least important part of #22510, since it's only the directory itself that would fail to get deleted, all the files inside would get deleted just fine.
`std.Io.tty.Config.detect` may be an expensive check (e.g. involving
syscalls), and doing it every time we need to print isn't really
necessary; under normal usage, we can compute the value once and cache
it for the whole program's execution. Since anyone outputting to stderr
may reasonably want this information (in fact they are very likely to),
it makes sense to cache it and return it from `lockStderrWriter`. Call
sites who do not need it will experience no significant overhead, and
can just ignore the TTY config with a `const w, _` destructure.
Previously, various doc comments heavily disagreed with the
implementation on both what lives where on the filesystem at what time,
and how that was represented in code. Notably, the combination of emit
paths outside the cache and `disable_lld_caching` created a kind of
ad-hoc "cache disable" mechanism -- which didn't actually *work* very
well, 'most everything still ended up in this cache. There was also a
long-standing issue where building using the LLVM backend would put a
random object file in your cwd.
This commit reworks how emit paths are specified in
`Compilation.CreateOptions`, how they are represented internally, and
how the cache usage is specified.
There are now 3 options for `Compilation.CacheMode`:
* `.none`: do not use the cache. The paths we have to emit to are
relative to the compiler cwd (they're either user-specified, or
defaults inferred from the root name). If we create any temporary
files (e.g. the ZCU object when using the LLVM backend) they are
emitted to a directory in `local_cache/tmp/`, which is deleted once
the update finishes.
* `.whole`: cache the compilation based on all inputs, including file
contents. All emit paths are computed by the compiler (and will be
stored as relative to the local cache directory); it is a CLI error to
specify an explicit emit path. Artifacts (including temporary files)
are written to a directory under `local_cache/tmp/`, which is later
renamed to an appropriate `local_cache/o/`. The caller (who is using
`--listen`; e.g. the build system) learns the name of this directory,
and can get the artifacts from it.
* `.incremental`: similar to `.whole`, but Zig source file contents, and
anything else which incremental compilation can handle changes for, is
not included in the cache manifest. We don't need to do the dance
where the output directory is initially in `tmp/`, because our digest
is computed entirely from CLI inputs.
To be clear, the difference between `CacheMode.whole` and
`CacheMode.incremental` is unchanged. `CacheMode.none` is new
(previously it was sort of poorly imitated with `CacheMode.whole`). The
defined behavior for temporary/intermediate files is new.
`.none` is used for direct CLI invocations like `zig build-exe foo.zig`.
The other cache modes are reserved for `--listen`, and the cache mode in
use is currently just based on the presence of the `-fincremental` flag.
There are two cases in which `CacheMode.whole` is used despite there
being no `--listen` flag: `zig test` and `zig run`. Unless an explicit
`-femit-bin=xxx` argument is passed on the CLI, these subcommands will
use `CacheMode.whole`, so that they can put the output somewhere without
polluting the cwd (plus, caching is potentially more useful for direct
usage of these subcommands).
Users of `--listen` (such as the build system) can now use
`std.zig.EmitArtifact.cacheName` to find out what an output will be
named. This avoids having to synchronize logic between the compiler and
all users of `--listen`.
Compile log output is now separated based on the `AnalUnit` which
perfomred the `@compileLog` call, so that we can omit the output for
unreferenced ("dead") units. The units are also sorted when collecting
the `ErrorBundle`, so that compile logs are always printed in a
consistent order, like compile errors are. This is important not only
for incremental compilation, but also for parallel analysis.
Resolves: #23609
The real problem here is that Git for Windows has horrendous defaults
which convert LF to CRLF. However, rather than changing this
configuration on the CI runners, it's worth supporting inexplicable CRLF
in these files so that anyone else cloning Zig on Windows doesn't get
unexpected test failures.
Uses of `@embedFile` register dependencies on the corresponding
`Zcu.EmbedFile`. At the start of every update, we iterate all embedded
files and update them if necessary, and invalidate the dependencies if
they changed.
In order to properly integrate with the lazy analysis model, failed
embed files are now reported by the `AnalUnit` which actually used
`@embedFile`; the filesystem error is stored in the `Zcu.EmbedFile`.
An incremental test is added covering incremental updates to embedded
files, and I have verified locally that dependency invalidation is
working correctly.
If no external executor is available for a successful binary, its
execution is silently skipped. This allows the CI to test, to the
fullest extent possible, incremental cross-compilation to targets whose
binaries can't be executed on the host.
This is contained in the `test` step, so is tested by CI.
This commit also includes some enhancements to the `incr-check` tool to
make this work correctly.
Matthew Lugg