Support excluding the generation of the standalone docs
For Ferrocene we need to exclude the generation of the standalone docs (which include the index page, which we want to replace with our own), but with the way bootstrap is currently implemented that proved not possible. This PR aims to support that.
The first problem is that the `doc::Standalone` step did two things: it generated the "standalone" documentation (which includes the index page and all the pages at the root of the documentation tree), but it also generated some files like `rust.css` and `version_info.html` that other step like `doc::TheBook` required. This meant generating the book required generating the index page, which made disabling the index page generation problematic.
The approach I took to fix the first problem is to split the step into `doc::Standalone` and `doc::SharedAssets`, with `doc::TheBook` now depending on `doc::SharedAssets`.
The second problem is that disabling the `doc::Standalone` proved to be tricky due to its path, `src/doc`. The path is accurate, as the source files for that step are `src/doc/*.md`. The problem is, bootstrap treats `--exclude` as a *suffix*, and so it also excluded the Cargo book whose source lives at `src/tools/cargo/src/doc`.
The approach I took to fix the second problem is to add the `standalone` path in addition to `src/doc`, so that you can pass `--exclude standalone`. I'm not fully happy with the solution, and the other idea I had was to just move the standalone docs source code to `src/doc/standalone`. I feel that second approach is cleaner, but also requires more changes and might require more consensus.
This PR is best reviewed commit-by-commit.
r? `@jyn514`
update Miri
I had to use a hacked version of josh to create this, so let's be careful with merging this and maybe wait a bit to see if the josh issue becomes more clear. But the history looks good to me, we are not adding duplicates of rustc commits that were previously mirrored to Miri.
Also I want to add some cross-testing of Miri in x.py.
All tools meant to be shipped with host toolchains only should be marked
as `ONLY_HOSTS = true`, but rust-analyzer was marked as `ONLY_HOSTS =
false` incorrectly. This meant that bootstrap attempted to build
rust-analyzer for cross-compilation-only targets, causing errors because
libstd is not present on some of them.
It will still be possible to cross-compile rust-analyzer by passing a
different --host flag to ./x, like you can cross-compile other tools.
tools/remote-test-{server,client}: Use /data/local/tmp on Android
The /data/tmp directory does not exist, at least not on recent versions of Android, which currently leads to test failures on that platform. I checked a virtual device running AOSP master and a Nexus 5 running Android Marshmallow and on both devices the /data/tmp directory does not exist and /data/local/tmp does, so let's switch to /data/local/tmp.
update to syn-1.0.102
This update removes the only `.gitignore` found in `rustc-src`:
vendor/syn/tests/.gitignore
vendor/syn-1.0.91/tests/.gitignore
vendor/syn-1.0.95/tests/.gitignore
To check-in `rustc-src` for hermetic builds in environments with
restrictive `.gitignore` policies, one has to remove these
`tests/.gitignore` and patch the respective
`.cargo-checksum.json`.`syn` >1.0.101 includes dtolnay/syn@3c49303bed,
which removes its `tests/.gitignore`. Now the `syn` crates.io package
has no `.gitignore`.
[`rustc-src`'s `vendor`][] is produced from the root `Cargo.toml`,
`src/tools/rust-analyzer/Cargo.toml`,
`compiler/rustc_codegen_cranelift/Cargo.toml`, and
`src/bootstrap/Cargo.toml`. `rustc_codegen_cranelift` does not use
`syn`.
[`rustc-src`'s `vendor`]:
https://github.com/rust-lang/rust/blob/c0784109daa0/src/bootstrap/dist.rs#L934-L940
This was produced with:
cargo update --package syn --precise 1.0.102 \
cargo update --package syn --precise 1.0.102 \
--manifest-path src/tools/rust-analyzer/Cargo.toml
cargo update --package syn --precise 1.0.102 \
--manifest-path src/bootstrap/Cargo.toml
Use BOLT in CI to optimize LLVM
This PR adds an optimization step in the Linux `dist` CI pipeline that uses [BOLT](https://github.com/llvm/llvm-project/tree/main/bolt) to optimize the `libLLVM.so` library built by boostrap.
Steps:
- [x] Use LLVM 15 as a bootstrap compiler and use it to build BOLT
- [x] Compile LLVM with support for relocations (`-DCMAKE_SHARED_LINKER_FLAGS="-Wl,-q"`)
- [x] Gather profile data using instrumented LLVM
- [x] Apply profile to LLVM that has already been PGOfied
- [x] Run with BOLT profiling on more benchmarks
- [x] Decide on the order of optimization (PGO -> BOLT?)
- [x] Decide how we should get `bolt` (currently we use the host `bolt`)
- [x] Clean up
The latest perf results can be found [here](https://github.com/rust-lang/rust/pull/94381#issuecomment-1258269440). The current CI build time with BOLT applied is around 1h 55 minutes.
Update CI to use Android NDK r25b
This commit updates the CI definitions to use the most recent Android LTS NDK release: r25b. Changes since the last NDK used by Rust negate the need to generate "standalone toolchains" and newer NDKs can be used in-place.
See https://developer.android.com/ndk/guides/other_build_systems#overview
Fix llvm-tblgen for cross compiling
- Let llvm-config tell us where to find its tools
- Add llvm-tblgen to rust-dev for cross-compiling
Fixes#86890.
r? ````@jyn514````
Make the `config.src` handling for downloadable bootstrap more conservative
In particular, this supports build directories within an unrelated git repository. Fixes https://github.com/rust-lang/rust/issues/102562.
As a side effect, it will fall back to the old logic when the source directory is being built from a tarball within an unrelated git repository. However, that second case is unsupported and untested; we reserve the right to break it in the future.
`@cr1901` can you confirm this fixes your problem?
cc `@kleisauke,` I believe this will also fix your issue (although your use case still isn't supported).
r? `@Mark-Simulacrum`
The /data/tmp directory does not exist, at least not on recent versions
of Android, which currently leads to test failures on that platform. I
checked a virtual device running AOSP master and a Nexus 5 running
Android Marshmallow and on both devices the /data/tmp directory does
not exist and /data/local/tmp does, so let's switch to /data/local/tmp.
Remove `-Ztime`
Because it has a lot of overlap with `-Ztime-passes` but is generally less useful. Plus some related cleanups.
Best reviewed one commit at a time.
r? `@davidtwco`
The compiler currently has `-Ztime` and `-Ztime-passes`. I've used
`-Ztime-passes` for years but only recently learned about `-Ztime`.
What's the difference? Let's look at the `-Zhelp` output:
```
-Z time=val -- measure time of rustc processes (default: no)
-Z time-passes=val -- measure time of each rustc pass (default: no)
```
The `-Ztime-passes` description is clear, but the `-Ztime` one is less so.
Sounds like it measures the time for the entire process?
No. The real difference is that `-Ztime-passes` prints out info about passes,
and `-Ztime` does the same, but only for a subset of those passes. More
specifically, there is a distinction in the profiling code between a "verbose
generic activity" and an "extra verbose generic activity". `-Ztime-passes`
prints both kinds, while `-Ztime` only prints the first one. (It took me
a close reading of the source code to determine this difference.)
In practice this distinction has low value. Perhaps in the past the "extra
verbose" output was more voluminous, but now that we only print stats for a
pass if it exceeds 5ms or alters the RSS, `-Ztime-passes` is less spammy. Also,
a lot of the "extra verbose" cases are for individual lint passes, and you need
to also use `-Zno-interleave-lints` to see those anyway.
Therefore, this commit removes `-Ztime` and the associated machinery. One thing
to note is that the existing "extra verbose" activities all have an extra
string argument, so the commit adds the ability to accept an extra argument to
the "verbose" activities.
Miri sync
This is a Miri sync created with my experimental fork of josh. We should probably not merge this yet, but we can use this to check if the sync looks the way it should.
r? `@oli-obk`
Before this commit, the path for the doc::Standalone step was "src/doc",
which is accurate as the standalone docs source files live at the root
of the "src/doc" directory tree.
Unfortunately, that caused bad interactions when trying to exclude it
with `--exclude src/doc`. When an exclusion is passed to bootstrap, it
will exclude all steps whose path *ends with* the exclusion, which
results in the Cargo book (src/tools/cargo/src/doc) to also be excluded.
To work around this problem, this commit adds the "standalone" path as
an alternate path for doc::Standalone, allowing `--exclude standalone`
to work without side effects.
Before this commit, the step to generate the standalone docs (which
included the index page and other HTML files at the root of the
documentation) was bundled with the code copying files needed by
multiple pieces of documentation. This means it wasn't possible to avoid
generating the standalone docs.
This commit splits the step into two, allowing the standalone docs
generation to be excluded while still building the rest of the docs.
This commit updates the CI definitions to use the most recent Android
LTS NDK release: r25b. Changes since the last NDK used by Rust negate
the need to generate "standalone toolchains" and newer NDKs can be used
in-place.
See https://developer.android.com/ndk/guides/other_build_systems#overview
Package `rust-docs-json` into nightly components (take 3)
`dist` creates a `rust-docs-json.tar.xz` tarfile. But build-manifest expected it to be named `rust-docs-json-preview.tar.xz`. Change build-manifest to allow the name without the `-preview` suffix.
I haven't actually tested this :( build-manifest is a pain to run locally.
rust-lang/rust#100557 removed the `git-commit-hash` file and replaced it
with `git-commit-info`. However, build-manifest relies on the
`git-commit-hash` file being present, so this adds it back.
fix: use git-commit-info for version information
Fixes#33286.
Fixes#86587.
This PR changes the current `git-commit-hash` file that `./x.py` dist puts in the `rustc-{version}-src.tar.{x,g}z` to contain the hash, the short hash, and the commit date from which the tarball was created, assuming git was available when it was. It uses this for reading the version so that rustc has all the appropriate metadata.
# Testing
Testing this is kind of a pain. I did it with something like
```sh
./x.py dist # ensure that `ignore-git` is `false` in config.toml
cp ./build/dist/rustc-1.65.0-dev-src.tar.gz ../rustc-1.65.0-dev-src.tar.gz
cd .. && tar -xzf rustc-1.65.0-dev-src && cd rustc-1.65.0-dev-src
./x.py build
```
Then, the output of `rustc -vV` with the stage1 compiler should have the `commit-hash` and `commit-date` fields filled, rather than be `unknown`. To be completely sure, you can use `rustc --sysroot` with the stdlib that the original `./x.py dist` made, which will require that the metadata matches.
In particular, this supports build directories within an unrelated git repository.
As a side effect, it will fall back to the old logic when the source directory is being built from a tarball within an unrelated git repository.
However, that second case is unsupported and untested; we reserve the right to break it in the future.
fix issue with x.py setup running into explicit panic
Fixes problem with [Issue #102555](https://github.com/rust-lang/rust/issues/102555) causing `x.py` setup to fail. Simply requires `rustfmt` be downloaded a little later.
Allow passing rustix_use_libc cfg using RUSTFLAGS
Before this would error with
```
error: unexpected `rustix_use_libc` as condition name
|
= note: `-D unexpected-cfgs` implied by `-D warnings`
= help: was set with `--cfg` but isn't in the `--check-cfg` expected names
```
I'm setting rustix_use_libc when testing bootstrapping rustc with cg_clif as I'm disabling inline asm here.
Change argument handling in `remote-test-server` and add new flags
This PR updates `remote-test-server` to add two new flags:
* `--sequential` disables parallel test execution, accepting one connection at the time instead. We need this for Ferrocene as one of our emulators occasionally deadlocks when running multiple tests in parallel.
* `--bind <ip:port>` allows customizing the IP and port `remote-test-server` binds to, rather than using the default value.
While I was changing the flags, and [after chatting on what to do on Zulip](https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/remote-test-server.20flags), I took this opportunity to cleanup argument handling in `remote-test-server`, which is a breaking change:
* The `verbose` argument has been renamed to the `--verbose` flag.
* The `remote` argument has been removed in favor of the `--bind 0.0.0.0:12345` flag. The only thing the argument did was to change the bound IP to 0.0.0.0, which can easily be replicated with `--bind` and also is not secure as our "remote" default.
I'm also open to keep the old arguments with deprecation warnings.
r? `@Mark-Simulacrum`
Dylan DPC