Use `tracing-forest` instead of `tracing-tree` for bootstrap tracing
I find the `tracing-forest` output easier to comprehend.
Note that this is not a strict improvement -- `tracing-forest` output contains some emojis and redundant log levels, but customizing it seems to be... non-trivial. Despite this, I still find `tracing-forest` easier to follow than `tracing-tree`, even when I tried to tune `tracing-tree` output.
### Preview
```bash
BOOTSTRAP_TRACING=debug ./x test library/std --dry-run
```
With `tracing-forest` (this PR), it looks like
With `tracing-tree` (before this PR), it looked like
r? `@Kobzol`
Rename run always
This PR renames run_always to run_to_dry_run for better clarity, making the field's purpose more explicit and avoiding confusion with command caching behavior.
r? ``````@Kobzol``````
make `tidy-alphabetical` use a natural sort
The idea here is that these lines should be correctly sorted, even though a naive string comparison would say they are not:
```
foo2
foo10
```
This is the ["natural sort order"](https://en.wikipedia.org/wiki/Natural_sort_order).
There is more discussion in [#t-compiler/help > tidy natural sort](https://rust-lang.zulipchat.com/#narrow/channel/182449-t-compiler.2Fhelp/topic/tidy.20natural.20sort/with/519111079)
Unfortunately, no standard sorting tools are smart enough to to this automatically (casting some doubt on whether we should make this change). Here are some sort outputs:
```
> cat foo.txt | sort
foo
foo1
foo10
foo2
mp
mp1e2
np",
np1e2",
> cat foo.txt | sort -n
foo
foo1
foo10
foo2
mp
mp1e2
np",
np1e2",
> cat foo.txt | sort -V
foo
foo1
foo2
foo10
mp
mp1e2
np1e2",
np",
```
Disappointingly, "numeric" sort does not actually have the behavior we want. It only sorts by numeric value if the line starts with a number. The "version" sort looks promising, but does something very unintuitive if you look at the final 4 values. None of the other options seem to have the desired behavior in all cases:
```
-b, --ignore-leading-blanks ignore leading blanks
-d, --dictionary-order consider only blanks and alphanumeric characters
-f, --ignore-case fold lower case to upper case characters
-g, --general-numeric-sort compare according to general numerical value
-i, --ignore-nonprinting consider only printable characters
-M, --month-sort compare (unknown) < 'JAN' < ... < 'DEC'
-h, --human-numeric-sort compare human readable numbers (e.g., 2K 1G)
-n, --numeric-sort compare according to string numerical value
-R, --random-sort shuffle, but group identical keys. See shuf(1)
--random-source=FILE get random bytes from FILE
-r, --reverse reverse the result of comparisons
--sort=WORD sort according to WORD:
general-numeric -g, human-numeric -h, month -M,
numeric -n, random -R, version -V
-V, --version-sort natural sort of (version) numbers within text
```
r? ```@Noratrieb``` (it sounded like you know this code?)
Enforce in bootstrap that build must have stage at least 1
This PR is a step towards https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Proposal.20to.20cleanup.20stages.20and.20steps.20after.20the.20redesign/with/523586917. It's very hard or me to make self-contained changes to bootstrap at this moment, so this PR kind of does several things:
1) (first two commits) Try to reduce the usage of `Std::new` in bootstrap, and replace it with a `Builder::std` method (similar to `Builder::compiler`). This is mostly to remove executions of the `Std` step for stage 0, which doesn't make a lot of sense; I'd like to ideally have the invariant that when a step is invoked, it actually builds or does something. Eventually, I'd like for everything to go through `Builder::std`. (Note: I'm not totally married to this idea, if you don't like it, we can remove it from this PR. I mostly did it right now to remove stage 0 std steps from snapshot tests, which shouldn't be there, but we can also filter them out in a different way)
2) Make sure that when you pass `x build compiler`, only the `Assemble` root level step will be invoked, and not the `Rustc` step. Before, both were invoked, which actually ran `Rustc` twice, once with all `crates` filled, and once with no crates (but both actually represent the same situation). Since the `Rustc::make_run` step actually requests a compile that is one stage below it, this actually made `build compiler --stage 0` work, which we don't want to have anymore.
3) Enforce a bootstrap-global invariant that all `build` commands are always on stage `>=1`. If you try to `build` anything on stage 0, it will print a warning and exit bootstrap. This follows the intuition from the new staging rules after the stage redesign; artifacts that are "stage 0" come outside of bootstrap, and we can't really build something for which we don't have source (although we can still test it, but that's for another PR).
Now the logic for build should be quite simple. For pretty much everything except for `Std`, you first use the stage0 compiler to build stage 1. Then you can build a stage 2 <something> using the previously built stage 1 (and then you can continue to stage 3 etc.). And that's it. The default build stage for everything is 1 (modulo download-ci-rustc, but that's a separate can of worms).
The snapshot test infra isn't super nice at the moment, as one of next steps I want to create some simple Builder pattern that will allow us to configure the bootstrap invocations in a more "forward-compatible" way (e.g. now it's not possible to modify the config passed to `configure_with_args`).
There are some things not yet fully resolved for build stage 0:
1) Cargo is still a `ModeRustc` tool, even though it doesn't really have to be, it is buildable with the stage0 compiler
2) bootstrap tools (`opt-dist`, `build-manifest` etc.) are still called stage0 tools, and in the bootstrap output it says something like "stage 0 rustc builds stage 0 opt-dist". Which is a bit weird, but functionally there's no difference, it's just a slightly inconsistent output. We still haven't decided if we should make these tools ignore staging altogether (which is IMO the right choice) or if we want to allow building stage 1/2/3/... bootstrap tools.
r? `@jieyouxu`
try-job: x86_64-rust-for-linux
Turn `stdarch` into a Josh subtree
In a similar vein as https://github.com/rust-lang/rust/pull/141229, this PR makes the `stdarch` repository a Josh subtree (it was previously a submodule). The initial commit of `stdarch` upon this is based is `5a7342fc16b208b1b16624e886937ed8509a6506`, which is the previous commit SHA of the `stdarch` submodule. The sync was performed according to https://hackmd.io/7pOuxnkdQDaL1Y1FQr65xg.
This was decided in https://github.com/rust-lang/stdarch/issues/1655.
Test pull PR on my fork: https://github.com/Kobzol/stdarch/pull/1
Test push PR on my fork: https://github.com/Kobzol/rust/pull/59
I plan to use the same Rust (miri-inspired) tooling that we use for `rustc-dev-guide` to enable pulls/pushes on stdarch.
Note that this repository currently doesn't have any stdarch-specific tests, so before that, the subtree should only be modified through this repository only when dealing with changes that contain "cyclical dependencies" between stdarch and rustc. The long term vision is to integrate stdarch into rust-lang/rust completely.
CC `@Amanieu`
try-job: aarch64-apple
try-job: aarch64-gnu
try-job: `x86_64-msvc-*`
try-job: x86_64-gnu
try-job: x86_64-gnu-aux
Fix hang in --print=file-names in bootstrap
In an interactive context, the subprocess inherited a real tty stdin, which lead it it waiting for something to happen, even though nothing happened. By explicitly passing null as stdin we make sure an empty file is passed, which achieves the desired behavior.
Fixesrust-lang/rust#142926 (verified locally by cherry-picking the patch onto beta where I was building).
In an interactive context, the subprocess inherited a real tty stdin,
which lead it it waiting for something to happen, even though nothing
happened. By explicitly passing null as stdin we make sure an empty file
is passed, which achieves the desired behavior.
It was confusing that the `Rustc` step was invoked twice, once with all crates and once with no crates (even though both of these mean the same thing).
Pass -Cpanic=abort for the panic_abort crate
The panic_abort crate must be compiled with panic=abort, but cargo doesn't allow setting the panic strategy for a single crate the usual way using `panic="abort"`, but luckily per-package rustflags do allow this. Bootstrap previously handled this in its rustc wrapper, but for example the build systems of cg_clif and cg_gcc don't use the rustc wrapper, so they would either need to add one, patch the standard library or be unable to build a sysroot suitable for both panic=abort and panic=unwind (as is currently the case).
Required for https://github.com/rust-lang/rustc_codegen_cranelift/issues/1567
Make sure to rebuild rustdoc if `src/rustdoc-json-types` is changed
I think `rustdoc-json-types` was more recently split out, so this download-rustc logic became outdated as it wasn't tracked. This PR adds `src/rustdoc-json-types` to be tracked for difference versus upstream, so that we properly rebuild rustdoc if it has changes versus upstream.
Fixesrust-lang/rust#142738.
### Local testing
This is not so easy to test locally because it requires download-rustc. To test this, you need to:
1. Disable `download-rustc` inhibition from bootstrap changes versus upstream, by including `:!src/bootstrap` in https://github.com/rust-lang/rust/blob/255aa220821c05c3eac7605fce4ea1c9ab2cbdb4/src/bootstrap/src/core/config/config.rs#L67-L74.
2. Then, use a config like `profile = "tools"` which by default uses `download-rustc = "if-unchanged"`.
3. Run `./x test tests/rustdoc-json` one time, to "prime" initial build caches.
4. Change the `FORMAT_VERSION` in `src/rustdoc-json-types`, i.e.
```diff
diff --git a/src/rustdoc-json-types/lib.rs b/src/rustdoc-json-types/lib.rs
index 1f93895ae07..72a3720c7b4 100644
--- a/src/rustdoc-json-types/lib.rs
+++ b/src/rustdoc-json-types/lib.rs
``@@`` -38,7 +38,7 ``@@``
// are deliberately not in a doc comment, because they need not be in public docs.)
//
// Latest feature: Pretty printing of inline attributes changed
-pub const FORMAT_VERSION: u32 = 48;
+pub const FORMAT_VERSION: u32 = 666;
```
5. Observe that without this patch, `rustdoc-json` tests fail because `FORMAT_VERSION` mismatch. Observe that with this patch, rustdoc gets properly rebuilt and `rustdoc-json` tests pass.
cc ``@aDotInTheVoid``
r? Kobzol
Add config builder for bootstrap tests
I started writing a bunch of snapshot tests for build/check steps, and quickly realized that the current interface for defining them won't be enough, so I created a simple builder, which can scale to pretty much any kind of configuration in the future.
The panic_abort crate must be compiled with panic=abort, but cargo
doesn't allow setting the panic strategy for a single crate the usual
way using panic="abort", but luckily per-package rustflags do allow
this. Bootstrap previously handled this in its rustc wrapper, but for
example the build systems of cg_clif and cg_gcc don't use the rustc
wrapper, so they would either need to add one, patch the standard
library or be unable to build a sysroot suitable for both panic=abort
and panic=unwind (as is currently the case).
Use jemalloc for Clippy
The tool macros are annoying, we should IMO just get rid of them, create separate steps for each tool and (re)use some builders in them to share the build code.
r? `@ghost`
Tshepang Mbambo