set `rust.channel` properly in source tarballs
This change sets the appropriate channel by default when using nightly, beta or stable source tarballs.
Fixes#124618
`x86_64-unknown-linux-gnu` has switched to using the self-contained linker
by default (unless asked not to), so we have to build rust-lld:
- when we build our own llvm
- when we use download-ci-llvm
- otherwise, when using an external llvm we can't enable it
This profile has only undergone minimal tweaks since it was originally
drafted. I asked a number of compiler contributors and they said they
set rust.debug explicitly. This was even true for one contributor that
set `rust.debug` = false! Almost everyone seems slightly surprised that
`rust.debug = true` is not the default.
However, adding full debuginfo at this level costs multiple gigabytes!
We can still get much better debuginfo by setting "line-tables-only"
at the cost of only 150~200 MB.
use key-value format in stage0 file
Currently, we are working on the python removal task on bootstrap. Which means we have to extract some data from the stage0 file using shell scripts. However, parsing values from the stage0.json file is painful because shell scripts don't have a built-in way to parse json files.
This change simplifies the stage0 file format to key-value pairs, which makes it easily readable from any environment.
See the zulip thread for more details: https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Using.20different.20format.20in.20the.20stage0.20file
Display walltime benchmarks with subnanosecond precision
With modern CPUs running at more than one cycle per nanosecond the current precision is insufficient to resolve differences worth several cycles per iteration.
Granted, walltime benchmarks often are noisy but occasionally, especially when no allocations are involved, the difference really is just a few cycles.
example results when benchmarking 1-4 serialized ADD instructions and an empty bench body
```
running 4 tests
test add ... bench: 0.24 ns/iter (+/- 0.00)
test add2 ... bench: 0.48 ns/iter (+/- 0.01)
test add3 ... bench: 0.72 ns/iter (+/- 0.01)
test add4 ... bench: 0.96 ns/iter (+/- 0.01)
test empty ... bench: 0.24 ns/iter (+/- 0.00)
```
Upgrade pre-built Clang used in MSVC and MacOS builds, move MSVC builds to Server 2022
Fixes#92948
Example working MacOS and Windows builds: <https://github.com/rust-lang/rust/actions/runs/8989360201>
There is a [bug in Clang 18](https://github.com/llvm/llvm-project/pull/81849) that causes issues when building for Arm64 in later parts of the build (specifically `libgit2`). As a workaround, we will still use the pre-built Clang to build LLVM but will use MSVC for the rest of the Arm64 build.
Ignore empty RUSTC_WRAPPER in bootstrap
This change ignores the RUSTC_WRAPPER_REAL environment variable if it's set to the empty string. This matches cargo behaviour and allows users to easily shadow a globally set RUSTC_WRAPPER (which they might have set for non-rustc projects).
I hit this because I have RUSTC_WRAPPER set to `sccache` in my fish universal env vars, and I can only shadow those locally with an empty string, I can't unset it entirely.
This change ignores the RUSTC_WRAPPER_REAL environment variable if it's
set to the empty string. This matches cargo behaviour and allows users
to easily shadow a globally set RUSTC_WRAPPER (which they might have set
for non-rustc projects).
Reason:
In order to build the Windows version of the Rust toolchain for the Android platform, the following patch to the cc is crate is required to avoid incorrectly determining that we are building with the Android NDK: https://github.com/rust-lang/cc-rs/commit/57853c4bf8a89a0f4c9137eb367ac580305c6919
This patch is present in version 1.0.80 and newer versions of the cc crate. The rustc source distribution currently has 3 different versions of cc in the vendor directory, only one of which has the necessary fix.
We (the Android Rust toolchain) are currently maintaining local patches to upgrade the cc crate dependency versions, which we would like to upstream.
Furthermore, beyond the specific reason, the cc crate in bootstrap is currently pinned at an old version due to problems in the past when trying to update it. It is worthwhile to figure out and resolve these problems so we can keep the dependency up-to-date.
Other fixes:
As of cc v1.0.78, object files are prefixed with a 16-character hash.
Update src/bootstrap/src/core/build_steps/llvm.rs to account for this to
avoid failures when building libunwind and libcrt. Note that while the hash
prefix was introduced in v1.0.78, in order to determine the names of the
object files without scanning the directory, we rely on the compile_intermediates
method, which was introduced in cc v1.0.86
As of cc v1.0.86, compilation on MacOS uses the -mmacosx-version-min flag.
A long-standing bug in the CMake rules for compiler-rt causes compilation
to fail when this flag is specified. So we add a workaround to suppress this
flag.
Updating to cc v1.0.91 and newer requires fixes to bootstrap unit tests.
The unit tests use targets named "A", "B", etc., which fail a validation
check introduced in 1.0.91 of the cc crate.
Fix bootstrap panic when build from tarball
Got this error when build from beta tarball (2024-05-03)
regression of https://github.com/rust-lang/rust/commit/6f4f39a8d56968a1ea120e6903c0640eb2a13ee9
panic info:
```
thread 'main' panicked at src/core/builder.rs:583:25:
`should_run.paths` should correspond to real on-disk paths - use `alias` if there is no relevant path: src/llvm-project/compiler-rt/lib/crt
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: bootstrap::core::builder::ShouldRun::paths::{{closure}}
at ./src/bootstrap/src/core/builder.rs:583:25
3: core::iter::adapters::map::map_fold::{{closure}}
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/iter/adapters/map.rs:89:28
4: <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::fold
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/slice/iter/macros.rs:232:27
5: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/iter/adapters/map.rs:129:9
6: core::iter::traits::iterator::Iterator::for_each
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/iter/traits/iterator.rs:817:9
7: alloc::vec::Vec<T,A>::extend_trusted
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/vec/mod.rs:3020:17
8: <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/vec/spec_extend.rs:26:9
9: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/vec/spec_from_iter_nested.rs:62:9
10: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/vec/spec_from_iter.rs:33:9
11: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/vec/mod.rs:2894:9
12: core::iter::traits::iterator::Iterator::collect
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/iter/traits/iterator.rs:2003:9
13: <alloc::collections::btree::set::BTreeSet<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/collections/btree/set.rs:1191:34
14: core::iter::traits::iterator::Iterator::collect
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/iter/traits/iterator.rs:2003:9
15: bootstrap::core::builder::ShouldRun::paths
at ./src/bootstrap/src/core/builder.rs:578:13
16: bootstrap::core::builder::ShouldRun::path
at ./src/bootstrap/src/core/builder.rs:562:9
17: <bootstrap::core::build_steps::llvm::CrtBeginEnd as bootstrap::core::builder::Step>::should_run
at ./src/bootstrap/src/core/build_steps/llvm.rs:1174:9
18: bootstrap::core::builder::StepDescription::run::{{closure}}
at ./src/bootstrap/src/core/builder.rs:416:25
19: core::iter::adapters::map::map_fold::{{closure}}
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/iter/adapters/map.rs:89:28
20: <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::fold
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/slice/iter/macros.rs:232:27
21: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/iter/adapters/map.rs:129:9
22: core::iter::traits::iterator::Iterator::for_each
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/iter/traits/iterator.rs:817:9
23: alloc::vec::Vec<T,A>::extend_trusted
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/vec/mod.rs:3020:17
24: <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/vec/spec_extend.rs:26:9
25: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/vec/spec_from_iter_nested.rs:62:9
26: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/vec/spec_from_iter.rs:33:9
27: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/vec/mod.rs:2894:9
28: core::iter::traits::iterator::Iterator::collect
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/iter/traits/iterator.rs:2003:9
29: bootstrap::core::builder::StepDescription::run
at ./src/bootstrap/src/core/builder.rs:414:27
30: bootstrap::core::builder::Builder::run_step_descriptions
at ./src/bootstrap/src/core/builder.rs:1047:9
31: bootstrap::core::builder::Builder::execute_cli
at ./src/bootstrap/src/core/builder.rs:1028:9
32: bootstrap::Build::build
at ./src/bootstrap/src/lib.rs:683:17
33: bootstrap::main
at ./src/bootstrap/src/bin/main.rs:79:5
34: core::ops::function::FnOnce::call_once
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
```
add support to override lldb binary path for ./x test
When running debuginfo tests I couldn't set custom build of lldb. The `src/bootstrap/src/core/build_steps/test.rs` has "lldb" hardcoded. I ended up hacking `src/bootstrap/src/core/build_steps/test.rs` just to get the tests running the way I wanted.
Then I've found out that we can override `gdb` under [build] section. This PR enables the same for `lldb`
Change `SIGPIPE` ui from `#[unix_sigpipe = "..."]` to `-Zon-broken-pipe=...`
In the stabilization [attempt](https://github.com/rust-lang/rust/pull/120832) of `#[unix_sigpipe = "sig_dfl"]`, a concern was [raised ](https://github.com/rust-lang/rust/pull/120832#issuecomment-2007394609) related to using a language attribute for the feature: Long term, we want `fn lang_start()` to be definable by any crate, not just libstd. Having a special language attribute in that case becomes awkward.
So as a first step towards the next stabilization attempt, this PR changes the `#[unix_sigpipe = "..."]` attribute to a compiler flag `-Zon-broken-pipe=...` to remove that concern, since now the language is not "contaminated" by this feature.
Another point was [also raised](https://github.com/rust-lang/rust/pull/120832#issuecomment-1987023484), namely that the ui should not leak **how** it does things, but rather what the **end effect** is. The new flag uses the proposed naming. This is of course something that can be iterated on further before stabilization.
Tracking issue: https://github.com/rust-lang/rust/issues/97889
Allow fmt to run on rmake.rs test files
As discussed with `@jieyouxu,` `rmake.rs` from the `run-make` testsuite would benefit from being formatted as well.
Only thing needed to be done for it to work: allow support for `!` in our `rustfmt.toml` file parsing.
r? `@onur-ozkan`
handle the targets that are missing in stage0
During sanity checks, we search for target names to determine if they exist in the compiler's built-in target list (`rustc --print target-list`). While a target name may be present in the stage2 compiler, it might not yet be included in stage0. This PR handles that difference.
Follow-up of https://github.com/rust-lang/rust/pull/123546
In the stabilization attempt of `#[unix_sigpipe = "sig_dfl"]`, a concern
was raised related to using a language attribute for the feature: Long
term, we want `fn lang_start()` to be definable by any crate, not just
libstd. Having a special language attribute in that case becomes
awkward.
So as a first step towards towards the next stabilization attempt, this
PR changes the `#[unix_sigpipe = "..."]` attribute to a compiler flag
`-Zon-broken-pipe=...` to remove that concern, since now the language
is not "contaminated" by this feature.
Another point was also raised, namely that the ui should not leak
**how** it does things, but rather what the **end effect** is. The new
flag uses the proposed naming. This is of course something that can be
iterated on further before stabilization.
Write `git-commit-{sha,info}` for Cargo in source tarballs
Right now Cargo doesn't populate the commit hash or date in its version output when it's built from the plain source tarball. That's because we don't include the git information for it, and Cargo's build script doesn't pick it up.
This PR *partially* solves the problem by storing the git information for Cargo in `src/tools/cargo` in the plain source tarball. We store separate information because even when built in CI Cargo uses its own git information rather than Rust's.
This PR will also require a change in the Cargo repository to consume this information (https://github.com/rust-lang/cargo/pull/13832), but it doesn't have to be blocked on the Cargo PR being merged.
drop deprecated value `if-available` for `download-ci-llvm` option
It's been 5 months since we deprecated this. It should be fine to drop its support now.
bootstrap: Document `struct Builder` and its fields
I'm exploring the code of bootstrap and had a bit of a hard time understanding exactly what `Builder` is for at first. I decided to help document it and its field to help future explorers.
bootstrap: Describe build_steps modules
One of my preferred ways to understand source code is to start with its API. This implies the code is documented reasonably accurately, even if it is a private API. The description of one of these modules had not been updated since 2015 and so was both terse and confusing, so I rewrote it. Then I noticed many others went unremarked, so I offered some remarks.
`x vendor`
This PR implements `x vendor` on bootstrap; enabling dependency vendoring without the need for developers to have `cargo` installed on their system (previously, we suggested running `cargo vendor ...` but now we can accomplish the same task with `x vendor`).
In addition, fixes#112391 problem.
Rollup of 7 pull requests
Successful merges:
- #124370 (Fix substitution parts having a shifted underline in some cases)
- #124394 (Fix ICE on invalid const param types)
- #124425 (Do not ICE on invalid consts when walking mono-reachable blocks)
- #124434 (Remove lazycell and once_cell from compiletest dependencies)
- #124437 (doc: Make the `mod.rs` in the comment point to the correct location)
- #124443 (Elaborate in comment about `statx` probe)
- #124445 (bootstrap: Change `global(true)` to `global = true` for flags for consistency)
r? `@ghost`
`@rustbot` modify labels: rollup
bors