When running the `install src` command I'm seeing failures as the
`builder.doc_out(host)` directory does not exist. This is because
`match_paths_to_steps_and_run()` doesn't actually build any
documentation as the `paths.is_empty()` causes an early return. This
results in install failures as the `*/doc` src directory doesn't exist.
This patch ensures that the builder.doc_out(host) directory exists.
This fixes installing the Rust source when `build.docs = false`.
This fixes installing the Rust source code in OpenEmbedded.
Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
Signed-off-by: Jieyou Xu <jieyouxu@outlook.com>
Introduce --ci flag in tidy
## Context
Currently tidy doesn't have `--ci` flag. Even if bootstrap has --ci=true, it isn't passed to tidy. As a result, some parts of tidy where we need to turn on `ci mode` are never executed on our local.
## Change
This PR introduces --ci flag in tidy, and modify CiInfo and other parts of tidy where currently use `CiEnv::is_ci()` for checking if the running environment is ci or not.
* add --ci flag in tidy
This commit introduces --ci flag in tidy because currently bootstrap can't pass its ci env information to tidy. It also modifies how CiInfo initialize its ci_env variable. tidy codes which uses CiEnv::is_ci for checking ci are now using ci_env in CiInfo.
* address review
- Fix comment
- Use Option for ci flag in order to have true/false explicitly or unspecified (implicit false)
* integrate CiInfo into TidyCtx
* remove CiInfo
* CiEnv::current() should be called when ci flag is not added
* extract base_commit() to a separate function
* use &TidyCtx instead of clone
Previously this would sometimes just say "unexpected output ``", which
was extremely unhelpful. The actual issue was that I had a `.jj`
directory but not a `.git` directory, so it couldn't run `git submodule
update.`
Revert relative paths for std links in rustc-docs
Reverts rust-lang/rust#152243.
The relative path `../` passed via `--extern-html-root-url` produces broken links in compiler documentation. The path is wrong in both the published layout (`doc.rust-lang.org/nightly/nightly-rustc/` vs `doc.rust-lang.org/nightly/core/`) and local builds (`compiler-doc/` vs `doc/`).
This restores absolute URLs via `#[doc(html_root_url)]`.
Fixesrust-lang/rust#152917
Re-opens rust-lang/rust#151496
cc @eggyal
Enable rust.remap-debuginfo in the dist profile
Anyone who distributes rustc almost certainly wants to enable this option. It is necessary for reproducibility and for a distributed rustc local paths are useless anyway. And finally it improves privacy if you distribute a local build.
Anyone who distributes rustc almost certainly wants to enable this
option. It is necessary for reproducibility and for a distributed rustc
local paths are useless anyway. And finally it improves privacy if you
distribute a local build.
Add build.rustdoc option to bootstrap config
This adds a bootstrap config option `build.rustdoc` to be able to override the stage0 `rustdoc` binary. When unspecified, this defaults to the existing behavior of using the `rustdoc` binary in the same directory as `rustc`, making this a backward-compatible change.
### Motivation
The existing behavior does not seem to be documented and can be surprising. By adding the new option, the behavior gets documented in `bootstrap.example.toml`.
I ran into this because I was experimenting with a build with a configuration like this:
```
build.rustc = "/usr/bin/rustc-1.92"
build.cargo = "/usr/bin/cargo-1.92"
```
This was on Ubuntu where the packages `rustc-1.92` and `cargo-1.92` place symlinks at these locations. This resulted in failure as the bootstrap process tried to run doc tests on `tidy` using a binary `/usr/bin/rustdoc` which did not exist. It took some digging to understand where that path `/usr/bin/rustdoc` was coming from.
@rustbot label +A-bootstrap-config
Build: Add `stdenv.cc.cc.lib` to Nix dependencies
Otherwise, it systematically fails on:
```
error: process didn't exit successfully: `[...]/rust/build/bootstrap/debug/rustc [...]/rust/build/bootstrap/debug/rustc -vV` (exit status: 127)
--- stderr
[...]/rust/build/x86_64-unknown-linux-gnu/stage1/bin/rustc: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory
```
for us at least. Closesrust-lang/rust#127620.
@rustbot label T-bootstrap A-reproducibility
Add bootstrap snapshot tests for {`install`, `install src`}
And `install src` with `build.docs = false`.
This is mostly to get coverage for the baseline to make it easier to review rust-lang/rust#150845.
bootstrap: respect POSIX jobserver
When bootstrapping Rust, the `-j N` flag was passed to CMake, which was
then forwarded to Ninja. This prevents the jobserver from being used,
and as a result leads to oversubscription when Rust is just one of the
many packages built as part of a larger software stack.
Since Cargo and the Rust compiler have long supported the jobserver, it
would be good if also bootstrapping Rust itself would participate in the
protocol, leading to composable parallelism.
This change allows bootstrapping to respect an existing FIFO based
jobserver. Old pipe based jobservers are not supported, because they are
brittle: currently the Python scripts in bootstrap do not inherit the
file descriptors, but do pass on `MAKEFLAGS`, which has lead to errors
like "invalid file descriptor" in the past. Because Ninja only supports
FIFO based jobservers, it's better to focus on new jobservers only,
which shouldn't suffer from the "invalid file descriptor" issue.
In summary:
* Bootstrap Cargo passes `MAKEFLAGS` verbatim to subprocesses if it
advertises a FIFO style jobserver, otherwise it unsets it. This ensures
subprocesses respect the jobserver during bootstrap.
* `llvm.rs` does not pass `-j` to `cmake` when a FIFO style jobserver is
set in `MAKEFLAGS`. This ensures Ninja respects the jobserver.
* Bootstrap Cargo no longer unsets `MKFLAGS`: from git blame, GNU Make
considered it a historical artifact back in 1992, and it is never read
by GNU Make, it's only set for backwards compatibility in case sub-Makefiles
read it.
---
I've tested this with the [Spack package manager](https://github.com/spack/spack) starting the POSIX jobserver,
building node.js and rust in parallel with `-j16`, which looks like this:
```console
$ pstree 382710
python3─┬─python3
└─python3─┬─python3─┬─make───make───6*[ccache───g++───cc1plus]
│ └─{python3}
└─python3─┬─python3.11───bootstrap───cmake───ninja-build───10*[sh───ccache───g++───cc1plus]
└─{python3}
```
As you can see there are 10 `g++` processes running for rust, and `6` for node.js, and
with a mix of `make` and `ninja` as build tools :).
(The only violation I see now is `rust-lld`, but I think that'll be fixed with the LLVM 23
release)
Otherwise, it systematically fails on:
```
error: process didn't exit successfully: `[...]/rust/build/bootstrap/debug/rustc [...]/rust/build/bootstrap/debug/rustc -vV` (exit status: 127)
--- stderr
[...]/rust/build/x86_64-unknown-linux-gnu/stage1/bin/rustc: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory
```
for us at least.
Signed-off-by: Paul Mabileau <paul.mabileau@harfanglab.fr>
Respect the `--ci` flag in more places in bootstrap
### Motivation
Currently, the way of checking CI environment in bootstrap is not unified. Sometimes `--ci` flag is respected, but in other cases only GITHUB_ACTIONS env var is checked.
### Change
This PR modifies the way of treating the flag in bootstrap. If `--ci` (or `--ci=true`) is added, bootstrap's config set ci_env as `CiEnv::GithubActions`. This PR also modifies some lines in bootstrap to respect the config's CiEnv instance.
### Note
I haven't touched anything in tidy, because I want to raise another PR for introducing --ci flag in tidy. In the PR, I will need to change how tidy treats CI information. So currently I leave it as it is.
Support JSON target specs in bootstrap
JSON target specs were destabilized in https://github.com/rust-lang/rust/pull/150151 and https://github.com/rust-lang/rust/pull/151534. However, this broke trying to build rustc itself with a JSON target spec. This is because in a few places bootstrap is manually calling `rustc` without the ability for the user to provide additional flags (primarily, `-Zunstable-options` to enable JSON targets).
There's a few different ways to fix this. One would be to change these calls to `rustc` to include flags provided by the user (such as `RUSTFLAGS_NOT_BOOTSTRAP`). Just to keep things simple, this PR proposes to just unconditionally pass `-Zunstable-options`.
Another consideration here is how maintainable this is. A possible improvement here would be to have a function somewhere (BootstrapCommand, TargetSelection, free function) that would handle appropriately adding the `--target` flag. For example, that's what cargo does in [`CompileKind::add_target_arg`](https://github.com/rust-lang/cargo/blob/592058c7ce08a2ba2628b01e7dc4ce1e72b6bdff/src/cargo/core/compiler/compile_kind.rs#L144-L154).
I have only tested building the compiler and a few tools like rustdoc. I have not tested doing things like building other tools, running tests, etc.
This would be much easier if there was a Docker image for testing the use case of building rustc with a custom target spec (and even better if that ran in CI).
After the next beta branch, using target JSON specs will become more cumbersome because target specs with the `.json` extension will now require passing `-Zjson-target-spec` (from
https://github.com/rust-lang/cargo/pull/16557). This does not affect target specs without the `.json` extension (such as those from RUST_TARGET_PATH). From my testing, it should be sufficient to pass `CARGOFLAGS_NOT_BOOTSTRAP="-Zjson-target-spec"`. I think that should be fine, since this is not a particularly common use case AFAIK. We could extend bootstrap to auto-detect if the target is a file path, and pass `-Zjson-target-spec` appropriately. I tried something similar in https://github.com/ehuss/rust/commit/f0bdd354835a0993febaf1de214cda2abd9a8500, which could be adapted if desired.
It would be nice if all of this is documented somewhere. https://rustc-dev-guide.rust-lang.org/building/new-target.html does not really say how to build the compiler with a custom json target.
Fixes https://github.com/rust-lang/rust/issues/151729
Install LLVM DLL in the right place on Windows
Continuation of https://github.com/rust-lang/rust/pull/151795 towards https://github.com/rust-lang/rust/issues/151774.
Unlike other systems, Windows requires runtime libraries to be present in `PATH` or right next to the binary.
So, we copy the library next to the binary as the easier solution.
Tested building `rust-openssl` in debug and release modes, but the difference is within noise margin.
JSON target specs were destabilized in
https://github.com/rust-lang/rust/pull/150151 and
https://github.com/rust-lang/rust/pull/151534. However, this broke
trying to build rustc itself with a JSON target spec. This is because in
a few places bootstrap is manually calling `rustc` without the ability
for the user to provide additional flags (primarily,
`-Zunstable-options` to enable JSON targets).
There's a few different ways to fix this. One would be to change these
calls to `rustc` to include flags provided by the user (such as
`RUSTFLAGS_NOT_BOOTSTRAP`). Just to keep things simple, this PR proposes
to just unconditionally pass `-Zunstable-options`.
Another consideration here is how maintainable this is. A possible
improvement here would be to have a function somewhere
(BootstrapCommand, TargetSelection, free function) that would handle
appropriately adding the `--target` flag. For example, that's what cargo
does in
[`CompileKind::add_target_arg`](https://github.com/rust-lang/cargo/blob/592058c7ce08a2ba2628b01e7dc4ce1e72b6bdff/src/cargo/core/compiler/compile_kind.rs#L144-L154).
I have only tested building the compiler and a few tools like rustdoc. I
have not tested doing things like building other tools, running tests,
etc.
This would be much easier if there was a Docker image for testing the
use case of building rustc with a custom target spec (and even better if
that ran in CI).
After the next beta branch, using target JSON specs will become more
cumbersome because target specs with the `.json` extension will now
require passing `-Zjson-target-spec` (from
https://github.com/rust-lang/cargo/pull/16557). This does not affect
target specs without the `.json` extension (such as those from
RUST_TARGET_PATH). From my testing, it should be sufficient to pass
`CARGOFLAGS_NOT_BOOTSTRAP="-Zjson-target-spec"`. I think that should be
fine, since this is not a particularly common use case AFAIK. We could
extend bootstrap to auto-detect if the target is a file path, and pass
`-Zjson-target-spec` appropriately. I tried something similar in
https://github.com/ehuss/rust/commit/f0bdd354835a0993febaf1de214cda2abd9a8500,
which could be adapted if desired.
It would be nice if all of this is documented somewhere.
https://rustc-dev-guide.rust-lang.org/building/new-target.html does not
really say how to build the compiler with a custom json target.
Fixes https://github.com/rust-lang/rust/issues/151729
Unlike other systems, Windows requires runtime libraries to be present
in `PATH` or right next to the binary.
So, we copy the library next to the binary as the easier solution.
bootstrap: add explicit UTF-8 encoding to text-mode open() calls
Make text-mode `open()` calls in `bootstrap.py`
explicitly use `encoding="utf-8"`.
This avoids relying on platform-dependent default encodings
when running Python < 3.15.
Binary-mode opens remain unchanged.
Start using pattern types in libcore (NonZero and friends)
part of rust-lang/rust#136006
This PR only changes the internal representation of `NonZero`, `NonMax`, ... and other integral range types in libcore. This subsequently affects other types made up of it, but nothing really changes except that the field of `NonZero` is now accessible safely in contrast to the `rustc_layout_scalar_range_start` attribute, which has all kinds of obscure rules on how to properly access its field.
Add `s390x-unknown-none-softfloat` with `RustcAbi::Softfloat`
followup on rust-lang/rust#150766
add an `s390x-unknown-none-softfloat` target to use for kernel compilation, as the Linux kernel does not wish to pay the overhead of saving float registers by default on kernel switch. this target's `extern "C"` ABI is unspecified, so it is unstable and subject to change between versions, just like the Linux intrakernel ABI and `extern "Rust"` ABIs are unstable.
enforce target feature incompatibility by adding `RustcAbi::Softfloat`. this is itself just a rename of `RustcAbi::X86Softfloat`, accepting both "x86-softfloat" and "softfloat" as valid strings in the target.json format. the target-features of `"soft-float"` and `"vector"` are incompatible for s390x, so issue a compatibility warning if they are combined.
This target is intended to be used for kernel development. Becasue on s390x
float and vector registers overlap we have to disable the vector extension.
The default s390x-unknown-gnu-linux target will not allow use of
softfloat.
Co-authored-by: Jubilee <workingjubilee@gmail.com>
bootstrap: always propagate `CARGO_TARGET_{host}_LINKER`
We were already setting `CARGO_TARGET_{target}_LINKER` when there is a
setting in `bootstrap.toml`, and when the host and target are the same,
this is also used for build scripts and proc-macros.
However, the host value wasn't set when building for any other target,
and Cargo would see that as a fingerprint change for those build
artifacts, rebuilding them.
If we always set the `CARGO_TARGET_{host}_LINKER`, then those build
scripts will keep a consistent Cargo fingerprint, so they'll remain
cached no matter how we're alternating targets.
Convert to inline diagnostics in `rustc_lint`
For https://github.com/rust-lang/rust/issues/151366
r? @jdonszelmann
This was a big one! Second to last one, `rustc_parse` is the last one and even bigger :P
Stuart Cook