Load only the crate header for `locator::crate_matches`
Previously, we used the following info to determine whether to load the crate:
1. The METADATA_HEADER, which includes a METADATA_VERSION constant
2. The embedded rustc version
3. Various metadata in the `CrateRoot`, including the SVH
This worked ok most of the time. Unfortunately, when building locally the rustc version is always
the same because `omit-git-hash` is on by default. That meant that we depended only on 1 and 3, and
we are not very good about bumping METADATA_VERSION (it's currently at 7) so in practice we were
only depending on 3. `CrateRoot` is a very large struct and changes somewhat regularly, so this led
to a steady stream of crashes from trying to load it.
Change the logic to add an intermediate step between 2 and 3: introduce a new `CrateHeader` struct
that contains only the minimum info needed to decide whether the crate should be loaded or not. That
avoids having to load all of `CrateRoot`, which in practice means we should crash much less often.
Note that this works because the SVH should be different between any two dependencies, even if the
compiler has changed, because we use `-Zbinary-dep-depinfo` in bootstrap. See
https://github.com/rust-lang/rust/pull/111329#issuecomment-1538303474 for more details about how the
original crash happened.
Previously, we used the following info to determine whether to load the crate:
1. The METADATA_HEADER, which includes a METADATA_VERSION constant
2. The embedded rustc version
3. Various metadata in the `CrateRoot`, including the SVH
This worked ok most of the time. Unfortunately, when building locally the rustc version is always
the same because `omit-git-hash` is on by default. That meant that we depended only on 1 and 3, and
we are not very good about bumping METADATA_VERSION (it's currently at 7) so in practice we were
only depending on 3. `CrateRoot` is a very large struct and changes somewhat regularly, so this led
to a steady stream of crashes from trying to load it.
Change the logic to add an intermediate step between 2 and 3: introduce a new `CrateHeader` struct
that contains only the minimum info needed to decide whether the crate should be loaded or not. That
avoids having to load all of `CrateRoot`, which in practice means we should crash much less often.
Note that this works because the SVH should be different between any two dependencies, even if the
compiler has changed, because we use `-Zbinary-dep-depinfo` in bootstrap. See
https://github.com/rust-lang/rust/pull/111329#issuecomment-1538303474 for more details about how the
original crash happened.
Support #[global_allocator] without the allocator shim
This makes it possible to use liballoc/libstd in combination with `--emit obj` if you use `#[global_allocator]`. This is what rust-for-linux uses right now and systemd may use in the future. Currently they have to depend on the exact implementation of the allocator shim to create one themself as `--emit obj` doesn't create an allocator shim.
Note that currently the allocator shim also defines the oom error handler, which is normally required too. Once `#![feature(default_alloc_error_handler)]` becomes the only option, this can be avoided. In addition when using only fallible allocator methods and either `--cfg no_global_oom_handling` for liballoc (like rust-for-linux) or `--gc-sections` no references to the oom error handler will exist.
To avoid this feature being insta-stable, you will have to define `__rust_no_alloc_shim_is_unstable` to avoid linker errors.
(Labeling this with both T-compiler and T-lang as it originally involved both an implementation detail and had an insta-stable user facing change. As noted above, the `__rust_no_alloc_shim_is_unstable` symbol requirement should prevent unintended dependence on this unstable feature.)
Fix dependency tracking for debugger visualizers
This PR fixes dependency tracking for debugger visualizer files by changing the `debugger_visualizers` query to an `eval_always` query that scans the AST while it is still available. This way the set of visualizer files is already available when dep-info is emitted. Since the query is turned into an `eval_always` query, dependency tracking will now reliably detect changes to the visualizer script files themselves.
TODO:
- [x] perf.rlo
- [x] Needs a bit more documentation in some places
- [x] Needs regression test for the incr. comp. case
Fixes https://github.com/rust-lang/rust/issues/111226
Fixes https://github.com/rust-lang/rust/issues/111227
Fixes https://github.com/rust-lang/rust/issues/111295
r? `@wesleywiser`
cc `@gibbyfree`
Only depend on CFG_VERSION in rustc_interface
This avoids having to rebuild the whole compiler on each commit when `omit-git-hash = false`.
cc https://github.com/rust-lang/rust/issues/76720 - this won't fix it, and I'm not suggesting we turn this on by default, but it will make it less painful for people who do have `omit-git-hash` on as a workaround.
Merge query property modules into one
This merges all the query modules that defines types into a single module per query with a normal naming convention for type aliases.
r? ``@cjgillot``
Handle error body in generator layout
Fixes#111468
I feel like making this query return `Option<GeneratorLayout>` might be better but had some issues with that approach
Error message all end up passing into a function as an `impl
Into<{D,Subd}iagnosticMessage>`. If an error message is creatd as
`&format("...")` that means we allocate a string (in the `format!`
call), then take a reference, and then clone (allocating again) the
reference to produce the `{D,Subd}iagnosticMessage`, which is silly.
This commit removes the leading `&` from a lot of these cases. This
means the original `String` is moved into the
`{D,Subd}iagnosticMessage`, avoiding the double allocations. This
requires changing some function argument types from `&str` to `String`
(when all arguments are `String`) or `impl
Into<{D,Subd}iagnosticMessage>` (when some arguments are `String` and
some are `&str`).
Introduce `DynSend` and `DynSync` auto trait for parallel compiler
part of parallel-rustc #101566
This PR introduces `DynSend / DynSync` trait and `FromDyn / IntoDyn` structure in rustc_data_structure::marker. `FromDyn` can dynamically check data structures for thread safety when switching to parallel environments (such as calling `par_for_each_in`). This happens only when `-Z threads > 1` so it doesn't affect single-threaded mode's compile efficiency.
r? `@cjgillot`
Apply simulate-remapped-rust-src-base even if remap-debuginfo is set in config.toml
This is really a mess. Here is the situation before this change:
- UI tests depend on not having `rust-src` available. In particular, <https://github.com/rust-lang/rust/blob/3f374128ee3924514aacadf96479e17fee8f9903/tests/ui/tuple/wrong_argument_ice.stderr#L7-L8> is depending on the `note` being a single line and not showing the source code.
- When `download-rustc` is disabled, we pass `-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX` `-Ztranslate-remapped-path-to-local-path=no`, which changes the diagnostic to something like ` --> /rustc/FAKE_PREFIX/library/alloc/src/collections/vec_deque/mod.rs:1657:12`
- When `download-rustc` is enabled, we still pass those flags, but they no longer have an effect. Instead rustc emits diagnostic paths like this: ` --> /rustc/39c6804b92aa202369e402525cee329556bc1db0/library/alloc/src/collections/vec_deque/mod.rs:1657:12`. Notice how there's a real commit and not `FAKE_PREFIX`. This happens because we set `CFG_VIRTUAL_RUST_SOURCE_BASE_DIR` during bootstrapping for CI artifacts, and rustc previously didn't allow for `simulate-remapped` to affect paths that had already been remapped.
- Pietro noticed this and decided the right thing was to normalize `/rustc/<commit>` to `$SRC_DIR` in compiletest: https://github.com/rust-lang/rust/commit/470423c3d2cde3a62d5d4ac23840d734e8145366
- After my change to `x test core`, which rebuilds stage 2 std from source so `build/stage2-std` and `build/stage2` use the same `.rlib` metadata, the compiler suddenly notices it has sources for `std` available and prints those in the diagnostic, causing the test to fail.
This changes `simulate-remapped-rust-src-base` to support remapping paths that have already been remapped, unblocking download-rustc.
Unfortunately, although this fixes the specific problem for
download-rustc, it doesn't seem to affect all the compiler's
diagnostics. In particular, various `mir-opt` tests are failing to
respect `simulate-remapped-path-prefix` (I looked into fixing this but
it seems non-trivial). As a result, we can't remove the normalization in
compiletest that maps `/rustc/<commit>` to `$SRC_DIR`, so this change is
currently untested anywhere except locally.
You can test this locally yourself by setting `rust.remap-debuginfo = true`, running any UI test with `ERROR` annotations, then rerunning the test manually with a dev toolchain to verify it prints `/rustc/FAKE_PREFIX`, not `/rustc/1.71.0`.
Helps with https://github.com/rust-lang/rust/issues/110352.
Encode `VariantIdx` so we can decode ADT variants in the right order
As far as I can tell, we don't guarantee anything about the ordering of `DefId`s and module children...
The code that motivated this PR (#111483) looks something like:
```rust
#[derive(Protocol)]
pub enum Data {
#[protocol(discriminator(0x00))]
Disconnect(Disconnect),
EncryptionRequest,
/* more variants... */
}
```
The specific macro ([`protocol`](https://github.com/dylanmckay/protocol)) doesn't really matter, but as far as I can tell (from calls to `build_reduced_graph`), the presence of that `#[protocol(..)]` helper attribute causes the def-id of the `Disconnect` enum variant to be collected *after* its siblings, and it shows up after the other variants in `module_children`.
When we decode the variants for `Data` in a child crate (an example test, in this case), this means that the `Disconnect` variant is moved to the end of the variants list, and all of the other variants now have incorrect relative discriminant data, causing the ICE.
This PR fixes this by sorting manually by variant index after they are decoded. I guess there are alternative ways of fixing this, such as not reusing `module_children_non_reexports` to encode the order-sensitive ADT variants, or to do some sorting in `rustc_resolve`... but none of those seemed particularly satisfying either.
~I really struggled to create a reproduction here -- it required at least 3 crates, one of which is a proc macro, and then some code to actually compute discriminants in the child crate... Needless to say, I failed to repro this in a test, but I can confirm that it fixes the regression in #111483.~ Test exists now.
r? `@petrochenkov` but feel free to reassign. ~Again, sorry for no test, but I hope the explanation at least suggests why a fix like this is likely necessary.~ Feedback is welcome.
Switch to `EarlyBinder` for `thir_abstract_const` query
Part of the work to finish https://github.com/rust-lang/rust/issues/105779.
This PR adds `EarlyBinder` to the return type of the `thir_abstract_const` query and removes `bound_abstract_const`.
r? `@compiler-errors`
Support linking to rust dylib with --crate-type staticlib
This allows for example dynamically linking libstd, while statically linking the user crate into an executable or C dynamic library. For this two unstable flags (`-Z staticlib-allow-rdylib-deps` and `-Z staticlib-prefer-dynamic`) are introduced. Without the former you get an error. The latter is the equivalent to `-C prefer-dynamic` for the staticlib crate type to indicate that dynamically linking is preferred when both options are available, like for libstd. Care must be taken to ensure that no crate ends up being merged into two distinct staticlibs that are linked together. Doing so will cause a linker error at best and undefined behavior at worst. In addition two distinct staticlibs compiled by different rustc may not be combined under any circumstances due to some rustc private symbols not being mangled.
To successfully link a staticlib, `--print native-static-libs` can be used while compiling to ask rustc for the linker flags necessary when linking the staticlib. This is an existing flag which previously only listed native libraries. It has been extended to list rust dylibs too. Trying to locate libstd yourself to link against it is not supported and may break if for example the libstd of multiple rustc versions are put in the same directory.
For an example on how to use this see the `src/test/run-make-fulldeps/staticlib-dylib-linkage/` test.
Add `force` option for `--extern` flag
When `--extern force:foo=libfoo.so` is passed to `rustc` and `foo` is not actually used in the crate, ~inject an `extern crate foo;` statement into the AST~ force it to be resolved anyway in `CrateLoader::postprocess()`. This allows you to, for instance, inject a `#[panic_handler]` implementation into a `#![no_std]` crate without modifying its source so that it can be built as a `dylib`. It may also be useful for `#![panic_runtime]` or `#[global_allocator]`/`#![default_lib_allocator]` implementations.
My work previously involved integrating Rust into an existing C/C++ codebase which was built with Buck and shipped on, among other platforms, Android. When targeting Android, Buck builds all "native" code with shared linkage* so it can be loaded from Java/Kotlin. My project was not itself `#![no_std]`, but many of our dependencies were, and they would fail to build with shared linkage due to a lack of a panic handler. With this change, that project can add the new `force` option to the `std` dependency it already explicitly provides to every crate to solve this problem.
*This is an oversimplification - Buck has a couple features for aggregating dependencies into larger shared libraries, but none that I think sustainably solve this problem.
~The AST injection happens after macro expansion around where we similarly inject a test harness and proc-macro harness. The resolver's list of actually-used extern flags is populated during macro expansion, and if any of our `--extern` arguments have the `force` option and weren't already used, we inject an `extern crate` statement for them. The injection logic was added in `rustc_builtin_macros` as that's where similar injections for tests, proc-macros, and std/core already live.~
(New contributor - grateful for feedback and guidance!)
Stabilize raw-dylib, link_ordinal, import_name_type and -Cdlltool
This stabilizes the `raw-dylib` feature (#58713) for all architectures (i.e., `x86` as it is already stable for all other architectures).
Changes:
* Permit the use of the `raw-dylib` link kind for x86, the `link_ordinal` attribute and the `import_name_type` key for the `link` attribute.
* Mark the `raw_dylib` feature as stable.
* Stabilized the `-Zdlltool` argument as `-Cdlltool`.
* Note the path to `dlltool` if invoking it failed (we don't need to do this if `dlltool` returns an error since it prints its path in the error message).
* Adds tests for `-Cdlltool`.
* Adds tests for being unable to find the dlltool executable, and dlltool failing.
* Fixes a bug where we were checking the exit code of dlltool to see if it failed, but dlltool always returns 0 (indicating success), so instead we need to check if anything was written to `stderr`.
NOTE: As previously noted (https://github.com/rust-lang/rust/pull/104218#issuecomment-1315895618) using dlltool within rustc is temporary, but this is not the first time that Rust has added a temporary tool use and argument: https://github.com/rust-lang/rust/pull/104218#issuecomment-1318720482
Big thanks to ``````@tbu-`````` for the first version of this PR (#104218)
lcnr