Fix MaybeUninit codegen using GVN
This is an alternative to https://github.com/rust-lang/rust/pull/142837, based on https://github.com/rust-lang/rust/pull/146355#discussion_r2421651968.
The general approach I took here is to aggressively propagate anything that is entirely uninitialized. GVN generally takes the approach of only synthesizing small types, but we need to generate large consts to fix the codegen issue.
I also added a special case to MIR dumps for this where now an entirely uninit const is printed as `const <uninit>`, because otherwise we end up with extremely verbose dumps of the new consts.
After GVN though, we still end up with a lot of MIR that looks like this:
```
StorageLive(_1);
_1 = const <uninit>;
_2 = &raw mut _1;
```
Which will break tests/codegen-llvm/maybeuninit-rvo.rs with the naive lowering. I think the ideal fix here is to somehow omit these `_1 = const <uninit>` assignments that come directly after a StorageLive, but I'm not sure how to do that. For now at least, ignoring such assignments (even if they don't come right after a StorageLive) in codegen seems to work.
Note that since GVN is based on synthesizing a `ConstValue` which has a defined layout, this scenario still gets deoptimized by LLVM.
```rust
#![feature(rustc_attrs)]
#![crate_type = "lib"]
use std::mem::MaybeUninit;
#[unsafe(no_mangle)]
pub fn oof() -> [[MaybeUninit<u8>; 8]; 8] {
#[rustc_no_mir_inline]
pub fn inner<T: Copy>() -> [[MaybeUninit<T>; 8]; 8] {
[[MaybeUninit::uninit(); 8]; 8]
}
inner()
}
```
This case can be handled correctly if enough inlining has happened, or it could be handled by post-mono GVN. Synthesizing `UnevaluatedConst` or some other special kind of const seems dubious.
Turn moves into copies after copy propagation
Previously copy propagation presumed that there is further unspecified distinction between move operands and copy operands in assignments and propagated moves from assignments into terminators. This is inconsistent with current operational semantics.
Turn moves into copies after copy propagation to preserve existing behavior.
Fixes https://github.com/rust-lang/rust/issues/137936.
Fixes https://github.com/rust-lang/rust/issues/146423.
r? `@cjgillot`
Previously copy propagation presumed that there is further unspecified
distinction between move operands and copy operands in assignments and
propagated moves from assignments into terminators. This is inconsistent
with current operational semantics.
Turn moves into copies after copy propagation to preserve existing behavior.
Fixes https://github.com/rust-lang/rust/issues/137936.
Fixes https://github.com/rust-lang/rust/issues/146423.
Rust's current mangling scheme depends on compiler internals; loses
information about generic parameters (and other things) which makes for
a worse experience when using external tools that need to interact with
Rust symbol names; is inconsistent; and can contain `.` characters
which aren't universally supported. Therefore, Rust has defined its own
symbol mangling scheme which is defined in terms of the Rust language,
not the compiler implementation; encodes information about generic
parameters in a reversible way; has a consistent definition; and
generates symbols that only use the characters `A-Z`, `a-z`, `0-9`, and
`_`.
Support for the new Rust symbol mangling scheme has been added to
upstream tools that will need to interact with Rust symbols (e.g.
debuggers).
This commit changes the default symbol mangling scheme from the legacy
scheme to the new Rust mangling scheme.
Signed-off-by: David Wood <david.wood@huawei.com>
use funnel shift as fallback impl for rotating shifts
That lets us remove this gnarly implementation from Miri and const-eval.
However, `rotate_left`/`rotate_right` are stable as const fn, so to do this we have to `rustc_allow_const_fn_unstable` a bunch of const trait stuff. Is that a bad idea? Cc `@oli-obk` `@fee1-dead`
The overflow-checks codegen test was failing on riscv64gc target
because the FileCheck pattern did not account for ABI extension
attributes. RISC-V LP64 ABI requires integer types smaller than
XLEN (64-bit) to be zero-extended or sign-extended to register width.
For u8 parameters, RISC-V generates:
i8 noundef zeroext %a, i8 noundef zeroext %b
While x86_64 and aarch64 generate:
i8 noundef %a, i8 noundef %b
The original CHECK pattern only matched the format without the
`zeroext` attribute, causing test failures on RISC-V.
This patch makes the zeroext attribute optional in the FileCheck
pattern using `{{( zeroext)?}}`, allowing the test to pass on
architectures that add ABI extension attributes (e.g., RISC-V).
Test results before fix:
- x86_64-unknown-linux-gnu: 3 passed
- aarch64-unknown-linux-gnu: 3 passed
- riscv64gc-unknown-linux-gnu: 1 passed, 2 failed
Test results after fix:
- x86_64-unknown-linux-gnu: 3 passed
- aarch64-unknown-linux-gnu: 3 passed
- riscv64gc-unknown-linux-gnu: 3 passed
Add `overflow_checks` intrinsic
This adds an intrinsic which allows code in a pre-built library to inherit the overflow checks option from a crate depending on it. This enables code in the standard library to explicitly change behavior based on whether `overflow_checks` are enabled, regardless of the setting used when standard library was compiled.
This is very similar to the `ub_checks` intrinsic, and refactors the two to use a common mechanism.
The primary use case for this is to allow the new `RangeFrom` iterator to yield the maximum element before overflowing, as requested [here](https://github.com/rust-lang/rust/issues/125687#issuecomment-2151118208). This PR includes a working `IterRangeFrom` implementation based on this new intrinsic that exhibits the desired behavior.
[Prior discussion on Zulip](https://rust-lang.zulipchat.com/#narrow/stream/219381-t-libs/topic/Ability.20to.20select.20code.20based.20on.20.60overflow_checks.60.3F)
This implements a new unstable compiler flag `-Zannotate-moves` that makes
move and copy operations visible in profilers by creating synthetic debug
information. This is achieved with zero runtime cost by manipulating debug
info scopes to make moves/copies appear as calls to `compiler_move<T, SIZE>`
and `compiler_copy<T, SIZE>` marker functions in profiling tools.
This allows developers to identify expensive move/copy operations in their
code using standard profiling tools, without requiring specialized tooling
or runtime instrumentation.
The implementation works at codegen time. When processing MIR operands
(`Operand::Move` and `Operand::Copy`), the codegen creates an `OperandRef`
with an optional `move_annotation` field containing an `Instance` of the
appropriate profiling marker function. When storing the operand,
`store_with_annotation()` wraps the store operation in a synthetic debug
scope that makes it appear inlined from the marker.
Two marker functions (`compiler_move` and `compiler_copy`) are defined
in `library/core/src/profiling.rs`. These are never actually called -
they exist solely as debug info anchors.
Operations are only annotated if the type:
- Meets the size threshold (default: 65 bytes, configurable via
`-Zannotate-moves=SIZE`)
- Has a non-scalar backend representation (scalars use registers,
not memcpy)
This has a very small size impact on object file size. With the default
limit it's well under 0.1%, and even with a very small limit of 8 bytes
it's still ~1.5%. This could be enabled by default.
Add alignment parameter to `simd_masked_{load,store}`
This PR adds an alignment parameter in `simd_masked_load` and `simd_masked_store`, in the form of a const-generic enum `core::intrinsics::simd::SimdAlign`. This represents the alignment of the `ptr` argument in these intrinsics as follows
- `SimdAlign::Unaligned` - `ptr` is unaligned/1-byte aligned
- `SimdAlign::Element` - `ptr` is aligned to the element type of the SIMD vector (default behavior in the old signature)
- `SimdAlign::Vector` - `ptr` is aligned to the SIMD vector type
The main motive for this is stdarch - most vector loads are either fully aligned (to the vector size) or unaligned (byte-aligned), so the previous signature doesn't cut it.
Now, stdarch will mostly use `SimdAlign::Unaligned` and `SimdAlign::Vector`, whereas portable-simd will use `SimdAlign::Element`.
- [x] `cg_llvm`
- [x] `cg_clif`
- [x] `miri`/`const_eval`
## Alternatives
Using a const-generic/"const" `u32` parameter as alignment (and we error during codegen if this argument is not a power of two). This, although more flexible than this, has a few drawbacks
- If we use an const-generic argument, then portable-simd somehow needs to pass `align_of::<T>()` as the alignment, which isn't possible without GCE
- "const" function parameters are just an ugly hack, and a pain to deal with in non-LLVM backends
We can remedy the problem with the const-generic `u32` parameter by adding a special rule for the element alignment case (e.g. `0` can mean "use the alignment of the element type), but I feel like this is not as expressive as the enum approach, although I am open to suggestions
cc `@workingjubilee` `@RalfJung` `@BoxyUwU`
Stabilize -Zno-jump-tables into -Cjump-tables=bool
I propose stabilizing the -Zno-jump-tables option into -Cjump-tables=<bool>.
# `-Zno-jump-tables` stabilization report
## What is the RFC for this feature and what changes have occurred to the user-facing design since the RFC was finalized?
No RFC was created for this option. This was a narrowly scoped option introduced in rust-lang/rust#105812 to support code generation requirements of the x86-64 linux kernel, and eventually other targets as Rust For Linux grows.
The tracking is rust-lang/rust#116592.
## What behavior are we committing to that has been controversial? Summarize the major arguments pro/con.
The behavior of this flag is well defined, and mimics the existing `-fno-jump-tables` option currently available with LLVM and GCC with some caveats:
* Unlike clang or gcc, this option may be ignored by the code generation backend. Rust can support multiple code-generation backends. For stabilization, only the LLVM backend honors this option.
* The usage of this option will not guarantee a library or binary is free of jump tables. To ensure a jump-table free binary, all crates in the build graph must be compiled with this option. This includes implicitly linked crates such as std or core.
* This option only enforces the crate being compiled is free of jump tables.
* No verification is done to ensure other crates are compiled with this option. Enforcing code generation options are applied across the crate graph is out of scope for this option.
What should the flag name be?
* As introduced, this option was named `-Zno-jump-tables`. However, other major toolchains allow both positive and negative variants of this option to toggle this feature. Renaming the option to `-Cjump-tables=<bool>` makes this option consistent, and if for some reason, expandable to other arguments in the future. Notably, many LLVM targets have a configurable and different thresholds for when to lower into a jump table.
## Are there extensions to this feature that remain unstable? How do we know that we are not accidentally committing to those.
No. This option is used exclusively to gate a very specific class of optimization.
## Summarize the major parts of the implementation and provide links into the code (or to PRs)
* The original PR rust-lang/rust#105812 by ```@ojeda```
* The stabilized CLI option is parsed as a bool:
https://github.com/pmur/rust/blob/68bfda9025ccb2778e2606e12e8021b9918f40d3/compiler/rustc_session/src/options.rs#L2025-L2026
* This options adds an attribute to each llvm function via:
https://github.com/pmur/rust/blob/68bfda9025ccb2778e2606e12e8021b9918f40d3/compiler/rustc_codegen_llvm/src/attributes.rs#L210-L215
* Finally, the rustc book is updated with the new option:
https://github.com/pmur/rust/blob/68bfda9025ccb2778e2606e12e8021b9918f40d3/src/doc/rustc/src/codegen-options/index.md?plain=1#L212-L223
## Has a call-for-testing period been conducted? If so, what feedback was received?
No. The option has originally created is being used by Rust For Linux to build the x86-64 kernel without issue.
## What outstanding bugs in the issue tracker involve this feature? Are they stabilization-blocking?
There are no outstanding issues.
## Summarize contributors to the feature by name for recognition and assuredness that people involved in the feature agree with stabilization
* ```@ojeda``` implemented this feature in rust-lang/rust#105815 as `-Zno-jump-tables`.
* ```@tgross35``` created and maintained the tracking issue rust-lang/rust#116592, and provided feedback about the naming of the cli option.
## What FIXMEs are still in the code for that feature and why is it ok to leave them there?
There are none.
## What static checks are done that are needed to prevent undefined behavior?
This option cannot cause undefined behavior. It is a boolean option with well defined behavior in both cases.
## In what way does this feature interact with the reference/specification, and are those edits prepared?
This adds a new cli option to `rustc`. The documentation is updated, and the unstable documentation cleaned up in this PR.
## Does this feature introduce new expressions and can they produce temporaries? What are the lifetimes of those temporaries?
No.
## What other unstable features may be exposed by this feature?
None.
## What is tooling support like for this feature, w.r.t rustdoc, clippy, rust-analzyer, rustfmt, etc.?
No support is required from other rust tooling.
## Open Items
- [x] Are there objections renaming `-Zno-jump-tables` to `-Cjump-tables=<bool>`? The consensus is no.
- [x] Is it desirable to keep `-Zno-jump-tables` for a period of time? The consensus is no.
---
Closesrust-lang/rust#116592
Add LLVM range attributes to slice length parameters
It'll take a bunch more work to do this in layout -- because of cycles in `struct Foo<'a>(&'a Foo<'a>);` -- so until we figure out how to do that well, just look for slices specifically and add the proper range for the length.
rustc assumes that regular `extern "Rust"` functions unwind only if the
`unwind` panic runtime is linked. `throw` was annotated as such, but
unwound unconditionally. This could cause UB when a crate built with `-C
panic=abort` called `throw` from `core` built with `-C panic=unwind`,
since no terminator was added to handle the panic arising from calling an
allegedly non-unwinding `extern "Rust"` function.
rustc was taught to recognize this condition since
https://github.com/rust-lang/rust/pull/144225 and prevented such
linkage, but this caused regressions in
https://github.com/rust-lang/rust/issues/148246, since this meant that
Emscripten projects could not be built with `-C panic=abort` without
recompiling std.
The most straightforward solution would be to move `throw` into the
`panic_unwind` crate, so that it's only compiled if the panic runtime is
guaranteed to be `unwind`, but this is messy due to our architecture.
Instead, move it into `unwind::wasm`, which is only compiled for
bare-metal targets that default to `panic = "abort"`, rendering the
issue moot.
When a MIR argument is spread at ABI level, deduced attributes are
potentially misapplied, since a spread argument can correspond to zero
or more arguments at ABI level.
Disable deduction for MIR using spread argument for the time being.
Deduce captures(none) for a return place and parameters
Extend attribute deduction to determine whether parameters using
indirect pass mode might have their address captured. Similarly to
the deduction of `readonly` attribute this information facilitates
memcpy optimizations.
Extend attribute deduction to determine whether parameters using
indirect pass mode might have their address captured. Similarly to
the deduction of `readonly` attribute this information facilitates
memcpy optimizations.
Stop passing resolver disambiguator state to AST lowering.
AST->HIR lowering can use a disjoint set of `DefPathData` as the resolver, so we don't need to pass the disambiguator state.
r? `@oli-obk`
This was a bit more invasive than I had kind of hoped. An alternate
approach would be to add an extra call_intrinsic_with_attrs() that would
have the new-in-this-change signature for call_intrinsic, but this felt
about equivalent and made it a little easier to audit the relevant
callsites of call_intrinsic().
Add a test for the cold attribute
This adds a test for the cold attribute to verify that it actually does something, and that it applies correctly in all the positions it is expected to work.
Offload host2
r? `@oli-obk`
A follow-up to my previous gpu host PR. With this, I can (in theory) run a sufficiently simple Rust function on GPUs. I tested it on AMD, where the amdgcn tartget of rustc causes issues due to Addressspace castings, which might not be valid. If I (manually) fix them, I can run the generated IR on an AMD GPU. This should conceptually also work on NVIDIA or Intel. I updated the dev-guide acordingly: https://rustc-dev-guide.rust-lang.org/offload/usage.html
I am unhappy with the amount of standalone functions in my offload code, so in my second commit I bundled some of the code around two structs which are Rust versions of the LLVM/Offload structs which they represent. The structs themselves only have doc comments. Since I directly lower everything to llvm-ir I didn't saw a big value in modelling the struct member variables.
Add vsx register support for ppc inline asm, and implement preserves_flag option
This should address the last(?) missing pieces of inline asm for ppc:
* Explicit VSX register support. ISA 2.06 (POWER7) added a 64x128b register overlay extending the fpr's to 128b, and unifies them with the vmx (altivec) registers. Implementations details within gcc/llvm percolate up, and require using the `x` template modifier. I have updated the inline asm to implicitly include this for vsx arguments which do not specify it. ~~Support for the gcc codegen backend is still a todo.~~
* Implement the `preserves_flags` option. All ABI's, and all ISAs store their flags in `cr`, and the carry bit lives inside `xer`. The other status registers hold sticky bits or control bits which do not affect branch instructions.
There is some interest in the e500 (powerpcspe) port. Architecturally, it has a very different FP ISA, and includes a simd extension called SPR (which is not IBM's cell SPE). Notably, it does not have altivec/fpr/vsx registers. It also has an SPE accumulator register which its ABI marks as volatile, but I am not sure if the compiler uses it.
Implemented preserves_flags on powerpc by making it do
nothing. This prevents having two different ways to mark
`cr0` as clobbered. clang and gcc alias `cr0` to `cc`.
The gcc inline documentation does not state what this does
on powerpc* targets, but inspection of the source shows
it is equivalent to condition register field `cr0`, so it
should not be added.
Where supported, VSX is a 64x128b register set which encompasses
both the floating point and vector registers.
In the type tests, xvsqrtdp is used as it is the only two-argument
vsx opcode supported by all targets on llvm. If you need to copy
a vsx register, the preferred way is "xxlor xt, xa, xa".
bors