tests: codegen-llvm: iter-repeat-n-trivial-drop: Allow non-zero lower bound to __rust_alloc size
LLVM emits a lower bound of 8 for the size parameter to `__rust_alloc` when targeting `x86_64-unknown-hermit`. Since that is also completely valid, relax the lower bound check.
I'm not really sure why LLVM is able to infer this - with the same setup targeting `x86_64-unknown-linux-gnu` I also see the lower bound of 0. Not that it's wrong, but I'd be curious to know which codegen options play into this.
num: Separate public API from internal implementations
Currently we have a single `core::num` module that contains both thin wrapper API and higher-complexity numeric routines. Restructure this by moving implementation details to a new `imp` module.
This results in a more clean separation of what is actually user-facing compared to items that have a stability attribute because they are public for testing.
The first commit does the actual change then the second moves a portion back.
Instead of defaulting to `None` it now defaults to `Align::ONE` i.e.
no alignment restriction. Codegen test changes are due to us now skipping
`align 1` annotations (they are useless; not skipping them makes all the
raw pointers gain an `align 1` annotation which doesn't seem any good)
Properly pass offload sizes to kernel args
This PRs prevents offload from creating an unnecessary alloca when all the arg sizes are static.
I'll implement the first dynamic-size data type in a follow up PR (slice support).
r? @ZuseZ4
fix autodiff parsing for non-trait impl
fixes: https://github.com/rust-lang/rust/issues/153322
@Sa4dUs Looks like we missed a case.
But also, going through the code, line 455 seems suspicious to me:
`Annotatable::AssocItem(d_fn, Impl { of_trait: false })`
Are we sure that this should always be an Impl, and never an impl of a trait?
r? @oli-obk
Update call-llvm-intrinsics test for Rust 1.94.0 IR
Rust 1.94 now passes constants directly to llvm.sqrt.f32 instead of
storing/loading via the stack.
- Updated the FileCheck pattern to match the new IR:
// CHECK: call float @llvm.sqrt.f32(float 4.000000e+00)
The test intent is unchanged: it still ensures the intrinsic is
emitted as a 'call' (not 'invoke').
- Removed unnecessary local variables and Drop usage to work in
`#![no_core]` mode with minicore.
- Added required crate attributes:
#![feature(no_core, lang_items)]
#![no_std]
#![no_core]
- Replaced `//@ only-riscv64` (host-based execution) with explicit
revisions for:
riscv32gc-unknown-linux-gnu
riscv64gc-unknown-linux-gnu
This ensures deterministic multi-target coverage in CI without
relying on the host architecture.
- Added `//@ needs-llvm-components: riscv` and
`//@ min-llvm-version: 21` for CI compatibility.
Fixesrust-lang/rust#153271
Rust 1.94 now passes constants directly to llvm.sqrt.f32 instead of
storing/loading via the stack.
- Updated the FileCheck pattern to match the new IR:
// CHECK: call float @llvm.sqrt.f32(float 4.000000e+00)
The test intent is unchanged: it still ensures the intrinsic is
emitted as a 'call' (not 'invoke').
- Removed unnecessary local variables and Drop usage to work in
`#![no_core]` mode with minicore.
- Added required crate attributes:
#![feature(no_core, lang_items)]
#![no_std]
#![no_core]
- Replaced `//@ only-riscv64` (host-based execution) with explicit
revisions for:
riscv32gc-unknown-linux-gnu
riscv64gc-unknown-linux-gnu
This ensures deterministic multi-target coverage in CI without
relying on the host architecture.
- Added `//@ needs-llvm-components: riscv` and
`//@ min-llvm-version: 21` for CI compatibility.
Signed-off-by: Deepesh Varatharajan <Deepesh.Varatharajan@windriver.com>
tests: codegen-llvm: vec-calloc: do not require the uwtable attribute
The `uwtable` attribute does not get emitted on targets that don't have unwinding tables, such as `x86_64-unknown-hermit`.
Updated slice tests to pass for big endian hosts for `multiple-option-or-permutations.rs`
It was discovered that the FileCheck tests when performing an `Option::or` operation on a slice was failing when tested on a big endian host.
The compiler explorer link is here outlining the codegen output differences - https://rust.godbolt.org/z/qdE7d3G4f
This MR relaxes the constraints for the `*slice_u8` variants of the test (by changing `CHECK-NEXT` to `CHECK-DAG`), whilst still maintaining the check for the necessary `or` logic.
Huge thanks to @Gelbpunkt for identifying this issue! It has been confirmed that this fix passes on a big endian target now as well.
Closesrust-lang/rust#151718
LLVM emits a lower bound of 8 for the size parameter to __rust_alloc
when targeting x86_64-unknown-hermit. Since that is also completely
valid, relax the lower bound check.
perf(codegen): Restore `noundef` On `PassMode::Cast` Args In Rust ABI
### Summary:
#### Problem:
Small aggregate arguments passed via `PassMode::Cast` in the Rust ABI (e.g. `[u32; 2]` cast to `i64`) are missing `noundef` in the emitted LLVM IR, even when the type contains no uninit bytes:
```rust
#[no_mangle]
pub fn f(v: [u32; 2]) -> u32 { v[0] }
```
```llvm
; expected: define i32 @f(i64 noundef %0)
; actual: define i32 @f(i64 %0) ← noundef missing
```
This blocks LLVM from applying optimizations that require value-defined semantics on function arguments.
#### Root Cause:
`adjust_for_rust_abi` calls `arg.cast_to(Reg::Integer)`, which internally creates a `CastTarget` with `ArgAttributes::new()` — always empty. Any validity attribute that was present before the cast is silently dropped.
This affects all `PassMode::Cast` arguments and return values in the Rust ABI: plain arrays, newtype wrappers, and any `BackendRepr::Memory` type small enough to fit in a register.
A prior attempt (rust-lang/rust#127210) used `Ty`/`repr` attributes to detect padding.
#### Solution:
After `adjust_for_rust_abi`, iterate all `PassMode::Cast` args and the return value. For each, call `layout_is_noundef` on the original layout; if it returns `true`, set `NoUndef` on the `CastTarget`'s `attrs`.
`layout_is_noundef` uses only the computed layout — `BackendRepr`, `FieldsShape`, `Variants`, `Scalar::is_uninit_valid()` — and never touches `Ty` or repr attributes. **Anything it cannot prove returns `false`.**
Covered cases:
- `Scalar` / `ScalarPair` (both halves initialized, fields contiguous)
- `FieldsShape::Array` (element type recursively uninit-free)
- `FieldsShape::Arbitrary` with `Variants::Single` (fields cover `0..size` with no gaps, each recursively uninit-free) — handles newtype wrappers, multi-field structs, single-variant enums, `repr(transparent)`, `repr(C)` wrappers
Conservatively excluded with FIXMEs:
- Multi-variant enums (per-variant padding analysis needed)
- Foreign-ABI casts (cast target may exceed layout size, needs a size guard)
### Changes:
- `compiler/rustc_ty_utils/src/abi.rs`: add restoration loop after `adjust_for_rust_abi`; add `layout_is_noundef` and `fields_cover_layout`.
- `tests/codegen-llvm/abi-noundef-cast.rs`: new FileCheck test covering arrays, newtype wrappers (`repr(Rust)`, `repr(transparent)`, `repr(C)`), multi-field structs, single-variant enums, return values, and negative cases (`MaybeUninit`, struct with trailing padding).
- `tests/codegen-llvm/debuginfo-dse.rs`: update one CHECK pattern — `Aggregate_4xi8` (`struct { i8, i8, i8, i8 }`) now correctly gets `noundef`.
Fixesrust-lang/rust#123183.
r? @RalfJung
`adjust_for_rust_abi` was casting small aggregates to an integer register
without propagating `noundef`, causing a performance regression (#123183)
— LLVM could no longer assume the bits were fully defined.
Add `layout_is_noundef` (conservative) + `fields_are_noundef` helper,
then use `cast_to_with_attrs` to forward `NoUndef` when proven:
Scalar → `!is_uninit_valid()`
ScalarPair → both scalars valid + `s1.size + s2.size == layout.size`
(size equality rejects layouts with inter-scalar padding)
Array → recurse into element; empty arrays unconditionally noundef
Arbitrary → `Variants::Single` required; walk fields in offset order —
any gap, non-noundef field, or trailing pad returns false
Union / Primitive / Simd → false (conservative)
Bless `pass-indirectly-attr.stderr` and `debuginfo-dse.rs` for the new
attribute on Cast args. Add `tests/codegen-llvm/abi-noundef-cast.rs`
covering positive Cast cases (arrays, plain structs, single-variant enum)
and negative Cast cases (MaybeUninit, multi-variant enum, field/pair gap,
trailing padding).
Fixes#123183.
Co-authored-by: Ralf Jung <post@ralfj.de>
Fix: On wasm targets, call `panic_in_cleanup` if panic occurs in cleanup
Previously this was not correctly implemented. Each funclet may need its own terminate block, so this changes the `terminate_block` into a `terminate_blocks` `IndexVec` which can have a terminate_block for each funclet. We key on the first basic block of the funclet -- in particular, this is the start block for the old case of the top level terminate function.
I also fixed the `terminate` handler to not be invoked when a foreign exception is raised, mimicking the behavior from msvc. On wasm, in order to avoid generating a `catch_all` we need to call `llvm.wasm.get.exception` and `llvm.wasm.get.ehselector`.
Previously this was not correctly implemented. Each funclet may need its own terminate
block, so this changes the `terminate_block` into a `terminate_blocks` `IndexVec` which
can have a terminate_block for each funclet. We key on the first basic block of the
funclet -- in particular, this is the start block for the old case of the top level
terminate function.
Rather than using a catchswitch/catchpad pair, I used a cleanuppad. The reason for the
pair is to avoid catching foreign exceptions on MSVC. On wasm, it seems that the
catchswitch/catchpad pair is optimized back into a single cleanuppad and a catch_all
instruction is emitted which will catch foreign exceptions. Because the new logic is
only used on wasm, it seemed better to take the simpler approach seeing as they do the
same thing.
extend unpin noalias tests to cover mutable references
https://github.com/rust-lang/rust/pull/152946 made a change to the logic for this attribute that the test should have flagged as problematic -- but the test only checked `Box`, not `&mut`, and those have independent code paths. So extend the test to also cover `&mut`.
@b-naber would be nice if you could confirm that the added tests do fail with your PR.
Simplify internals of `{Rc,Arc}::default`
This commit simplifies the internal implementation of `Default` for these two pointer types to have the same performance characteristics as before (a side effect of changes in rust-lang/rust#131460) while avoid use of internal private APIs of Rc/Arc. To preserve the same codegen as before some non-generic functions needed to be tagged as `#[inline]` as well, but otherwise the same IR is produced before/after this change.
The motivation of this commit is I was studying up on the state of initialization of `Arc` and `Rc` and figured it'd be nicer to reduce the use of internal APIs and instead use public stable APIs where possible, even in the implementation itself.
Simplify the canonical enum clone branches to a copy statement
I have overhauled MatchBranchSimplification in this PR. This pass tries to unify statements one by one, which is more readable and extensible.
This PR also unifies the following pattern that is mostly generated by GVN into one basic block that contains the copy statement:
```rust
match a {
Foo::A(_) => *a,
Foo::B => Foo::B
}
```
Fixes https://github.com/rust-lang/rust/issues/128081.
Pass alignments through the shim as `Alignment` (not `usize`)
We're using `Layout` on both sides, so might as well skip the transmutes back and forth to `usize`.
The mir-opt test shows that doing so allows simplifying the boxed-slice drop slightly, for example.
tests: adapt align-offset.rs for InstCombine improvements in LLVM 23
Upstream [has improved InstCombine](https://github.com/llvm/llvm-project/commit/8d2078332c23b10dcf3571adc1a186e5c65f82df) so that it can shrink added constants using known zeroes, which caused a little bit of change in this test. As far as I can tell either output is fine, so we just accept both.
@rustbot label: +llvm-main
UnsafePinned: implement opsem effects of UnsafeUnpin
This implements the next step for https://github.com/rust-lang/rust/issues/125735: actually making `UnsafePinned` have special opsem effects by suppressing the `noalias` *even if* the type is wrapped in an `Unpin` wrapper.
For backwards compatibility we also still keep the `Unpin` hack, i.e. a type must be both `Unpin` and `UnsafeUnpin` to get `noalias`.
Optimize indexing slices and strs with inclusive ranges
Instead of separately checking for `end == usize::MAX` and `end + 1 > slice.len()`, we can check for `end >= slice.len()`. Also consolidate all the str indexing related panic functions into a single function which reports the correct error depending on the arguments, as the slice indexing code already does.
The downside of all this is that the panic message is slightly less specific when trying to index with `[..=usize::MAX]`: instead of saying "attempted to index str up to maximum usize" it just says "end byte index {end} out of bounds". But this is a rare enough case that I think it is acceptable
This commit simplifies the internal implementation of `Default` for
these two pointer types to have the same performance characteristics as
before (a side effect of changes in 131460) while avoid use of internal
private APIs of Rc/Arc. To preserve the same codegen as before some
non-generic functions needed to be tagged as `#[inline]` as well, but
otherwise the same IR is produced before/after this change.
The motivation of this commit is I was studying up on the state of
initialization of `Arc` and `Rc` and figured it'd be nicer to reduce the
use of internal APIs and instead use public stable APIs where possible,
even in the implementation itself.
Make operational semantics of pattern matching independent of crate and module
The question of "when does matching an enum against a pattern of one of its variants read its discriminant" is currently an underspecified part of the language, causing weird behavior around borrowck, drop order, and UB.
Of course, in the common cases, the discriminant must be read to distinguish the variant of the enum, but currently the following exceptions are implemented:
1. If the enum has only one variant, we currently skip the discriminant read.
- This has the advantage that single-variant enums behave the same way as structs in this regard.
- However, it means that if the discriminant exists in the layout, we can't say that this discriminant being invalid is UB. This makes me particularly uneasy in its interactions with niches – consider the following example ([playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=5904a6155cbdd39af4a2e7b1d32a9b1a)), where miri currently doesn't detect any UB (because the semantics don't specify any):
<details><summary>Example 1</summary>
```rust
#![allow(dead_code)]
use core::mem::{size_of, transmute};
#[repr(u8)]
enum Inner {
X(u8),
}
enum Outer {
A(Inner),
B(u8),
}
fn f(x: &Inner) {
match x {
Inner::X(v) => {
println!("{v}");
}
}
}
fn main() {
assert_eq!(size_of::<Inner>(), 2);
assert_eq!(size_of::<Outer>(), 2);
let x = Outer::B(42);
let y = &x;
f(unsafe { transmute(y) });
}
```
</details>
2. For the purpose of the above, enums with marked with `#[non_exhaustive]` are always considered to have multiple variants when observed from foreign crates, but the actual number of variants is considered in the current crate.
- This means that whether code has UB can depend on which crate it is in: https://github.com/rust-lang/rust/issues/147722
- In another case of `#[non_exhaustive]` affecting the runtime semantics, its presence or absence can change what gets captured by a closure, and by extension, the drop order: https://github.com/rust-lang/rust/issues/147722#issuecomment-3674554872
- Also at the above link, there is an example where removing `#[non_exhaustive]` can cause borrowck to suddenly start failing in another crate.
3. Moreover, we currently make a more specific check: we only read the discriminant if there is more than one *inhabited* variant in the enum.
- This means that the semantics can differ between `foo<!>`, and a copy of `foo` where `T` was manually replaced with `!`: rust-lang/rust#146803
- Moreover, due to the privacy rules for inhabitedness, it means that the semantics of code can depend on the *module* in which it is located.
- Additionally, this inhabitedness rule is even uglier due to the fact that closure capture analysis needs to happen before we can determine whether types are uninhabited, which means that whether the discriminant read happens has a different answer specifically for capture analysis.
- For the two above points, see the following example ([playground](https://play.rust-lang.org/?version=nightly&mode=debug&edition=2024&gist=a07d8a3ec0b31953942e96e2130476d9)):
<details><summary>Example 2</summary>
```rust
#![allow(unused)]
mod foo {
enum Never {}
struct PrivatelyUninhabited(Never);
pub enum A {
V(String, String),
Y(PrivatelyUninhabited),
}
fn works(mut x: A) {
let a = match x {
A::V(ref mut a, _) => a,
_ => unreachable!(),
};
let b = match x {
A::V(_, ref mut b) => b,
_ => unreachable!(),
};
a.len(); b.len();
}
fn fails(mut x: A) {
let mut f = || match x {
A::V(ref mut a, _) => (),
_ => unreachable!(),
};
let mut g = || match x {
A::V(_, ref mut b) => (),
_ => unreachable!(),
};
f(); g();
}
}
use foo::A;
fn fails(mut x: A) {
let a = match x {
A::V(ref mut a, _) => a,
_ => unreachable!(),
};
let b = match x {
A::V(_, ref mut b) => b,
_ => unreachable!(),
};
a.len(); b.len();
}
fn fails2(mut x: A) {
let mut f = || match x {
A::V(ref mut a, _) => (),
_ => unreachable!(),
};
let mut g = || match x {
A::V(_, ref mut b) => (),
_ => unreachable!(),
};
f(); g();
}
```
</details>
In light of the above, and following the discussion at rust-lang/rust#138961 and rust-lang/rust#147722, this PR ~~makes it so that, operationally, matching on an enum *always* reads its discriminant.~~ introduces the following changes to this behavior:
- matching on a `#[non_exhaustive]` enum will always introduce a discriminant read, regardless of whether the enum is from an external crate
- uninhabited variants now count just like normal ones, and don't get skipped in the checks
As per the discussion below, the resolution for point (1) above is that it should land as part of a separate PR, so that the subtler decision can be more carefully considered.
Note that this is a breaking change, due to the aforementioned changes in borrow checking behavior, new UB (or at least UB newly detected by miri), as well as drop order around closure captures. However, it seems to me that the combination of this PR with rust-lang/rust#138961 should have smaller real-world impact than rust-lang/rust#138961 by itself.
Fixesrust-lang/rust#142394Fixesrust-lang/rust#146590Fixesrust-lang/rust#146803 (though already marked as duplicate)
Fixes parts of rust-lang/rust#147722Fixesrust-lang/miri#4778
r? @Nadrieril @RalfJung
@rustbot label +A-closures +A-patterns +T-opsem +T-lang
Augie Fackler