Add uwtable annotation to modules when required
When unwind tables are enabled with `-Cforce-unwind-tables=y`, Rust will annotate all functions with the `uwtable` annotation. However, this annotation is missing on modules, which leads to incorrect unwind tables being generated by LLVM for constructors (such as `asan.module_ctor`).
This was discovered because it leads to a crash in Linux when KASAN and dynamic shadow call stack are both enabled. In this scenario, the kernel uses the unwind tables to locate the `paciasp` and `autiasp` instructions in each function and patches the machine code at boot to use the shadow call stack instructions instead. However, LLVM's AArch64PointerAuth pass emits DWARF info for `paciasp` whenever `-g` is passed, but only emits DWARF info for `autiasp` when the `uwtable` attribute is present. Since the `uwtable` annotation is missing for modules, the relevant directives are generated for only the `autiasp` instruction in `asan.module_ctor`, and not for the `paciasp` instruction. This causes the kernel's dynamic SCS logic to patch the prolouge of `asan.module_ctor`, but not the epilogue. This leads to a crash as the shadow call stack becomes unbalanced.
The fact that LLVM doesn't use the same condition for whether to emit DWARF information for both instructions may be a separate bug in LLVM.
Relevant issue: https://github.com/llvm/llvm-project/issues/188234
AI assistance was used to determine the root cause of this crash from the observed symptoms, and to write the tests. Also thanks to @samitolvanen and @maurer for debugging this issue.
Similar to this previous PR of mine: rust-lang/rust#130824
codegen: re-enable FileCheck for scalable-vector tuple intrinsics
The FileCheck directives in `tests/codegen-llvm/scalable-vectors/tuple-intrinsics.rs` were not actually being checked: the test used `//@ build-pass`, and rust-lang/rust#155630 made that explicit by replacing it with `//@ skip-filecheck`.
This fixes the stale checks and re-enables FileCheck for the test. The `get` and `set` cases now use concrete index-0 wrappers, so they have stable `#[no_mangle]` names for FileCheck to match.
Fixesrust-lang/rust#155665.
r? @jieyouxu
The `needs-asm-mnemonic` directive was very general, but in practice was only
being used for `ret`. There are very few other mnemonics that it could
plausibly be useful for (e.g. `nop`), because any instruction that requires
arguments is probably going to be non-portable.
This PR replaces `needs-asm-mnemonic` with a simpler `needs-asm-ret` directive
that uses the same machinery as other simple needs directives.
If we happend to need more mnemonics in the future, we can just add more simple
directives as appropriate (e.g. `needs-asm-nop`).
Re-add call site inlining attributes
It is very odd that we are blindly setting the call site attributes to the same as the definition, because I think the point of the call site attributes is to differentiate between calls. Probably worth looking into later, but for now I think we should just be undoing the change we made by mistake.
This fixes the accidental regression from https://github.com/rust-lang/rust/pull/156242
Remove unsound `target_feature_inline_always` feature
## Summary
- Remove `target_feature_inline_always`
- Update stdarch generators to only use `#[inline]` & regenerate stdarch.
## Why?
Succinctly; the feature relies on LLVMs `AlwaysInlinerPass()` running before LLVMs heuristic based inliner pass. Which is not a basis for sound code.
This has been discussed in [the tracking issue](https://github.com/rust-lang/rust/issues/145574).
If the ordering of the passes were to change, of which they have in the past, it is very possible we could inline functions across callsites with mismatching target features leading to unsound code. Checks proposed in; https://github.com/rust-lang/rust/pull/155426 would only take into account caller -> callee which is not enough to guard against possibly of generating unsound code if the pass ordering were to change.
There doesn't seem to be a way, presently, this this mechanism to provide soundness guarantees nor does it seem like `AlwaysInlinerPass()` is a desired feature of LLVM, which this feature relies on.
r? @RalfJung
LLVM 23: Specify `returnaddress` intrinsic return type
https://github.com/llvm/llvm-project/pull/188464 made the return type of the intrinsic generic to support different pointer address spaces.
@rustbot label llvm-main
Have arrays' `drop_glue` just unsize and call the slice version
It's silly to emit two loops (because of the drop ladder -- just one in panic=abort) for every array length that's dropped when we can just polymorphize to the slice version.
Built atop rust-lang/rust#154327 to avoid conflicts later, so draft for now.
r? @WaffleLapkin
c-variadic: document `Clone` and `Drop` instances and require `VaArgSafe: Copy`
tracking issue: https://github.com/rust-lang/rust/issues/44930
Fixing some things that came up in the stabilization PR
r? tgross35
cc @kpreid
Fix: On wasm targets, call `panic_in_cleanup` if panic occurs in cleanup
Relies on https://github.com/rust-lang/llvm-project/pull/194.
Reland of https://github.com/rust-lang/rust/pull/151771.
Previously this was not correctly implemented. Each funclet may need its own terminate block, so this changes the `terminate_block` into a `terminate_blocks` `IndexVec` which can have a terminate_block for each funclet. We key on the first basic block of the funclet -- in particular, this is the start block for the old case of the top level terminate function.
Rather than using a catchswitch/catchpad pair, I used a cleanuppad. The reason for the pair is to avoid catching foreign exceptions on MSVC. On wasm, it seems that the catchswitch/catchpad pair is optimized back into a single cleanuppad and a catch_all instruction is emitted which will catch foreign exceptions. Because the new logic is only used on wasm, it seemed better to take the simpler approach seeing as they do the same thing.
- [ ] Add test for https://github.com/rust-lang/rust/issues/153948
Previously this was not correctly implemented. Each funclet may need its own terminate
block, so this changes the `terminate_block` into a `terminate_blocks` `IndexVec` which
can have a terminate_block for each funclet. We key on the first basic block of the
funclet -- in particular, this is the start block for the old case of the top level
terminate function.
Rather than using a catchswitch/catchpad pair, I used a cleanuppad. The reason for the
pair is to avoid catching foreign exceptions on MSVC. On wasm, it seems that the
catchswitch/catchpad pair is optimized back into a single cleanuppad and a catch_all
instruction is emitted which will catch foreign exceptions. Because the new logic is
only used on wasm, it seemed better to take the simpler approach seeing as they do the
same thing.
Prefer `-1` for `None`
Currently we pick "weird" numbers like `1114112` for `None::<char>`. While that's not *wrong*, it's kinda *unnatural* -- a human wouldn't make that choice.
This PR instead picks `-1` for thinge like `None::<char>` -- like [clang's `WEOF`](https://github.com/llvm/llvm-project/blob/63ae74b78a11f6c61136dbc445652929389eb9ab/libc/include/llvm-libc-macros/wchar-macros.h#L15) -- and `None::<bool>` and such.
Any enums with more than one niched value (so not `Result` nor `Option`) remain as they were before. Also we continue to use `0` when that's possible -- `-1` is only preferred when zero *isn't* possible.
---
Inspired when someone in discord posted an example like this <https://rust.godbolt.org/z/W94s9qdYW> and I thought it was odd that we're currently picking `-9223372036854775808` to be the value to store to mark an `Option<Vec<_>>` as `None`. (Especially since that needs an 8-byte immediate on x64, and writing `-1` is only a 4-byte immediate.)
* Implement core::arch::return_address and tests
Fix typo
Apply suggestions from code review
Wording/docs changes.
Co-authored-by: Ralf Jung <post@ralfj.de>
Change signature according to Ralf's comment
Fix call to `core::intrinsics::return_address()` according to the new signature
Add cranelift implementation for intrinsic
Change wording on `return_address!()` to be clear that returning a null pointer is best-effort.
Fix formatting of doc comment
Fix mistake in cranelift codegen
* Do not generate llvm.returnaddress on wasm
* Supress return_address test on miri
Currently we pick "weird" numbers like `1114112` for `None::<char>`. While that's not *wrong*, it's kinda *unnatural* -- a human wouldn't make that choice.
This PR instead picks `-1` for thinge like `None::<char>` -- like clang's `WEOF` -- and `None::<bool>` and such.
Any enums with more than one niched value (so not `Result` nor `Option`) remain as they were before.
Error on invalid macho section specifier
The macho section specifier used by `#[link_section = "..."]` is more strict than e.g. the one for elf. LLVM will error when you get it wrong, which is easy to do if you're used to elf. So, provide some guidance for the simplest mistakes, based on the LLVM validation.
Currently compilation fails with an LLVM error, see https://godbolt.org/z/WoE8EdK1K.
The LLVM validation logic is at
https://github.com/llvm/llvm-project/blob/a0f0d6342e0cd75b7f41e0e6aae0944393b68a62/llvm/lib/MC/MCSectionMachO.cpp#L199-L203
LLVM validates the other components of the section specifier too, but it feels a bit fragile to duplicate those checks. If you get that far, hopefully the LLVM errors will be sufficient to get unstuck.
---
sidequest from https://github.com/rust-lang/rust/pull/147811
r? JonathanBrouwer
specifically, is this the right place for this sort of validation? `rustc_attr_parsing` also does some validation.
Add intrinsic for launch-sized workgroup memory on GPUs
Workgroup memory is a memory region that is shared between all
threads in a workgroup on GPUs. Workgroup memory can be allocated
statically or after compilation, when launching a gpu-kernel.
The intrinsic added here returns the pointer to the memory that is
allocated at launch-time.
# Interface
With this change, workgroup memory can be accessed in Rust by
calling the new `gpu_launch_sized_workgroup_mem<T>() -> *mut T`
intrinsic.
It returns the pointer to workgroup memory guaranteeing that it is
aligned to at least the alignment of `T`.
The pointer is dereferencable for the size specified when launching the
current gpu-kernel (which may be the size of `T` but can also be larger
or smaller or zero).
All calls to this intrinsic return a pointer to the same address.
See the intrinsic documentation for more details.
## Alternative Interfaces
It was also considered to expose dynamic workgroup memory as extern
static variables in Rust, like they are represented in LLVM IR.
However, due to the pointer not being guaranteed to be dereferencable
(that depends on the allocated size at runtime), such a global must be
zero-sized, which makes global variables a bad fit.
# Implementation Details
Workgroup memory in amdgpu and nvptx lives in address space 3.
Workgroup memory from a launch is implemented by creating an
external global variable in address space 3. The global is declared with
size 0, as the actual size is only known at runtime. It is defined
behavior in LLVM to access an external global outside the defined size.
There is no similar way to get the allocated size of launch-sized
workgroup memory on amdgpu an nvptx, so users have to pass this
out-of-band or rely on target specific ways for now.
Tracking issue: rust-lang/rust#135516
Workgroup memory is a memory region that is shared between all
threads in a workgroup on GPUs. Workgroup memory can be allocated
statically or after compilation, when launching a gpu-kernel.
The intrinsic added here returns the pointer to the memory that is
allocated at launch-time.
# Interface
With this change, workgroup memory can be accessed in Rust by
calling the new `gpu_launch_sized_workgroup_mem<T>() -> *mut T`
intrinsic.
It returns the pointer to workgroup memory guaranteeing that it is
aligned to at least the alignment of `T`.
The pointer is dereferencable for the size specified when launching the
current gpu-kernel (which may be the size of `T` but can also be larger
or smaller or zero).
All calls to this intrinsic return a pointer to the same address.
See the intrinsic documentation for more details.
## Alternative Interfaces
It was also considered to expose dynamic workgroup memory as extern
static variables in Rust, like they are represented in LLVM IR.
However, due to the pointer not being guaranteed to be dereferencable
(that depends on the allocated size at runtime), such a global must be
zero-sized, which makes global variables a bad fit.
# Implementation Details
Workgroup memory in amdgpu and nvptx lives in address space 3.
Workgroup memory from a launch is implemented by creating an
external global variable in address space 3. The global is declared with
size 0, as the actual size is only known at runtime. It is defined
behavior in LLVM to access an external global outside the defined size.
There is no similar way to get the allocated size of launch-sized
workgroup memory on amdgpu an nvptx, so users have to pass this
out-of-band or rely on target specific ways for now.
Make `//@ skip-filecheck` a normal compiletest directive
The `skip-filecheck` directive is currently used by mir-opt tests, to suppress the default behaviour of running LLVM's `FileCheck` tool to check MIR output against FileCheck rules in the test file.
The `skip-filecheck` directive was not included in the big migration to `//@` directive syntax (https://github.com/rust-lang/rust/pull/121370), perhaps because it was parsed and processed in the *miropt-test-tools* helper crate, not in compiletest itself.
Recently I noticed that a small number of *codegen-llvm* tests were using the `//@ build-pass` directive, which has the non-obvious effect of skipping FileCheck in codegen tests. That's quite confusing, so I decided to have the mir-opt tests migrate over to a proper `//@ skip-filecheck` directive, which could then be used by codegen tests as well.
(I also added skip-filecheck support to assembly tests, which are very similar to codegen tests, though there are currently no assembly tests that actually use `//@ skip-filecheck`.)
---
Support for using `//@ build-pass` in codegen tests to skip FileCheck was introduced in https://github.com/rust-lang/rust/pull/113603. With hindsight, I think doing things that way was pretty clearly a mistake, and we'll be better off with `//@ skip-filecheck`.
r? jieyouxu
codegen: Copy to an alloca when the argument is neither by-val nor by-move for indirect pointer.
Fixes https://github.com/rust-lang/rust/issues/155241.
When a value is passed via an indirect pointer, the value needs to be copied to a new alloca. For x86_64-unknown-linux-gnu, `Thing` is the case:
```rust
#[derive(Clone, Copy)]
struct Thing(usize, usize, usize);
pub fn foo() {
let thing = Thing(0, 0, 0);
bar(thing);
assert_eq!(thing.0, 0);
}
#[inline(never)]
#[unsafe(no_mangle)]
pub fn bar(mut thing: Thing) {
thing.0 = 1;
}
```
Before passing the thing to the bar function, the thing needs to be copied to an alloca that is passed to bar.
```llvm
%0 = alloca [24 x i8], align 8
call void @llvm.memcpy.p0.p0.i64(ptr align 8 %0, ptr align 8 %thing, i64 24, i1 false)
call void @bar(ptr %0)
```
This patch applies the rule to the untupled arguments as well.
```rust
#![feature(fn_traits)]
#[derive(Clone, Copy)]
struct Thing(usize, usize, usize);
#[inline(never)]
#[unsafe(no_mangle)]
pub fn foo() {
let thing = (Thing(0, 0, 0),);
(|mut thing: Thing| {
thing.0 = 1;
}).call(thing);
assert_eq!(thing.0.0, 0);
}
```
For this case, this patch changes from
```llvm
; call example::foo::{closure#0}
call void @_RNCNvCs15qdZVLwHPA_7example3foo0B3_(ptr ..., ptr %thing)
```
to
```llvm
%0 = alloca [24 x i8], align 8
call void @llvm.memcpy.p0.p0.i64(ptr align 8 %0, ptr align 8 %thing, i64 24, i1 false)
; call example::foo::{closure#0}
call void @_RNCNvCs15qdZVLwHPA_7example3foo0B3_(ptr ..., ptr %0)
```
However, the same rule cannot be applied to tail calls that would be unsound, because the caller's stack frame is overwritten by the callee's stack frame. Fortunately, https://github.com/rust-lang/rust/pull/151143 has already handled the special case. We must not copy again.
No copy is needed for by-move arguments, because the argument is passed to the called "in-place".
No copy is also needed for by-val arguments, because the attribute implies that a hidden copy of the pointee is made between the caller and the callee.
NOTE: The patch has a trick for tail calls that we pass by-move. We can choose to copy an alloca even for by-move arguments, but tail calls require MUST-by-move.
Skipping FileCheck in codegen/assembly tests is normally not very useful, but a
small number of existing tests were using `//@ build-pass` to do so anyway, so
it's clearer for them to explicitly use `//@ skip-filecheck` instead.
Add autocast support for `x86amx`
Builds on rust-lang/rust#140763 by further adding autocasts for `x86amx` from/to vectors of size 8192 bits.
This also disables SIMD vector abi checks for the `"unadjusted"` abi because
- This is primarily used to link with LLVM intrinsics, which don't actually lower to function calls with vector arguments. Even with other cg backends, this is true.
- This ABI is internal and perma-unstable (and also super specific), so it is very unlikely that this will cause breakages.
- (The primary reason) Without doing this we can't actually use 8192 bit long vectors to represent `x86amx`
> Why do we need a bypass for `x86amx`? Can't we use a `#[lang_item]` or something?
If `x86amx` was a normal LLVM type, this approach would've worked and I would also prefer it. But LLVM specifies that
> No instruction is allowed for this type. There are no arguments, arrays, pointers, vectors or constants of this type.
So we can't treat it like a normal type at all -- even if we add it like a lang-item, we would still have to special-case everywhere to check if we are passing to the correct LLVM intrinsic, and only then use the `x86amx` type. IMO this is needlessly complex, and way worse than this solution, which just adds it to the autocast list in cg_llvm
r? codegen
Add regression test for dead code elimination with drop + panic
Add a codegen test for rust-lang/rust#114532.
The bug was that dead code elimination failed when a `Drop` impl contained a `panic!` and a potentially-panicking external function was called after the value was created. This was fixed since 1.82 but no regression test was added.
The test verifies that `foo()` compiles to just a call to `unknown()` + `ret void`, with no panic or panicking call in the function body.
Closesrust-lang/rust#114532
Remove fewer Storage calls in CopyProp and GVN
Modify the CopyProp and GVN MIR optimization passes to remove fewer `Storage{Live,Dead}` calls, allowing for better optimizations by LLVM - see rust-lang/rust#141649.
### Details
The idea is to use a new `MaybeUninitializedLocals` analysis and remove only the storage calls of locals that are maybe-uninit when accessed in a new location.
Guillaume Gomez