Add scalar support for offload
This PR adds scalar support to the offload feature. The scalar management has two main parts:
On the host side, each scalar arg is casted to `ix` type, zero extended to `i64` and passed to the kernel like that.
On the device, the each scalar arg (`i64` at that point), is truncated to `ix` and then casted to the original type.
r? @ZuseZ4
Generate openmp metadata
LLVM has an openmp-opt pass, which is part of the default O3 pipeline.
The pass bails if we don't have a global called openmp, so let's generate it if people enable our experimental offload feature. openmp is a superset of the offload feature, so they share optimizations.
In follow-up PRs I'll start verifying that LLVM optimizes Rust the way we want it.
r? compiler
Add amdgpu_dispatch_ptr intrinsic
There is an ongoing discussion in rust-lang/rust#150452 about using address spaces from the Rust language in some way.
As that discussion will likely not conclude soon, this PR adds one rustc_intrinsic with an addrspacecast to unblock getting basic information like launch and workgroup size and make it possible to implement something like `core::gpu`.
Add a rustc intrinsic `amdgpu_dispatch_ptr` to access the kernel dispatch packet on amdgpu.
The HSA kernel dispatch packet contains important information like the launch size and workgroup size.
The Rust intrinsic lowers to the `llvm.amdgcn.dispatch.ptr` LLVM intrinsic, which returns a `ptr addrspace(4)`, plus an addrspacecast to `addrspace(0)`, so it can be returned as a Rust reference.
The returned pointer/reference is valid for the whole program lifetime, and is therefore `'static`.
The return type of the intrinsic (`&'static ()`) does not mention the struct so that rustc does not need to know the exact struct type. An alternative would be to define the struct as lang item or add a generic argument to the function.
Is this ok or is there a better way (also, should it return a pointer instead of a reference)?
Short version:
```rust
#[cfg(target_arch = "amdgpu")]
pub fn amdgpu_dispatch_ptr() -> *const ();
```
Tracking issue: rust-lang/rust#135024
This reverts PR <https://github.com/rust-lang/rust/pull/130998> because
the added test seems to be flaky / non-deterministic, and has been
failing in unrelated PRs during merge CI.
Avoid should-fail in two ui tests and a codegen-llvm test
`should-fail` is only meant for testing the compiletest framework itself. It checks that the test runner itself panicked.
With this there are still a bunch of rustdoc-html tests that use it due to this test suite not supporting anything like `//@ doc-fail`.
Fix dso_local for external statics with linkage
Tracking issue of the feature: rust-lang/rust#127488
DSO local attributes are not correctly applied to extern statics with `#[linkage = "foo"]` as we generate an internal global for such statics, and the we evaluate (and apply) DSO attributes on the internal one instead.
Fix this by applying DSO local attributes on the actually extern ones, too.
Add a rustc intrinsic `amdgpu_dispatch_ptr` to access the kernel
dispatch packet on amdgpu.
The HSA kernel dispatch packet contains important information like the
launch size and workgroup size.
The Rust intrinsic lowers to the `llvm.amdgcn.dispatch.ptr` LLVM
intrinsic, which returns a `ptr addrspace(4)`, plus an addrspacecast to
`addrspace(0)`, so it can be returned as a Rust reference.
The returned pointer/reference is valid for the whole program lifetime,
and is therefore `'static`.
The return type of the intrinsic (`*const ()`) does not mention the
struct so that rustc does not need to know the exact struct type.
An alternative would be to define the struct as lang item or add a
generic argument to the function.
Short version:
```rust
#[cfg(target_arch = "amdgpu")]
pub fn amdgpu_dispatch_ptr() -> *const ();
```
Update offload test and verify that tgt_(un)register_lib have the right type
Apparently, we weren't running offload tests when Enzyme wasn't built. Time to fix that.
Also adds a test mode which generates the host IR, but does not expect device IR/artifacts. This way, we don't have to handle artifacts and paths in our tests.
Also removes some outdated documentation.
cc `@Kevinsala,` `@Sa4dUs`
closes: https://github.com/rust-lang/rust/issues/150415
~~blocked on `needs-offload` infrastructure landing in https://github.com/rust-lang/rust/pull/150427~~
Added codegen tests for different forms of `Option::or`
Adds tests to check the output of the different ways of writing `Option::or`
Fixesrust-lang/rust#124533
tests/codegen-llvm/some-non-zero-from-atomic-optimization.rs: New test
Closesrust-lang/rust#60044 which has one 👍 and one ❤️ vote and just **E-needs-test**.
Allow inline calls to offload intrinsic
Removes explicit insertion point handling and recovers the pointer at the end of the saved basic block.
r? `@ZuseZ4`
fixes: https://github.com/rust-lang/rust/issues/150413
This test currently doesn't fulfill its purpose, as `external dso_local`
can still match `external {{.*}}`. Fix this by using CHECK-NOT directives.
Also, this test is expanded to all platforms where it makes sense, instead
of restricting to loongarch.
Replace Rvalue::NullaryOp by a variant in mir::Operand.
Based on https://github.com/rust-lang/rust/pull/148151
This PR fully removes the MIR `Rvalue::NullaryOp`. After rust-lang/rust#148151, it was only useful for runtime checks like `ub_checks`, `contract_checks` and `overflow_checks`.
These are "runtime" checks, boolean constants that may only be `true` in codegen. It depends on a rustc flag passed to codegen, so we need to represent those flags cross-crate.
This PR replaces those runtime checks by special variants in MIR `ConstValue`. This allows code that expects constants to manipulate those as such, even if we may not always be able to evaluate them to actual scalars.
Move shared offload globals and define per-kernel globals once
This PR moves the shared LLVM global variables logic out of the `offload` intrinsic codegen and generates kernel-specific variables only ont he first call of the intrinsic.
r? `@ZuseZ4`
tracking:
- https://github.com/rust-lang/rust/issues/131513
`rustc_scalable_vector(N)`
Supercedes rust-lang/rust#118917.
Initial experimental implementation of rust-lang/rfcs#3838. Introduces a `rustc_scalable_vector(N)` attribute that can be applied to types with a single `[$ty]` field (for `u{16,32,64}`, `i{16,32,64}`, `f{32,64}`, `bool`). `rustc_scalable_vector` types are lowered to scalable vectors in the codegen backend.
As with any unstable feature, there will necessarily be follow-ups as we experiment and find cases that we've not considered or still need some logic to handle, but this aims to be a decent baseline to start from.
See rust-lang/rust#145052 for request for a lang experiment.
Introduces `BackendRepr::ScalableVector` corresponding to scalable
vector types annotated with `repr(scalable)` which lowers to a scalable
vector type in LLVM.
Co-authored-by: Jamie Cunliffe <Jamie.Cunliffe@arm.com>
fix va_list test by adding a llvmir signext check
s390x has no option to directly pass 32bit values therefor i32 parameters need an optional llvmir signext attribute.
Restrict spe_acc to PowerPC SPE targets
Update the tests, add powerpc-*-gnuspe testing, and create a distinct clobber_abi list for PowerPC SPE targets.
Note, the SPE target does not have vector, vector-scalar, or floating-point specific registers.
r? ```@Amanieu```
Add `ilog10` result range hints
This PR adds hints that the return value of `T::ilog10` will never exceed `T::MAX.ilog10()`.
This works because `ilog10` is a monotonically nondecreasing function, the maximum return value is reached at the max input value.
Assume the returned value in `.filter(…).count()`
Similar to how this helps in `slice::Iter::position`, LLVM sometimes loses track of how high this can get, so for `TrustedLen` iterators tell it what the upper bound is.
`c_variadic`: make `VaList` abi-compatible with C
tracking issue: https://github.com/rust-lang/rust/issues/44930
related PR: rust-lang/rust#144529
On some platforms, the C `va_list` type is actually a single-element array of a struct (on other platforms it is just a pointer). In C, arrays passed as function arguments expirience array-to-pointer decay, which means that C will pass a pointer to the array in the caller instead of the array itself, and modifications to the array in the callee will be visible to the caller (this does not match Rust by-value semantics). However, for `va_list`, the C standard explicitly states that it is undefined behaviour to use a `va_list` after it has been passed by value to a function (in Rust parlance, the `va_list` is moved, not copied). This matches Rust's pass-by-value semantics, meaning that when the C `va_list` type is a single-element array of a struct, the ABI will match C as long as the Rust type is always be passed indirectly.
In the old implementation, this ABI was achieved by having two separate types: `VaList` was the type that needed to be used when passing a `VaList` as a function parameter, whereas `VaListImpl` was the actual `va_list` type that was correct everywhere else. This however is quite confusing, as there are lots of footguns: it is easy to cause bugs by mixing them up (e.g. the C function `void foo(va_list va)` was equivalent to the Rust `fn foo(va: VaList)` whereas the C function `void bar(va_list* va)` was equivalent to the Rust `fn foo(va: *mut VaListImpl)`, not `fn foo(va: *mut VaList)` as might be expected); also converting from `VaListImpl` to `VaList` with `as_va_list()` had platform specific behaviour: on single-element array of a struct platforms it would return a `VaList` referencing the original `VaListImpl`, whereas on other platforms it would return a cioy,
In this PR, there is now just a single `VaList` type (renamed from `VaListImpl`) which represents the C `va_list` type and will just work in all positions. Instead of having a separate type just to make the ABI work, rust-lang/rust#144529 adds a `#[rustc_pass_indirectly_in_non_rustic_abis]` attribute, which when applied to a struct will force the struct to be passed indirectly by non-Rustic calling conventions. This PR then implements the `VaList` rework, making use of the new attribute on all platforms where the C `va_list` type is a single-element array of a struct.
Cleanup of the `VaList` API and implementation is also included in this PR: since it was decided it was OK to experiment with Rust requiring that not calling `va_end` is not undefined behaviour (https://github.com/rust-lang/rust/issues/141524#issuecomment-3028383594), I've removed the `with_copy` method as it was redundant to the `Clone` impl (the `Drop` impl of `VaList` is a no-op as `va_end` is a no-op on all known platforms).
Previous discussion: rust-lang/rust#141524 and [t-compiler > c_variadic API and ABI](https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/c_variadic.20API.20and.20ABI)
Tracking issue: https://github.com/rust-lang/rust/issues/44930
r? `@joshtriplett`
Stuart Cook