Auto merge of #154341 - RalfJung:retag-on-typed-copy, r=oli-obk

Make retags an implicit part of typed copies



Ever since Stacked Borrows was first implemented in Miri, that was done with `Retag` statements: given a place (usually a local variable), those statements find all references stored inside the place and refresh their tags to ensure the aliasing requirements are upheld. However, this is a somewhat unsatisfying approach for multiple reasons:
- It leaves open the [question](https://github.com/rust-lang/unsafe-code-guidelines/issues/371) of where to even put `Retag` statements. Over time, the AddRetag pass settled on one possible answer to this, but it wasn't very canonical.
- For assignments of the form `*ptr = expr`, if the assignment involves copying a reference, we probably want to do a retag -- but if we do a `Retag(*ptr)` as the next instruction, it can be non-trivial to argue that this even retags the right value, so we refrained from doing retags in that case. This has [come up](https://github.com/llvm/llvm-project/pull/160913#issuecomment-3341908717) as a potential issue for Rust making better use of LLVM "captures" annotations. (That said, there might be [other ways](https://github.com/rust-lang/unsafe-code-guidelines/issues/593#issuecomment-4328112031) to obtain this desired optimization.)
- Normal compilation avoids generating retags, but we still generate LLVM IR with `noalias`. What does that even mean? How do MIR optimization passes interact with retags? These are questions we have to figure out to make better use of aliasing information, but currently we can't even really ask such questions.

I think we should resolve all that by making retags part of what happens during a typed copy (a concept and interpreter infrastructure that did not exist yet when retags were initially introduced). Under this proposal, when executing a MIR assignment statement, what conceptually happens is as follows:
- We evaluate the LHS to a place.
- We evaluate the RHS to a value. This does a typed load from memory if needed, raising UB if memory does not contain a valid representation of the assignment's type.
- We walk that value, identify all references inside of it, and retag them. If this happens as part of passing a function argument, this is a protecting retag.
- We store (a representation of) the value into the place.

However, this semantics doesn't fully work: there's a mandatory MIR pass that turns expressions like `&mut ***ptr` into intermediate deref's. Those must *not* do any retags. So far this happened because the AddRetag pass did not add retags for assignments to deref temporaries, but that information is not recorded in cross-crate MIR. Therefore I instead added a field to `Rvalue::Use` to indicate whether this value should be retagged or not. A non-retagging copy seems like a sufficiently canonical primitive that we should be able to express it. Dealing with the fallout from that is a large chunk of the overall diff. (I also considered adding this field to `StatementKind::Assign` instead, but decided against that as we only actually need it for `Rvalue::Use`. I am not sure if this was the right call...)

This neatly answers the question of when retags should occur, and handles cases like `*ptr = expr`. It avoids traversing values twice in Miri. It makes codegen's use of `noalias` sound wrt the actual MIR that it is working on. It also gives us a target semantics to evaluate MIR opts against. However, I did not carefully check all MIR opts -- in particular, GVN needs a thorough look under the new semantics; it currently can turn alias-correct code into alias-incorrect code. (But this PR doesn't make things any worse for normal compilation where the retag indicator is anyway ignored.)

Another side-effect of this PR is that `-Zmiri-disable-validation` now also disables alias checking. It'd be nicer to keep them orthogonal but I find this an acceptable price to pay.

- [rustc benchmark results](https://github.com/rust-lang/rust/pull/154341#issuecomment-4125313290)
- [miri benchmark results](https://github.com/rust-lang/rust/pull/154341#issuecomment-4129840926)

compiler/rustc_borrowck/src/dataflow.rs

@@ -578,7 +578,6 @@ fn apply_primary_statement_effect(
             mir::StatementKind::FakeRead(..)
             | mir::StatementKind::SetDiscriminant { .. }
             | mir::StatementKind::StorageLive(..)
-            | mir::StatementKind::Retag { .. }
             | mir::StatementKind::PlaceMention(..)
             | mir::StatementKind::AscribeUserType(..)
             | mir::StatementKind::Coverage(..)

compiler/rustc_borrowck/src/diagnostics/conflict_errors.rs

@@ -4325,7 +4325,7 @@ fn annotate_argument_and_return_for_borrow(
                     // Otherwise, look at other types of assignment.
                     let assigned_from = match rvalue {
                         Rvalue::Ref(_, _, assigned_from) => assigned_from,
-                        Rvalue::Use(operand) => match operand {
+                        Rvalue::Use(operand, _) => match operand {
                             Operand::Copy(assigned_from) | Operand::Move(assigned_from) => {
                                 assigned_from
                             }

compiler/rustc_borrowck/src/diagnostics/explain_borrow.rs

@@ -875,7 +875,7 @@ fn was_captured_by_trait_object(&self, borrow: &BorrowData<'tcx>) -> bool {
                     match rvalue {
                         // If we see a use, we should check whether it is our data, and if so
                         // update the place that we're looking for to that new place.
-                        Rvalue::Use(operand) => match operand {
+                        Rvalue::Use(operand, _) => match operand {
                             Operand::Copy(place) | Operand::Move(place) => {
                                 if let Some(from) = place.as_local() {
                                     if from == target {

compiler/rustc_borrowck/src/diagnostics/mod.rs

@@ -243,7 +243,7 @@ pub(super) fn add_moved_or_invoked_closure_note(
         let mut target = place.local_or_deref_local();
         for stmt in &self.body[location.block].statements[location.statement_index..] {
             debug!("add_moved_or_invoked_closure_note: stmt={:?} target={:?}", stmt, target);
-            if let StatementKind::Assign(box (into, Rvalue::Use(from))) = &stmt.kind {
+            if let StatementKind::Assign(box (into, Rvalue::Use(from, _))) = &stmt.kind {
                 debug!("add_fnonce_closure_note: into={:?} from={:?}", into, from);
                 match from {
                     Operand::Copy(place) | Operand::Move(place)

compiler/rustc_borrowck/src/diagnostics/move_errors.rs

@@ -122,7 +122,7 @@ fn append_to_grouped_errors(
         // to a user variable is when initializing it.
         // If that ever stops being the case, then the ever initialized
         // flow could be used.
-        if let Some(StatementKind::Assign(box (place, Rvalue::Use(Operand::Move(move_from))))) =
+        if let Some(StatementKind::Assign(box (place, Rvalue::Use(Operand::Move(move_from), _)))) =
             self.body.basic_blocks[location.block]
                 .statements
                 .get(location.statement_index)

compiler/rustc_borrowck/src/diagnostics/mutability_errors.rs

@@ -1309,7 +1309,10 @@ fn suggest_make_local_mut(&self, err: &mut Diag<'_>, local: Local, name: Symbol)
                 if let Some(mir::Statement {
                     source_info: _,
                     kind:
-                        mir::StatementKind::Assign(box (_, mir::Rvalue::Use(mir::Operand::Copy(place)))),
+                        mir::StatementKind::Assign(box (
+                            _,
+                            mir::Rvalue::Use(mir::Operand::Copy(place), _),
+                        )),
                     ..
                 }) = first_assignment_stmt
                 {

compiler/rustc_borrowck/src/lib.rs

@@ -853,7 +853,6 @@ fn visit_after_early_statement_effect(
                 );
             }
             StatementKind::Nop
-            | StatementKind::Retag { .. }
             | StatementKind::SetDiscriminant { .. } => {
                 bug!("Statement not allowed in this MIR phase")
             }
@@ -1540,7 +1539,7 @@ fn consume_rvalue(
 
             Rvalue::ThreadLocalRef(_) => {}
 
-            Rvalue::Use(operand)
+            Rvalue::Use(operand, _)
             | Rvalue::Repeat(operand, _)
             | Rvalue::UnaryOp(_ /*un_op*/, operand)
             | Rvalue::Cast(_ /*cast_kind*/, operand, _ /*ty*/) => {
@@ -1693,7 +1692,7 @@ fn propagate_closure_used_mut_upvar(&mut self, operand: &Operand<'tcx>) {
                             StatementKind::Assign(box (
                                 _,
                                 Rvalue::Ref(_, _, source)
-                                | Rvalue::Use(Operand::Copy(source) | Operand::Move(source)),
+                                | Rvalue::Use(Operand::Copy(source) | Operand::Move(source), _),
                             )) => {
                                 propagate_closure_used_mut_place(self, source);
                             }

compiler/rustc_borrowck/src/polonius/legacy/loan_invalidations.rs

@@ -83,7 +83,6 @@ fn visit_statement(&mut self, statement: &Statement<'tcx>, location: Location) {
             }
             StatementKind::ConstEvalCounter
             | StatementKind::Nop
-            | StatementKind::Retag { .. }
             | StatementKind::BackwardIncompatibleDropHint { .. }
             | StatementKind::SetDiscriminant { .. } => {
                 bug!("Statement not allowed in this MIR phase")
@@ -294,7 +293,7 @@ fn consume_rvalue(&mut self, location: Location, rvalue: &Rvalue<'tcx>) {
 
             Rvalue::ThreadLocalRef(_) => {}
 
-            Rvalue::Use(operand)
+            Rvalue::Use(operand, _)
             | Rvalue::Repeat(operand, _)
             | Rvalue::UnaryOp(_ /*un_op*/, operand)
             | Rvalue::Cast(_ /*cast_kind*/, operand, _ /*ty*/) => {

compiler/rustc_borrowck/src/type_check/mod.rs

@@ -697,7 +697,6 @@ fn visit_statement(&mut self, stmt: &Statement<'tcx>, location: Location) {
             | StatementKind::FakeRead(..)
             | StatementKind::StorageLive(..)
             | StatementKind::StorageDead(..)
-            | StatementKind::Retag { .. }
             | StatementKind::Coverage(..)
             | StatementKind::ConstEvalCounter
             | StatementKind::PlaceMention(..)
@@ -1652,7 +1651,7 @@ fn visit_rvalue(&mut self, rvalue: &Rvalue<'tcx>, location: Location) {
                 .unwrap();
             }
 
-            Rvalue::Use(_)
+            Rvalue::Use(_, _)
             | Rvalue::UnaryOp(_, _)
             | Rvalue::CopyForDeref(_)
             | Rvalue::BinaryOp(..)
@@ -2215,8 +2214,8 @@ fn aggregate_field_ty(
     /// rvalue and will be unified with the inferred type.
     fn rvalue_user_ty(&self, rvalue: &Rvalue<'tcx>) -> Option<UserTypeAnnotationIndex> {
         match rvalue {
-            Rvalue::Use(_)
-            | Rvalue::ThreadLocalRef(_)
+            Rvalue::Use(..)
+            | Rvalue::ThreadLocalRef(..)
             | Rvalue::Repeat(..)
             | Rvalue::Ref(..)
             | Rvalue::RawPtr(..)

compiler/rustc_codegen_cranelift/src/base.rs

@@ -620,7 +620,7 @@ fn codegen_stmt<'tcx>(fx: &mut FunctionCx<'_, '_, 'tcx>, cur_block: Block, stmt:
             let lval = codegen_place(fx, to_place_and_rval.0);
             let dest_layout = lval.layout();
             match to_place_and_rval.1 {
-                Rvalue::Use(ref operand) => {
+                Rvalue::Use(ref operand, _) => {
                     let val = codegen_operand(fx, operand);
                     lval.write_cvalue(fx, val);
                 }
@@ -909,7 +909,6 @@ fn is_wide_ptr<'tcx>(fx: &FunctionCx<'_, '_, 'tcx>, ty: Ty<'tcx>) -> bool {
         | StatementKind::ConstEvalCounter
         | StatementKind::Nop
         | StatementKind::FakeRead(..)
-        | StatementKind::Retag { .. }
         | StatementKind::PlaceMention(..)
         | StatementKind::BackwardIncompatibleDropHint { .. }
         | StatementKind::AscribeUserType(..) => {}

compiler/rustc_codegen_cranelift/src/constant.rs

@@ -592,7 +592,7 @@ pub(crate) fn mir_operand_get_const_val<'tcx>(
                                         };
                                     computed_scalar_int = Some(scalar_int);
                                 }
-                                Rvalue::Use(operand) => {
+                                Rvalue::Use(operand, _) => {
                                     computed_scalar_int = mir_operand_get_const_val(fx, operand)
                                 }
                                 _ => return None,
@@ -613,7 +613,6 @@ pub(crate) fn mir_operand_get_const_val<'tcx>(
                         | StatementKind::SetDiscriminant { .. }
                         | StatementKind::StorageLive(_)
                         | StatementKind::StorageDead(_)
-                        | StatementKind::Retag(_, _)
                         | StatementKind::AscribeUserType(_, _)
                         | StatementKind::PlaceMention(..)
                         | StatementKind::Coverage(_)

compiler/rustc_codegen_ssa/src/mir/mod.rs

@@ -366,9 +366,12 @@ fn optimize_use_clone<'a, 'tcx, Bx: BuilderMethods<'a, 'tcx>>(
                 bb.terminator().source_info,
                 mir::StatementKind::Assign(Box::new((
                     *destination,
-                    mir::Rvalue::Use(mir::Operand::Copy(
-                        arg_place.project_deeper(&[mir::ProjectionElem::Deref], tcx),
-                    )),
+                    mir::Rvalue::Use(
+                        mir::Operand::Copy(
+                            arg_place.project_deeper(&[mir::ProjectionElem::Deref], tcx),
+                        ),
+                        mir::WithRetag::Yes,
+                    ),
                 ))),
             ));
 

compiler/rustc_codegen_ssa/src/mir/rvalue.rs

@@ -23,7 +23,7 @@ pub(crate) fn codegen_rvalue(
         rvalue: &mir::Rvalue<'tcx>,
     ) {
         match *rvalue {
-            mir::Rvalue::Use(ref operand) => {
+            mir::Rvalue::Use(ref operand, _) => {
                 if let mir::Operand::Constant(const_op) = operand {
                     let val = self.eval_mir_constant(&const_op);
                     if val.all_bytes_uninit(self.cx.tcx()) {
@@ -658,7 +658,7 @@ pub(crate) fn codegen_rvalue_operand(
                 };
                 OperandRef { val: OperandValue::Immediate(static_), layout, move_annotation: None }
             }
-            mir::Rvalue::Use(ref operand) => self.codegen_operand(bx, operand),
+            mir::Rvalue::Use(ref operand, _) => self.codegen_operand(bx, operand),
             mir::Rvalue::Repeat(ref elem, len_const) => {
                 // All arrays have `BackendRepr::Memory`, so only the ZST cases
                 // end up here. Anything else forces the destination local to be

compiler/rustc_codegen_ssa/src/mir/statement.rs

@@ -95,7 +95,6 @@ pub(crate) fn codegen_statement(&mut self, bx: &mut Bx, statement: &mir::Stateme
                 bx.memcpy(dst, align, src, align, bytes, crate::MemFlags::empty(), None);
             }
             mir::StatementKind::FakeRead(..)
-            | mir::StatementKind::Retag { .. }
             | mir::StatementKind::AscribeUserType(..)
             | mir::StatementKind::ConstEvalCounter
             | mir::StatementKind::PlaceMention(..)

compiler/rustc_const_eval/src/check_consts/check.rs

@@ -570,7 +570,7 @@ fn visit_rvalue(&mut self, rvalue: &Rvalue<'tcx>, location: Location) {
         match rvalue {
             Rvalue::ThreadLocalRef(_) => self.check_op(ops::ThreadLocalAccess),
 
-            Rvalue::Use(_)
+            Rvalue::Use(..)
             | Rvalue::CopyForDeref(..)
             | Rvalue::Repeat(..)
             | Rvalue::Discriminant(..) => {}
@@ -725,7 +725,6 @@ fn visit_statement(&mut self, statement: &Statement<'tcx>, location: Location) {
             | StatementKind::FakeRead(..)
             | StatementKind::StorageLive(_)
             | StatementKind::StorageDead(_)
-            | StatementKind::Retag { .. }
             | StatementKind::PlaceMention(..)
             | StatementKind::AscribeUserType(..)
             | StatementKind::Coverage(..)

compiler/rustc_const_eval/src/check_consts/qualifs.rs

@@ -231,7 +231,7 @@ pub fn in_rvalue<'tcx, Q, F>(
 
         Rvalue::CopyForDeref(place) => in_place::<Q, _>(cx, in_local, place.as_ref()),
 
-        Rvalue::Use(operand)
+        Rvalue::Use(operand, _)
         | Rvalue::Repeat(operand, _)
         | Rvalue::UnaryOp(_, operand)
         | Rvalue::Cast(_, operand, _) => in_operand::<Q, _>(cx, in_local, operand),

compiler/rustc_const_eval/src/const_eval/machine.rs

@@ -1,6 +1,6 @@
 use std::borrow::{Borrow, Cow};
-use std::fmt;
 use std::hash::Hash;
+use std::{fmt, mem};
 
 use rustc_abi::{Align, FIRST_VARIANT, Size};
 use rustc_ast::Mutability;
@@ -21,8 +21,8 @@
 use crate::errors::{LongRunning, LongRunningWarn};
 use crate::interpret::{
     self, AllocId, AllocInit, AllocRange, ConstAllocation, CtfeProvenance, FnArg, Frame,
-    GlobalAlloc, ImmTy, InterpCx, InterpResult, OpTy, PlaceTy, Pointer, RangeSet, Scalar,
-    compile_time_machine, ensure_monomorphic_enough, err_inval, interp_ok, throw_exhaust,
+    GlobalAlloc, ImmTy, InterpCx, InterpResult, OpTy, PlaceTy, Pointer, RangeSet, RetagMode,
+    Scalar, compile_time_machine, ensure_monomorphic_enough, err_inval, interp_ok, throw_exhaust,
     throw_inval, throw_ub, throw_ub_format, throw_unsup, throw_unsup_format,
     type_implements_dyn_trait,
 };
@@ -67,6 +67,9 @@ pub struct CompileTimeMachine<'tcx> {
 
     /// A cache of "data range" computations for unions (i.e., the offsets of non-padding bytes).
     union_data_ranges: FxHashMap<Ty<'tcx>, RangeSet>,
+
+    /// The current retag mode.
+    retag_mode: RetagMode,
 }
 
 #[derive(Copy, Clone)]
@@ -102,6 +105,7 @@ pub(crate) fn new(
             check_alignment,
             static_root_ids: None,
             union_data_ranges: FxHashMap::default(),
+            retag_mode: RetagMode::Default,
         }
     }
 }
@@ -824,9 +828,12 @@ fn before_access_global(
 
     fn retag_ptr_value(
         ecx: &mut InterpCx<'tcx, Self>,
-        _kind: mir::RetagKind,
         val: &ImmTy<'tcx, CtfeProvenance>,
-    ) -> InterpResult<'tcx, ImmTy<'tcx, CtfeProvenance>> {
+        _ty: Ty<'tcx>,
+    ) -> InterpResult<'tcx, Option<ImmTy<'tcx, CtfeProvenance>>> {
+        if matches!(ecx.machine.retag_mode, RetagMode::None | RetagMode::Raw) {
+            return interp_ok(None);
+        }
         // If it's a frozen shared reference that's not already immutable, potentially make it immutable.
         // (Do nothing on `None` provenance, that cannot store immutability anyway.)
         if let ty::Ref(_, ty, mutbl) = val.layout.ty.kind()
@@ -850,12 +857,23 @@ fn retag_ptr_value(
                 // even when there is interior mutability.)
                 place.map_provenance(CtfeProvenance::as_shared_ref)
             };
-            interp_ok(ImmTy::from_immediate(new_place.to_ref(ecx), val.layout))
+            interp_ok(Some(ImmTy::from_immediate(new_place.to_ref(ecx), val.layout)))
         } else {
-            interp_ok(val.clone())
+            interp_ok(None)
         }
     }
 
+    fn with_retag_mode<T>(
+        ecx: &mut InterpCx<'tcx, Self>,
+        mode: RetagMode,
+        f: impl FnOnce(&mut InterpCx<'tcx, Self>) -> InterpResult<'tcx, T>,
+    ) -> InterpResult<'tcx, T> {
+        let old_mode = mem::replace(&mut ecx.machine.retag_mode, mode);
+        let ret = f(ecx);
+        ecx.machine.retag_mode = old_mode;
+        ret
+    }
+
     fn before_memory_write(
         _tcx: TyCtxtAt<'tcx>,
         _machine: &mut Self,

compiler/rustc_const_eval/src/interpret/call.rs

@@ -7,7 +7,7 @@
 use either::{Left, Right};
 use rustc_abi::{self as abi, ExternAbi, FieldIdx, Integer, VariantIdx};
 use rustc_hir::def_id::DefId;
-use rustc_hir::find_attr;
+use rustc_hir::{LangItem, find_attr};
 use rustc_middle::ty::layout::{IntegerExt, TyAndLayout};
 use rustc_middle::ty::{self, AdtDef, Instance, Ty, Unnormalized, VariantDef};
 use rustc_middle::{bug, mir, span_bug};
@@ -16,12 +16,11 @@
 use tracing::{info, instrument, trace};
 
 use super::{
-    CtfeProvenance, FnVal, ImmTy, InterpCx, InterpResult, MPlaceTy, Machine, OpTy, PlaceTy,
-    Projectable, Provenance, ReturnAction, ReturnContinuation, Scalar, interp_ok, throw_ub,
-    throw_ub_format,
+    CtfeProvenance, EnteredTraceSpan, FnVal, ImmTy, InterpCx, InterpResult, MPlaceTy, Machine,
+    OpTy, PlaceTy, Projectable, Provenance, RetagMode, ReturnAction, ReturnContinuation, Scalar,
+    interp_ok, throw_ub, throw_ub_format,
 };
 use crate::enter_trace_span;
-use crate::interpret::EnteredTraceSpan;
 
 /// An argument passed to a function.
 #[derive(Clone, Debug)]
@@ -279,6 +278,7 @@ fn pass_argument<'x, 'y>(
         callee_arg: &mir::Place<'tcx>,
         callee_ty: Ty<'tcx>,
         already_live: bool,
+        is_drop_in_place: bool,
     ) -> InterpResult<'tcx>
     where
         'tcx: 'x,
@@ -323,7 +323,16 @@ fn pass_argument<'x, 'y>(
             self.storage_live_dyn(local, meta)?;
         }
         // Now we can finally actually evaluate the callee place.
-        let callee_arg = self.eval_place(*callee_arg)?;
+        let mut callee_arg = self.eval_place(*callee_arg)?;
+        // drop_in_place has a signature which says that the first argument is `*mut T`
+        // but really it's `&mut T`. This is where we handle that terrible hack in
+        // the MIR semantics.
+        // FIXME(#154274): remove this hack.
+        if is_drop_in_place && callee_arg_idx == 0 {
+            let pointee_ty = callee_arg.layout.ty.builtin_deref(true).unwrap();
+            let mutref_ty = Ty::new_mut_ref(*self.tcx, self.tcx.lifetimes.re_erased, pointee_ty);
+            callee_arg = callee_arg.transmute(self.layout_of(mutref_ty)?, self)?;
+        }
         // We allow some transmutes here.
         // FIXME: Depending on the PassMode, this should reset some padding to uninitialized. (This
         // is true for all `copy_op`, but there are a lot of special cases for argument passing
@@ -455,77 +464,93 @@ pub fn init_stack_frame(
         // Determine whether there is a special VaList argument. This is always the
         // last argument, and since arguments start at index 1 that's `arg_count`.
         let va_list_arg = callee_fn_abi.c_variadic.then(|| mir::Local::from_usize(body.arg_count));
-        for local in body.args_iter() {
-            // Update the span that we show in case of an error to point to this argument.
-            self.frame_mut().loc = Right(body.local_decls[local].source_info.span);
-            // Construct the destination place for this argument. At this point all
-            // locals are still dead, so we cannot construct a `PlaceTy`.
-            let dest = mir::Place::from(local);
-            // `layout_of_local` does more than just the instantiation we need to get the
-            // type, but the result gets cached so this avoids calling the instantiation
-            // query *again* the next time this local is accessed.
-            let ty = self.layout_of_local(self.frame(), local, None)?.ty;
-            if Some(local) == va_list_arg {
-                // This is the last callee-side argument of a variadic function.
-                // This argument is a VaList holding the remaining caller-side arguments.
-                self.storage_live(local)?;
+        // Part of the hack for #154274, see `pass_argument`.
+        let is_drop_in_place = {
+            let def_id = body.source.def_id();
+            self.tcx.is_lang_item(def_id, LangItem::DropInPlace)
+                || self.tcx.is_lang_item(def_id, LangItem::AsyncDropInPlace)
+        };
 
-                let place = self.eval_place(dest)?;
-                let mplace = self.force_allocation(&place)?;
+        // During argument passing, we want retagging with protectors.
+        M::with_retag_mode(self, RetagMode::FnEntry, |ecx| {
+            for local in body.args_iter() {
+                // Update the span that we show in case of an error to point to this argument.
+                ecx.frame_mut().loc = Right(body.local_decls[local].source_info.span);
+                // Construct the destination place for this argument. At this point all
+                // locals are still dead, so we cannot construct a `PlaceTy`.
+                let dest = mir::Place::from(local);
+                // `layout_of_local` does more than just the instantiation we need to get the
+                // type, but the result gets cached so this avoids calling the instantiation
+                // query *again* the next time this local is accessed.
+                let ty = ecx.layout_of_local(ecx.frame(), local, None)?.ty;
+                if Some(local) == va_list_arg {
+                    // This is the last callee-side argument of a variadic function.
+                    // This argument is a VaList holding the remaining caller-side arguments.
+                    ecx.storage_live(local)?;
 
-                // Consume the remaining arguments by putting them into the variable argument
-                // list.
-                let varargs = self.allocate_varargs(
-                    &mut caller_args,
-                    // "Ignored" arguments aren't actually passed, so the callee should also
-                    // ignore them. (`pass_argument` does this for regular arguments.)
-                    (&mut callee_args_abis).filter(|(_, abi)| !abi.is_ignore()),
-                )?;
-                // When the frame is dropped, these variable arguments are deallocated.
-                self.frame_mut().va_list = varargs.clone();
-                let key = self.va_list_ptr(varargs.into());
+                    let place = ecx.eval_place(dest)?;
+                    let mplace = ecx.force_allocation(&place)?;
 
-                // Zero the VaList, so it is fully initialized.
-                self.write_bytes_ptr(mplace.ptr(), (0..mplace.layout.size.bytes()).map(|_| 0u8))?;
+                    // Consume the remaining arguments by putting them into the variable argument
+                    // list.
+                    let varargs = ecx.allocate_varargs(
+                        &mut caller_args,
+                        // "Ignored" arguments aren't actually passed, so the callee should also
+                        // ignore them. (`pass_argument` does this for regular arguments.)
+                        (&mut callee_args_abis).filter(|(_, abi)| !abi.is_ignore()),
+                    )?;
+                    // When the frame is dropped, these variable arguments are deallocated.
+                    ecx.frame_mut().va_list = varargs.clone();
+                    let key = ecx.va_list_ptr(varargs.into());
 
-                // Store the "key" pointer in the right field.
-                let key_mplace = self.va_list_key_field(&mplace)?;
-                self.write_pointer(key, &key_mplace)?;
-            } else if Some(local) == body.spread_arg {
-                // Make the local live once, then fill in the value field by field.
-                self.storage_live(local)?;
-                // Must be a tuple
-                let ty::Tuple(fields) = ty.kind() else {
-                    span_bug!(self.cur_span(), "non-tuple type for `spread_arg`: {ty}")
-                };
-                for (i, field_ty) in fields.iter().enumerate() {
-                    let dest = dest.project_deeper(
-                        &[mir::ProjectionElem::Field(FieldIdx::from_usize(i), field_ty)],
-                        *self.tcx,
-                    );
+                    // Zero the VaList, so it is fully initialized.
+                    ecx.write_bytes_ptr(
+                        mplace.ptr(),
+                        (0..mplace.layout.size.bytes()).map(|_| 0u8),
+                    )?;
+
+                    // Store the "key" pointer in the right field.
+                    let key_mplace = ecx.va_list_key_field(&mplace)?;
+                    ecx.write_pointer(key, &key_mplace)?;
+                } else if Some(local) == body.spread_arg {
+                    // Make the local live once, then fill in the value field by field.
+                    ecx.storage_live(local)?;
+                    // Must be a tuple
+                    let ty::Tuple(fields) = ty.kind() else {
+                        span_bug!(ecx.cur_span(), "non-tuple type for `spread_arg`: {ty}")
+                    };
+                    for (i, field_ty) in fields.iter().enumerate() {
+                        let dest = dest.project_deeper(
+                            &[mir::ProjectionElem::Field(FieldIdx::from_usize(i), field_ty)],
+                            *ecx.tcx,
+                        );
+                        let (idx, callee_abi) = callee_args_abis.next().unwrap();
+                        ecx.pass_argument(
+                            &mut caller_args,
+                            callee_abi,
+                            idx,
+                            &dest,
+                            field_ty,
+                            /* already_live */ true,
+                            is_drop_in_place,
+                        )?;
+                    }
+                } else {
+                    // Normal argument. Cannot mark it as live yet, it might be unsized!
                     let (idx, callee_abi) = callee_args_abis.next().unwrap();
-                    self.pass_argument(
+                    ecx.pass_argument(
                         &mut caller_args,
                         callee_abi,
                         idx,
                         &dest,
-                        field_ty,
-                        /* already_live */ true,
+                        ty,
+                        /* already_live */ false,
+                        is_drop_in_place,
                     )?;
                 }
-            } else {
-                // Normal argument. Cannot mark it as live yet, it might be unsized!
-                let (idx, callee_abi) = callee_args_abis.next().unwrap();
-                self.pass_argument(
-                    &mut caller_args,
-                    callee_abi,
-                    idx,
-                    &dest,
-                    ty,
-                    /* already_live */ false,
-                )?;
             }
-        }
+            interp_ok(())
+        })?;
 
         // Don't forget to check the return type!
         self.frame_mut().loc = Right(body.local_decls[mir::RETURN_PLACE].source_info.span);

compiler/rustc_const_eval/src/interpret/machine.rs

@@ -40,6 +40,21 @@ pub enum ReturnAction {
     NoCleanup,
 }
 
+/// The currently active retagging mode.
+#[derive(Eq, PartialEq, Debug, Copy, Clone)]
+pub enum RetagMode {
+    /// A regular retag.
+    Default,
+    /// Retag preparing for a two-phase borrow.
+    TwoPhase,
+    /// The initial retag of arguments when entering a function.
+    FnEntry,
+    /// Retagging for reference-to-raw-pointer cast.
+    Raw,
+    /// No retagging.
+    None,
+}
+
 /// Whether this kind of memory is allowed to leak
 pub trait MayLeak: Copy {
     fn may_leak(self) -> bool;
@@ -481,25 +496,28 @@ fn before_memory_deallocation(
     }
 
     /// Executes a retagging operation for a single pointer.
-    /// Returns the possibly adjusted pointer.
+    /// Returns the possibly adjusted pointer. Return `None` if the pointer
+    /// was left unchanged.
+    ///
+    /// `ty` is the full type of the pointer. This is not the same as `val.layout.ty` for boxes
+    /// where `val` is just the inner raw pointer, but `ty` is the entire `Box` type.
     #[inline]
     fn retag_ptr_value(
         _ecx: &mut InterpCx<'tcx, Self>,
-        _kind: mir::RetagKind,
-        val: &ImmTy<'tcx, Self::Provenance>,
-    ) -> InterpResult<'tcx, ImmTy<'tcx, Self::Provenance>> {
-        interp_ok(val.clone())
+        _val: &ImmTy<'tcx, Self::Provenance>,
+        _ty: Ty<'tcx>,
+    ) -> InterpResult<'tcx, Option<ImmTy<'tcx, Self::Provenance>>> {
+        interp_ok(None)
     }
 
-    /// Executes a retagging operation on a compound value.
-    /// Replaces all pointers stored in the given place.
-    #[inline]
-    fn retag_place_contents(
-        _ecx: &mut InterpCx<'tcx, Self>,
-        _kind: mir::RetagKind,
-        _place: &PlaceTy<'tcx, Self::Provenance>,
-    ) -> InterpResult<'tcx> {
-        interp_ok(())
+    /// Invoke `f` in a state where calls to `retag_ptr_value` will use the given retag mode.
+    #[inline(always)]
+    fn with_retag_mode<T>(
+        ecx: &mut InterpCx<'tcx, Self>,
+        _mode: RetagMode,
+        f: impl FnOnce(&mut InterpCx<'tcx, Self>) -> InterpResult<'tcx, T>,
+    ) -> InterpResult<'tcx, T> {
+        f(ecx)
     }
 
     /// Called on places used for in-place function argument and return value handling.

compiler/rustc_const_eval/src/interpret/mod.rs

@@ -29,7 +29,9 @@
     HasStaticRootDefId, InternError, InternKind, intern_const_alloc_for_constprop,
     intern_const_alloc_recursive,
 };
-pub use self::machine::{AllocMap, Machine, MayLeak, ReturnAction, compile_time_machine};
+pub use self::machine::{
+    AllocMap, Machine, MayLeak, RetagMode, ReturnAction, compile_time_machine,
+};
 pub use self::memory::{AllocInfo, AllocKind, AllocRef, AllocRefMut, FnVal, Memory, MemoryKind};
 use self::operand::Operand;
 pub use self::operand::{ImmTy, Immediate, OpTy};

compiler/rustc_const_eval/src/interpret/place.rs

@@ -844,17 +844,20 @@ fn copy_op_inner(
     ) -> InterpResult<'tcx> {
         // These are technically *two* typed copies: `src` is a not-yet-loaded value,
         // so we're doing a typed copy at `src` type from there to some intermediate storage.
-        // And then we're doing a second typed copy from that intermediate storage to `dest`.
-        // But as an optimization, we only make a single direct copy here.
+        // And then we're doing a second typed copy at `dest` type from that intermediate storage to
+        // `dest`. But as an optimization, we only make a single direct copy here.
 
         // Do the actual copy.
         self.copy_op_no_validate(src, dest, allow_transmute)?;
 
         if M::enforce_validity(self, dest.layout()) {
             let dest = dest.to_place();
-            // Given that there were two typed copies, we have to ensure this is valid at both types,
-            // and we have to ensure this loses provenance and padding according to both types.
-            // But if the types are identical, we only do one pass.
+            // Given that there were two typed copies, we have to ensure this is valid at both
+            // types, and we have to ensure this loses provenance and padding according to both
+            // types. We also transmute both ways: when transmuting `*ptr` from `&T` to `*const T`,
+            // it seems nice to ensure that the resulting pointer value indeed is derived from a
+            // shared reference.
+            // But if the types are identical, that is strictly redundant so we only do one pass.
             if src.layout().ty != dest.layout().ty {
                 self.validate_operand(
                     &dest.transmute(src.layout(), self)?,

compiler/rustc_const_eval/src/interpret/step.rs

@@ -16,10 +16,9 @@
 use tracing::{info, instrument, trace};
 
 use super::{
-    FnArg, FnVal, ImmTy, Immediate, InterpCx, InterpResult, Machine, MemPlaceMeta, PlaceTy,
-    Projectable, interp_ok, throw_ub, throw_unsup_format,
+    EnteredTraceSpan, FnArg, FnVal, ImmTy, Immediate, InterpCx, InterpResult, Machine,
+    MemPlaceMeta, PlaceTy, Projectable, RetagMode, interp_ok, throw_ub, throw_unsup_format,
 };
-use crate::interpret::EnteredTraceSpan;
 use crate::{enter_trace_span, util};
 
 struct EvaluatedCalleeAndArgs<'tcx, M: Machine<'tcx>> {
@@ -112,12 +111,6 @@ pub fn eval_statement(&mut self, stmt: &mir::Statement<'tcx>) -> InterpResult<'t
             // interpreter is solely intended for borrowck'ed code.
             FakeRead(..) => {}
 
-            // Stacked Borrows.
-            Retag(kind, place) => {
-                let dest = self.eval_place(**place)?;
-                M::retag_place_contents(self, *kind, &dest)?;
-            }
-
             Intrinsic(box intrinsic) => self.eval_nondiverging_intrinsic(intrinsic)?,
 
             // Evaluate the place expression, without reading from it.
@@ -177,10 +170,11 @@ pub fn eval_rvalue_into_place(
                 self.write_pointer(ptr, &dest)?;
             }
 
-            Use(ref operand) => {
+            Use(ref operand, with_retag) => {
                 // Avoid recomputing the layout
                 let op = self.eval_operand(operand, Some(dest.layout))?;
-                self.copy_op(&op, &dest)?;
+                let mode = if with_retag.yes() { RetagMode::Default } else { RetagMode::None };
+                M::with_retag_mode(self, mode, |ecx| ecx.copy_op(&op, &dest))?;
             }
 
             CopyForDeref(_) => bug!("`CopyForDeref` in runtime MIR"),
@@ -214,18 +208,26 @@ pub fn eval_rvalue_into_place(
             Ref(_, borrow_kind, place) => {
                 let src = self.eval_place(place)?;
                 let place = self.force_allocation(&src)?;
-                let val = ImmTy::from_immediate(place.to_ref(self), dest.layout);
-                // A fresh reference was created, make sure it gets retagged.
-                let val = M::retag_ptr_value(
-                    self,
-                    if borrow_kind.is_two_phase_borrow() {
-                        mir::RetagKind::TwoPhase
-                    } else {
-                        mir::RetagKind::Default
-                    },
-                    &val,
-                )?;
-                self.write_immediate(*val, &dest)?;
+                let mut val = ImmTy::from_immediate(place.to_ref(self), dest.layout);
+                // A fresh reference was created, make sure it gets retagged with the right mode.
+                let mode = if borrow_kind.is_two_phase_borrow() {
+                    RetagMode::TwoPhase
+                } else {
+                    RetagMode::Default
+                };
+                M::with_retag_mode(self, mode, |ecx| {
+                    // If validation is disabled, we still want to do this retag. This is because
+                    // const-eval disables validation for performance reasons but wants to retag
+                    // shared references. So we add a bit of a hack here to do the retag manually
+                    // if the write would not incur validation.
+                    if !M::enforce_validity(ecx, val.layout) {
+                        if let Some(new_val) = M::retag_ptr_value(ecx, &val, val.layout.ty)? {
+                            val = new_val;
+                        }
+                    }
+                    // Now do the actual write.
+                    ecx.write_immediate(*val, &dest)
+                })?;
             }
 
             RawPtr(kind, place) => {
@@ -244,8 +246,11 @@ pub fn eval_rvalue_into_place(
                 if !place_base_raw && !kind.is_fake() {
                     // If this was not already raw, it needs retagging -- except for "fake"
                     // raw borrows whose defining property is that they do not get retagged.
-                    val = M::retag_ptr_value(self, mir::RetagKind::Raw, &val)?;
+                    val = M::with_retag_mode(self, RetagMode::Raw, |ecx| {
+                        interp_ok(M::retag_ptr_value(ecx, &val, val.layout.ty)?.unwrap_or(val))
+                    })?;
                 }
+                // This writes a raw pointer so it will not do any retags.
                 self.write_immediate(*val, &dest)?;
             }
 

compiler/rustc_const_eval/src/interpret/validity.rs

@@ -21,8 +21,7 @@
 use rustc_hir as hir;
 use rustc_middle::bug;
 use rustc_middle::mir::interpret::{
-    InterpErrorKind, InvalidMetaKind, Misalignment, Provenance, UnsupportedOpInfo, alloc_range,
-    interp_ok,
+    InterpErrorKind, InvalidMetaKind, Misalignment, Provenance, alloc_range, interp_ok,
 };
 use rustc_middle::ty::layout::{LayoutCx, TyAndLayout};
 use rustc_middle::ty::{self, Ty};
@@ -108,17 +107,17 @@ macro_rules! try_validation {
     }};
 }
 
-#[derive(Debug, Clone, Copy)]
-enum PointerKind {
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum PtrKind {
     Ref(Mutability),
     Box,
 }
 
-impl fmt::Display for PointerKind {
+impl fmt::Display for PtrKind {
     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
         let str = match self {
-            PointerKind::Ref(_) => "reference",
-            PointerKind::Box => "box",
+            PtrKind::Ref(_) => "reference",
+            PtrKind::Box => "box",
         };
         write!(f, "{str}")
     }
@@ -154,11 +153,11 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
     }
 }
 
-impl From<PointerKind> for ExpectedKind {
-    fn from(x: PointerKind) -> ExpectedKind {
+impl From<PtrKind> for ExpectedKind {
+    fn from(x: PtrKind) -> ExpectedKind {
         match x {
-            PointerKind::Box => ExpectedKind::Box,
-            PointerKind::Ref(_) => ExpectedKind::Reference,
+            PtrKind::Box => ExpectedKind::Box,
+            PtrKind::Ref(_) => ExpectedKind::Reference,
         }
     }
 }
@@ -545,34 +544,30 @@ fn read_scalar(
         interp_ok(self.read_immediate(val, expected)?.to_scalar())
     }
 
-    fn deref_pointer(
+    /// Given a place and a pointer loaded from that place, ensure that the place does
+    /// not store any more provenance than the pointer does. IOW, if any provenance
+    /// was discarded when loading the pointer, it will also get discarded in-memory.
+    fn reset_pointer_provenance(
         &mut self,
-        val: &PlaceTy<'tcx, M::Provenance>,
-        expected: ExpectedKind,
-    ) -> InterpResult<'tcx, MPlaceTy<'tcx, M::Provenance>> {
-        // Not using `ecx.deref_pointer` since we want to use our `read_immediate` wrapper.
-        let imm = self.read_immediate(val, expected)?;
-        // Reset provenance: ensure slice tail metadata does not preserve provenance,
-        // and ensure all pointers do not preserve partial provenance.
-        if self.reset_provenance_and_padding {
-            if matches!(imm.layout.backend_repr, BackendRepr::Scalar(..)) {
-                // A thin pointer. If it has provenance, we don't have to do anything.
-                // If it does not, ensure we clear the provenance in memory.
-                if matches!(imm.to_scalar(), Scalar::Int(..)) {
-                    self.ecx.clear_provenance(val)?;
-                }
-            } else {
-                // A wide pointer. This means we have to worry both about the pointer itself and the
-                // metadata. We do the lazy thing and just write back the value we got. Just
-                // clearing provenance in a targeted manner would be more efficient, but unless this
-                // is a perf hotspot it's just not worth the effort.
-                self.ecx.write_immediate_no_validate(*imm, val)?;
+        place: &PlaceTy<'tcx, M::Provenance>,
+        ptr: &ImmTy<'tcx, M::Provenance>,
+    ) -> InterpResult<'tcx> {
+        if matches!(ptr.layout.backend_repr, BackendRepr::Scalar(..)) {
+            // A thin pointer. If it has provenance, we don't have to do anything.
+            // If it does not, ensure we clear the provenance in memory.
+            if !matches!(ptr.to_scalar(), Scalar::Ptr(..)) {
+                // The loaded pointer has no provenance. Some bytes of its representation still
+                // might have provenance, which we have to clear.
+                self.ecx.clear_provenance(place)?;
             }
-            // The entire thing is data, not padding.
-            self.add_data_range_place(val);
+        } else {
+            // A wide pointer. This means we have to worry both about the pointer itself and the
+            // metadata. We do the lazy thing and just write back the value we got. Just
+            // clearing provenance in a targeted manner would be more efficient, but unless this
+            // is a perf hotspot it's just not worth the effort.
+            self.ecx.write_immediate_no_validate(**ptr, place)?;
         }
-        // Now turn it into a place.
-        self.ecx.imm_ptr_to_mplace(&imm)
+        interp_ok(())
     }
 
     fn check_wide_ptr_meta(
@@ -610,12 +605,22 @@ fn check_wide_ptr_meta(
     }
 
     /// Check a reference or `Box`.
+    ///
+    /// `ty` is the actual type of `value`; for a Box, `value` will be just the inner raw pointer.
     fn check_safe_pointer(
         &mut self,
         value: &PlaceTy<'tcx, M::Provenance>,
-        ptr_kind: PointerKind,
+        ty: Ty<'tcx>,
+        ptr_kind: PtrKind,
     ) -> InterpResult<'tcx> {
-        let place = self.deref_pointer(value, ptr_kind.into())?;
+        let ptr = self.read_immediate(value, ptr_kind.into())?;
+        if self.reset_provenance_and_padding {
+            // There's no padding in a pointer.
+            self.add_data_range_place(value);
+            // Resetting provenance is done below, together with retagging, to avoid
+            // redundant writes.
+        }
+        let place = self.ecx.imm_ptr_to_mplace(&ptr)?;
         // Handle wide pointers.
         // Check metadata early, for better diagnostics
         if place.layout.is_unsized() {
@@ -640,8 +645,9 @@ fn check_safe_pointer(
             // alignment should take attributes into account).
             .unwrap_or_else(|| (place.layout.size, place.layout.align.abi));
 
-        // If we're not allow to dangle, make sure this is dereferenceable.
-        if !self.may_dangle {
+        // If we're not allow to dangle, make sure this is dereferenceable and retag it for
+        // the aliasing model.
+        let adjusted_ptr = if !self.may_dangle {
             try_validation!(
                 self.ecx.check_ptr_access(
                     place.ptr(),
@@ -661,7 +667,44 @@ fn check_safe_pointer(
                 Ub(PointerUseAfterFree(..)) =>
                     format!("encountered a dangling {ptr_kind} (use-after-free)"),
             );
+            if self.reset_provenance_and_padding {
+                M::retag_ptr_value(self.ecx, &ptr, ty).map_err_kind(|e| match e {
+                    Ub(WriteToReadOnly(_)) => {
+                        err_validation_failure!(
+                            self.path,
+                            format!(
+                                "encountered {} pointing to read-only memory",
+                                if ptr_kind == PtrKind::Box { "box" } else { "mutable reference" },
+                            )
+                        )
+                    }
+                    InterpErrorKind::MachineStop(mut machine_err) => {
+                        // Enhance the aliasing model error with the current path.
+                        if !self.path.projs.is_empty() {
+                            let mut path = String::new();
+                            write_path(&mut path, &self.path.projs);
+                            machine_err.with_validation_path(path);
+                        }
+                        InterpErrorKind::MachineStop(machine_err)
+                    }
+                    e => e,
+                })?
+            } else {
+                // We can't retag if we're not resetting provenance.
+                None
+            }
+        } else {
+            // Pointer remains unchanged.
+            None
+        };
+        // If the pointer needs adjusting, write back adjusted pointer. This automatically
+        // also clears any excess provenance. Otherwise, just clear the provenance.
+        if let Some(ptr) = adjusted_ptr {
+            self.ecx.write_immediate_no_validate(*ptr, value)?;
+        } else if self.reset_provenance_and_padding {
+            self.reset_pointer_provenance(value, &ptr)?;
         }
+
         // Check alignment after dereferenceable (if both are violated, trigger the error above).
         try_validation!(
             self.ecx.check_ptr_align(
@@ -779,8 +822,8 @@ fn check_safe_pointer(
                     if size != Size::ZERO {
                         // Determine whether this pointer expects to be pointing to something mutable.
                         let ptr_expected_mutbl = match ptr_kind {
-                            PointerKind::Box => Mutability::Mut,
-                            PointerKind::Ref(mutbl) => {
+                            PtrKind::Box => Mutability::Mut,
+                            PtrKind::Ref(mutbl) => {
                                 // We do not take into account interior mutability here since we cannot know if
                                 // there really is an `UnsafeCell` inside `Option<UnsafeCell>` -- so we check
                                 // that in the recursive descent behind this reference (controlled by
@@ -883,14 +926,21 @@ fn try_visit_primitive(
                 interp_ok(true)
             }
             ty::RawPtr(..) => {
-                let place = self.deref_pointer(value, ExpectedKind::RawPtr)?;
+                let ptr = self.read_immediate(value, ExpectedKind::RawPtr)?;
+                if self.reset_provenance_and_padding {
+                    self.reset_pointer_provenance(value, &ptr)?;
+                    // There's no padding in a pointer.
+                    self.add_data_range_place(value);
+                }
+
+                let place = self.ecx.imm_ptr_to_mplace(&ptr)?;
                 if place.layout.is_unsized() {
                     self.check_wide_ptr_meta(place.meta(), place.layout)?;
                 }
                 interp_ok(true)
             }
             ty::Ref(_, _ty, mutbl) => {
-                self.check_safe_pointer(value, PointerKind::Ref(*mutbl))?;
+                self.check_safe_pointer(value, ty, PtrKind::Ref(*mutbl))?;
                 interp_ok(true)
             }
             ty::FnPtr(..) => {
@@ -1296,10 +1346,10 @@ fn visit_union(
     #[inline]
     fn visit_box(
         &mut self,
-        _box_ty: Ty<'tcx>,
+        box_ty: Ty<'tcx>,
         val: &PlaceTy<'tcx, M::Provenance>,
     ) -> InterpResult<'tcx> {
-        self.check_safe_pointer(val, PointerKind::Box)?;
+        self.check_safe_pointer(&val, box_ty, PtrKind::Box)?;
         interp_ok(())
     }
 
@@ -1567,9 +1617,12 @@ fn validate_operand_internal(
         .map_err_info(|err| {
             if !matches!(
                 err.kind(),
-                err_ub!(ValidationError { .. })
+                InterpErrorKind::UndefinedBehavior(ValidationError { .. })
                     | InterpErrorKind::InvalidProgram(_)
-                    | InterpErrorKind::Unsupported(UnsupportedOpInfo::ExternTypeField)
+                    | InterpErrorKind::Unsupported(_)
+                // We have to also ignore machine-specific errors since we do retagging
+                // during validation.
+                | InterpErrorKind::MachineStop(_)
             ) {
                 bug!("Unexpected error during validation: {}", format_interp_error(err));
             }

compiler/rustc_const_eval/src/interpret/visitor.rs

@@ -122,16 +122,17 @@ fn walk_value(&mut self, v: &Self::V) -> InterpResult<'tcx> {
                     phantom.layout().ty,
                 );
 
-                // ... that contains a `NonNull`... (gladly, only a single field here)
+                // ... that contains a `NonNull` whose only field finally is a raw ptr we can
+                // dereference.
                 assert_eq!(nonnull_ptr.layout().fields.count(), 1);
-                let pat_ty = self.ecx().project_field(&nonnull_ptr, FieldIdx::ZERO)?; // `*mut T is !null`
-                let base = match *pat_ty.layout().ty.kind() {
+                let pat_ptr = self.ecx().project_field(&nonnull_ptr, FieldIdx::ZERO)?; // `*mut T is !null`
+                let base = match *pat_ptr.layout().ty.kind() {
                     ty::Pat(base, _) => self.ecx().layout_of(base)?,
                     _ => unreachable!(),
                 };
-                let raw_ptr = pat_ty.transmute(base, self.ecx())?; // The actual raw pointer
+                let raw_ptr = pat_ptr.transmute(base, self.ecx())?; // The actual raw pointer
 
-                // ... whose only field finally is a raw ptr we can dereference.
+                // Hand this actual pointer to the visitor.
                 self.visit_box(ty, &raw_ptr)?;
 
                 // The second `Box` field is the allocator, which we recursively check for validity

compiler/rustc_interface/src/tests.rs

@@ -823,7 +823,6 @@ macro_rules! tracked {
     tracked!(merge_functions, Some(MergeFunctions::Disabled));
     tracked!(min_function_alignment, Some(Align::EIGHT));
     tracked!(min_recursion_limit, Some(256));
-    tracked!(mir_emit_retag, true);
     tracked!(mir_enable_passes, vec![("DestProp".to_string(), false)]);
     tracked!(mir_opt_level, Some(4));
     tracked!(mir_preserve_ub, true);

compiler/rustc_middle/src/mir/interpret/error.rs

@@ -772,7 +772,10 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
 }
 
 /// A trait for machine-specific errors (or other "machine stop" conditions).
-pub trait MachineStopType: Any + fmt::Display + fmt::Debug + Send {}
+pub trait MachineStopType: Any + fmt::Display + fmt::Debug + Send {
+    /// This error occurred during validation, inside a value at the given path.
+    fn with_validation_path(&mut self, _path: String) {}
+}
 
 impl dyn MachineStopType {
     #[inline(always)]

compiler/rustc_middle/src/mir/mod.rs

@@ -649,7 +649,7 @@ fn try_const_mono_switchint<'a>(
         }
 
         match rvalue {
-            Rvalue::Use(Operand::Constant(constant)) => {
+            Rvalue::Use(Operand::Constant(constant), _) => {
                 let bits = eval_mono_const(constant)?;
                 Some((bits, targets))
             }

compiler/rustc_middle/src/mir/pretty.rs

@@ -810,17 +810,6 @@ fn fmt(&self, fmt: &mut Formatter<'_>) -> fmt::Result {
             FakeRead(box (ref cause, ref place)) => {
                 write!(fmt, "FakeRead({cause:?}, {place:?})")
             }
-            Retag(ref kind, ref place) => write!(
-                fmt,
-                "Retag({}{:?})",
-                match kind {
-                    RetagKind::FnEntry => "[fn entry] ",
-                    RetagKind::TwoPhase => "[2phase] ",
-                    RetagKind::Raw => "[raw] ",
-                    RetagKind::Default => "",
-                },
-                place,
-            ),
             StorageLive(ref place) => write!(fmt, "StorageLive({place:?})"),
             StorageDead(ref place) => write!(fmt, "StorageDead({place:?})"),
             SetDiscriminant { ref place, variant_index } => {
@@ -1095,7 +1084,10 @@ fn fmt(&self, fmt: &mut Formatter<'_>) -> fmt::Result {
         use self::Rvalue::*;
 
         match *self {
-            Use(ref place) => write!(fmt, "{place:?}"),
+            Use(ref operand, with_retag) => {
+                // With retag is more common so we only print when it's without.
+                write!(fmt, "{}{operand:?}", if with_retag.no() { "no_retag " } else { "" })
+            }
             Repeat(ref a, b) => {
                 write!(fmt, "[{a:?}; ")?;
                 pretty_print_const(b, fmt, false)?;

compiler/rustc_middle/src/mir/statement.rs

@@ -52,7 +52,6 @@ pub const fn name(&self) -> &'static str {
             StatementKind::SetDiscriminant { .. } => "SetDiscriminant",
             StatementKind::StorageLive(..) => "StorageLive",
             StatementKind::StorageDead(..) => "StorageDead",
-            StatementKind::Retag(..) => "Retag",
             StatementKind::PlaceMention(..) => "PlaceMention",
             StatementKind::AscribeUserType(..) => "AscribeUserType",
             StatementKind::Coverage(..) => "Coverage",
@@ -763,7 +762,7 @@ pub fn is_safe_to_remove(&self) -> bool {
             // <https://www.ralfj.de/blog/2022/04/11/provenance-exposed.html>
             Rvalue::Cast(CastKind::PointerExposeProvenance, _, _) => false,
 
-            Rvalue::Use(_)
+            Rvalue::Use(_, _)
             | Rvalue::CopyForDeref(_)
             | Rvalue::Repeat(_, _)
             | Rvalue::Ref(_, _, _)
@@ -796,7 +795,7 @@ pub fn ty<D>(&self, local_decls: &D, tcx: TyCtxt<'tcx>) -> Ty<'tcx>
         D: ?Sized + HasLocalDecls<'tcx>,
     {
         match *self {
-            Rvalue::Use(ref operand) => operand.ty(local_decls, tcx),
+            Rvalue::Use(ref operand, _) => operand.ty(local_decls, tcx),
             Rvalue::Repeat(ref operand, count) => {
                 Ty::new_array_with_const_len(tcx, operand.ty(local_decls, tcx), count)
             }

compiler/rustc_middle/src/mir/syntax.rs

@@ -80,10 +80,6 @@ pub enum MirPhase {
     ///   StateTransform pass will expand those async drops or reset to sync.
     /// - Unwinding: in analysis MIR, unwinding from a function which may not unwind aborts. In
     ///   runtime MIR, this is UB.
-    /// - Retags: If `-Zmir-emit-retag` is enabled, analysis MIR has "implicit" retags in the same
-    ///   way that Rust itself has them. Where exactly these are is generally subject to change,
-    ///   and so we don't document this here. Runtime MIR has most retags explicit (though implicit
-    ///   retags can still occur at `Rvalue::{Ref,AddrOf}`).
     /// - Coroutine bodies: In analysis MIR, locals may actually be behind a pointer that user code
     ///   has access to. This occurs in coroutine bodies. Such locals do not behave like other
     ///   locals, because they e.g. may be aliased in surprising ways. Runtime MIR has no such
@@ -134,7 +130,6 @@ pub enum RuntimePhase {
     /// * [`LocalInfo::DerefTemp`](super::LocalInfo::DerefTemp)
     ///
     /// And the following variants are allowed:
-    /// * [`StatementKind::Retag`]
     /// * [`StatementKind::SetDiscriminant`]
     /// * [`PlaceElem::ConstantIndex`] / [`PlaceElem::Subslice`] after [`PlaceElem::Subslice`]
     ///
@@ -313,7 +308,10 @@ pub enum StatementKind<'tcx> {
     /// all). The *exact* way this works is undecided. It probably does something like evaluating
     /// the LHS to a place and the RHS to a value, and then storing the value to the place. Various
     /// parts of this may do type specific things that are more complicated than simply copying
-    /// bytes.
+    /// bytes. In particular, the assignment will typically erase the contents of padding,
+    /// erase provenance from non-pointer types, and implicitly "retag" all references and boxes
+    /// that it copies, meaning that the resulting value is not an exact duplicate for all intents
+    /// and purposes of the original value.
     ///
     /// **Needs clarification**: The implication of the above idea would be that assignment implies
     /// that the resulting value is initialized. I believe we could commit to this separately from
@@ -386,19 +384,6 @@ pub enum StatementKind<'tcx> {
     /// See `StorageLive` above.
     StorageDead(Local),
 
-    /// Retag references in the given place, ensuring they got fresh tags.
-    ///
-    /// This is part of the Stacked Borrows model. These statements are currently only interpreted
-    /// by miri and only generated when `-Z mir-emit-retag` is passed. See
-    /// <https://internals.rust-lang.org/t/stacked-borrows-an-aliasing-model-for-rust/8153/> for
-    /// more details.
-    ///
-    /// For code that is not specific to stacked borrows, you should consider retags to read and
-    /// modify the place in an opaque way.
-    ///
-    /// Only `RetagKind::Default` and `RetagKind::FnEntry` are permitted.
-    Retag(RetagKind, Box<Place<'tcx>>),
-
     /// This statement exists to preserve a trace of a scrutinee matched against a wildcard binding.
     /// This is especially useful for `let _ = PLACE;` bindings that desugar to a single
     /// `PlaceMention(PLACE)`.
@@ -506,18 +491,23 @@ pub enum NonDivergingIntrinsic<'tcx> {
     CopyNonOverlapping(CopyNonOverlapping<'tcx>),
 }
 
-/// Describes what kind of retag is to be performed.
+/// Describes whether this operand use performs a retag.
 #[derive(Copy, Clone, TyEncodable, TyDecodable, Debug, PartialEq, Eq, Hash, StableHash)]
 #[rustc_pass_by_value]
-pub enum RetagKind {
-    /// The initial retag of arguments when entering a function.
-    FnEntry,
-    /// Retag preparing for a two-phase borrow.
-    TwoPhase,
-    /// Retagging raw pointers.
-    Raw,
-    /// A "normal" retag.
-    Default,
+pub enum WithRetag {
+    Yes,
+    No,
+}
+
+impl WithRetag {
+    #[inline]
+    pub fn yes(self) -> bool {
+        matches!(self, Self::Yes)
+    }
+    #[inline]
+    pub fn no(self) -> bool {
+        matches!(self, Self::No)
+    }
 }
 
 /// The `FakeReadCause` describes the type of pattern why a FakeRead statement exists.
@@ -1366,8 +1356,8 @@ pub struct ConstOperand<'tcx> {
 /// value that an [`Operand`] produces.
 #[derive(Clone, TyEncodable, TyDecodable, Hash, StableHash, PartialEq, TypeFoldable, TypeVisitable)]
 pub enum Rvalue<'tcx> {
-    /// Yields the operand unchanged
-    Use(Operand<'tcx>),
+    /// Yields the operand unchanged, except for a potential retag.
+    Use(Operand<'tcx>, WithRetag),
 
     /// Creates an array where each element is the value of the operand.
     ///

compiler/rustc_middle/src/mir/visit.rs

@@ -170,15 +170,6 @@ fn visit_coverage(
                 self.super_coverage(kind, location);
             }
 
-            fn visit_retag(
-                &mut self,
-                kind: $(& $mutability)? RetagKind,
-                place: & $($mutability)? Place<'tcx>,
-                location: Location,
-            ) {
-                self.super_retag(kind, place, location);
-            }
-
             fn visit_place(
                 &mut self,
                 place: & $($mutability)? Place<'tcx>,
@@ -459,9 +450,6 @@ fn super_statement(
                             location
                         );
                     }
-                    StatementKind::Retag(kind, place) => {
-                        self.visit_retag($(& $mutability)? *kind, place, location);
-                    }
                     StatementKind::PlaceMention(place) => {
                         self.visit_place(
                             place,
@@ -704,7 +692,7 @@ fn super_rvalue(
                 location: Location
             ) {
                 match rvalue {
-                    Rvalue::Use(operand) => {
+                    Rvalue::Use(operand, _with_retag) => {
                         self.visit_operand(operand, location);
                     }
 
@@ -868,19 +856,6 @@ fn super_coverage(
             ) {
             }
 
-            fn super_retag(
-                &mut self,
-                _kind: $(& $mutability)? RetagKind,
-                place: & $($mutability)? Place<'tcx>,
-                location: Location
-            ) {
-                self.visit_place(
-                    place,
-                    PlaceContext::MutatingUse(MutatingUseContext::Retag),
-                    location,
-                );
-            }
-
             fn super_local_decl(
                 &mut self,
                 local: Local,

compiler/rustc_middle/src/ty/structural_impls.rs

@@ -220,11 +220,11 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
     crate::mir::MirPhase,
     crate::mir::Promoted,
     crate::mir::RawPtrKind,
-    crate::mir::RetagKind,
     crate::mir::SourceInfo,
     crate::mir::SourceScope,
     crate::mir::SourceScopeLocalData,
     crate::mir::SwitchTargets,
+    crate::mir::WithRetag,
     crate::traits::IsConstable,
     crate::traits::OverflowError,
     crate::ty::AdtKind,

compiler/rustc_mir_build/src/builder/cfg.rs

@@ -57,7 +57,7 @@ pub(crate) fn push_assign_constant(
             block,
             source_info,
             temp,
-            Rvalue::Use(Operand::Constant(Box::new(constant))),
+            Rvalue::Use(Operand::Constant(Box::new(constant)), WithRetag::Yes),
         );
     }
 
@@ -72,11 +72,14 @@ pub(crate) fn push_assign_unit(
             block,
             source_info,
             place,
-            Rvalue::Use(Operand::Constant(Box::new(ConstOperand {
-                span: source_info.span,
-                user_ty: None,
-                const_: Const::zero_sized(tcx.types.unit),
-            }))),
+            Rvalue::Use(
+                Operand::Constant(Box::new(ConstOperand {
+                    span: source_info.span,
+                    user_ty: None,
+                    const_: Const::zero_sized(tcx.types.unit),
+                })),
+                WithRetag::Yes,
+            ),
         );
     }
 

compiler/rustc_mir_build/src/builder/custom/parse/instruction.rs

@@ -23,9 +23,6 @@ pub(crate) fn parse_statement(&self, expr_id: ExprId) -> PResult<StatementKind<'
                 let op = self.parse_operand(args[0])?;
                 Ok(StatementKind::Intrinsic(Box::new(NonDivergingIntrinsic::Assume(op))))
             },
-            @call(mir_retag, args) => {
-                Ok(StatementKind::Retag(RetagKind::Default, Box::new(self.parse_place(args[0])?)))
-            },
             @call(mir_set_discriminant, args) => {
                 let place = self.parse_place(args[0])?;
                 let var = self.parse_integer_literal(args[1])? as u32;
@@ -287,7 +284,7 @@ fn parse_rvalue(&self, expr_id: ExprId) -> PResult<Rvalue<'tcx>> {
                     fields.iter().map(|f| self.parse_operand(f.expr)).collect::<Result<_, _>>()?
                 ))
             },
-            _ => self.parse_operand(expr_id).map(Rvalue::Use),
+            _ => self.parse_operand(expr_id).map(|op| Rvalue::Use(op, WithRetag::Yes)),
         )
     }
 

compiler/rustc_mir_build/src/builder/expr/as_rvalue.rs

@@ -346,11 +346,14 @@ pub(crate) fn as_rvalue(
             }
             ExprKind::Assign { .. } | ExprKind::AssignOp { .. } => {
                 block = this.stmt_expr(block, expr_id, None).into_block();
-                block.and(Rvalue::Use(Operand::Constant(Box::new(ConstOperand {
-                    span: expr_span,
-                    user_ty: None,
-                    const_: Const::zero_sized(this.tcx.types.unit),
-                }))))
+                block.and(Rvalue::Use(
+                    Operand::Constant(Box::new(ConstOperand {
+                        span: expr_span,
+                        user_ty: None,
+                        const_: Const::zero_sized(this.tcx.types.unit),
+                    })),
+                    WithRetag::Yes,
+                ))
             }
 
             ExprKind::Literal { .. }
@@ -361,7 +364,7 @@ pub(crate) fn as_rvalue(
             | ExprKind::ConstBlock { .. }
             | ExprKind::StaticRef { .. } => {
                 let constant = this.as_constant(expr);
-                block.and(Rvalue::Use(Operand::Constant(Box::new(constant))))
+                block.and(Rvalue::Use(Operand::Constant(Box::new(constant)), WithRetag::Yes))
             }
 
             ExprKind::WrapUnsafeBinder { source } => {
@@ -421,7 +424,7 @@ pub(crate) fn as_rvalue(
                         NeedsTemporary::No,
                     )
                 );
-                block.and(Rvalue::Use(operand))
+                block.and(Rvalue::Use(operand, WithRetag::Yes))
             }
 
             ExprKind::ByUse { expr, span: _ } => {
@@ -429,7 +432,7 @@ pub(crate) fn as_rvalue(
                     block =
                         this.as_operand(block, scope, expr, LocalInfo::Boring, NeedsTemporary::No)
                 );
-                block.and(Rvalue::Use(operand))
+                block.and(Rvalue::Use(operand, WithRetag::Yes))
             }
         }
     }
@@ -468,7 +471,7 @@ pub(crate) fn build_binary_op(
                 let err = AssertKind::Overflow(op, lhs, rhs);
                 block = self.assert(block, Operand::Move(of), false, err, span);
 
-                Rvalue::Use(Operand::Move(val))
+                Rvalue::Use(Operand::Move(val), WithRetag::Yes)
             }
             BinOp::Shl | BinOp::Shr if self.check_overflow && ty.is_integral() => {
                 // For an unsigned RHS, the shift is in-range for `rhs < bits`.

compiler/rustc_mir_build/src/builder/expr/into.rs

@@ -376,6 +376,7 @@ fn is_supported_loop_match_type(ty: Ty<'_>) -> bool {
                                             state_place,
                                             Rvalue::Use(
                                                 this.consume_by_copy_or_move(state_result_place),
+                                                WithRetag::Yes,
                                             ),
                                         );
                                         block.unit()
@@ -465,7 +466,7 @@ fn is_supported_loop_match_type(ty: Ty<'_>) -> bool {
                             source_info,
                             destination,
                             // Move from `b` so that does not get dropped any more.
-                            Rvalue::Use(Operand::Move(b)),
+                            Rvalue::Use(Operand::Move(b), WithRetag::Yes),
                         );
                         block.unit()
                     }
@@ -518,7 +519,7 @@ fn is_supported_loop_match_type(ty: Ty<'_>) -> bool {
                         block,
                         source_info,
                         destination,
-                        Rvalue::Use(Operand::Copy(place)),
+                        Rvalue::Use(Operand::Copy(place), WithRetag::Yes),
                     );
                     block.unit()
                 } else if this.infcx.type_is_use_cloned_modulo_regions(this.param_env, ty) {
@@ -555,7 +556,7 @@ fn is_supported_loop_match_type(ty: Ty<'_>) -> bool {
                         block,
                         source_info,
                         destination,
-                        Rvalue::Use(Operand::Move(place)),
+                        Rvalue::Use(Operand::Move(place), WithRetag::Yes),
                     );
                     block.unit()
                 }
@@ -837,7 +838,7 @@ fn is_supported_loop_match_type(ty: Ty<'_>) -> bool {
                 debug_assert!(Category::of(&expr.kind) == Some(Category::Place));
 
                 let place = unpack!(block = this.as_place(block, expr_id));
-                let rvalue = Rvalue::Use(this.consume_by_copy_or_move(place));
+                let rvalue = Rvalue::Use(this.consume_by_copy_or_move(place), WithRetag::Yes);
                 this.cfg.push_assign(block, source_info, destination, rvalue);
                 block.unit()
             }
@@ -852,7 +853,7 @@ fn is_supported_loop_match_type(ty: Ty<'_>) -> bool {
                 }
 
                 let place = unpack!(block = this.as_place(block, expr_id));
-                let rvalue = Rvalue::Use(this.consume_by_copy_or_move(place));
+                let rvalue = Rvalue::Use(this.consume_by_copy_or_move(place), WithRetag::Yes);
                 this.cfg.push_assign(block, source_info, destination, rvalue);
                 block.unit()
             }

compiler/rustc_mir_build/src/builder/matches/mod.rs

@@ -2737,7 +2737,9 @@ fn bind_matched_candidate_for_arm_body<'b>(
                 self.schedule_drop_for_binding(binding.var_id, binding.span, OutsideGuard);
             }
             let rvalue = match binding.binding_mode.0 {
-                ByRef::No => Rvalue::Use(self.consume_by_copy_or_move(binding.source)),
+                ByRef::No => {
+                    Rvalue::Use(self.consume_by_copy_or_move(binding.source), WithRetag::Yes)
+                }
                 ByRef::Yes(pinnedness, mutbl) => {
                     let rvalue =
                         Rvalue::Ref(re_erased, util::ref_pat_borrow_kind(mutbl), binding.source);

compiler/rustc_mir_build/src/builder/matches/test.rs

@@ -278,7 +278,12 @@ pub(super) fn perform_test(
 
                 // actual = len(place)
                 let length_op = self.len_of_slice_or_array(block, place, test.span, source_info);
-                self.cfg.push_assign(block, source_info, actual, Rvalue::Use(length_op));
+                self.cfg.push_assign(
+                    block,
+                    source_info,
+                    actual,
+                    Rvalue::Use(length_op, WithRetag::Yes),
+                );
 
                 // expected = <N>
                 let expected = self.push_usize(block, source_info, len);

compiler/rustc_mir_build/src/builder/mod.rs

@@ -885,7 +885,7 @@ fn find_unreachable_code_from(
             for stmt in &bb.statements {
                 match &stmt.kind {
                     // Ignore the implicit `()` return place assignment for unit functions/blocks
-                    StatementKind::Assign(box (_, Rvalue::Use(Operand::Constant(const_))))
+                    StatementKind::Assign(box (_, Rvalue::Use(Operand::Constant(const_), _)))
                         if const_.ty().is_unit() =>
                     {
                         continue;

compiler/rustc_mir_build/src/builder/scope.rs

@@ -957,7 +957,7 @@ pub(crate) fn break_const_continuable_scope(
                 (state_ty.discriminant_ty(self.tcx), Rvalue::Discriminant(scope.state_place))
             }
             ty::Uint(_) | ty::Int(_) | ty::Float(_) | ty::Bool | ty::Char => {
-                (state_ty, Rvalue::Use(Operand::Copy(scope.state_place)))
+                (state_ty, Rvalue::Use(Operand::Copy(scope.state_place), WithRetag::Yes))
             }
             _ => span_bug!(state_decl.source_info.span, "unsupported #[loop_match] state"),
         };

compiler/rustc_mir_dataflow/src/impls/liveness.rs

@@ -243,7 +243,6 @@ pub fn can_be_removed_if_dead<'tcx>(
             StatementKind::FakeRead(_)
             | StatementKind::StorageLive(_)
             | StatementKind::StorageDead(_)
-            | StatementKind::Retag(..)
             | StatementKind::AscribeUserType(..)
             | StatementKind::PlaceMention(..)
             | StatementKind::Coverage(..)

compiler/rustc_mir_dataflow/src/impls/storage_liveness.rs

@@ -169,7 +169,6 @@ fn apply_early_statement_effect(
             | StatementKind::FakeRead(..)
             | StatementKind::ConstEvalCounter
             | StatementKind::Nop
-            | StatementKind::Retag(..)
             | StatementKind::Intrinsic(..)
             | StatementKind::BackwardIncompatibleDropHint { .. }
             | StatementKind::StorageLive(..) => {}

compiler/rustc_mir_dataflow/src/move_paths/builder.rs

@@ -410,8 +410,7 @@ fn gather_statement(&mut self, stmt: &Statement<'tcx>) {
                     "SetDiscriminant/Deinit should not exist during borrowck"
                 );
             }
-            StatementKind::Retag { .. }
-            | StatementKind::AscribeUserType(..)
+            StatementKind::AscribeUserType(..)
             | StatementKind::PlaceMention(..)
             | StatementKind::Coverage(..)
             | StatementKind::Intrinsic(..)
@@ -424,7 +423,7 @@ fn gather_statement(&mut self, stmt: &Statement<'tcx>) {
     fn gather_rvalue(&mut self, rvalue: &Rvalue<'tcx>) {
         match *rvalue {
             Rvalue::ThreadLocalRef(_) => {} // not-a-move
-            Rvalue::Use(ref operand)
+            Rvalue::Use(ref operand, _)
             | Rvalue::Repeat(ref operand, _)
             | Rvalue::Cast(_, ref operand, _)
             | Rvalue::UnaryOp(_, ref operand)

compiler/rustc_mir_dataflow/src/rustc_peek.rs

@@ -97,7 +97,7 @@ fn sanity_check_via_rustc_peek<'tcx, A>(tcx: TyCtxt<'tcx>, mut cursor: ResultsCu
             (PeekCallKind::ByRef, mir::Rvalue::Ref(_, _, place))
             | (
                 PeekCallKind::ByVal,
-                mir::Rvalue::Use(mir::Operand::Move(place) | mir::Operand::Copy(place)),
+                mir::Rvalue::Use(mir::Operand::Move(place) | mir::Operand::Copy(place), _),
             ) => {
                 let loc = Location { block: bb, statement_index };
                 cursor.seek_before_primary_effect(loc);

compiler/rustc_mir_dataflow/src/value_analysis.rs

@@ -441,7 +441,7 @@ fn propagate_assignments(&mut self, tcx: TyCtxt<'tcx>, body: &Body<'tcx>) {
             for stmt in bbdata.statements.iter() {
                 let Some((lhs, rhs)) = stmt.kind.as_assign() else { continue };
                 match rhs {
-                    Rvalue::Use(Operand::Move(rhs) | Operand::Copy(rhs))
+                    Rvalue::Use(Operand::Move(rhs) | Operand::Copy(rhs), _)
                     | Rvalue::CopyForDeref(rhs) => {
                         let Some(lhs) = self.register_place_and_discr(tcx, body, *lhs) else {
                             continue;

compiler/rustc_mir_transform/src/add_moves_for_packed_drops.rs

@@ -100,7 +100,7 @@ fn add_move_for_packed_drop<'tcx>(
     ));
 
     patch.add_statement(loc, StatementKind::StorageLive(temp));
-    patch.add_assign(loc, Place::from(temp), Rvalue::Use(Operand::Move(*place)));
+    patch.add_assign(loc, Place::from(temp), Rvalue::Use(Operand::Move(*place), WithRetag::Yes));
     patch.patch_terminator(
         loc.block,
         TerminatorKind::Drop {

compiler/rustc_mir_transform/src/add_retag.rs

@@ -1,187 +0,0 @@
-//! This pass adds validation calls (AcquireValid, ReleaseValid) where appropriate.
-//! It has to be run really early, before transformations like inlining, because
-//! introducing these calls *adds* UB -- so, conceptually, this pass is actually part
-//! of MIR building, and only after this pass we think of the program has having the
-//! normal MIR semantics.
-
-use rustc_middle::mir::*;
-use rustc_middle::ty::{self, Ty, TyCtxt};
-
-pub(super) struct AddRetag;
-
-/// Determine whether this type may contain a reference (or box), and thus needs retagging.
-/// We will only recurse `depth` times into Tuples/ADTs to bound the cost of this.
-fn may_contain_reference<'tcx>(ty: Ty<'tcx>, depth: u32, tcx: TyCtxt<'tcx>) -> bool {
-    match ty.kind() {
-        // Primitive types that are not references
-        ty::Bool
-        | ty::Char
-        | ty::Float(_)
-        | ty::Int(_)
-        | ty::Uint(_)
-        | ty::RawPtr(..)
-        | ty::FnPtr(..)
-        | ty::Str
-        | ty::FnDef(..)
-        | ty::Never => false,
-        ty::Pat(base, ..) => may_contain_reference(*base, depth, tcx),
-        // References and Boxes (`noalias` sources)
-        ty::Ref(..) => true,
-        ty::Adt(..) if ty.is_box() => true,
-        // Compound types: recurse
-        ty::Array(ty, _) | ty::Slice(ty) => {
-            // This does not branch so we keep the depth the same.
-            may_contain_reference(*ty, depth, tcx)
-        }
-        ty::Tuple(tys) => {
-            depth == 0 || tys.iter().any(|ty| may_contain_reference(ty, depth - 1, tcx))
-        }
-        ty::Adt(adt, args) => {
-            depth == 0
-                || adt.variants().iter().any(|v| {
-                    v.fields.iter().any(|f| may_contain_reference(f.ty(tcx, args), depth - 1, tcx))
-                })
-        }
-        // Conservative fallback
-        _ => true,
-    }
-}
-
-impl<'tcx> crate::MirPass<'tcx> for AddRetag {
-    fn is_enabled(&self, sess: &rustc_session::Session) -> bool {
-        sess.opts.unstable_opts.mir_emit_retag
-    }
-
-    fn run_pass(&self, tcx: TyCtxt<'tcx>, body: &mut Body<'tcx>) {
-        // We need an `AllCallEdges` pass before we can do any work.
-        super::add_call_guards::AllCallEdges.run_pass(tcx, body);
-
-        let basic_blocks = body.basic_blocks.as_mut();
-        let local_decls = &body.local_decls;
-        let needs_retag = |place: &Place<'tcx>| {
-            // We're not really interested in stores to "outside" locations, they are hard to keep
-            // track of anyway.
-            !place.is_indirect_first_projection()
-                && may_contain_reference(place.ty(&*local_decls, tcx).ty, /*depth*/ 3, tcx)
-                && !local_decls[place.local].is_deref_temp()
-        };
-
-        // PART 1
-        // Retag arguments at the beginning of the start block.
-        {
-            // Gather all arguments, skip return value.
-            let places = local_decls.iter_enumerated().skip(1).take(body.arg_count).filter_map(
-                |(local, decl)| {
-                    let place = Place::from(local);
-                    needs_retag(&place).then_some((place, decl.source_info))
-                },
-            );
-
-            // Emit their retags.
-            basic_blocks[START_BLOCK].statements.splice(
-                0..0,
-                places.map(|(place, source_info)| {
-                    Statement::new(
-                        source_info,
-                        StatementKind::Retag(RetagKind::FnEntry, Box::new(place)),
-                    )
-                }),
-            );
-        }
-
-        // PART 2
-        // Retag return values of functions.
-        // We collect the return destinations because we cannot mutate while iterating.
-        let returns = basic_blocks
-            .iter_mut()
-            .filter_map(|block_data| {
-                match block_data.terminator().kind {
-                    TerminatorKind::Call { target: Some(target), destination, .. }
-                        if needs_retag(&destination) =>
-                    {
-                        // Remember the return destination for later
-                        Some((block_data.terminator().source_info, destination, target))
-                    }
-
-                    // `Drop` is also a call, but it doesn't return anything so we are good.
-                    TerminatorKind::Drop { .. } => None,
-                    // Not a block ending in a Call -> ignore.
-                    _ => None,
-                }
-            })
-            .collect::<Vec<_>>();
-        // Now we go over the returns we collected to retag the return values.
-        for (source_info, dest_place, dest_block) in returns {
-            basic_blocks[dest_block].statements.insert(
-                0,
-                Statement::new(
-                    source_info,
-                    StatementKind::Retag(RetagKind::Default, Box::new(dest_place)),
-                ),
-            );
-        }
-
-        // PART 3
-        // Add retag after assignments.
-        for block_data in basic_blocks {
-            // We want to insert statements as we iterate. To this end, we
-            // iterate backwards using indices.
-            for i in (0..block_data.statements.len()).rev() {
-                let (retag_kind, place) = match block_data.statements[i].kind {
-                    // Retag after assignments of reference type.
-                    StatementKind::Assign(box (ref place, ref rvalue)) => {
-                        let add_retag = match rvalue {
-                            // Ptr-creating operations already do their own internal retagging, no
-                            // need to also add a retag statement. *Except* if we are deref'ing a
-                            // Box, because those get desugared to directly working with the inner
-                            // raw pointer! That's relevant for `RawPtr` as Miri otherwise makes it
-                            // a NOP when the original pointer is already raw.
-                            Rvalue::RawPtr(_mutbl, place) => {
-                                // Using `is_box_global` here is a bit sketchy: if this code is
-                                // generic over the allocator, we'll not add a retag! This is a hack
-                                // to make Stacked Borrows compatible with custom allocator code.
-                                // It means the raw pointer inherits the tag of the box, which mostly works
-                                // but can sometimes lead to unexpected aliasing errors.
-                                // Long-term, we'll want to move to an aliasing model where "cast to
-                                // raw pointer" is a complete NOP, and then this will no longer be
-                                // an issue.
-                                if place.is_indirect_first_projection()
-                                    && body.local_decls[place.local].ty.is_box_global(tcx)
-                                {
-                                    Some(RetagKind::Raw)
-                                } else {
-                                    None
-                                }
-                            }
-                            Rvalue::Ref(..) => None,
-                            _ => {
-                                if needs_retag(place) {
-                                    Some(RetagKind::Default)
-                                } else {
-                                    None
-                                }
-                            }
-                        };
-                        if let Some(kind) = add_retag {
-                            (kind, *place)
-                        } else {
-                            continue;
-                        }
-                    }
-                    // Do nothing for the rest
-                    _ => continue,
-                };
-                // Insert a retag after the statement.
-                let source_info = block_data.statements[i].source_info;
-                block_data.statements.insert(
-                    i + 1,
-                    Statement::new(source_info, StatementKind::Retag(retag_kind, Box::new(place))),
-                );
-            }
-        }
-    }
-
-    fn is_required(&self) -> bool {
-        true
-    }
-}

compiler/rustc_mir_transform/src/check_enums.rs

@@ -430,11 +430,14 @@ fn insert_uninhabited_enum_check<'tcx>(
         source_info,
         StatementKind::Assign(Box::new((
             is_ok,
-            Rvalue::Use(Operand::Constant(Box::new(ConstOperand {
-                span: source_info.span,
-                user_ty: None,
-                const_: Const::Val(ConstValue::from_bool(false), tcx.types.bool),
-            }))),
+            Rvalue::Use(
+                Operand::Constant(Box::new(ConstOperand {
+                    span: source_info.span,
+                    user_ty: None,
+                    const_: Const::Val(ConstValue::from_bool(false), tcx.types.bool),
+                })),
+                WithRetag::Yes, // it's a bool, retag doesn't matter
+            ),
         ))),
     ));
 

compiler/rustc_mir_transform/src/copy_prop.rs

@@ -152,7 +152,7 @@ fn visit_statement(&mut self, stmt: &mut Statement<'tcx>, loc: Location) {
 
         // Do not leave tautological assignments around.
         if let StatementKind::Assign(box (lhs, ref rhs)) = stmt.kind
-            && let Rvalue::Use(Operand::Copy(rhs) | Operand::Move(rhs)) = *rhs
+            && let Rvalue::Use(Operand::Copy(rhs) | Operand::Move(rhs), _) = *rhs
             && lhs == rhs
         {
             stmt.make_nop(true);

compiler/rustc_mir_transform/src/coroutine.rs

@@ -237,17 +237,20 @@ fn insert_none_ret_block(&self, body: &mut Body<'tcx>) -> BasicBlock {
                 let ty::Adt(_poll_adt, args) = *self.old_yield_ty.kind() else { bug!() };
                 let ty::Adt(_option_adt, args) = *args.type_at(0).kind() else { bug!() };
                 let yield_ty = args.type_at(0);
-                Rvalue::Use(Operand::Constant(Box::new(ConstOperand {
-                    span: source_info.span,
-                    const_: Const::Unevaluated(
-                        UnevaluatedConst::new(
-                            self.tcx.require_lang_item(LangItem::AsyncGenFinished, body.span),
-                            self.tcx.mk_args(&[yield_ty.into()]),
+                Rvalue::Use(
+                    Operand::Constant(Box::new(ConstOperand {
+                        span: source_info.span,
+                        const_: Const::Unevaluated(
+                            UnevaluatedConst::new(
+                                self.tcx.require_lang_item(LangItem::AsyncGenFinished, body.span),
+                                self.tcx.mk_args(&[yield_ty.into()]),
+                            ),
+                            self.old_yield_ty,
                         ),
-                        self.old_yield_ty,
-                    ),
-                    user_ty: None,
-                })))
+                        user_ty: None,
+                    })),
+                    WithRetag::Yes,
+                )
             }
         };
 
@@ -306,22 +309,25 @@ fn make_state(
                     let ty::Adt(_poll_adt, args) = *self.old_yield_ty.kind() else { bug!() };
                     let ty::Adt(_option_adt, args) = *args.type_at(0).kind() else { bug!() };
                     let yield_ty = args.type_at(0);
-                    Rvalue::Use(Operand::Constant(Box::new(ConstOperand {
-                        span: source_info.span,
-                        const_: Const::Unevaluated(
-                            UnevaluatedConst::new(
-                                self.tcx.require_lang_item(
-                                    LangItem::AsyncGenFinished,
-                                    source_info.span,
+                    Rvalue::Use(
+                        Operand::Constant(Box::new(ConstOperand {
+                            span: source_info.span,
+                            const_: Const::Unevaluated(
+                                UnevaluatedConst::new(
+                                    self.tcx.require_lang_item(
+                                        LangItem::AsyncGenFinished,
+                                        source_info.span,
+                                    ),
+                                    self.tcx.mk_args(&[yield_ty.into()]),
                                 ),
-                                self.tcx.mk_args(&[yield_ty.into()]),
+                                self.old_yield_ty,
                             ),
-                            self.old_yield_ty,
-                        ),
-                        user_ty: None,
-                    })))
+                            user_ty: None,
+                        })),
+                        WithRetag::Yes,
+                    )
                 } else {
-                    Rvalue::Use(val)
+                    Rvalue::Use(val, WithRetag::Yes)
                 }
             }
             CoroutineKind::Coroutine(_) => {
@@ -545,19 +551,13 @@ fn make_coroutine_state_argument_pinned<'tcx>(tcx: TyCtxt<'tcx>, body: &mut Body
     let pin_field = tcx.mk_place_field(SELF_ARG.into(), FieldIdx::ZERO, ref_coroutine_ty);
 
     let statements = &mut body.basic_blocks.as_mut_preserves_cfg()[START_BLOCK].statements;
-    // Miri requires retags to be the very first thing in the body.
-    // We insert this assignment just after.
-    let insert_point = statements
-        .iter()
-        .position(|stmt| !matches!(stmt.kind, StatementKind::Retag(..)))
-        .unwrap_or(statements.len());
     statements.insert(
-        insert_point,
+        0,
         Statement::new(
             source_info,
             StatementKind::Assign(Box::new((
                 unpinned_local.into(),
-                Rvalue::Use(Operand::Copy(pin_field)),
+                Rvalue::Use(Operand::Copy(pin_field), WithRetag::Yes),
             ))),
         ),
     );
@@ -626,7 +626,7 @@ fn eliminate_get_context_call<'tcx>(bb_data: &mut BasicBlockData<'tcx>) -> Local
     let [arg] = *Box::try_from(args).unwrap();
     let local = arg.node.place().unwrap().local;
 
-    let arg = Rvalue::Use(arg.node);
+    let arg = Rvalue::Use(arg.node, WithRetag::Yes);
     let assign =
         Statement::new(terminator.source_info, StatementKind::Assign(Box::new((destination, arg))));
     bb_data.statements.push(assign);
@@ -1369,7 +1369,7 @@ fn create_cases<'tcx>(
                         source_info,
                         StatementKind::Assign(Box::new((
                             point.resume_arg,
-                            Rvalue::Use(Operand::Move(CTX_ARG.into())),
+                            Rvalue::Use(Operand::Move(CTX_ARG.into()), WithRetag::Yes),
                         ))),
                     ));
                 }
@@ -1599,7 +1599,7 @@ fn run_pass(&self, tcx: TyCtxt<'tcx>, body: &mut Body<'tcx>) {
             args_iter.filter_map(|local| {
                 let (ty, variant_index, idx) = transform.remap[local]?;
                 let lhs = transform.make_field(variant_index, idx, ty);
-                let rhs = Rvalue::Use(Operand::Move(local.into()));
+                let rhs = Rvalue::Use(Operand::Move(local.into()), WithRetag::Yes);
                 let assign = StatementKind::Assign(Box::new((lhs, rhs)));
                 Some(Statement::new(source_info, assign))
             }),
@@ -1755,7 +1755,6 @@ fn visit_statement(&mut self, statement: &Statement<'tcx>, location: Location) {
             | StatementKind::SetDiscriminant { .. }
             | StatementKind::StorageLive(_)
             | StatementKind::StorageDead(_)
-            | StatementKind::Retag(..)
             | StatementKind::AscribeUserType(..)
             | StatementKind::PlaceMention(..)
             | StatementKind::Coverage(..)

compiler/rustc_mir_transform/src/coroutine/drop.rs

@@ -328,7 +328,7 @@ pub(super) fn expand_async_drops<'tcx>(
         // First state-loop yield for mainline
         let context_ref_place =
             Place::from(body.local_decls.push(LocalDecl::new(context_mut_ref, source_info.span)));
-        let arg = Rvalue::Use(Operand::Move(Place::from(CTX_ARG)));
+        let arg = Rvalue::Use(Operand::Move(Place::from(CTX_ARG)), WithRetag::Yes);
         body[bb].statements.push(Statement::new(
             source_info,
             StatementKind::Assign(Box::new((context_ref_place, arg))),

compiler/rustc_mir_transform/src/coverage/from_mir.rs

@@ -91,7 +91,6 @@ fn filtered_statement_span(statement: &Statement<'_>) -> Option<Span> {
         )
         | StatementKind::Assign(_)
         | StatementKind::SetDiscriminant { .. }
-        | StatementKind::Retag(_, _)
         | StatementKind::PlaceMention(..)
         | StatementKind::AscribeUserType(_, _) => Some(statement.source_info.span),
 

compiler/rustc_mir_transform/src/dataflow_const_prop.rs

@@ -182,9 +182,6 @@ fn handle_statement(&self, statement: &Statement<'tcx>, state: &mut State<FlatSe
                     FlatSet::<Scalar>::BOTTOM,
                 );
             }
-            StatementKind::Retag(..) => {
-                // We don't track references.
-            }
             StatementKind::ConstEvalCounter
             | StatementKind::Nop
             | StatementKind::FakeRead(..)
@@ -304,7 +301,7 @@ fn handle_assign(
         state: &mut State<FlatSet<Scalar>>,
     ) {
         match rvalue {
-            Rvalue::Use(operand) => {
+            Rvalue::Use(operand, _) => {
                 state.flood(target.as_ref(), &self.map);
                 if let Some(target) = self.map.find(target.as_ref()) {
                     self.assign_operand(state, target, operand);
@@ -470,7 +467,7 @@ fn handle_rvalue(
                 }
             }
             Rvalue::Discriminant(place) => state.get_discr(place.as_ref(), &self.map),
-            Rvalue::Use(operand) => return self.handle_operand(operand, state),
+            Rvalue::Use(operand, _) => return self.handle_operand(operand, state),
             Rvalue::CopyForDeref(_) => bug!("`CopyForDeref` in runtime MIR"),
             Rvalue::Ref(..) | Rvalue::RawPtr(..) => {
                 // We don't track such places.
@@ -981,7 +978,7 @@ fn visit_after_primary_statement_effect(
         location: Location,
     ) {
         match statement.kind {
-            StatementKind::Assign(box (_, Rvalue::Use(Operand::Constant(_)))) => {
+            StatementKind::Assign(box (_, Rvalue::Use(Operand::Constant(_), _))) => {
                 // Don't overwrite the assignment if it already uses a constant (to keep the span).
             }
             StatementKind::Assign(box (place, _)) => {
@@ -1024,7 +1021,11 @@ fn visit_statement(&mut self, statement: &mut Statement<'tcx>, location: Locatio
         if let Some(value) = self.assignments.get(&location) {
             match &mut statement.kind {
                 StatementKind::Assign(box (_, rvalue)) => {
-                    *rvalue = Rvalue::Use(self.make_operand(*value));
+                    let old_retag = match rvalue {
+                        Rvalue::Use(_, retag) => *retag,
+                        _ => WithRetag::Yes,
+                    };
+                    *rvalue = Rvalue::Use(self.make_operand(*value), old_retag);
                 }
                 _ => bug!("found assignment info for non-assign statement"),
             }

compiler/rustc_mir_transform/src/deref_separator.rs

@@ -58,7 +58,11 @@ fn visit_place(&mut self, place: &mut Place<'tcx>, cntxt: PlaceContext, loc: Loc
                         if self.add_deref_metadata {
                             Rvalue::CopyForDeref(deref_place)
                         } else {
-                            Rvalue::Use(Operand::Copy(deref_place))
+                            // FIXME: Unfortunately, `add_deref_metadata` is not documented. So who
+                            // knows what is supposed to happen here -- retag or not? `CopyForDeref`
+                            // later turns into a no-retag assignment so probably maybe that's also
+                            // what we need here.
+                            Rvalue::Use(Operand::Copy(deref_place), WithRetag::No)
                         },
                     );
                     place_local = temp;

compiler/rustc_mir_transform/src/dest_prop.rs

@@ -284,7 +284,7 @@ fn visit_statement(&mut self, statement: &mut Statement<'tcx>, location: Locatio
         match &statement.kind {
             StatementKind::Assign(box (dest, rvalue)) => {
                 match rvalue {
-                    Rvalue::Use(Operand::Copy(place) | Operand::Move(place)) => {
+                    Rvalue::Use(Operand::Copy(place) | Operand::Move(place), _) => {
                         // These might've been turned into self-assignments by the replacement
                         // (this includes the original statement we wanted to eliminate).
                         if dest == place {
@@ -398,7 +398,7 @@ impl<'tcx> Visitor<'tcx> for FindAssignments<'_, 'tcx> {
     fn visit_statement(&mut self, statement: &Statement<'tcx>, _: Location) {
         if let StatementKind::Assign(box (
             lhs,
-            Rvalue::Use(Operand::Copy(rhs) | Operand::Move(rhs)),
+            Rvalue::Use(Operand::Copy(rhs) | Operand::Move(rhs), _),
         )) = &statement.kind
             && let Some(src) = lhs.as_local()
             && let Some(dest) = rhs.as_local()
@@ -574,7 +574,7 @@ fn save_as_intervals<'tcx>(
                 StatementKind::Assign(box (
                     lhs,
                     Rvalue::CopyForDeref(rhs)
-                    | Rvalue::Use(Operand::Copy(rhs) | Operand::Move(rhs)),
+                    | Rvalue::Use(Operand::Copy(rhs) | Operand::Move(rhs), _),
                 )) => lhs.projection == rhs.projection,
                 _ => false,
             };

compiler/rustc_mir_transform/src/elaborate_drop.rs

@@ -359,11 +359,10 @@ fn build_async_drop(
             let obj_ref_place = Place::from(self.new_temp(unwrap_ty));
             call_statements.push(self.assign(
                 obj_ref_place,
-                Rvalue::Use(Operand::Copy(tcx.mk_place_field(
-                    pin_obj_place,
-                    FieldIdx::ZERO,
-                    unwrap_ty,
-                ))),
+                Rvalue::Use(
+                    Operand::Copy(tcx.mk_place_field(pin_obj_place, FieldIdx::ZERO, unwrap_ty)),
+                    WithRetag::Yes,
+                ),
             ));
 
             let obj_ptr_place = Place::from(self.new_temp(obj_ptr_ty));
@@ -1290,7 +1289,7 @@ fn drop_loop_trio_for_slice(&mut self, ety: Ty<'tcx>) -> BasicBlock {
                         Operand::Copy(Place::from(self.place.local)),
                     ),
                 ),
-                self.assign(cur.into(), Rvalue::Use(zero)),
+                self.assign(cur.into(), Rvalue::Use(zero, WithRetag::Yes)),
             ],
             Some(Terminator {
                 source_info: self.source_info,

compiler/rustc_mir_transform/src/elaborate_drops.rs

@@ -403,11 +403,14 @@ fn elaborate_drops(&mut self) {
     }
 
     fn constant_bool(&self, span: Span, val: bool) -> Rvalue<'tcx> {
-        Rvalue::Use(Operand::Constant(Box::new(ConstOperand {
-            span,
-            user_ty: None,
-            const_: Const::from_bool(self.tcx, val),
-        })))
+        Rvalue::Use(
+            Operand::Constant(Box::new(ConstOperand {
+                span,
+                user_ty: None,
+                const_: Const::from_bool(self.tcx, val),
+            })),
+            WithRetag::Yes,
+        )
     }
 
     fn set_drop_flag(&mut self, loc: Location, path: MovePathIndex, val: DropFlagState) {

compiler/rustc_mir_transform/src/erase_deref_temps.rs

@@ -16,7 +16,9 @@ fn tcx(&self) -> TyCtxt<'tcx> {
 
     fn visit_rvalue(&mut self, rvalue: &mut Rvalue<'tcx>, _: Location) {
         if let &mut Rvalue::CopyForDeref(place) = rvalue {
-            *rvalue = Rvalue::Use(Operand::Copy(place))
+            // We do *NOT* want a retag here! This assignment might copy a mutable reference we
+            // can't actually copy, we just need it temporarily to create another pointer.
+            *rvalue = Rvalue::Use(Operand::Copy(place), WithRetag::No)
         }
     }
 

compiler/rustc_mir_transform/src/gvn.rs

@@ -1061,7 +1061,7 @@ fn simplify_rvalue(
     ) -> Option<VnIndex> {
         let value = match *rvalue {
             // Forward values.
-            Rvalue::Use(ref mut operand) => return self.simplify_operand(operand, location),
+            Rvalue::Use(ref mut operand, _) => return self.simplify_operand(operand, location),
 
             // Roots.
             Rvalue::Repeat(ref mut op, amount) => {
@@ -1263,7 +1263,8 @@ fn simplify_aggregate(
         if let Some(value) = self.simplify_aggregate_to_copy(ty, variant_index, &fields) {
             if let Some(place) = self.try_as_place(value, location, true) {
                 self.reused_locals.insert(place.local);
-                *rvalue = Rvalue::Use(Operand::Copy(place));
+                // FIXME: Is it correct to make these retagging assignments?
+                *rvalue = Rvalue::Use(Operand::Copy(place), WithRetag::Yes);
             }
             return Some(value);
         }
@@ -2022,13 +2023,13 @@ fn visit_assign(
 
         let value = self.simplify_rvalue(lhs, rvalue, location);
         if let Some(value) = value {
+            // FIXME: Is it correct to make these retagging assignments?
             if let Some(const_) = self.try_as_constant(value) {
-                *rvalue = Rvalue::Use(Operand::Constant(Box::new(const_)));
+                *rvalue = Rvalue::Use(Operand::Constant(Box::new(const_)), WithRetag::Yes);
             } else if let Some(place) = self.try_as_place(value, location, false)
-                && *rvalue != Rvalue::Use(Operand::Move(place))
-                && *rvalue != Rvalue::Use(Operand::Copy(place))
+                && !matches!(rvalue, Rvalue::Use(Operand::Move(p) | Operand::Copy(p), _) if p == &place)
             {
-                *rvalue = Rvalue::Use(Operand::Copy(place));
+                *rvalue = Rvalue::Use(Operand::Copy(place), WithRetag::Yes);
                 self.reused_locals.insert(place.local);
             }
         }

compiler/rustc_mir_transform/src/inline.rs

@@ -966,7 +966,7 @@ fn dest_needs_borrow(place: Place<'_>) -> bool {
                 callsite.source_info,
                 StatementKind::Assign(Box::new((
                     dest,
-                    Rvalue::Use(Operand::Move(destination_local.into())),
+                    Rvalue::Use(Operand::Move(destination_local.into()), WithRetag::Yes),
                 ))),
             ));
             n += 1;
@@ -1138,7 +1138,7 @@ fn create_temp_if_necessary<'tcx, I: Inliner<'tcx>>(
     let local = new_call_temp(caller_body, callsite, arg_ty, return_block);
     caller_body[callsite.block].statements.push(Statement::new(
         callsite.source_info,
-        StatementKind::Assign(Box::new((Place::from(local), Rvalue::Use(arg)))),
+        StatementKind::Assign(Box::new((Place::from(local), Rvalue::Use(arg, WithRetag::Yes)))),
     ));
     local
 }
@@ -1275,16 +1275,6 @@ fn visit_basic_block_data(&mut self, block: BasicBlock, data: &mut BasicBlockDat
         self.in_cleanup_block = false;
     }
 
-    fn visit_retag(&mut self, kind: &mut RetagKind, place: &mut Place<'tcx>, loc: Location) {
-        self.super_retag(kind, place, loc);
-
-        // We have to patch all inlined retags to be aware that they are no longer
-        // happening on function entry.
-        if *kind == RetagKind::FnEntry {
-            *kind = RetagKind::Default;
-        }
-    }
-
     fn visit_statement(&mut self, statement: &mut Statement<'tcx>, location: Location) {
         if let StatementKind::StorageLive(local) | StatementKind::StorageDead(local) =
             statement.kind

compiler/rustc_mir_transform/src/instsimplify.rs

@@ -107,16 +107,16 @@ fn simplify_bool_cmp(&self, rvalue: &mut Rvalue<'tcx>) {
         let Rvalue::BinaryOp(op @ (BinOp::Eq | BinOp::Ne), box (a, b)) = &*rvalue else { return };
         *rvalue = match (op, self.try_eval_bool(a), self.try_eval_bool(b)) {
             // Transform "Eq(a, true)" ==> "a"
-            (BinOp::Eq, _, Some(true)) => Rvalue::Use(a.clone()),
+            (BinOp::Eq, _, Some(true)) => Rvalue::Use(a.clone(), WithRetag::Yes),
 
             // Transform "Ne(a, false)" ==> "a"
-            (BinOp::Ne, _, Some(false)) => Rvalue::Use(a.clone()),
+            (BinOp::Ne, _, Some(false)) => Rvalue::Use(a.clone(), WithRetag::Yes),
 
             // Transform "Eq(true, b)" ==> "b"
-            (BinOp::Eq, Some(true), _) => Rvalue::Use(b.clone()),
+            (BinOp::Eq, Some(true), _) => Rvalue::Use(b.clone(), WithRetag::Yes),
 
             // Transform "Ne(false, b)" ==> "b"
-            (BinOp::Ne, Some(false), _) => Rvalue::Use(b.clone()),
+            (BinOp::Ne, Some(false), _) => Rvalue::Use(b.clone(), WithRetag::Yes),
 
             // Transform "Eq(false, b)" ==> "Not(b)"
             (BinOp::Eq, Some(false), _) => Rvalue::UnaryOp(UnOp::Not, b.clone()),
@@ -145,10 +145,24 @@ fn simplify_ref_deref(&self, rvalue: &mut Rvalue<'tcx>) {
             && let Some((base, ProjectionElem::Deref)) = place.as_ref().last_projection()
             && rvalue.ty(self.local_decls, self.tcx) == base.ty(self.local_decls, self.tcx).ty
         {
-            *rvalue = Rvalue::Use(Operand::Copy(Place {
-                local: base.local,
-                projection: self.tcx.mk_place_elems(base.projection),
-            }));
+            *rvalue = Rvalue::Use(
+                Operand::Copy(Place {
+                    local: base.local,
+                    projection: self.tcx.mk_place_elems(base.projection),
+                }),
+                // This might have been a two-phase borrow, which we should not upgrade
+                // to a full `&mut` reborrow.
+                // FIXME: Once Stacked Borrows is fully removed, we can use `Yes` here as
+                // Tree Borrows treats two-phase and full borrows the same.
+                if matches!(
+                    rvalue,
+                    Rvalue::Ref(_, BorrowKind::Mut { kind: MutBorrowKind::TwoPhaseBorrow }, _)
+                ) {
+                    WithRetag::No
+                } else {
+                    WithRetag::Yes
+                },
+            );
         }
     }
 
@@ -172,7 +186,7 @@ fn simplify_cast(&self, rvalue: &mut Rvalue<'tcx>) {
 
         let operand_ty = operand.ty(self.local_decls, self.tcx);
         if operand_ty == *cast_ty {
-            *rvalue = Rvalue::Use(operand.clone());
+            *rvalue = Rvalue::Use(operand.clone(), WithRetag::Yes);
         } else if *kind == CastKind::Transmute
             // Transmuting an integer to another integer is just a signedness cast
             && let (ty::Int(int), ty::Uint(uint)) | (ty::Uint(uint), ty::Int(int)) =
@@ -236,9 +250,10 @@ fn simplify_primitive_clone(
             terminator.source_info,
             StatementKind::Assign(Box::new((
                 *destination,
-                Rvalue::Use(Operand::Copy(
-                    arg_place.project_deeper(&[ProjectionElem::Deref], self.tcx),
-                )),
+                Rvalue::Use(
+                    Operand::Copy(arg_place.project_deeper(&[ProjectionElem::Deref], self.tcx)),
+                    WithRetag::Yes,
+                ),
             ))),
         ));
         terminator.kind = TerminatorKind::Goto { target: *destination_block };
@@ -294,7 +309,10 @@ fn simplify_size_or_align_of_val(
             );
             statements.push(Statement::new(
                 source_info,
-                StatementKind::Assign(Box::new((*destination, Rvalue::Use(const_op)))),
+                StatementKind::Assign(Box::new((
+                    *destination,
+                    Rvalue::Use(const_op, WithRetag::Yes),
+                ))),
             ));
             terminator.kind = TerminatorKind::Goto { target: *destination_block };
         }

compiler/rustc_mir_transform/src/jump_threading.rs

@@ -403,7 +403,6 @@ fn mutated_statement(
             StatementKind::StorageLive(local) | StatementKind::StorageDead(local) => {
                 Some((Place::from(local), None))
             }
-            StatementKind::Retag(..)
             | StatementKind::Intrinsic(box NonDivergingIntrinsic::Assume(..))
             // copy_nonoverlapping takes pointers and mutated the pointed-to value.
             | StatementKind::Intrinsic(box NonDivergingIntrinsic::CopyNonOverlapping(..))
@@ -512,7 +511,7 @@ fn process_assign(
     ) {
         let Some(lhs) = self.place(*lhs_place, None) else { return };
         match rvalue {
-            Rvalue::Use(operand) => self.process_operand(lhs, operand, state),
+            Rvalue::Use(operand, _) => self.process_operand(lhs, operand, state),
             // Transfer the conditions on the copy rhs.
             Rvalue::Discriminant(rhs) => {
                 let Some(rhs) = self.place(*rhs, Some(TrackElem::Discriminant)) else { return };

compiler/rustc_mir_transform/src/known_panics_lint.rs

@@ -549,7 +549,7 @@ fn eval_rvalue(&mut self, rvalue: &Rvalue<'tcx>, dest: &Place<'tcx>) -> Option<(
         let val: Value<'_> = match *rvalue {
             ThreadLocalRef(_) => return None,
 
-            Use(ref operand) | WrapUnsafeBinder(ref operand, _) => {
+            Use(ref operand, _) | WrapUnsafeBinder(ref operand, _) => {
                 self.eval_operand(operand)?.into()
             }
 

compiler/rustc_mir_transform/src/large_enums.rs

@@ -51,7 +51,7 @@ fn run_pass(&self, tcx: TyCtxt<'tcx>, body: &mut Body<'tcx>) {
             for (statement_index, st) in data.statements.iter_mut().enumerate() {
                 let StatementKind::Assign(box (
                     lhs,
-                    Rvalue::Use(Operand::Copy(rhs) | Operand::Move(rhs)),
+                    Rvalue::Use(Operand::Copy(rhs) | Operand::Move(rhs), _),
                 )) = &st.kind
                 else {
                     continue;
@@ -83,7 +83,7 @@ fn run_pass(&self, tcx: TyCtxt<'tcx>, body: &mut Body<'tcx>) {
                         tmp_ty,
                     ),
                 };
-                let rval = Rvalue::Use(Operand::Constant(Box::new(constant_vals)));
+                let rval = Rvalue::Use(Operand::Constant(Box::new(constant_vals)), WithRetag::No);
                 let const_assign = StatementKind::Assign(Box::new((place, rval)));
 
                 let discr_place =
@@ -100,10 +100,14 @@ fn run_pass(&self, tcx: TyCtxt<'tcx>, body: &mut Body<'tcx>) {
                 let size_place = Place::from(patch.new_temp(tcx.types.usize, span));
                 let store_size = StatementKind::Assign(Box::new((
                     size_place,
-                    Rvalue::Use(Operand::Copy(Place {
-                        local: size_array_local,
-                        projection: tcx.mk_place_elems(&[PlaceElem::Index(discr_cast_place.local)]),
-                    })),
+                    Rvalue::Use(
+                        Operand::Copy(Place {
+                            local: size_array_local,
+                            projection: tcx
+                                .mk_place_elems(&[PlaceElem::Index(discr_cast_place.local)]),
+                        }),
+                        WithRetag::No,
+                    ),
                 )));
 
                 let dst = Place::from(patch.new_temp(Ty::new_mut_ptr(tcx, ty), span));

compiler/rustc_mir_transform/src/lib.rs

@@ -22,7 +22,7 @@
 use rustc_middle::mir::{
     AnalysisPhase, Body, CallSource, ClearCrossCrate, ConstOperand, ConstQualifs, LocalDecl,
     MirPhase, Operand, Place, ProjectionElem, Promoted, RuntimePhase, Rvalue, START_BLOCK,
-    SourceInfo, Statement, StatementKind, TerminatorKind,
+    SourceInfo, Statement, StatementKind, TerminatorKind, WithRetag,
 };
 use rustc_middle::ty::{self, TyCtxt, TypeVisitableExt};
 use rustc_middle::util::Providers;
@@ -122,7 +122,6 @@ macro_rules! pass_names {
     mod abort_unwinding_calls : AbortUnwindingCalls;
     mod add_call_guards : AddCallGuards { AllCallEdges, CriticalCallEdges };
     mod add_moves_for_packed_drops : AddMovesForPackedDrops;
-    mod add_retag : AddRetag;
     mod add_subtyping_projections : Subtyper;
     mod check_inline : CheckForceInline;
     mod check_call_recursion : CheckCallRecursion, CheckDropRecursion;
@@ -275,7 +274,7 @@ fn remap_mir_for_const_eval_select<'tcx>(
                                 SourceInfo::outermost(fn_span),
                                 StatementKind::Assign(Box::new((
                                     local.into(),
-                                    Rvalue::Use(tupled_args.node.clone()),
+                                    Rvalue::Use(tupled_args.node.clone(), WithRetag::Yes),
                                 ))),
                             ));
                             (Operand::Move, local.into())
@@ -647,9 +646,6 @@ fn run_runtime_lowering_passes<'tcx>(tcx: TyCtxt<'tcx>, body: &mut Body<'tcx>) {
         // AddMovesForPackedDrops needs to run after drop
         // elaboration.
         &add_moves_for_packed_drops::AddMovesForPackedDrops,
-        // `AddRetag` needs to run after `ElaborateDrops` but before `ElaborateBoxDerefs`.
-        // Otherwise it should run fairly late, but before optimizations begin.
-        &add_retag::AddRetag,
         &erase_deref_temps::EraseDerefTemps,
         &elaborate_box_derefs::ElaborateBoxDerefs,
         &coroutine::StateTransform,

compiler/rustc_mir_transform/src/liveness.rs

@@ -391,7 +391,7 @@ fn find_self_assignments<'tcx>(
                     let Some(assign) = body.basic_blocks[*target].statements.first() else {
                         continue;
                     };
-                    let StatementKind::Assign(box (dest, Rvalue::Use(Operand::Move(temp)))) =
+                    let StatementKind::Assign(box (dest, Rvalue::Use(Operand::Move(temp), _))) =
                         assign.kind
                     else {
                         continue;
@@ -709,8 +709,7 @@ fn find_dead_assignments(
                     | StatementKind::SetDiscriminant { box place, .. } => {
                         check_place(*place, AccessKind::Assign, statement.source_info, live);
                     }
-                    StatementKind::Retag(_, _)
-                    | StatementKind::StorageLive(_)
+                    StatementKind::StorageLive(_)
                     | StatementKind::StorageDead(_)
                     | StatementKind::Coverage(_)
                     | StatementKind::Intrinsic(_)

compiler/rustc_mir_transform/src/lower_intrinsics.rs

@@ -35,7 +35,7 @@ fn run_pass(&self, tcx: TyCtxt<'tcx>, body: &mut Body<'tcx>) {
                             terminator.source_info,
                             StatementKind::Assign(Box::new((
                                 *destination,
-                                Rvalue::Use(Operand::RuntimeChecks(op)),
+                                Rvalue::Use(Operand::RuntimeChecks(op), WithRetag::Yes),
                             ))),
                         ));
                         terminator.kind = TerminatorKind::Goto { target };
@@ -46,11 +46,14 @@ fn run_pass(&self, tcx: TyCtxt<'tcx>, body: &mut Body<'tcx>) {
                             terminator.source_info,
                             StatementKind::Assign(Box::new((
                                 *destination,
-                                Rvalue::Use(Operand::Constant(Box::new(ConstOperand {
-                                    span: terminator.source_info.span,
-                                    user_ty: None,
-                                    const_: Const::zero_sized(tcx.types.unit),
-                                }))),
+                                Rvalue::Use(
+                                    Operand::Constant(Box::new(ConstOperand {
+                                        span: terminator.source_info.span,
+                                        user_ty: None,
+                                        const_: Const::zero_sized(tcx.types.unit),
+                                    })),
+                                    WithRetag::Yes,
+                                ),
                             ))),
                         ));
                         terminator.kind = TerminatorKind::Goto { target };
@@ -165,7 +168,7 @@ fn run_pass(&self, tcx: TyCtxt<'tcx>, body: &mut Body<'tcx>) {
                             terminator.source_info,
                             StatementKind::Assign(Box::new((
                                 *destination,
-                                Rvalue::Use(Operand::Copy(derefed_place)),
+                                Rvalue::Use(Operand::Copy(derefed_place), WithRetag::Yes),
                             ))),
                         ));
                         terminator.kind = match *target {

compiler/rustc_mir_transform/src/match_branches.rs

@@ -72,7 +72,9 @@ fn unify_if_equal_const(
         }) {
             Some(StatementKind::Assign(Box::new((
                 dest,
-                Rvalue::Use(Operand::Constant(Box::new(first_const.clone()))),
+                // We didn't remember the `WithRetag` of the original assignments, so in case
+                // one of them had "no", we also have to use "no" here.
+                Rvalue::Use(Operand::Constant(Box::new(first_const.clone())), WithRetag::No),
             ))))
         } else {
             None
@@ -202,7 +204,7 @@ fn unify_by_int_to_int(
         }) {
             let operand = Operand::Copy(Place::from(self.discr_local()));
             let rval = if first_const.ty() == self.discr_ty {
-                Rvalue::Use(operand)
+                Rvalue::Use(operand, WithRetag::No)
             } else {
                 Rvalue::Cast(CastKind::IntToInt, operand, first_const.ty())
             };
@@ -262,7 +264,7 @@ fn unify_by_copy(
         for &(case, rvalue) in rvals.iter() {
             match rvalue {
                 // Check if `_3 = const Foo::B` can be transformed to `_3 = copy *_1`.
-                Rvalue::Use(Operand::Constant(box constant))
+                Rvalue::Use(Operand::Constant(box constant), _)
                     if let Const::Val(const_, ty) = constant.const_ =>
                 {
                     let (ecx, op) = mk_eval_cx_for_const_val(
@@ -280,7 +282,7 @@ fn unify_by_copy(
                         return None;
                     }
                 }
-                Rvalue::Use(Operand::Copy(src_place)) if *src_place == copy_src_place => {}
+                Rvalue::Use(Operand::Copy(src_place), _) if *src_place == copy_src_place => {}
                 // Check if `_3 = Foo::B` can be transformed to `_3 = copy *_1`.
                 Rvalue::Aggregate(box AggregateKind::Adt(_, variant_index, _, _, None), fields)
                     if fields.is_empty()
@@ -290,7 +292,12 @@ fn unify_by_copy(
                 _ => return None,
             }
         }
-        Some(StatementKind::Assign(Box::new((dest, Rvalue::Use(Operand::Copy(copy_src_place))))))
+        // We didn't remember the `WithRetag` of the original assignments, so in case
+        // one of them had "no", we also have to use "no" here.
+        Some(StatementKind::Assign(Box::new((
+            dest,
+            Rvalue::Use(Operand::Copy(copy_src_place), WithRetag::No),
+        ))))
     }
 
     /// Returns a new statement if we can use the statement replace all statements.
@@ -427,7 +434,7 @@ fn simplify_match<'tcx>(
     let mut patch = simplify_match.patch;
     if let Some(discr_local) = simplify_match.discr_local {
         patch.add_statement(parent_end, StatementKind::StorageLive(discr_local));
-        patch.add_assign(parent_end, Place::from(discr_local), Rvalue::Use(discr));
+        patch.add_assign(parent_end, Place::from(discr_local), Rvalue::Use(discr, WithRetag::No));
     }
     for new_stmt in new_stmts {
         patch.add_statement(parent_end, new_stmt);
@@ -501,8 +508,9 @@ fn candidate_const<'tcx, 'a>(
     rvals: &'a [(u128, &'a Rvalue<'tcx>)],
     otherwise: Option<&'a Rvalue<'tcx>>,
 ) -> Option<(Vec<(u128, &'a ConstOperand<'tcx>)>, Option<&'a ConstOperand<'tcx>>)> {
+    // We ignore the retag mode here, which means the `Use` we insert later must be without retag.
     let otherwise = if let Some(otherwise) = otherwise {
-        let Rvalue::Use(Operand::Constant(box const_)) = otherwise else {
+        let Rvalue::Use(Operand::Constant(box const_), _) = otherwise else {
             return None;
         };
         Some(const_)
@@ -512,7 +520,7 @@ fn candidate_const<'tcx, 'a>(
     let consts = rvals
         .into_iter()
         .map(|&(case, rval)| {
-            let Rvalue::Use(Operand::Constant(box const_)) = rval else { return None };
+            let Rvalue::Use(Operand::Constant(box const_), _) = rval else { return None };
             Some((case, const_))
         })
         .try_collect()?;

compiler/rustc_mir_transform/src/promote_consts.rs

@@ -310,7 +310,7 @@ fn validate_place(&mut self, place: PlaceRef<'tcx>) -> Result<(), Unpromotable>
                 if let Some(local) = place_base.as_local()
                     && let TempState::Defined { location, .. } = self.temps[local]
                     && let Left(def_stmt) = self.body.stmt_at(location)
-                    && let Some((_, Rvalue::Use(Operand::Constant(c)))) = def_stmt.kind.as_assign()
+                    && let Some((_, Rvalue::Use(Operand::Constant(c), _))) = def_stmt.kind.as_assign()
                     && let Some(did) = c.check_static_ptr(self.tcx)
                     // Evaluating a promoted may not read statics except if it got
                     // promoted from a static (this is a CTFE check). So we
@@ -327,7 +327,7 @@ fn validate_place(&mut self, place: PlaceRef<'tcx>) -> Result<(), Unpromotable>
                 // Only accept if we can predict the index and are indexing an array.
                 if let TempState::Defined { location: loc, .. } = self.temps[local]
                     && let Left(statement) =  self.body.stmt_at(loc)
-                    && let Some((_, Rvalue::Use(Operand::Constant(c)))) = statement.kind.as_assign()
+                    && let Some((_, Rvalue::Use(Operand::Constant(c), _))) = statement.kind.as_assign()
                     && self.should_evaluate_for_promotion_checks(c.const_)
                     && let Some(idx) = c.const_.try_eval_target_usize(self.tcx, self.typing_env)
                     // Determine the type of the thing we are indexing.
@@ -424,7 +424,12 @@ fn validate_ref(&mut self, kind: BorrowKind, place: &Place<'tcx>) -> Result<(),
 
     fn validate_rvalue(&mut self, rvalue: &Rvalue<'tcx>) -> Result<(), Unpromotable> {
         match rvalue {
-            Rvalue::Use(operand)
+            Rvalue::Use(_operand, WithRetag::No) => {
+                // This shouldn't actually happen, but just to be safe: we'll later add the promoted
+                // with retagging, so don't promote anything that didn't already have retagging.
+                return Err(Unpromotable);
+            }
+            Rvalue::Use(operand, _)
             | Rvalue::Repeat(operand, _)
             | Rvalue::WrapUnsafeBinder(operand, _) => {
                 self.validate_operand(operand)?;
@@ -793,11 +798,14 @@ fn promote_temp(&mut self, temp: Local) -> Local {
                     if self.keep_original {
                         rhs.clone()
                     } else {
-                        let unit = Rvalue::Use(Operand::Constant(Box::new(ConstOperand {
-                            span: statement.source_info.span,
-                            user_ty: None,
-                            const_: Const::zero_sized(self.tcx.types.unit),
-                        })));
+                        let unit = Rvalue::Use(
+                            Operand::Constant(Box::new(ConstOperand {
+                                span: statement.source_info.span,
+                                user_ty: None,
+                                const_: Const::zero_sized(self.tcx.types.unit),
+                            })),
+                            WithRetag::Yes,
+                        );
                         mem::replace(rhs, unit)
                     },
                     statement.source_info,
@@ -918,7 +926,9 @@ fn promote_candidate(
                 statement.source_info,
                 StatementKind::Assign(Box::new((
                     Place::from(promoted_ref),
-                    Rvalue::Use(Operand::Constant(Box::new(promoted_operand))),
+                    // We can retag here because we wouldn't promote non-retagged values (they get
+                    // rejected in validate_rvalue).
+                    Rvalue::Use(Operand::Constant(Box::new(promoted_operand)), WithRetag::Yes),
                 ))),
             );
             self.extra_statements.push((loc, promoted_ref_statement));

compiler/rustc_mir_transform/src/ref_prop.rs

@@ -247,7 +247,7 @@ fn compute_replacement<'tcx>(
             // This is a copy, just use the value we have in store for the previous one.
             // As we are visiting in `assignment_order`, i.e. reverse postorder, `rhs` should
             // have been visited before.
-            Rvalue::Use(Operand::Copy(place) | Operand::Move(place)) => {
+            Rvalue::Use(Operand::Copy(place) | Operand::Move(place), _) => {
                 if let Some(rhs) = place.as_local()
                     && ssa.is_ssa(rhs)
                 {

compiler/rustc_mir_transform/src/remove_noop_landing_pads.rs

@@ -102,7 +102,7 @@ fn is_nop_landing_pad(
                     // These are all noops in a landing pad
                 }
 
-                StatementKind::Assign(box (place, Rvalue::Use(_) | Rvalue::Discriminant(_))) => {
+                StatementKind::Assign(box (place, Rvalue::Use(..) | Rvalue::Discriminant(_))) => {
                     if place.as_local().is_some() {
                         // Writing to a local (e.g., a drop flag) does not
                         // turn a landing pad to a non-nop
@@ -113,8 +113,7 @@ fn is_nop_landing_pad(
 
                 StatementKind::Assign { .. }
                 | StatementKind::SetDiscriminant { .. }
-                | StatementKind::Intrinsic(..)
-                | StatementKind::Retag { .. } => {
+                | StatementKind::Intrinsic(..) => {
                     return false;
                 }
             }

compiler/rustc_mir_transform/src/remove_zsts.rs

@@ -124,7 +124,6 @@ fn visit_statement(&mut self, statement: &mut Statement<'tcx>, loc: Location) {
             }
             StatementKind::SetDiscriminant { box place, variant_index: _ }
             | StatementKind::AscribeUserType(box (place, _), _)
-            | StatementKind::Retag(_, box place)
             | StatementKind::PlaceMention(box place)
             | StatementKind::FakeRead(box (_, place)) => Some(place),
             StatementKind::StorageLive(local) | StatementKind::StorageDead(local) => {

compiler/rustc_mir_transform/src/shim.rs

@@ -296,40 +296,6 @@ fn local_decls_for_sig<'tcx>(
         .collect()
 }
 
-fn dropee_emit_retag<'tcx>(
-    tcx: TyCtxt<'tcx>,
-    body: &mut Body<'tcx>,
-    mut dropee_ptr: Place<'tcx>,
-    span: Span,
-) -> Place<'tcx> {
-    if tcx.sess.opts.unstable_opts.mir_emit_retag {
-        let source_info = SourceInfo::outermost(span);
-        // We want to treat the function argument as if it was passed by `&mut`. As such, we
-        // generate
-        // ```
-        // temp = &mut *arg;
-        // Retag(temp, FnEntry)
-        // ```
-        // It's important that we do this first, before anything that depends on `dropee_ptr`
-        // has been put into the body.
-        let reborrow = Rvalue::Ref(
-            tcx.lifetimes.re_erased,
-            BorrowKind::Mut { kind: MutBorrowKind::Default },
-            tcx.mk_place_deref(dropee_ptr),
-        );
-        let ref_ty = reborrow.ty(body.local_decls(), tcx);
-        dropee_ptr = body.local_decls.push(LocalDecl::new(ref_ty, span)).into();
-        let new_statements = [
-            StatementKind::Assign(Box::new((dropee_ptr, reborrow))),
-            StatementKind::Retag(RetagKind::FnEntry, Box::new(dropee_ptr)),
-        ];
-        for s in new_statements {
-            body.basic_blocks_mut()[START_BLOCK].statements.push(Statement::new(source_info, s));
-        }
-    }
-    dropee_ptr
-}
-
 fn build_drop_shim<'tcx>(tcx: TyCtxt<'tcx>, def_id: DefId, ty: Option<Ty<'tcx>>) -> Body<'tcx> {
     debug!("build_drop_shim(def_id={:?}, ty={:?})", def_id, ty);
 
@@ -360,7 +326,6 @@ fn build_drop_shim<'tcx>(tcx: TyCtxt<'tcx>, def_id: DefId, ty: Option<Ty<'tcx>>)
 
     // The first argument (index 0), but local 1 (after the return place).
     let dropee_ptr = Place::from(Local::arg(0));
-    let dropee_ptr = dropee_emit_retag(tcx, &mut body, dropee_ptr, span);
 
     if ty.is_some() {
         let patch = {
@@ -627,7 +592,7 @@ fn copy_shim(&mut self) {
         let rcvr = self.tcx.mk_place_deref(Place::from(Local::arg(0)));
         let ret_statement = self.make_statement(StatementKind::Assign(Box::new((
             Place::return_place(),
-            Rvalue::Use(Operand::Copy(rcvr)),
+            Rvalue::Use(Operand::Copy(rcvr), WithRetag::Yes),
         ))));
         self.block(vec![ret_statement], TerminatorKind::Return, false);
     }

compiler/rustc_mir_transform/src/shim/async_destructor_ctor.rs

@@ -118,13 +118,12 @@ pub(super) fn build_async_drop_shim<'tcx>(
         return body;
     }
 
-    let mut dropee_ptr = Place::from(body.local_decls.push(LocalDecl::new(drop_ptr_ty, span)));
+    let dropee_ptr = Place::from(body.local_decls.push(LocalDecl::new(drop_ptr_ty, span)));
     let st_kind = StatementKind::Assign(Box::new((
         dropee_ptr,
-        Rvalue::Use(Operand::Move(coroutine_layout_dropee)),
+        Rvalue::Use(Operand::Move(coroutine_layout_dropee), WithRetag::Yes),
     )));
     body.basic_blocks_mut()[START_BLOCK].statements.push(Statement::new(source_info, st_kind));
-    dropee_ptr = dropee_emit_retag(tcx, &mut body, dropee_ptr, span);
 
     let dropline = body.basic_blocks.last_index();
 
@@ -235,7 +234,7 @@ fn build_adrop_for_coroutine_shim<'tcx>(
                 source_info,
                 StatementKind::Assign(Box::new((
                     Place::from(proxy_ref_local),
-                    Rvalue::Use(Operand::Copy(proxy_ref_place)),
+                    Rvalue::Use(Operand::Copy(proxy_ref_place), WithRetag::Yes),
                 ))),
             ),
         );
@@ -256,7 +255,7 @@ fn build_adrop_for_coroutine_shim<'tcx>(
                         source_info,
                         StatementKind::Assign(Box::new((
                             Place::from(cor_ptr_local),
-                            Rvalue::Use(Operand::Copy(impl_ptr_place)),
+                            Rvalue::Use(Operand::Copy(impl_ptr_place), WithRetag::Yes),
                         ))),
                     ),
                 );
@@ -323,7 +322,7 @@ fn build_adrop_for_adrop_shim<'tcx>(
         source_info,
         StatementKind::Assign(Box::new((
             Place::from(proxy_ref_local),
-            Rvalue::Use(Operand::Copy(proxy_ref_place)),
+            Rvalue::Use(Operand::Copy(proxy_ref_place), WithRetag::Yes),
         ))),
     ));
 
@@ -339,7 +338,7 @@ fn build_adrop_for_adrop_shim<'tcx>(
                 source_info,
                 StatementKind::Assign(Box::new((
                     Place::from(cor_ptr_local),
-                    Rvalue::Use(Operand::Copy(impl_ptr_place)),
+                    Rvalue::Use(Operand::Copy(impl_ptr_place), WithRetag::Yes),
                 ))),
             ));
         }

compiler/rustc_mir_transform/src/simplify.rs

@@ -568,7 +568,6 @@ impl<'tcx> Visitor<'tcx> for UsedLocals {
     fn visit_statement(&mut self, statement: &Statement<'tcx>, location: Location) {
         match statement.kind {
             StatementKind::Intrinsic(..)
-            | StatementKind::Retag(..)
             | StatementKind::Coverage(..)
             | StatementKind::FakeRead(..)
             | StatementKind::PlaceMention(..)

compiler/rustc_mir_transform/src/simplify_branches.rs

@@ -63,7 +63,7 @@ fn try_get_const<'tcx, 'a>(
                         continue 'blocks;
                     }
                 } else if let StatementKind::Assign(box (lhs, ref rvalue)) = stmt.kind
-                    && let Rvalue::Use(Operand::Constant(c)) = rvalue
+                    && let Rvalue::Use(Operand::Constant(c), _) = rvalue
                 {
                     pre_place_const = Some((lhs, c));
                 }

compiler/rustc_mir_transform/src/single_use_consts.rs

@@ -59,7 +59,7 @@ fn run_pass(&self, tcx: TyCtxt<'tcx>, body: &mut Body<'tcx>) {
             };
             let (place, rvalue) = *place_and_rvalue;
             assert_eq!(place.as_local(), Some(local));
-            let Rvalue::Use(operand) = rvalue else { bug!("No longer a use?") };
+            let Rvalue::Use(operand, _) = rvalue else { bug!("No longer a use?") };
 
             let mut replacer = LocalReplacer { tcx, local, operand: Some(operand) };
 
@@ -114,7 +114,7 @@ struct SingleUseConstsFinder {
 impl<'tcx> Visitor<'tcx> for SingleUseConstsFinder {
     fn visit_assign(&mut self, place: &Place<'tcx>, rvalue: &Rvalue<'tcx>, location: Location) {
         if let Some(local) = place.as_local()
-            && let Rvalue::Use(operand) = rvalue
+            && let Rvalue::Use(operand, _) = rvalue
             && let Operand::Constant(_) = operand
         {
             let locations = &mut self.locations[local];

compiler/rustc_mir_transform/src/sroa.rs

@@ -348,7 +348,7 @@ fn visit_statement(&mut self, statement: &mut Statement<'tcx>, location: Locatio
                             // Replace mentions of SROA'd locals that appear in the operand.
                             self.visit_operand(&mut operand, location);
 
-                            let rvalue = Rvalue::Use(operand);
+                            let rvalue = Rvalue::Use(operand, WithRetag::Yes);
                             self.patch.add_statement(
                                 location,
                                 StatementKind::Assign(Box::new((new_local.into(), rvalue))),
@@ -368,13 +368,13 @@ fn visit_statement(&mut self, statement: &mut Statement<'tcx>, location: Locatio
             // ...
             // ```
             // ConstProp will pick up the pieces and replace them by actual constants.
-            StatementKind::Assign(box (place, Rvalue::Use(Operand::Constant(_)))) => {
+            StatementKind::Assign(box (place, Rvalue::Use(Operand::Constant(_), retag))) => {
                 if let Some(final_locals) = self.replacements.place_fragments(place) {
                     // Put the deaggregated statements *after* the original one.
                     let location = location.successor_within_block();
                     for (field, ty, new_local) in final_locals {
                         let rplace = self.tcx.mk_place_field(place, field, ty);
-                        let rvalue = Rvalue::Use(Operand::Move(rplace));
+                        let rvalue = Rvalue::Use(Operand::Move(rplace), retag);
                         self.patch.add_statement(
                             location,
                             StatementKind::Assign(Box::new((new_local.into(), rvalue))),
@@ -394,7 +394,7 @@ fn visit_statement(&mut self, statement: &mut Statement<'tcx>, location: Locatio
             // ```
             StatementKind::Assign(box (
                 lhs,
-                Rvalue::Use(ref op @ (Operand::Copy(rplace) | Operand::Move(rplace))),
+                Rvalue::Use(ref op @ (Operand::Copy(rplace) | Operand::Move(rplace)), retag),
             )) => {
                 let copy = match *op {
                     Operand::Copy(_) => true,
@@ -411,9 +411,9 @@ fn visit_statement(&mut self, statement: &mut Statement<'tcx>, location: Locatio
                             .unwrap_or(rplace);
                         debug!(?rplace);
                         let rvalue = if copy {
-                            Rvalue::Use(Operand::Copy(rplace))
+                            Rvalue::Use(Operand::Copy(rplace), retag)
                         } else {
-                            Rvalue::Use(Operand::Move(rplace))
+                            Rvalue::Use(Operand::Move(rplace), retag)
                         };
                         self.patch.add_statement(
                             location,

compiler/rustc_mir_transform/src/ssa.rs

@@ -298,7 +298,7 @@ fn compute_copy_classes(ssa: &mut SsaLocals, body: &Body<'_>) {
     let mut copies = IndexVec::from_fn_n(|l| l, body.local_decls.len());
 
     for (local, rvalue, _) in ssa.assignments(body) {
-        let Rvalue::Use(Operand::Copy(place) | Operand::Move(place)) = rvalue else {
+        let Rvalue::Use(Operand::Copy(place) | Operand::Move(place), _) = rvalue else {
             continue;
         };
 

compiler/rustc_mir_transform/src/trivial_const.rs

@@ -86,7 +86,7 @@ pub(crate) fn trivial_const<'a, 'tcx: 'a, F, B>(
         return None;
     }
 
-    let Rvalue::Use(Operand::Constant(c)) = rvalue else {
+    let Rvalue::Use(Operand::Constant(c), _) = rvalue else {
         return None;
     };
     match c.const_ {

compiler/rustc_mir_transform/src/validate.rs

@@ -320,14 +320,6 @@ fn visit_statement(&mut self, statement: &Statement<'tcx>, location: Location) {
                     self.fail(location, "`SetDiscriminant`is not allowed until deaggregation");
                 }
             }
-            StatementKind::Retag(kind, _) => {
-                // FIXME(JakobDegen) The validator should check that `self.body.phase <
-                // DropsLowered`. However, this causes ICEs with generation of drop shims, which
-                // seem to fail to set their `MirPhase` correctly.
-                if matches!(kind, RetagKind::TwoPhase) {
-                    self.fail(location, format!("explicit `{kind:?}` is forbidden"));
-                }
-            }
             StatementKind::Coverage(kind) => {
                 if self.body.phase >= MirPhase::Analysis(AnalysisPhase::PostCleanup)
                     && let CoverageKind::BlockMarker { .. } | CoverageKind::SpanMarker { .. } = kind
@@ -1019,7 +1011,7 @@ macro_rules! check_kinds {
             };
         }
         match rvalue {
-            Rvalue::Use(_) => {}
+            Rvalue::Use(_, _) => {}
             Rvalue::CopyForDeref(_) => {
                 if self.body.phase >= MirPhase::Runtime(RuntimePhase::Initial) {
                     self.fail(location, "`CopyForDeref` should have been removed in runtime MIR");
@@ -1571,14 +1563,6 @@ fn visit_statement(&mut self, statement: &Statement<'tcx>, location: Location) {
                     );
                 }
             }
-            StatementKind::Retag(kind, _) => {
-                // FIXME(JakobDegen) The validator should check that `self.body.phase <
-                // DropsLowered`. However, this causes ICEs with generation of drop shims, which
-                // seem to fail to set their `MirPhase` correctly.
-                if matches!(kind, RetagKind::TwoPhase) {
-                    self.fail(location, format!("explicit `{kind:?}` is forbidden"));
-                }
-            }
             StatementKind::StorageLive(_)
             | StatementKind::StorageDead(_)
             | StatementKind::Coverage(_)

compiler/rustc_public/src/mir/body.rs

@@ -439,11 +439,9 @@ pub enum FakeReadCause {
 
 /// Describes what kind of retag is to be performed
 #[derive(Copy, Clone, Debug, Eq, PartialEq, Hash, Serialize)]
-pub enum RetagKind {
-    FnEntry,
-    TwoPhase,
-    Raw,
-    Default,
+pub enum WithRetag {
+    Yes,
+    No,
 }
 
 #[derive(Copy, Clone, Debug, Eq, PartialEq, Hash, Serialize)]
@@ -480,7 +478,6 @@ pub enum StatementKind {
     SetDiscriminant { place: Place, variant_index: VariantIdx },
     StorageLive(Local),
     StorageDead(Local),
-    Retag(RetagKind, Place),
     PlaceMention(Place),
     AscribeUserType { place: Place, projections: UserTypeProjection, variance: Variance },
     Coverage(Coverage),
@@ -587,14 +584,14 @@ pub enum Rvalue {
     /// return a value with the same type as their operand.
     UnaryOp(UnOp, Operand),
 
-    /// Yields the operand unchanged
-    Use(Operand),
+    /// Yields the operand unchanged, except for possibly a retag.
+    Use(Operand, WithRetag),
 }
 
 impl Rvalue {
     pub fn ty(&self, locals: &[LocalDecl]) -> Result<Ty, Error> {
         match self {
-            Rvalue::Use(operand) => operand.ty(locals),
+            Rvalue::Use(operand, _) => operand.ty(locals),
             Rvalue::Repeat(operand, count) => {
                 Ok(Ty::new_array_with_const_len(operand.ty(locals)?, count.clone()))
             }

compiler/rustc_public/src/mir/pretty.rs

@@ -108,7 +108,6 @@ fn pretty_statement<W: Write>(writer: &mut W, statement: &StatementKind) -> io::
         StatementKind::StorageDead(local) => {
             writeln!(writer, "{INDENT}StorageDead(_{local});")
         }
-        StatementKind::Retag(kind, place) => writeln!(writer, "Retag({kind:?}, {place:?});"),
         StatementKind::PlaceMention(place) => {
             writeln!(writer, "{INDENT}PlaceMention({place:?};")
         }
@@ -389,7 +388,12 @@ fn pretty_rvalue<W: Write>(writer: &mut W, rval: &Rvalue) -> io::Result<()> {
         Rvalue::UnaryOp(un, op) => {
             write!(writer, "{:?}({})", un, pretty_operand(op))
         }
-        Rvalue::Use(op) => write!(writer, "{}", pretty_operand(op)),
+        Rvalue::Use(op, retag) => write!(
+            writer,
+            "{}{}",
+            if matches!(retag, crate::mir::WithRetag::No) { "no_retag " } else { "" },
+            pretty_operand(op)
+        ),
     }
 }
 

compiler/rustc_public/src/mir/visit.rs

@@ -169,8 +169,7 @@ fn super_statement(&mut self, stmt: &$($mutability)? Statement, location: Locati
                     StatementKind::FakeRead(_, place) | StatementKind::PlaceMention(place) => {
                         self.visit_place(place, PlaceContext::NON_MUTATING, location);
                     }
-                    StatementKind::SetDiscriminant { place, .. }
-                    | StatementKind::Retag(_, place) => {
+                    StatementKind::SetDiscriminant { place, .. } => {
                         self.visit_place(place, PlaceContext::MUTATING, location);
                     }
                     StatementKind::StorageLive(local) | StatementKind::StorageDead(local) => {
@@ -278,7 +277,7 @@ fn super_rvalue(&mut self, rvalue: &$($mutability)? Rvalue, location: Location)
                         self.visit_ty_const(constant, location);
                     }
                     Rvalue::ThreadLocalRef(_) => {}
-                    Rvalue::UnaryOp(_, op) | Rvalue::Use(op) => {
+                    Rvalue::UnaryOp(_, op) | Rvalue::Use(op, _) => {
                         self.visit_operand(op, location);
                     }
                 }

compiler/rustc_public/src/unstable/convert/stable/mir.rs

@@ -157,9 +157,6 @@ fn stable<'cx>(
             mir::StatementKind::StorageDead(place) => {
                 crate::mir::StatementKind::StorageDead(place.stable(tables, cx))
             }
-            mir::StatementKind::Retag(retag, place) => {
-                crate::mir::StatementKind::Retag(retag.stable(tables, cx), place.stable(tables, cx))
-            }
             mir::StatementKind::PlaceMention(place) => {
                 crate::mir::StatementKind::PlaceMention(place.stable(tables, cx))
             }
@@ -195,7 +192,9 @@ fn stable<'cx>(
     ) -> Self::T {
         use rustc_middle::mir::Rvalue::*;
         match self {
-            Use(op) => crate::mir::Rvalue::Use(op.stable(tables, cx)),
+            Use(op, retag) => {
+                crate::mir::Rvalue::Use(op.stable(tables, cx), retag.stable(tables, cx))
+            }
             Repeat(op, len) => {
                 let len = len.stable(tables, cx);
                 crate::mir::Rvalue::Repeat(op.stable(tables, cx), len)
@@ -462,15 +461,13 @@ fn stable(&self, _: &mut Tables<'_, BridgeTys>, _: &CompilerCtxt<'_, BridgeTys>)
     }
 }
 
-impl<'tcx> Stable<'tcx> for mir::RetagKind {
-    type T = crate::mir::RetagKind;
+impl<'tcx> Stable<'tcx> for mir::WithRetag {
+    type T = crate::mir::WithRetag;
     fn stable(&self, _: &mut Tables<'_, BridgeTys>, _: &CompilerCtxt<'_, BridgeTys>) -> Self::T {
-        use rustc_middle::mir::RetagKind;
+        use rustc_middle::mir::WithRetag;
         match self {
-            RetagKind::FnEntry => crate::mir::RetagKind::FnEntry,
-            RetagKind::TwoPhase => crate::mir::RetagKind::TwoPhase,
-            RetagKind::Raw => crate::mir::RetagKind::Raw,
-            RetagKind::Default => crate::mir::RetagKind::Default,
+            WithRetag::Yes => crate::mir::WithRetag::Yes,
+            WithRetag::No => crate::mir::WithRetag::No,
         }
     }
 }

compiler/rustc_session/src/options.rs

@@ -2450,9 +2450,6 @@ pub(crate) fn parse_assert_incr_state(
         "align all functions to at least this many bytes. Must be a power of 2"),
     min_recursion_limit: Option<usize> = (None, parse_opt_number, [TRACKED],
         "set a minimum recursion limit (final limit = max(this, recursion_limit_from_crate))"),
-    mir_emit_retag: bool = (false, parse_bool, [TRACKED],
-        "emit Retagging MIR statements, interpreted e.g., by miri; implies -Zmir-opt-level=0 \
-        (default: no)"),
     mir_enable_passes: Vec<(String, bool)> = (Vec::new(), parse_list_with_polarity, [TRACKED],
         "use like `-Zmir-enable-passes=+DestinationPropagation,-InstSimplify`. Forces the \
         specified passes to be enabled, overriding all other checks. In particular, this will \

compiler/rustc_span/src/symbol.rs

@@ -1295,7 +1295,6 @@
         mir_move,
         mir_offset,
         mir_ptr_metadata,
-        mir_retag,
         mir_return,
         mir_return_to,
         mir_set_discriminant,

compiler/rustc_ty_utils/src/abi.rs

@@ -584,6 +584,11 @@ fn fn_abi_new_uncached<'tcx>(
         extra_args
     };
 
+    // For some functions we treat their first argument as-if it was a mutable reference
+    // even if it is a raw pointer. This is a terrible hack since it becomes effectively
+    // part of the MIR semantics that every MIR consumer needs to be aware of.
+    // Do not add more functions here! Instead, put the actually intended type in the signature.
+    // FIXME(#154274): remove this hack.
     let is_drop_in_place = determined_fn_def_id.is_some_and(|def_id| {
         tcx.is_lang_item(def_id, LangItem::DropInPlace)
             || tcx.is_lang_item(def_id, LangItem::AsyncDropInPlace)

library/alloc/src/boxed.rs

@@ -1430,9 +1430,13 @@ pub unsafe fn from_non_null(ptr: NonNull<T>) -> Self {
     pub fn into_raw(b: Self) -> *mut T {
         // Avoid `into_raw_with_allocator` as that interacts poorly with Miri's Stacked Borrows.
         let mut b = mem::ManuallyDrop::new(b);
-        // We go through the built-in deref for `Box`, which is crucial for Miri to recognize this
-        // operation for it's alias tracking.
-        &raw mut **b
+        // We need to give Miri (specifically, Stacked Borrows) a chance to recognize this as a
+        // safe-to-raw-pointer cast. To achieve this, we first create a mutable reference, and then
+        // cast that to a raw pointer -- this cast is recognized by the aliasing model and leads to
+        // a suitable retag.
+        // It would be wrong for `into_raw_with_allocator` to do the same as that would induce
+        // uniqueness assumptions (from the `&mut`) that we only want with the default allocator.
+        (&mut **b) as *mut T
     }
 
     /// Consumes the `Box`, returning a wrapped `NonNull` pointer.

library/core/src/intrinsics/mir.rs

@@ -229,7 +229,7 @@
 //!
 //! #### Statements
 //!  - Assign statements work via normal Rust assignment.
-//!  - [`Retag`], [`StorageLive`], [`StorageDead`] statements have an associated function.
+//!  - [`StorageLive`], [`StorageDead`] statements have an associated function.
 //!
 //! #### Rvalues
 //!
@@ -407,7 +407,6 @@ fn UnwindResume()
     "mir_ptr_metadata",
     fn PtrMetadata<P: ?Sized>(place: *const P) -> <P as ::core::ptr::Pointee>::Metadata
 );
-define!("mir_retag", fn Retag<T>(place: T));
 define!("mir_move", fn Move<T>(place: T) -> T);
 define!("mir_static", fn Static<T>(s: T) -> &'static T);
 define!("mir_static_mut", fn StaticMut<T>(s: T) -> *mut T);

src/tools/clippy/clippy_lints/src/redundant_clone.rs

@@ -293,7 +293,7 @@ fn find_stmt_assigns_to<'tcx>(
     })?;
 
     match (by_ref, rvalue) {
-        (true, mir::Rvalue::Ref(_, _, place)) | (false, mir::Rvalue::Use(mir::Operand::Copy(place))) => {
+        (true, mir::Rvalue::Ref(_, _, place)) | (false, mir::Rvalue::Use(mir::Operand::Copy(place), _)) => {
             Some(base_local_and_movability(cx, mir, *place))
         },
         (false, mir::Rvalue::Ref(_, _, place)) => {

src/tools/clippy/clippy_utils/src/mir/possible_borrower.rs

@@ -155,7 +155,7 @@ fn rvalue_locals(rvalue: &mir::Rvalue<'_>, mut visit: impl FnMut(mir::Local)) {
     };
 
     match rvalue {
-        Use(op) | Repeat(op, _) | Cast(_, op, _) | UnaryOp(_, op) => visit_op(op),
+        Use(op, _) | Repeat(op, _) | Cast(_, op, _) | UnaryOp(_, op) => visit_op(op),
         Aggregate(_, ops) => ops.iter().for_each(visit_op),
         BinaryOp(_, box (lhs, rhs)) => {
             visit_op(lhs);

src/tools/clippy/clippy_utils/src/mir/possible_origin.rs

@@ -46,7 +46,7 @@ fn visit_assign(&mut self, place: &mir::Place<'tcx>, rvalue: &mir::Rvalue<'_>, _
             mir::Rvalue::Ref(_, mir::BorrowKind::Mut { .. }, borrowed) |
             // _2: &mut _;
             // _3 = move _2
-            mir::Rvalue::Use(mir::Operand::Move(borrowed))  |
+            mir::Rvalue::Use(mir::Operand::Move(borrowed), _)  |
             // _3 = move _2 as &mut _;
             mir::Rvalue::Cast(_, mir::Operand::Move(borrowed), _)
                 => {

src/tools/clippy/clippy_utils/src/qualify_min_const_fn.rs

@@ -134,7 +134,7 @@ fn check_rvalue<'tcx>(
         },
         Rvalue::CopyForDeref(place) => check_place(cx, *place, span, body, msrv),
         Rvalue::Repeat(operand, _)
-        | Rvalue::Use(operand)
+        | Rvalue::Use(operand, _)
         | Rvalue::WrapUnsafeBinder(operand, _)
         | Rvalue::Cast(
             CastKind::PointerWithExposedProvenance
@@ -244,7 +244,6 @@ fn check_statement<'tcx>(
         // These are all NOPs
         StatementKind::StorageLive(_)
         | StatementKind::StorageDead(_)
-        | StatementKind::Retag { .. }
         | StatementKind::AscribeUserType(..)
         | StatementKind::PlaceMention(..)
         | StatementKind::Coverage(..)

src/tools/miri/README.md

@@ -432,14 +432,13 @@ to Miri failing to detect cases of undefined behavior in a program.
   be detected. Using this flag is **unsound** (but the affected soundness rules
   are experimental). Later flags take precedence: borrow tracking can be reactivated
   by `-Zmiri-tree-borrows`.
-* `-Zmiri-disable-validation` disables enforcing validity invariants, which are
-  enforced by default. This only disables these checks for typed copies; using
-  invalid values in any other operation will still cause an error. This is mostly useful 
-  to focus on other failures (such as out-of-bounds accesses) first. Setting this 
-  flag means Miri can miss bugs in your program. However, this can also help to 
-  make Miri run faster. Using this flag is **unsound**.
-* `-Zmiri-disable-weak-memory-emulation` disables the emulation of some C++11 weak
-  memory effects.
+* `-Zmiri-disable-validation` disables enforcing validity invariants, which are enforced by default.
+  This only disables these checks for typed copies; using invalid values in any other operation will
+  still cause an error. This also disables the aliasing model (Stacked/Tree Borrows). This is mostly
+  useful to focus on other failures (such as out-of-bounds accesses) first. Setting this flag means
+  Miri can miss bugs in your program. However, this can also help to make Miri run faster. Using
+  this flag is **unsound**.
+* `-Zmiri-disable-weak-memory-emulation` disables the emulation of some C++11 weak memory effects.
 * `-Zmiri-fixed-schedule` disables preemption (like `-Zmiri-preemption-rate=0.0`) and furthermore
   disables the randomization of the next thread to be picked, instead fixing a round-robin schedule.
   Note however that other aspects of Miri's concurrency behavior are still randomize; use
@@ -522,17 +521,6 @@ to Miri failing to detect cases of undefined behavior in a program.
 
 [function ABI]: https://doc.rust-lang.org/reference/items/functions.html#extern-function-qualifier
 
-Some native rustc `-Z` flags are also very relevant for Miri:
-
-* `-Zmir-opt-level` controls how many MIR optimizations are performed.  Miri
-  overrides the default to be `0`; be advised that using any higher level can
-  make Miri miss bugs in your program because they got optimized away.
-* `-Zalways-encode-mir` makes rustc dump MIR even for completely monomorphic
-  functions.  This is needed so that Miri can execute such functions, so Miri
-  sets this flag per default.
-* `-Zmir-emit-retag` controls whether `Retag` statements are emitted. Miri
-  enables this per default because it is needed for [Stacked Borrows] and [Tree Borrows].
-
 Moreover, Miri recognizes some environment variables:
 
 * `MIRIFLAGS` defines extra flags to be passed to Miri.

src/tools/miri/src/bin/miri.rs

@@ -721,6 +721,11 @@ fn main() -> ExitCode {
         }
     }
 
+    // Disabling validation also disables aliasing checks (as retags are done during validation).
+    if miri_config.validation == ValidationMode::No {
+        miri_config.borrow_tracker = None;
+    }
+
     // Native calls and strict provenance are not compatible.
     if !miri_config.native_lib.is_empty() && miri_config.provenance_mode == ProvenanceMode::Strict {
         fatal_error!("strict provenance is not compatible with calling native functions");

src/tools/miri/src/borrow_tracker/mod.rs

@@ -1,10 +1,10 @@
 use std::cell::RefCell;
-use std::fmt;
 use std::num::NonZero;
+use std::{fmt, mem};
 
 use rustc_abi::Size;
 use rustc_data_structures::fx::{FxHashMap, FxHashSet};
-use rustc_middle::mir::RetagKind;
+use rustc_middle::ty::Ty;
 use smallvec::SmallVec;
 
 use crate::*;
@@ -103,6 +103,8 @@ fn visit_provenance(&self, visit: &mut VisitWith<'_>) {
 pub struct GlobalStateInner {
     /// Borrow tracker method currently in use.
     borrow_tracker_method: BorrowTrackerMethod,
+    /// The currently active retag mode.
+    retag_mode: RetagMode,
     /// Next unused pointer ID (tag).
     next_ptr_tag: BorTag,
     /// Table storing the "root" tag for each allocation.
@@ -157,6 +159,7 @@ pub fn new(
     ) -> Self {
         GlobalStateInner {
             borrow_tracker_method,
+            retag_mode: RetagMode::Default,
             next_ptr_tag: BorTag::one(),
             root_ptr_tags: FxHashMap::default(),
             protected_tags: FxHashMap::default(),
@@ -267,36 +270,43 @@ impl<'tcx> EvalContextExt<'tcx> for crate::MiriInterpCx<'tcx> {}
 pub trait EvalContextExt<'tcx>: crate::MiriInterpCxExt<'tcx> {
     fn retag_ptr_value(
         &mut self,
-        kind: RetagKind,
         val: &ImmTy<'tcx>,
-    ) -> InterpResult<'tcx, ImmTy<'tcx>> {
-        let _trace = enter_trace_span!(borrow_tracker::retag_ptr_value, ?kind, ?val.layout);
+        ty: Ty<'tcx>,
+    ) -> InterpResult<'tcx, Option<ImmTy<'tcx>>> {
+        let _trace = enter_trace_span!(borrow_tracker::retag_ptr_value, ?ty);
         let this = self.eval_context_mut();
-        let method = this.machine.borrow_tracker.as_ref().unwrap().borrow().borrow_tracker_method;
+        let state = this.machine.borrow_tracker.as_mut().unwrap().get_mut();
+        let method = state.borrow_tracker_method;
+        let retag_mode = state.retag_mode;
+        info!("retag_ptr_value: type={ty}, mode={retag_mode:?}, val={:?}", **val);
         match method {
-            BorrowTrackerMethod::StackedBorrows => this.sb_retag_ptr_value(kind, val),
-            BorrowTrackerMethod::TreeBorrows { .. } => this.tb_retag_ptr_value(kind, val),
+            BorrowTrackerMethod::StackedBorrows => this.sb_retag_ptr_value(val, ty, retag_mode),
+            BorrowTrackerMethod::TreeBorrows { .. } => this.tb_retag_ptr_value(val, ty, retag_mode),
         }
     }
 
-    fn retag_place_contents(
+    fn with_retag_mode<T>(
         &mut self,
-        kind: RetagKind,
-        place: &PlaceTy<'tcx>,
-    ) -> InterpResult<'tcx> {
-        let _trace = enter_trace_span!(borrow_tracker::retag_place_contents, ?kind, ?place);
-        let this = self.eval_context_mut();
-        let method = this.machine.borrow_tracker.as_ref().unwrap().borrow().borrow_tracker_method;
-        match method {
-            BorrowTrackerMethod::StackedBorrows => this.sb_retag_place_contents(kind, place),
-            BorrowTrackerMethod::TreeBorrows { .. } => this.tb_retag_place_contents(kind, place),
-        }
+        mode: RetagMode,
+        f: impl FnOnce(&mut Self) -> InterpResult<'tcx, T>,
+    ) -> InterpResult<'tcx, T> {
+        // Set up new mode, remembering the old mode.
+        let state = self.eval_context_mut().machine.borrow_tracker.as_mut().unwrap().get_mut();
+        let old_mode = mem::replace(&mut state.retag_mode, mode);
+
+        let ret = f(self);
+
+        // Restore old mode.
+        let state = self.eval_context_mut().machine.borrow_tracker.as_mut().unwrap().get_mut();
+        state.retag_mode = old_mode;
+
+        ret
     }
 
     fn protect_place(&mut self, place: &MPlaceTy<'tcx>) -> InterpResult<'tcx, MPlaceTy<'tcx>> {
         let _trace = enter_trace_span!(borrow_tracker::protect_place, ?place);
         let this = self.eval_context_mut();
-        let method = this.machine.borrow_tracker.as_ref().unwrap().borrow().borrow_tracker_method;
+        let method = this.machine.borrow_tracker.as_mut().unwrap().get_mut().borrow_tracker_method;
         match method {
             BorrowTrackerMethod::StackedBorrows => this.sb_protect_place(place),
             BorrowTrackerMethod::TreeBorrows { .. } => this.tb_protect_place(place),
@@ -321,7 +331,7 @@ fn give_pointer_debug_name(
         name: &str,
     ) -> InterpResult<'tcx> {
         let this = self.eval_context_mut();
-        let method = this.machine.borrow_tracker.as_ref().unwrap().borrow().borrow_tracker_method;
+        let method = this.machine.borrow_tracker.as_mut().unwrap().get_mut().borrow_tracker_method;
         match method {
             BorrowTrackerMethod::StackedBorrows => {
                 this.tcx.tcx.dcx().warn("Stacked Borrows does not support named pointers; `miri_pointer_name` is a no-op");
@@ -334,11 +344,11 @@ fn give_pointer_debug_name(
 
     fn print_borrow_state(&mut self, alloc_id: AllocId, show_unnamed: bool) -> InterpResult<'tcx> {
         let this = self.eval_context_mut();
-        let Some(borrow_tracker) = &this.machine.borrow_tracker else {
+        let Some(borrow_tracker) = &mut this.machine.borrow_tracker else {
             eprintln!("attempted to print borrow state, but no borrow state is being tracked");
             return interp_ok(());
         };
-        let method = borrow_tracker.borrow().borrow_tracker_method;
+        let method = borrow_tracker.get_mut().borrow_tracker_method;
         match method {
             BorrowTrackerMethod::StackedBorrows => this.print_stacks(alloc_id),
             BorrowTrackerMethod::TreeBorrows { .. } => this.print_tree(alloc_id, show_unnamed),

src/tools/miri/src/borrow_tracker/stacked_borrows/diagnostics.rs

@@ -191,7 +191,6 @@ struct RetagOp {
 #[derive(Debug, Clone, Copy, PartialEq)]
 pub struct RetagInfo {
     pub cause: RetagCause,
-    pub in_field: bool,
 }
 
 #[derive(Debug, Clone, Copy, PartialEq)]
@@ -386,10 +385,7 @@ pub(super) fn grant_error(&self, stack: &Stack) -> InterpErrorKind<'tcx> {
             self.history.id,
             self.offset.bytes(),
         );
-        let mut helps = vec![operation_summary(&op.info.summary(), self.history.id, op.range)];
-        if op.info.in_field {
-            helps.push(format!("errors for retagging in fields are fairly new; please reach out to us (e.g. at <https://rust-lang.zulipchat.com/#narrow/stream/269128-miri>) if you find this error troubling"));
-        }
+        let helps = vec![operation_summary(&op.info.summary(), self.history.id, op.range)];
         err_sb_ub(
             format!("{action}{}", error_cause(stack, op.orig_tag)),
             helps,
@@ -505,16 +501,12 @@ fn error_cause(stack: &Stack, prov_extra: ProvenanceExtra) -> &'static str {
 
 impl RetagInfo {
     fn summary(&self) -> String {
-        let mut s = match self.cause {
+        match self.cause {
             RetagCause::Normal => "retag",
             RetagCause::FnEntry => "function-entry retag",
             RetagCause::InPlaceFnPassing => "in-place function argument/return passing protection",
             RetagCause::TwoPhase => "two-phase retag",
         }
-        .to_string();
-        if self.in_field {
-            s.push_str(" (of a reference/box inside this compound value)");
-        }
-        s
+        .to_string()
     }
 }

src/tools/miri/src/borrow_tracker/stacked_borrows/mod.rs

@@ -5,13 +5,13 @@
 mod item;
 mod stack;
 
+use std::cmp;
 use std::fmt::Write;
 use std::sync::atomic::AtomicBool;
-use std::{cmp, mem};
 
 use rustc_abi::Size;
 use rustc_data_structures::fx::FxHashSet;
-use rustc_middle::mir::{Mutability, RetagKind};
+use rustc_middle::mir::Mutability;
 use rustc_middle::ty::layout::HasTypingEnv;
 use rustc_middle::ty::{self, Ty};
 
@@ -59,13 +59,13 @@ enum NewPermission {
 impl NewPermission {
     /// A key function: determine the permissions to grant at a retag for the given kind of
     /// reference/pointer.
-    fn from_ref_ty<'tcx>(ty: Ty<'tcx>, kind: RetagKind, cx: &crate::MiriInterpCx<'tcx>) -> Self {
-        let protector = (kind == RetagKind::FnEntry).then_some(ProtectorKind::StrongProtector);
+    fn from_ref_ty<'tcx>(ty: Ty<'tcx>, mode: RetagMode, cx: &crate::MiriInterpCx<'tcx>) -> Self {
+        let protector = (mode == RetagMode::FnEntry).then_some(ProtectorKind::StrongProtector);
         match ty.kind() {
             ty::Ref(_, pointee, Mutability::Mut) => {
-                if kind == RetagKind::TwoPhase {
+                if mode == RetagMode::TwoPhase {
                     // We mostly just give up on 2phase-borrows, and treat these exactly like raw pointers.
-                    assert!(protector.is_none()); // RetagKind can't be both FnEntry and TwoPhase.
+                    assert!(protector.is_none()); // RetagMode can't be both FnEntry and TwoPhase.
                     NewPermission::Uniform {
                         perm: Permission::SharedReadWrite,
                         access: None,
@@ -90,7 +90,8 @@ fn from_ref_ty<'tcx>(ty: Ty<'tcx>, kind: RetagKind, cx: &crate::MiriInterpCx<'tc
                 }
             }
             ty::RawPtr(_, Mutability::Mut) => {
-                assert!(protector.is_none()); // RetagKind can't be both FnEntry and Raw.
+                assert!(mode == RetagMode::Raw);
+                assert!(protector.is_none()); // RetagMode can't be both FnEntry and Raw.
                 // Mutable raw pointer. No access, not protected.
                 NewPermission::Uniform {
                     perm: Permission::SharedReadWrite,
@@ -114,7 +115,8 @@ fn from_ref_ty<'tcx>(ty: Ty<'tcx>, kind: RetagKind, cx: &crate::MiriInterpCx<'tc
                 }
             }
             ty::RawPtr(_, Mutability::Not) => {
-                assert!(protector.is_none()); // RetagKind can't be both FnEntry and Raw.
+                assert!(mode == RetagMode::Raw);
+                assert!(protector.is_none()); // RetagMode can't be both FnEntry and Raw.
                 // `*const T`, when freshly created, are read-only in the frozen part.
                 NewPermission::FreezeSensitive {
                     freeze_perm: Permission::SharedReadOnly,
@@ -128,7 +130,7 @@ fn from_ref_ty<'tcx>(ty: Ty<'tcx>, kind: RetagKind, cx: &crate::MiriInterpCx<'tc
         }
     }
 
-    fn from_box_ty<'tcx>(ty: Ty<'tcx>, kind: RetagKind, cx: &crate::MiriInterpCx<'tcx>) -> Self {
+    fn from_box_ty<'tcx>(ty: Ty<'tcx>, mode: RetagMode, cx: &crate::MiriInterpCx<'tcx>) -> Self {
         // `ty` is not the `Box` but the field of the Box with this pointer (due to allocator handling).
         let pointee = ty.builtin_deref(true).unwrap();
         if pointee.is_unpin(*cx.tcx, cx.typing_env())
@@ -139,7 +141,7 @@ fn from_box_ty<'tcx>(ty: Ty<'tcx>, kind: RetagKind, cx: &crate::MiriInterpCx<'tc
             NewPermission::Uniform {
                 perm: Permission::Unique,
                 access: Some(AccessKind::Write),
-                protector: (kind == RetagKind::FnEntry).then_some(ProtectorKind::WeakProtector),
+                protector: (mode == RetagMode::FnEntry).then_some(ProtectorKind::WeakProtector),
             }
         } else {
             // `!Unpin` boxes do not get `noalias` nor `dereferenceable`.
@@ -664,7 +666,7 @@ fn sb_reborrow(
 
         if size == Size::ZERO {
             trace!(
-                "reborrow of size 0: reference {:?} derived from {:?} (pointee {})",
+                "reborrow of size 0: reference {:?} derived from {:?} (pointee {}) with permissions {new_perm:?}",
                 new_tag,
                 place.ptr(),
                 place.layout.ty,
@@ -699,7 +701,7 @@ fn sb_reborrow(
         log_creation(this, Some((alloc_id, base_offset, orig_tag)))?;
 
         trace!(
-            "reborrow: reference {:?} derived from {:?} (pointee {}): {:?}, size {}",
+            "reborrow: reference {:?} derived from {:?} (pointee {}) with permissions {new_perm:?}: {:?}, size {}",
             new_tag,
             orig_tag,
             place.layout.ty,
@@ -849,136 +851,38 @@ fn sb_retag_place(
         // one must also be `Some`.)
         interp_ok(place.clone().map_provenance(|_| new_prov.unwrap()))
     }
-
-    /// Retags an individual pointer, returning the retagged version.
-    /// `kind` indicates what kind of reference is being created.
-    fn sb_retag_reference(
-        &mut self,
-        val: &ImmTy<'tcx>,
-        new_perm: NewPermission,
-        info: RetagInfo, // diagnostics info about this retag
-    ) -> InterpResult<'tcx, ImmTy<'tcx>> {
-        let this = self.eval_context_mut();
-        let place = this.imm_ptr_to_mplace(val)?;
-        let new_place = this.sb_retag_place(&place, new_perm, info)?;
-        interp_ok(ImmTy::from_immediate(new_place.to_ref(this), val.layout))
-    }
 }
 
 impl<'tcx> EvalContextExt<'tcx> for crate::MiriInterpCx<'tcx> {}
 pub trait EvalContextExt<'tcx>: crate::MiriInterpCxExt<'tcx> {
     fn sb_retag_ptr_value(
         &mut self,
-        kind: RetagKind,
         val: &ImmTy<'tcx>,
-    ) -> InterpResult<'tcx, ImmTy<'tcx>> {
+        ty: Ty<'tcx>,
+        mode: RetagMode,
+    ) -> InterpResult<'tcx, Option<ImmTy<'tcx>>> {
         let this = self.eval_context_mut();
-        let new_perm = NewPermission::from_ref_ty(val.layout.ty, kind, this);
-        let cause = match kind {
-            RetagKind::TwoPhase => RetagCause::TwoPhase,
-            RetagKind::FnEntry => unreachable!(),
-            RetagKind::Raw | RetagKind::Default => RetagCause::Normal,
+        let cause = match mode {
+            RetagMode::TwoPhase => RetagCause::TwoPhase,
+            RetagMode::FnEntry => RetagCause::FnEntry,
+            RetagMode::Raw | RetagMode::Default => RetagCause::Normal,
+            RetagMode::None => return interp_ok(None), // no retagging
         };
-        this.sb_retag_reference(val, new_perm, RetagInfo { cause, in_field: false })
-    }
-
-    fn sb_retag_place_contents(
-        &mut self,
-        kind: RetagKind,
-        place: &PlaceTy<'tcx>,
-    ) -> InterpResult<'tcx> {
-        let this = self.eval_context_mut();
-        let retag_cause = match kind {
-            RetagKind::TwoPhase => unreachable!(), // can only happen in `retag_ptr_value`
-            RetagKind::FnEntry => RetagCause::FnEntry,
-            RetagKind::Default | RetagKind::Raw => RetagCause::Normal,
+        let new_perm = if ty.is_box() {
+            if ty.is_box_global(*this.tcx) {
+                NewPermission::from_box_ty(val.layout.ty, mode, this)
+            } else {
+                // Boxes with local allocator are not retagged.
+                return interp_ok(None);
+            }
+        } else {
+            NewPermission::from_ref_ty(val.layout.ty, mode, this)
         };
-        let mut visitor = RetagVisitor { ecx: this, kind, retag_cause, in_field: false };
-        return visitor.visit_value(place);
 
-        // The actual visitor.
-        struct RetagVisitor<'ecx, 'tcx> {
-            ecx: &'ecx mut MiriInterpCx<'tcx>,
-            kind: RetagKind,
-            retag_cause: RetagCause,
-            in_field: bool,
-        }
-        impl<'ecx, 'tcx> RetagVisitor<'ecx, 'tcx> {
-            #[inline(always)] // yes this helps in our benchmarks
-            fn retag_ptr_inplace(
-                &mut self,
-                place: &PlaceTy<'tcx>,
-                new_perm: NewPermission,
-            ) -> InterpResult<'tcx> {
-                let val = self.ecx.read_immediate(&self.ecx.place_to_op(place)?)?;
-                let val = self.ecx.sb_retag_reference(
-                    &val,
-                    new_perm,
-                    RetagInfo { cause: self.retag_cause, in_field: self.in_field },
-                )?;
-                self.ecx.write_immediate(*val, place)?;
-
-                interp_ok(())
-            }
-        }
-        impl<'ecx, 'tcx> ValueVisitor<'tcx, MiriMachine<'tcx>> for RetagVisitor<'ecx, 'tcx> {
-            type V = PlaceTy<'tcx>;
-
-            #[inline(always)]
-            fn ecx(&self) -> &MiriInterpCx<'tcx> {
-                self.ecx
-            }
-
-            fn visit_box(&mut self, box_ty: Ty<'tcx>, place: &PlaceTy<'tcx>) -> InterpResult<'tcx> {
-                // Only boxes for the global allocator get any special treatment.
-                if box_ty.is_box_global(*self.ecx.tcx) {
-                    // Boxes get a weak protectors, since they may be deallocated.
-                    let new_perm = NewPermission::from_box_ty(place.layout.ty, self.kind, self.ecx);
-                    self.retag_ptr_inplace(place, new_perm)?;
-                }
-                interp_ok(())
-            }
-
-            fn visit_value(&mut self, place: &PlaceTy<'tcx>) -> InterpResult<'tcx> {
-                // If this place is smaller than a pointer, we know that it can't contain any
-                // pointers we need to retag, so we can stop recursion early.
-                // This optimization is crucial for ZSTs, because they can contain way more fields
-                // than we can ever visit.
-                if place.layout.is_sized() && place.layout.size < self.ecx.pointer_size() {
-                    return interp_ok(());
-                }
-
-                // Check the type of this value to see what to do with it (retag, or recurse).
-                match place.layout.ty.kind() {
-                    ty::Ref(..) | ty::RawPtr(..) => {
-                        if matches!(place.layout.ty.kind(), ty::Ref(..))
-                            || self.kind == RetagKind::Raw
-                        {
-                            let new_perm =
-                                NewPermission::from_ref_ty(place.layout.ty, self.kind, self.ecx);
-                            self.retag_ptr_inplace(place, new_perm)?;
-                        }
-                    }
-                    ty::Adt(adt, _) if adt.is_box() => {
-                        // Recurse for boxes, they require some tricky handling and will end up in `visit_box` above.
-                        // (Yes this means we technically also recursively retag the allocator itself
-                        // even if field retagging is not enabled. *shrug*)
-                        self.walk_value(place)?;
-                    }
-                    ty::Adt(adt, _) if adt.is_maybe_dangling() => {
-                        // Skip traversing for everything inside of `MaybeDangling`
-                    }
-                    _ => {
-                        // Not a reference/pointer/box. Recurse.
-                        let in_field = mem::replace(&mut self.in_field, true); // remember and restore old value
-                        self.walk_value(place)?;
-                        self.in_field = in_field;
-                    }
-                }
-
-                interp_ok(())
-            }
-        }
+        let info = RetagInfo { cause };
+        let place = this.imm_ptr_to_mplace(val)?;
+        let new_place = this.sb_retag_place(&place, new_perm, info)?;
+        interp_ok(Some(ImmTy::from_immediate(new_place.to_ref(this), val.layout)))
     }
 
     /// Protect a place so that it cannot be used any more for the duration of the current function
@@ -994,11 +898,7 @@ fn sb_protect_place(&mut self, place: &MPlaceTy<'tcx>) -> InterpResult<'tcx, MPl
             access: Some(AccessKind::Write),
             protector: Some(ProtectorKind::StrongProtector),
         };
-        this.sb_retag_place(
-            place,
-            new_perm,
-            RetagInfo { cause: RetagCause::InPlaceFnPassing, in_field: false },
-        )
+        this.sb_retag_place(place, new_perm, RetagInfo { cause: RetagCause::InPlaceFnPassing })
     }
 
     /// Mark the given tag as exposed. It was found on a pointer with the given AllocId.

src/tools/miri/src/borrow_tracker/tree_borrows/mod.rs

@@ -1,5 +1,5 @@
 use rustc_abi::Size;
-use rustc_middle::mir::{Mutability, RetagKind};
+use rustc_middle::mir::Mutability;
 use rustc_middle::ty::layout::HasTypingEnv;
 use rustc_middle::ty::{self, Ty};
 
@@ -125,13 +125,17 @@ impl<'tcx> NewPermission {
     fn new(
         pointee: Ty<'tcx>,
         ref_mutability: Option<Mutability>,
-        retag_kind: RetagKind,
+        mode: RetagMode,
         cx: &crate::MiriInterpCx<'tcx>,
     ) -> Option<Self> {
+        if mode == RetagMode::None {
+            return None;
+        }
+
         let ty_is_unpin = pointee.is_unpin(*cx.tcx, cx.typing_env())
             && pointee.is_unsafe_unpin(*cx.tcx, cx.typing_env());
         let ty_is_freeze = pointee.is_freeze(*cx.tcx, cx.typing_env());
-        let is_protected = retag_kind == RetagKind::FnEntry;
+        let is_protected = mode == RetagMode::FnEntry;
 
         // Check if the implicit writes check has been enabled for this function using the `-Zmiri-tree-borrows-implicit-writes` flag
         let implicit_writes = cx
@@ -460,18 +464,6 @@ fn tb_retag_place(
         // one must also be `Some`.)
         interp_ok(place.clone().map_provenance(|_| new_prov.unwrap()))
     }
-
-    /// Retags an individual pointer, returning the retagged version.
-    fn tb_retag_reference(
-        &mut self,
-        val: &ImmTy<'tcx>,
-        new_perm: NewPermission,
-    ) -> InterpResult<'tcx, ImmTy<'tcx>> {
-        let this = self.eval_context_mut();
-        let place = this.imm_ptr_to_mplace(val)?;
-        let new_place = this.tb_retag_place(&place, new_perm)?;
-        interp_ok(ImmTy::from_immediate(new_place.to_ref(this), val.layout))
-    }
 }
 
 impl<'tcx> EvalContextExt<'tcx> for crate::MiriInterpCx<'tcx> {}
@@ -480,112 +472,36 @@ pub trait EvalContextExt<'tcx>: crate::MiriInterpCxExt<'tcx> {
     /// raw pointers are never reborrowed.
     fn tb_retag_ptr_value(
         &mut self,
-        kind: RetagKind,
         val: &ImmTy<'tcx>,
-    ) -> InterpResult<'tcx, ImmTy<'tcx>> {
+        ty: Ty<'tcx>,
+        mode: RetagMode,
+    ) -> InterpResult<'tcx, Option<ImmTy<'tcx>>> {
         let this = self.eval_context_mut();
-        let new_perm = match val.layout.ty.kind() {
+        let new_perm = match ty.kind() {
+            _ if ty.is_box_global(*this.tcx) => {
+                // The `None` marks this as a Box.
+                NewPermission::new(ty.builtin_deref(true).unwrap(), None, mode, this)
+            }
             &ty::Ref(_, pointee, mutability) =>
-                NewPermission::new(pointee, Some(mutability), kind, this),
-            _ => None,
+                NewPermission::new(pointee, Some(mutability), mode, this),
+
+            &ty::RawPtr(..) => {
+                assert!(mode == RetagMode::Raw);
+                // We don't give new tags to raw pointers.
+                None
+            }
+            _ if ty.is_box() => {
+                // No retagging for boxes with local allocators.
+                None
+            }
+            _ => panic!("tb_retag_ptr_value: invalid type {ty}"),
         };
         if let Some(new_perm) = new_perm {
-            this.tb_retag_reference(val, new_perm)
+            let place = this.imm_ptr_to_mplace(val)?;
+            let new_place = this.tb_retag_place(&place, new_perm)?;
+            interp_ok(Some(ImmTy::from_immediate(new_place.to_ref(this), val.layout)))
         } else {
-            interp_ok(val.clone())
-        }
-    }
-
-    /// Retag all pointers that are stored in this place.
-    fn tb_retag_place_contents(
-        &mut self,
-        kind: RetagKind,
-        place: &PlaceTy<'tcx>,
-    ) -> InterpResult<'tcx> {
-        let this = self.eval_context_mut();
-        let mut visitor = RetagVisitor { ecx: this, kind };
-        return visitor.visit_value(place);
-
-        // The actual visitor.
-        struct RetagVisitor<'ecx, 'tcx> {
-            ecx: &'ecx mut MiriInterpCx<'tcx>,
-            kind: RetagKind,
-        }
-        impl<'ecx, 'tcx> RetagVisitor<'ecx, 'tcx> {
-            #[inline(always)] // yes this helps in our benchmarks
-            fn retag_ptr_inplace(
-                &mut self,
-                place: &PlaceTy<'tcx>,
-                new_perm: Option<NewPermission>,
-            ) -> InterpResult<'tcx> {
-                if let Some(new_perm) = new_perm {
-                    let val = self.ecx.read_immediate(&self.ecx.place_to_op(place)?)?;
-                    let val = self.ecx.tb_retag_reference(&val, new_perm)?;
-                    self.ecx.write_immediate(*val, place)?;
-                }
-                interp_ok(())
-            }
-        }
-        impl<'ecx, 'tcx> ValueVisitor<'tcx, MiriMachine<'tcx>> for RetagVisitor<'ecx, 'tcx> {
-            type V = PlaceTy<'tcx>;
-
-            #[inline(always)]
-            fn ecx(&self) -> &MiriInterpCx<'tcx> {
-                self.ecx
-            }
-
-            /// Regardless of how `Unique` is handled, Boxes are always reborrowed.
-            /// When `Unique` is also reborrowed, then it behaves exactly like `Box`
-            /// except for the fact that `Box` has a non-zero-sized reborrow.
-            fn visit_box(&mut self, box_ty: Ty<'tcx>, place: &PlaceTy<'tcx>) -> InterpResult<'tcx> {
-                // Only boxes for the global allocator get any special treatment.
-                if box_ty.is_box_global(*self.ecx.tcx) {
-                    let pointee = place.layout.ty.builtin_deref(true).unwrap();
-                    let new_perm =
-                        NewPermission::new(pointee, /* not a ref */ None, self.kind, self.ecx);
-                    self.retag_ptr_inplace(place, new_perm)?;
-                }
-                interp_ok(())
-            }
-
-            fn visit_value(&mut self, place: &PlaceTy<'tcx>) -> InterpResult<'tcx> {
-                // If this place is smaller than a pointer, we know that it can't contain any
-                // pointers we need to retag, so we can stop recursion early.
-                // This optimization is crucial for ZSTs, because they can contain way more fields
-                // than we can ever visit.
-                if place.layout.is_sized() && place.layout.size < self.ecx.pointer_size() {
-                    return interp_ok(());
-                }
-
-                // Check the type of this value to see what to do with it (retag, or recurse).
-                match place.layout.ty.kind() {
-                    &ty::Ref(_, pointee, mutability) => {
-                        let new_perm =
-                            NewPermission::new(pointee, Some(mutability), self.kind, self.ecx);
-                        self.retag_ptr_inplace(place, new_perm)?;
-                    }
-                    ty::RawPtr(_, _) => {
-                        // We definitely do *not* want to recurse into raw pointers -- wide raw
-                        // pointers have fields, and for dyn Trait pointees those can have reference
-                        // type!
-                        // We also do not want to reborrow them.
-                    }
-                    ty::Adt(adt, _) if adt.is_box() => {
-                        // Recurse for boxes, they require some tricky handling and will end up in `visit_box` above.
-                        // (Yes this means we technically also recursively retag the allocator itself
-                        // even if field retagging is not enabled. *shrug*)
-                        self.walk_value(place)?;
-                    }
-                    ty::Adt(adt, _) if adt.is_maybe_dangling() => {
-                        // Skip traversing for everything inside of `MaybeDangling`
-                    }
-                    _ => {
-                        // Not a reference/pointer/box. Recurse.
-                        self.walk_value(place)?;
-                    }
-                }
-                interp_ok(())
-            }
+            interp_ok(None)
         }
     }