Rollup merge of #154087 - aytey:fix-fragment-specifier-whitespace, r=Kivooeo

Fix whitespace after fragment specifiers in macro pretty printing

When a macro-generating-macro captures fragment specifier tokens (like `$x:ident`) as `tt` metavariables and replays them before a keyword (like `where`), the pretty printer concatenates them into an invalid fragment specifier (e.g. `$x:identwhere` instead of `$x:ident where`).

This happens because `tt` captures preserve the original token spacing. When the fragment specifier name (e.g. `ident`) was originally the last token before a closing delimiter, it retains `JointHidden` spacing. The `print_tts` function only checks `space_between` for `Spacing::Alone` tokens, so `JointHidden` tokens skip the space check entirely, causing adjacent identifier-like tokens to merge.

The fix adds a check in `print_tts` to insert a space between adjacent identifier-like tokens (identifiers and keywords) regardless of the original spacing, preventing them from being concatenated into invalid tokens.

This is similar to the existing `space_between` mechanism that prevents token merging for `Spacing::Alone` tokens, extended to also handle `Joint`/`JointHidden` cases where two identifier-like tokens would merge.

## Example

**before** (`rustc 1.96.0-nightly (3b1b0ef4d 2026-03-11)`):

```rust
#![feature(prelude_import)]
#![no_std]
extern crate std;
#[prelude_import]
use ::std::prelude::rust_2015::*;
//@ pretty-mode:expanded
//@ pp-exact:macro-fragment-specifier-whitespace.pp

// Test that fragment specifier names in macro definitions are properly
// separated from the following keyword/identifier token when pretty-printed.
// This is a regression test for a bug where `$x:ident` followed by `where`
// was pretty-printed as `$x:identwhere` (an invalid fragment specifier).

macro_rules! outer {
    ($d:tt $($params:tt)*) =>
    {
        #[macro_export] macro_rules! inner
        { ($($params)* where $d($rest:tt)*) => {}; }
    };
}
#[macro_export]
macro_rules! inner { ($x:identwhere $ ($rest : tt)*) => {}; }

fn main() {}
```

**after** (this branch):

```rust
#![feature(prelude_import)]
#![no_std]
extern crate std;
#[prelude_import]
use ::std::prelude::rust_2015::*;
//@ pretty-mode:expanded
//@ pp-exact:macro-fragment-specifier-whitespace.pp

// Test that fragment specifier names in macro definitions are properly
// separated from the following keyword/identifier token when pretty-printed.
// This is a regression test for a bug where `$x:ident` followed by `where`
// was pretty-printed as `$x:identwhere` (an invalid fragment specifier).

macro_rules! outer {
    ($d:tt $($params:tt)*) =>
    {
        #[macro_export] macro_rules! inner
        { ($($params)* where $d($rest:tt)*) => {}; }
    };
}
#[macro_export]
macro_rules! inner { ($x:ident where $ ($rest : tt)*) => {}; }

fn main() {}
```

Notice the `$x:identwhere` in the before — an invalid fragment specifier that causes a hard parse error. The after correctly separates it as `$x:ident where`.
This commit is contained in:
Stuart Cook
2026-03-20 15:33:09 +11:00
committed by GitHub
3 changed files with 63 additions and 0 deletions
@@ -329,6 +329,19 @@ fn print_crate_inner<'a>(
/// - #63896: `#[allow(unused,` must be printed rather than `#[allow(unused ,`
/// - #73345: `#[allow(unused)]` must be printed rather than `# [allow(unused)]`
///
/// Returns `true` if both token trees are identifier-like tokens that would
/// merge into a single token if printed without a space between them.
/// E.g. `ident` + `where` would merge into `identwhere`.
fn idents_would_merge(tt1: &TokenTree, tt2: &TokenTree) -> bool {
fn is_ident_like(tt: &TokenTree) -> bool {
matches!(
tt,
TokenTree::Token(Token { kind: token::Ident(..) | token::NtIdent(..), .. }, _,)
)
}
is_ident_like(tt1) && is_ident_like(tt2)
}
fn space_between(tt1: &TokenTree, tt2: &TokenTree) -> bool {
use Delimiter::*;
use TokenTree::{Delimited as Del, Token as Tok};
@@ -811,6 +824,13 @@ fn print_tts(&mut self, tts: &TokenStream, convert_dollar_crate: bool) {
if let Some(next) = iter.peek() {
if spacing == Spacing::Alone && space_between(tt, next) {
self.space();
} else if spacing != Spacing::Alone && idents_would_merge(tt, next) {
// When tokens from macro `tt` captures preserve their
// original `Joint`/`JointHidden` spacing, adjacent
// identifier-like tokens can be concatenated without a
// space (e.g. `$x:identwhere`). Insert a space to
// prevent this.
self.space();
}
}
}
@@ -0,0 +1,24 @@
#![feature(prelude_import)]
#![no_std]
extern crate std;
#[prelude_import]
use ::std::prelude::rust_2015::*;
//@ pretty-mode:expanded
//@ pp-exact:macro-fragment-specifier-whitespace.pp
// Test that fragment specifier names in macro definitions are properly
// separated from the following keyword/identifier token when pretty-printed.
// This is a regression test for a bug where `$x:ident` followed by `where`
// was pretty-printed as `$x:identwhere` (an invalid fragment specifier).
macro_rules! outer {
($d:tt $($params:tt)*) =>
{
#[macro_export] macro_rules! inner
{ ($($params)* where $d($rest:tt)*) => {}; }
};
}
#[macro_export]
macro_rules! inner { ($x:ident where $ ($rest : tt)*) => {}; }
fn main() {}
@@ -0,0 +1,19 @@
//@ pretty-mode:expanded
//@ pp-exact:macro-fragment-specifier-whitespace.pp
// Test that fragment specifier names in macro definitions are properly
// separated from the following keyword/identifier token when pretty-printed.
// This is a regression test for a bug where `$x:ident` followed by `where`
// was pretty-printed as `$x:identwhere` (an invalid fragment specifier).
macro_rules! outer {
($d:tt $($params:tt)*) => {
#[macro_export]
macro_rules! inner {
($($params)* where $d($rest:tt)*) => {};
}
};
}
outer!($ $x:ident);
fn main() {}