- Add detail on origin of current parser when reaching EOF and stop
saying "found <eof>" and point at the end of macro calls
- Handle empty `cfg_attr` attribute
- Reword empty `derive` attribute error
Add stream_to_parser_with_base_dir
This PR adds `stream_to_parser_with_base_dir`, which creates a parser from a token stream and a base directory.
Context: I would like to parse `cfg_if!` macro and get a list of modules defined inside it from rustfmt so that rustfmt can format those modules (cc https://github.com/rust-lang/rustfmt/issues/3253). To do so, I need to create a parser from `TokenStream` and set the directory of `Parser` to the same directory as the parent directory of a file which contains `cfg_if!` invocation. AFAIK there is no way to achieve this, and hence this PR.
Alternatively, I could change the visibility of `Parser.directory` from `crate` to `pub` so that the value can be modified after initializing a parser. I don't have a preference over either approach (or others, as long as it works).
Move token tree related lexer state to a separate struct
Just a types-based refactoring.
We only used a bunch of fields when tokenizing into a token tree, so let's move them out of the base lexer
Identify when a stmt could have been parsed as an expr
There are some expressions that can be parsed as a statement without
a trailing semicolon depending on the context, which can lead to
confusing errors due to the same looking code being accepted in some
places and not others. Identify these cases and suggest enclosing in
parenthesis making the parse non-ambiguous without changing the
accepted grammar.
Fix#54186, cc #54482, fix#59975, fix#47287.
introduce unescape module
A WIP PR to gauge early feedback
Currently, we deal with escape sequences twice: once when we [lex](https://github.com/rust-lang/rust/blob/112f7e9ac564e2cfcfc13d599c8376a219fde1bc/src/libsyntax/parse/lexer/mod.rs#L928-L1065) a string, and a second time when we [unescape](https://github.com/rust-lang/rust/blob/112f7e9ac564e2cfcfc13d599c8376a219fde1bc/src/libsyntax/parse/mod.rs#L313-L366) literals. Note that we also produce different sets of diagnostics in these two cases.
This PR aims to remove this duplication, by introducing a new `unescape` module as a single source of truth for character escaping rules.
I think this would be a useful cleanup by itself, but I also need this for https://github.com/rust-lang/rust/pull/59706.
In the current state, the PR has `unescape` module which fully (modulo bugs) deals with string and char literals. I am quite happy about the state of this module
What this PR doesn't have yet are:
* [x] handling of byte and byte string literals (should be simple to add)
* [x] good diagnostics
* [x] actual removal of code from lexer (giant `scan_char_or_byte` should go away completely)
* [x] performance check
* [x] general cleanup of the new code
Diagnostics will be the most labor-consuming bit here, but they are mostly a question of just correctly adjusting spans to sub-tokens. The current setup for diagnostics is that `unescape` produces a plain old `enum` with various problems, and they are rendered into `Handler` separately. This bit is not actually required (it is possible to just pass the `Handler` in), but I like the separation between diagnostics and logic this approach imposes, and such separation should again be useful for #59706
cc @eddyb , @petrochenkov
Currently, we deal with escape sequences twice: once when we lex a
string, and a second time when we unescape literals. This PR aims to
remove this duplication, by introducing a new `unescape` mode as a
single source of truth for character escaping rules
There are some expressions that can be parsed as a statement without
a trailing semicolon depending on the context, which can lead to
confusing errors due to the same looking code being accepted in some
places and not others. Identify these cases and suggest enclosing in
parenthesis making the parse non-ambiguous without changing the
accepted grammar.
Rollup of 24 pull requests
Successful merges:
- #58080 (Add FreeBSD armv6 and armv7 targets)
- #58204 (On return type `impl Trait` for block with no expr point at last semi)
- #58269 (Add librustc and libsyntax to rust-src distribution.)
- #58369 (Make the Entry API of HashMap<K, V> Sync and Send)
- #58861 (Expand where negative supertrait specific error is shown)
- #58877 (Suggest removal of `&` when borrowing macro and appropriate)
- #58883 (Suggest appropriate code for unused field when destructuring pattern)
- #58891 (Remove stray ` in the docs for the FromIterator implementation for Option)
- #58893 (race condition in thread local storage example)
- #58906 (Monomorphize generator field types for debuginfo)
- #58911 (Regression test for #58435.)
- #58912 (Regression test for #58813)
- #58916 (Fix release note problems noticed after merging.)
- #58918 (Regression test added for an async ICE.)
- #58921 (Add an explicit test for issue #50582)
- #58926 (Make the lifetime parameters of tcx consistent.)
- #58931 (Elide invalid method receiver error when it contains TyErr)
- #58940 (Remove JSBackend from config.toml)
- #58950 (Add self to mailmap)
- #58961 (On incorrect cfg literal/identifier, point at the right span)
- #58963 (libstd: implement Error::source for io::Error)
- #58970 (delay_span_bug in wfcheck's ty.lift_to_tcx unwrap)
- #58984 (Teach `-Z treat-err-as-bug` to take a number of errors to emit)
- #59007 (Add a test for invalid const arguments)
Failed merges:
- #58959 (Add release notes for PR #56243)
r? @ghost
`-Z treat-err-as-bug=0` will cause `rustc` to panic after the first
error is reported. `-Z treat-err-as-bug=2` will cause `rustc` to
panic after 3 errors have been reported.
Rename rustc_errors dependency in rust 2018 crates
I think this is a better solution than `use rustc_errors as errors` in `lib.rs` and `use crate::errors` in modules.
Related: rust-lang/cargo#5653
cc #58099
r? @Centril
Simplify `TokenStream` some more
These commits simplify `TokenStream`, remove `ThinTokenStream`, and avoid some clones. The end result is simpler code and a slight perf win on some benchmarks.
r? @petrochenkov
Vadim Petrochenkov