Instead of generating a standalone executable to test `unicode_data`,
generate normal tests in `coretests`. This ensures tests are always
generated, and will be run as part of the normal testsuite.
Also change the generated tests to loop over lookup tables, rather than
generating a separate `assert_eq!()` statement for every codepoint. The
old approach produced a massive (20,000 lines plus) file which took
minutes to compile!
This PR reverts RUST-147622 for several reasons:
1. The RUST-147622 PR would format the generated core library code using
an arbitrary `rustfmt` picked up from `PATH`, which will cause
hard-to-debug failures when the `rustfmt` used to format the
generated unicode data code versus the `rustfmt` used to format the
in-tree library code.
2. Previously, the `unicode-table-generator` tests were not run under CI
as part of `coretests`, and since for `x86_64-gnu-aux` job we run
library `coretests` with `miri`, the generated tests unfortunately
caused an unacceptably large Merge CI time regression from ~2 hours
to ~3.5 hours, making it the slowest Merge CI job (and thus the new
bottleneck).
3. This PR also has an unintended effect of causing a diagnostic
regression (RUST-148387), though that's mostly an edge case not
properly handled by `rustc` diagnostics.
Given that these are three distinct causes with non-trivial fixes, I'm
proposing to revert this PR to return us to baseline. This is not
prejudice against relanding the changes with these issues addressed, but
to alleviate time pressure to address these non-trivial issues.
Instead of generating a standalone executable to test `unicode_data`,
generate normal tests in `coretests`. This ensures tests are always
generated, and will be run as part of the normal testsuite.
Also change the generated tests to loop over lookup tables, rather than
generating a separate `assert_eq!()` statement for every codepoint. The
old approach produced a massive (20,000 lines plus) file which took
minutes to compile!
To make the final output code easier to see:
* Get rid of the unnecessary line-noise of `.unwrap()`ing calls to
`write!()` by moving the `.unwrap()` into a macro.
* Join consecutive `write!()` calls using a single multiline format
string.
* Replace `.push()` and `.push_str(format!())` with `write!()`.
* If after doing all of the above, there is only a single `write!()`
call in the function, just construct the string directly with
`format!()`.
Remove `#[rustfmt::skip]` from all the generated modules in
`unicode_data.rs`. This means we won't have to worry so much about
getting indetation and formatting right when generating code.
Exempted for now some tables which would be too big when formatted by
`rustfmt`.
Instead of `include_str!()`ing `range_search.rs`, just make it a normal
module under `core::unicode`. This means the same source code doesn't
have to be checked in twice, and it plays nicer with IDEs.
Also rename it to `rt` since it includes functions for searching the
bitsets and case conversion tables as well as the range
represesentation.
The ASCII subset of Unicode is fixed and will never change, so we don't
need to generate tables for it with every new Unicode version. This
saves a few bytes of static data and speeds up `char::is_control` and
`char::is_grapheme_extended` on ASCII inputs.
Since the table lookup functions exported from the `unicode` module will
give nonsensical errors on ASCII input (and in fact will panic in debug
mode), I had to add some private wrapper methods to `char` which check
for ASCII-ness first.
Hard-code `char::is_control`
Split off from https://github.com/rust-lang/rust/pull/145219
According to
https://www.unicode.org/policies/stability_policy.html#Property_Value, the set of codepoints in `Cc` will never change. So we can hard-code the patterns to match against instead of using a table.
This doesn't change the generated assembly, since the lookup table is small enough that[ LLVM is able to inline the whole search](https://godbolt.org/z/bG8dM37YG). But this does reduce the chance of regressions if LLVM's heuristics change in the future, and means less generated Rust code checked in to `unicode-data.rs`.
make char::is_whitespace unstably const
I am adding this to the existing https://github.com/rust-lang/rust/issues/132241 feature gate, since `is_digit` and `is_whitespace` seem similar enough that one can group them together.
some const cleanup: remove unnecessary attributes, add const-hack indications
I learned that we use `FIXME(const-hack)` on top of the "const-hack" label. That seems much better since it marks the right place in the code and moves around with the code. So I went through the PRs with that label and added appropriate FIXMEs in the code. IMO this means we can then remove the label -- Cc ``@rust-lang/wg-const-eval.``
I also noticed some const stability attributes that don't do anything useful, and removed them.
r? ``@fee1-dead``
`use` declarations will be reformatted in #125443. Very rarely, there is
a desire to force a group of `use` declarations together in a way that
auto-formatting will break up. E.g. when you want a single comment to
apply to a group. #126776 dealt with all of these in the codebase,
ensuring that no comments intended for multiple `use` declarations would
end up in the wrong place. But some people were unhappy with it.
This commit uses `#[rustfmt::skip]` to create these custom `use` groups
in an idiomatic way for a few of the cases changed in #126776. This
works because rustfmt treats any `use` item annotated with
`#[rustfmt::skip]` as a barrier and won't reorder other `use` items
around it.
There are some comments describing multiple subsequent `use` items. When
the big `use` reformatting happens some of these `use` items will be
reordered, possibly moving them away from the comment. With this
additional level of formatting it's not really feasible to have comments
of this type. This commit removes them in various ways:
- merging separate `use` items when appropriate;
- inserting blank lines between the comment and the first `use` item;
- outright deletion (for comments that are relatively low-value);
- adding a separate "top-level" comment.
We also entirely skip formatting for four library files that contain
nothing but `pub use` re-exports, where reordering would be painful.
This adds a dedicated check for the lower bound
(if it is outside of ASCII range) to the output of the `unicode-table-generator` tool.
This generalized the ASCII-only fast-path, but only for the `Grapheme_Extend` property for now,
as that is the only one with a lower bound outside of ASCII.
The indices are encoded as `u32`s in the range of invalid `char`s, so
that we know that if any mapping fails to parse as a `char` we should
use the value for lookup in the multi-table.
This avoids the second binary search in cases where a multi-`char`
mapping is needed.
Idea from @nikic
The majority of char case replacements are single char replacements,
so storing them as [char; 3] wastes a lot of space.
This commit splits the replacement tables for both `to_lower` and
`to_upper` into two separate tables, one with single-character mappings
and one with multi-character mappings.
This reduces the binary size for programs using all of these tables
with roughly 24K bytes.
Since ascii chars are already handled by a special case in the
`to_lower` and `to_upper` functions, there's no need to waste space on
them in the LUTs.
Jules Bertholet