`TokenStream` used to be a complex type, but it is now just a newtype
around a `Lrc<Vec<TreeAndJoint>>`. Currently it uses custom encoding
that discards the `IsJoint` and custom decoding that adds `NonJoint`
back in for every token tree. This requires building intermediate
`Vec<TokenTree>`s.
This commit makes `TokenStream` derive `Rustc{En,De}codable`. This
simplifies the code, and avoids the creation of the intermediate
vectors, saving up to 3% on various benchmarks. It also changes the AST
JSON output in one test.
Likewise for `NestedMetaItem::tokens()`. Also, add
`MetaItemKind::token_trees_and_joints()`, which `MetaItemKind::tokens()`
now calls.
This avoids some unnecessary `TokenTree` to `TokenStream` conversions,
and removes the need for the clumsy
`TokenStream::append_to_tree_and_joint_vec()`.
The current code has this impl:
```
impl<T: Into<TokenStream>> iter::FromIterator<T> for TokenStream
```
If given an `IntoIterator<Item = TokenTree>`, it will convert each individual
`TokenTree` to a `TokenStream` (at the cost of two allocations: a `Vec`
and an `Lrc`). It will then merge those `TokenStream`s into a single
`TokenStream`. This is inefficient.
This commit changes the impl to this less general one:
```
impl iter::FromIterator<TokenTree> for TokenStream
```
It collects the `TokenTree`s into a single `Vec` first and then converts that
to a `TokenStream` by wrapping it in a single `Lrc`. The previous generality
was unnecessary; no other code needs changing.
This change speeds up several benchmarks by up to 4%.
It means an allocation is required to create an empty `TokenStream`, but
all other operations are simpler and marginally faster due to not having
to check for `None`. Overall it simplifies the code for a negligible
performance effect.
The commit also removes `TokenStream::empty` by implementing `Default`,
which is now possible.
Currently, when two tokens must be glued together, this function duplicates
large chunks of the existing streams. This can cause quadratic behaviour.
This commit changes the function so that it overwrites the last token with a
glued token, which avoids the quadratic behaviour. This removes the need for
`TokenStreamBuilder::push_all_but_{first,last}_tree`.
The commit also restructures `push` somewhat, by removing
`TokenStream::{first_tree_and_joint,last_tree_if_joint}` in favour of more
pattern matching and some comments. This makes the code shorter, and in my
opinion, more readable.
Currently, this function creates a new empty stream, and then appends
the elements from each given stream onto that stream. This can cause
quadratic behaviour.
This commit changes the function so that it modifies the first stream
(which can be long) by extending it with the elements from the
subsequent streams (which are almost always short), which avoids the
quadratic behaviour.
This reduces by 12% the number of allocations done for a "clean
incremental" of `webrender_api`, which reduces the instruction count by
about 0.5%.
It also reduces instruction counts by up to 1.4% across a range of
rustc-perf benchmark runs.
Overhaul `syntax::fold::Folder`.
This PR changes `syntax::fold::Folder` from a functional style
(where most methods take a `T` and produce a new `T`) to a more
imperative style (where most methods take and modify a `&mut T`), and
renames it `syntax::mut_visit::MutVisitor`.
This makes the code faster and more concise.
This commit changes `syntax::fold::Folder` from a functional style
(where most methods take a `T` and produce a new `T`) to a more
imperative style (where most methods take and modify a `&mut T`), and
renames it `syntax::mut_visit::MutVisitor`.
The first benefit is speed. The functional style does not require any
reallocations, due to the use of `P::map` and
`MoveMap::move_{,flat_}map`. However, every field in the AST must be
overwritten; even those fields that are unchanged are overwritten with
the same value. This causes a lot of unnecessary memory writes. The
imperative style reduces instruction counts by 1--3% across a wide range
of workloads, particularly incremental workloads.
The second benefit is conciseness; the imperative style is usually more
concise. E.g. compare the old functional style:
```
fn fold_abc(&mut self, abc: ABC) {
ABC {
a: fold_a(abc.a),
b: fold_b(abc.b),
c: abc.c,
}
}
```
with the imperative style:
```
fn visit_abc(&mut self, ABC { a, b, c: _ }: &mut ABC) {
visit_a(a);
visit_b(b);
}
```
(The reductions get larger in more complex examples.)
Overall, the patch removes over 200 lines of code -- even though the new
code has more comments -- and a lot of the remaining lines have fewer
characters.
Some notes:
- The old style used methods called `fold_*`. The new style mostly uses
methods called `visit_*`, but there are a few methods that map a `T`
to something other than a `T`, which are called `flat_map_*` (`T` maps
to multiple `T`s) or `filter_map_*` (`T` maps to 0 or 1 `T`s).
- `move_map.rs`/`MoveMap`/`move_map`/`move_flat_map` are renamed
`map_in_place.rs`/`MapInPlace`/`map_in_place`/`flat_map_in_place` to
reflect their slightly changed signatures.
- Although this commit renames the `fold` module as `mut_visit`, it
keeps it in the `fold.rs` file, so as not to confuse git. The next
commit will rename the file.
This requires a pre-pass over the input streams. But that is cheap
compared to the quadratic blowup associated with reallocating the
accumulating vector on-the-fly.
Nicholas Nethercote