This commit adds a new tool, `build-manifest`, which is used to generate a
distribution manifest of all produced artifacts. This tool is intended to
replace the `build-rust-manifest.py` script that's currently located on the
buildmaster. The intention is that we'll have a builder which periodically:
* Downloads all artifacts for a commit
* Runs `./x.py dist hash-and-sign`. This will generate `sha256` and `asc` files
as well as TOML manifests.
* Upload all generated hashes and manifests to the directory the artifacts came
from.
* Upload *all* artifacts (tarballs and hashes and manifests) to an archived
location.
* If necessary, upload all artifacts to the main location.
This script is intended to just be the second step here where orchestrating
uploads and such will all happen externally from the build system itself.
This commit adds a new flag to the configure script,
`--enable-extended`, which is intended for specifying a desire to
compile the full suite of Rust tools such as Cargo, the RLS, etc. This
is also an indication that the build system should create combined
installers such as the pkg/exe/msi artifacts.
Currently the `--enable-extended` flag just indicates that combined
installers should be built, and Cargo is itself not compiled just yet
but rather only downloaded from its location. The intention here is to
quickly get to feature parity with the current release process and then
we can start improving it afterwards.
All new files in this PR inside `src/etc/installer` are copied from the
rust-packaging repository.
rustbuild: Actually don't build stage0 target rustc
This was attempted in #38853 but erroneously forgot one more case of where the
compiler was compiled. This commit fixes that up and adds a test to ensure this
doesn't sneak back in.
This was attempted in #38853 but erroneously forgot one more case of where the
compiler was compiled. This commit fixes that up and adds a test to ensure this
doesn't sneak back in.
This commit starts adding the infrastructure for uploading release artifacts
from AppVeyor/Travis on each commit. The idea is that eventually we'll upload a
full release to AppVeyor/Travis in accordance with plans [outlined earlier].
Right now this configures Travis/Appveyor to upload all tarballs in the `dist`
directory, and various images are updated to actually produce tarballs in these
directories. These are nowhere near ready to be actual release artifacts, but
this should allow us to play around with it and test it out. Once this commit
lands we should start seeing artifacts uploaded on each commit.
[outlined earlier]: https://internals.rust-lang.org/t/rust-ci-release-infrastructure-changes/4489
The `doc-book` and `doc-nomicon` steps accidentally depended on a rustbook
compiled by a cross-compiled compiler, which isn't necessary. Be sure to set the
`host` on these dependency edges to the build compiler to ensure that we're
always using a tool compiled for the host platform.
This was discovered trawling the build logs for the new dist bots and
discovering that they're building one too many compilers in stage0.
This commit optimizes the compile time for creating tarballs of cross-host
compilers and as a proof of concept adds two to the standard Travis matrix. Much
of this commit is further refactoring and refining of the `step.rs` definitions
along with the interpretation of `--target` and `--host` flags. This has gotten
confusing enough that I've also added a small test suite to
`src/bootstrap/step.rs` to ensure what we're doing works and doesn't regress.
After this commit when you execute:
./x.py dist --host $MY_HOST --target $MY_HOST
the build system will compile two compilers. The first is for the build platform
and the second is for the host platform. This second compiler is then packaged
up and placed into `build/dist` and is ready to go. With a fully cached LLVM and
docker image I was able to create a cross-host compiler in around 20 minutes
locally.
Eventually we plan to add a whole litany of cross-host entries to the Travis
matrix, but for now we're just adding a few before we eat up all the extra
capacity.
cc #38531
rustbuild: Compile all support tools in stage0
This commit changes all tools and such to get compiled in stage0, not in
later stages. The purpose of this commit is to cut down dependencies on later
stages for future modifications to the build system. Notably we're going to be
adding builders that produce a full suite of cross-compiled artifacts for a
particular host, and that shouldn't compile the `x86_64-unknown-linux-gnu`
compiler more than once. Currently dependencies on, for example, the error index
end up compiling the `x86_64-unknown-linux-gnu` compiler more than necessary.
As a result here we move many dependencies on these tools to being produced by a
stage0 compiler, not a stage1+ compiler. None of these tools actually need to be
staged at all, so they'll exhibit consistent behavior across the stages.
The android-copy-libs step is crucial for running tests on the Android target as
it copies necessary scripts and such to the emulator. We must run that before
running any tests there, but we erroneously only did it for compiletest test
suites!
This commit changes all tools and such to get compiled in stage0, not in
later stages. The purpose of this commit is to cut down dependencies on later
stages for future modifications to the build system. Notably we're going to be
adding builders that produce a full suite of cross-compiled artifacts for a
particular host, and that shouldn't compile the `x86_64-unknown-linux-gnu`
compiler more than once. Currently dependencies on, for example, the error index
end up compiling the `x86_64-unknown-linux-gnu` compiler more than necessary.
As a result here we move many dependencies on these tools to being produced by a
stage0 compiler, not a stage1+ compiler. None of these tools actually need to be
staged at all, so they'll exhibit consistent behavior across the stages.
rustbuild: Move pretty test suites to host-only
In an ongoing effort to optimize the runtime of the Android cross builder this
commit updates the pretty test suites to run only for host platforms, not for
target platforms as well. This means we'll still keep running all the suites but
we'll only run them for configured hosts, not for configured targets. This
notably means that we won't be running these suites on Android or musl targets,
for example.
rustbuild: Compile rustc twice, not thrice
This commit switches the rustbuild build system to compiling the
compiler twice for a normal bootstrap rather than the historical three
times.
Rust is a bootstrapped language which means that a previous version of
the compiler is used to build the next version of the compiler. Over
time, however, we change many parts of compiler artifacts such as the
metadata format, symbol names, etc. These changes make artifacts from
one compiler incompatible from another compiler. Consequently if a
compiler wants to be able to use some artifacts then it itself must have
compiled the artifacts.
Historically the rustc build system has achieved this by compiling the
compiler three times:
* An older compiler (stage0) is downloaded to kick off the chain.
* This compiler now compiles a new compiler (stage1)
* The stage1 compiler then compiles another compiler (stage2)
* Finally, the stage2 compiler needs libraries to link against, so it
compiles all the libraries again.
This entire process amounts in compiling the compiler three times.
Additionally, this process always guarantees that the Rust source tree
can compile itself because the stage2 compiler (created by a freshly
created compiler) would successfully compile itself again. This
property, ensuring Rust can compile itself, is quite important!
In general, though, this third compilation is not required for general
purpose development on the compiler. The third compiler (stage2) can
reuse the libraries that were created during the second compile. In
other words, the second compilation can produce both a compiler and the
libraries that compiler will use. These artifacts *must* be compatible
due to the way plugins work today anyway, and they were created by the
same source code so they *should* be compatible as well.
So given all that, this commit switches the default build process to
only compile the compiler two times, avoiding this third compilation
by copying artifacts from the previous one. Along the way a new entry in
the Travis matrix was also added to ensure that our full bootstrap can
succeed. This entry does not run tests, though, as it should not be
necessary.
To restore the old behavior of a full bootstrap (three compiles) you can
either pass:
./configure --enable-full-bootstrap
or if you're using config.toml:
[build]
full-bootstrap = true
Overall this will hopefully be an easy 33% win in build times of the
compiler. If we do 33% less work we should be 33% faster! This in turn
should affect cycle times and such on Travis and AppVeyor positively as
well as making it easier to work on the compiler itself.
This commit relegates all pretty tests to not get run by default and rather get
run as part of an "aux" test suite. This "aux" suite is renamed from the old
"cargotest" suite to just collect tests that don't need to run everywhere but
should at least pass on Unix/Windows.
In an ongoing effort to optimize the runtime of the Android cross builder this
commit updates the pretty test suites to run only for host platforms, not for
target platforms as well. This means we'll still keep running all the suites but
we'll only run them for configured hosts, not for configured targets. This
notably means that we won't be running these suites on Android or musl targets,
for example.
This commit switches the rustbuild build system to compiling the
compiler twice for a normal bootstrap rather than the historical three
times.
Rust is a bootstrapped language which means that a previous version of
the compiler is used to build the next version of the compiler. Over
time, however, we change many parts of compiler artifacts such as the
metadata format, symbol names, etc. These changes make artifacts from
one compiler incompatible from another compiler. Consequently if a
compiler wants to be able to use some artifacts then it itself must have
compiled the artifacts.
Historically the rustc build system has achieved this by compiling the
compiler three times:
* An older compiler (stage0) is downloaded to kick off the chain.
* This compiler now compiles a new compiler (stage1)
* The stage1 compiler then compiles another compiler (stage2)
* Finally, the stage2 compiler needs libraries to link against, so it
compiles all the libraries again.
This entire process amounts in compiling the compiler three times.
Additionally, this process always guarantees that the Rust source tree
can compile itself because the stage2 compiler (created by a freshly
created compiler) would successfully compile itself again. This
property, ensuring Rust can compile itself, is quite important!
In general, though, this third compilation is not required for general
purpose development on the compiler. The third compiler (stage2) can
reuse the libraries that were created during the second compile. In
other words, the second compilation can produce both a compiler and the
libraries that compiler will use. These artifacts *must* be compatible
due to the way plugins work today anyway, and they were created by the
same source code so they *should* be compatible as well.
So given all that, this commit switches the default build process to
only compile the compiler three times, avoiding this third compilation
by copying artifacts from the previous one. Along the way a new entry in
the Travis matrix was also added to ensure that our full bootstrap can
succeed. This entry does not run tests, though, as it should not be
necessary.
To restore the old behavior of a full bootstrap (three compiles) you can
either pass:
./configure --enable-full-bootstrap
or if you're using config.toml:
[build]
full-bootstrap = true
Overall this will hopefully be an easy 33% win in build times of the
compiler. If we do 33% less work we should be 33% faster! This in turn
should affect cycle times and such on Travis and AppVeyor positively as
well as making it easier to work on the compiler itself.
`arr` is the actual list of targets participating in steps construction,
but due to #38468 the hosts array now consists of only the build triple
for the `dist` steps, hence all non-build-triple targets are lost for
the host-only rules.
Fix this by using the original non-shadowed hosts array in `arr`
calculation. This should unbreak the nightly packaging process.
Fixes#38637.
The comment touched, as originally written, only concerned itself with
the `test` steps. However, since #38468 the `arr` variable actually has
gained an indirect relationship with the `dist` steps too. The comment
failed to convey the extra meaning, contributing to the misunderstanding
which eventually lead to #38637. Fix that by moving the comment into the
right place near the relevant condition, and properly documenting
`arr`'s purpose.
rustbuild: Run debuginfo tests by default
This fixes an accidental regression in rustbuild which stopped running debuginfo
tests by default. Here we flag the test suites as `default(true)` to ensure that
they're run on bots, for example.
rustbuild: Eliminate duplication of dist tarballs
Fixes#38365 by not constructing the duplicate steps in the first place, as suggested. The source package step is lacking the check as in other steps, so it is added as well.
Tested locally with the `alexcrichton/rust-slave-linux-cross:2016-11-11` container (with the build slave init replaced with no-op, of course).
r? @alexcrichton
rustbuild: Stop building docs for libtest by default
They cause the search index from the std docs to get overwritten just like #34800.
Part of #38319.
This fixes an accidental regression in rustbuild which stopped running debuginfo
tests by default. Here we flag the test suites as `default(true)` to ensure that
they're run on bots, for example.
We only want to package each host/target once for `dist`. The obvious
solution takes the form of step dependency, which is implemented at
least for the `dist-rustc` step. Unfortunately since the steps are
created from `hosts x targets` during planning and *not* de-duplicated
afterwards, the problem still persists.
We therefore move the check inside `plan()` instead, to avoid creating
the duplicate steps in the first place.
rustbuild: Package rust-mingw by default
This fixes the `make dist` step on MinGW to package the `rust-mingw` component
by default. This should hopefully be the last step in fixing nightlies.
rustbuild: Add cli option --keep-stage
This option is intended to be used like:
./x.py build --stage 1 --keep-stage 0
Which skips all stage 0 steps, so that stage 1 can be recompiled
directly (even if for example libcore has changes).
This is useful when working on `cfg(not(stage0))` parts of the
libraries or when re-running stage 1 tests in libraries in general.
Fixes#38326
This option is intended to be used like:
./x.py build --stage 1 --keep-stage 0
Which skips all stage 0 steps, so that stage 1 can be recompiled
directly (even if for example libcore has changes).
This is useful when working on `cfg(not(stage0))` parts of the
libraries, or when re-running stage 1 tests in libraries in general.
This commit implements the `distcheck` target for rustbuild which is only ever
run on our nightly bots. This essentially just creates a tarball, un-tars it,
and then runs a full build, validating that the release tarballs do indeed have
everything they need to build Rust.
rustbuild: Use src/rustc for assembled compilers
The `src/rustc` path is intended for assembling a compiler (e.g. the bare bones)
not actually compiling the whole compiler itself. This path was accidentally
getting hijacked to represent the whole compiler being compiled, so let's
redirect that elsewhere for that particular cargo project.
Closes#38039
This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2016-01-02.
[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368
This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
Closes#37858
The `src/rustc` path is intended for assembling a compiler (e.g. the bare bones)
not actually compiling the whole compiler itself. This path was accidentally
getting hijacked to represent the whole compiler being compiled, so let's
redirect that elsewhere for that particular cargo project.
Closes#38039
If a rule is flagged with `default(true)` then the pseudo-rule `default:foo`
will include that. If a rule is also flagged with `.host(true)`, however, then
the rule shouldn't be included for targets that aren't in the host array. This
adds a filter to ensure we don't pull in host rules for targets by accident.
Alex Crichton