This commit changes our Windows CI to always use the latest Python
interpreter available in the GHA tool cache instead of hardcoding Python
3.7.6. This is needed because occasionally GitHub bumps the installed
version, deleting the previous one.
Stop explicitly depending on python 2
This PR revises our previous policy of officially only supporting and testing with python 2 in the CI environment to instead test with python 3. It also changes the defaults to python 3 in our various scripts (usually, by way of `python` rather than `python3` to preserve compatibility with systems that do not have a python 3 available).
The effect of this is that we expect all new patches to support python 3 (and will test as such). We explicitly also expect that patches support python 2.7 as well -- and test as such, though only on one builder. This is intended as a temporary, though likely long-lived, measure to preserve compatibility while looking towards the future which is likely to be a python 3 only world. We do not at this point set a timeline for when we'll drop support for python 2.7; it's plausible that this is months or years into the future, depending on how quickly the ecosystem drops support and how painful it is for us to maintain that support over time.
Closes#65063 (as far as I can tell; please file explicit and separate issues or PRs if not).
There are some builders that are running out of disk space while
building the Docker images, such as arm-android. This moves and symlinks
/var/lib/docker to the /mnt partition on Linux GHA.
The /mnt mount point has 53GB of free disk space at the time of writing
this commit, so this moves the build there to avoid running out of disk
space during builds.
On Azure Pipeliones, the C: filesystem is huge with a lot of free space,
while D: is small. By default builds happened in D:, so we added a
script to symlink the big directories to C:, granting us more space.
Filesystem Size Used Avail Use%
C: 256G 143G 114G 56%
D: 14G 2.0G 13G 15%
On GitHub Actions instead C: is almost full, and we have a lot of free
space on D:, where the build happens.
Filesystem Size Used Avail Use%
C: 128G 114G 15G 89%
D: 56G 4.8G 52G 9%
This commit stops creating the symlink on GitHub Actions, fixing the out
of disk space errors we were seeing on some Windows builders.
Apparently the old path we were using for Python 2 on Windows was not
documented, and eventually got removed. This switches our CI to use the
correct path.
This helps us have enough disk space for our builders to be able to complete
successfully. For now, the choices are ad-hoc and 'definitely not needed'. This
should never fail the build, as everything our build needs should be inside
Docker.
In their infinite wisdom, Apple decided that (starting from macOS 10.15
onwards) /usr/include is not the location we should all search in for
our beloved C headers. Instead, we should look inside the extremely
intuitive and easily guessable new path:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include
Because why not.
Some environment variables (like DEPLOY or DEPLOY_ALT for dist builders,
or IMAGE on Linux builders) are set on a lot of builders, and whether
they should be present or not can be detected automatically based on the
builder name and the platform.
This commit simplifies the CI configuration by automatically setting
those environment variables.
This commit replaces the mirrors base URL contained in the MINGW_URL
with a CUSTOM_MINGW=1 environment variable. The mirrors base URL will be
fetched instead through the MIRRORS_BASE environment variable, defined
in src/ci/shared.sh.
ci: revert msys2 ca-certificates hack
The hack was added because upstream msys2 broke the ca-certificates package, but since then it has been fixed. This reverts CI to use the upstream package.
Part of #65767
Upload toolstates.json to rust-lang-ci2
This PR does two things:
* Following up with https://github.com/rust-lang/rust/pull/65202, it migrates deploying artifacts to CI in a script. Both uploading release artifacts and CPU stats were merged into the same script, designing it to be easily extended.
* Uploads the toolstate JSON to `rust-lang-ci2` along with the release artifacts, both for Linux and Windows. This is needed because @RalfJung wants to stop shipping MIRI when its tests are failing, and the toolstate repo doesn't have entries for each commit. Having the toolstate data (just for that specific commit) on `rust-lang-ci2` will simplify the code a lot.
r? @alexcrichton
cc @RalfJung
Before this commit toolstates.json was stored in /tmp and it wasn't
mounted outside the build container. That caused uploading the file in
the upload-artifacts task to fail, as the file was missing on the host.
Mounting /tmp/toolstates.json alone is not the best approach: if the
file is missing when the container is started the Docker engine will
create a *directory* named /tmp/toolstates.json.
The Docker issue could be solved by pre-creating an empty file named
/tmp/toolstates.json, but doing that could cause problems if bootstrap
fails to generate the file and the toolstate scripts receive an empty
JSON.
The approach I took in this commit is to instead mount a /tmp/toolstate
directory inside Docker, and create the toolstates.json file in it. That
also required a small bootstrap change to ensure the directory is
created if it's missing.
The hack was added because upstream msys2 broke the ca-certificates
package, but since then it has been fixed. This reverts CI to use the
upstream package.
Uploading the toolstate data for each commit will help our release
tooling understand which components are failing, to possibly skip
shipping broken tools to users.
Mateusz Mikuła