[CI] Upload average CPU utilization of CI jobs to DataDog
This PR adds a new CI step that uploads the average CPU utilization of the current GH job to Datadog. I want to add more metrics in follow-up PRs.
r? `@jdno`
try-job: dist-i686-msvc
try-job: aarch64-apple
try-job: x86_64-gnu-llvm-18
This partially reverts commit fe7c97c2e7.
I kept a mv, not a cp, for the one that shuffles major artifacts around,
because the size of those artifacts are big enough to matter, sometimes.
I don't think the diagnostic info will be that heavy, by comparison.
CI: add a script for dynamically computing CI job matrix
It would be great if was easier to run specific CI workflows locally, and also to allow us to spawn a specific CI workflow by bors, to enable running arbitrary try builds. See discussion [here](https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/CI.20workflows.20refactoring).
This PR is a first step in that direction.
- Moves the definition of CI runners and (for now) PR jobs into a separate `jobs.yml` file.
- Adds a simple Python script that reads the file, decides which jobs should be active for the current CI workflow, and prints them as JSON to their output.
- The PR job then reads this output and generates its job matrix based on it.
By moving the job definitions from `ci.yml` into a separate file, we can handle it programmatically, which should make it easier to both do local execution of CI jobs and also to do arbitrary try builds.
Errors started showing up, and I read somewhere that this might be
because of old ninja versions. This ninja version is indeed *ancient*.
```
multiple outputs aren't (yet?) supported by depslog; bring this up on the mailing list if it affects you
```
Improvements to building and CI for mingw/msys
I was getting error messages when trying to follow the build instructions the mingw build for Rust, and managed to track the issue down to an incomparability of Rust's bootstrap program with MSYS2's version of git. Essentially, the problem is that MSYS2's git works in emulated unix-y paths, but bootstrap expects a Windows path. I found a workaround for this by using relative paths instead of absolute paths.
Along with that fix, this PR also updates the build instructions for MinGW to be compatible with modern versions of MSYS2, and some changes to CI to make sure that MSYS2's version of git is tested. In particular, I'm suggesting using the [MSYS2 github action](https://github.com/marketplace/actions/setup-msys2) specially made for this purpose, which is much less hacky than the old approach and gives us more control of what packages are installed. I also cleaned up as many alternate versions of key tools as I could find from PATH, to avoid accidental usage, and cleaned up some abuses of the `CUSTOM_MINGW` environment variable.
This fixes https://github.com/rust-lang/rust/issues/105696 and fixes https://github.com/rust-lang/rust/issues/117567
Allowing the Xcode version to "float" based on whatever default GitHub
selects creates an unreliable environment. When GitHub changes the
default, we can have multiple jobs in the same run using different
versions as it rolls out across machines. It can also cause oscillation
between runs as different machines are used. It also causes
unpredictable timing when the updates happen.
This change helps ensure that the version that is used is pinned. The
downside is that it requires manually bumping the version, and the risk
that if we take too long, older Xcodes will be removed and that will
break the build.
Jakub Beránek