44d679b902
Auto merge of #124985 - dpaoliello:rebalance, r=Mark-Simulacrum
...
Reduce builder size of jobs that take less than an hour
The current longest build time is ~2hr for the `dist-x86_64-linux-alt` build. This is already on a 16-core builder, so we can't make it any faster (by throwing more hardware at it).
Given that overall build times will be at least 2hrs, we can reduce build costs by reducing the builder size for any job that takes less than 1hr since it will still complete before `dist-x86_64-linux-alt` does.
Note that scaling isn't linear, halving the core count increases end-to-end build times by about 25-50%. In [this sample build](https://github.com/rust-lang/rust/actions/runs/9037235792/usage?pr=124985 ) `arm-android` went from ~52m to 1h 5m and `dist-arm-linux` went from ~55m to 1h 17m (then failed due to missing metrics).
Current job builder sizes and times and proposed new sizes:
| Job | Size | Proposed | Run 1 | Run 2 | Run 3 | Run 4 |
|-----|------|----------|-------|-------|-------|-------|
| aarch64-gnu | - | | 1h 9m 1s | 1h 8m 47s | 1h 8m 45s | 1h 9m 6s |
| arm-android | 8c | 4c | 52m 32s | 52m 38s | 51m 30s | 53m 13s |
| armhf-gnu | 8c | 4c | 37m 30s | 37m 40s | 38m 41s | 37m 56s |
| dist-aarch64-linux | 8c | 4c | 57m 11s | 56m 48s | 55m 53s | 56m 19s |
| dist-android | 8c | 4c | 24m 37s | 25m 13s | 25m 15s | 24m 17s |
| dist-arm-linux | 16c | 8c | 53m 34s | 55m 11s | 56m 1s | 54m 29s |
| dist-armhf-linux | 8c | 4c | 42m 1s | 43m 32s | 43m 27s | 41m 55s |
| dist-armv7-linux | 8c | 4c | 44m 51s | 44m 35s | 43m 34s | 46m 2s |
| dist-i586-gnu-i586-i686-musl | 8c | 4c | 37m 59s | 37m 56s | 38m 4s | 38m 24s |
| dist-i686-linux | 8c | 4c | 52m 20s | 51m 3s | 52m 53s | 50m 38s |
| dist-loongarch64-linux | 8c | 4c | 40m 39s | 40m 20s | 41m 6s | 40m 44s |
| dist-ohos | 8c | 4c | 25m 5s | 24m 34s | 25m 18s | 23m 40s |
| dist-powerpc-linux | 8c | 4c | 42m 31s | 43m 53s | 42m 35s | 41m 56s |
| dist-powerpc64-linux | 8c | 4c | 42m 52s | 44m 36s | 45m 32s | 43m 51s |
| dist-powerpc64le-linux | 8c | 4c | 43m 41s | 44m 11s | 43m 2s | 44m 21s |
| dist-riscv64-linux | 8c | 4c | 41m 25s | 42m 41s | 41m 52s | 43m 47s |
| dist-s390x-linux | 8c | 4c | 46m 48s | 47m 18s | 47m 27s | 46m 49s |
| dist-various-1 | 8c | 4c | 42m 14s | 43m 20s | 43m 20s | 41m 41s |
| dist-various-2 | 8c | 4c | 36m 18s | 38m 15s | 37m 41s | 39m 28s |
| dist-x86_64-freebsd | 8c | 4c | 39m 21s | 39m 40s | 40m 1s | 40m 2s |
| dist-x86_64-illumos | 8c | 4c | 45m 35s | 46m 43s | 46m 2s | 46m 4s |
| dist-x86_64-linux | 16c | | 1h 53m 10s | 1h 51m 15s | 1h 52m 18s | 1h 52m 26s |
| dist-x86_64-linux-alt | 16c | | 2h 3m 33s | 2h 3m 31s | 2h 4m 12s | 2h 2m 21s |
| dist-x86_64-musl | 8c | | 1h 5m 42s | 1h 6m 13s | 1h 7m 49s | 1h 6m 6s |
| dist-x86_64-netbsd | 8c | 4c | 40m 4s | 39m 48s | 40m 16s | 39m 43s |
| i686-gnu | 8c | | 1h 13m 38s | 1h 13m 39s | 1h 13m 48s | 1h 13m 12s |
| i686-gnu-nopt | 8c | | 1h 17m 44s | 1h 18m 14s | 1h 19m 55s | 1h 18m 44s |
| mingw-check | 4c | | 28m 15s | 27m 39s | 28m 36s | 28m 38s |
| test-various | 8c | 4c | 37m 45s | 37m 17s | 38m 26s | 38m 11s |
| x86_64-gnu | 4c | | 1h 34m 1s | 1h 31m 51s | 1h 30m 35s | 1h 32m 53s |
| x86_64-gnu-stable | 4c | | 1h 28m 26s | 1h 28m 11s | 1h 29m 40s | 1h 46m 28s |
| x86_64-gnu-aux | 4c | | 1h 33m 32s | 1h 31m 57s | 1h 34m 8s | 1h 32m 57s |
| x86_64-gnu-integration | 8c | | 1h 22m 2s | 1h 20m 14s | 1h 19m 46s | 1h 21m 24s |
| x86_64-gnu-debug | 8c | 4c | 52m 41s | 53m 40s | 51m 51s | 56m 9s |
| x86_64-gnu-distcheck | 8c | | 1h 9m 14s | 1h 5m 31s | 1h 6m 29s | 1h 5m 50s |
| x86_64-gnu-llvm-18 | 8c | | 1h 39m 47s | 1h 37m 57s | 1h 38m 40s | 1h 37m 38s |
| x86_64-gnu-llvm-17 | 8c | | 1h 41m 50s | 1h 45m 43s | 1h 45m 4s | 1h 43m 4s |
| x86_64-gnu-nopt | 4c | | 1h 20m 42s | 1h 21m 38s | 1h 20m 4s | 1h 22m 11s |
| x86_64-gnu-tools | 8c | | 1h 5m 0s | 1h 5m 30s | 1h 3m 1s | 1h 3m 20s |
| dist-x86_64-apple | xl | | 1h 35m 1s | 1h 39m 57s | 2h 2m 31s | 1h 47m 37s |
| dist-apple-various | xl | | 1h 18m 54s | 1h 22m 31s | 1h 13m 19s | 1h 38m 18s |
| x86_64-apple-1 | xl | | 1h 32m 8s | 1h 40m 12s | 1h 51m 28s | 1h 40m 26s |
| x86_64-apple-2 | xl | | 1h 0m 32s | 1h 4m 5s | 1h 9m 0s | 1h 7m 17s |
| dist-aarch64-apple | m1 | | 1h 3m 9s | 1h 1m 14s | 1h 2m 6s | 1h 2m 24s |
| aarch64-apple | m1 | | 53m 38s | 1h 1m 5s | 1h 3m 15s | 1h 6m 11s |
| x86_64-msvc | 8c | | 1h 27m 48s | 1h 29m 38s | 1h 29m 55s | 1h 28m 4s |
| i686-msvc | 8c | | 1h 38m 28s | 1h 34m 7s | 1h 39m 19s | 1h 39m 28s |
| x86_64-msvc-ext | 8c | | 1h 44m 5s | 1h 38m 40s | 1h 45m 21s | 1h 44m 19s |
| i686-mingw | 8c | | 1h 49m 57s | 1h 45m 1s | 1h 52m 4s | 1h 51m 4s |
| x86_64-mingw | 8c | | 1h 44m 2s | 1h 37m 36s | 1h 49m 58s | 1h 47m 5s |
| dist-x86_64-msvc | 8c | | 1h 57m 14s | 1h 49m 43s | 1h 52m 53s | 1h 52m 35s |
| dist-i686-msvc | 8c | | 1h 8m 5s | 1h 4m 9s | 1h 9m 26s | 1h 12m 0s |
| dist-aarch64-msvc | 8c | | 1h 18m 40s | 1h 14m 4s | 1h 22m 1s | 1h 19m 6s |
| dist-i686-mingw | 8c | | 1h 15m 36s | 1h 14m 36s | 1h 16m 38s | 1h 16m 2s |
| dist-x86_64-mingw | 8c | | 1h 11m 54s | 1h 16m 12s | 1h 16m 54s | 1h 18m 2s |
| dist-x86_64-msvc-alt | 8c | | 1h 11m 17s | 1h 10m 0s | 1h 11m 8s | 1h 13m 14s |
2024-05-20 11:30:58 +00:00
7b6a3d029b
CI: fix toolstate publishing
2024-05-15 15:48:52 +02:00
1b5d91b446
Reduce size of builders that take less than an hour
2024-05-10 14:09:39 -07:00
a6e87c546d
Auto merge of #124850 - dpaoliello:clang2022, r=Kobzol
...
Upgrade pre-built Clang used in MSVC and MacOS builds, move MSVC builds to Server 2022
Fixes #92948
Example working MacOS and Windows builds: <https://github.com/rust-lang/rust/actions/runs/8989360201 >
There is a [bug in Clang 18](https://github.com/llvm/llvm-project/pull/81849 ) that causes issues when building for Arm64 in later parts of the build (specifically `libgit2`). As a workaround, we will still use the pre-built Clang to build LLVM but will use MSVC for the rest of the Arm64 build.
2024-05-10 00:04:22 +00:00
5212e07a4e
Upgrade the version of Clang used in the build, move MSVC builds to Server 2022
2024-05-09 12:18:09 -07:00
b3e9060178
CI: fix auto builds and make sure that we always have at least a single CI job
2024-05-05 14:06:23 +02:00
bf8bcc4c2d
Address review comments
2024-05-05 14:06:23 +02:00
e2e280610d
Make the regex more robust
2024-05-05 14:06:23 +02:00
74dbe8ae9e
Parse try build CI job name from commit message
2024-05-05 14:06:23 +02:00
6778ecf960
Use sum type for WorkflowRunType
2024-05-05 14:06:23 +02:00
6d721dd0b8
Auto merge of #124562 - dpaoliello:mingwserver2022, r=Mark-Simulacrum
...
Use Server 2022 in CI for mingw jobs
From my [previous testing](https://github.com/rust-lang/rust/issues/92948#issuecomment-2077771347 ), there is nothing blocking using Server 2022 for the Windows MinGW jobs.
This change:
* Adds new `os` entries for Server 2022.
* Renamed the Server 2019 `os` entries (as we'll be removing these over time).
* Moves the mingw jobs to Server 2022 and leaves the msvc jobs on Server 2019.
Partially fixes #92948
2024-05-04 07:49:50 +00:00
22456309ea
CI: remove env-x86_64-apple-tests YAML anchor
2024-05-02 21:23:52 +02:00
bc294b857e
Use Server 2022 in CI for mingw jobs
2024-04-30 09:50:55 -07:00
b194d5ce44
Output run_type from the matrix calculation job
2024-04-29 21:33:17 +02:00
24bf3594a5
Rename JobType to WorkflowRunType
2024-04-29 21:33:17 +02:00
0b1ecf1d8d
Remove the expand-yaml-anchors tool
2024-04-29 21:33:17 +02:00
1ca92c0857
Replace dynamically generated ci.yml file with the original template
2024-04-29 21:32:35 +02:00
23341500a6
Unify outcome jobs
2024-04-29 21:28:52 +02:00
a700897d5a
Add missing checkout step
2024-04-27 15:48:35 +02:00
163ea3328b
Move TOOLSTATE_REPO to top-level env
2024-04-27 11:36:42 +02:00
e82c28debc
Remove master push hook
2024-04-25 11:06:27 +02:00
86da0e758f
Remove useless condition
...
Neither `env.DEPLOY` nor `env.DEPLOY_ALT` should be present in this job.
2024-04-25 10:47:19 +02:00
744dc2aa1d
Inline base-ci-job YAML anchor and remove unused anchors
2024-04-25 10:40:09 +02:00
144bdf5bf1
Remove master CI job
2024-04-25 10:40:09 +02:00
c36d78ed64
Turn CI_ONLY_WHEN_CHANNEL from an environment variable to a job attribute
2024-04-24 12:44:13 +02:00
c59bc9a06a
Fix documentation
2024-04-24 12:15:17 +02:00
4942a35f24
Remove step YAML anchor and env.SKIP_JOB
2024-04-24 11:12:39 +02:00
246ee53637
Remove should-skip-this.sh
2024-04-24 11:12:17 +02:00
e25735908f
Skip jobs based on the active channel in Python
2024-04-24 11:11:16 +02:00
2a7fe14acd
Use a type alias for a CI job
2024-04-24 11:09:42 +02:00
2632c6139d
Fix empty job matrix
2024-04-23 09:01:51 +02:00
162892b030
Make separate envs for try and auto builds
2024-04-23 08:55:35 +02:00
2f2d59dd06
Fix image field in auto jobs
2024-04-23 08:53:07 +02:00
c251abc7ec
Load GitHub context from environment variables
2024-04-23 08:52:28 +02:00
d2059aef27
Move auto jobs to calculate-job-matrix.py
2024-04-20 09:54:47 +02:00
6f0ff0b03a
Refactor calculate-job-matrix.py
2024-04-20 09:21:55 +02:00
7a90679e28
Perform PR and try builds dynamically
2024-04-20 09:07:11 +02:00
9fec43ddf1
Move calculate-job-matrix.py to ci/github-actions
2024-04-20 09:06:04 +02:00
ccfcd950b3
Auto merge of #123451 - Kobzol:arbitrary-try-build, r=pietroalbini
...
CI: add a script for dynamically computing CI job matrix
It would be great if was easier to run specific CI workflows locally, and also to allow us to spawn a specific CI workflow by bors, to enable running arbitrary try builds. See discussion [here](https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/CI.20workflows.20refactoring ).
This PR is a first step in that direction.
- Moves the definition of CI runners and (for now) PR jobs into a separate `jobs.yml` file.
- Adds a simple Python script that reads the file, decides which jobs should be active for the current CI workflow, and prints them as JSON to their output.
- The PR job then reads this output and generates its job matrix based on it.
By moving the job definitions from `ci.yml` into a separate file, we can handle it programmatically, which should make it easier to both do local execution of CI jobs and also to do arbitrary try builds.
2024-04-15 21:28:25 +00:00
72104e245b
Add comments
2024-04-15 21:21:45 +02:00
c22c81cbc1
Generate CI job matrix for PR jobs in Python
2024-04-04 14:45:02 +02:00
ad75760637
CI: Redirect stderr to stdout to order GHA logs
2024-04-02 12:08:47 +02:00
24a0d7daff
Update upload-artifact to v4
...
This contains a breaking change around artifact merging no longer being
done. This was not relied on, so it's fine.
2024-03-23 17:06:02 +01:00
29430554f6
Update the minimum external LLVM to 17
2024-03-17 10:11:04 -07:00
5aad51d015
Auto merge of #112268 - lqd:osx-1cgu, r=Kobzol
...
Build `rustc` with 1CGU on `x86_64-apple-darwin`
Distribute `x86_64-apple-darwin` artifacts built with `rust.codegen-units=1`, like we already do on Linux.
1) effect on code size on `x86_64-apple-darwin`: it's a 11.14% reduction on `librustc_driver.dylib`
- before, [`41d97c8a5dea2731b0e56fe97cd7cb79e21cff79`](https://ci-artifacts.rust-lang.org/rustc-builds/41d97c8a5dea2731b0e56fe97cd7cb79e21cff79/rustc-nightly-x86_64-apple-darwin.tar.xz ): 161232048
- after, [`7549dbdc09f0c4f6cc84002ac03081828054784b`](https://ci-artifacts.rust-lang.org/rustc-builds/7549dbdc09f0c4f6cc84002ac03081828054784b/rustc-nightly-x86_64-apple-darwin.tar.xz ): 143256928
2) time it took on CI:
- the [first `try` build](https://github.com/rust-lang-ci/rust/actions/runs/8155512915/job/22291187124 ) took: 1h 33m
- the [second `try` build](https://github.com/rust-lang-ci/rust/actions/runs/8157057880/job/22295839911 ) took: 1h 45m
3) most recent perf results on (a noisy) x64 mac are [here](https://github.com/rust-lang/rust/pull/112268#issuecomment-1575635073 ).
Related tracking issue for build configuration: #103595
2024-03-12 02:02:32 +00:00
4a0cc881dc
Auto merge of #112267 - lqd:win-1cgu, r=Kobzol
...
Build `rustc` with 1CGU on `x86_64-pc-windows-msvc`
Distribute `x86_64-pc-windows-msvc` artifacts built with `rust.codegen-units=1`, like we already do on Linux.
1) effect on code size on `x86_64-pc-windows-msvc`: it's a 3.67% reduction on `rustc_driver.dll`
- before, [`41d97c8a5dea2731b0e56fe97cd7cb79e21cff79`](https://ci-artifacts.rust-lang.org/rustc-builds/41d97c8a5dea2731b0e56fe97cd7cb79e21cff79/rustc-nightly-x86_64-pc-windows-msvc.tar.xz ): 137605632
- after, [`704aaa875e4acccc973cbe4579e66afbac425691`](https://ci-artifacts.rust-lang.org/rustc-builds/704aaa875e4acccc973cbe4579e66afbac425691/rustc-nightly-x86_64-pc-windows-msvc.tar.xz ): 132551680
2) time it took on CI
- the [first `try` build](https://github.com/rust-lang-ci/rust/actions/runs/8155647651/job/22291592507 ) took: 1h 31m
- the [second `try` build](https://github.com/rust-lang-ci/rust/actions/runs/8157043594/job/22295790552 ) took: 1h 32m
3) most recent perf results:
- on a slightly noisy desktop [here](https://github.com/rust-lang/rust/pull/112267#issuecomment-1575758155 )
- ChrisDenton's results [here](https://github.com/rust-lang/rust/pull/112267#issuecomment-1583295338 )
Related tracking issue for build configuration: #103595
2024-03-11 21:40:34 +00:00
bef1cd80d8
ci: add a runner for vanilla LLVM 18
2024-03-07 18:30:04 -08:00
7d6969bb28
build rustc with 1CGU on x86_64-pc-windows-msvc
2024-03-05 16:50:48 +00:00
b36fa70917
build rustc with 1CGU on x86_64-apple-darwin
2024-03-05 16:42:42 +00:00
5674f0662f
Promote OpenHarmony targets to tier 2
...
MCP: rust-lang/compiler-team#719
2024-03-02 18:11:25 +00:00
bors