Rollup merge of #152057 - haampie:hs/fix/bootstrap-respect-jobserver-protocol, r=clubby789

bootstrap: respect POSIX jobserver

When bootstrapping Rust, the `-j N` flag was passed to CMake, which was
then forwarded to Ninja. This prevents the jobserver from being used,
and as a result leads to oversubscription when Rust is just one of the
many packages built as part of a larger software stack.

Since Cargo and the Rust compiler have long supported the jobserver, it
would be good if also bootstrapping Rust itself would participate in the
protocol, leading to composable parallelism.

This change allows bootstrapping to respect an existing FIFO based
jobserver. Old pipe based jobservers are not supported, because they are
brittle: currently the Python scripts in bootstrap do not inherit the
file descriptors, but do pass on `MAKEFLAGS`, which has lead to errors
like "invalid file descriptor" in the past. Because Ninja only supports
FIFO based jobservers, it's better to focus on new jobservers only,
which shouldn't suffer from the "invalid file descriptor" issue.

In summary:

* Bootstrap Cargo passes `MAKEFLAGS` verbatim to subprocesses if it
  advertises a FIFO style jobserver, otherwise it unsets it. This ensures
  subprocesses respect the jobserver during bootstrap.
* `llvm.rs` does not pass `-j` to `cmake` when a FIFO style jobserver is
  set in `MAKEFLAGS`. This ensures Ninja respects the jobserver.
* Bootstrap Cargo no longer unsets `MKFLAGS`: from git blame, GNU Make
  considered it a historical artifact back in 1992, and it is never read
  by GNU Make, it's only set for backwards compatibility in case sub-Makefiles
  read it.

---

I've tested this with the [Spack package manager](https://github.com/spack/spack) starting the POSIX jobserver,
building node.js and rust in parallel with `-j16`, which looks like this:

```console
$ pstree 382710
python3─┬─python3
        └─python3─┬─python3─┬─make───make───6*[ccache───g++───cc1plus]
                  │         └─{python3}
                  └─python3─┬─python3.11───bootstrap───cmake───ninja-build───10*[sh───ccache───g++───cc1plus]
                            └─{python3}
```

As you can see there are 10 `g++` processes running for rust, and `6` for node.js, and
with a mix of `make` and `ninja` as build tools :).

(The only violation I see now is `rust-lld`, but I think that'll be fixed with the LLVM 23
release)

src/bootstrap/src/core/build_steps/llvm.rs

@@ -773,7 +773,15 @@ fn configure_cmake(
         .define("CMAKE_CXX_COMPILER", sanitize_cc(&cxx))
         .define("CMAKE_ASM_COMPILER", sanitize_cc(&cc));
 
-    cfg.build_arg("-j").build_arg(builder.jobs().to_string());
+    // If we are running under a FIFO jobserver, we should not pass -j to CMake; otherwise it
+    // overrides the jobserver settings and can lead to oversubscription.
+    let has_modern_jobserver = env::var("MAKEFLAGS")
+        .map(|flags| flags.contains("--jobserver-auth=fifo:"))
+        .unwrap_or(false);
+
+    if !has_modern_jobserver {
+        cfg.build_arg("-j").build_arg(builder.jobs().to_string());
+    }
     let mut cflags = ccflags.cflags.clone();
     // FIXME(madsmtm): Allow `cmake-rs` to select flags by itself by passing
     // our flags via `.cflag`/`.cxxflag` instead.

src/bootstrap/src/core/builder/cargo.rs

@@ -549,9 +549,17 @@ pub fn bare_cargo(
             assert_eq!(target, compiler.host);
         }
 
-        // Remove make-related flags to ensure Cargo can correctly set things up
-        cargo.env_remove("MAKEFLAGS");
-        cargo.env_remove("MFLAGS");
+        // Bootstrap only supports modern FIFO jobservers. Older pipe-based jobservers can run into
+        // "invalid file descriptor" errors, as the jobserver file descriptors are not inherited by
+        // scripts like bootstrap.py, while the environment variable is propagated. So, we pass
+        // MAKEFLAGS only if we detect a FIFO jobserver, otherwise we clear it.
+        let has_modern_jobserver = env::var("MAKEFLAGS")
+            .map(|flags| flags.contains("--jobserver-auth=fifo:"))
+            .unwrap_or(false);
+
+        if !has_modern_jobserver {
+            cargo.env_remove("MAKEFLAGS");
+        }
 
         cargo
     }