diff --git a/src/doc/rustc-dev-guide/src/tests/adding.md b/src/doc/rustc-dev-guide/src/tests/adding.md index 10483265c468..f1c40d7324f3 100644 --- a/src/doc/rustc-dev-guide/src/tests/adding.md +++ b/src/doc/rustc-dev-guide/src/tests/adding.md @@ -2,12 +2,13 @@ **In general, we expect every PR that fixes a bug in rustc to come accompanied by a regression test of some kind.** This test should fail in `main` but pass -after the PR. These tests are really useful for preventing us from repeating the +after the PR. +These tests are really useful for preventing us from repeating the mistakes of the past. -The first thing to decide is which kind of test to add. This will depend on the -nature of the change and what you want to exercise. Here are some rough -guidelines: +The first thing to decide is which kind of test to add. +This will depend on the nature of the change and what you want to exercise. +Here are some rough guidelines: - The majority of compiler tests are done with [compiletest]. - The majority of compiletest tests are [UI](ui.md) tests in the [`tests/ui`] @@ -24,14 +25,17 @@ guidelines: `library/${crate}tests/lib.rs`. - If the code is part of an isolated system, and you are not testing compiler output, consider using a [unit or integration test](intro.md#package-tests). -- Need to run rustdoc? Prefer a `rustdoc` or `rustdoc-ui` test. Occasionally - you'll need `rustdoc-js` as well. +- Need to run rustdoc? + Prefer a `rustdoc` or `rustdoc-ui` test. + Occasionally you'll need `rustdoc-js` as well. - Other compiletest test suites are generally used for special purposes: - - Need to run gdb or lldb? Use the `debuginfo` test suite. - - Need to inspect LLVM IR or MIR IR? Use the `codegen` or `mir-opt` test - suites. - - Need to inspect the resulting binary in some way? Or if all the other test - suites are too limited for your purposes? Then use `run-make`. + - Need to run gdb or lldb? + Use the `debuginfo` test suite. + - Need to inspect LLVM IR or MIR IR? + Use the `codegen` or `mir-opt` test suites. + - Need to inspect the resulting binary in some way? + Or if all the other test suites are too limited for your purposes? + Then use `run-make`. - Use `run-make-cargo` if you need to exercise in-tree `cargo` in conjunction with in-tree `rustc`. - Check out the [compiletest] chapter for more specialized test suites. @@ -47,14 +51,16 @@ modified several years later, how can we make it easier for them?). ## UI test walkthrough The following is a basic guide for creating a [UI test](ui.md), which is one of -the most common compiler tests. For this tutorial, we'll be adding a test for an -async error message. +the most common compiler tests. +For this tutorial, we'll be adding a test for an async error message. ### Step 1: Add a test file The first step is to create a Rust source file somewhere in the [`tests/ui`] -tree. When creating a test, do your best to find a good location and name (see -[Test organization](ui.md#test-organization) for more). Since naming is the +tree. +When creating a test, do your best to find a good location and name (see +[Test organization](ui.md#test-organization) for more). +Since naming is the hardest part of development, everything should be downhill from here! Let's place our async test at `tests/ui/async-await/await-without-async.rs`: @@ -77,19 +83,23 @@ A few things to notice about our test: - The top should start with a short comment that [explains what the test is for](#explanatory_comment). - The `//@ edition:2018` comment is called a [directive](directives.md) which - provides instructions to compiletest on how to build the test. Here we need to + provides instructions to compiletest on how to build the test. + Here we need to set the edition for `async` to work (the default is edition 2015). -- Following that is the source of the test. Try to keep it succinct and to the - point. This may require some effort if you are trying to minimize an example +- Following that is the source of the test. + Try to keep it succinct and to the point. + This may require some effort if you are trying to minimize an example from a bug report. -- We end this test with an empty `fn main` function. This is because the default +- We end this test with an empty `fn main` function. + This is because the default for UI tests is a `bin` crate-type, and we don't want the "main not found" - error in our test. Alternatively, you could add `#![crate_type="lib"]`. + error in our test. + Alternatively, you could add `#![crate_type="lib"]`. ### Step 2: Generate the expected output -The next step is to create the expected output snapshots from the compiler. This -can be done with the `--bless` option: +The next step is to create the expected output snapshots from the compiler. +This can be done with the `--bless` option: ```sh ./x test tests/ui/async-await/await-without-async.rs --bless @@ -99,8 +109,8 @@ This will build the compiler (if it hasn't already been built), compile the test, and place the output of the compiler in a file called `tests/ui/async-await/await-without-async.stderr`. -However, this step will fail! You should see an error message, something like -this: +However, this step will fail! +You should see an error message, something like this: > error: /rust/tests/ui/async-await/await-without-async.rs:7: unexpected > error: '7:10: 7:16: `await` is only allowed inside `async` functions and @@ -112,7 +122,8 @@ annotations in the source file. ### Step 3: Add error annotations Every error needs to be annotated with a comment in the source with the text of -the error. In this case, we can add the following comment to our test file: +the error. +In this case, we can add the following comment to our test file: ```rust,ignore fn bar() { @@ -136,7 +147,8 @@ It should now pass, yay! ### Step 4: Review the output Somewhat hand-in-hand with the previous step, you should inspect the `.stderr` -file that was created to see if it looks like how you expect. If you are adding +file that was created to see if it looks like how you expect. +If you are adding a new diagnostic message, now would be a good time to also consider how readable the message looks overall, particularly for people new to Rust. @@ -161,9 +173,9 @@ You may notice some things look a little different than the regular compiler output. - The `$DIR` removes the path information which will differ between systems. -- The `LL` values replace the line numbers. That helps avoid small changes in - the source from triggering large diffs. See the - [Normalization](ui.md#normalization) section for more. +- The `LL` values replace the line numbers. + That helps avoid small changes in the source from triggering large diffs. + See the [Normalization](ui.md#normalization) section for more. Around this stage, you may need to iterate over the last few steps a few times to tweak your test, re-bless the test, and re-review the output. @@ -171,8 +183,10 @@ to tweak your test, re-bless the test, and re-review the output. ### Step 5: Check other tests Sometimes when adding or changing a diagnostic message, this will affect other -tests in the test suite. The final step before posting a PR is to check if you -have affected anything else. Running the UI suite is usually a good start: +tests in the test suite. +The final step before posting a PR is to check if you +have affected anything else. +Running the UI suite is usually a good start: ```sh ./x test tests/ui @@ -188,16 +202,19 @@ You may also need to re-bless the output with the `--bless` flag. ## Comment explaining what the test is about The first comment of a test file should **summarize the point of the test**, and -highlight what is important about it. If there is an issue number associated -with the test, include the issue number. +highlight what is important about it. +If there is an issue number associated with the test, include the issue number. -This comment doesn't have to be super extensive. Just something like "Regression +This comment doesn't have to be super extensive. +Just something like "Regression test for #18060: match arms were matching in the wrong order." might already be enough. These comments are very useful to others later on when your test breaks, since -they often can highlight what the problem is. They are also useful if for some +they often can highlight what the problem is. +They are also useful if for some reason the tests need to be refactored, since they let others know which parts -of the test were important. Often a test must be rewritten because it no longer +of the test were important. +Often a test must be rewritten because it no longer tests what it was meant to test, and then it's useful to know what it *was* meant to test exactly.