| 1 | # The `run-make` test suite |
| 2 | |
| 3 | The `run-make` test suite contains tests which are the most flexible out of all the [rust-lang/rust](https://github.com/rust-lang/rust) test suites. `run-make` tests can basically contain arbitrary code, and are supported by the [`run_make_support`] library. |
| 4 | |
| 5 | ## Infrastructure |
| 6 | |
| 7 | A `run-make` test is a test recipe source file `rmake.rs` accompanied by its parent directory (e.g. `tests/run-make/foo/rmake.rs` is the `foo` `run-make` test). |
| 8 | |
| 9 | The implementation for collecting and building the `rmake.rs` recipes are in [`src/tools/compiletest/src/runtest.rs`](../../src/tools/compiletest/src/runtest.rs), in `run_rmake_test`. |
| 10 | |
| 11 | The setup for the `rmake.rs` can be summarized as a 3-stage process: |
| 12 | |
| 13 | 1. First, we build the [`run_make_support`] library in bootstrap as a tool lib. |
| 14 | 2. Then, we compile the `rmake.rs` "recipe" linking the support library and its dependencies in, and provide a bunch of env vars. We setup a directory structure within `build/<target>/test/run-make/` |
| 15 | |
| 16 | ``` |
| 17 | <test-name>/ |
| 18 | rmake.exe # recipe binary |
| 19 | rmake_out/ # sources from test sources copied over |
| 20 | ``` |
| 21 | |
| 22 | and copy non-`rmake.rs` input support files over to `rmake_out/`. The support library is made available as an [*extern prelude*][extern_prelude]. |
| 23 | 3. Finally, we run the recipe binary and set `rmake_out/` as the working directory. |
| 24 | |
| 25 | ## External dependencies |
| 26 | |
| 27 | `compiletest` passes tool paths and target-specific flags to `rmake.rs` through |
| 28 | environment variables. Prefer using the helpers in [`run_make_support`] over |
| 29 | reading these variables directly. The helpers keep command construction |
| 30 | consistent across hosts and targets, and avoid relying on whichever tools happen |
| 31 | to be first in `PATH`. |
| 32 | |
| 33 | Commonly used helpers include: |
| 34 | |
| 35 | - `rustc()` and `rustdoc()` for the compiler and rustdoc under test, from |
| 36 | `RUSTC` and `RUSTDOC`. |
| 37 | - `cc()` and `cxx()` for the target C and C++ compilers, from `CC`/`CXX` plus |
| 38 | `CC_DEFAULT_FLAGS`/`CXX_DEFAULT_FLAGS`. |
| 39 | - `gcc()` for tests that specifically need `gcc`; unlike `cc()`, this assumes a |
| 40 | suitable `gcc` is available in `PATH` and does not add `CC_DEFAULT_FLAGS`. |
| 41 | - `llvm_ar()`, `llvm_nm()`, `llvm_objdump()`, `llvm_readobj()`, and similar |
| 42 | LLVM tools from `LLVM_BIN_DIR`; `llvm_filecheck()` uses `LLVM_FILECHECK`. |
| 43 | - `python_command()` for the Python interpreter selected by bootstrap, from |
| 44 | `PYTHON`. |
| 45 | - `clang()` for tests that explicitly require clang-based testing, from `CLANG`. |
| 46 | - `cargo()` for in-tree cargo, from `CARGO`; this is only available in the |
| 47 | `run-make-cargo` and `build-std` suites, not in plain `run-make`. |
| 48 | - `htmldocck()` for the in-tree `src/etc/htmldocck.py` script, invoked through |
| 49 | `python_command()`. |
| 50 | |
| 51 | Some of these variables are only set when the configured builder can provide the |
| 52 | corresponding tool. Tests that require optional tools or LLVM target components |
| 53 | should declare the matching compiletest directive, such as |
| 54 | `//@ needs-force-clang-based-tests` or `//@ needs-llvm-components: x86`, instead |
| 55 | of failing later in the recipe. If a test needs a tool that is not represented by |
| 56 | `run_make_support::external_deps`, add a small helper there rather than open |
| 57 | coding the lookup in individual tests. |
| 58 | |
| 59 | [`run_make_support`]: ../../src/tools/run-make-support |
| 60 | [extern_prelude]: https://doc.rust-lang.org/reference/names/preludes.html#extern-prelude |