articles
Thanks!
rustdoc, the Rust documentation generator, just got a massive improvement which greatly reduces the time required for doctests in this pull request. This blog post will explain what are doctests, how they work and finally how we were able to improve them to this point.
You can write documentation in your source code with ///. This documentation follows the commonmark specification. With it, you can add "code blocks" which can be used as code examples:
Run/// This function prints "hello". You can use it like this:
///
/// ```
/// hello();
/// ```
pub fn hello() {
print!("hello");
}
In here, the code example is not very useful, but now we all know what it is. Now if you run cargo test, you will realize that the crate's unit tests are run, but also doctests! All code examples in the documentation are tested. It allows for code examples to never be outdated but not only! Doctests even allow to ensure that a code doesn't compile or a few other things. You can find more information about doctests in the rustdoc book.
Now the interesting question is: how is rustdoc running these doctests? When running rustdoc in "test mode", it will go through all items, retrieve their documentation and extract all code examples from them.
Then, each doctest will be generated into its own file with some changes to allow them to compile. If there is no main function, we will add one for example. But if there are crate attributes, we need to be careful keeping them out of the main function, which requires some parsing, so we can generate the doctest correctly.
Once we have the doctest source code generated, we call the libtest crate and we provide it information about the doctest such as:
Then, each doctest is compiled on its own and (maybe) run.
Like mentioned previously, each doctest was compiled on its own. That's the main issue. If we take a look at where libcore time is spent when running its doctests:
total compile time: 775.204s
total runtime: 15.487sSo out of a total of 790 seconds, only 15 seconds are actually used for runtime (less than 2%!). 98% of the time is lost in compilation.
So needless to say, this compilation for each doctest is taking a huge amount of time. The obvious idea to improve things is to run as few compilations as possible. However, some doctests cannot be merged together because of various reasons:
compile_fail code block attribute)--nocapture option is used.--show-output option is used.test_harness code block attribute.As you can see, there are quite a lot of cases where a doctest cannot be merged with others. If a doctest cannot be merged, we compile it on its own as we did before and move on.
Now we enter more interesting cases: what happens if a merged doctest failed to compile? We cannot guess which doctest failed compilation, so in this case we revert to the previous behaviour and compile each doctest on its own so users will be able to know which one(s) failed to compile so they can fix them.
Originally, the feature was implemented so that all doctests were running in the same process using threads. However, this brought two issues:
exit function is called, everything stops.A good example to illustrate the non-thread-local statics can be displayed here:
Run/// ```
/// init_log();
/// log_info("hello");
/// ```
pub fn log_info(msg: &str) {
// ...
}
/// ```
/// init_log();
/// log_error("hello");
/// ```
pub fn log_error(msg: &str) {
// ...
}
fn init_log() {
static mut CONTEXT: Option<String> = None;
unsafe {
if CONTEXT.is_some() {
panic!("`init_log` must only be called once!");
}
CONTEXT = Some(String::new());
}
}
If you run those doctests in the same process, init_log will panic. To prevent this issue, each doctest runs in its own process.
I mentioned above that if we encountered crate-level attributes, we didn't put the doctest into the merged doctests. However there is one exception:
Run/// ```
/// #![allow(some_warning)]
/// let x = 12;
/// ```
pub fn foo() {}
For allow, warn, deny, we still keep the doctest and move the attribute with it.
Now comes the interesting part! What does the generated merged doctests source code look like? Let's first start with the "common" code:
Run#![allow(unused_extern_crates)]
#![allow(internal_features)]
#![feature(test)]
#![feature(rustc_attrs)]
// All `#![doc(test(attr(...)))]` attributes are added here.
extern crate test; // importing libtest
// Some utils.
mod _doctest_mod {
use std::sync::OnceLock;
use std::path::PathBuf;
pub static BINARY_PATH: OnceLock<PathBuf> = OnceLock::new();
pub const RUN_OPTION: &str = "*doctest-inner-test";
pub const BIN_OPTION: &str = "*doctest-bin-path";
#[allow(unused)]
pub fn doctest_path() -> Option<&'static PathBuf> {
self::BINARY_PATH.get()
}
#[allow(unused)]
pub fn doctest_runner(
bin: &std::path::Path,
test_nb: usize,
) -> Result<(), String> {
let out = std::process::Command::new(bin)
.arg(self::RUN_OPTION)
.arg(test_nb.to_string())
.output()
.expect("failed to run command");
if !out.status.success() {
Err(String::from_utf8_lossy(&out.stderr).to_string())
} else {
Ok(())
}
}
}
// Where the merged doctests are placed. We will see the code of a doctest
// just below.
// This attribute allows us to tell `rustc` that this `main` function is the
// REAL one. Remember: all doctests are wrapped in a `main` function! It makes
// it much easier for us to know what's the starting point of all doctests too.
#[rustc_main]
fn main() -> std::process::ExitCode {
// We list all doctests in this constant. In this case, it means that
// there is only one doctest.
const TESTS: [test::TestDescAndFn; 1] = [_doctest_0::TEST];
let bin_marker = std::ffi::OsStr::new(_doctest_mod::BIN_OPTION);
let test_marker = std::ffi::OsStr::new(_doctest_mod::RUN_OPTION);
// Test args provided to rustdoc, so none in this case.
let test_args = &[];
let mut args = std::env::args_os().skip(1);
while let Some(arg) = args.next() {
if arg == bin_marker {
let Some(binary) = args.next() else {
panic!(
"missing argument after `{}`", _doctest_mod::BIN_OPTION);
};
if crate::_doctest_mod::BINARY_PATH.set(binary.into()).is_err() {
panic!(
"`{}` option was used more than once",
bin_marker.to_string_lossy(),
);
}
// Starting to run `libtest` with all doctests.
return std::process::Termination::report(test::test_main(
test_args,
Vec::from(TESTS),
None,
));
} else if arg == test_marker {
let Some(nb_test) = args.next() else {
panic!(
"missing argument after `{}`",
_doctest_mod::RUN_OPTION,
);
};
if let Some(nb_test) = nb_test.to_str().and_then(|nb| {
nb.parse::<usize>().ok()
}) {
if let Some(test) = TESTS.get(nb_test) {
if let test::StaticTestFn(f) = test.testfn {
return std::process::Termination::report(f());
}
}
}
panic!("Unexpected value after `{}`", _doctest_mod::RUN_OPTION);
}
}
eprintln!("WARNING: No argument provided so doctests will be run in the same process");
std::process::Termination::report(test::test_main(
test_args,
Vec::from(TESTS),
None,
))
}
As you can see, the binary has two paths: one to call all doctests and the other to run specifically ONE doctest. It works as follows:
Now let's take this doctest (it's the full content of a file named foo.rs):
Run/// ```
/// let x = 12;
/// ```
Now let's see what its (formatted) generated code looks like:
Runmod _doctest_0 {
#![allow(some_warning)]
// This small piece of code is the actual doctest!
fn main() {
let x = 12;
}
#[rustc_test_marker = "foo.rs - foo (line 1)"]
pub const TEST: test::TestDescAndFn = test::TestDescAndFn::new_doctest(
"foo.rs - foo (line 1)", // test name
false, // is `ignore`
"foo.rs", // name of the file where the doctest is
1, // line where the doctest is located
false, // is `no_run`
false, // should panic
test::StaticTestFn(
|| {
// If we have a binary path, we call it with the ID of the
// current doctest (so `0` in here).
if let Some(bin_path) = crate::_doctest_mod::doctest_path() {
test::assert_test_result(
crate::_doctest_mod::doctest_runner(bin_path, 0),
)
// If we don't have a binary path, it means we can actually run
// the doctest, so let's go!
} else {
test::assert_test_result(self::main())
}
},
),
);
}
For two reasons: the first one is that in some extremely rare cases, we cannot detect that a doctests cannot be merged and thus will make it fail. For example if you want to test the location of your doctest with std::panic::Location. Obviously, it will not work if the doctest source code is moved into a merged doctest. It's for such cases that we added the new standalone code block attribute. More information about it in the rustdoc book.
The second reason is because it might change the output. Let's take this code as example:
Run/// ```
/// #![allow(some_warning)]
/// let x = 12;
/// ```
///
/// ```compile_fail
/// x
/// ```
pub fn foo() {}
Without the merged doctests feature, if you run doctest you will have:
running 2 tests
test foo.rs - foo (line 6) - compile fail ... ok
test foo.rs - foo (line 1) ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.06sBut with the feature enabled you will have:
running 1 test
test foo.rs - foo (line 1) ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
running 1 test
test foo.rs - foo (line 6) - compile fail ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03sIt's because there are now different doctests passes where doctests which cannot be merged with others are run at the end.
Just to show off, here are some comparison numbers:
| crate | before this feature | with this feature |
|---|---|---|
| sysinfo | 4.6s | 1.11s |
| geos | 3.95s | 0.45s |
| core | 54.08s | 13.5s (merged: 0.9s, standalone: 12.6s) |
| std | 12s | 3.56s (merged: 2.1s, standalone: 1.46s) |
| jiff | 4min39 | 7.2s |
I'm quite happy with this improvement... but it was only the first step! I hope to achieve other improvements, like the possibility to have useful doctests on binary crates by allowing them to have access to the crate API like unit tests. I have a few ideas on how to achieve it. We will see if we will be able to make it a reality!
All this was done thanks to enlightenment(?) from my cat:
