Keeping Rust projects' README.md code examples up-to-date

Because keeping documentation up-to-date is very important (and should be a must have!), we need to check if the examples are still valid after every new updates.

Luckily, rustdoc already makes such things very easy:

/// This a nice example:
///
/// ```
/// let x = 2;
/// assert!(x & 1 == 0);
/// ```
/// pub fn foo() {}

When you run cargo test, this example will be compiled and run (or not depending on the options, but that's another topic!). If any failure occurs, tests will fail and you will know right away where it failed so you can update it.

But now, let's say you have examples outside of your Rust code. How can they be tested?

Generate test files from markdown

To do so, the first thing I came across was the very nice skeptic crate.

The idea of this crate is to parse your markdown file(s), extract code blocks and then generate tests for all of them in a Rust file. To do so, you need to create a build.rs file:

Runextern crate skeptic;

fn main() { 
    // generates doc tests for `README.md`.
    skeptic::generate_doc_tests(&["README.md"]);
}

Unfortunately, you also have to mark which examples you want skeptic to extract with the "skt-template" tag. Let's take an example with our README.md file:

This is a nice README.

## What about a title?

And here comes the example I want to check:

```rust,skt-template
let x = 2;
assert!(x & 1 == 0);
```

And that's it! (Don't forget to add the skeptic dependency though but I trust you on that!)

Now every time you'll run a cargo build or a cargo test, test files will be generated.

Downsides

As is, I'm not a big fan because it comes with what I consider as a great downside: it rebuilds every time your sources are built, therefore your crate is rebuilt every time as well. Not ideal...

A solution I thought about was to only build those tests when running cargo test. So I added #[cfg(test)] in the build.rs as follows:

Run#[cfg(test)]
extern crate skeptic;

#[cfg(test)]
fn main() { 
    // generates doc tests for `README.md`.
    skeptic::generate_doc_tests(&["README.md"]);
}

#[cfg(not(test))]
fn main() {}

I then ran cargo build (twice to be sure it doesn't rebuild everything every time). Surprise: it worked! Time to check cargo test now! What could go wrong?

Well, the tests were not generated with cargo test either. After some search, it appeared that you can't check if you're running in a "test mode" in your build.rs file, only release and debug. It's still a debate going on.

So I had to go back to my first version of my build.rs. But again, it rebuilds everything every time. The only way to avoid it is to execute the generated binaries directly. Not ideal once again...

Time to look for alternatives!

doc include

There is actually a rustdoc feature that allows me to do exactly what I want: #[doc(include = "...")]!

Let's give it a try!

Run#[cfg(test)]
#[doc(include = "../README.md")]
fn foo() {}

Surprise! It doesn't work... What could have gone wrong? Ah right! We want rustdoc to test those code examples, therefore the cfg condition is invalid. Let's fix it:

Run#[cfg(rustdoc)]
#[doc(include = "../README.md")]
fn foo() {}

It worked this time! The tests are run just as expected. Everything's fine. We're at the end of the road! Let's push it and then, once CI is happy, we can merge.

CI failed.

Wait what?! Only nightly version passed: it's a nightly-only feature, it hasn't been stabilised yet. However, I need it to work on stable. I can't accept this solution. Too bad...

It's frustrating to feel so close to the solution but failing so close of the "all green" CI checks. However, I think I have a good lead to check this file. I just need to include it in another way. Wait, that reminds me something I did a while ago!

doc-comment crate

For the record, I wrote the doc-comment crate with antoyo a while ago to fix a standard doc issue Rust had with its primitive types: we generated documentation for them with a macro. The problem was that the code examples were always the same, whatever the type (so a i32 would have i8 examples). This wasn't really a big issue but it's way better now!

For the curious ones, you can take a look at the PR here.

We fixed this issue by using the not very known #[doc = ""] attribute. For those who doesn't know it, /// is a sugar syntax over this attribute. Anyway, time to use the macro we created: doc_comment!.

A little add to the Cargo.toml file:

[dependencies]
doc-comment = "0.1"

Then we use it as follows:

Run#[macro_use]
extern crate doc_comment;

doc_comment! {
    include_str!("../README.md"),
    fn foo() {}
};

This time, everything works. stable, beta and nightly all passed. We now test our code examples in our README.md file by just running cargo test. The executable isn't rebuilt at every cargo command. We reached our goal! But can we do better?

Actually yes, there is one last small improvement that can be done:

Rundoc_comment! {
    include_str!("../README.md")
};

If we don't provide a second argument to the macro, it generates an extern {} item. Big advantage: it never appears in the documentation generated by rustdoc.

This time, we've done it! Now, you don't have any more excuses to not test your examples in any external markdown files so please add this little check. If not for yourself, at least for your users!

EDIT: Thanks to a suggestion made by HeroicKatora on reddit, I added the doctest! macro the doc-comment crate. If you want to test your README.md file now, you can do it like this:

Rundoctest!("../README.md");