New NodeJS framework to improve rustdoc UI tests

Hi everyone!

Today, I'd like to introduce to you a new NodeJS framework called browser-ui-test. You can find it on github and on npmjs. Its default behaviour is to compare screenshots after a list of actions and compare them to detect regressions. But it's much more than that!

Before presenting the framework, let's talk about what it has to do with rustdoc testing.

Testing issues

rustdoc is a software between worlds. It allows to test codes inside documentation and convert AST/Markdown into HTML. As you can guess, multiple test suites exist. But that's not all! The rustdoc search engine also needs to be tested. So to sum this up, we need to test:

• HTML generation
• generated HTML interactions
• code blocks detection into markdown
• code blocks execution
• search engine execution and results

That's already a lot and I didn't speak about testing the terminal UI, options check and a few other things on purpose. We'll only focus on "generated HTML interactions".

Through updates and new features, the rustdoc UI changed. A lot. Obviously, regressions appeared. I guess most people know than testing UIs is a whole kind of hell. It has different support for HTML, CSS and Javascript, different screen sizes and different devices. Trying to go through all this is very complicated and most websites just drop support for old browsers. Funny enough, I had the most issues with safari on iPhone and not with the infamous Internet Exporer. Times change I guess!

A lot of approaches were tested. I decided to go for screenshots comparison. It has the big advantage to fail if any pixel isn't where it should be, but it has also a big disavantadge of failing if any pixel is different. To put it simply, if you are writing a brand new website, this is definitely not a good strategy. However, if you have a somewhat stable UI with few changes, then it's definitely worth trying it. Therefore, rustdoc is a perfect match.

Why not selenium or equivalent?

Amongst the framework I checked, selenium was a very interesting one. Unfortunately, you still have to write a lot of "code" which needs to be maintained. Same problem with puppeteer. However, I didn't get rid of those completely since browser-ui-test is using puppeteer under the hood. It reads commands and convert the code into Javascript which uses puppeteer.

Birth of browser-ui-test

Now that I picked a way of testing, time to start implementing it! You can see the project here. At first, I made an all-in: the testing code was directly into the server itself. I then decided to write a small DSL (domain specific language) to make writing tests faster and easier to maintain. The goml language was born! It looks just like this:

command: arguments

Pretty simple. However, the implementation was very basic and lacking and error-prone. To say it more simply: it was bad but more than good enough to comfort me that my approach was the right one. So much that I started using it for other projects. It was time to go to the second step and outsource it. It made me discover the npm world and how to publish something there. browser-ui-test was born!

Growing and improving

One of the first things I did was to reimplement the whole parsing to handle it way more generically instead of having all commands implement their own. Funnily enough, the code grew bigger and yet became much easier to read.

I also added more options (used when running browser-ui-test binary) and commands, but not only! With the framework growing, I had to add tests as well. Instead of using big ones, I wrote a small class to count errors and show where they happen and used it to test everything present in the framework: parsers, commands and options.

Last step: adding documentation. Would be funny if the framework used to test a documentation tool wasn't documented itself, right? Irony...

Once everything was done and reached a satisfying enough state, I released it then updated the test-rust-docs-ui repository to use it.

Now, time to present the final result!

browser-ui-test

We talked briefly about the goml language. Let's take a concrete example:

screenshot: false
goto: file://|CURRENT_DIR|/|DOC_PATH|/basic.html
assert: ("#button", "Go somewhere else!")
text: ("#button", "hello")
assert: ("#button", "hello")

Let's explain line by line what's happening in there:

screenshot: false

This line disables the screenshot comparison at the end of the script.

goto: file://|CURRENT_DIR|/|DOC_PATH|/basic.html

This line uses variables (CURRENT_DIR and DOC_PATH) defined outside of the script. The goto command goes to the specified file/url.

assert: ("#button", "Go somewhere else!")

assert is used to check a few things. If the condition is false, the script fails (as expected of an assertion!). In this case, we check if the DOM element with the "button" id (written "#button" in CSS) has the text "Go somewhere else!". It can also checks other things such as the number of occurences of an element, check an element's attributes and CSS properties, etc...

text: ("#button", "hello")

This command sets the "#button" element's text to "hello".

assert: ("#button", "hello")

And finally, it checks if the text has been updated as expected.

With this, you got a basic idea of how it works. More information is available directly into the project's README.

Back to rustdoc

Another interesting thing about moving to this framework is that now, it's even possible to test the rustdoc search engine, which would allow to reduce once again the number of test suites. In any case, this simplification really did some good and will make my work easier in the future.