Today, I'd like to introduce to you a new NodeJS framework called browser-ui-test. You can find it on github and on npmjs. Its default behaviour is to compare screenshots after a list of actions and compare them to detect regressions. But it's much more than that!
Before presenting the framework, let's talk about what it has to do with
rustdoc is a software between worlds. It allows to test codes inside documentation and convert AST/Markdown into HTML. As you can guess, multiple test suites exist. But that's not all! The
rustdoc search engine also needs to be tested. So to sum this up, we need to test:
That's already a lot and I didn't speak about testing the terminal UI, options check and a few other things on purpose. We'll only focus on "generated HTML interactions".
Through updates and new features, the
A lot of approaches were tested. I decided to go for screenshots comparison. It has the big advantage to fail if any pixel isn't where it should be, but it has also a big disavantadge of failing if any pixel is different. To put it simply, if you are writing a brand new website, this is definitely not a good strategy. However, if you have a somewhat stable UI with few changes, then it's definitely worth trying it. Therefore,
rustdoc is a perfect match.
Amongst the framework I checked,
selenium was a very interesting one. Unfortunately, you still have to write a lot of "code" which needs to be maintained. Same problem with
puppeteer. However, I didn't get rid of those completely since
browser-ui-test is using
Now that I picked a way of testing, time to start implementing it! You can see the project here. At first, I made an all-in: the testing code was directly into the server itself. I then decided to write a small DSL (domain specific language) to make writing tests faster and easier to maintain. The
goml language was born! It looks just like this:
Pretty simple. However, the implementation was very basic and lacking and error-prone. To say it more simply: it was bad but more than good enough to comfort me that my approach was the right one. So much that I started using it for other projects. It was time to go to the second step and outsource it. It made me discover the
npm world and how to publish something there. browser-ui-test was born!
One of the first things I did was to reimplement the whole parsing to handle it way more generically instead of having all commands implement their own. Funnily enough, the code grew bigger and yet became much easier to read.
I also added more options (used when running browser-ui-test binary) and commands, but not only! With the framework growing, I had to add tests as well. Instead of using big ones, I wrote a small class to count errors and show where they happen and used it to test everything present in the framework: parsers, commands and options.
Last step: adding documentation. Would be funny if the framework used to test a documentation tool wasn't documented itself, right? Irony...
Once everything was done and reached a satisfying enough state, I released it then updated the test-rust-docs-ui repository to use it.
Now, time to present the final result!
We talked briefly about the
goml language. Let's take a concrete example:
screenshot: false goto: file://|CURRENT_DIR|/|DOC_PATH|/basic.html assert: ("#button", "Go somewhere else!") text: ("#button", "hello") assert: ("#button", "hello")
Let's explain line by line what's happening in there:
This line disables the screenshot comparison at the end of the script.
This line uses variables (
DOC_PATH) defined outside of the script. The
goto command goes to the specified file/url.
assert: ("#button", "Go somewhere else!")
assert is used to check a few things. If the condition is false, the script fails (as expected of an assertion!). In this case, we check if the DOM element with the "button" id (written "#button" in CSS) has the text "Go somewhere else!". It can also checks other things such as the number of occurences of an element, check an element's attributes and CSS properties, etc...
text: ("#button", "hello")
This command sets the "#button" element's text to "hello".
assert: ("#button", "hello")
And finally, it checks if the text has been updated as expected.
With this, you got a basic idea of how it works. More information is available directly into the project's README.
Another interesting thing about moving to this framework is that now, it's even possible to test the
rustdoc search engine, which would allow to reduce once again the number of test suites. In any case, this simplification really did some good and will make my work easier in the future.