How we do automated quality assurance on more than a hundred WordPress theme demos
Theme quality is something that’s not negotiable here at CSSIgniter. For over five years now we carry the promise of new theme releases month to month for our customers and we focus all of our efforts in maintaining the high quality standard we’ve set for them ever since we began this awesome endeavour. Five years in now and except for quality we’ve now found ourselves to also have to deal with another (a bit spookier) word: quantity. At the moment of this writing, our theme catalogue lists 89 premium and free WordPress themes, not counting the ones on Themeforest or our premium plugins or awesome Elementor landing pages.
Counting everything, we’re racking up about 150 (!) WordPress theme, plugin, and landing page products all with their own user demos and marketing pages.
Our theme and plugin demos are really important to us. They are quite literally the storefront of our business and the first point of friction that any of our users will have with our products. Anyone in the website development business can tell you how a website is a bit of a living organism that needs serious maintenance; browsers come and go, specifications change, external content on the web (i.e. YouTube videos) is volatile and our theme demos (or any website) have to be prepared and adapted to all such changes.
As our product family grew, checking a hundred and fifty product demos (and all pages within a demo!) every day, week, or month, became counter-productive, not practical and inaccurate. There are a lot of moving parts on a demo that need to be checked: External imagery, broken images (or anything that can return with a 404 error), plugin incompatibilities causing all kinds of errors, missing embeds (we’re using a lot of YouTube and Vimeo videos on our demos), and the list goes on.
To help us solve the problem of wasting time manually checking each and every demo page in our effort to ensure that all our content is intact we implemented a fully automated and abstract end-to-end testing solution which takes away a lot of the pain, especially for mundane tasks. Currently we’re running about 3,500 automated tests (yeap, that’s three and a half thousand – with more to come) across all our products, twice a day, getting notifications when something breaks. In this article we’re going to cover a few bits and pieces of what exactly we’re interested in testing, our general architecture, while also providing some useful snippets for some parts of it in case you want to implement it yourself.
What we test and why
At the most basic implementation we need to test the following:
- If the demo and all its pages are actually loading (basically a smoke test, you can never be sure what can go wrong in a new version deployment)
- All kinds of embeds and if they’re working (i.e. YouTube, Vimeo, Instagram, etc, just have a look at our theme Doberman’s article page demo to get an idea). As a potential customer, seeing a broken YouTube video on a theme demo is certainly not very appealing, and the theme feels neglected.
- Missing or broken images
- Possible third-party plugins breakage (e.g. need to make sure that nothing will break after we update a plugin in our multisite demo installation)
- Third party API integrations (i.e. our Weather API calls in most of our hotel themes, or our crypto widgets in our very recent Blockchain theme)
We’ll soon see how we’ve managed, with very little code, to check all our theme demos and all their pages for at least the above scenarios.
Choosing a testing framework
Back when we started thinking about this project there weren’t many options for tried and robust end-to-end testing frameworks. We very much needed to run all tests on actual browsers (not headless ones) and we also required the ability to actually play videos or sound (for video and music embeds). We also needed the tests to be written in a language that the team is (and will be) comfortable with in the future. With the above in mind we decided to go with Nightwatch.js which is an end-to-end testing framework based on NodeJS and runs on Selenium. Nowadays, there are a few more options (some more modern, granted), but so far we’re satisfied with the above solution.
Nightwatch can be pretty daunting at first, and this is because it uses Selenium under the hood. A lot of the internals do indeed require a good Selenium knowledge. For anyone that would like to implement some of the snippets provided in this post, installation and setting up Nightwatch.js along with Selenium will be assumed as the Getting Started section of the documentation is a very good resource.
Nightwatch tests are easy to write, as with any end-to-end testing framework, you supply a series of instructions (what you want to happen in the browser) as if you were a user, and then verify certain assertions (Nightwatch comes with its own assertion libraries).
As an example (taken directly from Nightwatch’s documentation) let’s see how we’d test the Google homepage for some basic functionality:
Let’s see what the above does line by line:
- We export the test as a node module (Nightwatch picks it up and runs it automatically in the runtime).
- Every test suite (module) can have multiple tests, in line 2 we define a single test and give it a name. Every test gets a callback function with a
clientreference (which is the browser instance).
- We start chaining instructions to the
- Opens up
- We wait for 1 second (
1000 milliseconds) for the
bodyelement to become visible on the page and verify that it indeed does (if it doesn’t, after 1 second the test will immediately fail). This is a good smoke test to run on every page.
- Using a CSS selector (which Nightwatch conveniently supports) we target an form input (the one at Google’s homepage) and set its value as “nightwatch”.
- Wait 1 second for the
buttonelement with the
btnGto appear on the page.
- Click on the button.
- Pause the test for 1 second (waiting for the results page to load).
- Assert that the page that now has loaded contains the text “Night Watch” anywhere in it.
- End the test suite (and close the browser instance).
And this is it, our first Nightwatch.js test which mimics a user searching for “nightwatch” in Google’s homepage. As you see the way we write tests is very straightforward and declarative. Nightwatch.js allows us to write our own custom assertions (i.e. similar to
elementPresent) and custom commands (i.e.
Since we need to test a lot of pages, it would be impractical to write each test by hand for every single page on all demos. Imagine about 10 pages for each one of 150 demos and we’re quickly skyrocketing to manually typing out 1,500 tests. That’s no good. Thankfully we’re lazy and (kind of) smart and we can abstract the bigger bulk of our tests into catch-all custom assertions and/or commands.
The above is the very basic boilerplate for any kind of test suite we write, and although minimal, it’s quite powerful. We open up the homepage and after verifying that it loads (
elementPresent) we extract all the page URLs of the demo via its main navigation menu with the useful
getPagesFromNav utility we’ve wrote exactly for this purpose:
Standard Selenium implementation, looping all
anchor elements in a supplied selector (our nav menu) and storing every
href attribute in an array which we later return.
Then, in a second test, we loop over all extracted page URLs of that particular demo and run other global tests with
verifyGlobalTests is a custom command which encompasses, as the name implies, global tests that we want to run on every single page. A toned down (but similar to our actual one) version is:
(Note: have you found a more elegant way to traverse and test URLs extracted by a navigation menu? Feel free to share in the comments below!)
Verifying all embeds in one go
Let’s delve a bit deeper into how we test all kinds of externally embedded content such as YouTube, Vimeo, or even Facebook, here’s the implementation of the `verifyEmbeds` custom command:
Without getting into boring details, the gist is that we first locate all
<frame> elements in a page (embeds mostly always come in iframe embeds, at least for the providers we’re interested in), then we loop each frame and check its
src attribute. If that source attribute matches any provider of interest (i.e. youtube, vimeo, etc) we run an assertion against that frame, verifying that it loads properly.
For example, the Facebook like box, if it loads correctly contains the string
likes (e.g. “1,990 likes” next to the Like button), so we check just that. For Vimeo videos, the assertion is similarly simple. If we detect a certain class (
.play-icon) we can be sure the video loaded up correctly. Other tests are more involved, like YouTube for example. For YouTube, we have to specifically enter its frame and actually force the video to play in order to figure out if it’s broken or not (for some reason some broken YouTube videos allow you to start playing them before notifying you that they don’t exist any more, so simply checking if a play button is visible doesn’t do the trick).
But how we’ve reached to what needs to be checked for each provider in the first place? A lot of research and trial and error, uploading videos and other embeds and then deleting them, or deactivating them, or breaking them and testing against them. The assertions for each provider are indeed fragile, i.e. a provider might change their HTML markup or CSS class naming, but that’s OK. We’ve abstracted all this in a good way so that it’s perfectly maintainable. If we detect that something has changed in a provider’s assertion we only need to adjust it in a single place instead of hundreds.
Also note that the beauty of this abstraction is that it blindly tries to check every single frame on a page, if the page doesn’t contain any it simply does nothing, but the developer needs not know which pages specifically contain embeds, they can just run everything against it and be sure that if some contain embeds they will be properly tested.
Checking for console errors
The browser’s console can inform us for a lot of things going wrong. One major benefit it has is that it points out any kind of broken images or content that returns 404 (they log as actual errors which is very helpful). For this task we’ve used a custom assertion (
client.verify.noConsoleErrors). The implementation is pretty straightforward with a small twist:
A bit different syntax than custom commands, Nightwatch’s custom assertions API documentation can be found here. What we’re doing in simple terms is fetching all browser console messages with
this.api.getLog('browser')and filtering out everything but the
SEVERE level (which corresponds to errors).
Then, as we’re not interested in errors generated by third party APIs, services, embeds, or browser extensions we also filter these out by creating a dictionary of string patterns such errors might include (see the
ignoredErrorMessagePatterns class property).
Wrapping it up
Our end-to-end tests are always a work in progress. Although we’ve displayed our main architecture and the implementation of some important assertions, we are constantly thinking of new test cases. We have different assertions for WooCommerce, hotel, or business themes, and we’re always extending our test suites with other fine grained assertions according to each theme’s requirements. Automating our testing has cut down a lot of work in theme demo maintenance, tracking errors, and making sure we detect them before we’re notified by our users, but they’re not meant to completely substitute the experienced eye of a human, rather more of an aid.
We’re also experimenting with augmenting our end-to-end testing flow with screenshot diffing (via Wraith.js), and when that comes to fruition you can expect a new blog post describing the whole process. Until then, feel free to let us know your thoughts or techniques for automated browser testing in the comments below!