In a previous blog post about automation, I explained that End-to-End (E2E) tests are inherently slow and that you need to focus on parallelism if you want throughput. With that in mind, you should strive to build your tests as reliably as possible in all possible environments; just because it works on your local machine does not mean that it will work consistently in your staging environment or when the tests are run on SauceLabs.
That leads us to the second lesson of the saga:
Slow and steady wins the race
In the Selenium world, many intermittent failures are caused by elements not showing up in time or not being enabled when the test tries to access it. In general, this is because the element has an unexpected transient state when the action is being performed.
The quick and dirty solution is to wait for an arbitrary amount of time, hoping that the element has the final expected state. This is such a terrible idea that I plan to dedicate an entire blog post to it in the future. A common scenario would be:
In this scenario, the test is waiting for 3 seconds, hoping that the element will be there by that time. The test then searches for the element and, finally, performs a click. This approach fails in several scenarios:
- What if the app was slow and the element was not there yet? The test fails.
- What if the element was there but was not enabled, so it was still not clickable? The test fails.
- What if the element was part of an animation and it was moving while you tried to click it? The test fails.
To address these issues, Selenium introduced the concept of Explicit Waits, which allows you to wait until the element has a particular state (“clickable” or “visible” for example). In AugmentedDriver, the driver has some extra functionality bundled in the augmented() library that helps you in those cases.
The previous example could be written:
With this approach, the click() will be performed only when the element is actually clickable. Augmented() has many other functionalities, such as findElementNotMoving(by) that waits until the animation has finished and the element is static, or findElementContain(by, text) that will wait until the element contains some text (Here is the interface where all that handy functionality is defined for Web and Mobile, here is the extra functionality that applies only to IOS and Android).
In Summary, whenever you are writing E2E automation you should be defensively coding so the tests are resilient to some degree of performance variance that are typical on Web or Mobile Applications. Adding extra checks on your tests might slow down your suites, but it’s a small price to pay for reliability and at the end that will make you win the race.
Stay up to date
Don’t miss out! Subscribe to our digest to get the latest about feature flags, continuous delivery, experimentation, and more.
New Relic and Split enable customers to make data-driven decisions using feature flags, with a shared belief that you can’t improve what you can’t measure.
When versioning your API, consider the tradeoffs between distinct microservices and feature flags.
It’s easy to think of the user interface as the primary target for new functionality, with product teams eagerly watching important business metrics such as conversion rates and user engagement for improvements after each release. But behind the scenes, engineers are continually working on server-side innovations such as changes in…