This is the process of writing code and using tools to automate the user interface (UI) testing of front-end components of websites, desktop applications or mobile applications.
But, getting it right goes beyond the coding of the tests and the tools you use. You need to make sure you understand things like what you’re testing, how your tests need to fit your team’s workflow and your release plan.
Why do UI Test Automation?
While many software teams might have at the minimum, some code inspection tools or unit tests, often because of their size or scope of the software, the time pressures on a team or the ingrained habits mean that UI testing is often either nonexistent or manually achieved through business acceptance testing or manual UI regression testing.
While the cost for UI automation can be high and maintenance costs increase for the test code, the benefits lead to faster regression testing, cross-browser testing, better coverage, consistent UI tests, and reduced load on manual testing / user acceptance testing, meaning those testing the application manually can instead focus on exploratory testing or really paying attention to the newer features that are being released and ensuring their quality and functionality, and allowing automation to confirm the state of past tests.
In addition, it’s often the case that repetitive testing of a UI might be frustrating and time-consuming, for example, testing an application form with hundreds of different combinations of user input. A write-once automation test that can re-run itself with different inputs can take away that painful time-consuming testing from a manual tester as well.
Key Concepts
1. Why Automate?
Quality of software costs time and it costs money. By shifting testing left and establishing test automation in the software development process, we can reduce risk, time and cost in discovering issues and/or defects.
Automation leads to faster processing of testing of your code, quicker turnaround times and assists in scaling of businesses. All of this leads to faster times to quality, as well as consistent quality. Automated regression testing also helps detect recurring defects, and the ability to be more competitive by delivering your features quickly and confidently.
Automation can also be used to expand test coverage, by creating tests not easily performed manually, running far more cases than could be executed by hand, with tight integration with CI / DevOps.
2. Who Automates?
This depends on the type of automation. There are no-code record and playback tools like Selenium IDE that allow a user with little to no experience in coding to record a test and have it playback as much as required.
If, however, you’re wanting to have a large test suite, with code reuse, possibly with other access to APIs or databases as well, coding is usually required. There should be dedicated automation testers to help build an effective and efficient UI automation framework, or assisting developers themselves in building it as part of your integration tests.
3. What to Automate?
The common, often invalid assumption is that we should automate everything! However, the cost, time, value/quality project management triangle comes into play. Some tests are trivial to automate, and maybe useful in fragile areas of code or high risk items. Some tests might be very, very difficult to automate and maybe in an area that hardly ever changes. Sometimes a decision needs to be made as to what we care about if it isn’t caught by a failing test.
A rule of thumb is to look at areas of code most commonly executed, critical features, or features that rely on a lot of other components and balance the time vs the cost. There’s usually little point in spending weeks automating part of a UI that three users a year click on, but at the same time, if it’s a critical page, say, the only place a user can change their email address, you’ll need to consider the impact if a failure here is missed.
While the UI tools can easily be used to navigate, click and generally interact with the site/application under test, consideration should also be given to the data behind it. It’s often useful to automate the test data creation directly into a database, instead of creating say 50 new test users slowly via the UI.
4. When to Automate?
Another area of debate is when some say automate core testing code, others automate a regression suite of tests. Ideally, if you do have a regression test suite or set of integration tests that are automated, they should run in a matter of minutes, not hours, allowing developers to make changes frequently with confidence.
If using an Agile development process, the start of a sprint can be utilised to identify which stories will require automation. It’s useful to add this to the acceptance criteria for an individual story, so it’s part of the team’s ‘definition of done’.
5. How to Automate?
Your team will likely have either a preference of tools, or be open to what to use, and the platform under test should be taken into consideration. The de facto standard is to use a code-based automation tool like Selenium, but as per the ‘Who automates’ section, this requires some coding experience. Record and playback tools like Selenium IDE can be easier. More modern tools like Cypress allow you to ‘step back’ in time through the test so you can see what changed, and when.
The idea behind all these is that the tools emulate a user. They open a web browser, perform actions on a website, and validate the result.
By creating a UI framework for tests, with a defined set of test environments, tests can then be added over time and ideally executed as part of CI / CD.
6. Where to Automate?
While with UI automation, the person or team building the test will usually execute them the most, integration with a build pipeline is of immense importance. This allows developers to commit code and get automated feedback on whether they’ve broken anything. It also allows reporting to be added and produced as part of this build pipeline and to show the team where and if anything has changed.
Key Inputs
The key inputs for UI Test Automation are:
- Stories: A very useful point of input into the automation is in the planning and kick off ceremonies for stories. During planning and estimation, the team can decide if the story could/should be automated, and depending on the estimation policy of the team, include this estimate in their story sizing.
At kick-off: When a developer picks up a story, the ‘three amigos’ (business / product person, developer and tester) can again discuss the story, their understanding, and what can or should be done regarding automation. For example, will the developer’s unit tests be enough to test the change, should integration tests be added, or should full UI automation tests be built. - Effort: Another key input for the automation of stories is the amount of time and effort it will take to automate your test, and the perceived benefits vs having the current test remain as purely manual in execution. The desire to ‘just automate everything’ is often strong, but considerations to the usefulness of the specific test, the reuse of it, the stability of the code and how complex it is to automate should all be taken into consideration.
- Scope: Finally, when automating testing of the UI, scope is important. When automating, will the test build the test data as well? Will it just execute the happy path (Everything works) or try and ‘break’ with negative cases as well? Will tests run on mobile, or desktop devices? And what about operating system, or browser? All modern browsers or just say, Firefox?
Generally, an approach to this is to inspect your analytics and determine what percentage of users would be covered with a reasonable amount of effort; perhaps the 80 / 20 rule as per Pareto’s Principle.
Preparation
Two of the key moments in preparing for automation of the UI are sprint planning/estimation and a story kick off meeting.
At planning/estimation, stories that would require estimation can be identified, and code could be written in advance, or as part of the story. The kick off: when a story begins, is the perfect time for the three amigos (business, developer and tester) to define the scope of the automation for the story, who will do it, and what it will achieve.
Key Activities
The Key Activities at a high level:
- Discovery
- Execution
- Release
- Evaluation
Let’s walk through the UI Test Automation specific elements of each activity.
1. Discovery
As new stories are onboarded to the team, this stage allows the QA team to gather requirements knowledge, investigate tools of choice, define a test strategy and set project milestones, in collaboration with the rest of the agile team.
This is also an important time to work through roles and responsibilities within the team, the definition of done for stories, and how much automation this will include, as well as whether automation is happening as part of stories, or as separate work.
2. Execution
This is the sprint cycle time, during which user stories and acceptance criteria will be finalised and documented. It helps the team develop velocity estimates, get the strategy for QA and automation approved by the rest of the team, and begin development of test automation code, in parallel with the development team. Collaboration at this stage is key to success, to ensure that there’s no gaps, or duplication of automation effort.
3. Release
Post sprint, ideally a demo will occur in which the efforts will be showcased. Acceptance criteria must be met, tests should pass, and acceptance by the product owner. If releasing to production, a roll out / release plan should be developed, and an indication of what regression testing / automation will be included to make sure it’s ready for a release.
4. Evaluation
Support and evaluation is an ongoing process. Monitoring of tests, bug fixes, continuous testing pipeline processes and maintenance, as well as reflection from retrospectives in each sprint to see how the team can improve going forward.
Outputs
Out of the UI Test Automation, you can expect:
- Code: Depending on the strategy taken, UI automation would likely include code – a test framework, in source control, and integrated with continuous integration to facilitate regression testing.
- Documentation: This would include an indication of what is tested, the scope and reach of the tests, how to execute the tests, and potentially how to analyse results.
- Reports: Test execution runs would generate reporting, ideally with statistics of the number of tests executed, time taken to complete, pass rate, and errors that occurred which could be used in debugging.
- Integration with CI: The benefit of automation is truly realised upon repeated execution of tests – automatically performing regression testing upon each commit/branch as part of continuous integration.
Further Reading
Articles
Videos
Books
Podcasts
Mark Mayo
Senior Quality Engineer
Mark Mayo is a Senior Quality Engineer at Terem. He has nearly two decades of software development and quality assurance experience across a variety of industries over four countries, including aviation, networking, real estate, travel, video games and banking.
LinkedIn: linkedin.com/in/mark-mayo