Testing Modernisation Playbook

In today’s rapidly evolving technological landscape, staying competitive often hinges on a company’s ability to adapt and modernise its processes. This is particularly true in the realm of software testing, where traditional methods can no longer keep pace with the demands of modern software development. As businesses strive to deliver high-quality software at an ever-increasing speed, modernising testing processes becomes not just beneficial, but essential.

Testing Modernisation is critical for businesses to remain competitive in an increasingly digital world, one that demands faster delivery of high-quality products. By modernising testing processes, businesses can accelerate their time-to-market, reduce costs, and improve the overall quality of their software. This shift not only enhances the ability to detect and fix issues early, but also supports a more agile and responsive development process.

This playbook focuses on the essential elements of Testing Modernisation, including the key concepts, inputs, activities, and outputs. It has been constructed and back-tested against the many discoveries we’ve run at Terem. It’s inspired by thinking and practices from globally recognised thought leadership and frameworks including Agile Methodology, Lean Software Development, Continuous Integration and Continuous Deployment (CI/CD) principles, DevOps culture, the Scrum framework, principles of Test-Driven Development (TDD) and Behavior-Driven Development (BDD), as well as the Modern Testing Principles. These methodologies and frameworks have been instrumental in shaping modern software development and testing practices, emphasising efficiency, collaboration, and continuous improvement.

Key Concepts

What is Testing Process Modernisation?

Testing Modernisation is the process of updating and refining the methods and tools used for testing software to meet the challenges of modern software development. In a world where the pace of software releases is accelerating, Modernisation involves adopting new technologies, methodologies, and practices to make testing more efficient, accurate, and aligned with continuous integration and deployment practices.

Why Do I Need to Modernise My Test Process? 

Modernising and automating test processes generally is valuable in accelerating your time to market and reducing cost. Before diving into how to modernise your company’s testing processes, it’s crucial to understand why this shift is necessary and what specific benefit you are looking for. Traditional testing methods – often manual and linear, unnecessarily complex and checkbox-driven – are becoming increasingly inefficient and unable to handle the complexity and speed required in today’s development cycles. This lag not only slows down the release of new features but also impacts the overall quality of the software.

Key Inputs

There are several things you need to understand or baseline before you think about modernising your testing technology or processes.

Health Check

This is where we need to evaluate the current state of Quality Assurance (QA) within your team or organisation.  This can be achieved by taking stock of a series of analyses of different goals, states, and gaps within the practice.

1. Review Existing QA Processes and Documentation: Examine the current QA processes in place. This includes reviewing all existing documentation, such as test plans, test cases, quality control procedures, and QA policies. Assess if these are up-to-date and aligned with your objectives.
2. Assess Tools and Technologies: Evaluate the tools and technologies currently being used for QA. This includes software for test automation, bug tracking, and performance monitoring. Determine if these tools are effective, up-to-date, and well integrated into your QA processes.
3. Analyse QA Team Skills and Training: Review the skill set and expertise of your QA team. Are they adequately trained? Do they have the necessary knowledge and tools to perform their tasks effectively? Assess if there’s a need for additional training or hiring.
4. Measure Quality Metrics: Analyse the quality metrics that have been collected, such as bug counts, test coverage, defect escape rates, and customer feedback on product quality. This will give you an insight into the effectiveness of your current QA practices.
5. Conduct Stakeholder Interviews and Surveys: Engage with stakeholders, including QA team members, developers, project managers, and customers, to gather feedback on the QA process. This can include conducting interviews or surveys to understand their perspectives and identify areas for improvement.
6. Perform a Gap Analysis: Compare your current QA state with the defined objectives and industry best practices. Identify gaps in processes, skills, tools, and technologies. This analysis will help in pinpointing areas that require enhancement.

After this, you’ll ideally have a clear understanding of what you have in terms of a testing / QA practice, how you measure it, and what you’re missing.

Discovery

Goals

No software is perfect. And even software that perfectly matches the requirements as defined by the team may not match the customer’s expectations or goals.  Our objective here is to decide what we want from quality. Things to consider:

1. Is quality just a ‘gate’ – a safety net at the end that happens before release of software? Or could it be an iterative process, with quality baked in from the start?
2. Who is responsible for quality? Hint: Everyone! 
3. What is your core quality objective – e.g. compliance, reliability, customer satisfaction?
4. What are the industry standards (e.g. meeting MISRA standards in automobile software)?
5. What resources would we need?
6. How will we measure success? (We will cover this in the next section).

Measurements

Peter Drucker is often attributed with the quote “You can’t improve what you don’t measure”. 

Much has been written about the meaning of this, and there are certainly things you can’t measure well – team collaboration, and integrity, for example.  However, there are more than enough data points that can and should be monitored and measured. It’s also important to do so for several reasons.

1. Ensures Reliability and Performance: Regular measurement allows developers to gauge the reliability and performance of software. This ensures that the software consistently meets user expectations and functions optimally under various conditions.
2. Identifies Improvement Areas: By measuring different aspects of software quality, developers can identify specific areas that need improvement. This targeted approach ensures efficient use of resources and time in enhancing the software.
3. Increases User Satisfaction: High-quality software leads to greater user satisfaction. Measuring software quality helps in maintaining high standards, which in turn attracts and retains users.
4. Facilitates Better Decision Making: Quantitative data obtained from measuring software quality aids in informed decision-making. It provides clear insights into the software’s performance, helping stakeholders make better strategic decisions.
5. Supports Compliance and Standards Adherence: In many industries, software must adhere to specific standards and regulations. Quality measurement ensures compliance with these requirements, avoiding legal and financial repercussions.
6. Reduces Costs in the Long Run: Identifying and addressing issues early through quality measurement can significantly reduce the cost of future fixes. It’s more cost-effective to maintain ongoing quality than to rectify major issues later.  Similarly, it’s often cheaper to fix an issue sooner rather than later.
7. Encourages Continuous Improvement: Regular quality measurement fosters a culture of continuous improvement. It encourages teams to strive for higher quality standards, leading to innovative and more robust software solutions.

Based on your goals in the previous section, measurements of quality can be established to confirm that you’re on track / meeting your goals.  For the areas that you’re not on track, you now have a measurement for what you need to improve on/against.

It’s important, however, to make sure you’re tracking useful metrics of quality, rather than vanity metrics.

See Useful Test Automation Metrics You Should Be Doing for further guidance on this..

Key Activities for Testing Modernisation

Now that you have a clearer understanding of the current state of your QA systems, the goals you are striving towards, and the metrics you will use to assess alignment with those goals, you are ready to begin the modernisation journey. Below are a series of recommended next steps, in no particular order.

1. Embrace Automation

In terms of modernising a team’s approach to shippable quality, the decision to embrace automation is pivotal. Automation in testing primarily involves the use of software tools to execute pre-scripted tests on the software before it is released into production. This shift is crucial for several reasons. 

1. It significantly reduces the time required for repetitive testing, which is especially beneficial for regression testing.
2. It enhances accuracy by eliminating the possibility of human error in repetitive tasks. Automation also allows for more extensive and thorough testing, which can be conducted more frequently, ensuring that more parts of the application are tested and potential defects are identified early. Leading the team in identifying the right tools and frameworks for automation, and training team members to write effective automated tests, is a key step in modernising your QA processes.

Example: A practical example could be the implementation of a set of Appium scripts for automating mobile application testing. This could involve setting up an automated regression suite that runs every time there is a new build, ensuring that key functionalities work as expected. For instance, if we’re working on an e-commerce application, automated scripts could be written to test the entire shopping cart process, from adding items to checkout, on both Android or iOS. This not only speeds up the testing process but also ensures that critical functionalities are always tested consistently, reducing the risk of human error.

2. Implement CI and CD (Including Tests)

Continuous Integration (CI) and Continuous Deployment (CD) are practices that are integral to modern software development and testing. CI involves the regular integration of code changes into a shared repository, followed by automated builds and tests.

CD extends this to ensure that every change that passes the automated tests can be automatically deployed to production. Implementing CI/CD streamlines the development process, allowing for faster, more frequent releases with reduced manual intervention. For QA, this means shifting the focus towards developing and maintaining robust automated test suites that can validate changes quickly and reliably. Guiding the team in integrating their tests into the CI/CD pipeline and ensuring that these tests are reliable and efficient is crucial for maintaining a high standard of shippable quality.

Example: To implement CI/CD, you might integrate a tool like Jenkins, or use Github Actions for continuous integration. Every time a developer commits code to the repository, Jenkins/GH Actions automatically triggers a build and runs the test suite. If the build and tests pass, the code can be automatically deployed to a staging environment (also in above tools). For example, in a web development project, this approach ensures that any changes made to the codebase are immediately tested and ready for rapid deployment, significantly reducing the time to release.

3. Foster a Culture of Quality

Modernising shippable quality is not just about implementing new tools or processes; it’s equally about fostering a culture of quality within the team. This involves ensuring that every team member, not just those in QA, understands the importance of quality and their role in achieving it. It means advocating for quality at every stage of the software development lifecycle, from requirements gathering to design, development, testing, and deployment. Encouraging open communication, collaboration, and a shared responsibility for quality helps in building a more quality-centric mindset. Leading by example and mentoring others in quality practices, while also encouraging feedback and learning from failures, are key strategies in cultivating this culture.

Example: This could involve regular cross-functional workshops where developers, testers, and business analysts collaborate to discuss quality goals. For instance, before starting a new feature, a joint session could be held to discuss potential risks and how to mitigate them. This approach not only improves understanding across the team but also ensures everyone is aligned on the quality objectives from the outset.

4. Identify Issues/Bottlenecks and Continuously Iterate and Improve to  Help Accelerate the Team’s Progress 

In a modern QA environment, continuous improvement is vital. This involves regularly evaluating and updating testing processes, tools, and strategies to ensure they remain effective and aligned with current industry standards and technologies. It’s about being proactive in identifying areas for improvement, whether it’s in test coverage, test execution speed, or defect detection rates. This also includes staying abreast of new testing methodologies and technologies and assessing their applicability to our current processes. Leading regular review meetings, encouraging innovation, and fostering an environment where team members are motivated to suggest and implement improvements plays a crucial role in the ongoing modernisation of our QA practices.

Example: An example of this continuous improvement could be the periodic review of our test automation scripts. After noticing that certain scripts are flaky and often fail due to minor, irrelevant changes in the UI, we could refactor these tests to be more resilient. This might involve using more reliable locators or adding intelligent waits. Regularly revisiting and refining these scripts ensures they remain effective and efficient.

In addition to the above, this is where the metrics in the earlier section come in. By continuously monitoring and measuring, you can see changes / improvements in your quality, time to deployment, time to fix and more.

It’s important to closely monitor the testing lifecycle, identify these bottlenecks, and work collaboratively with the team to find solutions. This might involve optimising test suites for faster execution, improving test environment management, or enhancing collaboration between developers and testers through better integration of tools and processes.

Second Example: To accelerate the team and mitigate bottlenecks, we might conduct a ‘Value Stream Mapping’ exercise. This would involve mapping out each step in the software development process to identify where delays typically occur. For example, if we find that the handover from development to QA is a consistent bottleneck, we might introduce a shift-left approach, where testers get involved earlier in the development cycle to provide feedback sooner.

6. Utilise Machine Learning / AI

Incorporating machine learning and AI into QA processes represents a cutting-edge approach to modernising shippable quality. These technologies can be used to predict potential quality issues, optimise test cases, and even identify areas of the application that require more rigorous testing based on historical data. Machine learning models can analyse vast amounts of data from past test runs to identify patterns and predict outcomes, thereby helping in making informed decisions about where to focus testing efforts. Exploring and implementing ML/AI solutions requires staying updated with the latest advancements in these fields and understanding how they can be applied effectively within the team’s testing processes.

Example: An application of machine learning in QA could be using a tool like Testim.io or Applitools, which leverage AI for visual testing. These tools can automatically detect visual anomalies in web applications. For instance, when testing a UI with numerous visual elements, these tools can quickly identify if any element is out of place or not rendered correctly across different browsers, something that would be time-consuming and error-prone if done manually.

7. Stop Using QA as a Safety Net

Modern QA practices emphasise the idea that quality assurance is not merely a safety net at the end of the development cycle but an integral part of the entire software development process. This means shifting the mindset from finding bugs to preventing them. It involves working closely with developers, business analysts, and other stakeholders from the beginning of the project to understand requirements clearly, set quality standards, and identify potential issues early on. Promoting this proactive approach to quality and ensuring that QA is involved throughout the software development life cycle is crucial for early defect detection and reducing the cost and effort of fixing bugs later in the process.

Example: Consider a scenario where QA is involved right from the requirement-gathering phase. In this case, testers would actively participate in discussions with business analysts and developers to understand the requirements clearly. This early involvement allows testers to identify potential ambiguities or issues in the requirements themselves, preventing costly fixes down the line.

8. Reduce Reliance on Spreadsheets of Test Cases

Moving away from traditional methods like maintaining spreadsheets of test cases is a significant step in modernising QA. This old approach is often time-consuming, error-prone, and difficult to maintain. Modern QA teams use test management tools that provide better organisation, version control, and traceability of test cases. These tools often integrate seamlessly with other tools used in the software development lifecycle, providing real-time insights and reporting capabilities. Guiding the team in the transition to an effective use of these modern test management tools is essential. This not only improves efficiency and accuracy but also ensures that the team can quickly adapt to changes in requirements or scope.

In addition, more modern techniques like acceptance based testing, Test Driven Development and exploratory/ad-hoc testing can reduce the documentation effort overhead that comes with maintaining thousands of potentially redundant test cases.

If traceability of every case IS required, we could adopt a test management tool like TestRail or Zephyr. These tools allow us to create, manage, and track test cases in a more organised and efficient way. For example, when working on a new release, all test cases can be easily updated and linked to specific requirements in these tools. This ensures better traceability and easier maintenance compared to traditional spreadsheets, especially as the project grows in complexity.

Summary

Continually modernising testing processes is essential for maintaining a competitive edge in today’s rapidly evolving technological landscape. By leveraging modern toolsets, teams can significantly enhance efficiency and accuracy, reducing the time to market and improving the overall quality of the product. Metrics play a crucial role in this evolution, providing tangible data to measure improvements and identify areas needing attention. They enable teams to track progress over time, from reduced bug rates to faster deployment cycles, offering clear insights into the effectiveness of current strategies. Furthermore, modernising processes fosters better teamwork and collaboration. It encourages a culture where quality is everyone’s responsibility, breaking down silos between developers, testers, and other stakeholders. This holistic approach not only streamlines workflows but also ensures a more robust and reliable product, ultimately leading to higher customer satisfaction and business success.