In an increasingly digital world, high-quality enterprise software is becoming more crucial. Software quality assurance (QA) plays a pivotal role in ensuring the highest quality standards. However, companies must navigate the delicate balance between achieving maximum quality and minimizing effort – the essence of mastering enterprise software quality assurance.

In this articles series, I will present solutions that can guide to an optimal equilibrium between excellence and efficiency in QA.

Part 1: Key Challenges Faced by Every Quality Assurance Endeavor

In the demanding cadence of software development, quality assurance can become the vulnerability when challenges and risks accumulate. Both these factors have the potential to consume substantial time and resources while diminishing quality. Thus, a meticulous examination of these aspects becomes indispensable. In this first part of the article series, I will focus on diverse challenges that every QA team confronts – obstacles that directly influence product quality, time commitments, and resource allocation.

Help! I Need to Write Test Cases

Let’s consider the following situation: I’ve been assigned a ticket. I am supposed to come up with good test cases for a requirement. But what exactly are “good test cases,” and how are they evaluated? My understanding of the business is somewhat vague, to say the least. Many terms are being used that I can’t comprehend. There are too many processes, details, and intricacies! Help! I prefer to focus closely on the ticket description and come up with something based on that. When it comes to writing the test cases, I struggle to define a clear objective. Do I need to describe every click in the test case? What goes into the test case expectation? I divide the clicks into multiple smaller test cases. Wow, now I have about 20 test cases easily! That’s more than enough for a single ticket. My test cases are surely good since I am verifying the correctness of multiple clicks.

Congratulations! You have unnecessarily inflated your tests by 20 test cases. In doing so, you have increased the test execution time and maintenance efforts for minimal added value. Unfortunately, this impact extends not only to the current sprint but also to future sprints. What’s worse, you now have a hidden testing gap that leaves potential vulnerabilities undiscovered, thereby compromising the long-term stability and reliability of the software product. In fact, by working on the ticket, you have effectively reduced quality instead of enhancing it.

As the number of “bloated” test cases grows, the quality diminishes while the efforts escalate.

Many Reasons for Quality Losses in Quality Assurance

The goal of every QA process is inherently to improve the quality of materials, products, and processes. However, in the realm of enterprise software QA, quality losses are not uncommon. This can stem from various factors:

• Insufficient or lacking experience in quality assurance
• Absent or incorrect test strategy and test analysis
• Missing or misguided prioritization and risk assessment
• Deficient test design methods and limited coverage
• Incorrect selection of test data
• Poor implementation of test cases
• Slow and/or inaccurate bug analysis
• Inadequate or absent business knowledge
• Inadequate or absent depth in test content
• Insufficient or missing combination of business
• Lack of holistic perspective
• Slow test execution
• Poor, missing, recurring, or continuously growing test maintenance

When a QA process is plagued by quality losses, it becomes costly and inefficient over time. Therefore, it is highly important to identify and effectively rectify these losses.

Time-Consuming Aspects in Enterprise Software Quality Assurance

In addition to the potential quality losses, time losses in QA can also prove to be a significant cost factor. Every minute spent on inefficient testing processes consumes valuable resources and can lead to project schedule delays. Some of the potential time-consuming aspects in QA include:

Provisioning of the Testing Environment, Test Data and Test Prerequisites

Setting up the testing environment requires precise configuration of a multi-layered technical infrastructure, including servers, networks, and virtual machines, to ensure, for example, the most realistic representation of the production environment. The accurate preparation or generation of test data that is both representative and sufficiently extensive presents an ongoing challenge. Furthermore, the identification and management of specific test prerequisites, such as certain system states or configurations, is a complex task that needs to be repeatedly performed. The effort involved in creating and maintaining these test prerequisites can be substantial, as they may change over the course of development cycles.

The recurring necessity to keep the testing environment, data, and prerequisites synchronized and up-to-date requires continuous monitoring, updating, and versioning. To manage these tasks, deep understanding of software architecture, comprehensive technical expertise, and a methodical approach are required. Overall, provisioning the testing environment, test data, and test prerequisites represents a demanding, complex, time-consuming, costly, and recurring endeavor.

Test Execution

The time-consuming nature of test execution in quality assurance for enterprise software arises from the need to ensure comprehensive coverage of various application scenarios, input variables, edge cases, and interactions. This requires the design and implementation of a wide range of test cases and test data that must be executed in different environments with varying configurations.

Executing these tests demands significant resources and time, as performing the tests, collecting and analyzing the results, as well as potential error diagnosis and resolution, all take a considerable amount of time. Additionally, tests may introduce additional complexity and time due to changes in application logic, system architecture, or test infrastructure.

Here’s a brief summary of potential time-consuming aspects:

• Clarifying whether an issue is truly a bug or an intended feature
• Investigating and analyzing errors and log files
• Pinpointing elusive errors that are not easily reproducible
• Creating error tickets with detailed information such as steps to reproduce the error, important notes, logs, screenshots, or a helpful video recording
• Follow-up errors: Errors caused by another error that disappear when the original error is fixed. Analyzing and identifying such errors is time-consuming.
• “Ping-pong” errors: Multiple builds and test runs are required to fix a single error.
• Side effects: Fixing one error causes one or more new errors, which then need to be resolved.
• Forward and backward changes: Changes are implemented in one release but rolled back in the next. Such changes lead to unnecessary test adaptation efforts.
• Regression: Repeating the same test cases multiple times.
• Re-runs (release candidates, patches, etc.): Repeating all test cases multiple times.
• Repeating tests in different environments, browsers, etc.
• Repeating tests at different development stages.
• Scaling test maintenance and execution time, especially by complex or long-lived enterprise software.
• Frequent changes to test plans or test scope.
• High documentation effort for test results.

With escalating time-consumers, quality assurance becomes more costly and time-intensive. This poses an elevated risk, especially during testing phases.

Challenges Amidst the Testing Phase: An Insight into Risks from a Quality Assurance Perspective

Within the critical realm of software development lies the testing phase where unforeseen barriers can emerge, potentially impeding the seamless flow of quality assurance and placing the timely delivery of software at risk. Below, we illuminate some of these potential hazards:

• Unplanned reduction of QA resources.
• Hardware issues occur, such as a VM failure.
• The code freeze is not adhered to.
• The QA team is not informed of a change or is informed too late.
• The testing phase is shortened, for instance, due to a late addition of a requirement or prolonged implementation.
• Fixing bugs leads to the introduction of new bugs.
• Fixing bugs introduces new features or changes despite a code freeze.
• The bug fix causes adjustment efforts in the QA area.
• The bug fix includes a blocker for test execution. For example: the detection of a central component on the UI is no longer possible and the automated UI tests cannot be executed.
• More test runs are required than planned.
• The number of discovered bugs does not decrease.
• Blockers or critical bugs continue to be found at the end of the testing phase.

Undiscovered risks during the testing phase can lead to increased costs and hinder test efficiency. Therefore, it is crucial to identify emerging risks early and take appropriate measures to reduce or prevent potential impacts.

Interim conclusion

In summary, the highlighted challenges and risks vividly illustrate their potential for significant losses in terms of quality, time, and resources. A thorough consideration of these aspects proves not only essential but also a strategic investment to ensure sustainably efficient software quality assurance.

How Do I Envision an Efficient QA?

Efficient quality assurance needs to be cost-effective, swift, and exemplary.

What does cost-effective mean?

• Utilizing minimal resources
• Short test execution times
• Minimal maintenance efforts

How is swift defined?

• Swift error analysis
• Swift test results
• Swift test reports

What constitutes “exemplary”?

• Risk minimization
• High test coverage

How many QA resources do I typically need?

• In a team of six developers, on average, I need one QA resource.
• For every manual QA resource, on average, I need half a QA resource for test automation.

What are my expectations?

• All use cases are thoroughly tested from begin to end.
• Despite increasing requirements, the testing phase ideally remains consistently the same duration.
• No (critical) bugs are found in production, especially not immediately after delivery.

To enhance efficiency in QA, it initially requires a strategic superiority. A deliberate focus on quality strategies not only boosts product quality but also remarkably optimizes time and resource consumption.

More insights on this topic in Part 2 “Strategic superiority”.