When is the Right Time to Stop Testing?

By: Lamis Ahmed

At one point or another, testers have asked themselves these questions: should I stop testing the product? Is the job truly finished?

Testing though very tedious a task is an exceedingly important part in developing software, therefore it is ideal to only stop testing until the defects are 100% corrected or removed. The problem with aiming for a perfect software without flaws lie in the amount of time and resources needed. We all eventually have to stop testing and deliver the product, but the question is: when should you do so?

 

160817_WRF_WhenStopTest02

Testing involves these three essentials: budget, time, and quality. Sometimes if you’re really lucky or just really good you can satisfy all three, but realistically there’s always a trade-off among them. The negative rule – which is unfortunately the approach most often used to stop testing – is to stop whenever any, if not all of the allocated resources are exhausted. The positive rule is to stop testing when either the reliability required of the software is met or if the benefits of continuing the testing can outweigh the cost. These two usually need the use of reliability models that help evaluate and predict the reliability of the software. The process of evaluation however, requires repetition of this cycle: failure data gathering, modeling, and prediction. This method however, is not suitable for very dependable systems because the accumulation of real field failure data will take too long.

Enough is enough: How to determine when to stop testing

Can anyone ever say without a shred of doubt that a piece of software is 100% functional, reliable, and without defect? Unless you’re testing a software as simple as ‘Hello World’, we can’t really say anything is guaranteed. Complete testing is something that is not feasible in practice and that has been a widely accepted fact, but in all projects there will always come a time when you need to step away and stop testing. There are things like deadlines that would force most of us to pull the plug on testing. Other times it would be because of a dwindling budget or lack or resources. Then there are instances when the team handling the project needs to make the call: Should we continue testing or are we done? Unfortunately in the real world, answers with magic formulas are exclusive to story books and fairytales. However, we can always appeal to logic and make educated as well as informed decisions by taking into account the following factors.

 

Criteria that help determine when to stop testing:

1-    Checking the test exit criteria is imperative before claiming that testing is complete. The product quality should be measured alongside completion criteria before ending the testing process.

Test Coverage

In an ideal world 100% test coverage is to be expected, but in reality such things as time and cost constraints exist which make it nearly impossible. Thus, when test cases meet a certain percentage it can be considered completed and the test coverage achieved. Here are some examples:

  • The ratio of passed to failed test cases– below are three methods with which you can interpret this:
    • When all cases are passed;
    • When the minimum portion that is needed to be passed is reached;
    • When the maximum limit of the allowed failures of test cases is met;
  • Exhausting all test cases available for execution during the test run.

2-    Defect Patterns

Website and software projects are unique and therefore no two are alike. Defects may run in patterns or in a group; when there is an identified problem area in the code it may need further testing as opposed to less troublesome areas. Defects may also become a major setback, so if the testers run into problems with increasing frequency on a constant basis, then clearly testing should not be stopped any time soon.

 

  • Bug Rates

The larger the added lines of code, the more likely it is to have a bug lurking in there somewhere. Each added line introduces a new place for a bug to exist. According to Caper Jones we introduce five bugs per function point depending on the system being built, bugs therefore are inevitable. The problem is that there are no tests where you can determine explicitly the bugs in your software but they crop up when there are unusual or obscure test cases or environmental conditions. However, with testing we can lessen the number of bugs and with further testing the frequency with which they appear will be greatly diminished.

 

  • Showstoppers

There may be times when it is the software that commands the testing phase to be done, well, at least temporarily. The existence of major structural defects and critical errors, though highly improbable is not impossible. These may be large enough to cause you to go back to square one or scrap the project entirely. Either way further testing would not only be unnecessary, but a waste of resources.

 

  • Testing budget of the project.

When the cost of continuing the testing does not justify the project cost or when the losses outweigh the gains it is best to stop.

 

  • Project deadline and test completion deadline.

In every project there is always a deadline to be met. If your project is way behind schedule or cannot meet the deadline then you obviously have to speed things up in order to meet it. Testing also needs to be planned and scheduled so that it’s done before the project deadline.

 

  • Duration of beta or alpha testing periods.

The alpha test is the first phase of testing which is comprised of unit testing, component testing and system testing. Beta testing is the second phase which is also known as field testing or pre-release testing.

In any project, it must always time bound. The outline in the test plan must include the optimal targets for these factors at the early stages of the project. This will also serve as a guide on the scheduling of testing as well as the target dates to meet, the duration and dates of which, are discussed with the customer and agreed upon by the team. The key testing metrics are then determined. Once those are met the testing is done.

 

Risks on “When to stop testing”

The risks on stopping testing can be measured by Risk Analysis but for projects with a short duration or those with low budget or resources, it can be inferred simply by the following:

  • Measuring Test Coverage.
  • Number of test cycles.
  • Number of high priority bugs

 

In conclusion

Software testing is an important yet extremely time consuming and resource draining activity which could potentially become an endless activity. But as pointed out over and over again by the criteria, there are certain limitations that will force you to stop doing so. Knowing how to determine and when to determine for testing to be stopped is not only essential to the project but to your personal well-being, and sanity, too!

As logical and scientific as this guide has tried to make it sound in order to appeal to your logic and seem more legit, it still boils down to your intuition and your gut feel that has been honed by experience.

While the testing experts have internalized if not mastered the process of testing to the point that it may have become second nature or an unconscious process to them, the green horned testers of today may still find themselves consumed by the darkness which is endless testing. Thus, we offer you this ray of light, the testing metrics. Just be careful not to obsess over it too much and be blinded.

Happy testing!

 

References

https://users.ece.cmu.edu/~koopman/des_s99/sw_testing/

http://www.softwaretestingtimes.com/2010/04/when-to-stop-testing-testing-completion.html

https://leantesting.com/resources/stop-testing/

http://csqa.info/when_to_stop_testing

http://reqtest.com/blog/when-do-i-stop-testing/

Our Services

Worpress

Lorem ipsum dolor amet consectetur adipiscing elit. Nunc mi dui placerat eu faucibus vel, efficitur quis magna. Pellentesque habitant morbi tristique senectus et netus et malesuada.

Read More

Web Design

Lorem ipsum dolor amet consectetur adipiscing elit. Nunc mi dui placerat eu faucibus vel, efficitur quis magna. Pellentesque habitant morbi tristique senectus et netus et malesuada.

Read More

Ecommerce

Lorem ipsum dolor amet consectetur adipiscing elit. Nunc mi dui placerat eu faucibus vel, efficitur quis magna. Pellentesque habitant morbi tristique senectus et netus et malesuada.

Brand Now

SEO

Lorem ipsum dolor amet consectetur adipiscing elit. Nunc mi dui placerat eu faucibus vel, efficitur quis magna. Pellentesque habitant morbi tristique senectus et netus et malesuada.

Read More

Our Team

ghada3

Sara Naimi

(HR Officer)

“Our continued success in the GCC market comes from our specialized, unique services — and that requires a highly competent, motivated and committed team. So what…

Our Top Management Team

Our Top Management Team

  • ghada3

    “Our continued success in the GCC market comes from our specialized, unique services — and that requires a highly competent, motivated and committed team. So what do we offer you? We provide a stimulating, challenging and dynamic environment that allows you to grow your career as far as your talent and ambition will take you. If you want to develop your skills and expertise, grow your career and contribute to the success of our team, then you are exactly what we are looking for.”

    Sara Naimi
    Wareef United HR Officer

Our Partners