Importance of Metrics and KPIs in Software Testing. How to Measure Your Quality Success?

Important topic on quality metrics and how to measure defect leakage and test efficiency indicators during test life cycle. We build a product MetricsAlert which will help you to measure KPIs Check this out www.metricsalert.com/

[@whereweregoingwedontneedey4374]

For your 3% quarantee I guess things like exploratory testing aren't options (or if they are then it supplements a more formal, structured approach) - so assuming that you set the criteria under which the 3% guarantee applies? It seems a bold claim to me :D

[@QAMENTORTESTING]

Things such as exploratory testing is a MUST and should be part of any SIT or SAT phase. It has nothing to do with Defect Leakage. Also please elaborate why 3% guarantee is a bold statement when it is part of our Contract and SLA?

[@whereweregoingwedontneedey4374]

@@QAMENTORTESTING Hi, Maybe "bold" is the wrong word as in a don't mean unrealistic - its a very strong statement to make. If I may elaborate -- for you to have the 3% SLA I guess you almost decide on the test plan and strategy? Would you apply an Exploratory or session based testing approach alone and guarantee the 3%, or do have a mix of structured and Session based?

[@khurrambhatti9470]

Apparently that so called "complicated formula" is (number of defects in sit * their relative weight * 100)/(number of defects in prod * their relative weight + number of defects in sit * their relative weight). In that example there is 1 critical defect in sit (1 * 4), and 3 critical defects in prod (3 * 4), where 4 is the weight in this case, and we get a 25% test execution efficiency by calculating (4 * 100/16).
Lets use another example with only 1 low defect in prod. Do you know what the outcome is? Feel free to validate my answer but the test execution efficiency in this case is 0% 🤣 🤣. Can you please explain that. I would love to hear from you. Thanks.

[@QAMENTORTESTING]

Yes, your test efficiency is ZERO because your test strategy including test design/test coverage was not able to reveal that low severity issue. What do you think we did not know that it will be a ZERO because you find nothing during the testing? And what is the probability of this to be a ZERO if you measure 1000 release cycles or sprints? Probably same probability for you to win a lottery. Keep smiling and be happy.

[@khurrambhatti9470]

@@QAMENTORTESTING I did my planning, and I had a 1000 scenarios to go with, but missed a minor UX defect. Lets say that the color in one of the pages header was RGB(100, 100, 100) instead of RGB(101, 101, 101) which is pretty had to notice. According to your formula, my test efficiency is 0%. At the same time another tester did his planning, and the same number of scenarios. And he missed 100 major defects. His efficiency is also 0%. How in the world is that fair. Look Ruslan, I am not completely against metrics, they have their uses. I am not trying to argue here and put you down. But really with a calm mind think of the formula, may be it needs to be revisited. Or may be you are right, and I am completely lost and not making sense. In that case, my apologies and I leave you alone. Have fun.

[@QAMENTORTESTING]

@@khurrambhatti9470 if you see there are better ways to improve the formula please suggest. Mathematically if you have a value of ZERO you cannot do any division. In your particular scenario we all agree it is not fair, but you do not take numbers for granted. You will have analytical part done on them and you will question the number. Again, your example is a very rare case because you will always find defects during the testing cycle so you will never have Test Efficiency equals to zero. Do you agree?