Testing methods

Here I would lkie to give a short review of the article called "Evaluating testing methods by Delivered Reliability" as I find its approach to the testing process to be very interesting and slightly different from the classical one.

The paper states that there are two majour approaches in testing:
a) "debug testing" where the objective is to probe the software for defects so those can be removed in order to achieve adequate quality
and
b) "operational testing" where the objective is to gain confidence that the software is reliable assessing its existing quality.

Although both methods look very similar, that is a devil in details. The debug testing concentrates on probing all possible (available) features and their combinations and the operational testing concentrates on "real life" scenarios of using the software.

In the result the debug testing covers sufficiently more during the test process, but could actually lead to finding mostly unimportant bugs and force the team to spend valuable resources on fixing those (leaving us without test/dev team time and resource to test more important cases).
Here the importance of a bug is calculated as a probability that the bug case will appear in the real use of the software and if the probability is very low, then the bug is unimportant.
Notice that in the result of this, we can even state that sometimes the debug testing will find bugs, but will not lead to the real quality improvement as the massive low importance bug fixing is not worse to do in compare to fixing one real case bug.

The debug testing by its nature is an active hunting for bugs, while the operational testing is a passive one: test cases are just executed and testers are waiting for a bug to happen, rather than trying to imagine cases that could lead to errors.

As we are dealing with selection of scenarios to be used in the operational testing, we can say that the operational testing is a probabilistic testing: it selects a set of test cases to be used from all possible combinations basing on the probability of each one to appear in real customers processes (use). Therefore this kind of testing greatly depends on experts who are assigning those probabilities or write required real-life use cases.

Finally we would like to mention that nowadays wide believe that the testing is a very expensive and time/resource consuming process is produced mainly by the fact that industry uses exclusively the debug testing. There is just a very minor number of companies where the operational testing is applied, although the common shortage of resources should obviously stimulate wider use of the operational testing approach.