Hi Again!

My past post on ‘apparently untestable systems‘ turned out to be a very cool experience. I talked with a lot of friends about it, and also had a very informative thread running on the comp.software.testing usenet group. The discussion was called “Is every single program testable? How do you deal with ‘untestable’ software?”

That thread was so helpful to further my understanding of testing, that I’ll put here a summary. It surely will make a good reference for me on the future, and I hope this will be useful for others too.

So, in order to continue and understand, read the ‘He Who Can Not Be Tested‘ post. All the discussion started from the questions in there:
– Do you think there are types of software which cannot be tested to a satisfactory degree?
– So how would you test software like this (of Dr. Parnas)? Or would you, as was his approach, just refuse to write and test it?

I’ll condense the responses here, and will add a note (and maybe a link) about each person. Note: The responses are edited and rearranged from the original replies, not really verbatim (although I changed the wording very little).

1)
H.S. helped a lot. He has a weblog were he discuss matters of development and test in depth, which was not updated lately, but there is still information to be found — hopefully he’ll add more content soon.

> Define ‘satisfactory’. B-)

Actually, all the responses we will see can be shortened to this question. Until we do not (re)define the satisfaction we are aiming to, we can not continue with our test approach.

> Yet another set of problems involves results that are not predictable. An example would be a module to shuffle a deck of cards for a game. One cannot directly test the randomness of the shuffle implementation; one must resort to statistical sampling and analysis.

(This is similar to the Fermat question in my previous post.)

> In fact, one can argue that exhaustive testing of any large application is simply not feasible because of combinatorial problems between inputs and logic paths.
> OTOH, if one is using testing to simply monitor the development process, then it may be possible to test any complex program, even nondeterministic ones. One can build a high degree of confidence in the quality of the development process (e.g., defect prevention) itself.
> Shops doing mission critical software do not use testing as a product reliability gate. They depend on defect prevention in the development process and only use testing to ensure the process is working correctly.
There is also a cool explanation of the testing role in the development process. Go and read it at the usenet.

2)
Ernie Englehart explained in a very practical way:

> One of the key components of a good test plan is a risk analysis. The calculation of impact x likelihood gives a risk factor. We then prioritize our test cases to focus on the highest risk items first.  By prioritizing the test cases based on risk we are mitigating and minimizing the risk as much as we can.
> For defense systems, obviously much of the testing needs to be performed with simulators. The design and functionality of those simulators is obviously very important. They can be very complex. The simulator projects follow the same methodologies of performing a risk analysis and then prioritizing the test cases accordingly.

3)
A response from Vladimir:

> To prove the concept you may not need to execute every single line of your code in every possible condition with every possible data value. Testing is about finding the least that covering the most (or even everything in the mission critical systems). Testing is also about probability. The more [of smart] testing you do the least probability is there that you have missed a defect.

4)
And I was much honored to receive a reply from Michael Bolton. He is one of my favorite authors since my beginnings on Software Testing. See his site (I keep him in my sidebar) and read his articles.

> Obtaining or attaining confidence in something is very different from testing something.  The testing of a product might be entirely satisfactory, where the information revealed by testing will reveal that the product is not satisfactory.
> The decision to build or not to build something isn’t a testing decision; it’s an engineering decision.  Engineering frequently involves building stuff that gets its first end-to-end system test on its first use.  Testing is a heuristic process that tells us things about the state of our engineering.  Some things don’t work even after extensive testing, and some things do work even after very superficial testing.

5)
Some conclusions reached on the usenet thread:   

> If a system was decided to be built, we can test it.
> There’s a difference between product satisfaction and testing satisfaction. One outcome of a satisfactory testing can be a conclusion that a product will not be satisfactory.
> A degree of confidence should be agreed between all the involved, based on the project limitations and testing constrains.
> As no system is 100% good and trustworthy, we can reach a result that defines a system as “good and trustworthy to an agreed level of coverage, based on a known analysis of the risks”. Such objective can always be reached, concerning testing.
> “Difficult” does not equal “impossible”!

Hope this collection of ideas is useful to you too.
Please feel free to add your comment, or to ask for more information on any of the ideas mentioned.