Well… No, you won’t get this prize either.
Read the previous BotT (about a bug in the lottery software) in order to be in the mood for this one.

This BotT tells a story. A story about a car dealership that had the great idea of mailing thousands of scratch-off games to promote their business. What they did not know is that, by mistake, all of the 50000 cards had a winning notice behind the scratch-off layer. Instead of having to pay a US$1.000 prize, they soon discovered that they had to pay US$50.000.000!!
Go to this link to read the story, or at least to see the videos of the angry customers that came to demand their US$1.000: Koat News link.

I am not pointing a finger at any culprit, it is clear that it was a big confusion. And although it is an interesting story, the point that I want to emphasize is: How do you test a system that can not be executed?

Open MatchboxYou can not scratch the Scratch-Off cards in order to ensure they are Ok. It would render them useless.
Think of it as a matchbox – once upon a time, there was a blackout on some neighborhood. The head of the family went to the kitchen to bring the matches, so they would have a little light. After failing to light even one single match, the man exclaims: “I can’t believe it! I tested each one of these matches just after I bought them!” :). You get the point: How do you test a system that can not be executed?

Oh boy, let’s open a new trend: Process testing.
By process testing I mean testing any deliverable, even if it is not software.

So, how would you test the Scratch-Off cards, without scratching the 50.000 cards? Here are some examples I can jot down quickly:

  1. Requirements Review: Ask if it is clear that 1/50000 should be prized. Ask again, and show it in the contract – if you show concerned enough, there is a high probability that the concern will spread to the publisher, and he will make sure to see the process is as expected.
  2. Black Box Statistics: Open 5 random tickets. If you found 5 winning cards, return all the boxes (there should be only one prize!). If you found one winning prize – try some more, statistically you’ll never be that lucky.
  3. White Box: Scrutinize the printing batch script, or whatever controls the machine. If you work together with someone from the publishing house that knows the language, you’ll be able to grasp exactly what percentage of winning cards is being printed.
  4. Interface testing: See a draft printout of the prized and a not prized card artwork. They can send it even, by email, and you will ensure that the publisher is using a unique artwork as model.
  5. Sample: Ask for a blank sample and for a prized sample. The rationale is similar to the previous one.
  6. Process Analysis: When the printed card boxes come, ask to be shown which box has the prized card. If all is well, there will be only ONE marked box. Legal disclaimer: It may be illegal to know where the winning prize is – as it would spoil the surprise and competition of the ‘game’, making it unfair.
  7. Legal Protection: Maybe you can also add protections clause to the card, like “my company is not responsible, rather the publisher company is” or “prize is dependant on XXX, where XXX is some other condition”.
  8. Marketing Protection: Another option will be tying the prize to a discount when buying a car (the scratch cards intention was to grab the attention of those 49.999 people who did not won the prize. With this discount-tie, the attention-grabbing would be the same or bigger, and the risk taken would be 1000 for car sold.).Not all of these techniques really protect you, but if done, these would give you enough confidence on the correctness of the data.

I could finish here, without talking about software, just to make the point of abstraction.
But as I want to keep my readers, and make my blog good for the world, let’s see some software points to ponder:

Is there software which can not be tested?
Well, I believe that although any software is testable up to some extent, yes there is software which can not be thoroughly tested.

The most common examples are from Critical/Safety systems.
Dr. Parnas refused to build a defense system software against nuclear missiles, stating that “The inability to test a strategic defense system under field conditions before we actually need it will mean that no knowledgeable person would have much faith in the system.“. He was right, the developers/testers would be able to test and simulate some parts of the system, but the final system as a whole would be impossible to test until a real war was in course and you had nuclear ogives being shot at you.
(You can not even test in at a desert location with test-intended-nuclear-bombs, because the Nuclear Test-Ban Treaty prohibits.)

==> So how would you test software like this? Or would you, as Dr. Parnas’s approach, just refuse to write and test it?

Another kind of untestable program I found in the Wikiwikiweb of Software Development:

Imagine its a few years ago, and the application under tests is a program that has a feature to disprove Fermat’s Last Theorem. Within the code, the function for it is:

bool fermatIsWrong(x, y, n) {
// return true if n > 2 and nth root of (x^n + y^n) is an integer
// else return false
}

Little FermatAt the time you are testing this function, you do not know whether there exists a set of numbers {x, y, n} which lead to a true result. And if the set exists, neither you nor anybody else knows which the values are.
So, this is a program to which you will never be able to cover 100% of the code paths in your tests. You’ll be able to test extensively the ‘false’ return value, though… 🙂

==> Again, how would you test a software like this?

Feel free to use the comments below to reply. Let the world know your Testing Thoughts!