Warning: count(): Parameter must be an array or an object that implements Countable in /home/public/wp-includes/post-template.php on line 284

He Who Can Not Be Tested

Well… No, you won’t get this prize either.
Read the previous BotT (about a bug in the lottery software) in order to be in the mood for this one.

This BotT tells a story. A story about a car dealership that had the great idea of mailing thousands of scratch-off games to promote their business. What they did not know is that, by mistake, all of the 50000 cards had a winning notice behind the scratch-off layer. Instead of having to pay a US$1.000 prize, they soon discovered that they had to pay US$50.000.000!!
Go to this link to read the story, or at least to see the videos of the angry customers that came to demand their US$1.000: Koat News link.

I am not pointing a finger at any culprit, it is clear that it was a big confusion. And although it is an interesting story, the point that I want to emphasize is: How do you test a system that can not be executed?

Open MatchboxYou can not scratch the Scratch-Off cards in order to ensure they are Ok. It would render them useless.
Think of it as a matchbox – once upon a time, there was a blackout on some neighborhood. The head of the family went to the kitchen to bring the matches, so they would have a little light. After failing to light even one single match, the man exclaims: “I can’t believe it! I tested each one of these matches just after I bought them!” :). You get the point: How do you test a system that can not be executed?

Oh boy, let’s open a new trend: Process testing.
By process testing I mean testing any deliverable, even if it is not software.

So, how would you test the Scratch-Off cards, without scratching the 50.000 cards? Here are some examples I can jot down quickly:

  1. Requirements Review: Ask if it is clear that 1/50000 should be prized. Ask again, and show it in the contract – if you show concerned enough, there is a high probability that the concern will spread to the publisher, and he will make sure to see the process is as expected.
  2. Black Box Statistics: Open 5 random tickets. If you found 5 winning cards, return all the boxes (there should be only one prize!). If you found one winning prize – try some more, statistically you’ll never be that lucky.
  3. White Box: Scrutinize the printing batch script, or whatever controls the machine. If you work together with someone from the publishing house that knows the language, you’ll be able to grasp exactly what percentage of winning cards is being printed.
  4. Interface testing: See a draft printout of the prized and a not prized card artwork. They can send it even, by email, and you will ensure that the publisher is using a unique artwork as model.
  5. Sample: Ask for a blank sample and for a prized sample. The rationale is similar to the previous one.
  6. Process Analysis: When the printed card boxes come, ask to be shown which box has the prized card. If all is well, there will be only ONE marked box. Legal disclaimer: It may be illegal to know where the winning prize is – as it would spoil the surprise and competition of the ‘game’, making it unfair.
  7. Legal Protection: Maybe you can also add protections clause to the card, like “my company is not responsible, rather the publisher company is” or “prize is dependant on XXX, where XXX is some other condition”.
  8. Marketing Protection: Another option will be tying the prize to a discount when buying a car (the scratch cards intention was to grab the attention of those 49.999 people who did not won the prize. With this discount-tie, the attention-grabbing would be the same or bigger, and the risk taken would be 1000 for car sold.).Not all of these techniques really protect you, but if done, these would give you enough confidence on the correctness of the data.

I could finish here, without talking about software, just to make the point of abstraction.
But as I want to keep my readers, and make my blog good for the world, let’s see some software points to ponder:

Is there software which can not be tested?
Well, I believe that although any software is testable up to some extent, yes there is software which can not be thoroughly tested.

The most common examples are from Critical/Safety systems.
Dr. Parnas refused to build a defense system software against nuclear missiles, stating that “The inability to test a strategic defense system under field conditions before we actually need it will mean that no knowledgeable person would have much faith in the system.“. He was right, the developers/testers would be able to test and simulate some parts of the system, but the final system as a whole would be impossible to test until a real war was in course and you had nuclear ogives being shot at you.
(You can not even test in at a desert location with test-intended-nuclear-bombs, because the Nuclear Test-Ban Treaty prohibits.)

==> So how would you test software like this? Or would you, as Dr. Parnas’s approach, just refuse to write and test it?

Another kind of untestable program I found in the Wikiwikiweb of Software Development:

Imagine its a few years ago, and the application under tests is a program that has a feature to disprove Fermat’s Last Theorem. Within the code, the function for itย is:

bool fermatIsWrong(x, y, n) {
// return true if n > 2 and nth root of (x^n + y^n) is an integer
// else return false

Little FermatAt the time you are testing this function, you do not know whether there exists a set of numbers {x, y, n} which lead to a true result. And if the set exists, neither you nor anybody else knows which the values are.
So, this is a program to which you will never be able to cover 100% of the code paths in your tests. You’ll be able to test extensively the ‘false’ return value, though… ๐Ÿ™‚

==> Again, how would you test a software like this?

Feel free to use the comments below to reply. Let the world know your Testing Thoughts!

7 Responses

  1. […] Aspects of Strategic Defense Systems” by Dr David Lorge Parnas. I commented about it on a previous post about hard-to-test software. This paper taught me about the difficulties of software testing and the responsibilities it […]

  2. […] Now, here is more about testing untestable systems. Please read “He Who Can Not Be Tested” and “He Who Will Be Tested!” before you continue, if you want to be in the mood. […]

  3. […] My past post on ‘apparently untestable systems‘ turned out to be a very cool experience. I talked with a lot of friends about it, and also had a very informative thread running on the comp.software.testing usenet group. The discussion was called “Is every single program testable? How do you deal with ‘untestable’ software?” […]

  4. ๐Ÿ™‚
    Pradeep, thanks for honoring me with your visit. A follow-up post to this post is on it’s way, and it would be cool to insert your opinions too — post what you think about this, if you can.

    (Know Pradeep: He works at Satisfice with James Bach, and is a very experienced blogger (almost two years). I have a link for his blog in the sidebar list.)

  5. I enjoyed reading this post a lot. I would want to continue the enjoyment for a little while or even more and then come back after a day or two ( if I remember I have to come back ) to post my thoughts.

    Thanks a lot for this wonderful piece!

  6. […] And thanks to all those who commented on the previous post: Weblog Qjm.be, Shmuel Gershon, Jean Browman, Rosa Say, Karen Lynch, Edward Mills, Bruce Ramsey, Auto Parts for Brains, Michelle Van, Riyueyi, Carol Webb, Joanna Young, Joseph Down, Rhett, circulating, jdr, Gailtrail, JoLynn Braley, Tshombe, dailytri, Shawn, Henri van den Hoof Share This Tags: awareness, focus, potential […]

  7. The Scratch-Off story remembers me a joke:

    A young man bought a donkey from a farmer for $100.00. The farmer agreed to deliver the donkey the next day.
    The next day the farmer drove up and said, “Sorry son, but I have some
    bad news, the donkey died.”
    The young replied, “Well, then, just give me my money back.”
    The farmer said, “Can’t do that. I went and spent it already.”
    — Ok, then, just bring me the dead donkey.
    — What ya gonna do with him?
    — I’m going to raffle him off.
    — You can’t raffle off a dead donkey!
    — Sure I can. Watch me. I just won’t tell anybody he is dead…

    A month later the two met again.
    “What happened with that dead donkey?”
    The youngster said, “I raffled him off. I Sold 500 tickets at two dollars a piece and made a profit of $998.00.”
    The farmer said, “Didn’t anyone complain?”
    — Just the guy who won. So I gave him his two dollars back.

Leave a Reply

© Shmuel's Software Stuff
%d bloggers like this: