A couple of months ago I tried to access a site (now I don’t even remember which it was) and was greeted by the note below:

Let’s read that again: “The attack was not successful and our safeguards prevented intrusion“. Ah, nice. But Mr BudgetHostingWeb, can you explain how, if the intrusion was prevented, how the hell did the “server's hard drives failed during this process“?

Maybe the hard disk sensed that an attack was being attempted, and then committed suicidal, in some sort of self-aware sense that “it's better to die than have those hackers get the secret data stored in me!”

(Side note: If you ever had to deal with web hosting service’s backup, you know it is a nightmare. A service I used once had “hard drives problems” too and their last backup was a month old — even if they had claimed to do daily backups. No, I didn’t sue, and I think I wasn’t even surprised (Quality is Dead, right?), but I did change to a different host later.)

So now we have customers who lost all the data of his website.
If they had good backups, they’ll be able to upload it with the FTP Upgrade package — and if they hadn’t, well, will have to start from scratch.
And this is being communicated to him in such a lousy way… Note that the alert in the page is not targeting the site visitors (which do not upload files by FTP), it is targeting the web page owners. Why would an alert to the page owner be glued to the home page of a customer? Maybe because they didn’t send the communication in any other way?
(Maybe the hard drive with the customer information failed too, who knows?)

And the alert says “the attack was not successful“, which is what makes one (me) angry.
Mr BudgetHostingWeb, your customers are unhappy, they had a major data loss, and have to fix it themselves instead of doing business. Please re-write the message to something like “The attackers succeed to cause failures in some of our hard drives, and unfortunately we don't kept backups of your data. As far as we can discern, no information was stolen, but we are deeply sorry for this inconvenience.”

Because shaking yourself free of responsibility isn’t exactly the best experience you can provide your users.

Let’s sympathize with one of their customer, which is a great practice for software testers:

I did a search on Google to try to find sites that were part of the problem. Even thought a lot of time passed since the incident, Google found three (two of them do not have the text in their sites anymore, Google is outdated too, so it leaves us with only one customer specimen, Free Range Whippets):

Whippets are lovely pretty dogs, and the first link on the screenshot above sports the following notice:
 ”
 We regret that our host server was recently the victim of a serious hacking attack.
 The attack was not successful and the safeguards prevented intrusion however the server's hard drives failed during this process.
 All services have now been restored.  Unfortunately, our site was one of those affected, and we need to upload ALL the files again.
 Some links and photos will not work or appear until we re-upload.
 We hope to be fully back online soon but finding after six years of loading photos, they don't go back up overnight.
 Please visit us again and click all the links! 
 Thank you, Lori & The Whippets
 ”

6 years of work, gone! Is that what the safeguards were supposed to do?
The hacker attack wasn’t successful, but Lori still has got broken links in the whippets’ images .

The last BotT (Bug of this time) was long ago, when we talked about testing and the Excel bug.
So now we’ve got a cool one, on which the most notable point is not the bug, is the submitter .

Bug 439858 on Fedora (a Linux distribution) was (supposedly) submitted by Linus Torvalds himself. He started the Linux wave on August 91, and is still posting bugs on March 08 – you gotta admire him.
I am aware that the submitter signing “Linus Torvalds” may not be Linus. It actually doesn’t matter, let’s just pretend it is for sensationalism’s sake (regarding the admiration, you still gotta admire him even if the bug’s not his ).

Let’s try to analyze the bug report, and the bug itself, to see what we can learn:

Linus uses a lot of witty comments. The bug starts as “youtube no workee“, which is a nice way of writing things in a forum or a youtube post, but we should not use such mocking language in our bugs – jokes or irony in bug descriptions can offend developers, who already have a hard timing accepting their code has got errors. Cem Kaner alerts: “it is easy to read a joke or a remark as a flame”.
You should not do this especially if you are a leader, influent people or the inventor of the OS — you are an example, why foment bad practices?

Then there is “fedora 9 not usable for wife“. That’s actually a good point. It is easy for programmers to remember that the average user is not tech-savvy and won’t tolerate complex workarounds. If this is true for common commercial (and Windows) application (where the programmer is interested in the sales to a large audience that expects intuitive software), this can be even more true in the Linux community (where they’ve got an incredible good will and skills, but sometimes prefer to add features over simplicity).
So explaining the Customer Impact in clear language at the beginning of your bug is a very good idea.
Moreover, I don’t think anyone would want to frustrate Mrs. Linus Torvalds. First because she doesn’t deserve it, being a nice person with a child-care background, and second because she’s a black-belt Karate champion .

“Steps to Reproduce:
1. Install current Fedora 9
2. Rick-roll!
3. No profit!”
Now, these steps to reproduce aren’t very useful. Even the title of the bug is more descriptive: “swf mozilla plugin – no youtube“. Everyone laughed at the mention of rick-rolling, but it only added confusion: is this bug related only to the Rick-Roll video? Any other video?
Linus added specific info to clarify: “I didn’t try a lot of videos, but I couldn’t find a single one that actually worked“. This point is actually a very good one, which teaches an important principle on bug reporting: There are points that are “conditions” to the bug (the bug only reproduces with the conditions fulfilled), and there are points that are ”incidental” to the bug (a setup or configuration that was true at the time of the bug but we cannot be sure it is part of a condition). It is very important to note the distinction between these when submitting a bug. And, of course, any setting which is neither incidental nor conditional should in most cases be omitted.

“Some videos just show a light gray background, some give the play buttons etc,
but in the latter case show only a black screen even when the red ball at the
bottom seems to moves along..“. Good! Information that is really related to the bug! Adding the link of videos that show the gray background and videos that show the buttons would be of help here. You (the tester) already visited all of them, why should the programmer search for specific reproductions again?

There’s some more information in there. The information is a sum of statements that help debugging (“youtube videos are supposed to work with swfdec“, “the adobe player won’t install on current rawhide“), statements that define the impact (“wife will kill me if she doesn’t have her videos“) and humorous comments. When you report your next bug, remember to separate these into clear sections: a section to discuss the user impact, and a section for additional debugging info that can be helpful. The humorous section you can leave out .

Le Grande Finale:
““Obi-wan Kenobi, you’re our only hope”“. He actually finalized with a Star Wars quote. Which is very cool if you’re the founder of an open source OS.
They should print this bug as a book. It mixes hackers, nerds, memes, Star Wars and jokes – receipt for a best seller!

—–

Before you post hate comments on how I dare to criticize Linus: I don’t. He’s a hero of mine and accomplished incredible things. I just used his bug for didactic purposes.

Uh, you didn’t notice? Well neither did I until I read it all over the internet.
See the post on SlashDot for scoop, and see its comments for some good laughs.

There are explanations all around about how this bug came to appear in Excel.
The best one is probably Joel’s one (you may remember Joel from this post).

Go and read it. Joel is very deep in there and explains why the binary calculations can be confusing, and how the binary notation standards may or not help you developing better software.

I say “go and read it” because I’ll not repeat his explanation here
I will, however, quote the best parts from the questions at the end — very insightfull!

First insight is a matter for pondering about bugs, small bugs and the perils of fixing such:

Q: Isn’t this really, really bad?
A: IMHO, no, the chance that you would see this in real life calculations is microscopic. Better worry about getting hit by a meterorite. Microsoft, of course, will be forced to tell everyone “accuracy is extremely important to us” and I’m sure they’ll have a fix in a matter of days, and they’ll be subjected to all kinds of well-deserved ridicule, but since I don’t work there I’m free to tell you that the chance of this bug actually mattering to you as an individual is breathtakingly small.
(…)
And let’s face it — do you really want the bright sparks who work there now, and manage to break lots of perfectly good working code — rewriting the core calculating engine in Excel? Better keep them busy adding and removing dancing paper clips all day long.

Second insight, on the perils of automated tests (an automated test sees only what you ask it to see, hence it has blind spots and will not report for side-effects!).
(Of course, as Joel says, it is doubtfull that manual tests would ever exercise rare/random calculations):

Q: Shouldn’t they be testing for these kinds of things?
A: I’ll bet that most of the numeric testing done on the Excel team is done automatically with VBA code. Cells containing this value display as 100,000, but from VBA, they’re going to look like 65,535 (since the number would be passed into the Basic runtime in binary, before the display formatting.) I’m sure there’s plenty of code to test display formatting, but with a bug like this that only happens on 12 out of 18446744073709551616 possible floating point binary numbers, it’s unlikely that any set of black-box tests would cover this case.

This BotT tells a story. A story about a car dealership that had the great idea of mailing thousands of scratch-off games to promote their business. What they did not know is that, by mistake, all of the 50000 cards had a winning notice behind the scratch-off layer. Instead of having to pay a US$1.000 prize, they soon discovered that they had to pay US$50.000.000!!
Go to this link to read the story, or at least to see the videos of the angry customers that came to demand their US$1.000: Koat News link.

I am not pointing a finger at any culprit, it is clear that it was a big confusion. And although it is an interesting story, the point that I want to emphasize is: How do you test a system that can not be executed?

You can not scratch the Scratch-Off cards in order to ensure they are ok. It would render them useless.
Think of it as a matchbox – once upon a time, there was a blackout on some neighborhood. The head of the family went to the kitchen to bring the matches, so they would have a little light. After failing to light even one single match, the man exclaims: “I can’t believe it! I tested each one of these matches just after I bought them!” . You get the point: How do you test a system that can not be executed?

Oh boy, let’s open a new trend: Process testing.
By process testing I mean testing any deliverable, even if it is not software.

So, how would you test the Scratch-Off cards, without scratching the 50.000 cards? Here are some examples I can jot down quickly:

1. Requirements Review: Ask if it is clear that 1/50000 should be prized. Ask again, and show it in the contract – if you show concerned enough, there is a high probability that the concern will spread to the publisher, and he will make sure to see the process is as expected.
2. Black Box Statistics: Open 5 random tickets. If you found 5 winning cards, return all the boxes (there should be only one prize!). If you found one winning prize – try some more, statistically you’ll never be that lucky.
3. White Box: Scrutinize the printing batch script, or whatever controls the machine. If you work together with someone from the publishing house that knows the language, you’ll be able to grasp exactly what percentage of winning cards is being printed.
4. Interface testing: See a draft printout of the prized and a not prized card artwork. They can send it even, by email, and you will ensure that the publisher is using a unique artwork as model.
5. Sample: Ask for a blank sample and for a prized sample. The rationale is similar to the previous one.
6. Process Analysis: When the printed card boxes come, ask to be shown which box has the prized card. If all is well, there will be only ONE marked box. Legal disclaimer: It may be illegal to know where the winning prize is – as it would spoil the surprise and competition of the ‘game’, making it unfair.
7. Legal Protection: Maybe you can also add protections clause to the card, like “my company is not responsible, rather the publisher company is” or “prize is dependant on XXX, where XXX is some other condition”.
8. Marketing Protection: Another option will be tying the prize to a discount when buying a car (the scratch cards intention was to grab the attention of those 49.999 people who did not won the prize. With this discount-tie, the attention-grabbing would be the same or bigger, and the risk taken would be 1000 for car sold.).
    
   Not all of these techniques really protect you, but if done, these would give you enough confidence on the correctness of the data.

I could finish here, without talking about software, just to make the point of abstraction.
But as I want to keep my readers, and make my blog good for the world, let’s see some software points to ponder:

Is there software which can not be tested?
Well, I believe that although any software is testable up to some extent, yes there is software which can not be thoroughly tested.

The most common examples are from Critical/Safety systems.
Dr. Parnas refused to build a defense system software against nuclear missiles, stating that “The inability to test a strategic defense system under field conditions before we actually need it will mean that no knowledgeable person would have much faith in the system.“. He was right, the developers/testers would be able to test and simulate some parts of the system, but the final system as a whole would be impossible to test until a real war was in course and you had nuclear ogives being shot at you.
(You can not even test in at a desert location with test-intended-nuclear-bombs, because the Nuclear Test-Ban Treaty prohibits.)

==> So how would you test software like this? Or would you, as Dr. Parnas’s approach, just refuse to write and test it?

Another kind of untestable program I found in the Wikiwikiweb of Software Development:

Imagine its a few years ago, and the application under tests is a program that has a feature to disprove Fermat’s Last Theorem. Within the code, the function for it is:

bool fermatIsWrong(x, y, n) {
  // return true if n > 2 and nth root of (x^n + y^n) is an integer
  // else return false
}

At the time you are testing this function, you do not know whether there exists a set of numbers {x, y, n} which lead to a true result. And if the set exists, neither you nor anybody else knows which the values are.
So, this is a program to which you will never be able to cover 100% of the code paths in your tests. You’ll be able to test extensively the ‘false’ return value, though…

==> Again, how would you test a software like this?

Feel free to use the comments below to reply. Let the world know your Testing Thoughts!

Do you play lottery? Did you ever win?
No?? Well, maybe the reason of you not being yet a millionaire may be a nasty bug in your favorite lottery system…

In the B.C. Lottery Corp news page, the company explains and excuse the problem:

The BC Lottery Corporation wishes to notify Lottery players who use BCLC’s Internet Number Checker to check lottery results that a technical issue has occurred. On February 8, a software upgrade inadvertently prevented draw results from being presented correctly by secondary BCLC web addresses.

Those who checked tickets on sites other than www.bclc.com were informed that the Internet Number Checker had not detected a match to the official winning numbers. BCLC is concerned that some of these players may have, in fact, held a winning ticket.

Oh… YOU are concerned? I am concerned! Now let me find that old ticket…! ;)

More information in these news:

• http://www.cbc.ca/canada/british-columbia/story/2007/04/06/bc-lotto-glitch.html
• http://www.canada.com/vancouversun/news/story.html?id=1bc10553-328d-436a-961d-ffe1e3d264d1&k=41924

So, innocent people bought lottery tickets to fulfill their dream, but when checking in the automatic services whether they won or not, they may have received a ‘Looser’ message when they were entitled for a helluva lot of cash.

What does that says about B.C. Lottery Corp.? What does this says about their development staff – - and their testers?

A bug like this have uncountable bad effects, which range from lost of customer trust (!), to possible loose of Lottery Business Permit. Read, understand and beware: Don’t let your company be the next to frustrate customers and make the front page of newspapers with bad news!

This bug was also featured on StickyMind’s Between-the-Lines as Bug of the Month. And on their news site, here you have: http://www.stickyminds.com/newscenter.asp

I picked a bug from the FireFox BugZilla database (BugZilla is a (free) (opensource) (reliable) tool for tracking bugs).
The bug is Bug 233853 – cnn.com, Ctrl+Enter goes to http://www.cnn.com.com/.

This bug is very particular. Let’s first analyze the Bug Description:

User-Agent:
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040207 Firefox/0.8

I believe that’s good enough for a Build Identifier. It is the common practice on the DataBase to provide this info (check for yourself). And, from what I’ve seen, no much information on the system OS is given until asked by the programmers.
My belief on where this comes from: From one side, basic information on the environment can be very healthy for bug debugging. But on systems such as this, where the parameteres can change _so_ much from user to user, and people all over the world are running this application with different settings in different operating systems on different computers to do different things… Well, with so much differences, you better leave for the programmer to chose which information he wants .

* Enter ‘cnn.com’ in the URL field.
* Press CTRL-Enter
* Firefox changes the URL to ‘www.cnn.com.com’.  Firefox prepends ‘www’ and
appends ‘com’ to whatever URL I type.
This does not happen if I provide a protocol or start the URL with ‘www.’

That is a very simple bug, with very simple reproduction steps.
It happens all the time to a lot of people, but Rick was the first to decide and report it.
Anyone can reproduce it from these instructions.

I pressed CTRL-Enter as I’m used to opening links in a background tab in
Mozilla.  I guess Firefox does not yet support opening background tabs.

Reproducible: Always
Steps to Reproduce:
1. Type a URL (sans protocol & ‘www’) in the URL field. (e.g. cnn.com)
2. Press CTRL-Enter
3. Notice Firefox prepends ‘www.’ and appends ‘.com’ to whatever URL you entered
without first attempting to connect to the URL provided.

Actual Results: 
Firefox prepends ‘www.’ and appends ‘.com’ to whatever URL you entered.

Expected Results: 
Firefox should just attempt to connect to the URL as typed.

This is an extension of the first-and-quick reproduction process.
It has additional information (like the background tab statement), a reproduction page that is not specific (in order not to have the discussion limited to cnn.com) and the expected results.

Pretty good!

Wait, if it is pretty good, and simple, and quick, why it is featured as the BotT?
Answer: Take a look at the bug thread — it has 46 posts, since 11/02/2004 until 16/10/2006! In almost three years, the bug was rolling back and forth between sides, until they get to some conclusion.

The culprit is that there is no definition on what the exact behavior should be.
Let’s quote some comments from the bug:

• “Why the change from Ctrl+Enter that’s currently implemented in Mozilla?  ..and what is Ctrl+Enter supposed to do?“
• “It’s supposed to pretend to be IE and prepend http://www/. and append .com“
• “Just because it’s in IE doesn’t mean we should copy it“
• “You Ctrl+Click to open a link in a tab?  What about Ctrl+Enter on a link that has focus in the page?“
• “So now we have two choices of behaviour…“
• “Maybe add ftp: and https: ?“
• “What happens if a site name exists in two or more TLDs?“
• …
• …

The moral of this story:
You can not develop what you don’t have defined. More than that, you can not test what was not defined before being programmed.

This is a common disease in Open Source projects, because usually all they have is a good idea, and good programmers. There is no one that writes down exactly what the programmers need to develop and how.
These are one of the most important reasons to have a Benevolent Dictator in the development team of Open Source projects.

Not only Open Source suffers from lack of requirements.
Most software, commercial or not, lacks good requirements in different extents, and usually one is lucky if there is a specific document that defines them.

This feature we are dealing in the bug is a very confusing one, that can be origin to many discussion and polemics.

For instance, I did a quick search and found this page about auto-completion on IE7. They talk about pressing Ctrl+Enter, Shift+Enter, Shift+Ctrl+Enter, or any of these combinations with Alt.
The shortcuts using Shift or Ctrl+Shift presented in that site do not work on my standard installation of IE7. But Ctrl and Alt do work, so unless there’s a separated checkbox in the IE options for each autocomplete-shortcut, something weird is going on (probably they have a different plugin). One way or another, this is another prove that this feature is vague, and both developing or testing it is painful for its vagueness.

If you use Internet Explorer (I did the switch back from Firefox a while ago), you may want to know that you can change the way Ctrl-Enter behaves in your machine! See this link in the MVPS network for more details.
For FireFox, you can use the ‘browser.fixup.alternate.suffix’ in about:config.

This type of customization is one of the answers found to have a well-defined behavior on the shortcuts. “If the shortcut is not clear, then let each user decide what the command do.”, “If no one is sure what is the best result, let every user decide his own behavior!”. Seems ridiculous, but it is true.

You can see long-lasting threads on requests like this for Mozilla and for FireFox. Threads on this last years and end without resolution.

Wait! Should not this section be called “Bug of the Month“, or “of the Week“? No, no. I do not want to limit myself to just one bug per period. Neither I want to commit to bring a bug every week… So it’s the “Bug of this Time“!

At “This Time”, instead of listing a bug found elsewhere, I’ll describe a ‘Privacy Bug’ I once found in the printer of a company I worked for.They had some Xerox printers that performed as Fax as well.

In some (most) of the machines, when you wanted to print a Transmission Report on a delivered fax, you had to drill-down on the printers’ menu, and press the “Transmission Report” option. Send your fax, and after success, a neat report, with a reduced image of the first page of the fax sent is printed out. Cool.

Other of the machines, had a very similar work flow, but to get the report you had to press the “Transmission Report” entry after you sent the fax.
A nice day, I was preparing to send my fax on this machine, and did all the “Report Flow” before sending the fax (just as I would do in the normal machines). Surprise!
The machine printed immediately the Acknowledgement Report of the last fax sent, some day ago. As the report comes with a nice reduced picture of the fax, I was presented with personal information of one of my co-workers .

Of course, I shredded the info right away, and warned my colleague of what happened.

My tester soul pushed me to send my fax and press the option a number of times, and every time the same report was printed. The printer machine did not deleted the info from its memory after being used. So if someone wanted, he could simply come once in a while and print out al the faxes sent by anybody.

This is a very high privacy concern in my opinion, as these info can as well be Credit Card numbers or delicate health matters.

Expected Behavior from a Fax machine like this one (one built to be used on a SoHo shared space):

1. The machine should save the report information (front-page picture) _only_ if the choice was made through the menu _before_ sending the Fax.
2. After printed once, the picture should be deleted.

That’s my Bug of this Time.
If anybody sends me his, I’ll feature it as well. It is not hard at all, there are bugs everywhere! Let me know your Testing Thoughts!