Last month I visited the STAR East conference. It was over two weeks ago, which on the internet age makes it old news.
I follow my own clock, however, and will start posting about STAR East only now. Scott Rosenberg would call this type of blogging ‘history’ instead of ‘journalism’ (link), and it sounds just as fine.

In next blog posts, I’ll write about what happened at STAR. What happens in Orlando doesn’t stay there.
But on this one, it’s a bit about what happened in the way there.

For some time before travelling, I’ve been thinking about physical problems that resemble computer bugs. “Spirits in the material world”. For example:

In this picture you can (barely) see a water bottle that was forgotten by a worker on my neighbors’ roof. Just like a programmer would forget a temporary variable she invented to get some refreshment from the code inherent limitations, a worker will forget his own refreshment once it isn’t needed anymore.

This other picture is of a bathroom urinal row. I bet the urinals looked beautiful in the catalog, with their slim form that doesn’t stand out of the wall too much. But when “implementing” the bathroom, not only the beauty is lost when you build the necessary dividers, but the very same low-footprint of the urinals makes the rest of the environment unclean. How many times a company/developer makes a bad decision of, say, a coding library, just to discover that the very ‘flexible’ API promised by it is the part that gives them most trouble? How many times they decide to retrocede the and change the library? Ok, ok, tasteless example.

It seems that the same thinking process that makes a programmer/tester miss a detail when coding/testing happens in any other aspect of real life. I’m reading “Why we make mistakes” by Joe Hallinan and this idea seems to be congruent with many examples.

Well, with that mindset, it’s a sure thing that I would find bugs on my trip. Not because the trip was bad — au contraire, it was a mostly pleasant journey. But analyzing things and trying to assess their value is a good exercise for testers.

To make the exercise better, I decided to categorize the bugs and problems into the 8 parts of the HICCUPPS mnemonic consistency heuristic for analyzing a bug. It is a great way of building yourself an oracle to assess a potential problem, and I learned about it first from Michael Bolton‘s “Testing without a Map” article, highly recommended.

A quick explanation of the heuristic: One method of assessing any phenomenon is by comparing its consistency to a different (but comparable) source. The 8 aspects this mnemonic suggests are:

Is the phenomenon consistent with a comparable idea in:
H	History (e.g. the same function in the past)
I	Image (e.g. what the company would like customers to think of it)
C	Comparable products (e.g. the competitor’s )
C	Claims (e.g. a user manual statement)
U	Users’ expectations (e.g. what users have become used to)
P	Product itself (e.g. the same function elsewhere)
P	Purpose (e.g. the aim of this very function)
S	Statutes (e.g. a certification check)

To illustrate better the method and to share the fun of travelling, here goes a list of interesting events encountered on the trip:

[H - History] Airlines/Airports are stingier than in the past

It’s been a long time since I last travelled. So my recollection on how it works left me with some expectations, like having free carts at the airport, or having free peanuts during short flights, or having free headphones in the long ones.
Surprise: A cart costs US$4.00, a headphone costs US$5.00, and peanuts only if you brought from home! I was lucky I had my headphones in the carry-on case, but would be disappointed if I had to pay for airline ones.

This is interesting: Is that a problem? Is it a bug? Well, certainly it is inconsistent with the (far) history of tourist aviation, but it may be a good business decision. And fixing a bug (or deciding something is one) is a business decision.

[I - Image] Airlines seem to content with poor software

US Airways have a personal in-flight video system for each passenger (on long flights). It is great, has a touch-screen and showcases movies, music, even TV shows.
But the system is slow. It’s so slow that you often get confused by what it is doing — Is it stuck? Should I press again? Which of my last clicks is he processing now?

A funny thing is that at the beginning of the flight, the attendant announces: “please expect a delay after pressing the buttons on screen, it is part of the normal operation of the system”. What? In the age of iPad and other rapid touch-screens, is it really part of normal operation? I observed the other travellers, and saw at least two that were frantically pressing the screen waiting for answer.
We all also hope that the cockpit controls respond faster…

This one has two interesting points:
- I had a glimpse of the system rebooting when the aircraft landed. It is a Linux based system, and has a quote in its boot sequence that would make any software tester laugh (I laughed): “All bugs added by David S. Miller” , which turns out to be a real line in the Linux Kernel (David is a programmer for RedHat). Great attitude, Dave!
- US Airways have, apparently, tested a different in-flight system, that is not available on A330 aircrafts. According to this review, it is much faster.

[C - Comparable Products] Top storage is smaller in international flights?

The overhead storage in the transatlantic trip was only deep enough to have the luggagge stored sideways. Good? Bad? Well, we don’t know (yet).
The storage (called overhead bin in the lingo) is made by a company called C&D Zodiac, and they look very professional and thorough. So the storage space is good? We still don’t know.

One thing we can do, is to try to learn from comparison: The domestic flight had an overhead bin too, but in this case the carry-on luggage fit on its length too, almost twice the depth of the other.
How come? International flights have much longer duration, and it is normal that people will bring larger/more carry-on (sandwiches? Games? Stuff for the kids?). So why is storage room smaller in international flights? A bug?
OTOH, on domestic flights more people are expected to bring only a carry-on (they stay less time at their destination), so maybe there are more/bigger carry-on on these flights. So what? No bug?

And that is exactly the point of interest here:
The mnemonics/heuristics don’t tell you a definite answer, but they help you to think.

[C - Claims] Where’s the Kosher food?

US Airways cancelled my flight back, but I will not complain about that.
However, they did claim that there would be an acceptable solution, and that the airport hotel would be able to provide Kosher meals. And none of these claims stood to the truth.

The airport hotel had no Kosher preparation, and US Airways made it very difficult to find an acceptable solution. It took three hours of phone calling and walking between their offices until they agreed to provide suitable meals from outside.

A claim, even a verbal one, is a claim that your customer will expect you to fulfill.

[U - User Expectations] What’s with the default volume settings?

The aforementioned in-flight entertainment system had many settings. One of these was the volume setting, controlling how loud the movie you are seeing is.
However, this setting was re-set for every movie, which means that every time you moved to a different TV show, it will be at a loud volume again!
And I think (I did not ask for tools to verify it at the time) that every time the attendant/pilot talked on the announcement system, even when it was during a movie, the volume was at the loud volume – again.

My expectation, as a user of the system, is that once I set a volume level for one show, it stays on that level for other shows. I got startled by loudness once and again.

[P - Product Itself] Now, where can I change the diapers?

This is my favorite one.
Bathroom stalls at airplanes have this platform that you can detach from the wall and use it horizontally to change the diapers of your little kid.
Bathroom stalls at airplanes also have this new foamy liquid soap, strongly attached to the

But, as you can see on the picture, the soap was installed right in the place where the platform should be based on. There is no way to use the support in order to lay the platform down.

So this is a perfect example of two parts of a system that are inconsistent with themselves — or, in this case, not only inconsistent, but the feature added later, practically nullifies the original feature.

[P - Purpose] Is your seatbelt fastened? Is it? Not?

I noticed during the flight that the flight attendants had to check if passengers had their seatbelts fastened. So they went from passenger to passenger, trying to discern if the seatbelt is locked or not. Not always an easy task, as passengers are usually holding things or use a blanket.

But if this “check seatbelts” function exists, why isn’t the automatic system of the aircraft aiding? Why isn’t there a little row of LED lights that show which seatbelt is fastened and which isn’t? Attendants would (maybe) have it all much easier.
It appears to me that without this simple system (hey, cars have it) the aircraft system is not serving its purpose in every way it could.

[S - Statutes] This page intentionally left blank

This one is missing. Because:

• I don’t think I noticed anything that was against a statute or law.
• It would be irresponsible to suggest things that are against the law in this (or other) industry — especially as I am not a domain expert.

As you see, a 12 hours flight is a lot of time… These are my notes from trip to (and back from) Orlando.
I am interested in learning about other methods you use to analyze bugs/value in the physical world. Share in the comments!

Now let me start that post about the actual STAR East lectures and networking…

Recently a big argument was held (link in Portuguese, requires registration) on whether “testers need to know programming” or not. I made some contributions to the discussion, and I’ll try to translate my point of view to English here.
Please note that there were a lot of people participating, and the post below does not present the entire debate or even an entire opinion, just the messages I wrote as answers. I hope you enjoy them.

The proposed question was:

Do testers need programming skills? Until a few years ago, most people would definitely answer NO… But I’ve known both people that seek testing jobs as an alternative to coding, and testers that write code routinely. So, do testers need or not to know programming?

—

Do testers need programming skills? My comments:
The question as it is posed takes some assumptions that may be incorrect: Saying that “until a few years ago, most people would definitely answer NO” is like deciding the answer nowadays is YES .
First, from other discussions on this (there was one not long ago in the software-testing mailing list too), it sounds more like whoever used to say “NO” a few years ago still says “NO”, and whoever used to say “YES” still says “YES” . There was no significant technology or methodology novelty that caused a change in the approach to this (Cem Kaner says testing practice has evolved only a little over the years).
What does happen is that some people confuse the software testing we (software testers) do with the unit testing and other techniques that enhance and support development (and are mostly done programmatically). It is true that many testers work on this kind of tasks in their day-to-day, but I believe this is far from the essence of our job — and the distinction of testing and checking by Michael Bolton may help understanding that.
Second; if anything, the advance of years made testing without programming knowledge even easier. Jerry Weinberg describes the tests he did at the beginning of his career as written in punch cards on assembly and given for the machine to execute. At these times, it was much harder (bordering unfeasible) to test an app without knowing to program in the language, or details about the data types and system limitations. With time the advancements in interfaces made it, if anything, much easier to test without programming knowledge. Moreover, most languages are today capable of more or less the same things, so when testing a web page most tests will be similar if it was written in PHP or ASP.NET, and many tests will be similar for an app written on C# or Delphi. And in order to learn about internal details, you don’t need to know the code either — memory, performance and process analyzers can help with detailed views about the resources used.

That’s not to say a tester should not know how to code . Programming has many advantages to a tester:

• Being able to understand the type of problems that afflicts applications (knowing “what the bug is doing to the program” ).
  • Knowing about how software is built helps when building a mental model of the software functioning (where does the program makes decisions? when does it asks for resource? how does it communicates with libraries?).
    • Knowledge about the internal code of the application being tested does not necessarily helps, though. At times, this can be harmful as it limits the tests done more than it expands them. But this is another discussion, on the values of Black-Box testing and Glass-Box testing.
• Gives the option to automate tasks or functions when these are needed in a test (automated tests, automated loads, automated performance probes…).
  • And as important, the creation of little tools that aid the testing process (helping quick configuration of a system or gathering data for reports) can help the whole team, and make a tester an important player in the group.
• It helps communication with developers. Programmers are human beings that are easy to talk to when you know their language … Being on the same speaking-context as them really helps. And as it happens in many other geek linguistic elites, some programmers will respect code-speakers more than non-speakers.

So, should testers know how to program?
Well, it definitely helps in many facets. My non-authoritative answer is that it helps so much that no tester should ignore this area, given the opportunity.

—

With so many advantages, should we hire only the testers that know how to program?

Cem Kaner commented once that he “will not accept M.Sc. or doctoral students if they have weak communication skills. My experience is that these are more important, for success in my lab, than programming skills”. Kaner’s vision, on the articles that he writes about recruiting (this book, and this article), is that the teams should be heterogeneous and not everybody should have coding backgrounds — the “domain knowledge” about the area and business of the application are essential for effective tests.
For example, when building a team for testing an app that does stock market transactions, which kind of player would you recruit? People who know how to program, or that know the stock market trade? The best answer, I believe Cem would say, is “people from both sides”. Personally, if I had to choose one of the extremes, I would pick the stock market expert. Hey, but he doesn’t know how to code!! Well, then end user doesn’t know either.
In the comments to Testers don’t think like Developers think like Computers I was asked about recruiting programming testers, and this is the answered there (notice the example reuse ):

Coding isn’t a required background for testers. Coding skills will help on coding tasks, and sometimes with system understanding.
But a diverse background will make the team thrive and explore the system in fruitful ways.
A domain expert can bring much more value than a B.S. in Computer science.
For example, when building an application for trading stocks, a tester with a buying/selling stocks background will find valuable information.
(See lesson 235 in “Lessons Learned in Software Testing” by Cem Kaner, James Bach and Bret Pettichord – “Staff the testing team with diverse background”).

Of course, there is no 1-size-fits-all, and different projects may require a different team profile. Just as different managers and different contexts may.
In essence, I would agree that teams should be encouraged to hire non-coder testers. Or, if not encouraged, the practice should be considered perfectly acceptable.

Another point:
When I instruct testers, one of my first instructions is that operating system, programming language or data base limitations don’t matter. Be careful of *why* the software does the things it does… What matters is what the user wants, and knowledge about technology limitations can often come in the way.
In his article “A Letter to the Programmer“, Michael Bolton comments that the limit of text entry in the chat window of Skype is 32768 chars and a little problem he found with that. In the comments, a programmer quickly jumps to say that 32678 is the standard limit with the text_box element! This programmer did not notice that this doesn’t matter. If there is a problem here, a user doesn’t want to know about API limitations — and testers with coding skills risk “believing the system” instead of “defending the user”.

—

But what about TDD and Unit Tests?

It was commented in the thread that job opening notices more often than not require knowledge in coding language and in automation tools. From this, one sees that the focus on automation is very strong when recruiting testers. But, is this due to a sincere attempt to increase value? This tendency can very well be focusing cost reduction instead (which is good too, but can be harmful when done wrong).
This brings us back to the confusion commented above about the difference between “real tests” and tests (checks) done during programming that enhance and support development (Unit Tests, TDD suites etc). In this video commented on the thread (here, in Portuguese) a programmer explaining unit testing and TDD maintains that if you have great automatic tests that support your code, you don’t need a software testing team.
Ouch, that is a big mix-up!
When TDD tests are mixed up with the tests software testers do, testers are being approached as a programmer’s baby-sitter, looking over the developer shoulders to tell about coding errors. But our real function isn’t (solely) checking whether the “if x < 20″ was coded correctly, but to ask whether “20″ is enough, and in what extreme business conditions there will be 23 required, or when each of these 20 data-structs will need to be bigger.

Programmers make well to test their code thoroughly before releasing for testing, that way we can skip the dumb tests and focus on the real questions the customer is not asking. As it happens, for these questions, most automatic checks are less valuable than manual tests — be it because the automation will not find a bug again, because these tests do not evolve together with the system, or because the automation is not thinking beyond its code. If one could automate everything a tester does and think, then it would have been done and we testers would be having an easy time supervising robots . But you can’t, sapient testing is a human’s thing. See what James Bach writes in his article “Manual Tests Cannot Be Automated“.

(
Last, full disclosure:
- I have a computer systems development engineer degree, and worked as programmer for many years. Knowing how to program in many languages helps my testing job every day.
- Unrelated to this post, and not depending on me , most new hires in my department at work have background and/or formation in programming, this is a prerequisite.
)

This was an interesting exercise.
What took my interest in this one?
- First of all, I liked Pradeep’s post about learning to code.
Learning to code is an important skill for a tester. I can relate to it because I am trying to learn PowerShell. I program in some languages (C#, C++, PHP, VbScript…) and have a programming background (after my Computer Systems Engineer degree, the ‘natural’ path was to code, and I worked as a programmer for many years before finding Software Testing), but PowerShell has different approach and paradigms and learning it will definitely be fun.
- Second, I had an impromptu vacation day, and I thought it would be educational to use it for a peer testing puzzle. I thought I could win this one quickly (I was mistaken, it took me long).

So I downloaded Pradeep’s application, and got to work!
Rather than just trying to solve the puzzle, I looked at it as if the mission was:
“This is commercial 'roulette' style game. Stakeholders want to know it the game logic can be broken/learnt, which could mean a substantial loss of money when people start winning every time.”

Testing Hint: Cem Kaner says “Testing is an empirical, technical investigation of a product, done on behalf of stakeholders, with the intention of revealing quality-related information of the kind that they seek“. So it is important to set straight what is the information to seek. Which kind of bugs to look for?

In this blog I describe my attemp to answer this question in an exploratory way.

Now, Stop the press!
You may want to go to Pradeeps post, download the file, and try it for yourself before reading my solution. This post will be waiting right here when you’re done.

…

Here is the report of what I did, the line of thinking during my experiment, and the conclusion:

• 1. First, I ordered my environment to allow efficient and organized work:
  1. Increased the Screen Buffer Size of the Command line I was using to 500 (it turned out that I would do well on less, even half, than that (see point 2.b). But it didn’t hurt).
  2. Renamed the executable to be shorter and without number version: ren fnemo_1.7.6.exe fnemo.exe
  3. Changed the prompt to something shorter, cleaner, and that gave me context about this task against the other Cmd lines windows open: prompt $Cfnemo$F$S$G

Testing Hint: Organizing your testing environment before you start will likely help you during your tests. After you’re in the heat of fight, it will be harder to stop and get organized.

• I tried some basic executions of the program, to grasp the feeling of what it does and how hard it is to find Nemo. This also taught me what are the messages returned by the application when Nemo is found, and when Nemo isn’t found. They’re “Found Nemo this time!, Nemo Gill Bubbles SharkTooth Flow Phamplet Stinger” and “Ah! bad luck, didnt find Nemo this time!” respectively.
  1. I also learnt that Pradeep used Perl and Perl2Exe to do the app.
  2. Additionally, the application clears the screen at every execution, so the big command line buffer is not so helpful.

Testing Hint: Planning or scheduling your testing before you experiment a bit with the software can be done. But you risk being completely wrong in your assumptions about the app. In our case, there were no big surprises, but there could be.

• Sarmila commented on the Challenge page about disassembling the executable. Although by learning all the assembly one can learn about the rules that move the fishes around, it would be extremely difficult and overkill. But this made me think on how Perl2Exe works… Maybe it just wraps the Perl interpreter and the Perl script together? If so, what if the script is stored internally in clear? I tried to open the app in an Hex editor, but no, the Perl script isn’t in clear text there, it is obfuscated.
  • I also tried the useful Strings tool by SysInternals. My intention was to see if, maybe, the array of fishes could be seen at the app code, in order to learn the initial order of fishes. It couldn’t.
  • That ended my cheating session , from here on I used only a black box functional approach.
    • Well, it is not really cheating because there were no rules against that. If you do a full reverse engineer of the code, the answer is legit too. But I preferred the straightforward exploratory approach: learning from the software functionality as I use it, and use this learning to redirect my next steps.
• .
• By now, I knew that Nemo changed places between plays.
  1. Nemo also change places between invalid parameters (which may or not be a bug).
  2. Moreover, the other fishes change places too, even their placement related to Nemos’ placement changes.
  3. So I tried to see if Nemo will return to the same place in any consistent way or number of times. Running that is easy, you just enter “3″, “3″, “3″… and count where Nemo was found and where he wasn’t.
     • I discovered there is regularity (for “3″, its every 2, 3, 17, 5, 10, 5…), but the regularity was irregular enough, and also changed for different positions. This was likely a consequence of the real logic, rather than the logic itself. This correlates to the movement of fishes, but does not cause it.

Testing Hint: Take note of your steps while testing. Using the Session Tester tool is a great idea. For this session, I used MS OneNote because it supports rich text, and I could use tables.
Is that like saying that rich texts helps rich tests? .

• Another detail that helped me in the game was knowing the developer and the purpose of the challenge. This puts a lot of content in the context. Pradeep is a rather playful and would put some tricks into it.
  1. First I tried to see if Nemo could sometimes be in any placement bigger than 7 (there are 7 fishes, places 1 to 7, but there were no rules as to where the fishes were limited to be).
  2. I tried in a toss of up to 220 attempts, and Nemo wasn’t in place 8 even once. So I let this idea on StandBy for now.
  3. Then I tried… what if the “Minimum attempts” input at the beginning of the app affected where Nemo appears?
     • 10 is the default for the “Minimum Attempts“. Nemo is at place 1 at the beginning there.
     • 11 attempts chosen: Nemo is at place 1 at the beginning too. Maybe it does not affect?
     • 12 attempts chosen: Nemo is at place 4! Bingo!
     • 13, 14, 15, 16 showed that the original places were cycling at ’3′ intervals. That means that when trying to find Nemo on the default 10 attempts, it will be in the same placements as 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46, 49, 52…

Testing Hint: let the test and what you learn during the testing guide you. New information you find should set you in a new direction.

• You’re seeing that “220 attempts” above, and thinking if I really entered each number manually.
  1. By this time, I decided that I was in need of some automation to insert inputs to the application.
  2. The original app had no apparent automation capabilities, so I decided to use the internal input redirection of the Command line:
     • By using the “<” operator, I could redirect an input file “input.txt” to the app using “fnemo.exe < input.txt“
     • I extended that to “fnemo.exe < input.txt > output.txt“, which gives us with a complete dump of results that can be searched with Notepad.
  3. We had an easy automation now, and this can help our sapient testing. I was exploring, and using automation to help me try different things help me explore my possibilities and get a bigger picture of the situation when needed.

Testing Hint: Don’t be afraid of automation. It can help you talk with the software in different ways and interact with it differently. It helps a lot with data generation and logging, too. Another hint: Never let the automation take control of the testing. It is there to help you, not to drive the effort.

• With the knowledge and tool I had so far, I did a lot of trials in order to find where Nemo is at each iteration. This could give me a hint if he moved in any regular or consistent manner.
  • So I made a big file of winning moves (a lot of trial and error went in it).
• .
• It was now time to model the results:
  • First attempts at modeling the results:
  • I got this: Nemo’s position: 1, 3, 5, 3, 2, 7, 3, 4, 7, 4, 4, 1, 5, 5, 2, 5, 6…
    • Nemo was certainly jumping around without any clear regularity, even when I did it twice the times there was no repetition.
• On the Second modeling attempt, I tried to see if a more visual model could help:

• 1	2	3	4	5	6	7
  X
  					X
  				X
  		X
  					X
  						X

• Even in a table with many more results, it made no sense.

Testing Hint: When you look at a problem or thing, different models will give you different view and different details. Multiple viewpoints help — keep changing the viewpoint until you learn enough.

One interesting thing that happened by this time, is that in my manual trials, I was finding Nemo pretty easily.
Intuitively, I was seeing a pattern on the game, a pattern that was not conscious or explicit. It was as if my mind had already learned the rules without noticing them.
It reminded me of what Douglas Adams wrote in Dirk Gently’s Holistic Detective Agency:
“A ball flying through the air is responding to the force and direction with which it was thrown, the action of gravity, the friction of the air which it must expend its energy on overcoming, the turbulence of the air around its surface, and the rate and direction of the ball's spin.
And yet, someone who might have difficulty consciously trying to work out what 3 x 4 x 5 comes to would have no trouble in doing differential calculus and a whole host of related calculations so astoundingly fast that they can actually catch a flying ball.
People who call this instinct are merely giving the phenomenon a name, not explaining anything.”

• I still thought that looking at the other fishes will bring the breakthrough needed.
  1. In order to learn the movement of the fishes, I did a table of their placement in relation to Nemo at every run.
     1. i. This is what I got:

     2. Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
        Nemo	Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth
        Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
        Nemo	Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth
        Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
        Nemo	Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth
        Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger

     3. See anything interesting? The rows repeat themselves alternatingly! So the fishes were not really moving around, they were following Nemo!
        1. My take is that Pradeep has two different arrays: One for Odd rows, the other for Even rows.
           1. This may not be true, but it doesn’t matter for us. When testing, we not always know what exactly the programmer wrote in the code, but we infer a mental model. If it suits the needs, it is a good model even when not the real thing.
              1. Many times I think these fake mental models are even better than the real thing. It’s the best way for them to act as an intuitive Oracle when analyzing an application.

Testing hint: When you learn and infer about software, you make assumptions and a mental model of what the software does inside itself. This model does not need to be an exact depiction of the code underneath the app. As long as it answers your questions, and you change this model constantly as you learn, it is good.

• From this part on, I started to analyze Even rows and Odd rows separately.
  • Bingo! Seeing the pattern was easy now.
  • Odd rows follow this cyclic sequence: +4, +4, +1
  • Even rows follow this cyclic sequence: +0, +4, +4
• .
• But what with the differences when using different “Minimum Attempts“?
  • A quick look at the tables I had gathered showed that they follow the same rules, but start at different places.
• .
• At the end of this trials, these are the set of rules that allow me to know where’s Nemo at any given time:
• a. Where does Nemo start?
  1. “Minimum Attempts” of 10, 13, 16, 19…

  2. 	Place 1	Place 2	Place 3	Place 4	Place 5	Place 6	Place 7
     First Line	Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
     Second Line	Stinger	SharkTooth	Nemo	Phamplet	Gill	Bubbles	Flow

  1. “Minimum Attempts” of 11, 14, 17, 20…

  2. 	Place 1	Place 2	Place 3	Place 4	Place 5	Place 6	Place 7
     First Line	Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
     Second Line	Gill	Bubbles	Flow	Stinger	SharkTooth	Nemo	Phamplet

  3. “Minimum Attempts” of 12, 15, 18, 21….

  4. 	Place 1	Place 2	Place 3	Place 4	Place 5	Place 6	Place 7
     First Line	Flow	Phamplet	Stinger	Nemo	Gill	Bubbles	SharkTooth
     Second Line	Stinger	SharkTooth	Nemo	Phamplet	Gill	Bubbles	Flow

• b. Where does Nemo go?
  1. i. “Minimum Attempts” of 10, 13, 16, 19… Follow this cyclic rule, starting at step 1.
  2. ii. “Minimum Attempts” of 11, 14, 17, 20… Follow this cyclic rule, starting at step 2.
  3. iii. “Minimum Attempts” of 12, 15, 18, 21… Follow this cyclic rule, starting at step 3.
  4. Odd Rows: +4 | +4 | +1
  5. Even Rows: +0 | +4 | +4
• .
• How do I know it is right?
  • Proof!  I built another automation aid: An Excel spreadsheet that can tell you the position of nemo in all trials given an initial “Minimum Attempt“. It works!
  • You can download the spreadsheet to try it here. Choose any number of initial attempts, it will tell you all Nemo’s position accurately.
  • Is this the real logic behind Pradeep’s challenge? It may not be. It is probable that pradeep did something else, much simpler, and that only translates into this. However, this answers the mission of predicting Nemo’s moves.
    • I am sure that analyzing the findings above one can discover what was really that Pradeep did. I’ll just ask Pradeep on a mail for a suggestion. Testing hint: You can always reach out to the developer and ask him about his software and/or code!

Testing hint: Again, your mental model doesn’t always needs to be exact. As long as it works and answers the questions you have. Keep in mind that this mental model should be flexible and evolve with your knowledge.

• Conclusion sent to the ‘roulette’ stakeholders:
  Until a higher amount of randomness is added to the game, it is too risky to release the game as is now. A malicious player can find the logic of the game after some hours of work, and a pair of smart malicious players in even less time.

Thanks Pradeep, for the nice challenge.
It was a very nice experiment, and I learnt a lot. The best part, for me, was organizing my thoughts in this report, and I hope you like it too.

So, a few hours before January is over, I’ll transpose here an answer to Testing.StackExchange about the last decade on testing:

Question: What are the most important software testing developments of the decade?

My Answer:
The question asks about the most important developments… Not the best or the worst, the beneficial or the harmful.
I’ll try to answer here with considerations by me and others I found on the net. Not everybody will agree that all these are good — even I don’t agree with all — but my approach here is more of a reporter than a judge.

1. Blogs were the most important testing development, at least for me .
   It helps me grow my testing philosophy. It helps by being a quick and instrumental medium to quality discussions. Following the great authors as they write is a great experience (many authors started blogging only during the 2000′s, like James Bach and Michael Bolton).
   One great thing about blogs is that they allow us to understand the flow of ideas as they are being built upon — instead of simply receiving the ideas later, in book, as was the custom 10 years earlier.
   • Tweeting, forums, and other collaboration platforms of Web2.0 are cool too, but I believe their impact to be smaller.
   • Blogging also allows a simple guy like me to share his Testing Thoughts :).

Other points:

1. The Context-Driven School gathered momentum with this name, and became well known. I couldn’t find tracks on the full history of the context-driven school, but the earliest mentions I could find to it with this name are from the very late nineties. It is clear that Cem and James were publishing context-driven papers from the early 90′s (see here), maybe before — but the first places where I found the “context-driven” name where from 1999 (at the software-testing list. This seems corroborated by C2′s page).
   • Again, I don’t mean to say that it started on this decade. I believe the work of the context-driven people was context-driven for the last 50 years , and all had always sought excellency. It is not new from the 2000′s, but it looks like the movement got a name — and exposure as a movement — only in the last 10 years.
2. Certifications gathered momentum too. Without entering in the good/bad discussion, it is a movement that had a lot of action in the past 10 years and affected the way we discuss software testing today. It wouldn’t be fair to count what happened in the 2k’s without mentioning them.
   • The ISTQB was founded in 2002, and is a popular certification.
   • The ISEB was doing certifications before (so this too, isn’t entirely a new thing), but all got much more impetus.
3. Greater awareness and recognition of the benefits not only of Exploratory Testing, but of testing in general, and testing as a career too.
4. Michael Bolton started consulting in the testing arena, but moreover started writing and publishing. His essays about testing and checking are very cool, and Jon Bach considered it “one of the most (in)famous and important posts to come along in our industry in a long time“.

For the next 10 years…

1. The weekend testing meetings in India were highly praised by James, Michael and Pradeep on twitter. Seems like they believe it will make an impact on testing in the next years.
2. Testing.StackExchange appeared too late in 2009. But… will it appear in the 2020 list? I hope yes.

Demetri Martin

One best friend of mine introduced me to Mitch Hedberg and Demetri Martin, great one-liner comedians. They are/were two funny men!! Three, actually, if you count my friend which is also funny. 

After hearing the disks for over a year, not only the jokes aren’t any less funny, but I’ve started to find subliminal testing messages in them .
I’m writing down these “insights” because I find value in them. And even if they fail to teach you something… Hey! At least the jokes are pretty funny!
So here go some favorite quotes, and their parallel in testing: 

. 

1) “A drunk driver is very dangerous. So is a drunk backseat driver, if he’s persuasive!
'Dude, make a left.'
'But those are trees…!'
'Trust me...'“ 

We are testers. As such, most of the time we aren’t driving the company: Essential operational decisions are made by someone else (project manager, product owner, CEO, whatever you call him).
But if we are not the drivers, our position is certainly close to this of a backseat driver: we have our maps, good knowledge of the area and our experience; and we give advice that is used to take real decisions. So don’t be a drunken backseat driver, it’s dangerous! If you are inebriated by an obsessive desire to fix a bug, or by a personal quest against/for colleague, by a blind belief in a set of metrics or a unfounded trust in a fictitious “Best Practice”, you’ll be taking your company directly into the trees.
And, as with any drunk person… Whenever your judgment isn’t objective… At least recognize/admit that you may be drunk! Will make it easier for everyone else. 

. 

2) “I used to play sports... Then I realized you can buy trophies. Now I'm good at everything!“ 

A plaque, a crown, a card, a trophy… It is common to find people who treasure “victory symbols”, I know I do. They prove you’ve mastered a skill.
Trophies are nice and great, but only if they are accompanied by skills and real world practice. Trophies that can be attained without these traits are just empty cups*. 

One should beware when dealing with “achievement symbols”… At times, acquiring the symbol does not mean acquiring the achievement or skill too!
For example, just as buying the trophy doesn’t make Demetri good at tennis, getting a testing certification won’t make a tester good at testing… The certifications syllabi try to teach only specific lexicon and terms definition, but not the real practice of testing, because they don’t/can’t cover the interactions between persons and players.
For some people, the joke could be read as “I used to practice and study... Then I realized you can pass a certification. Now I can prove mastership - without the effort of gaining it!” Be sure to be from the ones who keep learning and carrying the skill. 

. 

3) “I am afraid of sharks, but only in a water situation. If I saw a shark on the street, I'd be like 'What? F#*k you!'
It's funny, that's like the opposite of how I am with lions!“ 

What’s the Best Practice for dealing with sharks? “Escaping” or “screaming”, right?
What if the shark’s on the sand? Or dead? Not hungry? Not dangerous? You have an anti-bite clothe? Anti-shark cage? Suddenly the ‘Best Practice’ looks more like a ‘so-so advice’, uh? 

The business of testing isn’t different. By looking at our surroundings and context, we can learn important details about our problems – and only then we can try to find an action path that solves it.
Without the context analysis, we may come with solutions that don’t address the real problem or does it badly. One size only seldom fits all. 

. 

4) “If I have to move up in a building, I choose the elevator over the escalator. Because one time I was riding the escalator and I tripped. I fell down the stairs... for an hour and a half.“

Automation is all nice and fine… Until you trip on it! Then it takes long time to recover from the fall (you could be long done by this time) and the fix to the system can be so crooked that it lets you prone to further tripping.
The elevator is a much more robust automation (not only it goes up and down, but it is much safer to trip in it ).
Anyway, who said you have to automate? An equally good way to go up and down is the stairs! You don’t need to wait for them to come, they are as quick as you need (or can make) them to be, you can stop to look around or change direction in the middle of the way, they don’t limit weight or number of users and they are never out-of-order.
Slipping, well, can hurt a bit . Just remember that manually doing things gives you many opportunities a machine cannot provide. 

. 

5) “I saw a sign on this door; it said, 'Exit Only'. So, I entered it and went up to the guy working there, and I was like, 'I have some good news. You have severely underestimated this door over here by, like, 100%, man!'“

Labeling is a natural instinct: The Bible depicts Adam naming animals as his first action in Eden, and Aristotle has been classifying everything since the 300 BC.
But the labels we use can limit ourselves or our tools.
“I'm a tester, I can't code“.
“I'm an engineer, I can't sell“.
“I'm a newbie, I can't help“.
Good news! You’ve underestimated yourself by at least 100%! 

. 

6) “I want to make a revolving door that says 'Pull' on it, just see how obedient people are.“ 

Meeting a Test Case that says “1) Reach to revolving door”, “2) Pull the door to open”. What do you do?
Maybe nothing is wrong. Maybe there are good reasons to make people walk backwards at the entrance.
Or maybe there isn’t, and then, who’s decides? The test case? The design? The developer? The tester?
What are your ways to discover about the door product and its users before deciding on obeying or not?

. 

7) “I was walking down the street, and this guy waved to me. Then he came up to me and said, 'I'm sorry, I thought you were someone else.' I said, 'I am.'“ 

Wow, that’s a funny joke!
Beginning a task with a very clear objective is usually a good thing – but failing to identify or even acknowledge alternative outcomes may not be.
A good friend was working in an academic project, trying to prove a theory about the nature of lasers. However, while progressing in the research, the calculations made clear that the theory wasn’t right. He was desolated – all this work and nothing to publish! He failed, momentarily, to notice that demonstrating a theory wrong has as much scientific value as proving it right, and the publication will advance science just as well.
Testing present us with many similar situations. We can start a test chasing a specific bug, and miss other bugs that may appear. Or we can keep our mind open for different outcomes, and grow your tests as we go from these outcomes. A different bug is hiding there, believe it . 

. 

8 ) “My plumbing is all screwed up. Because it turns out, I do not own a garbage disposal.“ 

How many times we rely on a garbage disposal when we don’t have one?
- Don’t worry, release it that way, we have a fantastic relationship with this customer. What if you don’t?
- It’s fine to release that way, the customer has a great quality assurance team and they’ll catch it if there’s a problem. What if they don’t?
- No problem, we can work Sundays — our testing department understands our needs and won’t care. What if they do care?
- This looks like a very nasty bug. But customers will always have the underlying framework, so it is fine. Will they always? 

. 

9) “I'm in a weird position, because I like rainbows, but I'm not gay. So whenever I go out wearing a rainbow shirt, I have to put 'Not gay'. But I'm not against gays, so under that I'll have to put '... but supportive'. It's weird how one group of people took refracted light. That's very greedy, gays.“ 

Once in a while a group of people will come, wrap common sense ideas in a new named package, and claim ownership, which is as reasonable as taking sole ownership of refracted light.
I could parallel this one to many things or groups, but I chose one where this happens with a lot of enthusiasm… The Agile movement.
Suddenly, you can’t focus on customer and organize a software project in a change-adapting way, without being called Agile. That group just took no-nonsense practices that had been used for decades and re-branded them as if they had invented it!
Don’t get me wrong – it is my opinion that what Agile preaches are often-good-practices-for-many-contexts and there are testimonies of companies, projects and souls being saved from a bureaucracy and poor quality hell by adopting Agile as a new approach. But this does not mean that anyone who is delivering quality in a “release early release often” way is Agile.
For example, even being a total newbie and never part of an Agile team until now, my personal testing approach focus on happy customers, working software, individual interactions and change — without me needing to be an Agilist.
More experienced great people had practiced all that, and test-first, and XP for decades. 

More than that, not only the practices’ refracted light was taken, but even the word “agile” itself was expropriated – see James Bach’s note at his ‘Who stole Agile’ post. 

——
Hope you enjoyed the article, or at least Demetri’s jokes.
If you have more insights and funny stuff to share, don’t hesitate to contact me – I promise to laugh.

Request: Are you a lawyer? Please send me corrections

Matters related to law, and all the discussions around it, interest me much — especially when related to Software.
This made me read about the subject and keep contact with the legal representatives within the company I work for. This also motivated me to learn and lecture about the legal guidelines in software development adopted by our company, and to lecture about legal matters on software in general at the last Israeli SIGiST conference.
Most important than all, this made people share with me a lot of comments, questions and stories pertaining to the law.

For example, one colleague brought to my attention a case in which he and some friends had bought a PlayStation 3 in the local Office Depot website for 220NIS – when the normal price is almost tenfold! They believed it was some special sale promotion, but at the end Office Depot announced it as a typing mistake and cancelled the sale after it had been acknowledged (sale confirmation by email).
(There is an article about the episode in Hebrew here if you want to see it).

What is my opinion on the legal aspects of the story? I was asked.
I don’t have one, as I am not a legal professional, I answered. I am, though, a Testing professional, and here is the tester rambling I sent by mail commenting the occurrence:

——
In fact, electronic transactions are special and different that normal stores. They can be built without any human interaction whatsoever, and the lack of human monitoring can bring new and unexpected problems.
Note that I can’t have an official Legal opinion, as I am not a lawyer. I’ll just write what I learn.
A usual understanding in contract law is that once both parts show agreement to the deal, then the deal is closed and valid (I think a lawyer would explain this by stating that there is a mutual offer and acceptance of goods or services). Such agreement is shown, for example, by accepting the payment money.
In such a case as this story, the electronic sale often includes a debit in the buyer’s Credit Card (that turns into a credit at the cancellation of sale). If a service provider accepts payment for a service, he is in obligation to provide the service at the agreed terms. If he didn’t agreed to the terms, he would not have taken the payment. So in our case the fact that a payment transaction was completed supposedly forces the two parts to honor it (the store should not cancel the delivery just as you should not cancel the money debit).

However, apart from the amateur philosophy, we can find a bit more of info in the net about this sort of things…
a) Not only the payment debit can be viewed as a contract acceptance, but also the confirmation mail you receive when buying online can be seen as such, depending on what it writes.
b) The Terms of Service of the website can be used for or against the store, depending on their presence and the judge interpretation of them.

• For example:
• Buy.Com (huge company, 290 million revenue) announced at $164 a $588 monitor. They tried to cancel the sales, and lost more than $60000 in a lawsuit (CNET link).
• On the other hand, Amazon.Com (huger, 19 billion revenue) announced at $11 a $449 handheld (CNET link). They cancelled the sales successfully, claiming that the Terms of Use state that there is no binding contract until the confirmation of shipment arrives.
• Buy.Com had no such protection in their terms, and after the lawsuit they updated the terms to include “have the right to refuse or cancel any such orders whether or not the order has been confirmed and your credit card charged”.
• If it will always help? Not sure. In this link a representative of the FTC says that it will depend greatly on the state the lawsuit is taking place.

In the US, there is a concept of “loss of bargain”, that could allow one to buy the PS3 at normal price and sue for the difference (which represents the bargain lost due to the store’s malpractice). Of course, if the judge identifies the buyer as having bought it to exploit a weakness or obvious typo, the attempt can backfire…

Now that we learned some principles of the domain… We go for the most important…
As I said, I have no legal opinion, but I do have a tester opinion.
First, this shows (again) that there is no final answer in the law yet for new technologies and new problems.

And I now know that for assessing risk on an e-store product, one has to answer these questions too:

• Does the system have any protection against mistakes
• This can be done by issuing an alert if there is a big discrepancy after:
• comparing the product to similar ones
• or comparing to the previous price
• or comparing to the price in the normal (brick) store
• Does the system require a manual human confirmation before sending the payment or confirmation mail?
• This can be bad for a large operation, but for small-medium business, can be a good way of catching mistakes
• Does the system alerts the administrator if there is a sudden surge in the purchases of a newly added product?
• Do the Terms and Conditions protect the company in case of mistakes?
• Are the Terms and Conditions clear to the end user? At least in a visible place?
• Is the user obliged to mark an acceptance to the Terms during log-on, or it is just a link in the page footer?

These are some of the questions I would ask when faced with such a product. There are more, certainly — please share your own ones in the comments or by email

——
One last comment…
Sometimes the typos make the prices higher than the norm… See this picture, a real example from Amazon selling this $400 card for a whopping $5 million!! That’s a lot of money for a graphics card…

Hummm… would they cancel this sale as a mistake too?

My colleague Issi Hazan was asked to teach the ITCQB syllabus to testing-oriented-students at TechCareer, but he bravely thought of pushing them to real tests instead, even before they find a job – this would allow them to build experience and get a job more easily (one way to solve the “hard to get a job without experience, hard to get experience without a job” problem).
So we built a lecture on how to “create a tester portfolio“. We explain how testing can be done even outside an enterprise environment: Lots of Open Source projects are seeking for good information from software testers, and a good record there is certainly sure to help. Crowdsourcing is also a  possible way to work on testing before getting a real job (see two posts on crowdsourced tests here and here).

The presentation is available below in flash format, and can be downloaded here: “Create your testing portfolio“.

But “PowerPoint is Evil™“, so I feel compelled to write it in real words too.
It goes like this:

Introduction

If you are searching for your first job in software testing, one of the big challenges you will face is how to have your résumé stand out. Either if this is your first job ever, or you’re changing career from a different background, the question remains: How can your résumé compete with seasoned testers’?

The solution to the 1st challenge constitutes your second one: actually acquiring this experience that differentiates the seasoned testers. And this is hard.
People looking for jobs often complain they “can't get a first job without experience, and can't get experience without getting a first job“. The logic sounds leak proof solid… where it not for the fact that, well, most people do get a first job.

So how can a newbie gain this differentiation?
Many people seek certifications, hoping that the additional line or two in the CV with the uppercase LETTERS will cover for the lack of skills and practice. Now, here’s a secret: they won’t.
Certifications do not give you skills or wisdom; they give you a determined lexicon and specific definition for terms. Some employers may want that, but they know it does not equal practice.
Software testing is an art. Young artists do not collect “painter certifications” or “canvas master certifications“… What young artists do is paint a lot, toss all the colored canvases in a big binder, and show them to galleries. Gallery owners can through this portfolio see their practice, evolution, style and commitment.

This is our suggestion to you: start testing, and be prepared to explain your tests to prospective employers.

Meet your Pseudo-Employer

Testing without having a tester job sounds farfetched, but it is actually within reach.
Of course, you can enter a bug-hunting-spree and find defects in any software from your desktop or the web. While you can learn techniques and tools this way, this method misses the most important (and fun) part of testing: The interactions.
The interactions with people involved in the software will give you focus on what is important for them to know. It will also redirect your tests as you go with feedback and comments, making sure you are always providing useful data. Last, being in touch with this human side of the software will give you the opportunity to explain, discuss and participate in fix/no-fix decisions – transforming your findings from simple bugs to valuable information on quality and risks.

And there are many ways to be involved with testing in the development of software.
Here are a few options:

–> Academy Projects

There is a lot of interesting software being done as final projects for Computer Science graduations. These students do all the work themselves, and they will be certainly glad to have help in testing the software.
One of the great things here is that the interaction is all 1 to 1 and face to face, allowing it to be a closer and more dynamic relationship. Feedback and bug-fixing times are quick; you can suggest a change and then see it done the day after.
Posting your ‘testing offer’ as a note in the wallboard, or asking from one of the project mentors to point you a needy student, are all good ways to get started.
And, for what it’s worth, you may appear in the credits on the project report!

–> Open Source

Sourceforge has hundreds of thousands of open source software projects under development. You can pick one that suits your interests and limitations by using their filtered search, and you can also take a look at their “Help Wanted” section, which many times ask for help in testing.
The open source community is thirsty for contributors, and they respect sound advice. If you show your competence and are provide consistent views/comments, the community will pay attention – and they don’t care about your degree, background or years of experience.
And, for what it’s worth, you may be given credits (or ‘tip of hats‘) on the release notes!

–> Crowdsourcing

There are websites where you can receive money for reporting bugs. The two examples I know of: uTest and Flash Mob Testing.
Testing in this environment may pose problems (see this post for a list of concerns) and the interaction you have here is by far less involved and meaningful than the previous two options.
We listed this in the slides because the couple dollars are tempting and people may prefer this option over the others.

Note that here, you won’t get public credit for your work, in case you wanted to showcase it in an interview.

–> Volunteer for a company

This one wasn’t at the slides’ first version, because I had forgotten about this history completely.
When I was at the university, I had a friend working for a software company, a small one without a testing team. To exercise my ‘criticism’ (I didn’t know I was ‘testing’, back then) I started to write him comments on their webpage and their application, notes that they triaged in their bug meetings and fixed in the next release (sometimes).
So it was volunteer testing for a real company.
(This entire story slipped my mind until 3 weeks ago I had a course in the same building as this company is located. It all came back to memory suddenly: that was my first testing gig!! ).

In summary

There are many ways to test without a testing job. Find or invent the option that better fits you, and start differentiating yourself as a real tester.
Try it!
With a little work from your side, you can demonstrate to employers your capabilities, style and passion. And having volunteered to test and provide data isn’t something a recruiting manager sees every day, so it is likely to attract his attention.

The next steps

After you’ve decided where to start testing, following a consistent way of work will make your testing much more efficient.
The slides include many suggestion on how to get a good start: Follow the forums discussing the software (and ask questions!), be sure to read the bug submitting guidelines so your information is treated with care, and find a lot of bugs

An Alternative

When applying for a job, capitalize on the experience you’ve already got.
Software projects need domain expertise just as they need testing expertise. If you were a stock-broker, a software house developing stock-market software can have great benefit from your testing. If you speak languages, a lot of specific software can benefit from this trait.
See my comment in this post: The entire experience you bring to a team is valuable; recruiters aren’t always only looking for super-testing-wizes.

Good luck!

Instead of a new post, I revisited and modified last month’s post, About youTesting with uTest.
It has now more content, and still has a discussion of pay-per-bug models.

The initial opinions are still there. While the pay-per-bug model presented by uTest is certainly innovative and interesting; the model still misses a lot. It will certainly be center of discussion many times in many circuits .

To some extent, trying to get quality assessment and counseling from isolated bugs from isolated testers is like getting medical advice from many different isolated doctors without them examining you personally even once.
But, to the other extent, bug reports do consist an important tool for quality evaluation and direction assessment. And well formed groups can benefit from the yet another list of bugs received, as long as they don’t confuse them with complete product testing.

In one of my discussions of the matter with friends, I was encouraged to ask the people working in uTest what they think of these points and how do they and the companies solve the apparent drawbacks.
It is a great idea, as they seem to be very forthcoming and they do have a lot of experience in software development and quality. The CEO, Mr Doron Reuveni, tweets and sports and seems approachable. Not many companies have such accessible CEOs, and I’ll try my luck with him. If he replies and answers our questions, it will be very cool, and I’ll add the info to the uTest posts here.

This is an interesting topic:
I’ve been involved lately in many conversations about uTest, or more specifically about its model.
uTest is a website where companies can post their software, along with some guidelines on focus areas, and users around the world can download the app, find bugs and get paid for bugs reported (as long as the bugs are accepted by the posting company).
There is a lot of confusion/discussion around the good parts and the bad parts of the model, so I will share here some of the points I had taken from these conversations (thanks to all the friends who shared insights with me)… Some attentive readers will notice the article is an almost copy paste from a reply in the software-testing group.

Please note that I am not saying “uTest considered harmful” or “don’t use uTest” or anything like that.
Please note that I am not saying “uTest is great” or “use uTest” or anything like that.
All I want to point here are some of the strengths and some of the weaknesses of the model, so every one (both testers and companies) can decide for his own context. I welcome debate over any of these points, and will update my post accordingly.

–> (A) Here we go then. Some strenghts of the model:

1. Crowdsourcing provides diversity:
   James Whittaker in his “Future of Testing” webinar considered this model of testing (there I heard the ‘crowdsourcing tests’ term the first time) the next evolution in software testing, as it allows a huge number of testers to test in very diverse environments and configurations.
   Full disclosure: The Webinar was given for uTest and at their website.
2. The managers are nice guys trying to create a nice community.
   In the case of uTest, they have blogs with guest writers such as James Whittaker and others. They are on twitter and occasionally twit a testing article (more often they do brand promotion, which is fine too), they offer bug battles and discussion forums.
   And at these venues they do good. I have heard that Doron Reuveni is a really nice guy, and I believe it from his twits and posts.
   Of course, people tell me that all this community building is about money and business, but that’s completely legitimate. I take money for testing at my job, too. Making money doesn’t make anything bad.
3. Simplified development cycle:
   For some companies, it appears that crowdsourcing makes their development lifecycle more efficient. See this article on how crowdsourcing supposedly slashes development time.
   The points commented in the article are a bit dangerous, in light of the concerns raised below, but for companies that have yet to build a testing team, starting to receive bugs before the testing team is complete can be a good thing.

–> (B) On the other hand, the uTest model has some weaknesses, like:

1. “Software testing” misinterpretation:
   The idea of this model is ‘pay per bug’. Companies are outsourcing testing by receiving bug reports from the community.
   But submitting bugs is only one part of the work that a tester does. Testing (functional too) requires involvement with the development life cycle. Testing requires understanding of the user needs and business needs.
   By putting the testers far away from the company and their decisions or business view, the testers have little option to ask good questions. They have little space to provide real information about risk (that’s what testing is about, right?), and thus they are likely to practice poor testing.
   Moreover, testers get their money for reporting bugs, so this will become the ultimate purpose of submitting them. In any other context many would agree that bugs aren’t there just to be reported or even just to be fixed — bugs raise questions about the application, about the value it provides and about the general direction the product is going.
   When you take the testers out of the fixing decision, you miss this healthy discussion. In this model, if a bug doesn’t get fixed, the tester isn’t probably alerted about this fact. What’s more, the tester may not even care, as he already cashed the bug’s money and “fixing or not fixing is a business decision where we don’t have a say”, right?     And with that, reporting well detailed and complete written bugs isn’t a goal for pay per bug testers too: If you spend valuable time writing a bug report with detailed information, you are not making money! Plus, another person may log this exact bug with a poor report before you finish your perfect report (you don’t receive money in this case).
   Sure, the model does not foster or defend bad practices on purpose. But it has its way to motivate them, in a sense.
2. Poor metrics (paying for bug reported):
   Companies like uTest didn’t invent the “bug quantity measurement metric“, of course. It is an old approach that has failed in many places from what I learnt (might’ve been successful in other places) and is definitely not suitable for all contexts.
   Even if applying companies receive great comprehensive dashboards displaying different quality aspects of the software, at the end of the day they are only seeing bug numbers.”You get what you pay for” — the company is paying for bug quantity and testers (as anyone else) will give you what you measure them for, a large quantity of bugs. It is easy to confuse this data with information about risk or information about value. uTest writes in the Terms of Use: “uTest makes no promises or warranties as to any specified quality standards, testing coverage, maturity, quality measurements, the general quality of services or feedback, or results provided on or through its website.” Now, this is only a standard disclaimer, and some may say these measurements are rubbish anyway, but what uTest is saying, in essence, is “you get only bug numbers, nothing else“, which is true. Companies must understand that bug reports don’t make for a quality or value assessment. See point (C/5) for a related comment.
3. Quick and Dirty is rewarded:
   The rules are: time consuming bugs pay the same as quick bugs. From their Terms of Use: “uTest shall ONLY pay you for bugs reported and approved by the applicable Company and NOT for the time you incurred by analyzing bugs or time incurred for any other purpose.”
   Which type of testers receive a higher score at this model? Testers that report simple bugs will be at higher ranks than the ones that hunt and investigate and find hard bugs.
   Consider this together with point (C/2)… and you’ll have testers that log a large number of shallow bugs.
   This kind of behavior is acquired by whoever practices it… And by the end of the day, some testers with high scores at uTest may have become accustomed to quick and dirty work.And what about bugs that don’t easily reproduce?
   Bugs that don’t always reproduce need special attention and a lot of time invested. Forget about those in crowdsourcing, as more attention equals less money… One is risking that the ones to find the tricky bugs will be the end customers.

–> (C) Some other points should also be considered:

1. Bad disputes fostering:
   Any discussion translates into money. And money, being a terrible motive , can make disputes between a company and a tester biased on both sides.
   Software companies will want fewer bugs, and software testers will want more. None of the two is a good goal, and both can lead to wrong stands.
2. Bad attitude from testers causing bad results:
   Testers from a ‘real world’ project might wish to receive better built software so they can jump into important matters instead of clinging at innumerous surface bugs.
   In the pay-per-bug model, testers will leave a mature software to go test an immature one (more bugs = more money)… so the tester tests worse apps and the software company loses the testers exactly when it more needs them (when the app is stabilizing, testers will migrate to a new, immature one).
3. Disruptive competition between testers:
   In the company I work for, if I share information with a colleague and he uses it to find a bug before me, everybody is happy (as there was a collaborative effort that generated a useful piece of info). Testers in a pay-per-bug competition will be less inclined to share information, as bugs opened by someone else mean less money and less score points.
   In their model, you compete with other testers, and this enters the testing culture of the participants. And testers who compete are les useful in a team than testers who collaborate.
4. The score feedback isn’t objective:
   An award winning uTest tester commented in one of the online discussion that the score for testers doesn’t represent testing skill.
   The skill set required to be a good tester isn’t exercised in such simple view of testing as ‘receive the app, try it, list the bugs’.
   And also, don’t forget that there’s money involved and this usually undermines objectivity .From uTest Terms of Use: “Testers shall have no right to either control or influence their fee rate or feedback regarding their reputation or success.“. Of course, here it is simply a technicality, explaining that the decision over what is a bug is in the hand of the software company hiring the tester.
   But this is an approach where one cannot get more reputation by learning more skills. Instead, one has to wait until someone decides how good he is, with few chances to appeal. Different people in different companies think differently, and the tester has no control over this process here.
5. Companies get the wrong picture:
   A large amount of bugs and activity from uTest-like websites is likely to inebriate the software developing company into thinking they’re getting great value (just as a large amount of activity around automation does in other cases).
6. ‘Unfair’ and harmful competition with your in-house testers:
   For companies that do have testers in-house, sending the application to be tested at crowdsourcing models may pose additional problems:
   - How do the ‘inside’ testers feel about that?
   - What happens when the outside users find bugs they missed?
   - And for the same bug… how come someone from the outside get paid for reporting it? What conflict of interest may appear when a tester realizes he can suddenly make money if he (or his wife, or his brother) reports a bug he found during work?
   - Does a bug-finding competition affects how the in-house testers do their work?
7. Update about different types of test:
   The first version of this article said that “There are companies that value regression tests, for example. As these usually generate fewer bugs, testers at pay-per-bug will not perform them. The same is true with other types of tests that raise questions and not bugs.”.
   Update: I recently discovered that companies that want to run regression or a set of specific tests do use uTest. They post the application with a list of scripted tests for people to follow, and pay for the completion of the scripted tests. So there are ways to do these tests.
   (However, in one such example that I looked at, there were no bugs reported in the script executions. What would be the results if instead of paying for the tests passing, the company asked for bugs? But this problem isn’t exclusive of uTest, it happens everywhere, I guess )

…

There may be more and other points for and against the model.
Please write them on the comments or in a private mail, get the conversation going on this. I may change my mind on this matter too, why not?

A couple of months ago I tried to access a site (now I don’t even remember which it was) and was greeted by the note below:

Let’s read that again: “The attack was not successful and our safeguards prevented intrusion“. Ah, nice. But Mr BudgetHostingWeb, can you explain how, if the intrusion was prevented, how the hell did the “server's hard drives failed during this process“?

Maybe the hard disk sensed that an attack was being attempted, and then committed suicidal, in some sort of self-aware sense that “it's better to die than have those hackers get the secret data stored in me!”

(Side note: If you ever had to deal with web hosting service’s backup, you know it is a nightmare. A service I used once had “hard drives problems” too and their last backup was a month old — even if they had claimed to do daily backups. No, I didn’t sue, and I think I wasn’t even surprised (Quality is Dead, right?), but I did change to a different host later.)

So now we have customers who lost all the data of his website.
If they had good backups, they’ll be able to upload it with the FTP Upgrade package — and if they hadn’t, well, will have to start from scratch.
And this is being communicated to him in such a lousy way… Note that the alert in the page is not targeting the site visitors (which do not upload files by FTP), it is targeting the web page owners. Why would an alert to the page owner be glued to the home page of a customer? Maybe because they didn’t send the communication in any other way?
(Maybe the hard drive with the customer information failed too, who knows?)

And the alert says “the attack was not successful“, which is what makes one (me) angry.
Mr BudgetHostingWeb, your customers are unhappy, they had a major data loss, and have to fix it themselves instead of doing business. Please re-write the message to something like “The attackers succeed to cause failures in some of our hard drives, and unfortunately we don't kept backups of your data. As far as we can discern, no information was stolen, but we are deeply sorry for this inconvenience.”

Because shaking yourself free of responsibility isn’t exactly the best experience you can provide your users.

Let’s sympathize with one of their customer, which is a great practice for software testers:

I did a search on Google to try to find sites that were part of the problem. Even thought a lot of time passed since the incident, Google found three (two of them do not have the text in their sites anymore, Google is outdated too, so it leaves us with only one customer specimen, Free Range Whippets):

Whippets are lovely pretty dogs, and the first link on the screenshot above sports the following notice:
 ”
 We regret that our host server was recently the victim of a serious hacking attack.
 The attack was not successful and the safeguards prevented intrusion however the server's hard drives failed during this process.
 All services have now been restored.  Unfortunately, our site was one of those affected, and we need to upload ALL the files again.
 Some links and photos will not work or appear until we re-upload.
 We hope to be fully back online soon but finding after six years of loading photos, they don't go back up overnight.
 Please visit us again and click all the links! 
 Thank you, Lori & The Whippets
 ”

6 years of work, gone! Is that what the safeguards were supposed to do?
The hacker attack wasn’t successful, but Lori still has got broken links in the whippets’ images .

I like books, better yet when they are good/useful books, and even more when they’re cheap — so I came to the fair prepared! I planned a budget (100 NIS) studied the catalog of discounted books and decided beforehand which books I wanted to buy:

- Book A: 43 NIS
- Book B: 30 NIS
- Book C: 12 NIS
- Book D: 12 NIS
- Book E:  9 NIS
        (106 NIS total)

Book A and B were books that I actually was interested in, and the other 3 I put on the list just because they were cheap , and were worth considering if they content looked cool.

When I got to the fair, I skimmed over the books and found out that books E and D weren’t that interesting, and that they would probably remain unread, so I dropped them from the list. Book C wasn’t that appealing too, but I left it on my cart meanwhile.

A few minutes later, I noticed that there was a very cool book on a subject that I am already studying — but that one cost 75 NIS!
My list was now over-budget:

- Book A: 43 NIS
- Book B: 30 NIS
- Book C: 12 NIS
- Book F: 75 NIS
  (160 NIS total)

As such, I called my wife, telling that I found other relevant books, but the whole list now exceeded the initial budget we agreed on. She gave me permission to double the budget and to spend up to 200 NIS. Yay!

Just before getting to the check-out teller, I discovered that there was a new volume from a set of books we have, and this was an opportunity to complete the series. Price of this book: 63 NIS.
Adding this one on top of the others would explode the budget again! I really wanted this book, and removing only book C wouldn’t help staying on plan — so I removed book B (a sacrifice for budget’s sake) and then C (not so interesting, anyway).

I went home with three books:

- Book A: 43 NIS
- Book F: 75 NIS
- Book G: 63 NIS
  (181 NIS)

I was happy with the books I took home, so was my wife, and plus: she was pleased that we stayed under the ‘second budget’.
And then it struck me…

So, this is my analogy for Exploratory Software Testing.
Many others are available:

• Michael Bolton has one here (charting unknown waters)
• James Bach has one here (and exploration and jigsaw solving)
• Ben Simo wrote one too (improvisational theater)
• Jonathan Kohl wrote his here (musical creation)
• … And now I present one of mine . I hope I am worthy of sitting with such eminent group.

But note: this analogy, as every other analogy, doesn’t have exact 1:1 similitude/parallelism!
It is intended as a line-of-thought, to help explaining the process and mindset… to clarify that exploratory testing isn’t a ‘quick, run and do whatever tests you can squeeze in an hour’.

He writes books, software, and gives interviews about the craft and business of software. Not only that, but (not surprisingly) he’s also got a blog.

Two months ago he wrote that reading your colleagues code check-ins is a good practice, and I think this is good advice. And good advice for software testers too: I read the bugs submitted by my team on a regular base, and it’s been very enlightening (often to me, at times to them too).

I don’t like copy-paste, but as an experience on the parallels between development/testing, I’ll copy his entire post, just changing the parts that are development-related to testing-related words.
“Great artists steal“, right? Let’s see if it is intelligible (my changes are in blue).

Do you work as part of a software testing team?  Here’s a piece of advice for you:

Read the bugs.

Every morning before you start your own testing tasks, use your favorite bug tracking tool to look at all the bugs that everybody else submitted the day before.

There is a reasonable chance that this advice is worth exactly how much you paid me for it. 

Still, testers at many of the best teams I have known do make this their habit.

Reading the bugs is very likely to produce two benefits:

1. The bug might get better. You might find something in the bug description that needs to be fixed. Or you may find a different direction to investigate and find the inherent fault that causes the failure. Or you may suggest another implication of the defect that increases the customer impact perception… So on.
2. You might learn something. Maybe one of your coworkers is using a technique you don’t know about. Maybe someone else has a tool you can use. Or maybe reading the bugs simply gives you a deeper understanding of the project you are working on.

Of course, use some common sense about whether this habit is practical in your situation. For example, if you work on a team of a thousand testers, this probably won’t work well for you. Then again, I assume if you work on a team of a thousand testers, almost nothing works well for you, so this habit might fit right in. 

(Posted with Eric’s permission)

– Hi, Nancy? (Name changed. And she wasn’t the testing manager, she was from the HR.)
– Yes.
– Hi, my name is Shmuel and I saw you’re offering a software testing position, the “QA engineer” opening…
– We have it, right.
– I saw that one of the requirements is to know the STD and STP methodologies, and I was wondering what are STD and STP.
– Well, if you have no experience on that, then the position is not relevant to you.
– No, no, I guess the experience is fine; it is just that the announcement made me curious. These acronyms can mean many things, you know? So I want to know how your company understands them.
– They mean specification documents.
– Does STD stand for Software Test Documentation? What does the STP mean?
– No, STD is for Software Test Design, and the P… Well, both mean specification documents.
– Why design? Is that the specification of the tests to run, of the testing system, or of the product?
– Why don’t you just look up the meanings on Google?
– I tried right before calling you. The acronyms are all over the web, but they have different meanings too (*).
– Listen, these are very important topics, if you don’t know them the position is not relevant. They’re very important, and very well known.
– I see. Thanks you Nancy!
– Thanks, bye bye.

Very important and well known. Well, I guess I’ll put them in my list of things to learn… As soon as I decide which meaning I want them to have…

Let’s see. I guess two of the best “methods” I use in testing are:

STD – Software Test Drawing: Scribbling in a whiteboard (pen+paper are good too, but my experience shows whiteboards are much better). This is especially useful when you’re working with someone else. With some squares and circles explanations become simple, concepts become clear, people can point when they talk, you can easily delete mistakes, and the best part is that usually there is always a joke at the end of the process. Whiteboard scribbling is a major testing tool for me.

STP – Software Test Peer (or Pal): If you are like me, your brain is constantly storming . Nothing beats thinking out loud or talking to peers and fellow testers/developers about the challenges you face — more often than not you’ll learn about a direction you hadn’t thought of or about a tool that can help. Works fantastically with STD .

I may start an “Alphabet of Really Useful Software Testing Things” — please contribute on the comments.

(*) In fact Google says D and P apparently can mean Design, Documentation, Development, Performance, Process, Plan, Programming…Pick your own definition. Most companies seem to accept STP as Software Test Plan, though.I still like that anyone can pick the definition that best suits his needs or business, though. So I’ll stick to mine .

Note to my employer: No, I am not looking for a job elsewhere yet .

I enjoyed the story very much. It is very exciting to see the passion he had (has) for his software and how he was committed to it. Plus, Ron is a great story teller.
The graphing calculator had all the ingredients of a cool app. It scratched a developer’s personal itch, and is a great example of NeoVictorian computing: built for people, built by people, crafted in workshop, inspired.
Actually, if we’re commenting on NeoVictorianism, Ron was one that really “woke up one day to find himself living in the software factory“. The night got very cold, they said the factory is going to close and he should move somewhere else. The cool part? He kept doing his individual craftsmanship inside the corporation. Secretly.

Now to our matter, testing. He’s got some very good comments on Software Testing (which he calls in the story as QA. Not his fault, but ours). So nice comments, that I’ll just transcript:
At (~20:24), “Our biggest problem was QA“. “Fundamentally, you can’t do your own QA — it’s a question of seeing your own blind spots“, and “it is expensive to do it well“.
At a moment in the story, two testing engineers came to offer help, as in their official work “they were really bored because they spend lots of times developing automatic test tools and now they’re at the point where they push the button and the test starts in the morning and they come back in the evening to see which API’s break“.
These testers were successful because they badly wanted to explore the new software, and also because they had good mathematical experience/background — closely fit skilled testers!

Ron later speaks about usability (~35:57).
He describes the process of the usability studies they conducted after they’ve been programming with users in mind. Ron (which understands well the importance of usability: “In a classroom, any time spent frustrated with the computer is time taken away from teaching“) says that even after all the attention to usability they had, they were surprised by the results of the study: “Sitting behind a two-way mirror, watching first-time users struggle with our software, reminded me that programmers are the least qualified people to design software for novices. Humbled after five days of this, Greg and I went back and painstakingly added feedback to the software, as if we were standing next to users, explaining it ourselves.” In the movie, he adds that they were “pounding on the glass” and that “all engineers should be forced to go through this, watching real users actually trying to use their software“.

There’s another interesting comment at the very beginning (~02:30). Ron comments on how straightforward typewriting work was on the firsts Macs, and how a newbie could go from zero to full speed in less than 5 minutes on MacWrite.
This is very curious: Even today, the computer is more useful as a typewriter than anything else. The uses had varied much, and nowadays professionals use the computer for every nitty-gritty important detail, but still, the first thing that comes to mind to most people when they think “computer” is “writing”. From journals to books to presentations, my perception is that the only thing people can do (on a computer) with no complex instructions is publishing. Want to publish? Open this app and start typing. But you want to render a 3D object? Search for a court lawsuit? Build a repository of data? Out of luck, will take years to learn it well.
Our hardware has evolved, and the purposes we use the computer for have changed too. But the form has stayed the same as 30 years ago. And (not) surprisingly, the interfaces of our apps still resemble the interface of that first MacWrite. Time to change how we interact with computers? I hope so. Science Daily had long ago an article that said “While much about the computer has changed over the last three decades–greater power, faster speeds, more memory… – what has not changed is the user interface.” They offered a gesture interface proposition, and other technologies like multitouch or brain-controlled interfaces are being researched. From my part, I’m ready to test the new paradigm as soon as it’s ready .

This is the full video, this is the link for the graphing calculator storry in text form (10 min instead of 55 ), and go here for a demo of the product at Apple’s launch.

Enjoy, and please let know your testing thoughts in the comments!

The last BotT (Bug of this time) was long ago, when we talked about testing and the Excel bug.
So now we’ve got a cool one, on which the most notable point is not the bug, is the submitter .

Bug 439858 on Fedora (a Linux distribution) was (supposedly) submitted by Linus Torvalds himself. He started the Linux wave on August 91, and is still posting bugs on March 08 – you gotta admire him.
I am aware that the submitter signing “Linus Torvalds” may not be Linus. It actually doesn’t matter, let’s just pretend it is for sensationalism’s sake (regarding the admiration, you still gotta admire him even if the bug’s not his ).

Let’s try to analyze the bug report, and the bug itself, to see what we can learn:

Linus uses a lot of witty comments. The bug starts as “youtube no workee“, which is a nice way of writing things in a forum or a youtube post, but we should not use such mocking language in our bugs – jokes or irony in bug descriptions can offend developers, who already have a hard timing accepting their code has got errors. Cem Kaner alerts: “it is easy to read a joke or a remark as a flame”.
You should not do this especially if you are a leader, influent people or the inventor of the OS — you are an example, why foment bad practices?

Then there is “fedora 9 not usable for wife“. That’s actually a good point. It is easy for programmers to remember that the average user is not tech-savvy and won’t tolerate complex workarounds. If this is true for common commercial (and Windows) application (where the programmer is interested in the sales to a large audience that expects intuitive software), this can be even more true in the Linux community (where they’ve got an incredible good will and skills, but sometimes prefer to add features over simplicity).
So explaining the Customer Impact in clear language at the beginning of your bug is a very good idea.
Moreover, I don’t think anyone would want to frustrate Mrs. Linus Torvalds. First because she doesn’t deserve it, being a nice person with a child-care background, and second because she’s a black-belt Karate champion .

“Steps to Reproduce:
1. Install current Fedora 9
2. Rick-roll!
3. No profit!”
Now, these steps to reproduce aren’t very useful. Even the title of the bug is more descriptive: “swf mozilla plugin – no youtube“. Everyone laughed at the mention of rick-rolling, but it only added confusion: is this bug related only to the Rick-Roll video? Any other video?
Linus added specific info to clarify: “I didn’t try a lot of videos, but I couldn’t find a single one that actually worked“. This point is actually a very good one, which teaches an important principle on bug reporting: There are points that are “conditions” to the bug (the bug only reproduces with the conditions fulfilled), and there are points that are ”incidental” to the bug (a setup or configuration that was true at the time of the bug but we cannot be sure it is part of a condition). It is very important to note the distinction between these when submitting a bug. And, of course, any setting which is neither incidental nor conditional should in most cases be omitted.

“Some videos just show a light gray background, some give the play buttons etc,
but in the latter case show only a black screen even when the red ball at the
bottom seems to moves along..“. Good! Information that is really related to the bug! Adding the link of videos that show the gray background and videos that show the buttons would be of help here. You (the tester) already visited all of them, why should the programmer search for specific reproductions again?

There’s some more information in there. The information is a sum of statements that help debugging (“youtube videos are supposed to work with swfdec“, “the adobe player won’t install on current rawhide“), statements that define the impact (“wife will kill me if she doesn’t have her videos“) and humorous comments. When you report your next bug, remember to separate these into clear sections: a section to discuss the user impact, and a section for additional debugging info that can be helpful. The humorous section you can leave out .

Le Grande Finale:
““Obi-wan Kenobi, you’re our only hope”“. He actually finalized with a Star Wars quote. Which is very cool if you’re the founder of an open source OS.
They should print this bug as a book. It mixes hackers, nerds, memes, Star Wars and jokes – receipt for a best seller!

—–

Before you post hate comments on how I dare to criticize Linus: I don’t. He’s a hero of mine and accomplished incredible things. I just used his bug for didactic purposes.

It is possible that I’ll post here a transcript or some text about this lecture. One option may be to add my comments from the lecture as notes in the powerpoint slides and post the slides here. But I don’t like powerpoint much as it does a poor job of passing information (it takes the written word and limits it into bad format and structure).
Finding the time, I’ll write the lecture in extense text.

In order to keep this post still useful, here goes a short summary of the talk:

Fault Injection: I base everything I say on the book “How to break software” by James Whittaker. The examples were done using Canned HEAT version 3. Canned HEAT is a good tool (with many bugs ) which comes free with the book.

Fuzz: The tools used on the examples were:
GUIFuzz: A custom dumb fuzzer based on the original fuzzer which can be found here.
FileFuzz: from iDefense Labs.
Driver fuzzer: Built with the Peach Fuzzing Framework.

All the tools and frameworks are free for commercial use as well.

When not from a lecture focus, then from a side comment or explanation. Below you’ll find some insights I gained from today’s lecture. When not from a lecture focus, the ideas come from a side comment or explanation:

1. Map everything.
   Vipul’s “I have a dream” talk described an utopic system, in which every single bit of information is not only saved, but mapped and related to relevant bits. In this dream, every single information is mapped, and you can traverse all info and predict taks efforts and chain-reactions.
   As he said, all the technology needed for such a system exists, it just have to be done (in an efficient way).    

   As such, we can start to gain the benefits of the systems by implementing easy thing. Like, for example, Requirements mapping to code. Why is that everybody talks about mapping the test-cases to the requirements, but no one mentions the Requirement to Source Code mapping?
   Such mapping would be very good to:

   • Prevent Gold Plating (number #30 here) (and see a programmer’s rant here);
   • Show the risks (for the application) of changing a requirement (and estimate the change effort);
   • Show the risks (for the requirements) of changing the application (due to a bug fix, for example).

   A Requirement mapping effort is not complete by merely mapping test-cases.

2. 
   Combine testing components.
   If two components are codependent (as in one’s functionality affects the other’s and vice versa), most of the times it is better to think of them as one (although composed) component.
   Whenever you have to do something in one, include the other. It sounds obvious when you speak about testing tasks, like doing a regression and/or a specific test. But it can also be true when you are dividing any ownership and resources. Keep close things close.
3. 
   Combine tests.
   Combine tests into composed scenarios. Run a number of tests – but do them sequencially as part of of a use-case. Instead of testing the find feature, do a combo hit: Open a document, find a text, replace it, save it again under the same name – 4 tests in 1!
   You still got to do all the tests, but the scenario combinations increase incidental complexity ,which usually causes smarter bugs.
4. 
   Software security does not end in the software.
   Business are made of software and data. Be it a collection of different software by different company, or one single solution by one company, the business workflow has a lot of stops and transitions. It is not enough to have a software adequately tested for security, if the workflow (wich is made of people) allows security problems. Authentication, authorization, configuration and more are problems that need to be dealt on the workflow level. This business process needs to be built with a security mindset since the beginning. As Vipul said a day before: “Security is not supposed to be tested, it is supposed to be tested for. And the same is true for performance and usability”.
5. 
   Models always lie.
   I am afraid of models. Since I discovered that many problems with requirements come from people thinking this representation is the real thing (instead of just a reflecting image with imperfections), I am afraid of models. In todays’ Test-Cases from UML lecture by Ofer Prat, he gave a wonderful example about modeling:
   London’s Underground was always mapped in it’s geographical form, it was a model drawn as close to reality as possible. Such a model was complex and almost unusable. Over the time, it evolved in a simpler and easy to follow model, but by then it was not accurate anymore – see this page for a graphical history.  

   So, models (and requirements) are meant to be straightforward directions. As such, they may lack from the accuracy and/or truth that reality presents.  If software is behaving in a bad way, and this bad way fits the requirements, it is time to re-tune the model.

More to come tomorrow!

For testers, thinking like developers is evil. If you think like a programmer, you’ll start excusing the software and will forgive the system’s bugs.

I am reading the very cool book “Dreaming in Code” by Scott Rosenberg, and I just understood a little bit more on why’s so bad sharing the developers mindset.

Scott starts its book with Chapter 0, a common practice in computer literature. When enumerating the reasons, he explains that “Programmers count from zero, not from one. (…) Why do programmers count from zero? Because computers count from zero!” and that developers are taught to think like computers.

This is a very interesting and basic concept.
Dr. Parnas in his “Strategic Defense System” paper (a real favorite of mine ) has a whole chapter called Why conventional software development does not produce reliable programs, in which he clarifies this problem — developers thinking (or trying to think) like a computer:

As we continue in our attempt to “think like a computer”, the amount we have to remember grows and grows. The simple rules defining how we got to certain points in a program become more complex as we branch there from other points. (…) Because one needs to remember so much (…), new problems are created when old problems are corrected.

The beauty in all this is that Testers should not exercise the programming rationale (when analyzing a software) because they’ll be mimicking the rationale of the very same software they are testing. They are limiting themselves to what the program knows about itself, instead of expanding their logic to people-centered matters — i.e. how the software will be used and what it is really meant to do.

Back to “Dreaming in Code”, Scott has an almost poetical way of writing the concepts developed by Dr. Parnas.
Because computers think from zero, “programming include little offsets, translations of ‘+1′ or ‘-1′”, he says. So:

There’s a space between (…) the way the machine counts and thinks and the way we count and think. When you search for explanations for software’s bugs and delay and stubborn resistance to human desires, that space is where you’ll find them.

Poetry.

I am fond of quoting.
Not sure why, but I like to quote. I guess it gives some legitimating to what I am saying.

So, the quote collection is available at this address: http://testing.gershon.info/quote-collection/. It will grow slowly, please check it regularly.

Uh, you didn’t notice? Well neither did I until I read it all over the internet.
See the post on SlashDot for scoop, and see its comments for some good laughs.

There are explanations all around about how this bug came to appear in Excel.
The best one is probably Joel’s one (you may remember Joel from this post).

Go and read it. Joel is very deep in there and explains why the binary calculations can be confusing, and how the binary notation standards may or not help you developing better software.

I say “go and read it” because I’ll not repeat his explanation here
I will, however, quote the best parts from the questions at the end — very insightfull!

First insight is a matter for pondering about bugs, small bugs and the perils of fixing such:

Q: Isn’t this really, really bad?
A: IMHO, no, the chance that you would see this in real life calculations is microscopic. Better worry about getting hit by a meterorite. Microsoft, of course, will be forced to tell everyone “accuracy is extremely important to us” and I’m sure they’ll have a fix in a matter of days, and they’ll be subjected to all kinds of well-deserved ridicule, but since I don’t work there I’m free to tell you that the chance of this bug actually mattering to you as an individual is breathtakingly small.
(…)
And let’s face it — do you really want the bright sparks who work there now, and manage to break lots of perfectly good working code — rewriting the core calculating engine in Excel? Better keep them busy adding and removing dancing paper clips all day long.

Second insight, on the perils of automated tests (an automated test sees only what you ask it to see, hence it has blind spots and will not report for side-effects!).
(Of course, as Joel says, it is doubtfull that manual tests would ever exercise rare/random calculations):

Q: Shouldn’t they be testing for these kinds of things?
A: I’ll bet that most of the numeric testing done on the Excel team is done automatically with VBA code. Cells containing this value display as 100,000, but from VBA, they’re going to look like 65,535 (since the number would be passed into the Basic runtime in binary, before the display formatting.) I’m sure there’s plenty of code to test display formatting, but with a bug like this that only happens on 12 out of 18446744073709551616 possible floating point binary numbers, it’s unlikely that any set of black-box tests would cover this case.

So I want to share with you these articles, with a bit of background on them, and hyperlinks wherever I can find them. They are presented in the (chronologic) order they came to my attention.

1)
The first of these articles is “Testing Job Interviews” by Jonathan Bach.
You may know his brother James Bach, who writes at http://www.satisfice.com/blog, but my contact with Jonathan came even before I was a tester.
As soon as I was told about the Testing Engineer position at Intel, I started reading about it in the web, just to make sure I know what I am entering into. After stumbling then upon some of the works by Jonathan, I sent him a mail thanking him for the great articles. Not only did he respond, but he wished me luck and sent me another article by him, about interviewing for a tester position. We exchanged a few emails more and that was all.
But this gave me a huge motivation for trying the testers’ position! The article summarized a bit of what a tester should think when facing a product, and gave me an insight on what a tester does and behaves. Jon’s reaction summarized how a great tester shares and communicates.
In summary, the “Testing Job Interviews” article is what started me into testing, and sure had the most impact. Unfortunately, I cannot find a link now , and I don’t have it within reach in order to attach it – will have to dig in the old Hard-Disks…

You can find writings by Jonathan on his old blog at Quardev or the new one here.

2)
The second is “When Should a Test Be Automated?” by Brian Marick.
If I had to analyze and review it thoroughly, it would take a series of post just for this matter. This is truly an article that grows with the reader. I read it at first when I was just starting testing software; I read it later when I was a bit more experienced. Every time I read it I learned something different, and I highly recommend this text.
One of the things it taught me was that time wasted doing other things (programming an automation, investigating a bug…) costs in bugs you did not find during this time. If for nothing else, read this paper for this.

Brian’s blog is at this link.

3)
The third text is “Software Aspects of Strategic Defense Systems” by Dr David Lorge Parnas.
I commented about it on a previous post about hard-to-test software.
This paper taught me about the difficulties of software testing and the responsibilities it comprises. It is written in a very organized and cleat way, you can read it in one sitting (I did it at the barber-shop ).

Just let me add another point:
4) Although not a specific text, the articles and writings by Michael Bolton are very good, and he is today my preferred author. Find him at http://www.developsense.com/.

Enjoy these, then!
And if you have your own must-read articles, send me your suggestions!

I found an article about this subject, which is called “Testing the Untestable: Reliability in the 21st Century“.

It is an article by the Los Alamos National Laboratory, and it has an ‘Approved for public release, distribution is unlimited‘ mark - nevertheless I am finding it hard to discover a link to the complete and final article, all I found is a comented draft here.
The complete article was supposed to be at this link (but the link is broken ), and apparently it is available here at the IEEE Xplore for members. Probably you can ask for it at the IRRA (Institute for Reliability and Risk Analysis) page.

In the paper, they reinterpret the concept of testing and metrics, due to the difficulty of understanding testing of complex systems. They say, “constraints imposed by the real world inhibit the ability of researchers to calculate the reliability efficiently and accurately”.

In the article, between other things, they state that testing must also be built on subjective and intuitive criterias together with the accurate approach academy has to testing. I believe this is correct, and also think that the idea is not so innovative… just read the two first posts, and the thread in the newsgroup.

So… if you want to read the article, download the draft at this link.

My past post on ‘apparently untestable systems‘ turned out to be a very cool experience. I talked with a lot of friends about it, and also had a very informative thread running on the comp.software.testing usenet group. The discussion was called “Is every single program testable? How do you deal with ‘untestable’ software?”

That thread was so helpful to further my understanding of testing, that I’ll put here a summary. It surely will make a good reference for me on the future, and I hope this will be useful for others too.

So, in order to continue and understand, read the ‘He Who Can Not Be Tested‘ post. All the discussion started from the questions in there:
- Do you think there are types of software which cannot be tested to a satisfactory degree?
- So how would you test software like this (of Dr. Parnas)? Or would you, as was his approach, just refuse to write and test it?

I’ll condense the responses here, and will add a note (and maybe a link) about each person. Note: The responses are edited and rearranged from the original replies, not really verbatim (although I changed the wording very little).

1)
H.S. helped a lot. He has a weblog were he discuss matters of development and test in depth, which was not updated lately, but there is still information to be found — hopefully he’ll add more content soon.

> Define ‘satisfactory’. B-)

Actually, all the responses we will see can be shortened to this question. Until we do not (re)define the satisfaction we are aiming to, we can not continue with our test approach.

> Yet another set of problems involves results that are not predictable. An example would be a module to shuffle a deck of cards for a game. One cannot directly test the randomness of the shuffle implementation; one must resort to statistical sampling and analysis.

(This is similar to the Fermat question in my previous post.)

> In fact, one can argue that exhaustive testing of any large application is simply not feasible because of combinatorial problems between inputs and logic paths.
> OTOH, if one is using testing to simply monitor the development process, then it may be possible to test any complex program, even nondeterministic ones. One can build a high degree of confidence in the quality of the development process (e.g., defect prevention) itself.
> Shops doing mission critical software do not use testing as a product reliability gate. They depend on defect prevention in the development process and only use testing to ensure the process is working correctly.
There is also a cool explanation of the testing role in the development process. Go and read it at the usenet.

2)
Ernie Englehart explained in a very practical way:

> One of the key components of a good test plan is a risk analysis. The calculation of impact x likelihood gives a risk factor. We then prioritize our test cases to focus on the highest risk items first.  By prioritizing the test cases based on risk we are mitigating and minimizing the risk as much as we can.
> For defense systems, obviously much of the testing needs to be performed with simulators. The design and functionality of those simulators is obviously very important. They can be very complex. The simulator projects follow the same methodologies of performing a risk analysis and then prioritizing the test cases accordingly.

3)
A response from Vladimir:

> To prove the concept you may not need to execute every single line of your code in every possible condition with every possible data value. Testing is about finding the least that covering the most (or even everything in the mission critical systems). Testing is also about probability. The more [of smart] testing you do the least probability is there that you have missed a defect.

4)
And I was much honored to receive a reply from Michael Bolton. He is one of my favorite authors since my beginnings on Software Testing. See his site (I keep him in my sidebar) and read his articles.

> Obtaining or attaining confidence in something is very different from testing something.  The testing of a product might be entirely satisfactory, where the information revealed by testing will reveal that the product is not satisfactory.
> The decision to build or not to build something isn’t a testing decision; it’s an engineering decision.  Engineering frequently involves building stuff that gets its first end-to-end system test on its first use.  Testing is a heuristic process that tells us things about the state of our engineering.  Some things don’t work even after extensive testing, and some things do work even after very superficial testing.

> If a system was decided to be built, we can test it.
> There’s a difference between product satisfaction and testing satisfaction. One outcome of a satisfactory testing can be a conclusion that a product will not be satisfactory.
> A degree of confidence should be agreed between all the involved, based on the project limitations and testing constrains.
> As no system is 100% good and trustworthy, we can reach a result that defines a system as “good and trustworthy to an agreed level of coverage, based on a known analysis of the risks”. Such objective can always be reached, concerning testing.
> “Difficult” does not equal “impossible”!

This BotT tells a story. A story about a car dealership that had the great idea of mailing thousands of scratch-off games to promote their business. What they did not know is that, by mistake, all of the 50000 cards had a winning notice behind the scratch-off layer. Instead of having to pay a US$1.000 prize, they soon discovered that they had to pay US$50.000.000!!
Go to this link to read the story, or at least to see the videos of the angry customers that came to demand their US$1.000: Koat News link.

I am not pointing a finger at any culprit, it is clear that it was a big confusion. And although it is an interesting story, the point that I want to emphasize is: How do you test a system that can not be executed?

You can not scratch the Scratch-Off cards in order to ensure they are ok. It would render them useless.
Think of it as a matchbox – once upon a time, there was a blackout on some neighborhood. The head of the family went to the kitchen to bring the matches, so they would have a little light. After failing to light even one single match, the man exclaims: “I can’t believe it! I tested each one of these matches just after I bought them!” . You get the point: How do you test a system that can not be executed?

Oh boy, let’s open a new trend: Process testing.
By process testing I mean testing any deliverable, even if it is not software.

So, how would you test the Scratch-Off cards, without scratching the 50.000 cards? Here are some examples I can jot down quickly:

1. Requirements Review: Ask if it is clear that 1/50000 should be prized. Ask again, and show it in the contract – if you show concerned enough, there is a high probability that the concern will spread to the publisher, and he will make sure to see the process is as expected.
2. Black Box Statistics: Open 5 random tickets. If you found 5 winning cards, return all the boxes (there should be only one prize!). If you found one winning prize – try some more, statistically you’ll never be that lucky.
3. White Box: Scrutinize the printing batch script, or whatever controls the machine. If you work together with someone from the publishing house that knows the language, you’ll be able to grasp exactly what percentage of winning cards is being printed.
4. Interface testing: See a draft printout of the prized and a not prized card artwork. They can send it even, by email, and you will ensure that the publisher is using a unique artwork as model.
5. Sample: Ask for a blank sample and for a prized sample. The rationale is similar to the previous one.
6. Process Analysis: When the printed card boxes come, ask to be shown which box has the prized card. If all is well, there will be only ONE marked box. Legal disclaimer: It may be illegal to know where the winning prize is – as it would spoil the surprise and competition of the ‘game’, making it unfair.
7. Legal Protection: Maybe you can also add protections clause to the card, like “my company is not responsible, rather the publisher company is” or “prize is dependant on XXX, where XXX is some other condition”.
8. Marketing Protection: Another option will be tying the prize to a discount when buying a car (the scratch cards intention was to grab the attention of those 49.999 people who did not won the prize. With this discount-tie, the attention-grabbing would be the same or bigger, and the risk taken would be 1000 for car sold.).
    
   Not all of these techniques really protect you, but if done, these would give you enough confidence on the correctness of the data.

I could finish here, without talking about software, just to make the point of abstraction.
But as I want to keep my readers, and make my blog good for the world, let’s see some software points to ponder:

Is there software which can not be tested?
Well, I believe that although any software is testable up to some extent, yes there is software which can not be thoroughly tested.

The most common examples are from Critical/Safety systems.
Dr. Parnas refused to build a defense system software against nuclear missiles, stating that “The inability to test a strategic defense system under field conditions before we actually need it will mean that no knowledgeable person would have much faith in the system.“. He was right, the developers/testers would be able to test and simulate some parts of the system, but the final system as a whole would be impossible to test until a real war was in course and you had nuclear ogives being shot at you.
(You can not even test in at a desert location with test-intended-nuclear-bombs, because the Nuclear Test-Ban Treaty prohibits.)

==> So how would you test software like this? Or would you, as Dr. Parnas’s approach, just refuse to write and test it?

Another kind of untestable program I found in the Wikiwikiweb of Software Development:

Imagine its a few years ago, and the application under tests is a program that has a feature to disprove Fermat’s Last Theorem. Within the code, the function for it is:

bool fermatIsWrong(x, y, n) {
  // return true if n > 2 and nth root of (x^n + y^n) is an integer
  // else return false
}

At the time you are testing this function, you do not know whether there exists a set of numbers {x, y, n} which lead to a true result. And if the set exists, neither you nor anybody else knows which the values are.
So, this is a program to which you will never be able to cover 100% of the code paths in your tests. You’ll be able to test extensively the ‘false’ return value, though…

==> Again, how would you test a software like this?

Feel free to use the comments below to reply. Let the world know your Testing Thoughts!

If you are an avid reader, and read the first post of this blog, you know that this blog is was password protected.

Since last week, the page is accessible to all, and may be read by everyone. As of now, no other page on the net links to Testing Thoughts, so it will be a calm place for a while.

In this opportunity I want to thank the handful of people who had password-access to the posts, and gave me so-much-wanted feedback. Thank you, you know who you are, and you can now forget the password .
I want to explicitly thank Josh for his motivation and explanation on why opening my blog to public view is good. Earned a link, see?

So old and new readers, you are welcome.
Please share and let all know your Testing Thoughts. Use the comments to contact me directly and ask questions (until I build the ‘contact’ page).

On to writing the first webwide-public post…

1) “I’ve never seen the same solution work in every place; or in more than one place”.
The above, is quoted (almost) literally from Bernard’s “Systems of Systems” lecture. He described the well known problem, but was unable to suggest a solution.
Any solution you find to any problem, can never work 1-to-1 in your problem. Adapt. Don’t be afraid to change you environment/process too, but usually both should change in order to succeed.

2) Critical tests are done at the end. Live with it – plan for it.
On the “Performance Tests” lecture, Lior Katz said that it is a fact for almost all systems that you can not do proper stress/load/performance tests until the system is stable — which usually means an advanced stage of the project.
This leads to important/critical bugs to be found at critical/important phases. One way to deal with it is to hope nothing very bad will happen… the other is to be conscious of the danger and be prepared to deal with urgent problems in the last milestones.

3) Care for the important level of your abstraction.
Apart from the obvious eye-opener we had in the “Building Blocks” lecture, which taught us that GUI automation should be done in layers, like any other automation, another insight was that you should not care (so much) about ugliness or repetition in the abstraction layer. Three reasons for that:
   1) The most important is that the test-cases layer is easy to implement and easy to maintain (or maintenance-less). That way new test-cases can be added quickly and by unskilled persons.
   2) The abstraction layer is always finite (as it deals with a finite number of states, or components).
   3) From another lecture, “Automation from day one”: “Do not test a system with an equally complex system“. If sometimes repetition or duplication can help your final layer to be simpler, it is better.

1) When you measure an object’s weight, the weight do not gets affected. Similarly, when you measure software’s quality, you don’t have influence on the amount of quality.
That’s from Rex Black’s lecture (10 worst things).
 The lecture was your standard lecture in which a lot of common-sense ideas are put on a slide, but the clarity, order and side-info gives all a better meaning and things get adjusted on your head. I like these lectures, they help to organize your mind, and surely help you to articulate these thoughts later.
 The idea in bold above is a very smart one: Don’t try or pretend to be the Quality Factor in the product you are testing. Don’t be a Quality Assuror. Just because you are a Testing Expert, and know how to measure the quality of an application, does not mean you have the power (or meanings) to increase its quality. Testing is not to improve quality, testing is to give a quality measurement. It is a decision making tool.

2) Don’t be afraid of producing a large amount of test cases. If you need to discard a part, at least you’ll know exactly what you’re giving up.
That’s pretty much self-explanatory. It was one of Vipul Kocher’s points on the “Verb and Noun technique” lecture.
This technique allows for a quick and easy mass-production of test cases. And, for those who are afraid of being overwhelmed by all the test cases, the answer is: “No one is telling you to test them all. But now your choice is conscious.”

3) – Company ‘X’ has no written development requirements whatsoever. It is a bad thing?
   – Yes!
   – It depends.
The above dialog happened during the “V Model” lecture by Geoff. He was talking about a branch of some company (he named it, I wont. Just know that it is quite big…) that systematically don’t have written requirements.
The categorical reply that one participant gave (and most if not all held too) was the obvious – you’ve got to have requirements to be a good boy. How can I test without requirements?
That’s why the answer is so surprising and so refreshing. If an answer is categorical, it is probably wrong. Your cognitive bias is fooling you again.
If this company has a process, on which all stakeholders agree, and on which the testing department has agreed – then it doesn’t matter so much whether there is place for a ‘Write Requirements’ checkbox on it. If the testing department agrees to test without requirements as we know them, perfect.
Summing up, not everything that is different is bad. If it is well analyzed and well controlled, any process can give you results. You can decide on what is important and how much you want to risk.

3.5) You can use a process model as a display of your place on the project, without having to follow the process.
That’s actually the summary of the lecture. Geoff presented a case study, where although the process was not led by the V-Model principles, and did not follow the V-Model path, the V-Model was used as a clever instrument to show all involved parts what are the dependencies between development and testing. It is a clear diagram to show what parts of testing can/cannot be done with/without proper effort from the development side.
Of course… this rule is true for any other diagram/process, not only V-Model. So you probably can choose the model that better suits your necessities. Just be sure not to be biased – anyone can pick the model that fits its own interest and try to apply it to the company reality… Managers should be smart enough to spot where the discrepancies are.

4) Don’t be the one to harms MS Windows, or your own application.
On the “Testing the Windows* registry” lecture, Michael Stahl explained that messing with the Windows Registry can surely disrupt your OS system. Or your application. There are even disclaimers on this everywhere. But he also explained that it only messes the system if you (ok, the programmer) let it happen. If all registry inputs are checked and validated before being used, nothing bad will/shall happen.
There is no excuse – the registry is under the developer control (not the other way around)!

In 1979 (yes, I am that young ), Glenford J. Myers wrote what was to be the first classic book on Software Testing.
Called “The Art of Software Testing“, this book is very old, and written with old software in mind – most technologies and methodologies existing today were not dreamed of back then.
Well, it turns out the principles of software – from writing it to testing it - are still pretty much the same. Software is still written on sort of punchcards, the punchcards just got more complex and flexible… The book content is surprisingly relevant for today’s testers – so much, that it was updated only once in almost 30 years.
If you haven’t, read the book. Unfortunately, I have read it only partially meanwhile.

Maybe the most known contribution of Mr. Glenford in this book is the testing self-assessment challenge in the introduction.

It goes like this:

A Software is given, which receives three inputs (numbers) that define the size of a triangle’s sides. The SW can, then, categorize the triangle in one of these categories: Equilateral, Scalene, Isosceles or Invalid.
How many test cases can you build for testing this triangle?
The more heterogeneous the cases the better, and: The more effective, better yet.

Myers brings then an assessment criteria. If the test cases you came up fit into enough categories such as the ones below, you are in a good way:

• Test cases that represents a valid scalene/isosceles/equilateral triangle.
• Test cases that represent valid isosceles triangles such that there are all three permutations of two equal sides.
• Test cases in which one side has a zero value, or negative value.
• Test cases that represent invalid triangles.
• … 

The example is famous, because it grasps clearly the distance between coding and testing. Writing such a program may take about 30 minutes of work to a programmer (I did this first version in 20 minutes), but testing the program can be a never-ending task.

We will use this problem as the example to do all the acceptance tests from FitNesse.

We’ll have this DLL – based on the C# code above – and we will connect this to our FitNesse server. We will then start checking inputs and outputs. All this in Part 2 of the FitNesse Series!

1. Google Search – “software testing triangle equilateral”
   http://www.google.com/search?q=%22software+testing%22+triangle+equilateral
   You’ll find most of what’s online and cool here.
2. Example with working application (!) for you to test:
   http://www.testobsessed.com/2007/03/21/testing-triangles-a-classic-exercise-updated-for-the-web/
   Quality Tree Software’s Elisabeth brought the triangle to the   attention of all again when she posted it on her website.
   The working example by Elisabeth: http://www.testobsessed.com/exercises/triangle.html
3. Challenge Definition by Alex Samurin
   http://www.geocities.com/xtremetesting/TriangleTest.html
   Alex has a short explanation of the original problem as it was presented by Myers.
4. Malik’s Code Inspections
   http://codeinspections.blogspot.com/
   Malik summarizes in two paragraphs the two most interesting points on the abovementioned application by Elisabeth: Floating point calculations and sorting algorithms.
5. The exercise, Bollywood style
   http://venkatreddyc.wordpress.com/2007/03/24/taking-on-testing-triangles-a-classic-excercise/
   Venkat has written some insights (in a dramatic voice) on the triangle javascript app. I recommend this one, because the pace and steps are very didactic.
6. Last but not least :
   My post on it: A Triangle mystery (not the Bermudas’). I turned this post into a page/article, ’cause it is a useful subject.
   Pay attention, I am not trying to really test the triangle application. This is new stuff – we are using the triangles to test the FitNesse framework! Go for it!

Joel says that nothing improves morale and efficiency like private walled offices. I’ve worked in both way, and in two occasions companies I worked switched methods (from closed environments to open spaces) - in this experience, the gains of working on wall-less and open spaces are visible. Suddenly, everything is quick and everything is clear. No more leaving things to deal later. No more company-wide procrastination.
My advice: break those doors and those walls.

Everybody loves Joel.

Everybody loves his articles, his jokes and his books.
More than that, people love to disagree with him, so they can look smart and judgmental. The funniest part is seeing the judgmental faces they do.

In 2000, Joel wrote a cool article on “Top Five (Wrong) Reasons You Don’t Have Testers“.
He’s got some great information there, and in this post I’ll just comment it (see the ‘Annotations’ tag?).

Ok, so go on (link) and read the paper.
In his paper, Joel debunk 5 miths of companies who won’t hire testers:

1. Bugs come from lazy programmers.
2. My software is on the web. I can fix bugs in a second.
3. My customers will test the software for me.
4. Anybody qualified to be a good tester doesn’t want to work as a tester.
5. I can’t afford testers!

Joel has intelligent advice on the first three points (you’ve read them, right?), so I’ll not discuss them. As for the last two… lots of comments!

Regarding point 4, Joel’s article is a bit outdated.

• Joel says that “most people who are that smart will tend to get bored with day-to-day testing, so the best testers tend to last for about 3 or 4 months and then move on“, and gives some ways to make your testing scene look more inviting.
  C’mon! If your testers become bored after 3 months, you are probably hiring the wrong people. Good testers are passionate about tests, and know what it takes to be a tester. They know the work can become boring if they let it, so they manage the work in a way it will never be boring. Good testers are built for testing, even before you hire them.
  Myself, I started to work on testing out of curiosity, after a few years of development. Man, this is fun! The analytics mindset of a tester cannot be compared to the algorithmic mindset of a developer. They’re two different things, that level on the same job-satisfaction. Humm? What do I mean with Passionate Testers? Read the blogs of Braidy Tester, and of Pradeep, you’ll know what I mean.
  So, Joel, hiring those non-traditional workers as your test-base will just give you big turnovers and you’ll never gather that cool techie testing group that laugh about bugs found.
  A funny quote from the article: “Recognize that you will have a lot of turnover among your top testers. Hire aggressively to keep a steady inflow of people.“. I doubt anyone would treat the development team like that. It should not be different in the testers team – - recruit a winning team, treat them like kings, and you can expect them to serve the company for years to follow.

Last point, point  5. Again, not a very accurate/scientific approach:

• We can summarize Joel’s reply in this point as “Don’t skimp on testers, because you can find some cheap testers”. That’s the same as saying “Don’t skimp on Customer Service, you can hire any outsorced company on India to answer your phones.”. Ahh! Doesn’t make that much sense now, does it?
  The real reason not to economize on testers is that bugs and defects will cost much more than your testing team if you don’t test. If your customer find a bug, one of these which you could have avoided, the sum of costs of all the activities related to fixing and re-releasing your product is big – - very big (not to mention the harm to your reputation!). If your software is a shelf-product, it is even harder and expensive. If you are on hardware, the mere thought is making you pale.

Of course, I am not expecting any answer from Joel, but I like to believe that he agrees with me.

I’ll (as soon as I get to do a lot of things I want to get done) post some impressions and examples on FitNesse, the cool tool for cool testers.

What I plan to do:

1. Install and set a FitNesse setup;
2. Make a simple standard easy program – the System Under Test!
3. Start testing and making the program better with FitNesse.
4. And all this, with screenshot goodness!

When will that be?

• Ouch. It may take a long time. Let’s talk about this again on July…

Meanwhile: “FitNesse is designed to support acceptance testing rather than unit testing in that it facilitates detailed readable description of system function”. More information can be read in FitNesse’s website.

In this country, 95% of all accidents are due to reasons which include “human factors”. While 2005 statistics show that the “Deaths in Accident” relation to the population-total was low when compared to other European contries, the same statistics demonstrate that Israel is one of the worse countries on the “Deaths in Accident” compared to number of vehicles.
That means: Our vehicles kill more people than vehicles in other countries do. This is a very scary conclusion, and I learn from it that our drivers are careless (or could not care less!) about road-accidents.

Ad Campaigns can effectively increase consciousness when done right.
Here is how the Road Safety Authority approached the problem, and how you can adapt this in your tests.

See the picture:

The Caption says: “He is close to your bumper. He is fast, he is furtive… He can be unexpected… And if you don’t look for him, you’ll not see him. Slow down and look for the kid in the road!”

Another strong advert in Radio went like this: “He is approaching, and he is fast. On the road, always look for the man with the helmet!”

See the pattern? It is not “look for some kid that may be around”, or “check if there’s a motorcycle”… Stop and find that motorcyclist driving overthere!
The common to both announcements is that you are left with the impression that there are children and motorcycles everywhere.
They ARE there, somewhere where you can’t see them. You don’t want to hit them, so please be careful.

Once, when talking about SW testing with a person new on the subject, I explained the process of bug fixing, something in the lines of: The developer fixes and reports the fix in the next oficial version. You then have to look for the bug to reappear until you find it, by either changing the environment or parameters. If there is no reproduction of the original bug, you try to find a different bug related to the first or to the fix. If at last you don’t suceed, you give up, acknowledge the fix, and go find new bugs using the normal methods.
The new guy was surprized by the approach: Is this the attitude to have? Do we really want the program to have many bugs?
To anyone with a bit of experience it is clear — the bugs *are* there. All around, all the time. If we don’t find them, it is because they’re either hiding or of very little effect.

As soon as you realize that the program you are testing is bug-filled, your outlook changes. You now are a bug-hunter looking for the prey. And no one can fool you by saying there are no such things, because you know it, you can feel them breathing.

Some statistics on our side of the story.
Steve McDonnel, in his fabulous Code Complete book [Section 25.4], brings some bugs statistics (please lookup for details):

• - Industry Average is about 15 to 50 errors per 1000 lines of code.
• - Microsoft, in one department, at one stage, experienced from 10 to 20 defects per 1000 lines of code (I usually don’t use KLOC notation, but this statistics are different)

I’ve heard people telling that there is a potential defect for every two lines of code (!), and that for every bug fixed there is an additional one inserted in the code (which is hopefully of minor impact). As I’ve never seen this written anywhere, and they seem a bit exagerated, I’ll not count it officialy. But it is the right mind-set!

The truth is, unless you believe/know that the bugs are there, you will not be able to find them.
And, for the programmers reading this, you will not be able to prevent them. As Steve says in the same book [Section 26.1], “It’s hard enough to find an error in your code when you’re looking for it; it is even harder when you’ve assumed your code is error free”. In “The Software Development Process” presentation, Robert Brady explains that the belief in bug-free code doomed IBM to work bad, slowly and lousy at the 80′s. Be sure to read the example.

In conclusion, there is no such thing as bug-free code, but the impact of bugs can be minimized with proper testing.
Remember! He is close to your customer. He is fast, he is furtive… He can be unexpected… And if you don’t look for him, you’ll not see him. Slow down and look for the bug on that program!