Last month I visited the STAR East conference. It was over two weeks ago, which on the internet age makes it old news.
I follow my own clock, however, and will start posting about STAR East only now. Scott Rosenberg would call this type of blogging ‘history’ instead of ‘journalism’ (link), and it sounds just as fine.

In next blog posts, I’ll write about what happened at STAR. What happens in Orlando doesn’t stay there.
But on this one, it’s a bit about what happened in the way there.

For some time before travelling, I’ve been thinking about physical problems that resemble computer bugs. “Spirits in the material world”. For example:

In this picture you can (barely) see a water bottle that was forgotten by a worker on my neighbors’ roof. Just like a programmer would forget a temporary variable she invented to get some refreshment from the code inherent limitations, a worker will forget his own refreshment once it isn’t needed anymore.

This other picture is of a bathroom urinal row. I bet the urinals looked beautiful in the catalog, with their slim form that doesn’t stand out of the wall too much. But when “implementing” the bathroom, not only the beauty is lost when you build the necessary dividers, but the very same low-footprint of the urinals makes the rest of the environment unclean. How many times a company/developer makes a bad decision of, say, a coding library, just to discover that the very ‘flexible’ API promised by it is the part that gives them most trouble? How many times they decide to retrocede the and change the library? Ok, ok, tasteless example.

It seems that the same thinking process that makes a programmer/tester miss a detail when coding/testing happens in any other aspect of real life. I’m reading “Why we make mistakes” by Joe Hallinan and this idea seems to be congruent with many examples.

Well, with that mindset, it’s a sure thing that I would find bugs on my trip. Not because the trip was bad — au contraire, it was a mostly pleasant journey. But analyzing things and trying to assess their value is a good exercise for testers.

To make the exercise better, I decided to categorize the bugs and problems into the 8 parts of the HICCUPPS mnemonic consistency heuristic for analyzing a bug. It is a great way of building yourself an oracle to assess a potential problem, and I learned about it first from Michael Bolton‘s “Testing without a Map” article, highly recommended.

A quick explanation of the heuristic: One method of assessing any phenomenon is by comparing its consistency to a different (but comparable) source. The 8 aspects this mnemonic suggests are:

Is the phenomenon consistent with a comparable idea in:
H	History (e.g. the same function in the past)
I	Image (e.g. what the company would like customers to think of it)
C	Comparable products (e.g. the competitor’s )
C	Claims (e.g. a user manual statement)
U	Users’ expectations (e.g. what users have become used to)
P	Product itself (e.g. the same function elsewhere)
P	Purpose (e.g. the aim of this very function)
S	Statutes (e.g. a certification check)

To illustrate better the method and to share the fun of travelling, here goes a list of interesting events encountered on the trip:

[H - History] Airlines/Airports are stingier than in the past

It’s been a long time since I last travelled. So my recollection on how it works left me with some expectations, like having free carts at the airport, or having free peanuts during short flights, or having free headphones in the long ones.
Surprise: A cart costs US$4.00, a headphone costs US$5.00, and peanuts only if you brought from home! I was lucky I had my headphones in the carry-on case, but would be disappointed if I had to pay for airline ones.

This is interesting: Is that a problem? Is it a bug? Well, certainly it is inconsistent with the (far) history of tourist aviation, but it may be a good business decision. And fixing a bug (or deciding something is one) is a business decision.

[I - Image] Airlines seem to content with poor software

US Airways have a personal in-flight video system for each passenger (on long flights). It is great, has a touch-screen and showcases movies, music, even TV shows.
But the system is slow. It’s so slow that you often get confused by what it is doing — Is it stuck? Should I press again? Which of my last clicks is he processing now?

A funny thing is that at the beginning of the flight, the attendant announces: “please expect a delay after pressing the buttons on screen, it is part of the normal operation of the system”. What? In the age of iPad and other rapid touch-screens, is it really part of normal operation? I observed the other travellers, and saw at least two that were frantically pressing the screen waiting for answer.
We all also hope that the cockpit controls respond faster…

This one has two interesting points:
- I had a glimpse of the system rebooting when the aircraft landed. It is a Linux based system, and has a quote in its boot sequence that would make any software tester laugh (I laughed): “All bugs added by David S. Miller” , which turns out to be a real line in the Linux Kernel (David is a programmer for RedHat). Great attitude, Dave!
- US Airways have, apparently, tested a different in-flight system, that is not available on A330 aircrafts. According to this review, it is much faster.

[C - Comparable Products] Top storage is smaller in international flights?

The overhead storage in the transatlantic trip was only deep enough to have the luggagge stored sideways. Good? Bad? Well, we don’t know (yet).
The storage (called overhead bin in the lingo) is made by a company called C&D Zodiac, and they look very professional and thorough. So the storage space is good? We still don’t know.

One thing we can do, is to try to learn from comparison: The domestic flight had an overhead bin too, but in this case the carry-on luggage fit on its length too, almost twice the depth of the other.
How come? International flights have much longer duration, and it is normal that people will bring larger/more carry-on (sandwiches? Games? Stuff for the kids?). So why is storage room smaller in international flights? A bug?
OTOH, on domestic flights more people are expected to bring only a carry-on (they stay less time at their destination), so maybe there are more/bigger carry-on on these flights. So what? No bug?

And that is exactly the point of interest here:
The mnemonics/heuristics don’t tell you a definite answer, but they help you to think.

[C - Claims] Where’s the Kosher food?

US Airways cancelled my flight back, but I will not complain about that.
However, they did claim that there would be an acceptable solution, and that the airport hotel would be able to provide Kosher meals. And none of these claims stood to the truth.

The airport hotel had no Kosher preparation, and US Airways made it very difficult to find an acceptable solution. It took three hours of phone calling and walking between their offices until they agreed to provide suitable meals from outside.

A claim, even a verbal one, is a claim that your customer will expect you to fulfill.

[U - User Expectations] What’s with the default volume settings?

The aforementioned in-flight entertainment system had many settings. One of these was the volume setting, controlling how loud the movie you are seeing is.
However, this setting was re-set for every movie, which means that every time you moved to a different TV show, it will be at a loud volume again!
And I think (I did not ask for tools to verify it at the time) that every time the attendant/pilot talked on the announcement system, even when it was during a movie, the volume was at the loud volume – again.

My expectation, as a user of the system, is that once I set a volume level for one show, it stays on that level for other shows. I got startled by loudness once and again.

[P - Product Itself] Now, where can I change the diapers?

This is my favorite one.
Bathroom stalls at airplanes have this platform that you can detach from the wall and use it horizontally to change the diapers of your little kid.
Bathroom stalls at airplanes also have this new foamy liquid soap, strongly attached to the

But, as you can see on the picture, the soap was installed right in the place where the platform should be based on. There is no way to use the support in order to lay the platform down.

So this is a perfect example of two parts of a system that are inconsistent with themselves — or, in this case, not only inconsistent, but the feature added later, practically nullifies the original feature.

[P - Purpose] Is your seatbelt fastened? Is it? Not?

I noticed during the flight that the flight attendants had to check if passengers had their seatbelts fastened. So they went from passenger to passenger, trying to discern if the seatbelt is locked or not. Not always an easy task, as passengers are usually holding things or use a blanket.

But if this “check seatbelts” function exists, why isn’t the automatic system of the aircraft aiding? Why isn’t there a little row of LED lights that show which seatbelt is fastened and which isn’t? Attendants would (maybe) have it all much easier.
It appears to me that without this simple system (hey, cars have it) the aircraft system is not serving its purpose in every way it could.

[S - Statutes] This page intentionally left blank

This one is missing. Because:

• I don’t think I noticed anything that was against a statute or law.
• It would be irresponsible to suggest things that are against the law in this (or other) industry — especially as I am not a domain expert.

As you see, a 12 hours flight is a lot of time… These are my notes from trip to (and back from) Orlando.
I am interested in learning about other methods you use to analyze bugs/value in the physical world. Share in the comments!

Now let me start that post about the actual STAR East lectures and networking…

This was an interesting exercise.
What took my interest in this one?
- First of all, I liked Pradeep’s post about learning to code.
Learning to code is an important skill for a tester. I can relate to it because I am trying to learn PowerShell. I program in some languages (C#, C++, PHP, VbScript…) and have a programming background (after my Computer Systems Engineer degree, the ‘natural’ path was to code, and I worked as a programmer for many years before finding Software Testing), but PowerShell has different approach and paradigms and learning it will definitely be fun.
- Second, I had an impromptu vacation day, and I thought it would be educational to use it for a peer testing puzzle. I thought I could win this one quickly (I was mistaken, it took me long).

So I downloaded Pradeep’s application, and got to work!
Rather than just trying to solve the puzzle, I looked at it as if the mission was:
“This is commercial 'roulette' style game. Stakeholders want to know it the game logic can be broken/learnt, which could mean a substantial loss of money when people start winning every time.”

Testing Hint: Cem Kaner says “Testing is an empirical, technical investigation of a product, done on behalf of stakeholders, with the intention of revealing quality-related information of the kind that they seek“. So it is important to set straight what is the information to seek. Which kind of bugs to look for?

In this blog I describe my attemp to answer this question in an exploratory way.

Now, Stop the press!
You may want to go to Pradeeps post, download the file, and try it for yourself before reading my solution. This post will be waiting right here when you’re done.

…

Here is the report of what I did, the line of thinking during my experiment, and the conclusion:

• 1. First, I ordered my environment to allow efficient and organized work:
  1. Increased the Screen Buffer Size of the Command line I was using to 500 (it turned out that I would do well on less, even half, than that (see point 2.b). But it didn’t hurt).
  2. Renamed the executable to be shorter and without number version: ren fnemo_1.7.6.exe fnemo.exe
  3. Changed the prompt to something shorter, cleaner, and that gave me context about this task against the other Cmd lines windows open: prompt $Cfnemo$F$S$G

Testing Hint: Organizing your testing environment before you start will likely help you during your tests. After you’re in the heat of fight, it will be harder to stop and get organized.

• I tried some basic executions of the program, to grasp the feeling of what it does and how hard it is to find Nemo. This also taught me what are the messages returned by the application when Nemo is found, and when Nemo isn’t found. They’re “Found Nemo this time!, Nemo Gill Bubbles SharkTooth Flow Phamplet Stinger” and “Ah! bad luck, didnt find Nemo this time!” respectively.
  1. I also learnt that Pradeep used Perl and Perl2Exe to do the app.
  2. Additionally, the application clears the screen at every execution, so the big command line buffer is not so helpful.

Testing Hint: Planning or scheduling your testing before you experiment a bit with the software can be done. But you risk being completely wrong in your assumptions about the app. In our case, there were no big surprises, but there could be.

• Sarmila commented on the Challenge page about disassembling the executable. Although by learning all the assembly one can learn about the rules that move the fishes around, it would be extremely difficult and overkill. But this made me think on how Perl2Exe works… Maybe it just wraps the Perl interpreter and the Perl script together? If so, what if the script is stored internally in clear? I tried to open the app in an Hex editor, but no, the Perl script isn’t in clear text there, it is obfuscated.
  • I also tried the useful Strings tool by SysInternals. My intention was to see if, maybe, the array of fishes could be seen at the app code, in order to learn the initial order of fishes. It couldn’t.
  • That ended my cheating session , from here on I used only a black box functional approach.
    • Well, it is not really cheating because there were no rules against that. If you do a full reverse engineer of the code, the answer is legit too. But I preferred the straightforward exploratory approach: learning from the software functionality as I use it, and use this learning to redirect my next steps.
• .
• By now, I knew that Nemo changed places between plays.
  1. Nemo also change places between invalid parameters (which may or not be a bug).
  2. Moreover, the other fishes change places too, even their placement related to Nemos’ placement changes.
  3. So I tried to see if Nemo will return to the same place in any consistent way or number of times. Running that is easy, you just enter “3″, “3″, “3″… and count where Nemo was found and where he wasn’t.
     • I discovered there is regularity (for “3″, its every 2, 3, 17, 5, 10, 5…), but the regularity was irregular enough, and also changed for different positions. This was likely a consequence of the real logic, rather than the logic itself. This correlates to the movement of fishes, but does not cause it.

Testing Hint: Take note of your steps while testing. Using the Session Tester tool is a great idea. For this session, I used MS OneNote because it supports rich text, and I could use tables.
Is that like saying that rich texts helps rich tests? .

• Another detail that helped me in the game was knowing the developer and the purpose of the challenge. This puts a lot of content in the context. Pradeep is a rather playful and would put some tricks into it.
  1. First I tried to see if Nemo could sometimes be in any placement bigger than 7 (there are 7 fishes, places 1 to 7, but there were no rules as to where the fishes were limited to be).
  2. I tried in a toss of up to 220 attempts, and Nemo wasn’t in place 8 even once. So I let this idea on StandBy for now.
  3. Then I tried… what if the “Minimum attempts” input at the beginning of the app affected where Nemo appears?
     • 10 is the default for the “Minimum Attempts“. Nemo is at place 1 at the beginning there.
     • 11 attempts chosen: Nemo is at place 1 at the beginning too. Maybe it does not affect?
     • 12 attempts chosen: Nemo is at place 4! Bingo!
     • 13, 14, 15, 16 showed that the original places were cycling at ’3′ intervals. That means that when trying to find Nemo on the default 10 attempts, it will be in the same placements as 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46, 49, 52…

Testing Hint: let the test and what you learn during the testing guide you. New information you find should set you in a new direction.

• You’re seeing that “220 attempts” above, and thinking if I really entered each number manually.
  1. By this time, I decided that I was in need of some automation to insert inputs to the application.
  2. The original app had no apparent automation capabilities, so I decided to use the internal input redirection of the Command line:
     • By using the “<” operator, I could redirect an input file “input.txt” to the app using “fnemo.exe < input.txt“
     • I extended that to “fnemo.exe < input.txt > output.txt“, which gives us with a complete dump of results that can be searched with Notepad.
  3. We had an easy automation now, and this can help our sapient testing. I was exploring, and using automation to help me try different things help me explore my possibilities and get a bigger picture of the situation when needed.

Testing Hint: Don’t be afraid of automation. It can help you talk with the software in different ways and interact with it differently. It helps a lot with data generation and logging, too. Another hint: Never let the automation take control of the testing. It is there to help you, not to drive the effort.

• With the knowledge and tool I had so far, I did a lot of trials in order to find where Nemo is at each iteration. This could give me a hint if he moved in any regular or consistent manner.
  • So I made a big file of winning moves (a lot of trial and error went in it).
• .
• It was now time to model the results:
  • First attempts at modeling the results:
  • I got this: Nemo’s position: 1, 3, 5, 3, 2, 7, 3, 4, 7, 4, 4, 1, 5, 5, 2, 5, 6…
    • Nemo was certainly jumping around without any clear regularity, even when I did it twice the times there was no repetition.
• On the Second modeling attempt, I tried to see if a more visual model could help:

• 1	2	3	4	5	6	7
  X
  					X
  				X
  		X
  					X
  						X

• Even in a table with many more results, it made no sense.

Testing Hint: When you look at a problem or thing, different models will give you different view and different details. Multiple viewpoints help — keep changing the viewpoint until you learn enough.

One interesting thing that happened by this time, is that in my manual trials, I was finding Nemo pretty easily.
Intuitively, I was seeing a pattern on the game, a pattern that was not conscious or explicit. It was as if my mind had already learned the rules without noticing them.
It reminded me of what Douglas Adams wrote in Dirk Gently’s Holistic Detective Agency:
“A ball flying through the air is responding to the force and direction with which it was thrown, the action of gravity, the friction of the air which it must expend its energy on overcoming, the turbulence of the air around its surface, and the rate and direction of the ball's spin.
And yet, someone who might have difficulty consciously trying to work out what 3 x 4 x 5 comes to would have no trouble in doing differential calculus and a whole host of related calculations so astoundingly fast that they can actually catch a flying ball.
People who call this instinct are merely giving the phenomenon a name, not explaining anything.”

• I still thought that looking at the other fishes will bring the breakthrough needed.
  1. In order to learn the movement of the fishes, I did a table of their placement in relation to Nemo at every run.
     1. i. This is what I got:

     2. Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
        Nemo	Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth
        Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
        Nemo	Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth
        Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
        Nemo	Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth
        Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger

     3. See anything interesting? The rows repeat themselves alternatingly! So the fishes were not really moving around, they were following Nemo!
        1. My take is that Pradeep has two different arrays: One for Odd rows, the other for Even rows.
           1. This may not be true, but it doesn’t matter for us. When testing, we not always know what exactly the programmer wrote in the code, but we infer a mental model. If it suits the needs, it is a good model even when not the real thing.
              1. Many times I think these fake mental models are even better than the real thing. It’s the best way for them to act as an intuitive Oracle when analyzing an application.

Testing hint: When you learn and infer about software, you make assumptions and a mental model of what the software does inside itself. This model does not need to be an exact depiction of the code underneath the app. As long as it answers your questions, and you change this model constantly as you learn, it is good.

• From this part on, I started to analyze Even rows and Odd rows separately.
  • Bingo! Seeing the pattern was easy now.
  • Odd rows follow this cyclic sequence: +4, +4, +1
  • Even rows follow this cyclic sequence: +0, +4, +4
• .
• But what with the differences when using different “Minimum Attempts“?
  • A quick look at the tables I had gathered showed that they follow the same rules, but start at different places.
• .
• At the end of this trials, these are the set of rules that allow me to know where’s Nemo at any given time:
• a. Where does Nemo start?
  1. “Minimum Attempts” of 10, 13, 16, 19…

  2. 	Place 1	Place 2	Place 3	Place 4	Place 5	Place 6	Place 7
     First Line	Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
     Second Line	Stinger	SharkTooth	Nemo	Phamplet	Gill	Bubbles	Flow

  1. “Minimum Attempts” of 11, 14, 17, 20…

  2. 	Place 1	Place 2	Place 3	Place 4	Place 5	Place 6	Place 7
     First Line	Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
     Second Line	Gill	Bubbles	Flow	Stinger	SharkTooth	Nemo	Phamplet

  3. “Minimum Attempts” of 12, 15, 18, 21….

  4. 	Place 1	Place 2	Place 3	Place 4	Place 5	Place 6	Place 7
     First Line	Flow	Phamplet	Stinger	Nemo	Gill	Bubbles	SharkTooth
     Second Line	Stinger	SharkTooth	Nemo	Phamplet	Gill	Bubbles	Flow

• b. Where does Nemo go?
  1. i. “Minimum Attempts” of 10, 13, 16, 19… Follow this cyclic rule, starting at step 1.
  2. ii. “Minimum Attempts” of 11, 14, 17, 20… Follow this cyclic rule, starting at step 2.
  3. iii. “Minimum Attempts” of 12, 15, 18, 21… Follow this cyclic rule, starting at step 3.
  4. Odd Rows: +4 | +4 | +1
  5. Even Rows: +0 | +4 | +4
• .
• How do I know it is right?
  • Proof!  I built another automation aid: An Excel spreadsheet that can tell you the position of nemo in all trials given an initial “Minimum Attempt“. It works!
  • You can download the spreadsheet to try it here. Choose any number of initial attempts, it will tell you all Nemo’s position accurately.
  • Is this the real logic behind Pradeep’s challenge? It may not be. It is probable that pradeep did something else, much simpler, and that only translates into this. However, this answers the mission of predicting Nemo’s moves.
    • I am sure that analyzing the findings above one can discover what was really that Pradeep did. I’ll just ask Pradeep on a mail for a suggestion. Testing hint: You can always reach out to the developer and ask him about his software and/or code!

Testing hint: Again, your mental model doesn’t always needs to be exact. As long as it works and answers the questions you have. Keep in mind that this mental model should be flexible and evolve with your knowledge.

• Conclusion sent to the ‘roulette’ stakeholders:
  Until a higher amount of randomness is added to the game, it is too risky to release the game as is now. A malicious player can find the logic of the game after some hours of work, and a pair of smart malicious players in even less time.

Thanks Pradeep, for the nice challenge.
It was a very nice experiment, and I learnt a lot. The best part, for me, was organizing my thoughts in this report, and I hope you like it too.

Demetri Martin

One best friend of mine introduced me to Mitch Hedberg and Demetri Martin, great one-liner comedians. They are/were two funny men!! Three, actually, if you count my friend which is also funny. 

After hearing the disks for over a year, not only the jokes aren’t any less funny, but I’ve started to find subliminal testing messages in them .
I’m writing down these “insights” because I find value in them. And even if they fail to teach you something… Hey! At least the jokes are pretty funny!
So here go some favorite quotes, and their parallel in testing: 

. 

1) “A drunk driver is very dangerous. So is a drunk backseat driver, if he’s persuasive!
'Dude, make a left.'
'But those are trees…!'
'Trust me...'“ 

We are testers. As such, most of the time we aren’t driving the company: Essential operational decisions are made by someone else (project manager, product owner, CEO, whatever you call him).
But if we are not the drivers, our position is certainly close to this of a backseat driver: we have our maps, good knowledge of the area and our experience; and we give advice that is used to take real decisions. So don’t be a drunken backseat driver, it’s dangerous! If you are inebriated by an obsessive desire to fix a bug, or by a personal quest against/for colleague, by a blind belief in a set of metrics or a unfounded trust in a fictitious “Best Practice”, you’ll be taking your company directly into the trees.
And, as with any drunk person… Whenever your judgment isn’t objective… At least recognize/admit that you may be drunk! Will make it easier for everyone else. 

. 

2) “I used to play sports... Then I realized you can buy trophies. Now I'm good at everything!“ 

A plaque, a crown, a card, a trophy… It is common to find people who treasure “victory symbols”, I know I do. They prove you’ve mastered a skill.
Trophies are nice and great, but only if they are accompanied by skills and real world practice. Trophies that can be attained without these traits are just empty cups*. 

One should beware when dealing with “achievement symbols”… At times, acquiring the symbol does not mean acquiring the achievement or skill too!
For example, just as buying the trophy doesn’t make Demetri good at tennis, getting a testing certification won’t make a tester good at testing… The certifications syllabi try to teach only specific lexicon and terms definition, but not the real practice of testing, because they don’t/can’t cover the interactions between persons and players.
For some people, the joke could be read as “I used to practice and study... Then I realized you can pass a certification. Now I can prove mastership - without the effort of gaining it!” Be sure to be from the ones who keep learning and carrying the skill. 

. 

3) “I am afraid of sharks, but only in a water situation. If I saw a shark on the street, I'd be like 'What? F#*k you!'
It's funny, that's like the opposite of how I am with lions!“ 

What’s the Best Practice for dealing with sharks? “Escaping” or “screaming”, right?
What if the shark’s on the sand? Or dead? Not hungry? Not dangerous? You have an anti-bite clothe? Anti-shark cage? Suddenly the ‘Best Practice’ looks more like a ‘so-so advice’, uh? 

The business of testing isn’t different. By looking at our surroundings and context, we can learn important details about our problems – and only then we can try to find an action path that solves it.
Without the context analysis, we may come with solutions that don’t address the real problem or does it badly. One size only seldom fits all. 

. 

4) “If I have to move up in a building, I choose the elevator over the escalator. Because one time I was riding the escalator and I tripped. I fell down the stairs... for an hour and a half.“

Automation is all nice and fine… Until you trip on it! Then it takes long time to recover from the fall (you could be long done by this time) and the fix to the system can be so crooked that it lets you prone to further tripping.
The elevator is a much more robust automation (not only it goes up and down, but it is much safer to trip in it ).
Anyway, who said you have to automate? An equally good way to go up and down is the stairs! You don’t need to wait for them to come, they are as quick as you need (or can make) them to be, you can stop to look around or change direction in the middle of the way, they don’t limit weight or number of users and they are never out-of-order.
Slipping, well, can hurt a bit . Just remember that manually doing things gives you many opportunities a machine cannot provide. 

. 

5) “I saw a sign on this door; it said, 'Exit Only'. So, I entered it and went up to the guy working there, and I was like, 'I have some good news. You have severely underestimated this door over here by, like, 100%, man!'“

Labeling is a natural instinct: The Bible depicts Adam naming animals as his first action in Eden, and Aristotle has been classifying everything since the 300 BC.
But the labels we use can limit ourselves or our tools.
“I'm a tester, I can't code“.
“I'm an engineer, I can't sell“.
“I'm a newbie, I can't help“.
Good news! You’ve underestimated yourself by at least 100%! 

. 

6) “I want to make a revolving door that says 'Pull' on it, just see how obedient people are.“ 

Meeting a Test Case that says “1) Reach to revolving door”, “2) Pull the door to open”. What do you do?
Maybe nothing is wrong. Maybe there are good reasons to make people walk backwards at the entrance.
Or maybe there isn’t, and then, who’s decides? The test case? The design? The developer? The tester?
What are your ways to discover about the door product and its users before deciding on obeying or not?

. 

7) “I was walking down the street, and this guy waved to me. Then he came up to me and said, 'I'm sorry, I thought you were someone else.' I said, 'I am.'“ 

Wow, that’s a funny joke!
Beginning a task with a very clear objective is usually a good thing – but failing to identify or even acknowledge alternative outcomes may not be.
A good friend was working in an academic project, trying to prove a theory about the nature of lasers. However, while progressing in the research, the calculations made clear that the theory wasn’t right. He was desolated – all this work and nothing to publish! He failed, momentarily, to notice that demonstrating a theory wrong has as much scientific value as proving it right, and the publication will advance science just as well.
Testing present us with many similar situations. We can start a test chasing a specific bug, and miss other bugs that may appear. Or we can keep our mind open for different outcomes, and grow your tests as we go from these outcomes. A different bug is hiding there, believe it . 

. 

8 ) “My plumbing is all screwed up. Because it turns out, I do not own a garbage disposal.“ 

How many times we rely on a garbage disposal when we don’t have one?
- Don’t worry, release it that way, we have a fantastic relationship with this customer. What if you don’t?
- It’s fine to release that way, the customer has a great quality assurance team and they’ll catch it if there’s a problem. What if they don’t?
- No problem, we can work Sundays — our testing department understands our needs and won’t care. What if they do care?
- This looks like a very nasty bug. But customers will always have the underlying framework, so it is fine. Will they always? 

. 

9) “I'm in a weird position, because I like rainbows, but I'm not gay. So whenever I go out wearing a rainbow shirt, I have to put 'Not gay'. But I'm not against gays, so under that I'll have to put '... but supportive'. It's weird how one group of people took refracted light. That's very greedy, gays.“ 

Once in a while a group of people will come, wrap common sense ideas in a new named package, and claim ownership, which is as reasonable as taking sole ownership of refracted light.
I could parallel this one to many things or groups, but I chose one where this happens with a lot of enthusiasm… The Agile movement.
Suddenly, you can’t focus on customer and organize a software project in a change-adapting way, without being called Agile. That group just took no-nonsense practices that had been used for decades and re-branded them as if they had invented it!
Don’t get me wrong – it is my opinion that what Agile preaches are often-good-practices-for-many-contexts and there are testimonies of companies, projects and souls being saved from a bureaucracy and poor quality hell by adopting Agile as a new approach. But this does not mean that anyone who is delivering quality in a “release early release often” way is Agile.
For example, even being a total newbie and never part of an Agile team until now, my personal testing approach focus on happy customers, working software, individual interactions and change — without me needing to be an Agilist.
More experienced great people had practiced all that, and test-first, and XP for decades. 

More than that, not only the practices’ refracted light was taken, but even the word “agile” itself was expropriated – see James Bach’s note at his ‘Who stole Agile’ post. 

——
Hope you enjoyed the article, or at least Demetri’s jokes.
If you have more insights and funny stuff to share, don’t hesitate to contact me – I promise to laugh.

Request: Are you a lawyer? Please send me corrections

Matters related to law, and all the discussions around it, interest me much — especially when related to Software.
This made me read about the subject and keep contact with the legal representatives within the company I work for. This also motivated me to learn and lecture about the legal guidelines in software development adopted by our company, and to lecture about legal matters on software in general at the last Israeli SIGiST conference.
Most important than all, this made people share with me a lot of comments, questions and stories pertaining to the law.

For example, one colleague brought to my attention a case in which he and some friends had bought a PlayStation 3 in the local Office Depot website for 220NIS – when the normal price is almost tenfold! They believed it was some special sale promotion, but at the end Office Depot announced it as a typing mistake and cancelled the sale after it had been acknowledged (sale confirmation by email).
(There is an article about the episode in Hebrew here if you want to see it).

What is my opinion on the legal aspects of the story? I was asked.
I don’t have one, as I am not a legal professional, I answered. I am, though, a Testing professional, and here is the tester rambling I sent by mail commenting the occurrence:

——
In fact, electronic transactions are special and different that normal stores. They can be built without any human interaction whatsoever, and the lack of human monitoring can bring new and unexpected problems.
Note that I can’t have an official Legal opinion, as I am not a lawyer. I’ll just write what I learn.
A usual understanding in contract law is that once both parts show agreement to the deal, then the deal is closed and valid (I think a lawyer would explain this by stating that there is a mutual offer and acceptance of goods or services). Such agreement is shown, for example, by accepting the payment money.
In such a case as this story, the electronic sale often includes a debit in the buyer’s Credit Card (that turns into a credit at the cancellation of sale). If a service provider accepts payment for a service, he is in obligation to provide the service at the agreed terms. If he didn’t agreed to the terms, he would not have taken the payment. So in our case the fact that a payment transaction was completed supposedly forces the two parts to honor it (the store should not cancel the delivery just as you should not cancel the money debit).

However, apart from the amateur philosophy, we can find a bit more of info in the net about this sort of things…
a) Not only the payment debit can be viewed as a contract acceptance, but also the confirmation mail you receive when buying online can be seen as such, depending on what it writes.
b) The Terms of Service of the website can be used for or against the store, depending on their presence and the judge interpretation of them.

• For example:
• Buy.Com (huge company, 290 million revenue) announced at $164 a $588 monitor. They tried to cancel the sales, and lost more than $60000 in a lawsuit (CNET link).
• On the other hand, Amazon.Com (huger, 19 billion revenue) announced at $11 a $449 handheld (CNET link). They cancelled the sales successfully, claiming that the Terms of Use state that there is no binding contract until the confirmation of shipment arrives.
• Buy.Com had no such protection in their terms, and after the lawsuit they updated the terms to include “have the right to refuse or cancel any such orders whether or not the order has been confirmed and your credit card charged”.
• If it will always help? Not sure. In this link a representative of the FTC says that it will depend greatly on the state the lawsuit is taking place.

In the US, there is a concept of “loss of bargain”, that could allow one to buy the PS3 at normal price and sue for the difference (which represents the bargain lost due to the store’s malpractice). Of course, if the judge identifies the buyer as having bought it to exploit a weakness or obvious typo, the attempt can backfire…

Now that we learned some principles of the domain… We go for the most important…
As I said, I have no legal opinion, but I do have a tester opinion.
First, this shows (again) that there is no final answer in the law yet for new technologies and new problems.

And I now know that for assessing risk on an e-store product, one has to answer these questions too:

• Does the system have any protection against mistakes
• This can be done by issuing an alert if there is a big discrepancy after:
• comparing the product to similar ones
• or comparing to the previous price
• or comparing to the price in the normal (brick) store
• Does the system require a manual human confirmation before sending the payment or confirmation mail?
• This can be bad for a large operation, but for small-medium business, can be a good way of catching mistakes
• Does the system alerts the administrator if there is a sudden surge in the purchases of a newly added product?
• Do the Terms and Conditions protect the company in case of mistakes?
• Are the Terms and Conditions clear to the end user? At least in a visible place?
• Is the user obliged to mark an acceptance to the Terms during log-on, or it is just a link in the page footer?

These are some of the questions I would ask when faced with such a product. There are more, certainly — please share your own ones in the comments or by email

——
One last comment…
Sometimes the typos make the prices higher than the norm… See this picture, a real example from Amazon selling this $400 card for a whopping $5 million!! That’s a lot of money for a graphics card…

Hummm… would they cancel this sale as a mistake too?

– Hi, Nancy? (Name changed. And she wasn’t the testing manager, she was from the HR.)
– Yes.
– Hi, my name is Shmuel and I saw you’re offering a software testing position, the “QA engineer” opening…
– We have it, right.
– I saw that one of the requirements is to know the STD and STP methodologies, and I was wondering what are STD and STP.
– Well, if you have no experience on that, then the position is not relevant to you.
– No, no, I guess the experience is fine; it is just that the announcement made me curious. These acronyms can mean many things, you know? So I want to know how your company understands them.
– They mean specification documents.
– Does STD stand for Software Test Documentation? What does the STP mean?
– No, STD is for Software Test Design, and the P… Well, both mean specification documents.
– Why design? Is that the specification of the tests to run, of the testing system, or of the product?
– Why don’t you just look up the meanings on Google?
– I tried right before calling you. The acronyms are all over the web, but they have different meanings too (*).
– Listen, these are very important topics, if you don’t know them the position is not relevant. They’re very important, and very well known.
– I see. Thanks you Nancy!
– Thanks, bye bye.

Very important and well known. Well, I guess I’ll put them in my list of things to learn… As soon as I decide which meaning I want them to have…

Let’s see. I guess two of the best “methods” I use in testing are:

STD – Software Test Drawing: Scribbling in a whiteboard (pen+paper are good too, but my experience shows whiteboards are much better). This is especially useful when you’re working with someone else. With some squares and circles explanations become simple, concepts become clear, people can point when they talk, you can easily delete mistakes, and the best part is that usually there is always a joke at the end of the process. Whiteboard scribbling is a major testing tool for me.

STP – Software Test Peer (or Pal): If you are like me, your brain is constantly storming . Nothing beats thinking out loud or talking to peers and fellow testers/developers about the challenges you face — more often than not you’ll learn about a direction you hadn’t thought of or about a tool that can help. Works fantastically with STD .

I may start an “Alphabet of Really Useful Software Testing Things” — please contribute on the comments.

(*) In fact Google says D and P apparently can mean Design, Documentation, Development, Performance, Process, Plan, Programming…Pick your own definition. Most companies seem to accept STP as Software Test Plan, though.I still like that anyone can pick the definition that best suits his needs or business, though. So I’ll stick to mine .

Note to my employer: No, I am not looking for a job elsewhere yet .

Today morning I was greeted by an Email from SQE: The subject read “Are you certifiable?“.
My first reaction was to discuss the term. If I am certifiable? I? In my mind, I was arguing whether a person can be considered certifiable or maybe the topic of certification is the one certifiable.
As in “Software Testing is (or not) a certifiable topic” against “Johnny is a certifiable (or not) software tester”.

I was puzzled over the confusing choice of words:

A lot of good people (Rex Black, in a moderated way Lee Copeland, Michael Stahl (link to an unrelated article)…) support certification, and a huge lot of good people think that either the software testing arena is immature to certify, or the certifications out there are inadequate, or both (James Bach, Michael Bolton, James Whitaker, Cem Kaner…). For transparency’s sake (so you get the context of this rambling), while I admire and have learned a lot from the first ones, the last ones are personal heroes (ok, Michael Stahl is a hero+guru too ).

So we did not even come to an agreement if the craft itself is certifiable, and you are talking about certifiable persons? How does that work?

.

.
At this moment I stopped musing and went to the dictionary:

Ok, my guess on the meaning of the work was correct: “a certifiable fact“, “capable of being guaranteed“.
So was the other meaning, about certifiable persons: “endorsed authoritatively as having met certain requirements” (in our case, an exam with multiple choice memory questions ).
But wow. That one in the middle is a curious meaning! “fit to be certified as insane (and treated accordingly)”
Certifiable == Insane. And while I don’t see a problem with insane people being ISTQB certified and vice-versa, it made it even a weirder word choice.

I had to search more in the internet to see that this “are you certifiable” idiom is a bit common in english. Microsoft even has a game with this name here.

So it was a joke! Ah, all a joke, but they kept me thinking.
I guess that most of the non-native speakers of the SQE newsletters didn’t get the joke. And they were this close to being offended (depends on the culture one comes from).
SQE (and all), it was good as an excuse for me to post the links for certification articles, but pay attention to your publications’ audience! It takes a lot of time to get your jokes overseas .

I enjoyed the story very much. It is very exciting to see the passion he had (has) for his software and how he was committed to it. Plus, Ron is a great story teller.
The graphing calculator had all the ingredients of a cool app. It scratched a developer’s personal itch, and is a great example of NeoVictorian computing: built for people, built by people, crafted in workshop, inspired.
Actually, if we’re commenting on NeoVictorianism, Ron was one that really “woke up one day to find himself living in the software factory“. The night got very cold, they said the factory is going to close and he should move somewhere else. The cool part? He kept doing his individual craftsmanship inside the corporation. Secretly.

Now to our matter, testing. He’s got some very good comments on Software Testing (which he calls in the story as QA. Not his fault, but ours). So nice comments, that I’ll just transcript:
At (~20:24), “Our biggest problem was QA“. “Fundamentally, you can’t do your own QA — it’s a question of seeing your own blind spots“, and “it is expensive to do it well“.
At a moment in the story, two testing engineers came to offer help, as in their official work “they were really bored because they spend lots of times developing automatic test tools and now they’re at the point where they push the button and the test starts in the morning and they come back in the evening to see which API’s break“.
These testers were successful because they badly wanted to explore the new software, and also because they had good mathematical experience/background — closely fit skilled testers!

Ron later speaks about usability (~35:57).
He describes the process of the usability studies they conducted after they’ve been programming with users in mind. Ron (which understands well the importance of usability: “In a classroom, any time spent frustrated with the computer is time taken away from teaching“) says that even after all the attention to usability they had, they were surprised by the results of the study: “Sitting behind a two-way mirror, watching first-time users struggle with our software, reminded me that programmers are the least qualified people to design software for novices. Humbled after five days of this, Greg and I went back and painstakingly added feedback to the software, as if we were standing next to users, explaining it ourselves.” In the movie, he adds that they were “pounding on the glass” and that “all engineers should be forced to go through this, watching real users actually trying to use their software“.

There’s another interesting comment at the very beginning (~02:30). Ron comments on how straightforward typewriting work was on the firsts Macs, and how a newbie could go from zero to full speed in less than 5 minutes on MacWrite.
This is very curious: Even today, the computer is more useful as a typewriter than anything else. The uses had varied much, and nowadays professionals use the computer for every nitty-gritty important detail, but still, the first thing that comes to mind to most people when they think “computer” is “writing”. From journals to books to presentations, my perception is that the only thing people can do (on a computer) with no complex instructions is publishing. Want to publish? Open this app and start typing. But you want to render a 3D object? Search for a court lawsuit? Build a repository of data? Out of luck, will take years to learn it well.
Our hardware has evolved, and the purposes we use the computer for have changed too. But the form has stayed the same as 30 years ago. And (not) surprisingly, the interfaces of our apps still resemble the interface of that first MacWrite. Time to change how we interact with computers? I hope so. Science Daily had long ago an article that said “While much about the computer has changed over the last three decades–greater power, faster speeds, more memory… – what has not changed is the user interface.” They offered a gesture interface proposition, and other technologies like multitouch or brain-controlled interfaces are being researched. From my part, I’m ready to test the new paradigm as soon as it’s ready .

This is the full video, this is the link for the graphing calculator storry in text form (10 min instead of 55 ), and go here for a demo of the product at Apple’s launch.

Enjoy, and please let know your testing thoughts in the comments!

Writing the Fitnesse posts turned to be harder than I thought.
I do have a bit of tests ready for the triangle case, but not enough text to make an interesting post. As I’m not using Fitnesse in my day-to-day work, it makes it harder to bring cool insights or to explore on the framework.

But I just discovered someone who not only uses Fitnesse at work, but also writes articles about Fitnesse that give the Fitnese feeling:

Stuart writes at http://working-thought.blogspot.com/.
It seems as if his company has been trying Fitnesse for a while, and they’re over the initial pains. Now they’re on more advanced pains , but with apparently good results.

If you ‘ve already seen the Two Minute Example at the Fitnesse page, then you know what Fitnesse if used for.
Then start reading from “Fitnesse for Dummies” at Stuart’s blog (and laugh at the ilarious GenericFixture.jar error page). There are further posts later, like the very insightful love/hate relationship with Fitnesse.

As for me, the post I liked most was the one about test automations, that has nothing to do with Fitnesse, but explains well many of the web-app testing tasks.

I wish luck to Stuart, and who knows? Maybe I’ll get back at posting about Fitnesse somewhen.

For testers, thinking like developers is evil. If you think like a programmer, you’ll start excusing the software and will forgive the system’s bugs.

I am reading the very cool book “Dreaming in Code” by Scott Rosenberg, and I just understood a little bit more on why’s so bad sharing the developers mindset.

Scott starts its book with Chapter 0, a common practice in computer literature. When enumerating the reasons, he explains that “Programmers count from zero, not from one. (…) Why do programmers count from zero? Because computers count from zero!” and that developers are taught to think like computers.

This is a very interesting and basic concept.
Dr. Parnas in his “Strategic Defense System” paper (a real favorite of mine ) has a whole chapter called Why conventional software development does not produce reliable programs, in which he clarifies this problem — developers thinking (or trying to think) like a computer:

As we continue in our attempt to “think like a computer”, the amount we have to remember grows and grows. The simple rules defining how we got to certain points in a program become more complex as we branch there from other points. (…) Because one needs to remember so much (…), new problems are created when old problems are corrected.

The beauty in all this is that Testers should not exercise the programming rationale (when analyzing a software) because they’ll be mimicking the rationale of the very same software they are testing. They are limiting themselves to what the program knows about itself, instead of expanding their logic to people-centered matters — i.e. how the software will be used and what it is really meant to do.

Back to “Dreaming in Code”, Scott has an almost poetical way of writing the concepts developed by Dr. Parnas.
Because computers think from zero, “programming include little offsets, translations of ‘+1′ or ‘-1′”, he says. So:

There’s a space between (…) the way the machine counts and thinks and the way we count and think. When you search for explanations for software’s bugs and delay and stubborn resistance to human desires, that space is where you’ll find them.

Poetry.

I am fond of quoting.
Not sure why, but I like to quote. I guess it gives some legitimating to what I am saying.

So, the quote collection is available at this address: http://testing.gershon.info/quote-collection/. It will grow slowly, please check it regularly.

Uh, you didn’t notice? Well neither did I until I read it all over the internet.
See the post on SlashDot for scoop, and see its comments for some good laughs.

There are explanations all around about how this bug came to appear in Excel.
The best one is probably Joel’s one (you may remember Joel from this post).

Go and read it. Joel is very deep in there and explains why the binary calculations can be confusing, and how the binary notation standards may or not help you developing better software.

I say “go and read it” because I’ll not repeat his explanation here
I will, however, quote the best parts from the questions at the end — very insightfull!

First insight is a matter for pondering about bugs, small bugs and the perils of fixing such:

Q: Isn’t this really, really bad?
A: IMHO, no, the chance that you would see this in real life calculations is microscopic. Better worry about getting hit by a meterorite. Microsoft, of course, will be forced to tell everyone “accuracy is extremely important to us” and I’m sure they’ll have a fix in a matter of days, and they’ll be subjected to all kinds of well-deserved ridicule, but since I don’t work there I’m free to tell you that the chance of this bug actually mattering to you as an individual is breathtakingly small.
(…)
And let’s face it — do you really want the bright sparks who work there now, and manage to break lots of perfectly good working code — rewriting the core calculating engine in Excel? Better keep them busy adding and removing dancing paper clips all day long.

Second insight, on the perils of automated tests (an automated test sees only what you ask it to see, hence it has blind spots and will not report for side-effects!).
(Of course, as Joel says, it is doubtfull that manual tests would ever exercise rare/random calculations):

Q: Shouldn’t they be testing for these kinds of things?
A: I’ll bet that most of the numeric testing done on the Excel team is done automatically with VBA code. Cells containing this value display as 100,000, but from VBA, they’re going to look like 65,535 (since the number would be passed into the Basic runtime in binary, before the display formatting.) I’m sure there’s plenty of code to test display formatting, but with a bug like this that only happens on 12 out of 18446744073709551616 possible floating point binary numbers, it’s unlikely that any set of black-box tests would cover this case.

In 1979 (yes, I am that young ), Glenford J. Myers wrote what was to be the first classic book on Software Testing.
Called “The Art of Software Testing“, this book is very old, and written with old software in mind – most technologies and methodologies existing today were not dreamed of back then.
Well, it turns out the principles of software – from writing it to testing it - are still pretty much the same. Software is still written on sort of punchcards, the punchcards just got more complex and flexible… The book content is surprisingly relevant for today’s testers – so much, that it was updated only once in almost 30 years.
If you haven’t, read the book. Unfortunately, I have read it only partially meanwhile.

Maybe the most known contribution of Mr. Glenford in this book is the testing self-assessment challenge in the introduction.

It goes like this:

A Software is given, which receives three inputs (numbers) that define the size of a triangle’s sides. The SW can, then, categorize the triangle in one of these categories: Equilateral, Scalene, Isosceles or Invalid.
How many test cases can you build for testing this triangle?
The more heterogeneous the cases the better, and: The more effective, better yet.

Myers brings then an assessment criteria. If the test cases you came up fit into enough categories such as the ones below, you are in a good way:

• Test cases that represents a valid scalene/isosceles/equilateral triangle.
• Test cases that represent valid isosceles triangles such that there are all three permutations of two equal sides.
• Test cases in which one side has a zero value, or negative value.
• Test cases that represent invalid triangles.
• … 

The example is famous, because it grasps clearly the distance between coding and testing. Writing such a program may take about 30 minutes of work to a programmer (I did this first version in 20 minutes), but testing the program can be a never-ending task.

We will use this problem as the example to do all the acceptance tests from FitNesse.

We’ll have this DLL – based on the C# code above – and we will connect this to our FitNesse server. We will then start checking inputs and outputs. All this in Part 2 of the FitNesse Series!

1. Google Search – “software testing triangle equilateral”
   http://www.google.com/search?q=%22software+testing%22+triangle+equilateral
   You’ll find most of what’s online and cool here.
2. Example with working application (!) for you to test:
   http://www.testobsessed.com/2007/03/21/testing-triangles-a-classic-exercise-updated-for-the-web/
   Quality Tree Software’s Elisabeth brought the triangle to the   attention of all again when she posted it on her website.
   The working example by Elisabeth: http://www.testobsessed.com/exercises/triangle.html
3. Challenge Definition by Alex Samurin
   http://www.geocities.com/xtremetesting/TriangleTest.html
   Alex has a short explanation of the original problem as it was presented by Myers.
4. Malik’s Code Inspections
   http://codeinspections.blogspot.com/
   Malik summarizes in two paragraphs the two most interesting points on the abovementioned application by Elisabeth: Floating point calculations and sorting algorithms.
5. The exercise, Bollywood style
   http://venkatreddyc.wordpress.com/2007/03/24/taking-on-testing-triangles-a-classic-excercise/
   Venkat has written some insights (in a dramatic voice) on the triangle javascript app. I recommend this one, because the pace and steps are very didactic.
6. Last but not least :
   My post on it: A Triangle mystery (not the Bermudas’). I turned this post into a page/article, ’cause it is a useful subject.
   Pay attention, I am not trying to really test the triangle application. This is new stuff – we are using the triangles to test the FitNesse framework! Go for it!

I’ll (as soon as I get to do a lot of things I want to get done) post some impressions and examples on FitNesse, the cool tool for cool testers.

What I plan to do:

1. Install and set a FitNesse setup;
2. Make a simple standard easy program – the System Under Test!
3. Start testing and making the program better with FitNesse.
4. And all this, with screenshot goodness!

When will that be?

• Ouch. It may take a long time. Let’s talk about this again on July…

Meanwhile: “FitNesse is designed to support acceptance testing rather than unit testing in that it facilitates detailed readable description of system function”. More information can be read in FitNesse’s website.