This was an interesting exercise.
What took my interest in this one?
- First of all, I liked Pradeep’s post about learning to code.
Learning to code is an important skill for a tester. I can relate to it because I am trying to learn PowerShell. I program in some languages (C#, C++, PHP, VbScript…) and have a programming background (after my Computer Systems Engineer degree, the ‘natural’ path was to code, and I worked as a programmer for many years before finding Software Testing), but PowerShell has different approach and paradigms and learning it will definitely be fun.
- Second, I had an impromptu vacation day, and I thought it would be educational to use it for a peer testing puzzle. I thought I could win this one quickly (I was mistaken, it took me long).

So I downloaded Pradeep’s application, and got to work!
Rather than just trying to solve the puzzle, I looked at it as if the mission was:
“This is commercial 'roulette' style game. Stakeholders want to know it the game logic can be broken/learnt, which could mean a substantial loss of money when people start winning every time.”

Testing Hint: Cem Kaner says “Testing is an empirical, technical investigation of a product, done on behalf of stakeholders, with the intention of revealing quality-related information of the kind that they seek“. So it is important to set straight what is the information to seek. Which kind of bugs to look for?

In this blog I describe my attemp to answer this question in an exploratory way.

Now, Stop the press!
You may want to go to Pradeeps post, download the file, and try it for yourself before reading my solution. This post will be waiting right here when you’re done.

…

Here is the report of what I did, the line of thinking during my experiment, and the conclusion:

• 1. First, I ordered my environment to allow efficient and organized work:
  1. Increased the Screen Buffer Size of the Command line I was using to 500 (it turned out that I would do well on less, even half, than that (see point 2.b). But it didn’t hurt).
  2. Renamed the executable to be shorter and without number version: ren fnemo_1.7.6.exe fnemo.exe
  3. Changed the prompt to something shorter, cleaner, and that gave me context about this task against the other Cmd lines windows open: prompt $Cfnemo$F$S$G

Testing Hint: Organizing your testing environment before you start will likely help you during your tests. After you’re in the heat of fight, it will be harder to stop and get organized.

• I tried some basic executions of the program, to grasp the feeling of what it does and how hard it is to find Nemo. This also taught me what are the messages returned by the application when Nemo is found, and when Nemo isn’t found. They’re “Found Nemo this time!, Nemo Gill Bubbles SharkTooth Flow Phamplet Stinger” and “Ah! bad luck, didnt find Nemo this time!” respectively.
  1. I also learnt that Pradeep used Perl and Perl2Exe to do the app.
  2. Additionally, the application clears the screen at every execution, so the big command line buffer is not so helpful.

Testing Hint: Planning or scheduling your testing before you experiment a bit with the software can be done. But you risk being completely wrong in your assumptions about the app. In our case, there were no big surprises, but there could be.

• Sarmila commented on the Challenge page about disassembling the executable. Although by learning all the assembly one can learn about the rules that move the fishes around, it would be extremely difficult and overkill. But this made me think on how Perl2Exe works… Maybe it just wraps the Perl interpreter and the Perl script together? If so, what if the script is stored internally in clear? I tried to open the app in an Hex editor, but no, the Perl script isn’t in clear text there, it is obfuscated.
  • I also tried the useful Strings tool by SysInternals. My intention was to see if, maybe, the array of fishes could be seen at the app code, in order to learn the initial order of fishes. It couldn’t.
  • That ended my cheating session , from here on I used only a black box functional approach.
    • Well, it is not really cheating because there were no rules against that. If you do a full reverse engineer of the code, the answer is legit too. But I preferred the straightforward exploratory approach: learning from the software functionality as I use it, and use this learning to redirect my next steps.
• .
• By now, I knew that Nemo changed places between plays.
  1. Nemo also change places between invalid parameters (which may or not be a bug).
  2. Moreover, the other fishes change places too, even their placement related to Nemos’ placement changes.
  3. So I tried to see if Nemo will return to the same place in any consistent way or number of times. Running that is easy, you just enter “3″, “3″, “3″… and count where Nemo was found and where he wasn’t.
     • I discovered there is regularity (for “3″, its every 2, 3, 17, 5, 10, 5…), but the regularity was irregular enough, and also changed for different positions. This was likely a consequence of the real logic, rather than the logic itself. This correlates to the movement of fishes, but does not cause it.

Testing Hint: Take note of your steps while testing. Using the Session Tester tool is a great idea. For this session, I used MS OneNote because it supports rich text, and I could use tables.
Is that like saying that rich texts helps rich tests? .

• Another detail that helped me in the game was knowing the developer and the purpose of the challenge. This puts a lot of content in the context. Pradeep is a rather playful and would put some tricks into it.
  1. First I tried to see if Nemo could sometimes be in any placement bigger than 7 (there are 7 fishes, places 1 to 7, but there were no rules as to where the fishes were limited to be).
  2. I tried in a toss of up to 220 attempts, and Nemo wasn’t in place 8 even once. So I let this idea on StandBy for now.
  3. Then I tried… what if the “Minimum attempts” input at the beginning of the app affected where Nemo appears?
     • 10 is the default for the “Minimum Attempts“. Nemo is at place 1 at the beginning there.
     • 11 attempts chosen: Nemo is at place 1 at the beginning too. Maybe it does not affect?
     • 12 attempts chosen: Nemo is at place 4! Bingo!
     • 13, 14, 15, 16 showed that the original places were cycling at ‘3′ intervals. That means that when trying to find Nemo on the default 10 attempts, it will be in the same placements as 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46, 49, 52…

Testing Hint: let the test and what you learn during the testing guide you. New information you find should set you in a new direction.

• You’re seeing that “220 attempts” above, and thinking if I really entered each number manually.
  1. By this time, I decided that I was in need of some automation to insert inputs to the application.
  2. The original app had no apparent automation capabilities, so I decided to use the internal input redirection of the Command line:
     • By using the “<” operator, I could redirect an input file “input.txt” to the app using “fnemo.exe < input.txt“
     • I extended that to “fnemo.exe < input.txt > output.txt“, which gives us with a complete dump of results that can be searched with Notepad.
  3. We had an easy automation now, and this can help our sapient testing. I was exploring, and using automation to help me try different things help me explore my possibilities and get a bigger picture of the situation when needed.

Testing Hint: Don’t be afraid of automation. It can help you talk with the software in different ways and interact with it differently. It helps a lot with data generation and logging, too. Another hint: Never let the automation take control of the testing. It is there to help you, not to drive the effort.

• With the knowledge and tool I had so far, I did a lot of trials in order to find where Nemo is at each iteration. This could give me a hint if he moved in any regular or consistent manner.
  • So I made a big file of winning moves (a lot of trial and error went in it).
• .
• It was now time to model the results:
  • First attempts at modeling the results:
  • I got this: Nemo’s position: 1, 3, 5, 3, 2, 7, 3, 4, 7, 4, 4, 1, 5, 5, 2, 5, 6…
    • Nemo was certainly jumping around without any clear regularity, even when I did it twice the times there was no repetition.
• On the Second modeling attempt, I tried to see if a more visual model could help:

• 1	2	3	4	5	6	7
  X
  					X
  				X
  		X
  					X
  						X

• Even in a table with many more results, it made no sense.

Testing Hint: When you look at a problem or thing, different models will give you different view and different details. Multiple viewpoints help — keep changing the viewpoint until you learn enough.

One interesting thing that happened by this time, is that in my manual trials, I was finding Nemo pretty easily.
Intuitively, I was seeing a pattern on the game, a pattern that was not conscious or explicit. It was as if my mind had already learned the rules without noticing them.
It reminded me of what Douglas Adams wrote in Dirk Gently’s Holistic Detective Agency:
“A ball flying through the air is responding to the force and direction with which it was thrown, the action of gravity, the friction of the air which it must expend its energy on overcoming, the turbulence of the air around its surface, and the rate and direction of the ball's spin.
And yet, someone who might have difficulty consciously trying to work out what 3 x 4 x 5 comes to would have no trouble in doing differential calculus and a whole host of related calculations so astoundingly fast that they can actually catch a flying ball.
People who call this instinct are merely giving the phenomenon a name, not explaining anything.”

• I still thought that looking at the other fishes will bring the breakthrough needed.
  1. In order to learn the movement of the fishes, I did a table of their placement in relation to Nemo at every run.
     1. i. This is what I got:

     2. Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
        Nemo	Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth
        Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
        Nemo	Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth
        Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
        Nemo	Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth
        Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger

     3. See anything interesting? The rows repeat themselves alternatingly! So the fishes were not really moving around, they were following Nemo!
        1. My take is that Pradeep has two different arrays: One for Odd rows, the other for Even rows.
           1. This may not be true, but it doesn’t matter for us. When testing, we not always know what exactly the programmer wrote in the code, but we infer a mental model. If it suits the needs, it is a good model even when not the real thing.
              1. Many times I think these fake mental models are even better than the real thing. It’s the best way for them to act as an intuitive Oracle when analyzing an application.

Testing hint: When you learn and infer about software, you make assumptions and a mental model of what the software does inside itself. This model does not need to be an exact depiction of the code underneath the app. As long as it answers your questions, and you change this model constantly as you learn, it is good.

• From this part on, I started to analyze Even rows and Odd rows separately.
  • Bingo! Seeing the pattern was easy now.
  • Odd rows follow this cyclic sequence: +4, +4, +1
  • Even rows follow this cyclic sequence: +0, +4, +4
• .
• But what with the differences when using different “Minimum Attempts“?
  • A quick look at the tables I had gathered showed that they follow the same rules, but start at different places.
• .
• At the end of this trials, these are the set of rules that allow me to know where’s Nemo at any given time:
• a. Where does Nemo start?
  1. “Minimum Attempts” of 10, 13, 16, 19…

  2. 	Place 1	Place 2	Place 3	Place 4	Place 5	Place 6	Place 7
     First Line	Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
     Second Line	Stinger	SharkTooth	Nemo	Phamplet	Gill	Bubbles	Flow

  1. “Minimum Attempts” of 11, 14, 17, 20…

  2. 	Place 1	Place 2	Place 3	Place 4	Place 5	Place 6	Place 7
     First Line	Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
     Second Line	Gill	Bubbles	Flow	Stinger	SharkTooth	Nemo	Phamplet

  3. “Minimum Attempts” of 12, 15, 18, 21….

  4. 	Place 1	Place 2	Place 3	Place 4	Place 5	Place 6	Place 7
     First Line	Flow	Phamplet	Stinger	Nemo	Gill	Bubbles	SharkTooth
     Second Line	Stinger	SharkTooth	Nemo	Phamplet	Gill	Bubbles	Flow

• b. Where does Nemo go?
  1. i. “Minimum Attempts” of 10, 13, 16, 19… Follow this cyclic rule, starting at step 1.
  2. ii. “Minimum Attempts” of 11, 14, 17, 20… Follow this cyclic rule, starting at step 2.
  3. iii. “Minimum Attempts” of 12, 15, 18, 21… Follow this cyclic rule, starting at step 3.
  4. Odd Rows: +4 | +4 | +1
  5. Even Rows: +0 | +4 | +4
• .
• How do I know it is right?
  • Proof!  I built another automation aid: An Excel spreadsheet that can tell you the position of nemo in all trials given an initial “Minimum Attempt“. It works!
  • You can download the spreadsheet to try it here. Choose any number of initial attempts, it will tell you all Nemo’s position accurately.
  • Is this the real logic behind Pradeep’s challenge? It may not be. It is probable that pradeep did something else, much simpler, and that only translates into this. However, this answers the mission of predicting Nemo’s moves.
    • I am sure that analyzing the findings above one can discover what was really that Pradeep did. I’ll just ask Pradeep on a mail for a suggestion. Testing hint: You can always reach out to the developer and ask him about his software and/or code!

Testing hint: Again, your mental model doesn’t always needs to be exact. As long as it works and answers the questions you have. Keep in mind that this mental model should be flexible and evolve with your knowledge.

• Conclusion sent to the ‘roulette’ stakeholders:
  Until a higher amount of randomness is added to the game, it is too risky to release the game as is now. A malicious player can find the logic of the game after some hours of work, and a pair of smart malicious players in even less time.

Thanks Pradeep, for the nice challenge.
It was a very nice experiment, and I learnt a lot. The best part, for me, was organizing my thoughts in this report, and I hope you like it too.

Related posts:

1. Exploratory Shopping – An analogy attempt
2. Software Testing is Funny! with Demetri Martin
3. Annotated Joel on Software – (Bad) Reasons not to test

So, a few hours before January is over, I’ll transpose here an answer to Testing.StackExchange about the last decade on testing:

Question: What are the most important software testing developments of the decade?

My Answer:
The question asks about the most important developments… Not the best or the worst, the beneficial or the harmful.
I’ll try to answer here with considerations by me and others I found on the net. Not everybody will agree that all these are good — even I don’t agree with all — but my approach here is more of a reporter than a judge.

1. Blogs were the most important testing development, at least for me .
   It helps me grow my testing philosophy. It helps by being a quick and instrumental medium to quality discussions. Following the great authors as they write is a great experience (many authors started blogging only during the 2000’s, like James Bach and Michael Bolton).
   One great thing about blogs is that they allow us to understand the flow of ideas as they are being built upon — instead of simply receiving the ideas later, in book, as was the custom 10 years earlier.
   • Tweeting, forums, and other collaboration platforms of Web2.0 are cool too, but I believe their impact to be smaller.
   • Blogging also allows a simple guy like me to share his Testing Thoughts :).

Other points:

1. The Context-Driven School gathered momentum with this name, and became well known. I couldn’t find tracks on the full history of the context-driven school, but the earliest mentions I could find to it with this name are from the very late nineties. It is clear that Cem and James were publishing context-driven papers from the early 90’s (see here), maybe before — but the first places where I found the “context-driven” name where from 1999 (at the software-testing list. This seems corroborated by C2’s page).
   • Again, I don’t mean to say that it started on this decade. I believe the work of the context-driven people was context-driven for the last 50 years , and all had always sought excellency. It is not new from the 2000’s, but it looks like the movement got a name — and exposure as a movement — only in the last 10 years.
2. Certifications gathered momentum too. Without entering in the good/bad discussion, it is a movement that had a lot of action in the past 10 years and affected the way we discuss software testing today. It wouldn’t be fair to count what happened in the 2k’s without mentioning them.
   • The ISTQB was founded in 2002, and is a popular certification.
   • The ISEB was doing certifications before (so this too, isn’t entirely a new thing), but all got much more impetus.
3. Greater awareness and recognition of the benefits not only of Exploratory Testing, but of testing in general, and testing as a career too.
4. Michael Bolton started consulting in the testing arena, but moreover started writing and publishing. His essays about testing and checking are very cool, and Jon Bach considered it “one of the most (in)famous and important posts to come along in our industry in a long time“.

For the next 10 years…

1. The weekend testing meetings in India were highly praised by James, Michael and Pradeep on twitter. Seems like they believe it will make an impact on testing in the next years.
2. Testing.StackExchange appeared too late in 2009. But… will it appear in the 2020 list? I hope yes.

Related posts:

1. Happy New Testing Year! Yet again!
2. Happy New Testing Year!
3. Happy New Testing Year Again! What? Too late?

Demetri Martin

One best friend of mine introduced me to Mitch Hedberg and Demetri Martin, great one-liner comedians. They are/were two funny men!! Three, actually, if you count my friend which is also funny. 

After hearing the disks for over a year, not only the jokes aren’t any less funny, but I’ve started to find subliminal testing messages in them .
I’m writing down these “insights” because I find value in them. And even if they fail to teach you something… Hey! At least the jokes are pretty funny!
So here go some favorite quotes, and their parallel in testing: 

. 

1) “A drunk driver is very dangerous. So is a drunk backseat driver, if he’s persuasive!
'Dude, make a left.'
'But those are trees…!'
'Trust me...'“ 

We are testers. As such, most of the time we aren’t driving the company: Essential operational decisions are made by someone else (project manager, product owner, CEO, whatever you call him).
But if we are not the drivers, our position is certainly close to this of a backseat driver: we have our maps, good knowledge of the area and our experience; and we give advice that is used to take real decisions. So don’t be a drunken backseat driver, it’s dangerous! If you are inebriated by an obsessive desire to fix a bug, or by a personal quest against/for colleague, by a blind belief in a set of metrics or a unfounded trust in a fictitious “Best Practice”, you’ll be taking your company directly into the trees.
And, as with any drunk person… Whenever your judgment isn’t objective… At least recognize/admit that you may be drunk! Will make it easier for everyone else. 

. 

2) “I used to play sports... Then I realized you can buy trophies. Now I'm good at everything!“ 

A plaque, a crown, a card, a trophy… It is common to find people who treasure “victory symbols”, I know I do. They prove you’ve mastered a skill.
Trophies are nice and great, but only if they are accompanied by skills and real world practice. Trophies that can be attained without these traits are just empty cups*. 

One should beware when dealing with “achievement symbols”… At times, acquiring the symbol does not mean acquiring the achievement or skill too!
For example, just as buying the trophy doesn’t make Demetri good at tennis, getting a testing certification won’t make a tester good at testing… The certifications syllabi try to teach only specific lexicon and terms definition, but not the real practice of testing, because they don’t/can’t cover the interactions between persons and players.
For some people, the joke could be read as “I used to practice and study... Then I realized you can pass a certification. Now I can prove mastership - without the effort of gaining it!” Be sure to be from the ones who keep learning and carrying the skill. 

. 

3) “I am afraid of sharks, but only in a water situation. If I saw a shark on the street, I'd be like 'What? F#*k you!'
It's funny, that's like the opposite of how I am with lions!“ 

What’s the Best Practice for dealing with sharks? “Escaping” or “screaming”, right?
What if the shark’s on the sand? Or dead? Not hungry? Not dangerous? You have an anti-bite clothe? Anti-shark cage? Suddenly the ‘Best Practice’ looks more like a ’so-so advice’, uh? 

The business of testing isn’t different. By looking at our surroundings and context, we can learn important details about our problems – and only then we can try to find an action path that solves it.
Without the context analysis, we may come with solutions that don’t address the real problem or does it badly. One size only seldom fits all. 

. 

4) “If I have to move up in a building, I choose the elevator over the escalator. Because one time I was riding the escalator and I tripped. I fell down the stairs... for an hour and a half.“

Automation is all nice and fine… Until you trip on it! Then it takes long time to recover from the fall (you could be long done by this time) and the fix to the system can be so crooked that it lets you prone to further tripping.
The elevator is a much more robust automation (not only it goes up and down, but it is much safer to trip in it ).
Anyway, who said you have to automate? An equally good way to go up and down is the stairs! You don’t need to wait for them to come, they are as quick as you need (or can make) them to be, you can stop to look around or change direction in the middle of the way, they don’t limit weight or number of users and they are never out-of-order.
Slipping, well, can hurt a bit . Just remember that manually doing things gives you many opportunities a machine cannot provide. 

. 

5) “I saw a sign on this door; it said, 'Exit Only'. So, I entered it and went up to the guy working there, and I was like, 'I have some good news. You have severely underestimated this door over here by, like, 100%, man!'“

Labeling is a natural instinct: The Bible depicts Adam naming animals as his first action in Eden, and Aristotle has been classifying everything since the 300 BC.
But the labels we use can limit ourselves or our tools.
“I'm a tester, I can't code“.
“I'm an engineer, I can't sell“.
“I'm a newbie, I can't help“.
Good news! You’ve underestimated yourself by at least 100%! 

. 

6) “I want to make a revolving door that says 'Pull' on it, just see how obedient people are.“ 

Meeting a Test Case that says “1) Reach to revolving door”, “2) Pull the door to open”. What do you do?
Maybe nothing is wrong. Maybe there are good reasons to make people walk backwards at the entrance.
Or maybe there isn’t, and then, who’s decides? The test case? The design? The developer? The tester?
What are your ways to discover about the door product and its users before deciding on obeying or not?

. 

7) “I was walking down the street, and this guy waved to me. Then he came up to me and said, 'I'm sorry, I thought you were someone else.' I said, 'I am.'“ 

Wow, that’s a funny joke!
Beginning a task with a very clear objective is usually a good thing – but failing to identify or even acknowledge alternative outcomes may not be.
A good friend was working in an academic project, trying to prove a theory about the nature of lasers. However, while progressing in the research, the calculations made clear that the theory wasn’t right. He was desolated – all this work and nothing to publish! He failed, momentarily, to notice that demonstrating a theory wrong has as much scientific value as proving it right, and the publication will advance science just as well.
Testing present us with many similar situations. We can start a test chasing a specific bug, and miss other bugs that may appear. Or we can keep our mind open for different outcomes, and grow your tests as we go from these outcomes. A different bug is hiding there, believe it . 

. 

8 ) “My plumbing is all screwed up. Because it turns out, I do not own a garbage disposal.“ 

How many times we rely on a garbage disposal when we don’t have one?
- Don’t worry, release it that way, we have a fantastic relationship with this customer. What if you don’t?
- It’s fine to release that way, the customer has a great quality assurance team and they’ll catch it if there’s a problem. What if they don’t?
- No problem, we can work Sundays — our testing department understands our needs and won’t care. What if they do care?
- This looks like a very nasty bug. But customers will always have the underlying framework, so it is fine. Will they always? 

. 

9) “I'm in a weird position, because I like rainbows, but I'm not gay. So whenever I go out wearing a rainbow shirt, I have to put 'Not gay'. But I'm not against gays, so under that I'll have to put '... but supportive'. It's weird how one group of people took refracted light. That's very greedy, gays.“ 

Once in a while a group of people will come, wrap common sense ideas in a new named package, and claim ownership, which is as reasonable as taking sole ownership of refracted light.
I could parallel this one to many things or groups, but I chose one where this happens with a lot of enthusiasm… The Agile movement.
Suddenly, you can’t focus on customer and organize a software project in a change-adapting way, without being called Agile. That group just took no-nonsense practices that had been used for decades and re-branded them as if they had invented it!
Don’t get me wrong – it is my opinion that what Agile preaches are often-good-practices-for-many-contexts and there are testimonies of companies, projects and souls being saved from a bureaucracy and poor quality hell by adopting Agile as a new approach. But this does not mean that anyone who is delivering quality in a “release early release often” way is Agile.
For example, even being a total newbie and never part of an Agile team until now, my personal testing approach focus on happy customers, working software, individual interactions and change — without me needing to be an Agilist.
More experienced great people had practiced all that, and test-first, and XP for decades. 

More than that, not only the practices’ refracted light was taken, but even the word “agile” itself was expropriated – see James Bach’s note at his ‘Who stole Agile’ post. 

——
Hope you enjoyed the article, or at least Demetri’s jokes.
If you have more insights and funny stuff to share, don’t hesitate to contact me – I promise to laugh.

Related posts:

1. Finding Nemo – Solving Pradeep’s software testing challenge (an exploratory approach)
2. Annotated Joel on Software – (Bad) Reasons not to test
3. “Create your testing portfolio” presentation

Request: Are you a lawyer? Please send me corrections

Matters related to law, and all the discussions around it, interest me much — especially when related to Software.
This made me read about the subject and keep contact with the legal representatives within the company I work for. This also motivated me to learn and lecture about the legal guidelines in software development adopted by our company, and to lecture about legal matters on software in general at the last Israeli SIGiST conference.
Most important than all, this made people share with me a lot of comments, questions and stories pertaining to the law.

For example, one colleague brought to my attention a case in which he and some friends had bought a PlayStation 3 in the local Office Depot website for 220NIS – when the normal price is almost tenfold! They believed it was some special sale promotion, but at the end Office Depot announced it as a typing mistake and cancelled the sale after it had been acknowledged (sale confirmation by email).
(There is an article about the episode in Hebrew here if you want to see it).

What is my opinion on the legal aspects of the story? I was asked.
I don’t have one, as I am not a legal professional, I answered. I am, though, a Testing professional, and here is the tester rambling I sent by mail commenting the occurrence:

——
In fact, electronic transactions are special and different that normal stores. They can be built without any human interaction whatsoever, and the lack of human monitoring can bring new and unexpected problems.
Note that I can’t have an official Legal opinion, as I am not a lawyer. I’ll just write what I learn.
A usual understanding in contract law is that once both parts show agreement to the deal, then the deal is closed and valid (I think a lawyer would explain this by stating that there is a mutual offer and acceptance of goods or services). Such agreement is shown, for example, by accepting the payment money.
In such a case as this story, the electronic sale often includes a debit in the buyer’s Credit Card (that turns into a credit at the cancellation of sale). If a service provider accepts payment for a service, he is in obligation to provide the service at the agreed terms. If he didn’t agreed to the terms, he would not have taken the payment. So in our case the fact that a payment transaction was completed supposedly forces the two parts to honor it (the store should not cancel the delivery just as you should not cancel the money debit).

However, apart from the amateur philosophy, we can find a bit more of info in the net about this sort of things…
a) Not only the payment debit can be viewed as a contract acceptance, but also the confirmation mail you receive when buying online can be seen as such, depending on what it writes.
b) The Terms of Service of the website can be used for or against the store, depending on their presence and the judge interpretation of them.

• For example:
• Buy.Com (huge company, 290 million revenue) announced at $164 a $588 monitor. They tried to cancel the sales, and lost more than $60000 in a lawsuit (CNET link).
• On the other hand, Amazon.Com (huger, 19 billion revenue) announced at $11 a $449 handheld (CNET link). They cancelled the sales successfully, claiming that the Terms of Use state that there is no binding contract until the confirmation of shipment arrives.
• Buy.Com had no such protection in their terms, and after the lawsuit they updated the terms to include “have the right to refuse or cancel any such orders whether or not the order has been confirmed and your credit card charged”.
• If it will always help? Not sure. In this link a representative of the FTC says that it will depend greatly on the state the lawsuit is taking place.

In the US, there is a concept of “loss of bargain”, that could allow one to buy the PS3 at normal price and sue for the difference (which represents the bargain lost due to the store’s malpractice). Of course, if the judge identifies the buyer as having bought it to exploit a weakness or obvious typo, the attempt can backfire…

Now that we learned some principles of the domain… We go for the most important…
As I said, I have no legal opinion, but I do have a tester opinion.
First, this shows (again) that there is no final answer in the law yet for new technologies and new problems.

And I now know that for assessing risk on an e-store product, one has to answer these questions too:

• Does the system have any protection against mistakes
• This can be done by issuing an alert if there is a big discrepancy after:
• comparing the product to similar ones
• or comparing to the previous price
• or comparing to the price in the normal (brick) store
• Does the system require a manual human confirmation before sending the payment or confirmation mail?
• This can be bad for a large operation, but for small-medium business, can be a good way of catching mistakes
• Does the system alerts the administrator if there is a sudden surge in the purchases of a newly added product?
• Do the Terms and Conditions protect the company in case of mistakes?
• Are the Terms and Conditions clear to the end user? At least in a visible place?
• Is the user obliged to mark an acceptance to the Terms during log-on, or it is just a link in the page footer?

These are some of the questions I would ask when faced with such a product. There are more, certainly — please share your own ones in the comments or by email

——
One last comment…
Sometimes the typos make the prices higher than the norm… See this picture, a real example from Amazon selling this $400 card for a whopping $5 million!! That’s a lot of money for a graphics card…

Hummm… would they cancel this sale as a mistake too?

Related posts:

1. Happy New Testing Year! Yet again!
2. Software Testing is Funny! with Demetri Martin
3. Three texts that changed my (testing) life

My colleague Issi Hazan was asked to teach the ITCQB syllabus to testing-oriented-students at TechCareer, but he bravely thought of pushing them to real tests instead, even before they find a job – this would allow them to build experience and get a job more easily (one way to solve the “hard to get a job without experience, hard to get experience without a job” problem).
So we built a lecture on how to “create a tester portfolio“. We explain how testing can be done even outside an enterprise environment: Lots of Open Source projects are seeking for good information from software testers, and a good record there is certainly sure to help. Crowdsourcing is also a  possible way to work on testing before getting a real job (see two posts on crowdsourced tests here and here).

The presentation is available below in flash format, and can be downloaded here: “Create your testing portfolio“.

But “PowerPoint is Evil™“, so I feel compelled to write it in real words too.
It goes like this:

Introduction

If you are searching for your first job in software testing, one of the big challenges you will face is how to have your résumé stand out. Either if this is your first job ever, or you’re changing career from a different background, the question remains: How can your résumé compete with seasoned testers’?

The solution to the 1st challenge constitutes your second one: actually acquiring this experience that differentiates the seasoned testers. And this is hard.
People looking for jobs often complain they “can't get a first job without experience, and can't get experience without getting a first job“. The logic sounds leak proof solid… where it not for the fact that, well, most people do get a first job.

So how can a newbie gain this differentiation?
Many people seek certifications, hoping that the additional line or two in the CV with the uppercase LETTERS will cover for the lack of skills and practice. Now, here’s a secret: they won’t.
Certifications do not give you skills or wisdom; they give you a determined lexicon and specific definition for terms. Some employers may want that, but they know it does not equal practice.
Software testing is an art. Young artists do not collect “painter certifications” or “canvas master certifications“… What young artists do is paint a lot, toss all the colored canvases in a big binder, and show them to galleries. Gallery owners can through this portfolio see their practice, evolution, style and commitment.

This is our suggestion to you: start testing, and be prepared to explain your tests to prospective employers.

Meet your Pseudo-Employer

Testing without having a tester job sounds farfetched, but it is actually within reach.
Of course, you can enter a bug-hunting-spree and find defects in any software from your desktop or the web. While you can learn techniques and tools this way, this method misses the most important (and fun) part of testing: The interactions.
The interactions with people involved in the software will give you focus on what is important for them to know. It will also redirect your tests as you go with feedback and comments, making sure you are always providing useful data. Last, being in touch with this human side of the software will give you the opportunity to explain, discuss and participate in fix/no-fix decisions – transforming your findings from simple bugs to valuable information on quality and risks.

And there are many ways to be involved with testing in the development of software.
Here are a few options:

–> Academy Projects

There is a lot of interesting software being done as final projects for Computer Science graduations. These students do all the work themselves, and they will be certainly glad to have help in testing the software.
One of the great things here is that the interaction is all 1 to 1 and face to face, allowing it to be a closer and more dynamic relationship. Feedback and bug-fixing times are quick; you can suggest a change and then see it done the day after.
Posting your ‘testing offer’ as a note in the wallboard, or asking from one of the project mentors to point you a needy student, are all good ways to get started.
And, for what it’s worth, you may appear in the credits on the project report!

–> Open Source

Sourceforge has hundreds of thousands of open source software projects under development. You can pick one that suits your interests and limitations by using their filtered search, and you can also take a look at their “Help Wanted” section, which many times ask for help in testing.
The open source community is thirsty for contributors, and they respect sound advice. If you show your competence and are provide consistent views/comments, the community will pay attention – and they don’t care about your degree, background or years of experience.
And, for what it’s worth, you may be given credits (or ‘tip of hats‘) on the release notes!

–> Crowdsourcing

There are websites where you can receive money for reporting bugs. The two examples I know of: uTest and Flash Mob Testing.
Testing in this environment may pose problems (see this post for a list of concerns) and the interaction you have here is by far less involved and meaningful than the previous two options.
We listed this in the slides because the couple dollars are tempting and people may prefer this option over the others.

Note that here, you won’t get public credit for your work, in case you wanted to showcase it in an interview.

–> Volunteer for a company

This one wasn’t at the slides’ first version, because I had forgotten about this history completely.
When I was at the university, I had a friend working for a software company, a small one without a testing team. To exercise my ‘criticism’ (I didn’t know I was ‘testing’, back then) I started to write him comments on their webpage and their application, notes that they triaged in their bug meetings and fixed in the next release (sometimes).
So it was volunteer testing for a real company.
(This entire story slipped my mind until 3 weeks ago I had a course in the same building as this company is located. It all came back to memory suddenly: that was my first testing gig!! ).

In summary

There are many ways to test without a testing job. Find or invent the option that better fits you, and start differentiating yourself as a real tester.
Try it!
With a little work from your side, you can demonstrate to employers your capabilities, style and passion. And having volunteered to test and provide data isn’t something a recruiting manager sees every day, so it is likely to attract his attention.

The next steps

After you’ve decided where to start testing, following a consistent way of work will make your testing much more efficient.
The slides include many suggestion on how to get a good start: Follow the forums discussing the software (and ask questions!), be sure to read the bug submitting guidelines so your information is treated with care, and find a lot of bugs

An Alternative

When applying for a job, capitalize on the experience you’ve already got.
Software projects need domain expertise just as they need testing expertise. If you were a stock-broker, a software house developing stock-market software can have great benefit from your testing. If you speak languages, a lot of specific software can benefit from this trait.
See my comment in this post: The entire experience you bring to a team is valuable; recruiters aren’t always only looking for super-testing-wizes.

Good luck!

Related posts:

1. Software Testing is Funny! with Demetri Martin
2. Testing Insights – The Graphing Calculator
3. Finding Nemo – Solving Pradeep’s software testing challenge (an exploratory approach)

Instead of a new post, I revisited and modified last month’s post, About youTesting with uTest.
It has now more content, and still has a discussion of pay-per-bug models.

The initial opinions are still there. While the pay-per-bug model presented by uTest is certainly innovative and interesting; the model still misses a lot. It will certainly be center of discussion many times in many circuits .

To some extent, trying to get quality assessment and counseling from isolated bugs from isolated testers is like getting medical advice from many different isolated doctors without them examining you personally even once.
But, to the other extent, bug reports do consist an important tool for quality evaluation and direction assessment. And well formed groups can benefit from the yet another list of bugs received, as long as they don’t confuse them with complete product testing.

In one of my discussions of the matter with friends, I was encouraged to ask the people working in uTest what they think of these points and how do they and the companies solve the apparent drawbacks.
It is a great idea, as they seem to be very forthcoming and they do have a lot of experience in software development and quality. The CEO, Mr Doron Reuveni, tweets and sports and seems approachable. Not many companies have such accessible CEOs, and I’ll try my luck with him. If he replies and answers our questions, it will be very cool, and I’ll add the info to the uTest posts here.

Related posts:

1. About youTesting with uTest

This is an interesting topic:
I’ve been involved lately in many conversations about uTest, or more specifically about its model.
uTest is a website where companies can post their software, along with some guidelines on focus areas, and users around the world can download the app, find bugs and get paid for bugs reported (as long as the bugs are accepted by the posting company).
There is a lot of confusion/discussion around the good parts and the bad parts of the model, so I will share here some of the points I had taken from these conversations (thanks to all the friends who shared insights with me)… Some attentive readers will notice the article is an almost copy paste from a reply in the software-testing group.

Please note that I am not saying “uTest considered harmful” or “don’t use uTest” or anything like that.
Please note that I am not saying “uTest is great” or “use uTest” or anything like that.
All I want to point here are some of the strengths and some of the weaknesses of the model, so every one (both testers and companies) can decide for his own context. I welcome debate over any of these points, and will update my post accordingly.

–> (A) Here we go then. Some strenghts of the model:

1. Crowdsourcing provides diversity:
   James Whittaker in his “Future of Testing” webinar considered this model of testing (there I heard the ‘crowdsourcing tests’ term the first time) the next evolution in software testing, as it allows a huge number of testers to test in very diverse environments and configurations.
   Full disclosure: The Webinar was given for uTest and at their website.
2. The managers are nice guys trying to create a nice community.
   In the case of uTest, they have blogs with guest writers such as James Whittaker and others. They are on twitter and occasionally twit a testing article (more often they do brand promotion, which is fine too), they offer bug battles and discussion forums.
   And at these venues they do good. I have heard that Doron Reuveni is a really nice guy, and I believe it from his twits and posts.
   Of course, people tell me that all this community building is about money and business, but that’s completely legitimate. I take money for testing at my job, too. Making money doesn’t make anything bad.
3. Simplified development cycle:
   For some companies, it appears that crowdsourcing makes their development lifecycle more efficient. See this article on how crowdsourcing supposedly slashes development time.
   The points commented in the article are a bit dangerous, in light of the concerns raised below, but for companies that have yet to build a testing team, starting to receive bugs before the testing team is complete can be a good thing.

–> (B) On the other hand, the uTest model has some weaknesses, like:

1. “Software testing” misinterpretation:
   The idea of this model is ‘pay per bug’. Companies are outsourcing testing by receiving bug reports from the community.
   But submitting bugs is only one part of the work that a tester does. Testing (functional too) requires involvement with the development life cycle. Testing requires understanding of the user needs and business needs.
   By putting the testers far away from the company and their decisions or business view, the testers have little option to ask good questions. They have little space to provide real information about risk (that’s what testing is about, right?), and thus they are likely to practice poor testing.
   Moreover, testers get their money for reporting bugs, so this will become the ultimate purpose of submitting them. In any other context many would agree that bugs aren’t there just to be reported or even just to be fixed — bugs raise questions about the application, about the value it provides and about the general direction the product is going.
   When you take the testers out of the fixing decision, you miss this healthy discussion. In this model, if a bug doesn’t get fixed, the tester isn’t probably alerted about this fact. What’s more, the tester may not even care, as he already cashed the bug’s money and “fixing or not fixing is a business decision where we don’t have a say”, right?     And with that, reporting well detailed and complete written bugs isn’t a goal for pay per bug testers too: If you spend valuable time writing a bug report with detailed information, you are not making money! Plus, another person may log this exact bug with a poor report before you finish your perfect report (you don’t receive money in this case).
   Sure, the model does not foster or defend bad practices on purpose. But it has its way to motivate them, in a sense.
2. Poor metrics (paying for bug reported):
   Companies like uTest didn’t invent the “bug quantity measurement metric“, of course. It is an old approach that has failed in many places from what I learnt (might’ve been successful in other places) and is definitely not suitable for all contexts.
   Even if applying companies receive great comprehensive dashboards displaying different quality aspects of the software, at the end of the day they are only seeing bug numbers.”You get what you pay for” — the company is paying for bug quantity and testers (as anyone else) will give you what you measure them for, a large quantity of bugs. It is easy to confuse this data with information about risk or information about value. uTest writes in the Terms of Use: “uTest makes no promises or warranties as to any specified quality standards, testing coverage, maturity, quality measurements, the general quality of services or feedback, or results provided on or through its website.” Now, this is only a standard disclaimer, and some may say these measurements are rubbish anyway, but what uTest is saying, in essence, is “you get only bug numbers, nothing else“, which is true. Companies must understand that bug reports don’t make for a quality or value assessment. See point (C/5) for a related comment.
3. Quick and Dirty is rewarded:
   The rules are: time consuming bugs pay the same as quick bugs. From their Terms of Use: “uTest shall ONLY pay you for bugs reported and approved by the applicable Company and NOT for the time you incurred by analyzing bugs or time incurred for any other purpose.”
   Which type of testers receive a higher score at this model? Testers that report simple bugs will be at higher ranks than the ones that hunt and investigate and find hard bugs.
   Consider this together with point (C/2)… and you’ll have testers that log a large number of shallow bugs.
   This kind of behavior is acquired by whoever practices it… And by the end of the day, some testers with high scores at uTest may have become accustomed to quick and dirty work.And what about bugs that don’t easily reproduce?
   Bugs that don’t always reproduce need special attention and a lot of time invested. Forget about those in crowdsourcing, as more attention equals less money… One is risking that the ones to find the tricky bugs will be the end customers.

–> (C) Some other points should also be considered:

1. Bad disputes fostering:
   Any discussion translates into money. And money, being a terrible motive , can make disputes between a company and a tester biased on both sides.
   Software companies will want fewer bugs, and software testers will want more. None of the two is a good goal, and both can lead to wrong stands.
2. Bad attitude from testers causing bad results:
   Testers from a ‘real world’ project might wish to receive better built software so they can jump into important matters instead of clinging at innumerous surface bugs.
   In the pay-per-bug model, testers will leave a mature software to go test an immature one (more bugs = more money)… so the tester tests worse apps and the software company loses the testers exactly when it more needs them (when the app is stabilizing, testers will migrate to a new, immature one).
3. Disruptive competition between testers:
   In the company I work for, if I share information with a colleague and he uses it to find a bug before me, everybody is happy (as there was a collaborative effort that generated a useful piece of info). Testers in a pay-per-bug competition will be less inclined to share information, as bugs opened by someone else mean less money and less score points.
   In their model, you compete with other testers, and this enters the testing culture of the participants. And testers who compete are les useful in a team than testers who collaborate.
4. The score feedback isn’t objective:
   An award winning uTest tester commented in one of the online discussion that the score for testers doesn’t represent testing skill.
   The skill set required to be a good tester isn’t exercised in such simple view of testing as ‘receive the app, try it, list the bugs’.
   And also, don’t forget that there’s money involved and this usually undermines objectivity .From uTest Terms of Use: “Testers shall have no right to either control or influence their fee rate or feedback regarding their reputation or success.“. Of course, here it is simply a technicality, explaining that the decision over what is a bug is in the hand of the software company hiring the tester.
   But this is an approach where one cannot get more reputation by learning more skills. Instead, one has to wait until someone decides how good he is, with few chances to appeal. Different people in different companies think differently, and the tester has no control over this process here.
5. Companies get the wrong picture:
   A large amount of bugs and activity from uTest-like websites is likely to inebriate the software developing company into thinking they’re getting great value (just as a large amount of activity around automation does in other cases).
6. ‘Unfair’ and harmful competition with your in-house testers:
   For companies that do have testers in-house, sending the application to be tested at crowdsourcing models may pose additional problems:
   - How do the ‘inside’ testers feel about that?
   - What happens when the outside users find bugs they missed?
   - And for the same bug… how come someone from the outside get paid for reporting it? What conflict of interest may appear when a tester realizes he can suddenly make money if he (or his wife, or his brother) reports a bug he found during work?
   - Does a bug-finding competition affects how the in-house testers do their work?
7. Update about different types of test:
   The first version of this article said that “There are companies that value regression tests, for example. As these usually generate fewer bugs, testers at pay-per-bug will not perform them. The same is true with other types of tests that raise questions and not bugs.”.
   Update: I recently discovered that companies that want to run regression or a set of specific tests do use uTest. They post the application with a list of scripted tests for people to follow, and pay for the completion of the scripted tests. So there are ways to do these tests.
   (However, in one such example that I looked at, there were no bugs reported in the script executions. What would be the results if instead of paying for the tests passing, the company asked for bugs? But this problem isn’t exclusive of uTest, it happens everywhere, I guess )

…

There may be more and other points for and against the model.
Please write them on the comments or in a private mail, get the conversation going on this. I may change my mind on this matter too, why not?

Related posts:

1. In August, a rewrite of July’s uTest post (and maybe official feedback)
2. “Create your testing portfolio” presentation
3. Annotated Joel on Software – (Bad) Reasons not to test

A couple of months ago I tried to access a site (now I don’t even remember which it was) and was greeted by the note below:

Let’s read that again: “The attack was not successful and our safeguards prevented intrusion“. Ah, nice. But Mr BudgetHostingWeb, can you explain how, if the intrusion was prevented, how the hell did the “server's hard drives failed during this process“?

Maybe the hard disk sensed that an attack was being attempted, and then committed suicidal, in some sort of self-aware sense that “it's better to die than have those hackers get the secret data stored in me!”

(Side note: If you ever had to deal with web hosting service’s backup, you know it is a nightmare. A service I used once had “hard drives problems” too and their last backup was a month old — even if they had claimed to do daily backups. No, I didn’t sue, and I think I wasn’t even surprised (Quality is Dead, right?), but I did change to a different host later.)

So now we have customers who lost all the data of his website.
If they had good backups, they’ll be able to upload it with the FTP Upgrade package — and if they hadn’t, well, will have to start from scratch.
And this is being communicated to him in such a lousy way… Note that the alert in the page is not targeting the site visitors (which do not upload files by FTP), it is targeting the web page owners. Why would an alert to the page owner be glued to the home page of a customer? Maybe because they didn’t send the communication in any other way?
(Maybe the hard drive with the customer information failed too, who knows?)

And the alert says “the attack was not successful“, which is what makes one (me) angry.
Mr BudgetHostingWeb, your customers are unhappy, they had a major data loss, and have to fix it themselves instead of doing business. Please re-write the message to something like “The attackers succeed to cause failures in some of our hard drives, and unfortunately we don't kept backups of your data. As far as we can discern, no information was stolen, but we are deeply sorry for this inconvenience.”

Because shaking yourself free of responsibility isn’t exactly the best experience you can provide your users.

Let’s sympathize with one of their customer, which is a great practice for software testers:

I did a search on Google to try to find sites that were part of the problem. Even thought a lot of time passed since the incident, Google found three (two of them do not have the text in their sites anymore, Google is outdated too, so it leaves us with only one customer specimen, Free Range Whippets):

Whippets are lovely pretty dogs, and the first link on the screenshot above sports the following notice:
 ”
 We regret that our host server was recently the victim of a serious hacking attack.
 The attack was not successful and the safeguards prevented intrusion however the server's hard drives failed during this process.
 All services have now been restored.  Unfortunately, our site was one of those affected, and we need to upload ALL the files again.
 Some links and photos will not work or appear until we re-upload.
 We hope to be fully back online soon but finding after six years of loading photos, they don't go back up overnight.
 Please visit us again and click all the links! 
 Thank you, Lori & The Whippets
 ”

6 years of work, gone! Is that what the safeguards were supposed to do?
The hacker attack wasn’t successful, but Lori still has got broken links in the whippets’ images .

Related posts:

1. BotT: No multi-million dollar prize for you. Again.
2. BotT: Ctrl+Enter on Firefox
3. BotT: Linus’ bug, youtube no workee!

I like books, better yet when they are good/useful books, and even more when they’re cheap — so I came to the fair prepared! I planned a budget (100 NIS) studied the catalog of discounted books and decided beforehand which books I wanted to buy:

- Book A: 43 NIS
- Book B: 30 NIS
- Book C: 12 NIS
- Book D: 12 NIS
- Book E:  9 NIS
        (106 NIS total)

Book A and B were books that I actually was interested in, and the other 3 I put on the list just because they were cheap , and were worth considering if they content looked cool.

When I got to the fair, I skimmed over the books and found out that books E and D weren’t that interesting, and that they would probably remain unread, so I dropped them from the list. Book C wasn’t that appealing too, but I left it on my cart meanwhile.

A few minutes later, I noticed that there was a very cool book on a subject that I am already studying — but that one cost 75 NIS!
My list was now over-budget:

- Book A: 43 NIS
- Book B: 30 NIS
- Book C: 12 NIS
- Book F: 75 NIS
  (160 NIS total)

As such, I called my wife, telling that I found other relevant books, but the whole list now exceeded the initial budget we agreed on. She gave me permission to double the budget and to spend up to 200 NIS. Yay!

Just before getting to the check-out teller, I discovered that there was a new volume from a set of books we have, and this was an opportunity to complete the series. Price of this book: 63 NIS.
Adding this one on top of the others would explode the budget again! I really wanted this book, and removing only book C wouldn’t help staying on plan — so I removed book B (a sacrifice for budget’s sake) and then C (not so interesting, anyway).

I went home with three books:

- Book A: 43 NIS
- Book F: 75 NIS
- Book G: 63 NIS
  (181 NIS)

I was happy with the books I took home, so was my wife, and plus: she was pleased that we stayed under the ’second budget’.
And then it struck me…

So, this is my analogy for Exploratory Software Testing.
Many others are available:

• Michael Bolton has one here (charting unknown waters)
• James Bach has one here (and exploration and jigsaw solving)
• Ben Simo wrote one too (improvisational theater)
• Jonathan Kohl wrote his here (musical creation)
• … And now I present one of mine . I hope I am worthy of sitting with such eminent group.

But note: this analogy, as every other analogy, doesn’t have exact 1:1 similitude/parallelism!
It is intended as a line-of-thought, to help explaining the process and mindset… to clarify that exploratory testing isn’t a ‘quick, run and do whatever tests you can squeeze in an hour’.

Related posts:

1. Finding Nemo – Solving Pradeep’s software testing challenge (an exploratory approach)

He writes books, software, and gives interviews about the craft and business of software. Not only that, but (not surprisingly) he’s also got a blog.

Two months ago he wrote that reading your colleagues code check-ins is a good practice, and I think this is good advice. And good advice for software testers too: I read the bugs submitted by my team on a regular base, and it’s been very enlightening (often to me, at times to them too).

I don’t like copy-paste, but as an experience on the parallels between development/testing, I’ll copy his entire post, just changing the parts that are development-related to testing-related words.
“Great artists steal“, right? Let’s see if it is intelligible (my changes are in blue).

Do you work as part of a software testing team?  Here’s a piece of advice for you:

Read the bugs.

Every morning before you start your own testing tasks, use your favorite bug tracking tool to look at all the bugs that everybody else submitted the day before.

There is a reasonable chance that this advice is worth exactly how much you paid me for it. 

Still, testers at many of the best teams I have known do make this their habit.

Reading the bugs is very likely to produce two benefits:

1. The bug might get better. You might find something in the bug description that needs to be fixed. Or you may find a different direction to investigate and find the inherent fault that causes the failure. Or you may suggest another implication of the defect that increases the customer impact perception… So on.
2. You might learn something. Maybe one of your coworkers is using a technique you don’t know about. Maybe someone else has a tool you can use. Or maybe reading the bugs simply gives you a deeper understanding of the project you are working on.

Of course, use some common sense about whether this habit is practical in your situation. For example, if you work on a team of a thousand testers, this probably won’t work well for you. Then again, I assume if you work on a team of a thousand testers, almost nothing works well for you, so this habit might fit right in. 

(Posted with Eric’s permission)

– Hi, Nancy? (Name changed. And she wasn’t the testing manager, she was from the HR.)
– Yes.
– Hi, my name is Shmuel and I saw you’re offering a software testing position, the “QA engineer” opening…
– We have it, right.
– I saw that one of the requirements is to know the STD and STP methodologies, and I was wondering what are STD and STP.
– Well, if you have no experience on that, then the position is not relevant to you.
– No, no, I guess the experience is fine; it is just that the announcement made me curious. These acronyms can mean many things, you know? So I want to know how your company understands them.
– They mean specification documents.
– Does STD stand for Software Test Documentation? What does the STP mean?
– No, STD is for Software Test Design, and the P… Well, both mean specification documents.
– Why design? Is that the specification of the tests to run, of the testing system, or of the product?
– Why don’t you just look up the meanings on Google?
– I tried right before calling you. The acronyms are all over the web, but they have different meanings too (*).
– Listen, these are very important topics, if you don’t know them the position is not relevant. They’re very important, and very well known.
– I see. Thanks you Nancy!
– Thanks, bye bye.

Very important and well known. Well, I guess I’ll put them in my list of things to learn… As soon as I decide which meaning I want them to have…

Let’s see. I guess two of the best “methods” I use in testing are:

STD – Software Test Drawing: Scribbling in a whiteboard (pen+paper are good too, but my experience shows whiteboards are much better). This is especially useful when you’re working with someone else. With some squares and circles explanations become simple, concepts become clear, people can point when they talk, you can easily delete mistakes, and the best part is that usually there is always a joke at the end of the process. Whiteboard scribbling is a major testing tool for me.

STP – Software Test Peer (or Pal): If you are like me, your brain is constantly storming . Nothing beats thinking out loud or talking to peers and fellow testers/developers about the challenges you face — more often than not you’ll learn about a direction you hadn’t thought of or about a tool that can help. Works fantastically with STD .

I may start an “Alphabet of Really Useful Software Testing Things” — please contribute on the comments.

(*) In fact Google says D and P apparently can mean Design, Documentation, Development, Performance, Process, Plan, Programming…Pick your own definition. Most companies seem to accept STP as Software Test Plan, though.I still like that anyone can pick the definition that best suits his needs or business, though. So I’ll stick to mine .

Note to my employer: No, I am not looking for a job elsewhere yet .

Today morning I was greeted by an Email from SQE: The subject read “Are you certifiable?“.
My first reaction was to discuss the term. If I am certifiable? I? In my mind, I was arguing whether a person can be considered certifiable or maybe the topic of certification is the one certifiable.
As in “Software Testing is (or not) a certifiable topic” against “Johnny is a certifiable (or not) software tester”.

I was puzzled over the confusing choice of words:

A lot of good people (Rex Black, in a moderated way Lee Copeland, Michael Stahl (link to an unrelated article)…) support certification, and a huge lot of good people think that either the software testing arena is immature to certify, or the certifications out there are inadequate, or both (James Bach, Michael Bolton, James Whitaker, Cem Kaner…). For transparency’s sake (so you get the context of this rambling), while I admire and have learned a lot from the first ones, the last ones are personal heroes (ok, Michael Stahl is a hero+guru too ).

So we did not even come to an agreement if the craft itself is certifiable, and you are talking about certifiable persons? How does that work?

.

.
At this moment I stopped musing and went to the dictionary:

Ok, my guess on the meaning of the work was correct: “a certifiable fact“, “capable of being guaranteed“.
So was the other meaning, about certifiable persons: “endorsed authoritatively as having met certain requirements” (in our case, an exam with multiple choice memory questions ).
But wow. That one in the middle is a curious meaning! “fit to be certified as insane (and treated accordingly)”
Certifiable == Insane. And while I don’t see a problem with insane people being ISTQB certified and vice-versa, it made it even a weirder word choice.

I had to search more in the internet to see that this “are you certifiable” idiom is a bit common in english. Microsoft even has a game with this name here.

So it was a joke! Ah, all a joke, but they kept me thinking.
I guess that most of the non-native speakers of the SQE newsletters didn’t get the joke. And they were this close to being offended (depends on the culture one comes from).
SQE (and all), it was good as an excuse for me to post the links for certification articles, but pay attention to your publications’ audience! It takes a lot of time to get your jokes overseas .

Related posts:

1. Testers don’t think like Developers think like Computers
2. Job Description
3. 4.5 things I learned at Sigist Conference, day 1

I enjoyed the story very much. It is very exciting to see the passion he had (has) for his software and how he was committed to it. Plus, Ron is a great story teller.
The graphing calculator had all the ingredients of a cool app. It scratched a developer’s personal itch, and is a great example of NeoVictorian computing: built for people, built by people, crafted in workshop, inspired.
Actually, if we’re commenting on NeoVictorianism, Ron was one that really “woke up one day to find himself living in the software factory“. The night got very cold, they said the factory is going to close and he should move somewhere else. The cool part? He kept doing his individual craftsmanship inside the corporation. Secretly.

Now to our matter, testing. He’s got some very good comments on Software Testing (which he calls in the story as QA. Not his fault, but ours). So nice comments, that I’ll just transcript:
At (~20:24), “Our biggest problem was QA“. “Fundamentally, you can’t do your own QA — it’s a question of seeing your own blind spots“, and “it is expensive to do it well“.
At a moment in the story, two testing engineers came to offer help, as in their official work “they were really bored because they spend lots of times developing automatic test tools and now they’re at the point where they push the button and the test starts in the morning and they come back in the evening to see which API’s break“.
These testers were successful because they badly wanted to explore the new software, and also because they had good mathematical experience/background — closely fit skilled testers!

Ron later speaks about usability (~35:57).
He describes the process of the usability studies they conducted after they’ve been programming with users in mind. Ron (which understands well the importance of usability: “In a classroom, any time spent frustrated with the computer is time taken away from teaching“) says that even after all the attention to usability they had, they were surprised by the results of the study: “Sitting behind a two-way mirror, watching first-time users struggle with our software, reminded me that programmers are the least qualified people to design software for novices. Humbled after five days of this, Greg and I went back and painstakingly added feedback to the software, as if we were standing next to users, explaining it ourselves.” In the movie, he adds that they were “pounding on the glass” and that “all engineers should be forced to go through this, watching real users actually trying to use their software“.

There’s another interesting comment at the very beginning (~02:30). Ron comments on how straightforward typewriting work was on the firsts Macs, and how a newbie could go from zero to full speed in less than 5 minutes on MacWrite.
This is very curious: Even today, the computer is more useful as a typewriter than anything else. The uses had varied much, and nowadays professionals use the computer for every nitty-gritty important detail, but still, the first thing that comes to mind to most people when they think “computer” is “writing”. From journals to books to presentations, my perception is that the only thing people can do (on a computer) with no complex instructions is publishing. Want to publish? Open this app and start typing. But you want to render a 3D object? Search for a court lawsuit? Build a repository of data? Out of luck, will take years to learn it well.
Our hardware has evolved, and the purposes we use the computer for have changed too. But the form has stayed the same as 30 years ago. And (not) surprisingly, the interfaces of our apps still resemble the interface of that first MacWrite. Time to change how we interact with computers? I hope so. Science Daily had long ago an article that said “While much about the computer has changed over the last three decades–greater power, faster speeds, more memory… – what has not changed is the user interface.” They offered a gesture interface proposition, and other technologies like multitouch or brain-controlled interfaces are being researched. From my part, I’m ready to test the new paradigm as soon as it’s ready .

This is the full video, this is the link for the graphing calculator storry in text form (10 min instead of 55 ), and go here for a demo of the product at Apple’s launch.

Enjoy, and please let know your testing thoughts in the comments!

Related posts:

1. “Create your testing portfolio” presentation
2. Testing questions and the Laws of eCommerce
3. Yay, another Happy New Testing Year! A decade in review…

Let’s recapitulate my 2008 blogging activity:

• This last year I did not do my self-goal of at least 12 posts, due to the account problem at the beginning of the year.
  From the new posts, one (Testers don’t think like Developers think like Computers) entered my list of ‘classic posts that new team members read’ :) together with A Bug with a Helmet and the “untestable things posts (1, 2, 3)”.
• This also means that, although the blog is kept unpublicized, I’ve got (compulsory) readers now .
• I had at least two very interesting to write BotT: BotT: Excel 2007 has algebra difficulties… and BotT: Linus’ bug, youtube no workee!.
• On the corporate blogs, I started a new series of post about Intel(R) AMT – not as popular as the ASF series, but I got good feedback. Read if you’re interested in the technology I work on.

Hope this next year comes with at least 12 posts, all of them useful.
Gotta start thinking on January’s post…

Related posts:

1. Happy New Testing Year Again! What? Too late?
2. Happy New Testing Year!
3. Yay, another Happy New Testing Year! A decade in review…

Writing the Fitnesse posts turned to be harder than I thought.
I do have a bit of tests ready for the triangle case, but not enough text to make an interesting post. As I’m not using Fitnesse in my day-to-day work, it makes it harder to bring cool insights or to explore on the framework.

But I just discovered someone who not only uses Fitnesse at work, but also writes articles about Fitnesse that give the Fitnese feeling:

Stuart writes at http://working-thought.blogspot.com/.
It seems as if his company has been trying Fitnesse for a while, and they’re over the initial pains. Now they’re on more advanced pains , but with apparently good results.

If you ‘ve already seen the Two Minute Example at the Fitnesse page, then you know what Fitnesse if used for.
Then start reading from “Fitnesse for Dummies” at Stuart’s blog (and laugh at the ilarious GenericFixture.jar error page). There are further posts later, like the very insightful love/hate relationship with Fitnesse.

As for me, the post I liked most was the one about test automations, that has nothing to do with Fitnesse, but explains well many of the web-app testing tasks.

I wish luck to Stuart, and who knows? Maybe I’ll get back at posting about Fitnesse somewhen.

Related posts:

1. FitNesse Series in Testing Thoughts (Part 0)
2. The Triangle problem as a starting point – FitNesse Series (Part 1)

The last BotT (Bug of this time) was long ago, when we talked about testing and the Excel bug.
So now we’ve got a cool one, on which the most notable point is not the bug, is the submitter .

Bug 439858 on Fedora (a Linux distribution) was (supposedly) submitted by Linus Torvalds himself. He started the Linux wave on August 91, and is still posting bugs on March 08 – you gotta admire him.
I am aware that the submitter signing “Linus Torvalds” may not be Linus. It actually doesn’t matter, let’s just pretend it is for sensationalism’s sake (regarding the admiration, you still gotta admire him even if the bug’s not his ).

Let’s try to analyze the bug report, and the bug itself, to see what we can learn:

Linus uses a lot of witty comments. The bug starts as “youtube no workee“, which is a nice way of writing things in a forum or a youtube post, but we should not use such mocking language in our bugs – jokes or irony in bug descriptions can offend developers, who already have a hard timing accepting their code has got errors. Cem Kaner alerts: “it is easy to read a joke or a remark as a flame”.
You should not do this especially if you are a leader, influent people or the inventor of the OS — you are an example, why foment bad practices?

Then there is “fedora 9 not usable for wife“. That’s actually a good point. It is easy for programmers to remember that the average user is not tech-savvy and won’t tolerate complex workarounds. If this is true for common commercial (and Windows) application (where the programmer is interested in the sales to a large audience that expects intuitive software), this can be even more true in the Linux community (where they’ve got an incredible good will and skills, but sometimes prefer to add features over simplicity).
So explaining the Customer Impact in clear language at the beginning of your bug is a very good idea.
Moreover, I don’t think anyone would want to frustrate Mrs. Linus Torvalds. First because she doesn’t deserve it, being a nice person with a child-care background, and second because she’s a black-belt Karate champion .

“Steps to Reproduce:
1. Install current Fedora 9
2. Rick-roll!
3. No profit!”
Now, these steps to reproduce aren’t very useful. Even the title of the bug is more descriptive: “swf mozilla plugin – no youtube“. Everyone laughed at the mention of rick-rolling, but it only added confusion: is this bug related only to the Rick-Roll video? Any other video?
Linus added specific info to clarify: “I didn’t try a lot of videos, but I couldn’t find a single one that actually worked“. This point is actually a very good one, which teaches an important principle on bug reporting: There are points that are “conditions” to the bug (the bug only reproduces with the conditions fulfilled), and there are points that are ”incidental” to the bug (a setup or configuration that was true at the time of the bug but we cannot be sure it is part of a condition). It is very important to note the distinction between these when submitting a bug. And, of course, any setting which is neither incidental nor conditional should in most cases be omitted.

“Some videos just show a light gray background, some give the play buttons etc,
but in the latter case show only a black screen even when the red ball at the
bottom seems to moves along..“. Good! Information that is really related to the bug! Adding the link of videos that show the gray background and videos that show the buttons would be of help here. You (the tester) already visited all of them, why should the programmer search for specific reproductions again?

There’s some more information in there. The information is a sum of statements that help debugging (“youtube videos are supposed to work with swfdec“, “the adobe player won’t install on current rawhide“), statements that define the impact (“wife will kill me if she doesn’t have her videos“) and humorous comments. When you report your next bug, remember to separate these into clear sections: a section to discuss the user impact, and a section for additional debugging info that can be helpful. The humorous section you can leave out .

Le Grande Finale:
““Obi-wan Kenobi, you’re our only hope”“. He actually finalized with a Star Wars quote. Which is very cool if you’re the founder of an open source OS.
They should print this bug as a book. It mixes hackers, nerds, memes, Star Wars and jokes – receipt for a best seller!

—–

Before you post hate comments on how I dare to criticize Linus: I don’t. He’s a hero of mine and accomplished incredible things. I just used his bug for didactic purposes.

Related posts:

1. BotT: Ctrl+Enter on Firefox
2. BotT: Excel 2007 has algebra difficulties…
3. BOtT: Smile, your data is gone!

It is possible that I’ll post here a transcript or some text about this lecture. One option may be to add my comments from the lecture as notes in the powerpoint slides and post the slides here. But I don’t like powerpoint much as it does a poor job of passing information (it takes the written word and limits it into bad format and structure).
Finding the time, I’ll write the lecture in extense text.

In order to keep this post still useful, here goes a short summary of the talk:

Fault Injection: I base everything I say on the book “How to break software” by James Whittaker. The examples were done using Canned HEAT version 3. Canned HEAT is a good tool (with many bugs ) which comes free with the book.

Fuzz: The tools used on the examples were:
GUIFuzz: A custom dumb fuzzer based on the original fuzzer which can be found here.
FileFuzz: from iDefense Labs.
Driver fuzzer: Built with the Peach Fuzzing Framework.

All the tools and frameworks are free for commercial use as well.

Related posts:

1. 3 things I learned at Sigist Conference, day 2
2. 4.5 things I learned at Sigist Conference, day 1
3. 5 things I learned at SIGiST Conference, day 1

When not from a lecture focus, then from a side comment or explanation. Below you’ll find some insights I gained from today’s lecture. When not from a lecture focus, the ideas come from a side comment or explanation:

1. Map everything.
   Vipul’s “I have a dream” talk described an utopic system, in which every single bit of information is not only saved, but mapped and related to relevant bits. In this dream, every single information is mapped, and you can traverse all info and predict taks efforts and chain-reactions.
   As he said, all the technology needed for such a system exists, it just have to be done (in an efficient way).    

   As such, we can start to gain the benefits of the systems by implementing easy thing. Like, for example, Requirements mapping to code. Why is that everybody talks about mapping the test-cases to the requirements, but no one mentions the Requirement to Source Code mapping?
   Such mapping would be very good to:

   • Prevent Gold Plating (number #30 here) (and see a programmer’s rant here);
   • Show the risks (for the application) of changing a requirement (and estimate the change effort);
   • Show the risks (for the requirements) of changing the application (due to a bug fix, for example).

   A Requirement mapping effort is not complete by merely mapping test-cases.

2. 
   Combine testing components.
   If two components are codependent (as in one’s functionality affects the other’s and vice versa), most of the times it is better to think of them as one (although composed) component.
   Whenever you have to do something in one, include the other. It sounds obvious when you speak about testing tasks, like doing a regression and/or a specific test. But it can also be true when you are dividing any ownership and resources. Keep close things close.
3. 
   Combine tests.
   Combine tests into composed scenarios. Run a number of tests – but do them sequencially as part of of a use-case. Instead of testing the find feature, do a combo hit: Open a document, find a text, replace it, save it again under the same name – 4 tests in 1!
   You still got to do all the tests, but the scenario combinations increase incidental complexity ,which usually causes smarter bugs.
4. 
   Software security does not end in the software.
   Business are made of software and data. Be it a collection of different software by different company, or one single solution by one company, the business workflow has a lot of stops and transitions. It is not enough to have a software adequately tested for security, if the workflow (wich is made of people) allows security problems. Authentication, authorization, configuration and more are problems that need to be dealt on the workflow level. This business process needs to be built with a security mindset since the beginning. As Vipul said a day before: “Security is not supposed to be tested, it is supposed to be tested for. And the same is true for performance and usability”.
5. 
   Models always lie.
   I am afraid of models. Since I discovered that many problems with requirements come from people thinking this representation is the real thing (instead of just a reflecting image with imperfections), I am afraid of models. In todays’ Test-Cases from UML lecture by Ofer Prat, he gave a wonderful example about modeling:
   London’s Underground was always mapped in it’s geographical form, it was a model drawn as close to reality as possible. Such a model was complex and almost unusable. Over the time, it evolved in a simpler and easy to follow model, but by then it was not accurate anymore – see this page for a graphical history.  

   So, models (and requirements) are meant to be straightforward directions. As such, they may lack from the accuracy and/or truth that reality presents.  If software is behaving in a bad way, and this bad way fits the requirements, it is time to re-tune the model.

More to come tomorrow!

Related posts:

1. 4.5 things I learned at Sigist Conference, day 1
2. 3 things I learned at Sigist Conference, day 2
3. SIGiST Conference, my lecture on Fuzz and Fault Injection

For testers, thinking like developers is evil. If you think like a programmer, you’ll start excusing the software and will forgive the system’s bugs.

I am reading the very cool book “Dreaming in Code” by Scott Rosenberg, and I just understood a little bit more on why’s so bad sharing the developers mindset.

Scott starts its book with Chapter 0, a common practice in computer literature. When enumerating the reasons, he explains that “Programmers count from zero, not from one. (…) Why do programmers count from zero? Because computers count from zero!” and that developers are taught to think like computers.

This is a very interesting and basic concept.
Dr. Parnas in his “Strategic Defense System” paper (a real favorite of mine ) has a whole chapter called Why conventional software development does not produce reliable programs, in which he clarifies this problem — developers thinking (or trying to think) like a computer:

As we continue in our attempt to “think like a computer”, the amount we have to remember grows and grows. The simple rules defining how we got to certain points in a program become more complex as we branch there from other points. (…) Because one needs to remember so much (…), new problems are created when old problems are corrected.

The beauty in all this is that Testers should not exercise the programming rationale (when analyzing a software) because they’ll be mimicking the rationale of the very same software they are testing. They are limiting themselves to what the program knows about itself, instead of expanding their logic to people-centered matters — i.e. how the software will be used and what it is really meant to do.

Back to “Dreaming in Code”, Scott has an almost poetical way of writing the concepts developed by Dr. Parnas.
Because computers think from zero, “programming include little offsets, translations of ‘+1′ or ‘-1′”, he says. So:

There’s a space between (…) the way the machine counts and thinks and the way we count and think. When you search for explanations for software’s bugs and delay and stubborn resistance to human desires, that space is where you’ll find them.

Poetry.

Related posts:

1. On certified testers and being certifiable, and on non native english speakers :)
2. Happy New Testing Year! Yet again!

I am fond of quoting.
Not sure why, but I like to quote. I guess it gives some legitimating to what I am saying.

So, the quote collection is available at this address: http://testing.gershon.info/quote-collection/. It will grow slowly, please check it regularly.

Related posts:

1. All might be free!