Back on December, Pradeep Soundararajan set a challenge up in his blog.
He built an application with the description below:

This was an interesting exercise.
What took my interest in this one?
– First of all, I liked Pradeep’s post about learning to code.
Learning to code is an important skill for a tester. I can relate to it because I am trying to learn PowerShell. I program in some languages (C#, C++, PHP, VbScript…) and have a programming background (after my Computer Systems Engineer degree, the ‘natural’ path was to code, and I worked as a programmer for many years before finding Software Testing), but PowerShell has different approach and paradigms and learning it will definitely be fun.
– Second, I had an impromptu vacation day, and I thought it would be educational to use it for a peer testing puzzle. I thought I could win this one quickly (I was mistaken, it took me long). 🙂

So I downloaded Pradeep’s application, and got to work!
Rather than just trying to solve the puzzle, I looked at it as if the mission was:
“This is commercial 'roulette' style game. Stakeholders want to know it the game logic can be broken/learnt, which could mean a substantial loss of money when people start winning every time.”

Testing Hint: Cem Kaner says “Testing is an empirical, technical investigation of a product, done on behalf of stakeholders, with the intention of revealing quality-related information of the kind that they seek“. So it is important to set straight what is the information to seek. Which kind of bugs to look for?

In this blog I describe my attemp to answer this question in an exploratory way.

Now, Stop the press!
You may want to go to Pradeeps post, download the file, and try it for yourself before reading my solution. This post will be waiting right here when you’re done. 🙂

…

Here is the report of what I did, the line of thinking during my experiment, and the conclusion:

  • 1. First, I ordered my environment to allow efficient and organized work:
    1. Increased the Screen Buffer Size of the Command line I was using to 500 (it turned out that I would do well on less, even half, than that (see point 2.b). But it didn’t hurt).
    2. Renamed the executable to be shorter and without number version: ren fnemo_1.7.6.exe fnemo.exe
    3. Changed the prompt to something shorter, cleaner, and that gave me context about this task against the other Cmd lines windows open: prompt $Cfnemo$F$S$G

Testing Hint: Organizing your testing environment before you start will likely help you during your tests. After you’re in the heat of fight, it will be harder to stop and get organized.

  • I tried some basic executions of the program, to grasp the feeling of what it does and how hard it is to find Nemo. This also taught me what are the messages returned by the application when Nemo is found, and when Nemo isn’t found. They’re “Found Nemo this time!, Nemo Gill Bubbles SharkTooth Flow Phamplet Stinger” and “Ah! bad luck, didnt find Nemo this time!” respectively.
    1. I also learnt that Pradeep used Perl and Perl2Exe to do the app.
    2. Additionally, the application clears the screen at every execution, so the big command line buffer is not so helpful.

Testing Hint: Planning or scheduling your testing before you experiment a bit with the software can be done. But you risk being completely wrong in your assumptions about the app. In our case, there were no big surprises, but there could be.

  • Sarmila commented on the Challenge page about disassembling the executable. Although by learning all the assembly one can learn about the rules that move the fishes around, it would be extremely difficult and overkill. But this made me think on how Perl2Exe works… Maybe it just wraps the Perl interpreter and the Perl script together? If so, what if the script is stored internally in clear? I tried to open the app in an Hex editor, but no, the Perl script isn’t in clear text there, it is obfuscated.
    • I also tried the useful Strings tool by SysInternals. My intention was to see if, maybe, the array of fishes could be seen at the app code, in order to learn the initial order of fishes. It couldn’t.
    • That ended my cheating session :), from here on I used only a black box functional approach.
      • Well, it is not really cheating because there were no rules against that. If you do a full reverse engineer of the code, the answer is legit too. But I preferred the straightforward exploratory approach: learning from the software functionality as I use it, and use this learning to redirect my next steps.
  • .
  • By now, I knew that Nemo changed places between plays.
    1. Nemo also change places between invalid parameters (which may or not be a bug).
    2. Moreover, the other fishes change places too, even their placement related to Nemos’ placement changes.
    3. So I tried to see if Nemo will return to the same place in any consistent way or number of times. Running that is easy, you just enter “3”, “3”, “3”… and count where Nemo was found and where he wasn’t.
      • I discovered there is regularity (for “3”, its every 2, 3, 17, 5, 10, 5…), but the regularity was irregular enough, and also changed for different positions. This was likely a consequence of the real logic, rather than the logic itself. This correlates to the movement of fishes, but does not cause it.

Testing Hint: Take note of your steps while testing. Using the Session Tester tool is a great idea. For this session, I used MS OneNote because it supports rich text, and I could use tables.
Is that like saying that rich texts helps rich tests? :).

  • Another detail that helped me in the game was knowing the developer and the purpose of the challenge. This puts a lot of content in the context. Pradeep is a rather playful 🙂 and would put some tricks into it.
    1. First I tried to see if Nemo could sometimes be in any placement bigger than 7 (there are 7 fishes, places 1 to 7, but there were no rules as to where the fishes were limited to be).
    2. I tried in a toss of up to 220 attempts, and Nemo wasn’t in place 8 even once. So I let this idea on StandBy for now.
    3. Then I tried… what if the “Minimum attempts” input at the beginning of the app affected where Nemo appears?
      • 10 is the default for the “Minimum Attempts“. Nemo is at place 1 at the beginning there.
      • 11 attempts chosen: Nemo is at place 1 at the beginning too. Maybe it does not affect?
      • 12 attempts chosen: Nemo is at place 4! Bingo!
      • 13, 14, 15, 16 showed that the original places were cycling at ‘3’ intervals. That means that when trying to find Nemo on the default 10 attempts, it will be in the same placements as 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46, 49, 52…

Testing Hint: let the test and what you learn during the testing guide you. New information you find should set you in a new direction.

  • You’re seeing that “220 attempts” above, and thinking if I really entered each number manually.
    1. By this time, I decided that I was in need of some automation to insert inputs to the application.
    2. The original app had no apparent automation capabilities, so I decided to use the internal input redirection of the Command line:
      • By using the “<” operator, I could redirect an input file “input.txt” to the app using “fnemo.exe < input.txt“
      • I extended that to “fnemo.exe < input.txt > output.txt“, which gives us with a complete dump of results that can be searched with Notepad.
    3. We had an easy automation now, and this can help our sapient testing. I was exploring, and using automation to help me try different things help me explore my possibilities and get a bigger picture of the situation when needed.

Testing Hint: Don’t be afraid of automation. It can help you talk with the software in different ways and interact with it differently. It helps a lot with data generation and logging, too. Another hint: Never let the automation take control of the testing. It is there to help you, not to drive the effort.

  • With the knowledge and tool I had so far, I did a lot of trials in order to find where Nemo is at each iteration. This could give me a hint if he moved in any regular or consistent manner.
    • So I made a big file of winning moves (a lot of trial and error went in it).
  • .
  • It was now time to model the results:
    • First attempts at modeling the results:
    • I got this: Nemo’s position: 1, 3, 5, 3, 2, 7, 3, 4, 7, 4, 4, 1, 5, 5, 2, 5, 6…
      • Nemo was certainly jumping around without any clear regularity, even when I did it twice the times there was no repetition.
  • On the Second modeling attempt, I tried to see if a more visual model could help:
  • 1 2 3 4 5 6 7
    X            
              X  
            X    
        X        
              X  
                X
  • Even in a table with many more results, it made no sense.
  • .
  • I changed my approach. Looking only at Nemo was not giving enough insight. Plus, I knew the other fishes moved along, so there might be a hint in their placement.
    • I got this table:
    • Nemo Gill Bubbles SharkTooth Flow Phamplet Stinger
      Stinger SharkTooth Nemo Phamplet Gill Bubbles Flow
      SharkTooth Flow Phamplet Stinger Nemo Gill Bubbles
      Stinger SharkTooth Nemo Phamplet Gill Bubbles Flow
      Stinger Nemo Gill Bubbles SharkTooth Flow Phamplet
      Phamplet Gill Bubbles Flow Stinger SharkTooth Nemo
      Phamplet Stinger Nemo Gill Bubbles SharkTooth Flow
      Flow Stinger SharkTooth Nemo Phamplet Gill Bubbles
      Gill Bubbles SharkTooth Flow Phamplet Stinger Nemo
      Flow Stinger SharkTooth Nemo Phamplet Gill Bubbles
      Flow Phamplet Stinger Nemo Gill Bubbles SharkTooth
      Nemo Phamplet Gill Bubbles Flow Stinger SharkTooth
      SharkTooth Flow Phamplet Stinger Nemo Gill Bubbles
    • That didn’t look good. Nemo moved spuriously, and the other fishes either jumped aimlessly or stayed at their place. Nothing made much sense.

    • Testing Hint: When you look at a problem or thing, different models will give you different view and different details. Multiple viewpoints help — keep changing the viewpoint until you learn enough.

    One interesting thing that happened by this time, is that in my manual trials, I was finding Nemo pretty easily.
    Intuitively, I was seeing a pattern on the game, a pattern that was not conscious or explicit. It was as if my mind had already learned the rules without noticing them.
    It reminded me of what Douglas Adams wrote in Dirk Gently’s Holistic Detective Agency:
    “A ball flying through the air is responding to the force and direction with which it was thrown, the action of gravity, the friction of the air which it must expend its energy on overcoming, the turbulence of the air around its surface, and the rate and direction of the ball's spin.
    And yet, someone who might have difficulty consciously trying to work out what 3 x 4 x 5 comes to would have no trouble in doing differential calculus and a whole host of related calculations so astoundingly fast that they can actually catch a flying ball.
    People who call this instinct are merely giving the phenomenon a name, not explaining anything.    ”

    • I still thought that looking at the other fishes will bring the breakthrough needed.
      1. In order to learn the movement of the fishes, I did a table of their placement in relation to Nemo at every run.
        1. i. This is what I got:
        2. Nemo Gill Bubbles SharkTooth Flow Phamplet Stinger
          Nemo Phamplet Gill Bubbles Flow Stinger SharkTooth
          Nemo Gill Bubbles SharkTooth Flow Phamplet Stinger
          Nemo Phamplet Gill Bubbles Flow Stinger SharkTooth
          Nemo Gill Bubbles SharkTooth Flow Phamplet Stinger
          Nemo Phamplet Gill Bubbles Flow Stinger SharkTooth
          Nemo Gill Bubbles SharkTooth Flow Phamplet Stinger
        3. See anything interesting? The rows repeat themselves alternatingly! So the fishes were not really moving around, they were following Nemo!
          1. My take is that Pradeep has two different arrays: One for Odd rows, the other for Even rows.
            1. This may not be true, but it doesn’t matter for us. When testing, we not always know what exactly the programmer wrote in the code, but we infer a mental model. If it suits the needs, it is a good model even when not the real thing.
              1. Many times I think these fake mental models are even better than the real thing. It’s the best way for them to act as an intuitive Oracle when analyzing an application.

    Testing hint: When you learn and infer about software, you make assumptions and a mental model of what the software does inside itself. This model does not need to be an exact depiction of the code underneath the app. As long as it answers your questions, and you change this model constantly as you learn, it is good.

    • From this part on, I started to analyze Even rows and Odd rows separately.
      • Bingo! Seeing the pattern was easy now.
      • Odd rows follow this cyclic sequence: +4, +4, +1
      • Even rows follow this cyclic sequence: +0, +4, +4
    • .
    • But what with the differences when using different “Minimum Attempts“?
      • A quick look at the tables I had gathered showed that they follow the same rules, but start at different places.
    • .
    • At the end of this trials, these are the set of rules that allow me to know where’s Nemo at any given time:
    • a. Where does Nemo start?
      1. “Minimum Attempts” of 10, 13, 16, 19…
      2.   Place 1 Place 2 Place 3 Place 4 Place 5 Place 6 Place 7
        First Line Nemo Gill Bubbles SharkTooth Flow Phamplet Stinger
        Second Line Stinger SharkTooth Nemo Phamplet Gill Bubbles Flow
      1. “Minimum Attempts” of 11, 14, 17, 20…
      2.   Place 1 Place 2 Place 3 Place 4 Place 5 Place 6 Place 7
        First Line Nemo Gill Bubbles SharkTooth Flow Phamplet Stinger
        Second Line Gill Bubbles Flow Stinger SharkTooth Nemo Phamplet
      3. “Minimum Attempts” of 12, 15, 18, 21….
      4.   Place 1 Place 2 Place 3 Place 4 Place 5 Place 6 Place 7
        First Line Flow Phamplet Stinger Nemo Gill Bubbles SharkTooth
        Second Line Stinger SharkTooth Nemo Phamplet Gill Bubbles Flow
    • b. Where does Nemo go?
      1. i. “Minimum Attempts” of 10, 13, 16, 19… Follow this cyclic rule, starting at step 1.
      2. ii. “Minimum Attempts” of 11, 14, 17, 20… Follow this cyclic rule, starting at step 2.
      3. iii. “Minimum Attempts” of 12, 15, 18, 21… Follow this cyclic rule, starting at step 3.
      4. Odd Rows: +4 | +4 | +1
      5. Even Rows: +0 | +4 | +4
    • .
    • How do I know it is right?
      • Proof! 🙂 I built another automation aid: An Excel spreadsheet that can tell you the position of nemo in all trials given an initial “Minimum Attempt“. It works!
      • You can download the spreadsheet to try it here. Choose any number of initial attempts, it will tell you all Nemo’s position accurately.
      • Is this the real logic behind Pradeep’s challenge? It may not be. It is probable that pradeep did something else, much simpler, and that only translates into this. However, this answers the mission of predicting Nemo’s moves.
        • I am sure that analyzing the findings above one can discover what was really that Pradeep did. I’ll just ask Pradeep on a mail for a suggestion. Testing hint: You can always reach out to the developer and ask him about his software and/or code! 🙂

    Testing hint: Again, your mental model doesn’t always needs to be exact. As long as it works and answers the questions you have. Keep in mind that this mental model should be flexible and evolve with your knowledge.

    • Conclusion sent to the ‘roulette’ stakeholders:
      Until a higher amount of randomness is added to the game, it is too risky to release the game as is now. A malicious player can find the logic of the game after some hours of work, and a pair of smart malicious players in even less time.

    Thanks Pradeep, for the nice challenge.
    It was a very nice experiment, and I learnt a lot. The best part, for me, was organizing my thoughts in this report, and I hope you like it too.