Archive for category Test Insight

2020 – the year that tested us. Happy New Testing Year!

This weekend I recalled that up to 10 years ago I used to write New Year posts. There was little to no action in the past 10 years on this blog, but another decade – and 2021 – deserves a “hello again” post!

2020 was by all means special. You may think it was bad, it was good, neutral – but you will agree it was special. 2020 made us rethink the way we live and relate, and the way we study and work.

A month ago, I presented at TestFlix, an online video conference to which The Test Tribe was kind enough to invite me. The conference was scheduled on Shabbath, so I risked missing the conference (and my own talk)… luckily it continued after the Shabbath was over and I got to attend a couple hours of it – including the weird experience of listening to myself.
As far as online conferences went, it was pretty fun, with a small expo and different chat rooms. You look around your room and there’s no one other than your coffee mug, and yet you’re in touch with people around the globe.
EuroSTAR had a similar conference recently, for which you can see the stunning (stunning, I say! Content is top-notch, but just look at that graphic design!) program here.

Everything is online now! The joys of remote working from home sitting in a small room from dawn to dusk!
We’re a household of 10, with 6-7 of the kids studying over Zoom/Teams/Classroom, encouraging a tired and sweating D-Link router to make it through one more Mb – You can do it, little D-Link, you’re a big Cisco!

Testing in the time of Corona… Covid-19 gave us opportunities for many interesting conversations on Testing. From the discussion on whether testing more causes more bugs (after all, looking at your clock makes it tick slower, right?), to the realization that testing is the most important thing ever, all the way through rethinking your work routine.

On a practical and personal note, one of the things that working remotely demonstrated is, well, that testing remotely is possible in embedded and invisible systems. That statement is not as trivial as it seems: You see, working with embedded software means your product does not fit in the contents of a USB-Key, or in a GitHub repo. The embedded product (software) is located in a big board with a specific processor, usually connected to a bigger power supply and having all sort of flimsy wires tied to weirdly named devices (JTAG? Aardvark? Saleae? Lauterbach? Kvaser? Who’s naming these things??) that you connect to your computer when you test. It looks so scary, that once we needed one abroad for a business trip and we mailed it instead of taking it on the plane because it looked so much like a bomb 🙂 .


The quickest way to test this software is to be relatively close to it so you can turn it off and on, look at the LEDs, disconnect cables, reconnect cables, plug in other analyzers, look at it angrily to see if it finally obeys, nudge it in despair, and so on. Can we do this with no hands?

In 2015 I presented at EuroSTAR a talk called ‘Demystifying Embedded Software‘ (it was frontal, not online, as the internet didn’t exist back then). We discussed the many similarities between a desktop/mobile application and embedded software running on a specialized processor and an OS, and also the differences (which include optimizations of storage and power, runtime accessibility, unusual interfaces and fuzzy IO, and limitations in the software’s lifecycle).

I debated with myself briefly on whether to include the fact that embedded software needs you near. I was testing sensor subsystems at the time, so tests included shaking, rotating, and flashing lights at our product – certainly much harder to do from a distance. However, we were able to work from a distance – as we had robots to rotate, lightboxes to illuminate, coils to reorient, and all sorts of fancy toys. The conclusion was clear: It is possible to test a system like that remotely (so this didn’t make the slides), it is much harder too.

Jump forward 5 years and we must test things remotely 24/6. I am not testing sensors anymore, but the above still holds true: While I can test remotely, it requires fancy solutions as a medium.

While everyone will quickly tell you that that makes the work slower, I now discovered slowness is not the major drawback. The problem is that mediation means talking with the product through a translator. And if in person you can get a feeling of the software’s mood (is it slowing down? is it in trouble? did it restart surreptitiously?), having an SSH commanding an automatic query makes it much harder to notice the software’s reactions (is the network slow? is the serial interface in trouble? did the JTAG interface flicker?).

If on one hand I have now a full remote setup that allows me to control the product, on the other hand I still haven’t overcome the alienation that distance brings: I can’t feel the product due to the noise caused by the input and output toolchains. Monitoring the tools in use is becoming as important as monitoring the product itself.

“Monitoring the tools in use is becoming as important as monitoring the product itself.”

— Previous Paragraph

My lesson? When working remotely, we have to be double alert.

It is an interesting place to be, and it will hopefully retranslate into sharper instincts once we come back to normal.

Tags: ,

Train Trainings, or missing the train for fun and profit

Yes, please Sir, you may take your train in platform 1 or 2 to Amsterdam Central.
These were the words that gave me a breath of confidence inside the Amsterdam Airport train station. Amidst many signs in the undecipherable Dutch language (insert jokes about Hebrew letters here), numbers are easy and I know these from tender age. Off I go to platform 1 (where a computer screen did indeed say “Amsterdam Central” in large fonts) hopping in the first train that arrives.

After a couple stations in which the small computer screen in the wagon did not show anything resembling “Amsterdam Central”, I asked the young boy sharing the bench with me whether I’m on the right direction. Turns out I was not, ha! Had to have waited for a later train in that now far-away platform 1.
In the last moments before the train left with me for another city altogether (Utrecht), I got off the train and a station operator helped me to a platform that had a direct train to Maastricht (for EuroSTAR 2015!). And I felt back in control.

Ok, not a remarkable story. Why am I breaking a 4 years blog-hiatus with such inanities (other than to humour a good friend)?
It’s because I’ll try to tie it with testing and learning. Let’s try it together:

Meanwhile I texted my Wife, explained that I’d lost time by getting in the wrong train. “OTOH,” she asked, “if you got on an earlier train and then did an exchange, you may have saved time. Right?
And that’s the point: I don’t know.

I learned a few things about Netherlands trains during the 20 minutes this happened: that platforms are used for many different lines and directions, that even though the main sign for a platform points to one destination there are smaller trains that also stop there before the big ones, that even though we are told to change trains in a station we can exchange in others… but I still didn’t know whether what I did was a delay or a lucky win. Even though I am empirically experiencing the trains first hand, I’ll have to wait to meet a friend to show me the train rails and timetables before I can start to understand more of what had happened. Either that, or opening the info by myself to make sense of it, or alternatively spending a couple days doing the trajectory between the stations taking notes.

Sitting in the train right now, it felt similar to what we feel testing software. We act, the application reacts, we do one thing and the software replies with another. Sometimes we go in a wrong direction, take notes of the new learnings and then go back. And, after a while testing, a question pops: We’ve been learning all day long, now this peculiar behavior — is it a bug or not? Is it a problem or perhaps a lucky feature?
And often we don’t know. Only after we admit that our learning is not complete, we can look for a friend to help us with the answer, or identify the materials needed to fill the knowledge gap, or decide on the next empirical steps required to elucidate the question… or at least get closer to an answer or recommendation.

Software is tricky (it’s invisible, it’s intangible) and it is infinite. Learning software is infinite as well. If we fall in the trap of feeling in control, we’ll not notice when big questions are yet unanswered.

Well, I should post this online, before I miss my station.

The Big Exploratory Testing Rolling Strategy Dice

Marlon Brando said in a role he played: “People win with loaded dice“…
This is the exercise we’re building today, loaded dices – for the win!

…I get by with a little help from my friends. One of the friends at work who pushes me into all sort of good experiences is Issi Hazan. At this example, he invited me to join him in a testing class for the Tech-Career project (a social wellbeing program in Israel that works with immigrants from Ethiopia to form them and prepare them for a job in the High-Tech industry. They were mentioned at this old post about building a portfolio, and at work we volunteer by giving them coding and testing classes).

Issi’s aim is to teach the group strategies for Exploratory Testing.
But how to do that in a single short out-of-the blue session? How do you convey the strategy, heuristics, coverage models, quality criteria in a practical easy-to-get form? How to make it dynamic and not-boring?

In this post we want to run through you this idea for an aiding tool for the instruction — and the execution too — of tests of exploratory nature. Read the rest of this entry »

Tags: , , , , , , , ,

Lightning Talks night with EuroStar 2010 (the Rebel Alliance, a conference after the conference with beer)

I am back from EuroStar.
I’ve got lots to write about EuroStar: the people, the lectures, the Test Lab, the venue. However, if my mails are any indication, the thing people are waiting most to read about is the “Rebel Alliance” night.

This night, which went by many names (“Rebel Alliance”, “Oprørsalliancen”, “Danish Alliance”), was an informal meeting of friends. As we did at StarEast, it was “a mini conference after the conference, with beer”. We had there very special people and some famous names from the software testing world, and we spent the night talking testing, debating testing, listening to lightning talks on testing, playing games and debating some more. I am a boring nerd, so for me it was the best party ever.
The content and energy were fantastic, it was a remarkable evening. In this post I’ve collected some pics, videos and links.

Who was there? I know I’ll miss a name, but here goes a list (random order):
Jesper L Ottosen | Joris Meerts | Dorothy Graham | James Lyndsay
Bart Knaack | Zeger Van Hese | Martin Jansson | Henrik Andersson
Michael Bolton | Andy Glover | John Stevenson | Rob Lambert
Carsten Feilberg | Ajay Balamurugadas | Markus Gaertner | Henrik Emilsson
Julian Harty | Rob Sabourin | Rikard Edgren | Shmuel Gershon
Lynn McKee | Rob Lugton

Read the rest of this entry »

Tags: , , , , , , , , , , ,

Alternative Paths for Self-Education in Software Testing by Markus Gaertner (EuroStar talk)

Markus Gartner is very skilled in speed-notes. He’s been around EuroStar speed-noting all the session’s he’s been into. He calls it live blogging, but I use the term Speed-Note because, well, I think I coined it ;).
He was live-blogging all the talks, but… who is live-blogging Markus’s talk? Who watches the watchmen? Well, I took the challenge. What you see below are my notes from his ~40 minutes talk.

Alternative Paths for Self-Education in Software Testing by Markus Gaertner

Markus’s main motivation for organizing this talk was noticing an economic downtime a few years back, where conventional commercial education was costly to follow and pay for by companies or individuals. But even in this situation, if you’re in a position where you want to learn and educate yourself, there are other ways…
Markus’s first conventional/official training course in Software Testing was in 2007, 1 year after starting his work as a tester. By that time he already had got responsibility leading people and projects, and he attributes this to spending that year learning about testing from books and people, thinking “how can I advance in this?”

Who is responsible for your education?
One can assign the responsibility over her education on the employer, on teachers, certifications, family… Markus main message on the lecture is that YOU are responsible for your own education. (editor’s note: I agree. In fact, I believe it can’t work any other way).

Shino (as he’s called informally by school friends) presented some options to self-education:

  • Feedback.
    • By writing a personal journal one can assess how his thoughts change over time — that’s self-feedback. By writing to a blog or magazine you receive a different feedback, which is external feedback by other people reading your stuff. For even more direct feedback, you can write on mailing lists or present at conferences. And there’s also an environments built specifically for getting quick and easy feedback, and it’s called, guess what, Social Media. On Twitter, on LinkedIn, the Software Testing Club or Weekend Testing, you can get immediate feedback to questions and affirmations (editor’s note: some of the links point to my own profiles).
  • Learn to Program
    • Programming can be a very useful skill for a tester.
      You can try to learn some scripting languages to help you during your tests, or study design patterns to understand more about software construction. Learning about programming techniques and technologies or pairing with a programmer can give you a great understanding of what goes on in the code.

There are many more options to learn and advance. Markus divided them by two:

  • *HYPOTHESIS* – Left Side of the Brain
    • Books
      • There are many books you can read, but Markus mentioned specifically three.
      • Testing Computer Software – Cem Kaner
        • Gives a very good understanding of software testing and it structure. Markus heard rumors that a 3rd edition might be on the way. (editor’s note: Can it be? Remember, you heard the scoop here first! 🙂 )
      • A Practitioner’s Guide to Software Test Design – Lee Copeland
        • This book structures techniques of software technique, in a very short form. This is Markus preferred and recommended book. (editor’s note: I like this book a lot, thanks Justin for the great gift. But the book I like most is Testing Comp Software by Kaner (and thanks Hayim for this gift!))
      • Lessons Learned in Software Testing – Kaner/Bach/Pettichord
        • An easy reading book with a multitude of different aspects and ideas about testing in all its aspects.
      • Secrets of a Buccaneer-Scholar
        • This book by James Bach includes heuristics for learning (like the SACKED SCOWS, Long Leash Heuristic, Obsess and Forget…) that can be used in most contexts.
    • RST
      • It’s a commercial course by James Bach and Michael Bolton. You have to take the course frontally in a class room. But the slides are available online (!) at and they include models, heuristics and other valuable points from the course.
    • BBST
      • Developed by James Bach and Cem Kaner, this course is an in-depth view of software techniques and our work as testers (it covers Bug Advocacy and Testing Techniques). One of the ways to take the course is by looking at the videos at the BBST website, the other is by taking the course with the AST association, which include — in addition to that material — collaboration and feedback with peers and instructors.
  • *SYNTHESIS* – Right Side of the Brain
    • Testing Challenges
      • Solving challenges as a game may condition you to solve challenges in real life. Markus reminded us that James Bach is willing to share testing challenges over Skype to anyone who contacts him ( Eusebiu Blindu, for example, has challenges on his website as well (link, link, link).
    • Dojos
      • On coding dojos, programmers pair and try to solve one specific coding problem. They include collaboration, a safe environment and a deliberate practice. Markus cloned this idea over to testing, and in Testing Dojos testers solve a problem (test this, evaluate tools, learn new approaches…) in pairs or individually to sharpen their skills.
    • Weekend Testing
      • A weekend testing session is an online ~2 hours session of testing where people work on their own or pairing. They are like dojos in matter of mission, and include a multi-person discussion over Skype on the mission and the results. (editor’s note: It’s certainly a fun way to spend the weekend! I was there and recommend).
    • Miagi-Do
      • This ‘school’ for software testing was founded by Matt Heusser, is non-commercial and zero-profit, and aims to improve testing skills through practice. By proving skills when solving testing challenges you earn colored belts. Markus is a Black-Belt Miagi-Do tester and can be contacted for more information or for initiation. (editor’s note: While is it true that there isn’t much you can do with the belts :), there’s much you can do with the skills.)

Summarizing his talk, Markus says that there are different styles for learning, and you are responsible to discover what your own style is and how you will advance on it. Markus recommends, in any case, trying as many types of learning as you can.

Tags: , , , , , , , ,

How do you evaluate testers? A question from StackExchange

I frequent many of online forums. It’s a great way to grow and learn, as the discussions are fantastic opportunities to challenge us with questions we don’t face on our day to day work. I also enjoy the questions that I do encounter in my day to day, because when answering these (or reading answers) for a context different from mine, I discover new ways of thinking about matters I do out of routine.

In English, an interesting site is the Testing.StackExchange” Questions&Answers site. Unfortunately the site is less active than it could be, but there are good questions and very good answers in the list. Names like Alan Page, Michael Bolton and BJ Rollison answer questions there, so it’s a good place for you to ask yours.

This week an interesting question popped at the Testing.StackExchange forum: How can you evaluate the performance of a tester? How can you compare testers’ efficiency?
This is a question that entertains me once a year, when my company does employee evaluations, and my trial at answering it is in this post.
Can you suggest other points of view? What else you recommend to consider? How do you evaluate testers?

Read the rest of this entry »

Tags: , , , , , , , , , , ,

The effect of time. And Shana Tova UMetuka (Happy and Sweet Year)

שנה הלכה שנה באה
אני כפי ארימה.
שנה טובה לך, אבא,
שנה טובה לך, אמא.(link)
“For auld lang syne, my dear,
for auld lang syne,
we’ll take a cup of kindness yet,
for auld lang syne. (link)”


Tomorrow we celebrate in Israel the New Year in the Hebrew calendar. Yes, 4 months before December ;), and guess what… we are now celebrating year 5771! The difference between the years counted in Gregorian calendar and the Hebrew calendar is ~3760 years.

New years, calendars and dates are extremely important for Software Testing.

Let me tell you a true story about value, about a bug. It was not a bug I found, but in fact a bug I had created. Read the rest of this entry »