Why we need realism in testing

I was sitting on the train. As the train attendant passed a woman right behind me called out "Excuse me, is that ticket reader from 2013?".

- I'm not sure, why do you ask?, responded the train attendant
- Well, this morning I noticed the train attendant's ticket reader seemed very bulky so I called the railway company and they rudely told me it wasn't a problem anymore since they had upgraded the model in 2013. So I was wondering if that's the new or old model.
- Well, I don't think there's a newer model but honestly I don't know, what I do know is this one is at least not built for me.

They started talking about various problems the train attendant and her colleagues had experienced. Of those these seemed to be the biggest ones:

• The device was made for much bigger hands than most train attendants'.
• The device was way too heavy.
• The device was tough to get a grip around and maneuver, probably even if you had "right sized hands".

The most alarming result was the amount of people on sick leave had grown notably since the current ticket reader was introduced, according to the train attendant, and the main reason was reportedly over-stretched arms.

The biggest issue with the current ticket reader seemed to be the humongous battery on the back. At this point I had curiously joined the conversation and commented that the battery's size seemed unreasonable knowing even several years old, cheap, small smartphones had batteries lasting for days doing much more complex tasks. The response to that was quite telling:

- The one we had before was much easier to carry but the battery didn't last long enough.

At this point I suspected two things:

1. Someone was informed: "We need to improve the battery on our ticket readers". Armed with that information, this someone wrote a spec saying "same functionality but better battery" and the humongous battery was the quick fix, a seemingly identical device but with more power. Implicit requirements such as "it should weight a maximum of ... grams" or "it should fit the hands of our train attendants" was most likely not considered.
    
2. Whoever tested this didn't test it under realistic conditions such as for many hours, trying to reach over other passengers, walking long distances carrying it and so forth. Another possibility is the casing and hardware was "just an ordered standard third-party device" (probably not built for this specific purpose) and all focus was on the software. A third option would be no testing was done since it was "just a change of battery, which will not change anything to the device, only improve it".

I'm still very curious if any acceptance testing was done by the railway company. If so, under which conditions, for how long and by who, actual train attendants? If not, why not? (I can list many plausible answers but still curious)

Continuing the conversation the woman initiating it started to list a few simple ideas, just from the top of her head, that could had eased/solved the problem. My personal favorite was: "Why don't design it so that the battery pack is in the belt". That seemed like a minimum effort, low cost solution to both the old and new problem. Knowing the current state though I suspect the manufacturer would probably had delivered a too short power cord.

I thanked this wonderful women who asked the question, as well as the train attendant. I got a valuable reminder of how important "dogfooding" and testing a product under realistic conditions are, as well as a good reminder of how much you can learn from just being curious.

My message with this story: Yes, it's often inconvenient to figure out and set up realistic conditions when testing but failing to do so can be catastrophic. As the train attendant pointed out: "Some days when my arm is aching I just look at the tickets and assume they are valid".

Update

Here's an image of the ticket reader. Notice the battery (bottom left) and how it obstructs a "natural grip" around the device. This train attendant also had notably larger hands than the train attendant mentioned in the original post (two different train attendants) to further emphasize the problem.

Finally, when I snapped this picture the train attendant immediately responded: "Are you gonna get us a better ticket reader?". Pretty telling comment.

Using STWC to explain testing

When I explained STWC (Software Testing World Cup) to a friend recently, I realized that story actually encapsulated testing in a pretty good and concise way:

Well, the general setup was pretty easy: Each team had 3 hours to test a product, revealed just before the competition started. Throughout the whole competition we had people in charge of the product available on a live video channel so that we could ask questions and get valuable information from them as we were testing. Finally, before the three hours were up, we had to file all our bugs and send in a test report containing stuff like critical bugs, suggestions for improvements, what we had covered and so on. The winner was then chosen based on value from bug reports, the test report and how well we communicated with stakeholders. And value of bug reports was not our bug count but rather severity of the bugs, if the programmers managed to understand and reproduce the bugs based on our descriptions and so on.

Our team started out by just clicking around in the application, trying understand how it worked. In parallel we watched an introduction given by the product owner where he shared what he wanted us to focus our testing on etc. After doing this for a few minutes we stopped to share what we had learned, split up the work and briefly talked test strategy. When done, we started exploring, or well test, the product, stopping every now and then for debriefs to ensure everyone had a good flow. At the end we finalized the report, made some improvements to our bug reports and well, that was it.

To just briefly explain concepts like debriefs, touring, the importance of questions and communication, common artifacts/output, value from bug reports and how the process of testing actually (can) look like, is hard! Being able to put them into a simple context like this, seems to me like an efficient way and if something is still unclear I can just zoom in on that specific detail, for instance:

Hang on, what do you mean with test strategy?

Well, in the intro the product owner shared what parts he wanted us to focus on; the email functionality, screen resolutions and the possibility to "steal" other user's customers. The strategy in this case was: Is this enough or should we focus on something else? Any general ideas on how to test these things? Who does what? How do we use our time, like how often to debrief and when to start with the test report? Also, as part of the strategy, we set some roles to make sure nothing was forgotten; so Agnetha was responsible for our communication with stakeholders, I was responsible for the test report and Sofia was responsible for the quality of our bug reports. All and all a plan on what to do, roughly how to do it and who should do what.

My message with this blog post is: If you are to explain something complex, it often helps to find a simple context to apply it on. The story is like a picture, even when brief it can tell a thousand words worth of complex, abstract details.

By the way: Gerald Weinberg is a master when it comes to explaining complex concepts using simple stories. I do recommend his books if you wanna see this in full action (guess that's kind of an unnecessary recommendation as I seem to be the last tester to have discovered his brilliance)