Why we need realism in testing

I was sitting on the train. As the train attendant passed a woman right behind me called out "Excuse me, is that ticket reader from 2013?".

- I'm not sure, why do you ask?, responded the train attendant
- Well, this morning I noticed the train attendant's ticket reader seemed very bulky so I called the railway company and they rudely told me it wasn't a problem anymore since they had upgraded the model in 2013. So I was wondering if that's the new or old model.
- Well, I don't think there's a newer model but honestly I don't know, what I do know is this one is at least not built for me.

They started talking about various problems the train attendant and her colleagues had experienced. Of those these seemed to be the biggest ones:

• The device was made for much bigger hands than most train attendants'.
• The device was way too heavy.
• The device was tough to get a grip around and maneuver, probably even if you had "right sized hands".

The most alarming result was the amount of people on sick leave had grown notably since the current ticket reader was introduced, according to the train attendant, and the main reason was reportedly over-stretched arms.

The biggest issue with the current ticket reader seemed to be the humongous battery on the back. At this point I had curiously joined the conversation and commented that the battery's size seemed unreasonable knowing even several years old, cheap, small smartphones had batteries lasting for days doing much more complex tasks. The response to that was quite telling:

- The one we had before was much easier to carry but the battery didn't last long enough.

At this point I suspected two things:

1. Someone was informed: "We need to improve the battery on our ticket readers". Armed with that information, this someone wrote a spec saying "same functionality but better battery" and the humongous battery was the quick fix, a seemingly identical device but with more power. Implicit requirements such as "it should weight a maximum of ... grams" or "it should fit the hands of our train attendants" was most likely not considered.
    
2. Whoever tested this didn't test it under realistic conditions such as for many hours, trying to reach over other passengers, walking long distances carrying it and so forth. Another possibility is the casing and hardware was "just an ordered standard third-party device" (probably not built for this specific purpose) and all focus was on the software. A third option would be no testing was done since it was "just a change of battery, which will not change anything to the device, only improve it".

I'm still very curious if any acceptance testing was done by the railway company. If so, under which conditions, for how long and by who, actual train attendants? If not, why not? (I can list many plausible answers but still curious)

Continuing the conversation the woman initiating it started to list a few simple ideas, just from the top of her head, that could had eased/solved the problem. My personal favorite was: "Why don't design it so that the battery pack is in the belt". That seemed like a minimum effort, low cost solution to both the old and new problem. Knowing the current state though I suspect the manufacturer would probably had delivered a too short power cord.

I thanked this wonderful women who asked the question, as well as the train attendant. I got a valuable reminder of how important "dogfooding" and testing a product under realistic conditions are, as well as a good reminder of how much you can learn from just being curious.

My message with this story: Yes, it's often inconvenient to figure out and set up realistic conditions when testing but failing to do so can be catastrophic. As the train attendant pointed out: "Some days when my arm is aching I just look at the tickets and assume they are valid".

Update

Here's an image of the ticket reader. Notice the battery (bottom left) and how it obstructs a "natural grip" around the device. This train attendant also had notably larger hands than the train attendant mentioned in the original post (two different train attendants) to further emphasize the problem.

Finally, when I snapped this picture the train attendant immediately responded: "Are you gonna get us a better ticket reader?". Pretty telling comment.