Software Testing World Cup, Time management

Credit
This is based on my team's effort in the European qualifying round for STWC (Software Testing World Cup). The team consisted of Agnetha Bennstam, Sofia Brusberg and me. Everything mentioned in this post is my memory/interpretation of our collective work.

What is the Software Testing World Cup?
STWC is exactly what it sounds like: Testers from all over the world competing in software testing. You can read about it on the STWC web site but here's the bare minimum you need to understand this post:

To get to the finals you have to win a qualifying round. Typically there's one qualifying round per continent with a maximum of 250 teams each round and 1-4 testers per team. A round is 3 hours long during which the contestants test an application, revealed just before the round starts. Throughout the competition teams can ask questions to the application's product owner who answers these in a live video channel available to everyone. To get a better idea of how this works Matt Heusser have shared recordings of the video feeds. At the end teams are scored mainly based on a test report and the bugs they've filed.

The plan
We had a beautiful plan starting the competition:

Personally I have a history of not following plans so I was surprised when we actually executed pretty much according to it (I give Sofia and Agnetha full credit for this). Here is an adjusted, and somewhat extended, version of what actually happened:

First 55 minutes

After some minor technical issues with the YouTube channel we were up and running. While Agnetha and I mainly focused on the stakeholder's live video channel, Sofia started touring the application. We kept going like this gradually focusing less and less on the video until Agnetha mentioned we should stop and debrief what we had learned so far as well as discuss test strategy and possibly adjust the time plan. I would say Agnetha was our time keeper for most of the competition.

During the debrief/strategy discussion we identified 3 main areas to focus our testing on, based on the stakeholders' requests:

• Email functionality
• Screen resolutions (mobile devices)
• Possibility to steal accounts (customers)

To make it easy we focused on one area each. As security is one of my better areas I went with the account task, Sofia had already started looking into the emailing functionality so she continued while Agnetha started playing around with the mobile devices we had/various resolution.

Whiteboard from the competition

Exploring and debriefing

For the next 65 minutes we tested the product and had a debrief after 30 and 50 minutes. When we stopped for the first debrief (once again thanks to Agnetha) we realized all of us were testing user settings even though this was only part of Agnetha's "scope". To some degree this was good because the user settings functionality was rather buggy, but at the same time the bugs we found weren't generally that severe and once again, this was not communicated as a high priority from the stakeholders. So after the first debrief I think we got on track. During the second debrief everyone seemed to be in the zone. We kept this debrief short, just briefly summarizing the more severe bugs we had found, what we were currently looking at and if we needed any help, to make sure the flow wasn't lost.

One big difference between our plan and what actually happened: We didn't pair test nearly as much as we thought we would. I think this was a good decision, it was simply too much to cover in too little time so we had to spread out our resources.

Wrap up

In the initial plan we thought 30 minutes would be enough to finalize the report, review bug reports etc. Luckily I was a bit nervous about the report (my focus area) and suggested we should start with just under an hour left. After all, we could always keep testing after the report was done.

Let's just say there was no time after the report! I reviewed my bug reports and realized some things were not clear or I realized I had forgot to add something, I added things to the test report and often questioned "do we really know that or are we just assuming/remembering it wrong" which required further investigations and so on. We also had to rewrite some bug descriptions to be more business oriented rather than technical. All and all, the last hour went by quickly.

Conclusion

Thanks to Richard Robinson inviting me to a 3 hour test session during Let's Test 2013, I had an idea of how short 3 hours actually is. Still... 3 hours is not much time! I'm glad we did a time plan before the competition and reviewed it during the competition. Without it I think the limited time would had hurt us much more!

... and it was fun to actually follow a plan, I might try that again in a few years .)

Finally: Thank you Agnetha and Sofia!

Lessons

• Writing a good bug report takes a lot of time
• Taking notes is crucial, we could had saved a lot of time when writing the test report with better notes
• 3 hours is not much time
• A (good) plan helps a lot when under pressure
• Debriefs are valuable to continuously set/correct direction
• Even when alone you can benefit from stopping regularly to review what you're doing
• Visualizing (like a schedule or the image in this blog post) a plan helps as it makes it easier to relate to (at least for me). I think splitting it up in a way where the size of each activity is relative to it's length would had improved our visualization.