Test plan questions

I recently gave a talk on test strategy and gave the participants a list of "test strategy questions" (which should be called test plan questions, adhering to these definitions). I think and hope these can be useful for more people so last in this post you'll find my most recent version.

How I use the questions

I use these questions as a quick way to form a test plan. The test plan might be everything from a high level, early plan for an upcoming test project to something small like the strategy for a specific test charter.

The way I practically use the list is pretty straight forward. I just run through the questions and basically strike out questions that are not relevant. Not relevant refers to questions like "any particular tours we should perform" when creating a high level plan, "what are the key test/release milestones" for a plan covering a 90 minutes test charter or "how do we report bugs and status" if this is already stated by our overall process.

The questions not struck out I try to answer; either myself or in a meeting with the people involved. Involved in this case might refer to anything from a second tester to everyone involved in a project. The outcome is typically:

• I/we lack the necessary information or competence and need help answering this question
• I/we have an idea but I/we still want help/input from someone else
• I/we think I/we know the answer well enough but still want a second opinion
• I/we think I/we know the answer well enough

The results can be turned into a formally documented test plan, some brief notes next to the questions, a mind map, an image and sometimes the discussion and actions taken are enough (no need to document anything). Do remember that a test plan document is not a test plan; it's a document describing some parts of your test plan.

A few additional notes:

• The list should be improved to fit your context; add, remove and modify questions on the list accordingly (if you for instance know which bug tracker to always use, which most do, remove that question completely).
   
• You always have to take time and value into consideration. For a minor issue with low priority it's probably not worth inviting everyone in the project as the cost of doing so is too high; Your own judgement is cheaper and probably enough.
   
• The list can help e.g. project managers, line managers and developers think about testing concerns so do spread (your modified version of) the list to these people.
   
• You can split the list into several smaller lists; e.g. one short list of questions to use when creating test charters and a longer/different one for project test plans.
   
• It might be wise to think about what's important for a test plan before you even look at the questions. The point here is sometimes key details that definitely should be in your test plan might not be highlighted by any of the questions. If you start by using the list you might miss these key details (your thinking is limited by the questions). Once again I think value is important: How much time and effort seems reasonable to spend on this particular test plan?

Credit

I created these questions together with my students so credit should go to all the students involved. I also discovered testers before me had created their lists as well and I would not have thought about some of the questions below if it wasn't for their efforts.

So, before I present "my" version I want to highlight and thank both the students, Michael Larsen, Ministry of testing, The test eye and Michael Bolton. The four lists I just linked to, all come with their own twist/benefit so do check them out.

Questions

Notice some questions are generally only applicable when consulting, some only in an agile context, others when testing and development is clearly separated etc. Like has already been pointed out: Do remove or modify questions not relevant to your context.

Initial

I typically want to answer these questions before I go on with the rest, because these five typically impact my approach for the strategy in general.

• How much time do we have? 
• Do we need to document the strategy? Why? How will we document it? 
• What’s the primary objective (testing mission)? 
• Are there any other objectives? 
• Where can we get the initial information about the product/feature/system we need?

Planning

• What resources do I have available?
  people, money, equipment, licenses, time, expertise, facilities... 
• Is this a realistic amount of resources for this project? 
• How flexible is the time plan and what happens if the product is not “ready” by the deadline? 
• What project risks exist?
  e.g. customers not clear about what they want.
• What is our backup plan if risk X happens? 
• What is our backup plan in general if everything fails? 
• What is likely to change?
  e.g. team setup, requirements, developers' focus, delivery dates… 
• Any meetings we should attend/discussions we should be part of? 
• How do we handle handovers/new members/lost members? 
• Who does what?
  e.g. responsibilities and roles 
• Any known problems along the road? 
• Are there any workarounds or solutions to the known problems? 
• Any regulations, rules, standards, certifications etc. limiting us or forcing us to work/act in a specific way? 
• What administrative tools are (almost) enforced and what else do we need/benefit from? 
• How do we plan the everyday work? 
• What are the key test/release milestones?
• How flexible is the scope - can the project be down-scaled if some unexpected problem happens?

Prioritization

• What is most important (to test)? 
• What is not important (to test)? 
• What can be skipped all together? 
• What quality characteristics are most/least important? 
• Any specific aspect of these characteristics that is more/less important? 
• What is covered by other teams? 
• How do we continuously verify we’re doing the right thing? 
• What is our done criteria
  e.g. strict deadline, customer acceptance tests or some other assessment of "good enough quality" and if so, by who? 
• What's the general requirement for quality?
  Aim for the stars/critical system or “just don’t explode... too badly”

Information

• Where can I get information about X and who do I contact if that's not enough? 
• Which claims exist? 
• Which customers exist, can we contact them? 
• Who can answer questions and which questions? 
• What is still unknown/not clear about the project? 
• How do we simplify the complexity?
  Simplify the process of learning the product. An example might be "what sort of visual models would help to create?". 
• Any particular tours we should perform?
• Are there any general guidelines to how we deal with learning activities and knowledge sharing? 
• How do we further inspire/reassure/encourage feedback, reviews and knowledge sharing? 
• How do we stay up to date with what is happening in the project?
• How do we communicate with various information owners?
  e.g. email (address), phone (number), instant messaging tool, via other person, meeting etc.

Support

• What kind of built in testability will help testers?
• Which modifications can/must be done to the process in general to support the testing? 
• What do we need to learn more about? 
• Any particular configuration or test data we can prepare? 
• Which tools can help us? 
• What other teams should/can/must we cooperate with? When, how and why? 
• Do I know who the developers are and can I talk to them? 
• Do the developers have time allotted for us? 
• Are there any problems getting their time/getting in touch with the developers? 
• Will they keep working on this?
• What will the developers test? How does this impact our testing? Can we make suggestions regarding their testing?

Testing

• How do I tell right from wrong?
  Find potential oracles 
• Which oracles are generally most trustworthy? 
• What testing risks exist?
  e.g. unstable test environment or lack of knowledge about something.
• Which test techniques might be useful? 
• What expertise do we lack? 
• Which scenarios/typical use cases exist? 
• Which heuristics might be useful? 
• What logical components/modules are there?
• Is there anything I'm not allowed to do?
• Any (testing) tips the developers can give to the testers?

Product

• Which product risks exist?
  e.g. complex algorithms likely to be buggy or new technology used.
• Is there any complexity we might be missing? 
• Which functions will the system/application/feature have? 
• Who’s the target audience? 
• Which platforms, systems etc. should the product support? 
• What requirements exist for this product? 
• What problem is the product expected to solve? For who? 
• What problems have happened in the past with this product? 
• Any existing functionality that is impacted? 
• What must the product never do?
  e.g. any data sent as plain text is strictly forbidden

Reporting

• What do I need to cover and how well? 
• How do we track and visualize coverage and progress? 
• Which stakeholders exist?
• How do we report bugs and status? To who? Why?
  "Why" as in: Which problems/questions will the receiver hope to solve/answer with our report. 
• What other artifacts/information do we need to share? To who? Why? 
• When do we need to report what?

Additional sources

• Michael Larsen
• The test eye
• Ministry of testing
• Michael Bolton (thank you David Greenlees for the reminder)

Time estimates in testing, part 2

Intro

Before I begin: Thanks to feedback from especially Jari Laakso, I've found several flaws in my first post which was what I hoped for when I published it as the flaws help me understand estimation better. I will highlight some of them in this post but I also want to point out that part 1 has been updated several times since I first published it to clarify or correct content.

How to interpret the graphic below

Do notice that these are not polarities where you switch from one to the other, instead they are continua. I will however describe one rather specific scenario for each box for the sake of simplicity, a chance for you to see the differences between "each corner". My hope is these examples can work as inspiration/help for you to think different.

One final remark: I will provide all these examples as cross-functional scrum teams consisting of only developers and testers. Of course this is just one of many ways a team can be set up but I'll use it once again for the sake of simplicity.

1 - Estimate testing stories

Separated planning with detailed estimates.

In this scenario testers create their own stories, estimate them and get them prioritized as part of the regular scrum flow (the cross-functional team will now have a mix of stories relevant to developers/testers/others). This allows testers to create their own plan optimized for testing needs and I find it easier to motivate developers to help out with testing as needed as the only content left in the sprint might be test work (and this is very much visible) but...

Since planning testing even a week forward is rather hard in most scenarios (open bugs blocking stories, open bugs delaying stories, late code deliveries, level of quality differs from the expected etc.) I find this approach somewhat chaotic (even though you might expect the complete opposite).

A second challenge with this approach is the risk of creating a gap, within the team, between testers and developers. This, I think, is because if you actually try to follow the plan (test stories) you will lose some of the flexibility to drop something in favor of something else in order to minimize the feedback loop between feature implemented and feature tested.

Finally my experience is it takes a certain skill as a tester and/or a certain group of developers to actually make developers engage in the test part of the sprint planning, further increasing the risk of a gap between testers and developers.

For me, session based test management is an approach to exploratory testing worth looking into if you want a separate test planning, with estimates, without making it too expensive.

If you choose this approach, I would strongly recommend you to share sprint plannings, backlog groomings etc. so that everyone sits down together. It's easy to "optimize" so that testers have their groomings or that developers don't engaging in those, which is like asking for a disconnect to happen.

Example of a dangerous archetype potentially advocating for this approach for "the wrong reasons":

The project manager/product owner/manager/test lead who pretends/thinks testing can be controlled.

2 - Estimate dev stories

Shared planning with detailed estimates.

In this scenario testers and developers share stories (DoD includes both implementation and testing). A story is estimated both in regards of testing effort and developer effort for a combined estimate. A story is considered open until "someone" decide that the implementation, testing and bug fixing is done. That "someone" is typically the concerned testers and developers together but if they disagree or the estimate turns out to be way off, a product owner or similar might have to step in.

My experience is that this fosters a very close bond between testers and developer which typically has a positive effect on e.g. tester status, information flow and code related testability. Other benefits are developers have a better motivation (and help) to test themselves as their work is not over until testing is "finished".

However, since the testers are normally the minority it's easy they become the ones always losing votes and debates which may impact how they test, when they stop testing etc. and often not in a good way. Also since the testing is so closely tied to the development things like integration testing, general quality assessments (e.g. security, performance, consistency, test legacy for support of changed context etc.) and regression testing may be missed (regression can be helped by developers taking a more active role in creating automated smoketests but that's just a part of it).

One last problem is if you actually create new tester-developer hybrids and the pressure builds up (important release, budget cuts etc.) teams might trick themselves by moving more members into developing and tamper with their normal quality standards (which in a few cases might be the right choice but often not in my experience, often because the "temporary need" becomes permanent).

If you choose this approach, I would strongly recommend you to introduce activities such as system tests or release tests. In this activity, also involve developers as part testers, part bug fixers, so that a gap between developing and testing isn't created (testers not being able to provide feedback quick enough in the new sprint). A different method is to have teams solely dealing with integration testing (more waterfall like/support team) but for this to work there has to be strong information channels established between the cross-functional teams and the integration test teams.

Example of a dangerous archetype potentially advocating for this approach for "the wrong reasons":

The R&D manager/developer who doesn't really understand there's a need to test anything else than the code that was just written or the developer who rush the testing part since "that's not that interesting".

3 - Follow development planning

Shared planning without detailed estimates.

Very similar to the above but without time estimates of the testing. The difference that happens is you save a bit of money (less administration) but testers are a bit more detached from the development process (but not necessarily from the developers) and if estimates can actually be somewhat accurate you lose that as a planning benefit/potential delay indicator; in this scenario sprint plans assume testers will manage time limits if developer manage these which is of course not always true. But do remember that estimates are often not that accurate to begin with (described in part 1) so the actual effect of no test estimates might be small, personally I would most often worry more about the potential disconnect between testers and developers.

If you choose to approach testing like this, I would strongly recommend you to closely monitor so that testers participate actively in all the typical planning activities (groomings and sprint plans). This is to help build a team where testers and developer keep working closely together. The goal is to  "just" remove the part where testers perform time estimates, nothing else, in the planning meetings.

Example of a dangerous archetype potentially advocating for this approach for "the wrong reasons":

Testers who simply dislike administration and don't even care to try estimates. A reason why this might be very troublesome is also because these testers could easily contribute to the disconnect between testers and developers in that they might want to use the removal of estimates to skip out on "irrelevant meetings" where key implementation decisions are taken.

4 - Report on/off track

Separate planning without detailed estimates.

In this scenario testers attend backlog groomings and sprint plans to stay up to date with the planning and the product as well as get attention on testability needs; but they have a complete separate planning done together with developers (costs time) or with testers only (may hurt developers' understanding of/respect for/interest in testing). From a project planning perspective testing "just exists" and whenever the team suspects there's a risk for the testing/bug fixing to be late this is reported and dealt with just like any other case where estimates are wrong or development delayed.

This separate planning would happen as soon as possible after the sprint plan is complete (so that the scope for the sprint is known). Testers discuss if there are any other activities they should plan other than the regular testing of stories. They also look into if stories should be tested together to avoid costly retests. This makes it easier to take testing needs into consideration without messing up the development planning and testers can make rapid changes in their plan without impacting development stories.

This has some interesting benefits but there's a great risk of testers forming a subteam within the team potentially hurting the developers interest and effort in testing (information flow should be less of a problem since groomings and sprint plans are key to be able to do the test planning).

If you choose this approach, I would strongly recommend you to leave a bit of time open for unplanned as bugs not related to development stories are more likely to be discovered and help testers need might not be taken into consideration when planning and thus that extra time has to still exist when testers need help. This might seem stupid but if you've been in a software project you know some sudden needs from testers (blocking bugs, needed support tool updates etc.) are hard to predict at the sprint start.

Finally the name for this is approach is a bit strange. If we're not estimating testing (at all) we can't possibly say if we're on track or off track. I thus want to again stress that in reality we always make estimates but rather than debating whether one of many tasks should be estimated to 3 or 5 hours we look at the whole picture and make a rough estimate if we'll make it or not in time.

If you choose to approach testing like this, be careful. Putting testers with low communication skills, reporting skills or unwillingness to make work transparent, in a team like this can lead to isolation and a negative mythic image of testing where developers neither really understand nor care about testing ("they don't seem to care for the planning and the team so why should we care about them").

Examples of two dangerous archetypes potentially advocating for this approach for "the wrong reasons":

Waterfall accustomed testers or developers who still haven't figured out how to work with testing in scrum and other agile approaches. Also testers who don't want to/have the ability to make their work reasonably transparent.

The actual implementation

In a real life scenario we move a bit in between two or more "squares"; for instance testers will sooner or later need to do work that only benefits testing, so completely following the developer's stories is a myth (or an example of a dysfunctional team). However, testers not feeling like they are allowed to test things outside the current development scope e.g. because of a big pressure to close stories, is something I've come across multiple times and with that said my goal is to highlight options. The examples in this post is meant to help you see alternatives; they're not models, ready to implement.

My hope is part 1 helped you understand the dynamics of estimates and planning while the second gave you some practical ideas to experiment with. Like in part 1 I want to emphasize that this is still thoughts in process but I can't wait forever to share them if I want to get feedback...

Finally: Thank you once again Jari Laakso for all your feedback on the first post!

Time estimates in testing, part 1

Why this blog post

Activity planning and time estimation in testing is complex to begin with; combining that with cross-functional teams (having to actively take other's planning into consideration) and I've seen a lot of confusion. In this blog post I hope I can arm you with some useful ideas on how to deal with activity planning and time estimates in testing.

... and that turned out to be a lot bigger task than I anticipated so in this first part I will speak about the more overall concepts and in the second part I'll provide more concrete explanations of each approach (see the graphic below).

Key attributes

When I started listing various ways I've been working with estimates I continually came back to two variables that seemed central:

1. To what degree does testers and developer share their planning?
2. How much trust and effort is put into time estimates?

For this reason I made this little graphic:

Do note that this is still thoughts in process so I do appreciate any feedback on whether this makes sense or not.

Separate planning vs Shared planning

I've been in teams where testers were only or almost only expected to test what had just been implemented (as in testing the exact work described in the implementation stories thus sharing the planning with developers) as well as teams where testers were expected to create their own planning based on the testing needs; both extremes come with their own benefits and drawbacks:

Shared planning:

 + less administration (as in cheaper)

 + closer relationsship between testers and developers

 + quick feedback loops in general

 + easier to get focus on testability improvements in the code

Separate planning

 + more focus put on regression, system (e.g. consistency) and integration testing

 + easier to get work only affecting testers prioritized

 + planning can be optimized to fit testing

Two dangerous mind traps I've experienced:

Shared planning
Risk: Testing seen as something very simplistic where you just test the product's new features with no or little regard to the big picture.

Negative effect: General testing activities, such as overall performance testing, integration testing or regression testing are down-prioritized.

Implications: Bugs related to e.g. integration, unforeseen legacy impacts or product inconsistency.

Coping: To deal with this it takes testers or other team members who are good at advocating for testing in general and can motivate why certain test activities makes sense even though obviously impacting code is not changed.

Separated planning
Risk: Testers and developers move apart and create two separated subteams within the team.

Negative effect: Impacts information sharing, understanding of the other's work (which in terms may impact the respect for one another's profession) and developer's losing the feeling of responsibility for quality.

Implications: Prolonged feedback loop (code developed to bug reported to bug fixed to fix verified), worse quality on code handed over from development, testers lacking important information making them less effective/making them take bad decisions.

Coping: Well functioning team and social management, testers who're good at communicating their work and good at speaking with developers on the developers terms or vice versa, developers putting a great interest in testers and what they do.

I'll elaborate a bit more on this in part 2.

Effort put into estimates

Let's talk about the two extremes first.

When we put a lot of effort into estimates we may for instance split the work into small tasks to make them easier to estimate and try to give each an as exact time estimate as possible; e.g. "these 6 tasks are estimated to 3, 4, 6, 3, 1 and 9 hours so the time to test this feature should be around 26 hours".

Little effort are put into time estimates when we for instance accept an external time constrain, e.g. "get the testing ready for the release" or "finish your testing within this sprint" and we more inform the one setting the time constrain what we roughly think the quality of the testing will be at this point.

Important: "Quality of the testing" in this case refers to how deep/well we expect we will be able to cover the various features. This is not the same as the "quality of the product". Why the latter is not what is (typically) talked about is because that oe is way too complex to estimate and often out of what we can actively influence. For instance it's not our call if a bug we've identified should actually be fixed, that's up to the stakeholder(s).

At this point the former might seem more reasonable but it's not that simple...

Accuracy of estimates

There's no "done" in testing. This is sort of true for development as well, we can always continue to improve the code itself or polish the actual feature, but at least we can observe the product and say "this is good enough" and, important, that state of "done" can be fairly well-defined/described.

Even though some try to pretend it's not true; testing does not at all work like this.

Testing happens inside you head, development too (to a large degree); However, the product of testing also stays in your head and is relying on your ability to communicate it while the product of development can be easily observed by others. This complicates estimates since we cannot keep discussions around a fairly well defined end product; instead we have to leave it to everyone's interpretation of the task and do our best to communicate our own interpretation so we roughly estimate the same thing at least. For this to work at all, tester's first need consciousness of what they actually do, the necessary communication skills to explain this to others and the receivers must have enough understanding to interpret this message correctly. No human on earth (as far as I know) do even one of these things flawlessly (aka: communication is hard).

With no well defined "done" everyone will have to rely on their interpretations of what they think needs to be done and what they think the receiver asks for. That in terms will impact the estimate's accuracy but this's just part of the problem...

... the other part

On top of estimating a task we cannot clearly define we also have to estimate something that is inherently plagued with great levels of uncertainty:

The time it’ll take to test something depends on external factors such as the quality of the code, the complexity of the chosen design (which is often not set when estimates are done), the stability (up-time and robustness) of necessary environments, the help we'll get to correct problems found/blocking etc. To continue the reasoning the quality of the code depends on things such as how hard something is to implement, the skill level of the developer(s), how stressed they are (if a developer has a tight schedule she can typically stress the design a bit to get it out the door in time which in terms affect the quality which in terms affect the effort required to reach the abstract "done state" when testing) and stress level depends on even more factors. I can go on but let’s just say the time it takes to test something to the “abstract, hard to reasonably accurately define, level we desire” depends on so many external factors that the estimate's uncertainty is huge.

Bullshit!

I once told a project manager this and he replied: "Bullshit! My testers ace their estimates almost every time!" (in a product with pretty crappy quality and expensive testing should be added). And here comes a conflicting truth:

We can't predict when we'll be "done" but we can be "done" by Wednesday if you want.

In development we need to produce some kind of agreed upon product based on, at least, our explicit requirements. Remember how testing did not have a defined "done"; well taken to its extreme we can (almost) say we're done at any point in time since it's unclear what done is anyhow: "This is the findings we've made so far (and are able to communicate). You told us to be "done" until today so we are "done" but our professional judgement is this needs more testing as the following areas are still considered unstable or only superficially tested...". In a sense we always do this, estimates or not, but don't even spend time trying to guess in the latter case, instead we do our best to put us in a good enough state when the time is up.

... and before someone talks about bad attitude or lack of professionalism; this is of course not how it's practically done. Some level of estimation is always performed but rather than saying: "We think this will take 10 hours, this 5 hours, this 2 hours and this 8 hours." we might say for instance "We think we're on track for the release but the quality at this point is unusually low so we're a bit worried this might change".

Planning based on time or effort

This topic is too big to cover in this post but a quick rundown.

When planning our testing we can either focus on effort or time. Effort means "we want to reach a state where we/someone else feels confident about this particular part before we leave it" while time means "we want to spend the fixed amount of time we have in a way so that we've looked at everything at least briefly". In the former we're doomed to not be "done" if something runs late and leave the project management with little choice but delay e.g. a release if they don't want to take on a considerable risk (leaving parts completely untested). But this also allows us to argue for the need of more testing better by saying "we haven't even looked at these four features" and we'll spend less time revisiting areas having to "relearn them" as an area is left more in a "done state".

In the latter we will hopefully have found at least the most obvious, serious problems in all areas of the product and can thus make a general assessment: "our testing is fairly shallow in most areas and we would like to spend more time with the product but from what we've seen all of the new features seems to be okey at a basic level". The drawback with this is it's harder to argue that we actually do need more time to test if stakeholders aren't well informed and comfortable with the concept of "uncertainty" as well as a greater risk of needing to revisit areas for further testing.

How does this relate to estimates? Well in my experience the effort approach prevails when we have estimates and plan our work around these actual stories since "we shouldn't" have too many open stories at once and when they are closed, they are closed so we have to "complete them". In the same way the time approach is more common when we skip estimates (my experience), at least when in combination with a separate test planning. If we have a set deadline (e.g. sprint end or release date) we can more naturally plan in a way so that we've at least looked at everything once.

I say this because most stakeholders I've been in contact with seem to prefer the time approach but still argue for the importance of estimates and these concepts seem a bit contradictory, at least in the context of testing. One last note: The contradiction problem still applies to development: If we get a bit of everything ready we will at least have some kind of product in the end but as described earlier: Since more has to be in place for a piece of software to make sense combined with that estimates should be easier to get accurate enough; the contradiction is not as big of a deal (my biased opinion, feel free to challenge this).

Wrap up of estimate or not

To wrap up:

Since time estimates of testing will always be plagued with great levels of uncertainty, no matter the estimation effort, the question is if detailed estimates really provide enough additional value to support their cost (both time and the risk of misleading stakeholder to believe we know when we'll be "done"). The "ideal" level of estimation is also highly context dependent, the context changes over time and we can't really objectively measure whether our approach is good or bad; so we'll have to experiment and rely on our ability to observe/analyze.

... and finally don't confuse time estimates with planning; Planning is made in both cases and the effort spent on planning has no/little correlation to the effort spent estimating (my experience).

Stay tuned for part 2...

Arguing for Exploratory Testing, part 2, Reuse

Intro
You can read a bit more about this series in the first post:
Arguing for Exploratory Testing, part 1, Traceability

The topic for our second Transpection Tuesday on "Arguing for Exploratory Testing" was Reuse.

We finished with two open questions:
Can we ensure we actually repeat the exact same test a second time?
How do you actually achieve reuse in exploratory testing (when it is desired)?

Reasons to reuse tests
First we tried to state reasons someone would want to reuse test cases:

• Save time during test design
• Functionality is changed and we want to rerun the full/part of the test scope
• We want to verify a (bug) fix

Preconceptions

Looking at reasons quickly led us to some preconceptions which became the topic for a big portion of the session:

• Effort = Value
• Equal execution = Equal value
• Our scope is (almost) complete
• Reuse = free testing
• A monkey can run a test case

Preconception: Effort = ValueSince we've invested so much time (as well as money and prestige) in writing test cases they must be worth more than a single execution.

• Even if presented with clear evidence we may reject it to defend out judgement
• We may overestimate what a test case is useful for (we want to get the most out of our work)
• It's my work, criticize it and you criticize me not the work! (common and unfortunate misconception)

It takes a lot of self-esteem to say: "Yeah I screwed up, could you help me?", especially in an environment where mistakes are not accepted. Notice many of the "so how can we make the most of this mistake" still communicates "so you made a mistake, now you'll have to suffer for it by telling us why you are a failure". It takes a lot of work to change this.

Preconception: Equal execution = Equal value

Let's say we execute the exact same steps in a scripted and an exploratory way, wouldn't that be two identical tests? We believe not.

1. Your goal differs. With test cases your goal is to finish as many test cases as possible (progress). That's how you measure "how much testing you were able to perform". In exploratory testing you are judged based on the information you provide thus you should be more incline to spend a few extra minutes observing/following something up even when it's not "part of your test".
2. Your focus differs. When you have a script you have to focus on following that script. In exploratory testing your goal is typically to find new leads to base the next test on. That means in one case your focus is on the product and in the other on an artifact. Think about the Invisible Gorilla experiment.
3. Scripts easier bias you not to observe. In a script you typically have verification steps e.g. "verify X=5". We believe this could bias you to not be as observant during the other steps: "this is just setup so nothing should happen that concerns me".

Preconception: Our scope is (almost) complete
We know a feature's boundaries (specifications, requirements) so when we set the scope for testing we can, and usually will, cover almost the entire feature.

• We can't know the boundaries of a feature:
• We will impact and use other components not "part of the feature" e.g. other code in the application, the operating system, surrounding applications, third party plugins, hardware, hardware states etc.
• We interpret planning documents differently, adding parts, discover things we couldn't had anticipated, correct mistakes or interpret something differently than intended by the author and/or interpreted by the tester.
• We can (almost) always tweak a test a little bit (e.g. change input data or timing). But testing all combinations (we recognize) is way too expensive. Also there are usually so many ways an application can be misused (intentionally or unintentionally) that even with a ton of creativity we can't figure out them all (ask any security expert .)

So our scope is basically a few small dots on a big canvas rather than a well colored map. But those dots are (hopefully) carefully selected to protect us from the greatest risks we can anticipate. Still, they are only dots.

As testers we easily support the preconception of full coverage by answering questions like "do we cover this feature now?", "is all testing done?" etc. with a simple "yes" or "almost". The more accurate answer would be "we cover the most important risks we identified limited by our current knowledge, time available and other constraints", but that answer is not very manager friendly which leads us to...

There is a general lack of knowledge and understanding of testing in most organizations. And we decided to stop there since that was a way too big question to tackle at the point we got there. But it's an important questions so please take a moment and think about it for yourself: How can you improve understanding and interest for test in your organization?

A final note. Since we only cover a small part, reusing a test scope will not help us catch the bugs we missed the first time. How big of a problem that is differs but repeat a few times and it may scale up in a nasty way.

Preconception: Reuse = free testing
We've already written the test case so wherever it's applicable (which should be self-explanatory) we can just paste it into out test scope and voí la! Free coverage!

The big issue here is the "self-explanatory" part. Problem is what fitted well in one feature might not do it in another even similar one. Even without needed tweaks we still have to figure out what the test case actually does, so that we know what we have covered with it and what we still need to cover in other ways.

This process is expensive, really expensive, so sure we save time not having to figure out the test and how to practically run it all over again but consider the time it takes to find the test case, analyse what it covers, analyse what it doesn't cover, analyse how it interacts with existing test cases, analyse if something has changed that impacts the test case compared to last time and so forth.

Preconception: A monkey can run a test case

• We all interpret things differently. Click can mean single click, double click, right click (already assuming the first two were left clicks), tab and use enter, middle button click, etc. Even a well written, simple test case can lead to different interpretations.
• One thing we're looking for is unexpected behavior and it's in the nature of "unexpected" to be something we can't plan for. Thus to get much use of a test case we need to handle the system well enough to investigate and identify when a behavior is unexpected or undesired.
• We do a ton more observations than we consciously think of. These observations takes practice, focus and skill. For example, when you boot your computer you would react to a lot more things than you would add in a "boot test". Examples: screen is blinking, smoke is coming out, lights in the room flickers, you hear strange mechanical sounds, all these should catch you attention but are unlikely written down.
  
  More skill and/or focus can lead to more valuable observations: The login screen looks different, memory calculations are wrong, it's slower than usual/expected, BIOS version is incorrect, the operating system's starting mode is wrong etc.
• When we don't fully understand something we tend to write it down less detailed (sucks to look stupid by writing down something incorrect and we're too lazy to investigate every detail we don't understand, it's easier to investigate as we get there).
• When we write a test case based on specifications, requirements and other "guesses" of how the end system will work even a flawless instruction will sometimes not correspond to how the system is actually working (including when working as desired). This of course requires the person executing to be able to correct the test case thus understand both the intention with the test case and how the system works.
• If we don't understand the system we may lose a lot of time setting up fully or partly irrelevant variables to the values stated in the instructions. The immediate comment is, if we have stated irrelevant variables in the test case we've failed. Consider then that the variable might be irrelevant to the test but mandatory to the system (e.g. you have to set a valid time server). Leave that out and the person executing once again needs to understand the system.

When is reuse actually beneficial?

• We have rewritten something from the ground up but want it to externally still work the same. Reuse could save time.
• We have some critical paths through the system that can't break.
• We need to quickly regression test a feature without having to dig in too deep in the feature itself.

But remember that the one executing still should understand the test and the system to ensure tweaks (using different input values, triggering different fault cases etc.) can be made and important observations are more likely to be made.

How can we achieve reuse in Exploratory Testing
Not covered much by this particular session but a few thoughts:

• Charters
• Debrief notes
• Test ideas
• Test plans

Try creating a report specifically used as a "feature summary" including valuable operational instructions, general test ideas, impacts, lessons, problems, tools, important details, testability etc. We did kind of this at my former company where we let the test plan continuously turn into an end report as we added lessons from our testing. This would not only help when retesting a similar feature but also as educational material or test plan input for instance. Important though is to stay concise, noise is a huge enemy! The number of readers of a document is inversely proportional to the number of pages in the document, you know .)

A few notes on test case storage
First off I love this post on having a big inventory of test cases by Kristoffer Nordström.

It's easy to think something you've already created is free, but there's no such thing. Having a large inventory to test cases costs in many different ways:

• Storage
• Noise (it's one more thing testers have to keep track of)
• Another tool/part of tool for testers to stay updated with / learn / understand
• For a test case to be fully reusable later it should be kept up to date. How many refactors all their old test cases as functionality is changed?
• ... if you do, that sounds really expensive.

Summary

Reuse has it's place but be careful!

Remember reuse means inheriting blind spots, has a cost and still requires the person "reusing" to know just as much about the feature, system and testing in general as if (s)he wasn't reusing old checks.

Take care, and I hope these Transpection Tuesday notes (even though somewhat messy) were helpful!

... and of course, thank you Helena!

Arguing for Exploratory Testing, part 1, Traceability

Background

The topic for my and Helena Jeret Mäe's last Transpection Tuesday was Arguing for Exploratory Testing. What we basically wanted to achieve was to get better at explaining the pros (and cons) about exploratory testing, in a concise way, as well as identify common preconceptions about scripted versus exploratory testing.

Input

We had defined 15 subtopics such as time estimations, credibility and making sure the important testing is done. The first item on this list was traceability which turned out to be enough material to fill the whole 2 hour session.

What is Traceability

First question was: What do we mean with traceability?

Our answer: Being able to track what has been tested, how, when and by who.

Why do we want Traceability

The next question was why we want traceability. We quickly formed a list but reading it now makes me realize we mixed together traceability and claimed benefits of having a trunk of test cases. But anyway:

• External demands
• Ensure work has been performed
• Base for further testing
• Support handovers
• Create a map
• Reuse

General thoughts
One thing we got back to over and over again was: The best way (often related to level of detail) to achieve good enough traceability is highly context dependent! For example having a simple mind map with short comments is good enough for one company while another requires every session to be recorded with the recordings being stored and indexed together with session notes, debrief summaries and saved logs. It all depends!

Another reoccurring theme was: "But do we really achieve that kind of traceability with test cases". I will not bring up those discussions much in this post but expect another one on "false assumptions about scripted and exploratory testing" soon.

Terms

Charter
Charter is basically an area to test, a way to break down a big testing mission. Notice though that as you test new charters might come up so it's by no means a definite plan. Read more >>

Test idea
Typically a one liner describing one or more tests you want to do. Read more >>

Session
A timeboxed, uninterrupted test sitting, typically 60-120 minutes. Read more >>

Debrief
Refers to an activity happening after a session where the tester explains what has been done to, for example, a test manager. This also includes clarifying questions, feedback and other kinds of dialog to help both parties learn from the session. Read more >>

Recording
We mainly refer to screen recording (video, either using a screen recording tool or an external video camera) but could as well mean record audio, save logs/traces or other ways to save what has been done. A good resource >>

External demands

This refers to regulated businesses (watch the excellent presentation What is good evidence by Griffin Jones), evidence in a potential lawsuit or customers demanding test data.

Possible solutions:

• Record the sessions, preferably with configuration (device, version, settings etc.) explained if that matters. Adding commentary might improve the value as well (communicating purpose, observations etc.). This is also typically a scenario where logs/traces can be a required addition to a video recording. Once again, watch What is good evidence.
• Store session notes
• Store session summaries
• Store charters
• Store debrief summaries
• Store test ideas (assuming they has been covered by your testing)

Creating support to find old information (index) seems key as well. For this charters, time stamps and/or categories might be useful to tag your save material with.

Ensure work has been performed

First question raised was: Is this really something we want to encourage? And our general answer is no; with the motivation that people in our experience tend to do things to look good rather than do what is needed/valuable when closely monitored. But being able to know that the testers actually do their job is closely connected to credibility and transparency so still a valid question.

Possible solutions:

• Debriefs
• Recordings
• Notes
• Bugs reported (a really bad metric for this but can indicates something!)

Debriefs seemed to most often be the preferred approach. During a good debrief the person being debriefed asks followup questions that will require the person debriefing to explain the testing done. A byproduct in this process would be to ensure that the tester actually did a good job / any job at all. But once again; if your focus is on monitoring, the people monitored (testers as well as non-testers) is likely to waste time proving job has been done rather than actually work!

Base for further testing

Let's say we've finished the prepared scope or are suddenly given an extra week to test something. If we can't go back and use already executed tests as inspiration, how do we know where to continue?

Possible solutions:

• Having a bulk of charters as inspiration
• Make comments about testing you've left out in your finished charters/sessions
• Review session notes

We also brought up if there's a value of actually looking at what has been done. Often we found that the time it takes to analyse the work already done might not be worth it (information being too detailed making it hard to overview and learn from quickly). Simply exploring using knowledge we might not had had the first time or by having a different tester from when we first tested, is often more than enough to add value. After all, the time we analyse is time we cannot test (which might or might not be well invested).

Support handovers

One tester leaves (quits, parental leave, other tasks etc.) and another has to take over, how can we manage such a change when not having a set scope of test cases? First of all the new tester do have to spend some time getting familiar with the feature in exploratory testing but this is also true for using test cases since we, for instance, can't predict what problems we will run into thus can't prepare instructions for those!

But we can make it easier:

• Charters (with status)
• Debrief
• Documented test ideas with already investigated ideas being marked
• Session notes or session summaries
• Mind maps or other test planning with already tested parts commented
• Documenting lessons learned (like operational instructions)

Debrief in this case refers to a general debrief of what has been done, what we know is left, problems seen, lessons learned, where information is stored, who to talk to etc. by the tester leaving. Of course if the switch happens very suddenly (e.g. sickness) performing this is not possible and in that case it's important testers are professional enough to document what has been done (mind maps, short plans, visualizations, debrief/session summaries, charters). This is once again true for both exploratory and scripted testing.

Create a map

A bulk of test cases combined with statuses can somewhat be used to draw a map of what has been covered and what is left to test. How can we visualize this without test cases?

Possible solutions:

• Charters
• A mind map describing what has been tested
• A picture/model of our product with comments about testing/coverage
• Other visualizations like diagrams
• The Low Tech Dashboard

A few important notes:

1. You sure have a map with test cases but is it actually anyway near accurate? Say we have two equally complex functions. One takes 1 argument, one takes 10. We likely will have at least 10 times as many test cases to cover the second function. So if we execute all the test cases for the second function, have we really covered over 90% (with "covered" only considering these 2 functions)?
2. Even if equally sized, that map would not cover what we didn't anticipate from the beginning so you still need to add an up to date judgement/evaluation (e.g. "wow that network protocol sure was more complex when we expected during the planning, we need more testing of it!").
3. Scale is really important. Do we want to see Tartu, Estonia, Europe, the world or the Milky Way galaxy? We might need different visualizations to create all the maps we need (once again, think about value, how much time can we spare to keep these updated).

Reuse

Later a similar feature or a feature impacting the one we just tested is developed and we want to reuse the work previously done. How can we do this without test cases?

First of all, reuse is one of the places where test cases are powerful. However you have the minesweeper problem: If you walk the same lane in a mine field over and over, as new mines are constantly added, it's likely that the number of mines beside your narrow track start to build up while few will happen to end up in your path. Meaning, running the same tests over and over is less likely catch new bugs as creating new tests are so value quickly diminishes (more tests executed is not equal to more valuable ground covered).

What we often would suggest is to use knowledge acquired the first time as foundation for new testing to speed it up. Think about the new risks introduced and what needs to be tested based on that (like with new functionality) rather than how old test cases might fit into your testing.

Possible solutions:

• Reuse of charters
• Reuse of test ideas
• Look at old session notes / summaries
• Use old recordings (the simpler the form of the recordings the better for this, watching several hours of screen recording is probably waste)
• Start a wiki page/document/similar for each feature and add lessons learned, where to find info, problems etc. as you test.

Summary

There are many ways of achieving traceability (and similar potential benefits of test case trunks) in exploratory testing, Session Based Test Management principles seems to be the most straight forward way but keeping track of test ideas or using other approaches works as well. All have their own contexts where they seem to work best (e.g. SBTM might add too much overhead for a simple project).

All and all, if someone claims "You lose traceability with exploratory testing", ask what that person means more precisely (e.g. present testing data to customer) and explain the alternatives. Notice this is only based on our two hour discussion and there are a whole lot more to add so think for yourself as well! Also question whether you actually achieve the kind of traceability requested using a scripted approach and to what cost. Finally question if the requested traceability is actually worth its cost no matter if exploratory or scripted testing is used. Doing unnecessary work is wasteful no matter what approach you use.

Finally: There are still contexts where a highly scripted approach is likely the best option but the closer you get to a pure scripted approach the fewer and more extreme the contexts become.

Thank you for reading!

And thank you Helena, see you next week!

Going Exploratory

Prerequisites

• Test process dictating a factory testing kind of approach.
• Progress was presented as pass-fail ratios with a brief comment at best
• Test cases were saved in a humongous, very general, database application.
• Bugs and requirements were saved in another humongous, very general, database application.
• Most documents were based on big, thorough templates.
• All development was split into features, a feature was then further split into smaller packages. A large feature could take over a year to finish (implementation description written to hand off to integration testing) while packages were typically released once ever two weeks.
• The product was a huge real time system (middle-node in an even bigger system), had legacy from 1975 and was mostly written in a language older than C.

March to June - Starting out
Early in 2012 I was asked if I wanted to join a pioneering (second), self directed, cross functional team at my former job. It was the golden opportunity I had been searching for, in my wish to introduce more context driven thinking. I answered yes immediately.

Eager to put everything I had been reading about into practice I started scouting my possibilities. First mission was to figure out what "self directed team" actually meant. The answer was rather disappointing: "Well, you control the inner process but you have to provide the same output as everyone else" (documents, test case storage etc.).

I decided to interpret output a bit more... general (ask Meike about my willingness to test or bend rules). I began asking the various receivers of progress reports, strategy documents and other output, what questions they tried to answer with these artifacts. I then decided to interpreted "answering the same questions" as "same output", which would later stir up a lot of criticism but luckily noone seemed to know who was i charge of us at this point so I got away for now.

I began to wildly change everything around me, more or less removing the whole foundation on which all our previous testing relied without adding any new. For a while it seemed fine. I got shit done! It was fun, it was quick, it felt efficient, I was -exploring-!

July - Chaos
Vacation time, everyone was calm, just mentally preparing for a month of freedom.

... except anyone in contact with the testing in my team...

Everything was in chaos! In the midst of trying out everything I wanted to implement (but didn't understand), I had lost track. To make matters worse I had to, in the middle of this chaos, hand over my work to an experienced tester who would join my team starting the first day on my vacation. I scratched my head trying to figure out what I had done and what I wanted him to do. I sat down and started writing, trying to at least make it look like I had things under control. Doing this I realized my team was not informed about half of the stuff I called "how we work with test in my team". The dreamy world I had been living in dimmed away as reality slowly materialized in the document in front of me. When done I could at least breathe a sigh of relief, I hadn't screwed up everything (yet). I handed over my work and went on vacation.


Vacation
A well packed schedule was turned upside down when my youngest son was hospitalized with a severe bacteria infection. All the traveling back and forth to the hospital gave me plenty of time to reflect on my work. I slowly started to see ways of turning my mess into something valuable.

August - The turning point
Vacation ended and I headed back to work. The day I came back my new testing colleague, who we can call Saam, since that's his name, left for his four weeks of freedom. I started looking at his testing. It was also chaotic, but a lot more structured (call it professional if you like) than mine. I started to clean up my mess by attacking the main problem at hand: How to know what have been tested?

The first solution was nothing fancy, but oh so important: A simple spreadsheet with functions to test on one axis and general stuff like how the function works under load or during restarts on the other. Each box that formed had a color representing its status, yellow for unknown, red for open bugs, white for invalid combinations and so forth. It was basically a simpler version of the Low Tech Dashboard, which I unfortunately didn't know about at the time (if I were to redo it I would use the Low Tech Dashboard as my foundation). The document started almost entirely yellow (unknown) but at least I had a plan.

When Saam came back we started talking about how to work with test in general and found out we had very similar ideas. From here on forward things worked better and better, thanks to brilliant chemistry and adventurous but realistic approaches.

September to October - Upping the pace
Saam loved to dig into technical details, I loved the more overlaying principles, together we rocked! As we learned more and more and started to form a routine I began documenting the process that had emerged. It all ended up as a concise, rather visual, easy to grasp, four page process description, probably the shortest document ever created inhouse (of course not exaggerating even the slightest).

It basically said:

1. Start a wiki page for the feature. Fill it with information like; lessons learned, operational instructions and where to find more detailed information. This wiki page is updated regularly throughout the whole feature and emphasis is on keeping it as short as possible, only including valuable, up to date material.
   
2. Learn about the feature, read docs, speak with programmers and experiment with what exists already. While doing this, document test ideas and a brief test strategy. Don't plan to much but make sure good ideas aren't lost.
   
3. The moment before code is shipped to us, grab the developers and possibly a testing expert/domain expert. As simple as possible, present our test ideas so far and ask for feedback.
   
4. When the code is shipped, use the test ideas as input, but don't in any way limit yourself to only test those. The goal is to steadily add new ones as more is learned about the system and feature.
   
5. As the perception about the status of a certain area changes, update the current status (color code) in the spreadsheet and add a few comments to help explain the situation. This spreadsheet serves as progress and coverage report.
   
6. When done with testing in an area (group of charters) sit down together, look at the test ideas now documented and see if there is anything important left out based on current knowledge. Consider inviting a developer.
   
7. When the whole feature is done, book a feature retrospective to reflect on the work done; what do we want to bring to future features, what could be improved, what should be avoided, new ideas, lessons learned, gaps in knowledge and so forth, both regarding the approach and the feature tested.
   
8. The output of this meeting is an end report. The end report is basically the wiki page with ending notes added. The receiver for the end report is integration testers, testers testing the same/similar features as well as stakeholders interested in the work done.
   
9. Anything in this process is subject to change if the context suggests so.

The example above is stripped from everything company specific but that more concerned how things were solved in our specific context anyway. Also the original process included a short description of how to actually work with exploratory testing in a way that provides visual structure and a bit concerning flow. It had it's flaws, sometimes based on lack of experience and sometimes on context (like how well we had to keep track of test ideas since test data in this form was sometimes requested by costumers), but it was a tremendous improvement.

By the way, the spreadsheet was later replaced by an 8 hour PHP hack... imagine being able to improve your situation by replacing a tool costing thousands of dollars with an 8 hour inhouse hack. I felt pretty good about myself after that day.

November to December - People finding out
Up until this point very few seemed to care about our work. Some liked our new approach and the qualitative reports we created was appreciated but nothing more.

Suddenly someone noticed we had left out basically everything described in the official test process. Our work was questioned, our judgement was questioned and people demanded we fixed our "mess" (document test cases done for instance).

Luckily for us we had prepared ourselves both to explain our decisions, how the new output fitted in the old system and how traceability was still achieved. We heard some grudges but since the testing in the feature seemed to be a success and the transition towards agile was in focus it was actually received rather well. Especially my boss at that time, and the two test managers, deserves a lot of credit for their reactions (they also knew to some degree what was happening in the team as we stopped making most of it a secret when things worked well again).

Today - Trying to support from the outside
The brilliant colleagues I had, including Saam, keeps pushing for the ideas we introduced. I try my best to support them but this blog post reminds me I could improve on that.

Important to note: The company in question has developed a lot since this story happened so it's not in any way representative for how they work now, it's just a story about change, nothing else!

Lesson: Breaking rules should not be your first option
If you think what I did was irresponsible, I definitely agree. I do think you sometimes have to bend or break rules to make change happen but it should really be your last resort if the stakes are high and done with great care. Speaking to whoever is in charge and change things that way is generally a much better idea... what would had happened if Saam never joined the team for instance?

If it's completely impossible to change something without going behind everyone's back, a new job is probably a much better option.

Lesson: You can change most things
To continue on my previous lesson: Notice that I've never found something that with the right skill, time and effort, didn't seem possible to change though. I know it exists (e.g. strictly regulated industries) but it's rarely the problem. So trying over and over again, staying naive at a healthy level, is a great skill/trait.

Lesson: Experience is invaluable
If someone tells me:
- This idea with lightweight test processes are just bullshit
My answer would be:
- Let me tell you about when me and Saam...

Lesson: Testing requires skill
I knew the theory about how to organize testing in an exploratory way already in June but it took me many hours of practicing to really become skilled enough to perform efficient exploratory testing. It's like driving, you can't learn just by reading, you actual have to practice (but reading can speed up the process).

Lesson: My unsuccessful experiments were just as valuable as my successful ones
When I started to reflect on my work during my vacation I used my experiences so far to find problems to solve. When doing this I developed an understanding of each problem that a successful experiment probably couldn't had provided me. This was useful later when I ran into other problems that I could somehow relate back to problems I already had encountered.

Lesson: Reflection is just as important as planning

Reality constantly crushed my plans but in the beginning I was so focused on getting forward I completely neglected this. When I finally started reflecting, it became an invaluable tool to keep me on track. Plans didn't take surprises (changes, my misconceptions, my simplifications or my lack of understanding) into consideration. All those required extensive reflection to figure out as I learned more and more.

Lesson: Adaptation is key
I tried to just pick an interesting concept and force it into my context, this failed badly most of the times. When I started to look at what I was trying to implement to see where and how it fitted, my attempts to introduce things suddenly became efficient. It's like solving a jigsaw puzzle with the difference you have an unlimited amount of pieces, only a few can fit in a good way, when you actually find a good piece you still need to cut it to make it fit and as you solve the jigsaw puzzle the picture you try to make changes... so kinda not like a jigsaw puzzle at all but maybe that gives you a useful mental image.

Lesson: Learn from others' experiences
One thing I used when trying to turn things around during my vacation was things other testers had told me, stuff like "we did like this but it failed due to that". Exploratory testing was nothing new at the company, great testers already used it in secrecy, they just couldn't solve the problems they had run into trying to fit it into our process (problems like knowing what they had covered and traceability). What they told me helped me realize problems I had to solve and raised questions I had to be comfortable answering as well as provide solutions to problems I already knew about.

Lesson: Discovering my behavioral warning signs was great

Today I know I'm in trouble when I for instance stop sharing things with others or when I start to give very many reasons (rather than a few very clear ones) to why we should do something. Both are signs I'm either off course or at least don't know what I'm doing good enough. The latter can sometimes be a necessary step but at least it's good to be aware. When I discover one of these signs I can immediately stop what I'm doing and start reflecting to ensure I'm not drifting off in a dangerous direction.

Lesson: Being the only judge is dangerous
In my team I was the only one with testing experience (until Saam joined). This led to the rest of the team trusting my judgement when it came to decisions concerning test (which is to some extent reasonable). As soon as Saam and I started to work together we began questioning each other's opinions, ideas and directions, helping us to stay on a healthy course.

Lesson: Reasons are not proof
I can provide reasons for pretty much everything (including why you should jump off a cliff) but that doesn't mean everything is right (including the part where you jump off a cliff). I argued for my decisions early on, providing tons of reasons, even though I was way off. Now I know better than to accept peoples reasons, instead I look for lack of real life experience, generalizations, false connections and other assumptions (including my own).

Lesson: It's important to have someone to discuss with
It's no coincidence everything started to improve when Saam and I started to work together. When I was alone (Saam came to similar conclusions) my creativity dropped, my willingness to stand up for the changes I wanted to perform decreased, my motivation dropped significantly, my feeling of trust in my results dropped, learning was not happening as rapidly and finally I got stuck a lot more often and for longer duration (even when Saam couldn't help me it started the process of involving someone who could cause obviously I wasn't stupid asking whatever question it was). Having someone to quickly turn to whenever I needed a second opinion or some quick guidance was key.

Lesson: Exploratory testing needs support from your process
I didn't understand how crippled my exploratory testing efforts would be by the not adapted surrounding test process (like test reporting, test strategy guidelines, traceability requirements etc.). For instance the test management tool everyone had to use had no support for rapidly changing test scope and qualitative progress reporting. To even give us a fair chance of succeeding we had to work that out first.

Considering what kind of environment we actually created, my advice would be to isolate yourself. So instead of trying to change the world around you (which many will object and in the end you will probably just be allowed to change half the stuff anyway severely crippling your chances of success) try to build a micro universe where as much as possible is adjusted to fit an exploratory approach without colliding with the outside world. In agile, having an independent team with tested code as the only demanded output and working in an isolated part of the product / team owned product, could be one way.

Lesson: Education can save time and improve quality of test
In the beginning the exploratory testing didn't work efficiently. One reason, I believe, was I didn't really incorporate learning. Instead of stopping my testing effort when I felt my knowledge was insufficient I continued just putting out each fire I ran into (learning the bare minimum to continue). This was sometimes the right decision but often not as it constantly interrupted my testing flow. As soon as I started to spend longer stretches educating myself in an area (not just to solve one single problem), I also started to get a better flow and I started to recognize valuable tests I had previously not seen.

Lesson: If you can't argue for something you don't understand it
I've found a lovely exercise I wish I had performed earlier. Before I suggest a change I run it in my head trying to defend it against all the critique I can imagine. If I find a question I can't answer I try to sort that out before I continue. Notice, that the answer is context dependent so you can't just "learn" answers, you have to go through this over and over (but after a few times the loop will be faster as long as the suggestion is not too complex).

Example:
- We should remove the storage of test cases
- But without storage we can't reuse them
- Have we ever done that? For instance, it takes more time finding an existing suitable test case than to write a new one?
- But we lose traceability
- We could store session debriefs or one liners of tests we've performed, it would be a lot cheaper. By the way, when do we need traceability on that scale?
- Well, what if a customer asks for exactly what we've tested... or our test manager?
- Has that ever happened? And is it really worth all this time and storage money just to achieve that?
- But what about new tester, where will they go to learn test
- Our test cases are used for this purpose only due to lack of something better. I think the teams can manage this a lot better with pair testing and internal competence boosts. Also the test scripts don't really explain why you should do certain things partly because people stop writing when it becomes to complex (easier to figure out as you go), so the competence value is minimal as it is today.
...

Wrap up
This is one of my proudest moments, most frustrating experiences, most motivating explorations and definitely one of my most educational adventures. The lessons listed is just a brief start, we learned so much and I still learn things from reading this now.

Finally, this blog post was almost entirely written in December last year. I can now see lacks in my understanding of exploratory testing and how colored I was from only have been working at the company described. That's a lesson in itself.

Thanks for reading!

Guided by Fun

This is actually a rewrite of my old blog post: Using Fun as your guideline. Reason being that I think the idea is good but it was poorly presented and too connected to our unique context in the old version.

Background
Documentation is generally not fun, so focusing on fun would leave us with chaos
// colleague

This summer we ran an unintended experiment in my team where almost all our time was spent testing and documentation only existed as casual notes not really stored anywhere.

What happened was we couldn't explain to stakeholders what we had done, why and what our current status was. However, we did get a lot of testing done so we pretty much failed to support stakeholders but did support programmers.

Was it fun? No! Why? It was frustrating and we felt unprofessional.

So we added some very basic documentation. When we encountered questions we couldn't answer, we first asked ourselves: Is this question really something we should be able to answer. If yes we looked at what we had to improve to be able to provide that answer, if no we politely explained this (I will touch the "it's not a choice" response in further down).

Guideline
At some point we realized that generally the better we worked the more fun we had. Turing it around would be: The more fun we had the better we seemed to work. Fun was a lot easier to relate to than "better" or "more efficient" so we stated what we felt was fun, asked ourselves if it seemed like a good way to work and, if so, tried to reinforce it. Here are our results:

• More time for testing = fun
• Understanding what we do (continuous learning) = fun
• Being able to explain our testing = required to have fun
• Knowing what is done and what is left = required to have fun

In practice we try to fulfill the bottom two requirements, in an as lightweight but sufficient way as possible so we can put as much time as possible into testing and learning (association to exploratory testing anyone?).

It's not a choice

When you don't really see a point in answering a certain question the response might be:

It's not a choice

Without going into reasons for that, both valid and invalid, let's talk about how to approach it. First, never (if possible) ask for what information someone wants, ask what questions you should be able to answer. This not only helps you find alternative information that might be sufficient but it's also something not too often asked so the stakeholder can't just go back and use the default answer.

If this isn't enough try to understand what the information/answer is used for (by asking questions, not by making assumptions). This may:

• Make it less frustrating to provide (since you understand its value)
• Help you find a better alternative
• Make the stakeholder more aware of gain compared to cost

Summary

Always trying to enforce things that make testing more fun is not a solution to every problem, it's a heuristic, it can be very useful in many situations but not all. Please also notice the opposite of fun is boring, not serious or professional. Instead I claim doing something in a boring way when there exists an equally good, or better, fun alternative, that is truly unprofessional.

Using Fun as your guideline

This post is rewritten, please read the new guided by fun post first.

Very interesting statement from colleague after my presentation on Exploratory Testing long ago:
Documentation is generally not fun, so focusing on fun would leave us with chaos

Last spring I started on a new team as part of an organizational change. The goal was to create something less cumbersome and depressing than our current waterfall implementation. In our old way of working tester's have had time to prepare detailed test cases with instructions and god knows what information.

Things just went bad. We were standing in a situation where a poorly written strategy was all we had while code was handed to us: "Please test so we don't have to wait for months before faults are discovered and we have to dig into code we already dislike" (like we did before) plead the programmers.

So we did the only reasonable thing: We tested, we tested and we tested. Life was great! No public documentation! Success!

Time went by and the first hand over was made between me and the other tester in the team due to vacations. Chaos! I knew to some extent what was tested but I couldn't really say if anything was "done". Soon after he handed the work back to me as he went on vacation. Chaos once again. The team even started to ask us for test cases just to get some kind of idea of what was tested.

Was it fun spending all our time testing? No! Why? Because it lead to tons of frustration and a feeling (a well motivated one) of not being professional as we couldn't answer fundamental questions about our work.

Fun as guideline
Today we use fun as our main guideline in our internal test process.

These are our criterias for fun:

• More time for testing = more fun 
• Understanding what we test = more fun 
• Knowing what's been done = basic requirement for fun 
• Knowing where we're heading = basic requirement for fun 
• Being able to answer questions about our work = basic requirement for fun 

To summarize: On top of actual testing (incl. learning anything we need to learn), we put a layer of administration, as small as possible, to reach the basic requirements for fun.

In practice

• More time for testing
  Exploratory testing, removing anything that does not provide value to us, asking stakeholders what questions why want us to answer rather than what documents they want.
• Understanding what we test
  Experience database, ask questions, ask for presentations, involve programmers in testing discussions, read, discuss, pair testing 
• Knowing what's been done
  Test matrix (I'll explain that in a later blog post), SBTM inspired team internal test management (still have a lot to learn about SBTM though) and (at least) weekly test discussions
• Knowing what's next to test
  Compact test strategy, test matrix  and (at least) weekly test discussions
• Being able to answer questions about our testing
  Test matrix, bug tracker, backlog, test strategy

Things we fell we've achieved

• We are more motivated than before 
• We are a lot more efficient than before 
• We know what we're doing to a much higher degree (surprising result!) 
• We constantly learn things about testing and our product (= more fun) 
• Less frustration 
• We understand the code better and can write our own corrections / add testability ourselves more often (in our case this is a bit special since our product supports injecting assember instructions in runtime). 
• The test matrix creates more interesting questions than the test case progress reports ever did (once again: I'll write a post on our Test Matrix later) 
• Minimal waste upon changes. This makes it far easier to work in an iterative way 
• More lightweight administration makes test more available to programmers 
• Quicker feedback to programmers 
• So far we think we've found more important bugs this way, especially important bugs that are indirectly a consequence of what we've implemented (impacts we could not had anticipated) 
• We take responsibility for our testing to a much higher degree ("we probably triggered that case" is not enough anymore, instead we learn how to do it, implement the necessary testability or admit it's not tested and explains why). 
• We've stopped considering issues as something we need to "revert from"/work around and rather consider them as a chance to learn and improve. We've also learned that we can't really know for sure what we actually need before we've lost it a valueable lesson.