Time estimates in testing, part 2

Intro

Before I begin: Thanks to feedback from especially Jari Laakso, I've found several flaws in my first post which was what I hoped for when I published it as the flaws help me understand estimation better. I will highlight some of them in this post but I also want to point out that part 1 has been updated several times since I first published it to clarify or correct content.

How to interpret the graphic below

Do notice that these are not polarities where you switch from one to the other, instead they are continua. I will however describe one rather specific scenario for each box for the sake of simplicity, a chance for you to see the differences between "each corner". My hope is these examples can work as inspiration/help for you to think different.

One final remark: I will provide all these examples as cross-functional scrum teams consisting of only developers and testers. Of course this is just one of many ways a team can be set up but I'll use it once again for the sake of simplicity.

1 - Estimate testing stories

Separated planning with detailed estimates.

In this scenario testers create their own stories, estimate them and get them prioritized as part of the regular scrum flow (the cross-functional team will now have a mix of stories relevant to developers/testers/others). This allows testers to create their own plan optimized for testing needs and I find it easier to motivate developers to help out with testing as needed as the only content left in the sprint might be test work (and this is very much visible) but...

Since planning testing even a week forward is rather hard in most scenarios (open bugs blocking stories, open bugs delaying stories, late code deliveries, level of quality differs from the expected etc.) I find this approach somewhat chaotic (even though you might expect the complete opposite).

A second challenge with this approach is the risk of creating a gap, within the team, between testers and developers. This, I think, is because if you actually try to follow the plan (test stories) you will lose some of the flexibility to drop something in favor of something else in order to minimize the feedback loop between feature implemented and feature tested.

Finally my experience is it takes a certain skill as a tester and/or a certain group of developers to actually make developers engage in the test part of the sprint planning, further increasing the risk of a gap between testers and developers.

For me, session based test management is an approach to exploratory testing worth looking into if you want a separate test planning, with estimates, without making it too expensive.

If you choose this approach, I would strongly recommend you to share sprint plannings, backlog groomings etc. so that everyone sits down together. It's easy to "optimize" so that testers have their groomings or that developers don't engaging in those, which is like asking for a disconnect to happen.

Example of a dangerous archetype potentially advocating for this approach for "the wrong reasons":

The project manager/product owner/manager/test lead who pretends/thinks testing can be controlled.

2 - Estimate dev stories

Shared planning with detailed estimates.

In this scenario testers and developers share stories (DoD includes both implementation and testing). A story is estimated both in regards of testing effort and developer effort for a combined estimate. A story is considered open until "someone" decide that the implementation, testing and bug fixing is done. That "someone" is typically the concerned testers and developers together but if they disagree or the estimate turns out to be way off, a product owner or similar might have to step in.

My experience is that this fosters a very close bond between testers and developer which typically has a positive effect on e.g. tester status, information flow and code related testability. Other benefits are developers have a better motivation (and help) to test themselves as their work is not over until testing is "finished".

However, since the testers are normally the minority it's easy they become the ones always losing votes and debates which may impact how they test, when they stop testing etc. and often not in a good way. Also since the testing is so closely tied to the development things like integration testing, general quality assessments (e.g. security, performance, consistency, test legacy for support of changed context etc.) and regression testing may be missed (regression can be helped by developers taking a more active role in creating automated smoketests but that's just a part of it).

One last problem is if you actually create new tester-developer hybrids and the pressure builds up (important release, budget cuts etc.) teams might trick themselves by moving more members into developing and tamper with their normal quality standards (which in a few cases might be the right choice but often not in my experience, often because the "temporary need" becomes permanent).

If you choose this approach, I would strongly recommend you to introduce activities such as system tests or release tests. In this activity, also involve developers as part testers, part bug fixers, so that a gap between developing and testing isn't created (testers not being able to provide feedback quick enough in the new sprint). A different method is to have teams solely dealing with integration testing (more waterfall like/support team) but for this to work there has to be strong information channels established between the cross-functional teams and the integration test teams.

Example of a dangerous archetype potentially advocating for this approach for "the wrong reasons":

The R&D manager/developer who doesn't really understand there's a need to test anything else than the code that was just written or the developer who rush the testing part since "that's not that interesting".

3 - Follow development planning

Shared planning without detailed estimates.

Very similar to the above but without time estimates of the testing. The difference that happens is you save a bit of money (less administration) but testers are a bit more detached from the development process (but not necessarily from the developers) and if estimates can actually be somewhat accurate you lose that as a planning benefit/potential delay indicator; in this scenario sprint plans assume testers will manage time limits if developer manage these which is of course not always true. But do remember that estimates are often not that accurate to begin with (described in part 1) so the actual effect of no test estimates might be small, personally I would most often worry more about the potential disconnect between testers and developers.

If you choose to approach testing like this, I would strongly recommend you to closely monitor so that testers participate actively in all the typical planning activities (groomings and sprint plans). This is to help build a team where testers and developer keep working closely together. The goal is to  "just" remove the part where testers perform time estimates, nothing else, in the planning meetings.

Example of a dangerous archetype potentially advocating for this approach for "the wrong reasons":

Testers who simply dislike administration and don't even care to try estimates. A reason why this might be very troublesome is also because these testers could easily contribute to the disconnect between testers and developers in that they might want to use the removal of estimates to skip out on "irrelevant meetings" where key implementation decisions are taken.

4 - Report on/off track

Separate planning without detailed estimates.

In this scenario testers attend backlog groomings and sprint plans to stay up to date with the planning and the product as well as get attention on testability needs; but they have a complete separate planning done together with developers (costs time) or with testers only (may hurt developers' understanding of/respect for/interest in testing). From a project planning perspective testing "just exists" and whenever the team suspects there's a risk for the testing/bug fixing to be late this is reported and dealt with just like any other case where estimates are wrong or development delayed.

This separate planning would happen as soon as possible after the sprint plan is complete (so that the scope for the sprint is known). Testers discuss if there are any other activities they should plan other than the regular testing of stories. They also look into if stories should be tested together to avoid costly retests. This makes it easier to take testing needs into consideration without messing up the development planning and testers can make rapid changes in their plan without impacting development stories.

This has some interesting benefits but there's a great risk of testers forming a subteam within the team potentially hurting the developers interest and effort in testing (information flow should be less of a problem since groomings and sprint plans are key to be able to do the test planning).

If you choose this approach, I would strongly recommend you to leave a bit of time open for unplanned as bugs not related to development stories are more likely to be discovered and help testers need might not be taken into consideration when planning and thus that extra time has to still exist when testers need help. This might seem stupid but if you've been in a software project you know some sudden needs from testers (blocking bugs, needed support tool updates etc.) are hard to predict at the sprint start.

Finally the name for this is approach is a bit strange. If we're not estimating testing (at all) we can't possibly say if we're on track or off track. I thus want to again stress that in reality we always make estimates but rather than debating whether one of many tasks should be estimated to 3 or 5 hours we look at the whole picture and make a rough estimate if we'll make it or not in time.

If you choose to approach testing like this, be careful. Putting testers with low communication skills, reporting skills or unwillingness to make work transparent, in a team like this can lead to isolation and a negative mythic image of testing where developers neither really understand nor care about testing ("they don't seem to care for the planning and the team so why should we care about them").

Examples of two dangerous archetypes potentially advocating for this approach for "the wrong reasons":

Waterfall accustomed testers or developers who still haven't figured out how to work with testing in scrum and other agile approaches. Also testers who don't want to/have the ability to make their work reasonably transparent.

The actual implementation

In a real life scenario we move a bit in between two or more "squares"; for instance testers will sooner or later need to do work that only benefits testing, so completely following the developer's stories is a myth (or an example of a dysfunctional team). However, testers not feeling like they are allowed to test things outside the current development scope e.g. because of a big pressure to close stories, is something I've come across multiple times and with that said my goal is to highlight options. The examples in this post is meant to help you see alternatives; they're not models, ready to implement.

My hope is part 1 helped you understand the dynamics of estimates and planning while the second gave you some practical ideas to experiment with. Like in part 1 I want to emphasize that this is still thoughts in process but I can't wait forever to share them if I want to get feedback...

Finally: Thank you once again Jari Laakso for all your feedback on the first post!

Time estimates in testing, part 1

Why this blog post

Activity planning and time estimation in testing is complex to begin with; combining that with cross-functional teams (having to actively take other's planning into consideration) and I've seen a lot of confusion. In this blog post I hope I can arm you with some useful ideas on how to deal with activity planning and time estimates in testing.

... and that turned out to be a lot bigger task than I anticipated so in this first part I will speak about the more overall concepts and in the second part I'll provide more concrete explanations of each approach (see the graphic below).

Key attributes

When I started listing various ways I've been working with estimates I continually came back to two variables that seemed central:

1. To what degree does testers and developer share their planning?
2. How much trust and effort is put into time estimates?

For this reason I made this little graphic:

Do note that this is still thoughts in process so I do appreciate any feedback on whether this makes sense or not.

Separate planning vs Shared planning

I've been in teams where testers were only or almost only expected to test what had just been implemented (as in testing the exact work described in the implementation stories thus sharing the planning with developers) as well as teams where testers were expected to create their own planning based on the testing needs; both extremes come with their own benefits and drawbacks:

Shared planning:

 + less administration (as in cheaper)

 + closer relationsship between testers and developers

 + quick feedback loops in general

 + easier to get focus on testability improvements in the code

Separate planning

 + more focus put on regression, system (e.g. consistency) and integration testing

 + easier to get work only affecting testers prioritized

 + planning can be optimized to fit testing

Two dangerous mind traps I've experienced:

Shared planning
Risk: Testing seen as something very simplistic where you just test the product's new features with no or little regard to the big picture.

Negative effect: General testing activities, such as overall performance testing, integration testing or regression testing are down-prioritized.

Implications: Bugs related to e.g. integration, unforeseen legacy impacts or product inconsistency.

Coping: To deal with this it takes testers or other team members who are good at advocating for testing in general and can motivate why certain test activities makes sense even though obviously impacting code is not changed.

Separated planning
Risk: Testers and developers move apart and create two separated subteams within the team.

Negative effect: Impacts information sharing, understanding of the other's work (which in terms may impact the respect for one another's profession) and developer's losing the feeling of responsibility for quality.

Implications: Prolonged feedback loop (code developed to bug reported to bug fixed to fix verified), worse quality on code handed over from development, testers lacking important information making them less effective/making them take bad decisions.

Coping: Well functioning team and social management, testers who're good at communicating their work and good at speaking with developers on the developers terms or vice versa, developers putting a great interest in testers and what they do.

I'll elaborate a bit more on this in part 2.

Effort put into estimates

Let's talk about the two extremes first.

When we put a lot of effort into estimates we may for instance split the work into small tasks to make them easier to estimate and try to give each an as exact time estimate as possible; e.g. "these 6 tasks are estimated to 3, 4, 6, 3, 1 and 9 hours so the time to test this feature should be around 26 hours".

Little effort are put into time estimates when we for instance accept an external time constrain, e.g. "get the testing ready for the release" or "finish your testing within this sprint" and we more inform the one setting the time constrain what we roughly think the quality of the testing will be at this point.

Important: "Quality of the testing" in this case refers to how deep/well we expect we will be able to cover the various features. This is not the same as the "quality of the product". Why the latter is not what is (typically) talked about is because that oe is way too complex to estimate and often out of what we can actively influence. For instance it's not our call if a bug we've identified should actually be fixed, that's up to the stakeholder(s).

At this point the former might seem more reasonable but it's not that simple...

Accuracy of estimates

There's no "done" in testing. This is sort of true for development as well, we can always continue to improve the code itself or polish the actual feature, but at least we can observe the product and say "this is good enough" and, important, that state of "done" can be fairly well-defined/described.

Even though some try to pretend it's not true; testing does not at all work like this.

Testing happens inside you head, development too (to a large degree); However, the product of testing also stays in your head and is relying on your ability to communicate it while the product of development can be easily observed by others. This complicates estimates since we cannot keep discussions around a fairly well defined end product; instead we have to leave it to everyone's interpretation of the task and do our best to communicate our own interpretation so we roughly estimate the same thing at least. For this to work at all, tester's first need consciousness of what they actually do, the necessary communication skills to explain this to others and the receivers must have enough understanding to interpret this message correctly. No human on earth (as far as I know) do even one of these things flawlessly (aka: communication is hard).

With no well defined "done" everyone will have to rely on their interpretations of what they think needs to be done and what they think the receiver asks for. That in terms will impact the estimate's accuracy but this's just part of the problem...

... the other part

On top of estimating a task we cannot clearly define we also have to estimate something that is inherently plagued with great levels of uncertainty:

The time it’ll take to test something depends on external factors such as the quality of the code, the complexity of the chosen design (which is often not set when estimates are done), the stability (up-time and robustness) of necessary environments, the help we'll get to correct problems found/blocking etc. To continue the reasoning the quality of the code depends on things such as how hard something is to implement, the skill level of the developer(s), how stressed they are (if a developer has a tight schedule she can typically stress the design a bit to get it out the door in time which in terms affect the quality which in terms affect the effort required to reach the abstract "done state" when testing) and stress level depends on even more factors. I can go on but let’s just say the time it takes to test something to the “abstract, hard to reasonably accurately define, level we desire” depends on so many external factors that the estimate's uncertainty is huge.

Bullshit!

I once told a project manager this and he replied: "Bullshit! My testers ace their estimates almost every time!" (in a product with pretty crappy quality and expensive testing should be added). And here comes a conflicting truth:

We can't predict when we'll be "done" but we can be "done" by Wednesday if you want.

In development we need to produce some kind of agreed upon product based on, at least, our explicit requirements. Remember how testing did not have a defined "done"; well taken to its extreme we can (almost) say we're done at any point in time since it's unclear what done is anyhow: "This is the findings we've made so far (and are able to communicate). You told us to be "done" until today so we are "done" but our professional judgement is this needs more testing as the following areas are still considered unstable or only superficially tested...". In a sense we always do this, estimates or not, but don't even spend time trying to guess in the latter case, instead we do our best to put us in a good enough state when the time is up.

... and before someone talks about bad attitude or lack of professionalism; this is of course not how it's practically done. Some level of estimation is always performed but rather than saying: "We think this will take 10 hours, this 5 hours, this 2 hours and this 8 hours." we might say for instance "We think we're on track for the release but the quality at this point is unusually low so we're a bit worried this might change".

Planning based on time or effort

This topic is too big to cover in this post but a quick rundown.

When planning our testing we can either focus on effort or time. Effort means "we want to reach a state where we/someone else feels confident about this particular part before we leave it" while time means "we want to spend the fixed amount of time we have in a way so that we've looked at everything at least briefly". In the former we're doomed to not be "done" if something runs late and leave the project management with little choice but delay e.g. a release if they don't want to take on a considerable risk (leaving parts completely untested). But this also allows us to argue for the need of more testing better by saying "we haven't even looked at these four features" and we'll spend less time revisiting areas having to "relearn them" as an area is left more in a "done state".

In the latter we will hopefully have found at least the most obvious, serious problems in all areas of the product and can thus make a general assessment: "our testing is fairly shallow in most areas and we would like to spend more time with the product but from what we've seen all of the new features seems to be okey at a basic level". The drawback with this is it's harder to argue that we actually do need more time to test if stakeholders aren't well informed and comfortable with the concept of "uncertainty" as well as a greater risk of needing to revisit areas for further testing.

How does this relate to estimates? Well in my experience the effort approach prevails when we have estimates and plan our work around these actual stories since "we shouldn't" have too many open stories at once and when they are closed, they are closed so we have to "complete them". In the same way the time approach is more common when we skip estimates (my experience), at least when in combination with a separate test planning. If we have a set deadline (e.g. sprint end or release date) we can more naturally plan in a way so that we've at least looked at everything once.

I say this because most stakeholders I've been in contact with seem to prefer the time approach but still argue for the importance of estimates and these concepts seem a bit contradictory, at least in the context of testing. One last note: The contradiction problem still applies to development: If we get a bit of everything ready we will at least have some kind of product in the end but as described earlier: Since more has to be in place for a piece of software to make sense combined with that estimates should be easier to get accurate enough; the contradiction is not as big of a deal (my biased opinion, feel free to challenge this).

Wrap up of estimate or not

To wrap up:

Since time estimates of testing will always be plagued with great levels of uncertainty, no matter the estimation effort, the question is if detailed estimates really provide enough additional value to support their cost (both time and the risk of misleading stakeholder to believe we know when we'll be "done"). The "ideal" level of estimation is also highly context dependent, the context changes over time and we can't really objectively measure whether our approach is good or bad; so we'll have to experiment and rely on our ability to observe/analyze.

... and finally don't confuse time estimates with planning; Planning is made in both cases and the effort spent on planning has no/little correlation to the effort spent estimating (my experience).

Stay tuned for part 2...