What happens if...

Intro

The quality categorization below derives from the work of James Bach / the Test Eye.

What happens if...

... we don't care about security

Internet is playing grounds for brilliant people who like to figure out how to get passed applications' moats or bring them down. Leaked credit card numbers, leaked personal or other sensitive information, leaked passwords, malware on the customers' computers, defaced company websites and services unavailable, all due to security shortcomings in software happens daily. This can lead to lawsuits, loss of customers, the company losing a lot of money or, worst case, bankruptcy.

Even if your application is not using network communication, no one will care the day someone hacks/steals a computer with (or even worse, due to) your app and get their hands on sensitive information. Also few (read no) customers will care if it was a third party product that failed or not. The "we didn't build it so we don't have to test it" mentality is simply not productive.

... we don't care about charisma

The amount of applications is huge and that has changed how we pick our preferred ones. Now you can install an application, look at it, dislike its design (not professional, fresh, modern, elegant, beautiful, simple, safe... looking enough) and quickly move on to the next, without even "using it". I repeat, you can lose customers simply because your product doesn't look like the product it is!

"It's just an [pick your type of application], no one will care about how it looks" is rarely true anymore. Also remember charisma is not just "cool design features", it can mean anything from simple/elegant to looking robust/secure (we do judge an application by its cover).

... we don't care about usability

Just because someone initially choose your application doesn't mean they'll stick with it. One annoying extra step or interrupt in their work flow might be enough to trigger an "uninstall and move on". Even if customers are tolerant and the competition isn't that fierce, too bad usability can render key features useless (thus wasted money developing).

Also, usability problems might add a lot of costs for customer support as they'll have to spend time explaining things to the customers that the product should make obvious.

One big enemy to usability in my experience is programmer shortcuts. "Well the framework doesn't really support dropdowns so we had to go with five links instead". Beware of just accepting this, usually technology isn't the limiting factor even though it might feel like that when in the middle of things (my experience as a tester and a programmer).

... we don't care about robustness

A crash or loss of service, even for a brief moment, can be enough to get your application uninstalled. Also, downtime can be extremely costly for customers and, worst case, a customer may not just stop using the product but also demand compensation. No matter what, common/long down times and other robustness problems can quickly hurt the company's/product's reputation. Remember, time is money.

One mistake often done is many testers just open the application, do their tests and close it down/reset. Customers on the other hand often work for hours or even weeks without closing stuff down, can your application handle that? Are you sure?

... we don't care about performance

We are an inpatient, stressed society, if the loading of a webpage/application/service is not quick enough we'll simply not wait to see if the result is pleasing or not. Slow apps are quickly gone.

Once again some experience: If a function is really slow and can't be tweaked to go faster (e.g. a complex algorithm), maybe it shouldn't be in the product at all even though it seems valuable. Or the algorithm should be simplified or changed to become some kind of background job. All this to avoid frustrated users (if a function is in the product it's expected to work and work well!).

... we don't care about compatibility

I happen to like my browser. If you don't support that browser (support means, work better than the competitors, not "well, it does but...") I won't even consider you an option... And by the way, the world is going mobile and Windows is no longer "the only" operating system on the market.

So what happens if you "support Windows, iOS and Linux" but only Windows is actually tested and in reality your product hasn't even started on Linux since three versions back? Well, a customer could demand money back, which adds administrative costs, or compensation for lost money installing/upgrading. A company buying your product might ditch you on all platforms due to distrust or you might be headlining a lot more blogs than you wished for (or rather, not for the reasons you wished for). Simply put, it would be considered lying and lying is rarely appreciated. Also shortening the list of compatible systems due to uncertainty (due to lakc of testing) is of course a deal breaker for potential customers on those systems.

... we don't care about IT-ability

Every install/reinstall and update/upgrade (or lack thereof) is a potential source of frustration and incentive to uninstall. Make sure you avoid any unnecessary friction or even a great application may find itself deleted.

Also in bigger companies there's likely someone maintaining/supporting your product internally. Those guys tend to have quite a bit of saying when it comes to selecting applications, so you don't want your application/brand/company to be on their black list. "It seems like a great product but out IT department says it's impossible to have in our software infrastructure".

Make time

We all have limited resources and have to prioritize. What I'm trying to explain above is not "we need to double the amount of testers" but rather "we need to prioritize testing and make sure the right people focus on the right stuff".

Of course a big part of this is arguing for the importance of quality, not just for the functions themselves. I hope the list above can aid you in this.

Programmers

Many things are better tested by programmers (knowledge about what has been implemented and ability to manipulate code for targeted, temporary testability) and no matter what, a simple bug caught by a programmer and solved immediately is a lot less time consuming than if a tester finds it, has to report it, may be stopped until it's fixed/application rebuilt, the programmer has to stop and context switching likely happens for both parties. Don't underestimate the saved cost of programmers that test their own code well and (almost) always hands over high quality stuff!

Product owners

My product owner does an awesome testing job! What he does is simply making sure what he gets is what he asked for and since he wrote the requirement/specification/user story that lead to the code he knows what he actually meant rather than what can be interpreted from the text (meaning is often lost/changed when thoughts are translated to written language). His job saves me (tester) a lot of time figuring out if he actually means what I interpret he means.

Domain experts

In my current company, much of the technical complexity of our main product is in the handling of security cameras. In this area I'm glad we have a great domain expert who both tests and technically supports my testing when needed. This once again saves me time to focus specifically on e.g. security in this area, which I have more knowledge about.

Combined effort

I (let's hope so at least) have a broader knowledge about testing and potential ways to put the system in risk of failing. However, programmers, product owners, domain experts, graphical designers/interaction designers etc. know their own domains much better. Teaming up when testing is rarely a bad idea as it's a way for both parties to learn from each other and better utilize the strengths of each person (e.g. the domain expert shows various ways to feed data into the system while the tester suggests various kinds of data to try).

Not everything can be tested in isolation

Testing "what has just been developed" is not enough (which some Scrum fans seems to find provoking). Often when testing we simply need to zoom out, for several reasons:

• E.g. Consistency, flow (as a usability aspect) and data flow (as in how data is transferred through the product) requires you too look at the whole, or a big part of the, product.
• We learn new things all the time which also concerns old functionality and a difference between programming and testing is testers don't need to make expensive refactoring to apply new knowledge on what already exists.
• We test how new functionality integrates with old, and sometimes those new changes also affect how old functionality integrates with other old functionality.
• Security, performance, robustness etc. is not always efficient to test for a specific function, instead you need regular sessions to test it for all functionality in a sprint, from the last month or, once again, the whole product.
• You don't always know what existing parts new functionality impacts and programmers are not always aware of what might, even on a code level, affect existing functionality (and if they are it's not necessarily communicated). Thus we need to simply "look over the whole thing" or look at a more general area every once in a while.
• ... and when looking at old functionality we inevitably find issues we've missed before due to new tests, new knowledge, changed context etc.

What sums this up I think is: A lot of what's needed to test is not efficient/possible to test isolated. It sure sucks to correct one year old bugs as a programmer, but you need to understand: Testing is not programming nor is it "extended unit tests". To a tester, modules interact and form something together, they aren't isolated blocks with api:s... Oh, and by the way testers and programmers, never expect two sides to have interpreted an api-spec the same way.

A way to solve this in e.g. Scrum is to add stories aimed towards a kind of testing rather than a functionality to test e.g. "Security survey" or "User tests/usability" and if needed, compensate this with less testing of each specific function developed (which requires more testing work, as well as better communicated testing work, by programmers and others). Another solution is to add "system test" periods before release where the product as a whole is tested with not too much regard to what has just been developed.

Let's make this a requirement

Some of this can sure be requirements and/or added to a specification, and I'm not saying it doesn't help, because I think having people aware of e.g. the quality characteristics, writing the requirements/specifications is great. But a specification will never include everything simply because even if you have all your experts doing their best, a product changes from the spec, reality is never as problem free as planning and the context (users, market windows, staff etc.) will change in one way or another as the project progress.

So, yes, I think considering quality characteristics when writing specification/requirements is great but, no, it will sure not cover everything.

Customers never ask for these things!

True, in many cases they don't explicitly ask for things like security and usability. But it's not because they don't care, it's because they simply assume it will be there and if you do deliver and don't get any feedback about bad quality, do all your customers come back? Do they recommend you to friends and business colleagues, creating new business opportunities?

And sure, if you care less about security you might save some money and still never end up in trouble. But if that chance taking doesn't go as planned it might, as already said, put you out of business. So make sure you understand the risks really well before taking such a chance.

Summary

Customers will not stick to your products (except in rare cases, and you're not one of them) and your reputation will not be impressive if your products have low quality. And what great quality means can never be fully predicted or documented so let's not kid ourselves and say checking based on the specification is enough.

Make time by explaining consequences (thus getting the right attention), educate the people around you and making sure everyone do their share of the testing job!

Good luck!

Lessons from testing my own code

The last 5 months I've spent almost every free minute developing a website for my kids' preschool. This blog post is devoted to observations done when testing my own code while creating it. Notice many of these are highly subjective and biased but if nothing else, I hope they can raise some interesting questions.

Observation 1: Finish developing (mentally) before testing
I found it hard to do good testing when I was in the middle of developing something. My brain seemed to be more focused on how to solve the next problem rather than critically question what existed.

On the other hand, when a function was done (both function and done is ambiguous, the key though is the feeling of being able to move on) I found it much easier to change my focus to "how can I improve this?" or "what are the problems with my solution?". I often setup a few tasks, completed those and then sat down and committed to make what already existed (including older stuff) awesome, before moving on planning the next tasks.

Lesson: Finish up mentally, before doing non-superficial testing.

Observation 2: Some testing was easier to do than other
Testing I do rather statically (e.g. following checklists, like OWASP checklist, or strict procedures) worked better than more creative or "subjective" testing (no right answer, more about judgment/opinion) compared to when I work with other people's products. In the latter case I often found it much harder to see the alternatives on how to solve problems and of course what I created was intuitive, right? .)

Two things I found useful when doing less static testing was scenarios and roles. When examining the product trying to observe it from the perspective of a certain user or had a certain task to perform I could better defocus and question the decisions I had already done... a bit like I was no longer responsible for the code "I'm the news editor now, not the developer".

Lesson: Scenarios and roles, I think, helped me fight "my solution is perfect" bias.

Lessons: I sometimes find checklists thought limiting when testing others' products but it did help to observe my own more critically/objectively.

Observation 3: Only testing new functionality was not as efficient
I discovered my "examine old functionality" sessions seemed to reveal more bugs, both in number and severity, compared to sessions focused on new and specific functionality. A few reasons for this, I think, is:

1. I learned new things about the product/environment that inspired new tests and applying these to existing functionality often revealed bugs I had previously missed.
2. I got some more distance to the functions making it easier to look at them as if I were a regular user, not the creator.
3. I found my affection for the specific functionality (wanting my product to work) lowered over time. At some point it was like "I don't want this to work perfectly just to show to myself that I've gotten better as a programmer".
4. Naturally I found it much easier to observe inconsistencies and problems with flows when looking at the product as a whole, not at a single function.
5. I found it much more motivating to "play around" with the whole product and following my own energy seemed to make me a better tester than if I tried to "force myself" to test a specific part.

Lesson: Instead of spending all my time testing new functionality, spending a short session on new functionality and then add longer (or several shorter) general sessions every once in a while was a more effective way for me to spend my testing time.

Notice, I've observed a similar pattern at my job. Although, when being a tester in that context I also have to consider the fact that new functionality will be delivered to customers so I can't "save testing for later" as much and as a programmer I do prefer to work with code I've recently written. Still, scheduling time to just generally go over the whole product has often proved to be a valuable practice for me even when there's a lot of new functionality to look into!

Observation 4: Testing my own code really helped me think about testability

I think any company would benefit from pairing a programmer with a tester, not so much for the tester to get quick feedback on bugs or help with how some functionality is meant to work, but for the programmer to see what parts are hard to observe or control (read: test). Turning it around, as a tester it has been interesting to play with what's actually possible to achieve. Also, when I have the code in front of me I noticed I do a lot of things I would never had done on a "regular testing project", like disable functionality to enable better/quicker testing of back end stuff and making printouts I would normally "have to live without".

Testability, and especially ways to achieve/improve it, has been one of the most interesting lessons learned during this project, both from the perspective of me being a programmer and of me being a tester!

Lesson: Testers benefits from observing programmers/program as it helps understand what's possible and programmers benefit from observing testers/test as it helps understanding what makes a product testable (not to mention what strange ways users can "misuse" their creation .)

Observation 5: Programmers do a lot of testing better than testers
I found knowledge and control over the code invaluable, especially when testing "functional correctness":

• I can manipulate code as I test
  e.g. remove client side form validation to quicker test server side validation
• I know about technical solutions affecting testing
  e.g. due to a technical limitation we need to handle two seemingly identical requests differently
• I know about the full implemented feature set, including "bonus functionality"
  e.g. no one requested drag and drop but it came with the framework, still it has to work
• I know about the code quality/problems
  e.g. the user form handler is very messy, give it some extra testing attention

Communication can help much of this but it's hard as a programmer to know what a tester needs, it's hard as a tester to know what you need when you don't know what's available and it's hard to be fully aware of what you actually take into consideration when testing no matter if you're a programmer or a tester.

As the programmer you're of course likely to agree with your solution and test for the things you've already built protection against, simply because those are the problems/scenarios you're familiar with/have identified. Thus a second, third and forth opinion is important but don't underestimate the awesome bulk of unique knowledge the programmer possesses!

Lesson: Programmers testing their own code is extremely important, much of the testing they do will not be done by a tester and/or as efficiently.

Lesson: Pairing the programmer with a tester/test coach is probably awesome (in fact, I think it is).

Observation 6: Mixing up customer value and technical practicality

"That checkbox is not vertically aligned but I know it's really messy to fix so it's not important".

This kind of mind traps I found hard to fight against, even though I'm normally a tester.

The only real protection against it, that I found, was to force myself to solve any problem I found. In the beginning it was time consuming and demotivating (slowed the progress quite a bit) but soon I noticed it greatly improved my "base quality" as many of the problems were reoccurring until I dig in and learned how to solve them.

Lesson: Extremely low tolerance to faults really helps you improve! Of course I've had to let a few problems slide but I've always given them significant time, weighing the options/workarounds and tried to find consensus that a problem is impossible/not worth solving within the professional community. The stuff inconvenient to solve, technically, often provided me the big programmer "a-ha moments" when solved.

Summary

Forcing programmers to test their own code more will not make testers obsolete as well as adding more testers won't solve the problem where programmers aren't given enough time to test their code. Both are important and both add unique elements to testing!

I think programmers can severely affect their ability to test by how they behave/plan. For me, testing intensively while in development and trying to make "objective judgments", didn't work very well. Checklists and waiting with non-superficial testing until "mentally done", seemed to greatly improve my ability to test my own code.

So can programmers test their own code?
Yes, and they should but not alone for best result, that's my experience!

Agree?

The ideal testing project

To get to the good stuff quickly without much explanation of why and how this list exists, jump to the "How to use the list" chapter.

What?
This "exercise" is rather simple:
Describe an ideal testing project

So first off, there's a lot of ambiguity in that statement. Ideal? To who? In what context? Testing project? Means what? What are the boundaries? Valid in what context?

Exactly what this means is not so important, and I'll explain why in the next chapter. But to give you a good enough idea to understand this post: "Using the experience you have, list the actions/rituals/behaviors etc. you think would make for the best possible testing". That still leaves you with a ton of ambiguity but hopefully you agree that's not the bigger issue here when I explain...

Why?
I often find when I try to look at my current context and how to improve it, I limit my thinking. By trying to list "ideal attributes" I can look at each of them and say: "do we do this today, to what extent, is it ideal in our context and how would the ideal scenario look in my specific context". Those questions, for me, tend to break that mental limitation and it's, once again for me, especially useful to see those, individually insignificant changes, that together really make for a huge improvement.

How?
Feel free to skip to the next chapter if you don't care about how this list was created.

I've tried this "exercise" three times. First time was in a workshop at Securitas Direct/Verisure on testability. One question was: "how can we improve our testability" and my brilliant colleague Niklas Grönberg, suggested we, instead of thinking about what we had, should think about how we wish things were. It generated several great ideas and everyone, no matter of current context, could add to the list.

Soon after I suggested trying the same thing on a Transpection Tuesday with Helena Jeret Mäe but focus on the project in general. We spent roughly three hours adding to the list and while doing it sharing ideas on how this could be achieved in our respective contexts.

Finally I tried the idea as a group exercise at EAST. In this case 8-10 people shouted out ideas that were discussed/further described and then added to lists on a whiteboard. When we were done everyone got a few minutes to look at the lists (we filled two whiteboards) and in an open forum we shared our key takeaways. I'll add a few notes from that at the end.

To support/guide the thinking a bit I added a few categories both when running it as an exercise on EAST and with Helena. This also (I think) helped explain what I was looking for. You will see a similar setup in this post.

How to use the list?
This is not attributes of an ideal project, it's attributes that to someone seemed ideal in some, to that person, relevant context. Thus you can't just take an item from the list and say: "well how do we achieve this". Instead I suggest you look at an idea and ask yourself, as described before:

• Is this relevant to my context?
• Why/why not?
• Do we do this today?
• To what extent?
• How would this ideally work in my unique context?

Important: I do not agree with all the items below. But I wanted to add everything I picked up as it was mentioned by some skilled tester as ideal in a context (s)he could relate to and thus potentially helpful to someone reading this.

Also notice some of the items are in conflict with each other. Once again something might be valid in one context or aspect of a context while something else is true in a different context.

General

• Clear boundaries/limitations throughout the whole project
  Time, resources etc. also knowledge about why it's limited and why the limit is set to what it is
• Clear and well communicated goals and milestones
  Iteratively challenged throughout the whole project
• Knowledge about users and buyers
  In bigger companies the buyer and user can be two very different groups
• Access (throughout the whole project) to users and buyers
• Knowledge about stakeholder
• Knowledge about market (e.g. competitors and market windows)
• Manageable amount of changes and quickly informed about these
• No hidden agendas
• A product you...
  ... like to work with
  ... find meaningful
  ... believe in
  ... want
• Balanced work flow
• Time to experiment
  Including product, working methods, techniques and tools
• Being included and listened to
  E.g. invited to meetings
• See the product being used
  And see that it matters to the customer
• Low level of (boring) repetition
• Some "hype factor"
• Test ideas generated from day one
• The agreed level of quality and customer focus is not lowered throughout the project
• Everyone strives to achieve better results when agreed upon
  Includes everything from quality/scope to efficiency/cost/time
• One focus at a time (no multitasking)
• Testing being a respected and prioritized activity that everyone cares about

Start of project and general

• Involved when it's a problem to solve not a solution to implement
• Well communicated why (why it's a problem, why it's important)
  This includes being presented the real why, not the "official" why.
• Kick off to meet, bond and discuss
• Being listened to regarding testing issues and general risks we see
• Everyone genuinely agrees upon what to build and why
  Also includes for instance salespeople and maintenance/support

Team

• Motivated
• Mixed group of expertise/experience
• Skilled to do the job
• A willingness to get better, both individually and as a team
• Problems and concerns are shared
• Having fun together!
• Everyone is listened to
• Respect for each other
• Working well together
• Domain knowledge
• Clear, effective communication

Implementation discussions

• Quality characteristics are discussed and clearly prioritized.
  This is an iterative process
  Stakeholders, customers, testers, programmers should be involved
  Most important is to state what is not that prioritized
• All solutions should have a known "why"
• Testable paper mocks or similar
• Customer feedback on these mocks, communicated to the testers
• Testability and relevant quality characteristics issues should be discussed as part of each solution
• Always 3 options to consider (<- much debate to "at least", "roughly" or "exactly" three)
• Key areas and lower prioritized areas are discussed and agreed upon

Test planning and strategy

• Risks collected from testers, customers, stakeholders etc. discussed and prioritized
• Discussions on how to test with programmers involved
• Test process revisited and tweaked to fit current context
  E.g. what documentation, for who and why. When/how to involve programmers etc.
• Testers, programmers, customers and stakeholders read, understand and commit to the strategy
• Strategy is continuously revisited throughout the whole project

Testing / Developing

• Feedback on errors (email, popup), rather than scanning a log
  (could be achieved with a tool scanning a log of course)
• Being able to start a flow at any relevant position
  (e.g. in the middle of a setup wizard)
• Being able to extract any relevant information from the product
• Early first delivery and frequent deliveries throughout the whole project
• Attention on bugs, not just new functionality
• Programmers being aware of and taking relevant quality characteristics into account when developing
• High quality code delivered
  Helped by code reviews, static code analysis, TDD/BDD, quick manual sanity tests, (automated) smoketests, skilled programmers taking pride in delivering great quality code etc.
• Testers pairing up with programmers, customers, stakeholders when testing
• No distance between programmers and testers (and stakeholders)
• No delays from code written to code up and running in the test environment
• All target systems/platforms/versions used by customers available to testers
• Regularly time to test with and speak to customers
• Why should be well explained and motivated for every demanded administrational action
  E.g. documentation and reporting
• Iterative reflection on processes and tools that everyone is engage in, genuinely want to engage in and given time to act upon.
• Progress is celebrated!
• Everyone is having fun!
• Testers are not the only ones testing
  Stakeholders, programmers and customers test the product as well, both while pairing with testers and alone.
• Mistakes being embraced
  Admitting mistakes should never lead to shame, everyone naturally want to share mistakes done rather than keep them a secret.
• Lots of questions asked and people taking time to answer questions
• Amazing tools that support the desired process, not dictating it
• Short optimized feedback loops
  Includes communication
• Testers focusing on testing not checking
• No stupid measurements and KPIs
• Stakeholders accept relevant information, no extra translation necessary
  Much of this is probably better explained as both parties striving to understand each other and to learn speaking each other's language.
• Team! Not individuals
• Continuously challenging goals and milestones as more is learned about the product
• Stakeholders clearly explaining why they chose to ignore certain risks communicated by testers

Finishing up

• Customers blown away
• World peace and end of world hunger
• Time to reflect
  ... and result of reflection being used as input for the next project
• Clear finish
• Getting the time to feel proud
• Being shown appreciation from managers, customers and stakeholders
• Lessons learned shared between various projects (if projects are run in parallel with different testers)

Some reflections from the wrap up during EAST

As I said before we ended the exercise during EAST with sharing lessons and ideas to bring back to our respective contexts. One observation was most of them related to communication like ideas to improve communication or information someone should ask for. Another was very different kinds of lessons were described but everyone seemed to identify something relevant to their context that they might not had thought about before, or now thought about in a different way, which to me indicates that the list can be valuable to most testers.

Feel free to share your own additions as comments below, I will gladly add them to the list!

And to all of you who've added to this list already thank you! This is not my work, it's our work! Actually I even tried to shut up during EAST! (which went... okey I guess, at least compared to how it usually ends up)

Thank you for reading!

So when will you be done?

Project manager: How's it going?
Tester: Right now we see a lot of problems in the export functionality but import seems a lot more stable than last week
Project manager: Okey, so when will you be done?

The next time you get this question, answer:

I can be done tomorrow, or next week, or in September if you want, when I'm done (testing) is irrelevant, what's relevant is how confident you want to be with the product when it's released.

When are we "done"?
We don't really decide when we're done, that's up to the people delivering something for us to test (developers, designers, people responsible for builds etc.) and the people deciding when a product is ready for the market (let's call that person(s) a project manager in this post). How is that?

First of all we don't create quality, developers etc. do, we only make a quality assessment. That means we don't control when the "desired level of quality" is reached, that's once again up to developers etc.

Second of all we don't decide the "desired level of quality" itself, that's up to people with better knowledge of the market, internal business situation etc., the person(s) I call the project managers in this post.

And finally we don't know about the actual level of quality in a product, we only try to make an, as good as possible, assessment. This poses another problem as we have to try and communicate the level of quality we think the product is at and communicate this in a way so that the person actually deciding if the quality is good enough (once again, the project manager) can take an as good decision as possible. This is typically where risk comes in: "If we release now, here are the risks I see, it's your call if we can live with those or not" and the last part, if we can live with those or not, that could be translated to "if I'm done or not".

Problem, QA
Now there are a couple of problems.

First, why doesn't testers then decide if we should release or not? After all, we have the most detailed view of the system? Well, we call that quality assurance (QA) and there's several reason why I highly dislike that concept: One is we don't know the business well enough! There might be a market window we can't miss or that will be very costly to miss, we might need more money now to make crucial investments and so on. Long story short: Quality, no matter if we like it or not, is far from the only thing deciding when a product is ready to release. And also, I strongly believe QA could, or rather would, hamper our ability to stay critical and "test minded" as we would be biased to fulfill other goals (like the wish/pressure to release).

Problem, estimates
Second I know someone is saying: "But as a project manager I can't just have a huge question mark in my time plan, I need an estimate of when we think we can release!". Sure, in my team test is currently not estimated (we do have some days between code freeze and release but they are set, and same for every sprint), instead I just report if I see problems a long the way that I think jeopardize our ability to release when planned (risk). That's one way to do it.

At my previous work we did specific estimates for testing, nothing wrong in that, but it doesn't change anything. Once again, that's for someone to be able to make a plan. In the middle of everything though, the time is suddenly irrelevant and we can only hope our assessment of how long it will take developers etc. to reach a perceived level of quality where the project manager feels confident to release, is anywhere near reality.

... Oh, and test cases, or rather pass-fail/pass-total ratios, don't change any of this, it's only a way to pretend you have control over something you don't control.

Summary
Never ask a tester when (s)he will be done, ask about current risks and what consequences (s)he thinks come with those risks.

As a tester, don't provide a "time left", it's not up to you when you're done. But do help the person asking to figure out an estimate for themselves to when the product/feature might be ready by presenting known problems, how deep areas are covered and other relevant information.

Credit
This comes from a Transpection Tuesday with Helena Jeret-Mäe.

Clarification
I received a comment and want to add a clarification: At my current work, this is not a problem and has never been (at least I haven't come in contact with it)... but it has been earlier in my career.

Rules can't stop me!

I love this picture! I think it brilliantly illustrates an important creative skill testers need: the ability to see beyond and challenge existing rules.

... And it got me thinking; what other ways can I think of that would/could fool a genie. Here's a quick list, I would love to hear your creative solutions as well:

• Wish for the ability to grant wishes
• Wish for every lamp you touch to contain a genie
• Wish that the genie will keep forgetting that it has already granted you a wish
• Wish that the genie stopped having/followed rules (seems dangerous though!)
• Wish that no wish you make counts towards the limit
• Wish there would pop out a different genie every time you rubbed the lamp
• Find out a way to express all your wishes as one wish
• Wish there was a much higher limit than 3
• Wish that 3 changed meaning to unlimited (consequences once again)
• Wish "I want to change into a new but identical person" as your third and final wish
• Wish the genie inscrutiating pain whenever he decides to decline you a wish.
• Wish... now it's your turn!

How many holes are in this shirt?

Saw this picture on Facebook and couldn't resist to add to the long stream of comments saying 6, 7, 8, .. holes:

First: Definition of a hole? You have separations between threads in the fabric for instance, do they count?

Second: Even with a definition it's impossible to say only based on that picture since you can't see holes on the back of the shirt, covered by the part of the front that is not ripped open. For instance, you can see right through so there must be holes on the back but is there two different holes or one big hole?

For the spirit of the question: At minimum zero, assuming the yellow parts are only clever design and the whole back is gone so that holes for arm, neck etc. are not really holes anymore... hard to call it shirt in that case tough.

... so what's the minimum requirement for it to be called a shirt?

You could of course continue and question things like if that's a shirt or a drawing, that a two dimensional object in general would work poorly as a shirt if you don't live in Flat Land and so on.

I wanted to share this just as an example of how many assumptions we make when giving a simple answer even to a simple question. I leave it to you to make something out of it but a start could be to look critically at simple, especially "universal", answers you get or give. What's required for them to be true and are you sure that reflects reality?

Arguing for Exploratory Testing, part 2, Reuse

Intro
You can read a bit more about this series in the first post:
Arguing for Exploratory Testing, part 1, Traceability

The topic for our second Transpection Tuesday on "Arguing for Exploratory Testing" was Reuse.

We finished with two open questions:
Can we ensure we actually repeat the exact same test a second time?
How do you actually achieve reuse in exploratory testing (when it is desired)?

Reasons to reuse tests
First we tried to state reasons someone would want to reuse test cases:

• Save time during test design
• Functionality is changed and we want to rerun the full/part of the test scope
• We want to verify a (bug) fix

Preconceptions

Looking at reasons quickly led us to some preconceptions which became the topic for a big portion of the session:

• Effort = Value
• Equal execution = Equal value
• Our scope is (almost) complete
• Reuse = free testing
• A monkey can run a test case

Preconception: Effort = ValueSince we've invested so much time (as well as money and prestige) in writing test cases they must be worth more than a single execution.

• Even if presented with clear evidence we may reject it to defend out judgement
• We may overestimate what a test case is useful for (we want to get the most out of our work)
• It's my work, criticize it and you criticize me not the work! (common and unfortunate misconception)

It takes a lot of self-esteem to say: "Yeah I screwed up, could you help me?", especially in an environment where mistakes are not accepted. Notice many of the "so how can we make the most of this mistake" still communicates "so you made a mistake, now you'll have to suffer for it by telling us why you are a failure". It takes a lot of work to change this.

Preconception: Equal execution = Equal value

Let's say we execute the exact same steps in a scripted and an exploratory way, wouldn't that be two identical tests? We believe not.

1. Your goal differs. With test cases your goal is to finish as many test cases as possible (progress). That's how you measure "how much testing you were able to perform". In exploratory testing you are judged based on the information you provide thus you should be more incline to spend a few extra minutes observing/following something up even when it's not "part of your test".
2. Your focus differs. When you have a script you have to focus on following that script. In exploratory testing your goal is typically to find new leads to base the next test on. That means in one case your focus is on the product and in the other on an artifact. Think about the Invisible Gorilla experiment.
3. Scripts easier bias you not to observe. In a script you typically have verification steps e.g. "verify X=5". We believe this could bias you to not be as observant during the other steps: "this is just setup so nothing should happen that concerns me".

Preconception: Our scope is (almost) complete
We know a feature's boundaries (specifications, requirements) so when we set the scope for testing we can, and usually will, cover almost the entire feature.

• We can't know the boundaries of a feature:
• We will impact and use other components not "part of the feature" e.g. other code in the application, the operating system, surrounding applications, third party plugins, hardware, hardware states etc.
• We interpret planning documents differently, adding parts, discover things we couldn't had anticipated, correct mistakes or interpret something differently than intended by the author and/or interpreted by the tester.
• We can (almost) always tweak a test a little bit (e.g. change input data or timing). But testing all combinations (we recognize) is way too expensive. Also there are usually so many ways an application can be misused (intentionally or unintentionally) that even with a ton of creativity we can't figure out them all (ask any security expert .)

So our scope is basically a few small dots on a big canvas rather than a well colored map. But those dots are (hopefully) carefully selected to protect us from the greatest risks we can anticipate. Still, they are only dots.

As testers we easily support the preconception of full coverage by answering questions like "do we cover this feature now?", "is all testing done?" etc. with a simple "yes" or "almost". The more accurate answer would be "we cover the most important risks we identified limited by our current knowledge, time available and other constraints", but that answer is not very manager friendly which leads us to...

There is a general lack of knowledge and understanding of testing in most organizations. And we decided to stop there since that was a way too big question to tackle at the point we got there. But it's an important questions so please take a moment and think about it for yourself: How can you improve understanding and interest for test in your organization?

A final note. Since we only cover a small part, reusing a test scope will not help us catch the bugs we missed the first time. How big of a problem that is differs but repeat a few times and it may scale up in a nasty way.

Preconception: Reuse = free testing
We've already written the test case so wherever it's applicable (which should be self-explanatory) we can just paste it into out test scope and voí la! Free coverage!

The big issue here is the "self-explanatory" part. Problem is what fitted well in one feature might not do it in another even similar one. Even without needed tweaks we still have to figure out what the test case actually does, so that we know what we have covered with it and what we still need to cover in other ways.

This process is expensive, really expensive, so sure we save time not having to figure out the test and how to practically run it all over again but consider the time it takes to find the test case, analyse what it covers, analyse what it doesn't cover, analyse how it interacts with existing test cases, analyse if something has changed that impacts the test case compared to last time and so forth.

Preconception: A monkey can run a test case

• We all interpret things differently. Click can mean single click, double click, right click (already assuming the first two were left clicks), tab and use enter, middle button click, etc. Even a well written, simple test case can lead to different interpretations.
• One thing we're looking for is unexpected behavior and it's in the nature of "unexpected" to be something we can't plan for. Thus to get much use of a test case we need to handle the system well enough to investigate and identify when a behavior is unexpected or undesired.
• We do a ton more observations than we consciously think of. These observations takes practice, focus and skill. For example, when you boot your computer you would react to a lot more things than you would add in a "boot test". Examples: screen is blinking, smoke is coming out, lights in the room flickers, you hear strange mechanical sounds, all these should catch you attention but are unlikely written down.
  
  More skill and/or focus can lead to more valuable observations: The login screen looks different, memory calculations are wrong, it's slower than usual/expected, BIOS version is incorrect, the operating system's starting mode is wrong etc.
• When we don't fully understand something we tend to write it down less detailed (sucks to look stupid by writing down something incorrect and we're too lazy to investigate every detail we don't understand, it's easier to investigate as we get there).
• When we write a test case based on specifications, requirements and other "guesses" of how the end system will work even a flawless instruction will sometimes not correspond to how the system is actually working (including when working as desired). This of course requires the person executing to be able to correct the test case thus understand both the intention with the test case and how the system works.
• If we don't understand the system we may lose a lot of time setting up fully or partly irrelevant variables to the values stated in the instructions. The immediate comment is, if we have stated irrelevant variables in the test case we've failed. Consider then that the variable might be irrelevant to the test but mandatory to the system (e.g. you have to set a valid time server). Leave that out and the person executing once again needs to understand the system.

When is reuse actually beneficial?

• We have rewritten something from the ground up but want it to externally still work the same. Reuse could save time.
• We have some critical paths through the system that can't break.
• We need to quickly regression test a feature without having to dig in too deep in the feature itself.

But remember that the one executing still should understand the test and the system to ensure tweaks (using different input values, triggering different fault cases etc.) can be made and important observations are more likely to be made.

How can we achieve reuse in Exploratory Testing
Not covered much by this particular session but a few thoughts:

• Charters
• Debrief notes
• Test ideas
• Test plans

Try creating a report specifically used as a "feature summary" including valuable operational instructions, general test ideas, impacts, lessons, problems, tools, important details, testability etc. We did kind of this at my former company where we let the test plan continuously turn into an end report as we added lessons from our testing. This would not only help when retesting a similar feature but also as educational material or test plan input for instance. Important though is to stay concise, noise is a huge enemy! The number of readers of a document is inversely proportional to the number of pages in the document, you know .)

A few notes on test case storage
First off I love this post on having a big inventory of test cases by Kristoffer Nordström.

It's easy to think something you've already created is free, but there's no such thing. Having a large inventory to test cases costs in many different ways:

• Storage
• Noise (it's one more thing testers have to keep track of)
• Another tool/part of tool for testers to stay updated with / learn / understand
• For a test case to be fully reusable later it should be kept up to date. How many refactors all their old test cases as functionality is changed?
• ... if you do, that sounds really expensive.

Summary

Reuse has it's place but be careful!

Remember reuse means inheriting blind spots, has a cost and still requires the person "reusing" to know just as much about the feature, system and testing in general as if (s)he wasn't reusing old checks.

Take care, and I hope these Transpection Tuesday notes (even though somewhat messy) were helpful!

... and of course, thank you Helena!

Arguing for Exploratory Testing, part 1, Traceability

Background

The topic for my and Helena Jeret Mäe's last Transpection Tuesday was Arguing for Exploratory Testing. What we basically wanted to achieve was to get better at explaining the pros (and cons) about exploratory testing, in a concise way, as well as identify common preconceptions about scripted versus exploratory testing.

Input

We had defined 15 subtopics such as time estimations, credibility and making sure the important testing is done. The first item on this list was traceability which turned out to be enough material to fill the whole 2 hour session.

What is Traceability

First question was: What do we mean with traceability?

Our answer: Being able to track what has been tested, how, when and by who.

Why do we want Traceability

The next question was why we want traceability. We quickly formed a list but reading it now makes me realize we mixed together traceability and claimed benefits of having a trunk of test cases. But anyway:

• External demands
• Ensure work has been performed
• Base for further testing
• Support handovers
• Create a map
• Reuse

General thoughts
One thing we got back to over and over again was: The best way (often related to level of detail) to achieve good enough traceability is highly context dependent! For example having a simple mind map with short comments is good enough for one company while another requires every session to be recorded with the recordings being stored and indexed together with session notes, debrief summaries and saved logs. It all depends!

Another reoccurring theme was: "But do we really achieve that kind of traceability with test cases". I will not bring up those discussions much in this post but expect another one on "false assumptions about scripted and exploratory testing" soon.

Terms

Charter
Charter is basically an area to test, a way to break down a big testing mission. Notice though that as you test new charters might come up so it's by no means a definite plan. Read more >>

Test idea
Typically a one liner describing one or more tests you want to do. Read more >>

Session
A timeboxed, uninterrupted test sitting, typically 60-120 minutes. Read more >>

Debrief
Refers to an activity happening after a session where the tester explains what has been done to, for example, a test manager. This also includes clarifying questions, feedback and other kinds of dialog to help both parties learn from the session. Read more >>

Recording
We mainly refer to screen recording (video, either using a screen recording tool or an external video camera) but could as well mean record audio, save logs/traces or other ways to save what has been done. A good resource >>

External demands

This refers to regulated businesses (watch the excellent presentation What is good evidence by Griffin Jones), evidence in a potential lawsuit or customers demanding test data.

Possible solutions:

• Record the sessions, preferably with configuration (device, version, settings etc.) explained if that matters. Adding commentary might improve the value as well (communicating purpose, observations etc.). This is also typically a scenario where logs/traces can be a required addition to a video recording. Once again, watch What is good evidence.
• Store session notes
• Store session summaries
• Store charters
• Store debrief summaries
• Store test ideas (assuming they has been covered by your testing)

Creating support to find old information (index) seems key as well. For this charters, time stamps and/or categories might be useful to tag your save material with.

Ensure work has been performed

First question raised was: Is this really something we want to encourage? And our general answer is no; with the motivation that people in our experience tend to do things to look good rather than do what is needed/valuable when closely monitored. But being able to know that the testers actually do their job is closely connected to credibility and transparency so still a valid question.

Possible solutions:

• Debriefs
• Recordings
• Notes
• Bugs reported (a really bad metric for this but can indicates something!)

Debriefs seemed to most often be the preferred approach. During a good debrief the person being debriefed asks followup questions that will require the person debriefing to explain the testing done. A byproduct in this process would be to ensure that the tester actually did a good job / any job at all. But once again; if your focus is on monitoring, the people monitored (testers as well as non-testers) is likely to waste time proving job has been done rather than actually work!

Base for further testing

Let's say we've finished the prepared scope or are suddenly given an extra week to test something. If we can't go back and use already executed tests as inspiration, how do we know where to continue?

Possible solutions:

• Having a bulk of charters as inspiration
• Make comments about testing you've left out in your finished charters/sessions
• Review session notes

We also brought up if there's a value of actually looking at what has been done. Often we found that the time it takes to analyse the work already done might not be worth it (information being too detailed making it hard to overview and learn from quickly). Simply exploring using knowledge we might not had had the first time or by having a different tester from when we first tested, is often more than enough to add value. After all, the time we analyse is time we cannot test (which might or might not be well invested).

Support handovers

One tester leaves (quits, parental leave, other tasks etc.) and another has to take over, how can we manage such a change when not having a set scope of test cases? First of all the new tester do have to spend some time getting familiar with the feature in exploratory testing but this is also true for using test cases since we, for instance, can't predict what problems we will run into thus can't prepare instructions for those!

But we can make it easier:

• Charters (with status)
• Debrief
• Documented test ideas with already investigated ideas being marked
• Session notes or session summaries
• Mind maps or other test planning with already tested parts commented
• Documenting lessons learned (like operational instructions)

Debrief in this case refers to a general debrief of what has been done, what we know is left, problems seen, lessons learned, where information is stored, who to talk to etc. by the tester leaving. Of course if the switch happens very suddenly (e.g. sickness) performing this is not possible and in that case it's important testers are professional enough to document what has been done (mind maps, short plans, visualizations, debrief/session summaries, charters). This is once again true for both exploratory and scripted testing.

Create a map

A bulk of test cases combined with statuses can somewhat be used to draw a map of what has been covered and what is left to test. How can we visualize this without test cases?

Possible solutions:

• Charters
• A mind map describing what has been tested
• A picture/model of our product with comments about testing/coverage
• Other visualizations like diagrams
• The Low Tech Dashboard

A few important notes:

1. You sure have a map with test cases but is it actually anyway near accurate? Say we have two equally complex functions. One takes 1 argument, one takes 10. We likely will have at least 10 times as many test cases to cover the second function. So if we execute all the test cases for the second function, have we really covered over 90% (with "covered" only considering these 2 functions)?
2. Even if equally sized, that map would not cover what we didn't anticipate from the beginning so you still need to add an up to date judgement/evaluation (e.g. "wow that network protocol sure was more complex when we expected during the planning, we need more testing of it!").
3. Scale is really important. Do we want to see Tartu, Estonia, Europe, the world or the Milky Way galaxy? We might need different visualizations to create all the maps we need (once again, think about value, how much time can we spare to keep these updated).

Reuse

Later a similar feature or a feature impacting the one we just tested is developed and we want to reuse the work previously done. How can we do this without test cases?

First of all, reuse is one of the places where test cases are powerful. However you have the minesweeper problem: If you walk the same lane in a mine field over and over, as new mines are constantly added, it's likely that the number of mines beside your narrow track start to build up while few will happen to end up in your path. Meaning, running the same tests over and over is less likely catch new bugs as creating new tests are so value quickly diminishes (more tests executed is not equal to more valuable ground covered).

What we often would suggest is to use knowledge acquired the first time as foundation for new testing to speed it up. Think about the new risks introduced and what needs to be tested based on that (like with new functionality) rather than how old test cases might fit into your testing.

Possible solutions:

• Reuse of charters
• Reuse of test ideas
• Look at old session notes / summaries
• Use old recordings (the simpler the form of the recordings the better for this, watching several hours of screen recording is probably waste)
• Start a wiki page/document/similar for each feature and add lessons learned, where to find info, problems etc. as you test.

Summary

There are many ways of achieving traceability (and similar potential benefits of test case trunks) in exploratory testing, Session Based Test Management principles seems to be the most straight forward way but keeping track of test ideas or using other approaches works as well. All have their own contexts where they seem to work best (e.g. SBTM might add too much overhead for a simple project).

All and all, if someone claims "You lose traceability with exploratory testing", ask what that person means more precisely (e.g. present testing data to customer) and explain the alternatives. Notice this is only based on our two hour discussion and there are a whole lot more to add so think for yourself as well! Also question whether you actually achieve the kind of traceability requested using a scripted approach and to what cost. Finally question if the requested traceability is actually worth its cost no matter if exploratory or scripted testing is used. Doing unnecessary work is wasteful no matter what approach you use.

Finally: There are still contexts where a highly scripted approach is likely the best option but the closer you get to a pure scripted approach the fewer and more extreme the contexts become.

Thank you for reading!

And thank you Helena, see you next week!

Going Exploratory

Prerequisites

• Test process dictating a factory testing kind of approach.
• Progress was presented as pass-fail ratios with a brief comment at best
• Test cases were saved in a humongous, very general, database application.
• Bugs and requirements were saved in another humongous, very general, database application.
• Most documents were based on big, thorough templates.
• All development was split into features, a feature was then further split into smaller packages. A large feature could take over a year to finish (implementation description written to hand off to integration testing) while packages were typically released once ever two weeks.
• The product was a huge real time system (middle-node in an even bigger system), had legacy from 1975 and was mostly written in a language older than C.

March to June - Starting out
Early in 2012 I was asked if I wanted to join a pioneering (second), self directed, cross functional team at my former job. It was the golden opportunity I had been searching for, in my wish to introduce more context driven thinking. I answered yes immediately.

Eager to put everything I had been reading about into practice I started scouting my possibilities. First mission was to figure out what "self directed team" actually meant. The answer was rather disappointing: "Well, you control the inner process but you have to provide the same output as everyone else" (documents, test case storage etc.).

I decided to interpret output a bit more... general (ask Meike about my willingness to test or bend rules). I began asking the various receivers of progress reports, strategy documents and other output, what questions they tried to answer with these artifacts. I then decided to interpreted "answering the same questions" as "same output", which would later stir up a lot of criticism but luckily noone seemed to know who was i charge of us at this point so I got away for now.

I began to wildly change everything around me, more or less removing the whole foundation on which all our previous testing relied without adding any new. For a while it seemed fine. I got shit done! It was fun, it was quick, it felt efficient, I was -exploring-!

July - Chaos
Vacation time, everyone was calm, just mentally preparing for a month of freedom.

... except anyone in contact with the testing in my team...

Everything was in chaos! In the midst of trying out everything I wanted to implement (but didn't understand), I had lost track. To make matters worse I had to, in the middle of this chaos, hand over my work to an experienced tester who would join my team starting the first day on my vacation. I scratched my head trying to figure out what I had done and what I wanted him to do. I sat down and started writing, trying to at least make it look like I had things under control. Doing this I realized my team was not informed about half of the stuff I called "how we work with test in my team". The dreamy world I had been living in dimmed away as reality slowly materialized in the document in front of me. When done I could at least breathe a sigh of relief, I hadn't screwed up everything (yet). I handed over my work and went on vacation.


Vacation
A well packed schedule was turned upside down when my youngest son was hospitalized with a severe bacteria infection. All the traveling back and forth to the hospital gave me plenty of time to reflect on my work. I slowly started to see ways of turning my mess into something valuable.

August - The turning point
Vacation ended and I headed back to work. The day I came back my new testing colleague, who we can call Saam, since that's his name, left for his four weeks of freedom. I started looking at his testing. It was also chaotic, but a lot more structured (call it professional if you like) than mine. I started to clean up my mess by attacking the main problem at hand: How to know what have been tested?

The first solution was nothing fancy, but oh so important: A simple spreadsheet with functions to test on one axis and general stuff like how the function works under load or during restarts on the other. Each box that formed had a color representing its status, yellow for unknown, red for open bugs, white for invalid combinations and so forth. It was basically a simpler version of the Low Tech Dashboard, which I unfortunately didn't know about at the time (if I were to redo it I would use the Low Tech Dashboard as my foundation). The document started almost entirely yellow (unknown) but at least I had a plan.

When Saam came back we started talking about how to work with test in general and found out we had very similar ideas. From here on forward things worked better and better, thanks to brilliant chemistry and adventurous but realistic approaches.

September to October - Upping the pace
Saam loved to dig into technical details, I loved the more overlaying principles, together we rocked! As we learned more and more and started to form a routine I began documenting the process that had emerged. It all ended up as a concise, rather visual, easy to grasp, four page process description, probably the shortest document ever created inhouse (of course not exaggerating even the slightest).

It basically said:

1. Start a wiki page for the feature. Fill it with information like; lessons learned, operational instructions and where to find more detailed information. This wiki page is updated regularly throughout the whole feature and emphasis is on keeping it as short as possible, only including valuable, up to date material.
   
2. Learn about the feature, read docs, speak with programmers and experiment with what exists already. While doing this, document test ideas and a brief test strategy. Don't plan to much but make sure good ideas aren't lost.
   
3. The moment before code is shipped to us, grab the developers and possibly a testing expert/domain expert. As simple as possible, present our test ideas so far and ask for feedback.
   
4. When the code is shipped, use the test ideas as input, but don't in any way limit yourself to only test those. The goal is to steadily add new ones as more is learned about the system and feature.
   
5. As the perception about the status of a certain area changes, update the current status (color code) in the spreadsheet and add a few comments to help explain the situation. This spreadsheet serves as progress and coverage report.
   
6. When done with testing in an area (group of charters) sit down together, look at the test ideas now documented and see if there is anything important left out based on current knowledge. Consider inviting a developer.
   
7. When the whole feature is done, book a feature retrospective to reflect on the work done; what do we want to bring to future features, what could be improved, what should be avoided, new ideas, lessons learned, gaps in knowledge and so forth, both regarding the approach and the feature tested.
   
8. The output of this meeting is an end report. The end report is basically the wiki page with ending notes added. The receiver for the end report is integration testers, testers testing the same/similar features as well as stakeholders interested in the work done.
   
9. Anything in this process is subject to change if the context suggests so.

The example above is stripped from everything company specific but that more concerned how things were solved in our specific context anyway. Also the original process included a short description of how to actually work with exploratory testing in a way that provides visual structure and a bit concerning flow. It had it's flaws, sometimes based on lack of experience and sometimes on context (like how well we had to keep track of test ideas since test data in this form was sometimes requested by costumers), but it was a tremendous improvement.

By the way, the spreadsheet was later replaced by an 8 hour PHP hack... imagine being able to improve your situation by replacing a tool costing thousands of dollars with an 8 hour inhouse hack. I felt pretty good about myself after that day.

November to December - People finding out
Up until this point very few seemed to care about our work. Some liked our new approach and the qualitative reports we created was appreciated but nothing more.

Suddenly someone noticed we had left out basically everything described in the official test process. Our work was questioned, our judgement was questioned and people demanded we fixed our "mess" (document test cases done for instance).

Luckily for us we had prepared ourselves both to explain our decisions, how the new output fitted in the old system and how traceability was still achieved. We heard some grudges but since the testing in the feature seemed to be a success and the transition towards agile was in focus it was actually received rather well. Especially my boss at that time, and the two test managers, deserves a lot of credit for their reactions (they also knew to some degree what was happening in the team as we stopped making most of it a secret when things worked well again).

Today - Trying to support from the outside
The brilliant colleagues I had, including Saam, keeps pushing for the ideas we introduced. I try my best to support them but this blog post reminds me I could improve on that.

Important to note: The company in question has developed a lot since this story happened so it's not in any way representative for how they work now, it's just a story about change, nothing else!

Lesson: Breaking rules should not be your first option
If you think what I did was irresponsible, I definitely agree. I do think you sometimes have to bend or break rules to make change happen but it should really be your last resort if the stakes are high and done with great care. Speaking to whoever is in charge and change things that way is generally a much better idea... what would had happened if Saam never joined the team for instance?

If it's completely impossible to change something without going behind everyone's back, a new job is probably a much better option.

Lesson: You can change most things
To continue on my previous lesson: Notice that I've never found something that with the right skill, time and effort, didn't seem possible to change though. I know it exists (e.g. strictly regulated industries) but it's rarely the problem. So trying over and over again, staying naive at a healthy level, is a great skill/trait.

Lesson: Experience is invaluable
If someone tells me:
- This idea with lightweight test processes are just bullshit
My answer would be:
- Let me tell you about when me and Saam...

Lesson: Testing requires skill
I knew the theory about how to organize testing in an exploratory way already in June but it took me many hours of practicing to really become skilled enough to perform efficient exploratory testing. It's like driving, you can't learn just by reading, you actual have to practice (but reading can speed up the process).

Lesson: My unsuccessful experiments were just as valuable as my successful ones
When I started to reflect on my work during my vacation I used my experiences so far to find problems to solve. When doing this I developed an understanding of each problem that a successful experiment probably couldn't had provided me. This was useful later when I ran into other problems that I could somehow relate back to problems I already had encountered.

Lesson: Reflection is just as important as planning

Reality constantly crushed my plans but in the beginning I was so focused on getting forward I completely neglected this. When I finally started reflecting, it became an invaluable tool to keep me on track. Plans didn't take surprises (changes, my misconceptions, my simplifications or my lack of understanding) into consideration. All those required extensive reflection to figure out as I learned more and more.

Lesson: Adaptation is key
I tried to just pick an interesting concept and force it into my context, this failed badly most of the times. When I started to look at what I was trying to implement to see where and how it fitted, my attempts to introduce things suddenly became efficient. It's like solving a jigsaw puzzle with the difference you have an unlimited amount of pieces, only a few can fit in a good way, when you actually find a good piece you still need to cut it to make it fit and as you solve the jigsaw puzzle the picture you try to make changes... so kinda not like a jigsaw puzzle at all but maybe that gives you a useful mental image.

Lesson: Learn from others' experiences
One thing I used when trying to turn things around during my vacation was things other testers had told me, stuff like "we did like this but it failed due to that". Exploratory testing was nothing new at the company, great testers already used it in secrecy, they just couldn't solve the problems they had run into trying to fit it into our process (problems like knowing what they had covered and traceability). What they told me helped me realize problems I had to solve and raised questions I had to be comfortable answering as well as provide solutions to problems I already knew about.

Lesson: Discovering my behavioral warning signs was great

Today I know I'm in trouble when I for instance stop sharing things with others or when I start to give very many reasons (rather than a few very clear ones) to why we should do something. Both are signs I'm either off course or at least don't know what I'm doing good enough. The latter can sometimes be a necessary step but at least it's good to be aware. When I discover one of these signs I can immediately stop what I'm doing and start reflecting to ensure I'm not drifting off in a dangerous direction.

Lesson: Being the only judge is dangerous
In my team I was the only one with testing experience (until Saam joined). This led to the rest of the team trusting my judgement when it came to decisions concerning test (which is to some extent reasonable). As soon as Saam and I started to work together we began questioning each other's opinions, ideas and directions, helping us to stay on a healthy course.

Lesson: Reasons are not proof
I can provide reasons for pretty much everything (including why you should jump off a cliff) but that doesn't mean everything is right (including the part where you jump off a cliff). I argued for my decisions early on, providing tons of reasons, even though I was way off. Now I know better than to accept peoples reasons, instead I look for lack of real life experience, generalizations, false connections and other assumptions (including my own).

Lesson: It's important to have someone to discuss with
It's no coincidence everything started to improve when Saam and I started to work together. When I was alone (Saam came to similar conclusions) my creativity dropped, my willingness to stand up for the changes I wanted to perform decreased, my motivation dropped significantly, my feeling of trust in my results dropped, learning was not happening as rapidly and finally I got stuck a lot more often and for longer duration (even when Saam couldn't help me it started the process of involving someone who could cause obviously I wasn't stupid asking whatever question it was). Having someone to quickly turn to whenever I needed a second opinion or some quick guidance was key.

Lesson: Exploratory testing needs support from your process
I didn't understand how crippled my exploratory testing efforts would be by the not adapted surrounding test process (like test reporting, test strategy guidelines, traceability requirements etc.). For instance the test management tool everyone had to use had no support for rapidly changing test scope and qualitative progress reporting. To even give us a fair chance of succeeding we had to work that out first.

Considering what kind of environment we actually created, my advice would be to isolate yourself. So instead of trying to change the world around you (which many will object and in the end you will probably just be allowed to change half the stuff anyway severely crippling your chances of success) try to build a micro universe where as much as possible is adjusted to fit an exploratory approach without colliding with the outside world. In agile, having an independent team with tested code as the only demanded output and working in an isolated part of the product / team owned product, could be one way.

Lesson: Education can save time and improve quality of test
In the beginning the exploratory testing didn't work efficiently. One reason, I believe, was I didn't really incorporate learning. Instead of stopping my testing effort when I felt my knowledge was insufficient I continued just putting out each fire I ran into (learning the bare minimum to continue). This was sometimes the right decision but often not as it constantly interrupted my testing flow. As soon as I started to spend longer stretches educating myself in an area (not just to solve one single problem), I also started to get a better flow and I started to recognize valuable tests I had previously not seen.

Lesson: If you can't argue for something you don't understand it
I've found a lovely exercise I wish I had performed earlier. Before I suggest a change I run it in my head trying to defend it against all the critique I can imagine. If I find a question I can't answer I try to sort that out before I continue. Notice, that the answer is context dependent so you can't just "learn" answers, you have to go through this over and over (but after a few times the loop will be faster as long as the suggestion is not too complex).

Example:
- We should remove the storage of test cases
- But without storage we can't reuse them
- Have we ever done that? For instance, it takes more time finding an existing suitable test case than to write a new one?
- But we lose traceability
- We could store session debriefs or one liners of tests we've performed, it would be a lot cheaper. By the way, when do we need traceability on that scale?
- Well, what if a customer asks for exactly what we've tested... or our test manager?
- Has that ever happened? And is it really worth all this time and storage money just to achieve that?
- But what about new tester, where will they go to learn test
- Our test cases are used for this purpose only due to lack of something better. I think the teams can manage this a lot better with pair testing and internal competence boosts. Also the test scripts don't really explain why you should do certain things partly because people stop writing when it becomes to complex (easier to figure out as you go), so the competence value is minimal as it is today.
...

Wrap up
This is one of my proudest moments, most frustrating experiences, most motivating explorations and definitely one of my most educational adventures. The lessons listed is just a brief start, we learned so much and I still learn things from reading this now.

Finally, this blog post was almost entirely written in December last year. I can now see lacks in my understanding of exploratory testing and how colored I was from only have been working at the company described. That's a lesson in itself.

Thanks for reading!

Transpection Tuesdays

A couple of weeks ago me and Helena Jeret Mäe sent a few cryptic tweets about "Transpection Tuesday". Since then Helena has written a great summary of our first session and to add to that here are my thoughts so far.

What is Transpection
The best explanations I've found are James Bach's explanation of transpection and Michael Bolton's transcript of a transpection session. But basically it's one person asking questions to another person with the twist that the one asking has already tried to answer those questions. The result is you get two sets of answers, hopefully less biased from each other.

What is Transpection Tuesday
When I first met Helena it was during Let's Test earlier this year. She had tried to grab me a few times but sessions and other conversations had interrupted us. So suddenly during lunch she stood up and said: "You sit here!", and pointed at the chair in front of her. And so I did, which was one of the smarter decisions I've taken as a tester.

Transpection Tuesday happened in a somewhat similar fashion:

Helena: Hey. I was thinking this morning that we could have Transpection Tuesdays (because it rhymes :P) or something. if there's something each of us is trying to work through and solve, then we could try this transpection thing out. It's a bit random thought
Erik: Sounds cool... I'll take care of the kids alone tonight since my fiancé is working so suits me perfect... with the possible problem that a kid wakes up and I have to leave very suddenly and might not come back .)
Helena: I understand that and won't hold a grudge :P
Erik: In that case I'm in! Usually kids are asleep after 19:30 CET, hopefully there's no change tonight

So after some technical difficulties we started a Skype call with video. By the way, video was awesome for this since it made sharing visualization easier, body language is useful when trying to explain something and it helped, at least me, stay focused.

So to summarize:
Transpection Tuesday is simply a Skype session ~3 hours long, happening every Tuesday evening between me and Helena, dedicated to talk about testing.

Confession
Okey, so what we've done so far has been far from just transpection. Instead it has more been like a weekly dose of conferring where we discuss topics that matters to us in various forms; where form depends on mood and topic. You could argue the name thus is misleading, and it probably is... But we like it.

What did we discuss?
Helena shared our combined notes, from the first session. I won't write anything detailed about it now but if something in those notes seems interesting, feel free to leave a comment and I can try to write a future blog post about it (my guess is some of the more "community challenging" stuff would fit that criteria).

How did we discuss?
I recognized five different "styles"/formats used during our talks, plus one suggested that we haven't tried yet:

1. Transpection
   I tried this briefly when speaking about why/if we should send people to courses and Helena did it much more extensively during the last TT when speaking about bias. When talking about these topics one of us asked questions we already had answers to but felt we wanted to improve/challenge. What makes this powerful is the person answering last get slightly less biased since he/she hasn't heard the first person's answers which gives you interesting answers to compare.
   
   
2. Conferring
   Most of the talking was just quick questions being thrown out and then we together tried to answer and discuss them. I would say this is the style we use most (it's like our "go to" style) and for those of you who've been to Let's Test or CAST, this is very similar to the discussions you typically end up in during lunches (which is a good thing!).
   
   
3. Challenging
   I plan to do a lot more of this going forward because it felt like an awesome way to learn. Essentially it's one of us asking the other to clarify something to a great level of detail or challenge a claim being made (similar to what happens during RST for instance). Very powerful as assumptions were uncovered and you constantly had to be on your toes.
   
   
4. Role play
   We take opposite roles (e.g. the manager who questions the professionalism in ET versus a context driven tester considering ET a good approach in the current context) and simply fight it off with the goal to pressure each other to explain/motivate something at a deep level, as well as highlight assumptions. We have actually done this very briefly but I don't think enough to really call it role play, more quick imitations and reactions to that.
   
   
5. Transfer
   Simply one of us telling the other something without any questioning going on; so basically a monologue about a topic. What was interesting to me was the information I received this way really seemed valuable but missing the interactive aspect hampered my ability to connect it to something else/make "lasting connections". Also I often felt the energy was lost a bit when one of us talk for long stretches.
   
   
6. Interview
   Similar to Transfer but more driven by questions so basically the receiver "choosing" what information he/she is interested in. Better than the monologue version but still a lot less valuable than any of the more interactive ways. It was like, even with questions, I didn't get my brain into gears, so new information simply didn't stick as it was when e.g. Conferring.

Improvements
Here are a few things I've noticed:

• Having a clear topic helps as we got fewer side tracks and thus easier could focus on going deep into the current topic. Side tracks are not evil per se, but too many and too big ones seem to make discussions rather shallow (my feeling so far).
• Having prepared followup questions would had been helpful, we have improved this for next week, will be interesting to see the result.
• We did get deeper into the topic during our third TT as we pressured each others to better explain what we meant and why even when agreeing. Think we can do this to much greater level though.
• I need to try to quickly structure the information in my head, before starting long rants/monologues about something.
• I think both of us could benefit from monitoring our own speaking trying to stop ourselves during long monologues. It's much harder for the other person to recognize a good break point.

Wrap up
So, Transpection Tuesdays are not so much about transpection but rather about weekly conferring. I already look forward to next week (preliminary topic: "Arguing for Exploratory Testing") and that's a great sign. I hope this turns into a long lasting tradition and it will definitely be the foundation for many future blog posts!