Peer conferences, part 1

Warning

This blog series is mainly for people who (would like to) arrange or attend peer conferences (explained below). People not interested in peer conferences will find rather little value in this post. One exception though could be if you want to arrange some workshop/conference at your work and you want inspiration for that. Finally; you can replace testing/software testing below with e.g. "programming", "change management" or something else as the experiences and methods should work for any "topic", I just happen to work with software testing.

Finally, these are my thoughts, not "the truth".

EDIT: If you want more ideas on the topic of peer conferences, check out James Thomas' blog post "Trying to be CEWT" and James Lyndsay's blog post "http://workroomprds.github.io/LEWT".
  

Peer conference

A peer conference in this case refers to a small group of experts, gathered to debate a specific topic, based on their own experiences rather than e.g. abstract models and they do this in a focused manner for at least one day. "Small group" refers to something like 6-16 people. "Expert" is harder to define but means something like "a person with a lot of skill, passion and/or experience in software testing".

With this definition I realize other events I've attended might qualify but I hope it's good enough to at least understand this article.
 

Data

This section is to help you understand what I base my thoughts on. Feel free to skip it if not interested. Do note that I attend a lot of other "peer conference -like" events such as Transpection Tuesday, local meetups etc. and I will take that experience into consideration but haven't listed it in this chapter for the sake of your (and my) sanity.

• SWET 4
  15 attendees
  James Bach present, lot's of well known names overall
  I was very inexperienced (compared to the other participants)
  Language: English
  Setup: LAWST inspired
  Topic: Models in testing
  Location: Sweden, fancy conference center
  Length: 1½ days (+optional 1/3 day before), Saturday morning to Sunday lunch
      
• SWET 7
  10 attendees
  James Bach present, very inexperienced group
  I was one of the most experienced attendees
  Language: English
  Setup: LAWST inspired
  Topic: Test coaching
  Location: Sweden, fancy conference center
  Length: 1½ days (+optional 1/3 day before), Saturday morning to Sunday lunch
    
• PEST 6
  9 attendees
  Michael Bolton present, the best of Estonia (which is pretty awesome btw.)
  I was an "international attendee" so a bit different
  Language: English
  Setup: LAWST inspired
  Topic: Gaining consciousness
  Location: Estonia, at Nortal (company)
  Length: 1½ days, Saturday morning to Sunday lunch
    
• SWETish
  10 attendees
  "regional peer conferences" with attendees mainly from Linköping and Örebro
  I was one of the most experienced attendees and co-organizer
  Language: Swedish
  Setup: LAWST inspired
  Topic: Exploratory testing
  Location: Sweden, fancy conference center
  Length: 1½ days, Saturday morning to Sunday lunch
    
• EASTish
  8 attendees
  Only attendees from Linköping, mainly from two specific companies
  I was one of the most experienced attendees and co-organizer
  Language: Swedish
  Setup: People brought their own topics so no "formal presentations"
  Topic: Any
  Location: Sweden, at Sectra (company)
  Length: 1 day (optional evening), Saturday
    
• SWET 8
  11 attendees
  Experienced group, rather mixed skillsets
  I was one of the more experienced attendees
  Language: Swedish
  Setup: LAWST inspired
  Topic: Testing that's not testing
  Location: Sweden, fancy conference center
  Length: 1 day (+optional 1/3 day before), Saturday morning to Sunday lunch
      

Size

So far my experience is ~9 is the minimum; below that the amount of conflicting ideas and experiences, which are important, starts to become an issue. EASTish was still a great peer conference but I think that conference would had gotten even better with a couple more attendees. PEST was right at the minimum limit but I personally did not feel the amount of people negatively impacted the quality of the conversations. Of course the people matter a lot in this case; more experienced people with more diverse experiences having a lot of passion and willingness to debate will likely mean you need fewer attendees and vice versa.

My personal upper limit is ~13; beyond that it seems like every single person gets too little time; especially if there are a few very talkative individuals in the group.

I would typically aim for 13 and since people will get sick, can't attend in the first place etc. that might make us end up with 11-12 which seems great. Notice that "aim" in this case does not mean "invite". For SWET 8 we had, for instance, at most 18 invitations out simultaneously but ended up with 12 who accepted and in the end 11 attendees as one had to cancel.

Language

I didn't think of language as much of an issue until I attended SWETish, my first peer conference in Swedish. It helps a lot even in a country where people generally speak pretty good English. To me, using the attendees' native language seems to help people "dare" to share more ideas, there seems to be way fewer misunderstandings and the overall flow is much better.

But it's a balance as using attendees native language rather than a language more broadly spoken (such as English) will limit who can attend... a problem we have in Sweden as well as several top notch testers here don't speak Swedish or at least not well enough (yet) to attend a fast paced peer conference in Swedish.
   

Experts

I have huge respect for people like James Bach and Michael Bolton; they always add a ton of value to a conversation about testing and especially given a format like the one typically used at peer conferences. Also seeing how James really helped a bunch of less experienced testers elevate during SWET 7 was awesome... however...

In Sweden it impacts the language, which I think is a problem (see Language). We also have a lot of talented testers so giving a spot to an expert will naturally stop someone else from attending and/or give some people less room to express themselves. Finally my experience is it steals a bit of focus as some people, knowingly or not, try too hard to impress the expert and/or not look stupid, hurting their overall performance at the conference.

I think inviting someone like James or Michael is amazing if the language is English anyway and/or the group is very experienced (hopefully lowering the "need" to impress) and/or it's hard to attract enough attendees so giving away a spot is not an issue while the expert can act as a motivation for other people to attend... but it's not necessary, you can have an amazing peer conferences without international, or even national, "experts" (e.g. at work, in your city or in you "region")... and I say this from the context: Linköping, Sweden; a city where we have a fairly active and skilled test community, just to make that clear.
   

Attendees

Experience (as in how much "testing" and "software development" the person has experienced/seen/participated in) helps as we base the discussions on experience. But mixing in a few rather inexperienced people can really add some interesting new points of view, as long as these inexperienced people feel safe sharing their thoughts. To summarize: Having a lot of  experience helps but lack of it is not a deal breaker.

Skill (as in your actual ability to test and understand testing) is, for me, key. Some people might not be known anywhere outside their own company, they might not have much experience neither as testers or in talking about testing but place them in a situation like this and they will provide value, as long as they themselves understand that their skill level is on par with everyone else's.

So on the topic of "experts", how much experience and skill does the average attendee need to have to make the peer conference amazing? My personal experience is: "a lot less than many seem to think".

Two other interesting attributes to me are passion and (verbal) communication skills.

Passion helps a lot but I think that usually comes naturally with wanting to spend a weekend (during which peer conferences or often organized) "just talking about testing"... be careful though about attendees who just think "it'll look good on their resumés" or who want to attend to advertise their own services or hire skilled testers.

Communication skills are important in general but do not mistake this for "talkative" attendees. The K-card system often used at peer conferences, for instance, can help less talkative people gather their thoughts and help them get into otherwise intense conversation and it can stop people who think in an extroverted way ("while talking") from filling all "the space". Instead, much more important is having attendees who can express something in a concise way, who respect the facilitator/format (primarily who won't speak when they shouldn't) and who can understand when their comments won't help a conversation forward/add value as well as dare to speak up when their comments will.
   

Invitation

My first ever peer conference, SWET 4, had an invite only format. My second had an open invitation (first 15 to sign up), my third had an open invitation (everyone may send in an abstract but a program board will select who will actually get an invitation among those) and later I also attended a "semi invite only" where ~15 people got a few days head start (personal invitation) until an open invitation was sent out. Long story short: You can do it in many different ways.

Invite only is great when you know exactly who you want to invite and want control over the group. I also find this to be most efficient as people feel selected and thus prioritze the conference more. It also helps to get those crucial first two or three attendees you need to start a buzz about the conference. There's also a risk that the group becomes too homogeneous; resulting in fewer conflicting ideas/experiences and thus fewer opportunities for people to challenge their own models.

Open invitation is great when you don't know the people you want to invite and/or who you want to come. It also relieves you from some "why did she get an invitation but not be" comments and allows you to better talk about the conference before it actually starts. However, it may make it harder to get those first attendees to sign up as they don't know if the group will be good; this can be somewhat helped by e.g. make in open invitation in a large group (e.g. a Meetup group) but where all the members should be good candidates. Also, there's a risk "the right people" will ignore the invitation because they don't understand they qualify or they, for whatever other reason, don't feel like they are the ones you're looking for.

I personally prefer invite only, even when I don't get an invite myself. I think that allows the organizers to better create a group that's suitable for the topic at hand... but that's my personal preference.

There are like I described in the beginning of this chapter many hybrids between the two but I think that should at least give you and idea. Finding the right format to send out the invitations is pretty straight forward for invite-only (email is pretty good for this) but depends completely on your context if it's an open invitation.

In the next part I'll share a checklist and example of what I think you should consider adding to an invitation, so stay tuned for part 2.

P.S.
For SWET 8 we wrote a personal note in each invitation explaining why we wanted that particular person to attend. This was definitely a win-win:

• Participants better understood our expectations
• Participants got to feel good about themselves
• It seemed to make more people accept the invitation
• For us, the organizers, it felt good to tell awesome people why they were awesome and it didn't cost much time or effort.

I'll provide an example of this in part 2.
     

Organizers

My preference so far is "as many as possible but not more than 4" (so I guess 3-4). Communication and taking decisions become a problem as soon as you're two but to me it's still worth the benefits (see below) until you're around 4 to 5. Some benefits of having more organizers are:

1. There are simply fewer people (easier) you need to invite to get a full group
2. The first person to accept the invitation will join an already established group
3. You are much less fragile, if one gets sick/life happens the work can still move forward
4. Larger personal network, key to avoid too many like-minded attendees when using invite only
5. Greater presence in social medias etc., key when using an open invitation
6. You have more options considering location, food etc.
7. More people mean each person needs to do less... and I'm lazy
8. It's easier to identify who you want to invite since you better know what's missing in the group
9. Each organizer can relax more during the conference (less pressure on each)
10. A lot less risk as with one sick organizer a peer conference might collapse if only having one or two organizers but it's much easier to handle if 3 or 4.

If the organizers can meet in person I think it's a benefit but having a good communication platform (e.g. Slack, Skype etc.) should be sufficient. For SWET 8 the only physical meeting we had together was the evening when we decided we wanted to organize a test conference, everything else was handled via Slack.

      

Format

All peer conferences, except EASTish, I've attended have used basically the same format:

• All attendees prepare ~20 min presentations based on something they've experienced.
• A few attendees will actually present, usually 2-4, for a 1½ day peer conference.
• After an experience report there's a facilitated discussion around that presentation. The discussion will continue until the group feel done with the topic (usually 1-6 hours).
• At some point there's time for lightning talks: ~5 min talks including open season.

While I think this is a great format I think other formats could work just as well and potentially even better as they have been explored less.

Two suggestions:

• Discussion topics and dot voting instead of presentations
• Solving an actual problem (doing something); with one or more debriefs

At EASTish rather than presentations, attendees got to write down a few topics each (typically in the form of a question or short scenario) and then we dot voted. This basically turned it into a prolonged lean coffee. There were some cool benefits to this format:

• People got help/got to discuss the exact topics/questions they were interested in
• Much less time to prepare for attendees
• People who feel uncomfortable to present didn't have that distraction 

Drawbacks could potentially be more abstract content rather than focus on experience (not my experience from EASTish), if attendees think the format means they don't have to prepare that may negatively impact the topics covered and the presentations (or rather preparation work needed) may act as a useful gate keeper; scaring off people who want to attend for the "wrong reasons". I don't know if any of these drawbacks are actually valid but no matter what; this is a format I would love to try at a peer conference fairly soon; with or without a specific topic/theme, but probably with.

The other suggestion is something I have tried at a local meetup where we split into smaller (mixed) groups, tested a specific application and finally spent a long time debriefing our testing including why we did the testing we did, how we organized ourselves, differences between individuals in the team etc. The idea has also been used at at least one peer conference before: PEST 4.5, where they tried to visualize various reports in testing in new and creative ways. I highly recommend reading about PEST 4.5 and ever since I heard about it I've wanted to attend/arrange something similar.

If you have ideas on other formats that could be useful, please comment and I'll add them to the post. This is one of the areas where I think we could really take the concept of peer conferences to a whole new level!
   

Topic/theme

Peer conferences typically have a topic/theme. This topic/theme is important to keep discussions focused, to get the right people interested in the conference and to help people prepare before the conference.

Generally topics should be broad enough to include some diversity but narrow enough so that we can actually get deep into the topic. You have several examples in the Data chapter and if you search for "peer conference software testing" you'll find more. One issue though is that the one liner rarely tells the full story as there are usually specific aspects you're interested in (the one-liner is too broad). To get a better idea what I mean, do check out the invitation for PEST 6, it explains their topic "Gaining consciousness" in a great way.

       

Presentations

This is relevant if you let people present, skip if you want a different format.

First of all, set a clear deadline for when you want attendees to inform you what they want to talk about. Informing you about what they want to talk about is typically done in the form of an abstract (~1 page description of the talk).

When you know what they want to talk about establish a speaker order and if possible, set who will facilitate each talk. You can read more about facilitation in the story behind K-cards. I'll try to share how I do this in some later part but if I forget, feel free to ask me.

If your peer conference looks anything like the ones I've attended you'll likely fit 2-4 presentations into a 1½ day conference, so my advice is to inform the top 5 participants (1 extra to deal with a potential cancellation) and let the rest focus on lightning talks.

I also recommend having appointed mentors (typically organizers, e.g. the facilitator) available for each speaker. This can help both experienced and inexperienced speakers present the "right thing" (an actual experience rather than some abstract concept).
    

Duration

Every time I've attend 1½ day peer conferences, I feel like half the group says "This sucks, I would like to continue" and the other half says: "I loved this, but now I need some sleep/think for myself" at the end. I don't know if that means the length is perfect or too short (or even too long) but I think it's a sign the length is quite good.

However, to avoid stagnation (few new ideas/low energy) when going beyond a day my experience is you need a larger group than ~10 and/or very passionate/skilled/experienced attendees. I for instance felt a bit of stagnation during SWET 7 but not during SWET 4 and SWET 8.

I would love to try a full 2 day version but when trying to find a good schedule I run into problems. If you want to avoid missing too much time from work (problem typically for consultants) you could either start by lunch on Friday and end by lunch on Sunday, this would cost one extra night (compared to the typical Saturday morning to Sunday lunch setup) and interfere with working hours especially for people with a long ride to the conference... or you could start Saturday morning but end Sunday evening instead, so you basically end after dinner (say everyone leave ~21:00). This would not cost an extra night but add quite a bit of conference time. Would suck though for people having e.g. a 3h+ drive home (common in Sweden)... Maybe end at say 17:00 or 18:00 and skip the extra dinner as part of the conference... I need to think about this a bit more.

The other option: Shorter, means you can cut costs (e.g. no nights) and make it simpler for people to "spare the time". We did this in Linköping (city with ~140k population) and it work out nice. I don't think shorter is an option when attendees have more than say 30mins to the conference but worked great if you want to introduce the concept of peer conferences to a more local group.
   

Facility and food

Facility and food are actually quite important since attendees will like sit for long durations and be exposed to a lot of information. When selecting a place to host the peer conference, take into consideration: costs, food, quality of the conference area, how much you have to fix yourself and facilities to use in the evenings.

For the evenings it seems like you want one of two things: An inspiring area to sit in (beautiful, unusual/creative, enough space etc.) or a relaxing pool area; e.g. a large outdoor jacuzzi... some alcohol (like a beer or two) also helps.

Being in an area that's great for taking a relaxed walk or jog also helps in my experience as people need some air after more or less a full day in a conference room. For this, choosing a location that's somewhat remote seems to help; this also has the benefit of helping people to fully commit to the conference as there are fewer distractions.

A conference center will greatly up the costs but also significantly lower the amount of work for the organizers. I've attended peer conferences hosted both at conference centers and at someones company, both work equally good to me as long as the organizers, in the latter case, have a good plan for e.g. food, evening location, some energy refillers (=candy/sweets) for the breaks etc.

Schedule

1. Make sure there's a schedule
2. Make sure the conference center, food catering etc. agree to your schedule
3. Be flexible; not interrupting a good discussion is more important than sticking to the plan
4. Schedule regular breaks but remember to not interrupt good discussions (see previous)

Evenings

During the evening(s) a lot of important processing, bonding and follow up discussions take place. Make sure there are good facilities for these, that attendees stay (except for the need of sleep or handle social overload) and that there are some "conversation/activity help".

"A good place" and "making attendees stay" were described in the "Facility and food" chapter above so let's focus on "conversation/activity help". A lesson learned (for me) during SWET 8 was I think the group benefits from being split up a bit in the evening. One simple example is having one or more tables devoted to e.g. the dice game, coin game, Test Sphere or Set as this will split up the group. If there's a pool area the size of the jacuzzi and the fact not everyone like spending time in a pool will automatically split up the group (not necessarily in an optimal way though, but hopefully good enough).

Other examples could be to actually schedule activities in the evening. One way would be to split into smaller groups doing some task, challenge or activity and then, in a simple and informal way, let the groups debrief their results to the rest of the attendees (either in the evening or the next morning). Another would be to set specific topics/tasks at different tables so people can rotate and discuss/do different things with different attendees. Be careful about ambitious plans though; it seems like as long as you provide a somewhat quiet area where people can easily split up into smaller groups themselves; you're basically set... but some help rarely hurts.
   

Summary

I think we can take the concept of peer conferences even further if we dare to challenge the current common setup by e.g. trying new formats, longer/shorter conferences, tinkering with the group we invite, try new locations etc. For instance my view of the "minimum viable product" for a peer conference (location, setup etc.) was significantly altered after I had attended PEST 6 which was the first peer conference I attended that wasn't hosted in a fancy conference center. PEST 6 then became important inspiration for how we arranged EASTish here in Linköping.

What's next

Part 2 will be a checklist for organizing a peer conference including an example of an invitation etc. The goal with this post was to help people learn new ways of organizing a peer conference, part 2 will hopefully inspire new people to organize them as they learn it's not that complicated.

Part 3 will deal with stuff related to the actual execution of the conference e.g. facilitation, check ins/check outs etc.

How do you help a team become awesome?

Background

I raised a question, first during a Transpection Tuesday, then in the TestSverige Slack chat and finally with all sorts of people I've met; mostly software testers. The question was:

How do you help a team become awesome?

Awesome in this case refers to the kind of team where everyone seems comfortable; they laugh, they communicate, they do silly things but don't seem embarrassed and at the same time they seem productive, motivated and ever evolving with low employee turnover rate.

This is my summary of those discussions.

Before we start: This is not specifically for managers, team leads, scrum masters etc.; it's everyone's responsibility and opportunity; anyone can improve a team's "mood".

Personal attributes/attitudes

Personal attributes and attitudes came up a lot during the discussions and they seemed to be the foundation on which you can add helpful activities. All of these work as self reinforcing systems so if you start to set a positive direction others will (eventually) follow. The same applies if you set a negative direction though, as this will start to create a deeper and deeper hole to get out of.

So why don't we just act "good"? Because we're imperfect, also known as being human: We're scared, we sense injustice, we want revenge, we get stressed, angry or sad, we're sometime egocentric and so forth.

For these reasons there are a few things you need to consider for each of the attributes listed below:

1. It'll take courage to set a new direction and you might get hurt... sometimes a lot
2. You'll need to consciously monitor yourself to avoid stress etc. getting the better of you
3. You'll need to nurture these attributes in the team primarily by making positive examples visible

So, without further ado; dare to...

• Be vulnerable
  "My uncle used to say that we like people for their qualities but we love them for their defects."
  /John Myers, Hellboy
  
  Share your struggles, admit you're scared, open up, allow people to come close and dare to be imperfect (aka. human) in general.
     
• Be transparent
  Share what you know and do, that's relevant to others even though this might make them question your decisions, force you to (temporarily) stop something or even use the information to personally attack you.
     
• Be accountable
  When you've messed up, take responsibility, apologize if appropriate and accept the consequences. Sometimes it's even beneficial to take responsibility for things you weren't responsible for just to get out of a negative loop.
     
• Appreciate
  Make it a habit to register when someone makes something good and tell them this. Make sure you're sincere, empty flatter is not helping. Another nice way to appreciate people is to be a proxy for appreciation e.g. "Just so you know, Bob gave a two minute speech this morning about how great he thought your design was".
     
• Trust people
  People want to do good so do trust them. Sometimes they'll let you down, sometimes you might even get stabbed in the back but keep trusting them. With that being said, of course bad behavior should be dealt with, e.g. see "be sincere" below, but as soon as you stop trusting people you're heading in a bad direction. After all, if you don't trust people they'll never be able to show you they can be trusted starting a rather destructive loop. Also people grow with responsibility. Finally: trusting people does not mean not helping them and/or helping them realize they need help.
     
• Be sincere
  Integrity is sexy; if you think someone, including yourself, is being singled out, is getting unfair criticism or for other reasons aren't treated in a fair way: Speak up! Especially when people aren't given a chance to defend themselves.
  
  However, stick to your observations not your interpretations. You don't know for sure if "this person is actively trying to hurt you" but you do know for instance that "the person was told to give you the latest version but you never got it". Sincere != Judgmental, quite the opposite actually.
     
• Care about people
  Caring about people costs very little and the main risk you face is simply to be creepy. Do notice that care does not mean micromanage, instead it's about genuinely trying to create a good situation for others. Carita Jansson Tsiantes gave a lovely example in the TestSverige Slack chat that went something like:
  
  When you boil water to make a cup of tea, don't just think about yourself; prepare water for your colleagues who might want tea as well.
     
• Help and support
  This can shortly be summarized as:
  "If someone has a problem, we have a problem".
  
  When asked for help do help and if people express frustration or confusion offer to help. Few people ask questions if they don't need to so rather than telling them "you should know that" try to help them learn how they can find the answer themselves; e.g. by introducing them to the right people, help them get access to some information system, help them get invited to a certain meeting/mailing list etc. An attitude to avoid is "it's not my job to help...". Sure this is sometimes true and you need to work too but then again: help the person help herself rather than ignore the request.
     
• Respect everyone
  No job, role or person is more important than any other. Of course some tasks might be more important to finish but then focus on getting them solved as a team. A key aspect in this is understanding your colleagues' tasks, challenges, frustrations and talents. Andreas Cederholm brought up a great example of how to nurture this attitude:
  
  We run team test sessions where the whole team test together. Add some cookies and laughs and it'll work even better.
     
• Try
  If you want to challenge status quo you'll have to try new things. Trying comes with an increased risk of failing and potentially making a fool of yourself but that's necessary and typically a great way to learn. Sometimes trying something you don't really believe in might still be beneficial simply to acknowledge that ideas are appreciated and that you trust in peoples judgement even when you might not agree with them.
     
• Auto forgive
  A psychiatrist once told me a very smart thing about eating disorders and how to react when people have not been able to fight the decease (generally applicable of course):
  
  Guess who'll feel worst when this has happens? You? No, the person who just "failed"! You don't need to remind them they "let you down", they'll know and they'll feel terrible about it.
  
  People mess up, people take bad decisions, people have bad days. You rarely need to remind them, it's typically much more constructive to say "don't worry, shit happens, let's fix this" and move on. This is also important to nurture previously mentioned attitudes such as "try" and "be transparent"; if people are scared about potential consequences (including reactions) the only thing they'll try is to cover stuff up.
     
• Smile (and laugh)
  Being met with a calm, warm smile is great medicine when you feel down or nervous about some bad news you have to deliver. Smiling also helps at least me stay calm making it a useful tool to manage feelings of anger or frustration.
       

I get the feeling all the attributes/attitudes above point back to some basic principle like "get unhelpful frustration off the table fast; both yours and others" or "always trust in peoples willingness to do good"... but I can't really put it into words. Feel free to help me.

Activities

If the personal attributes/attitudes are the foundation the various activities below represent important tools to speed up the process. Notice though that the activities by themselves are not silver bullets and overusing them or using them at the wrong time can actually have a negative impact. Focus on the list above first!

• Social activities outside of work
  E.g. cook together, sports or boardgames. Activities where everyone is active which is not necessarily true for e.g. your typical after work.
      
• Quirky things
  E.g. quote book, silly competitions, fun/silly "rules" or internal titles.
      
• Retrospectives taken seriously
  Not specifically the meeting, can be e.g. a continuous, everyday team reflection activity. All problems brought up are dealt with. Problems are taken seriously even by members not personally impacted.
         
• One on ones
  Allows people to raise concerns in a safe environment (assuming the person meeting members one on one has earned the members' respect).
      
• Do each others work
  An example of this is Team Test Sessions where the team test together (suggested by Andreas Cederholm, TestSverige) or move the other direction and try mob programming with testers included. Everyone (product owner, developers, testers, designers...) together attending e.g. courses in security or usability could also help support as this kind of activities creates some common ground. Yet another suggestion is team members meeting customers, accompanying sales/support people etc.
       
• Discussions about values
  E.g. take the "personal attributes/attitudes" list above and talk about each one described. Is this something you want to strive for in the team; can you change something to help nurture this behavior etc. Make it a team goal to improve and nurture the "mood" in the team in general.
      
• Personal values
  Most of the personal attributes and attitudes require consistency. An activity where you sit down an state you personal "manifest", goals or values can be important. For instance it might be hard to treat yourself in a fair way without some guidelines; either turning you into an asshole demanding more from other than yourself or a "victim" never treating yourself well enough.
      
• Clarify your attentions to your boss
  If you want to invest quite a bit of time in this, go to your boss, explain your intention and ask for her/his support. Making your boss, or if necessary, your boss' boss, an ally can provide access to several powerful tools (e.g. see "Supporting context" further down).
        

Symptoms

The list below represent "symptoms" that your team (or even company) is moving in the right direction:

• People laugh.
     
• You're met with a smile, even in bad times.
     
• You know what your colleagues like, both at work and outside. E.g. their hobbies,interests, spare time activities, important life milestones, work and private goals, "hidden talents" and previous experience.
     
• People talk about hobbies, spare time activities and the other things listed above.
      
• Conflicts are taken seriously and navigated swiftly.
    
• People blame themselves, if anyone, not others.
      
• High level of motivation.
      
• You rarely feel stupid (in a bad way).
      
• Stuff that "should be done", gets done.
      
• Ideas are taken seriously, people try new things and experiments are run frequently.
      
• People admit mistakes and challenges early as they're not afraid of the consequences.
       
• People meet outside work because they want to, not because they feel obligated to.
      
• Few taboos.
       
• Very limited "bullshit" or backtalk in the team.
      
• You know what's happening in the team and rarely get "unpleasant surprises".
      

Supporting context

These things might be hard for you to actively influence but be aware as they do seem to have an important impact:

• Reasonable pace
  People need time to do supporting, long term activities and when under immense pressure/unreasonably high pace this is quickly forgotten or down prioritized. These lost activities help you become faster tomorrow than today meaning they're long term, multiplicative investments.
       
• Stable organization
  Adding or losing team members can in worse case force the team to start over in their attempts to be awesome. If you're the manager; try not to change teams that work great together even though it might be tempting!
      
• Ethics
  A product you believe in and feel ethically good working with, helps. The same goes for the company's actions: If it feels like the company acts in an ethical way that seems to help people "invest" in the company in a way that's helpful.
       
• Good social (especially empathic) skills
  Having team members who like the social aspect and are good at nurturing positive social behavior (not to be mixed up with people "talking a lot") helps.
       
• Previous friends
  Not always true as the previous friends may create a "sub team" within the team but seems to sometimes help as the friends most likely have a healthy relationship towards each others which can spread.
       
• Management accepting problems
  Having a manager/management asking for "solutions, not problems" can suppress people's willingness to bring attention to important problems or make the company accept suboptimal solutions. The intention to focus on what's constructive is not bad but the message delivered can be. It's of course okey to ask the person if they have any ideas themselves on how to solve the problems they bring up but don't make the solutions a "requirement".
       
• Culture awareness
  Manager/management that genuinely cares about the company culture and how to improve it helps.
       

Misc

Some "quotes", all loosely translated from Swedish:

• "I ask myself: How can I make this person feel like I want her to feel?"
  Carita Jansson Tsiantes
      
• "It's professional to be personal"
  David Högberg
       
• "It's not unprofessional to have fun but to do something in a boring way when it can be achieved just as well in a fun way, that's unprofessional"
  Klas Hallberg, from his book: YCDBRALAI (Swedish).

Finally a comment I didn't know where to place:

• "If I say I can't talk about it, you know and accept this". Transparency is important but some information you mustn't share for various reasons. However, sometimes the mere knowledge you know some secret information can be enough to help people prepare for a big change, avoid unpleasant surprises etc. One example could be: "We will get a new boss, I know who it's most likely gonna be but I can't tell you until papers are signed; however, I can tell you I think this person will do a terrific job, so don't worry too much about it".
      

Conclusion

It makes perfect sense but didn't really occur to me when I first asked the question:

Making a team awesome is basically the same thing as making any relationship awesome and it starts with you and all the small decisions you make every day.

Good luck!

Learning about learning by teaching

I've undergone a tough education in software testing:

• 15 months long
• Tests 3 times a week, 7 hours each and in front of a crowd
• If you skip a test you'll have to do it again, typically within 5 days.
• The expected level of competence is: "Good enough to teach others"
• Little or no chance of "redo:s", you better do it right the first time, every time

In other words: I've been teaching a class in software testing.
The intense experience of teaching testing like this has of course taught me tons of things and with this post I want to share the positive effects this particular job had on my own learning. Each benefit (of teaching) comes with a "why I find it useful" and "how you can apply this knowledge in a more everyday context".

Benefit: Curriculum

Why?

I'm not a fan of committing to a plan, especially not when it comes to my learning. However, the education's curriculum did force me to look into many topics/material I would otherwise had skipped due to laziness, lack of interest or lack of understanding (not thinking it was useful to me). Some of these have definitely made me a more complete/skilled tester such as test framing, deeper understanding of test techniques and a better understanding of bias.

How?

• Use this existing curriculum in your own self-education.
• Complete the BBST courses (just for reference: I haven't done this myself)
• Create your own list of topics and focus on these in whatever order you prefer.
  Important though: It's easy to do this in a shallow way, go deep to learn something new!
  Some good resources to guide you: James Bach - Self-education, James Bach - My view of testing professionalism (especially page 7) and Huib Schoots - A road to awesomeness.

Benefit: Go deep

Why?

I've read tons of articles, watched hundreds of hours of presentations/videos and spent a lot of time practically practicing testing. However, I often end up looking at topics in a quite shallow way especially when I find the topic a bit boring (may still be very useful). When you are to talk about a specific topic for just a couple of hours, you're okey, there's little "need" to go deep. When you have to prepare several weeks of study material though, that's a whole different beast! Being forced to go deep into topics has enables me to better question, improve, explain and argue for the various choices I make (for example why I chose to place my testing mission at the top in a certain status report).

How?

• Dig deep into fundamental questions e.g. what is testing, why do we test, what is an oracle etc.
• Look into related topics. Say you want to improve your written reporting skills then look into e.g. rhetoric, design, how people read printed and digital documents, how people interpret colors or tutorials for your word processor/mind mapping tool/whatever. The point is: don't limit yourself to articles specifically about test reporting.
• Set a "topic of the month" and try to become as skilled as you can in this topic. Don't stop because you feel "done", continue beyond that.

Benefit: Giving feedback

Why?

An important part of my job is helping students understand what they do well and what they might need to improve. To do this I have to observe and analyze what they've done, what they think they've accomplished, what actually made them accomplish what they've accomplished etc., all this I have to do rather thorough in order to be able to explain it to them. This helps me create an understanding that is beyond "do this or do that because it works better".
An example of this is when grading various assignments and projects as students, at least on a general level, need to understand what they did good and what they would had to do to get a better grade. If they get the highest grade they need to know why, so they both know what to continue doing and what to improve. As testers we need these kinds of observation and communication skills all the time when working with developers, project managers etc.

How?

• Study your own testing and try to explain why it was good and how it could be improved.
• One area where I've found this pretty easy to practice (can't prove that the practice translates to other parts of testing but I think it does) is watching presentations (e.g. YouTube) and try to give feedback to the presenter. What specifically did she/he do good and bad?
• Study other testers and try to figure out why you find them good/bad testers. Be as specific as you can.
• When testing, try to find positive and negative patterns: "The log entries are (almost) always relevant and well described making my work so much easier" or "The UI components often have poor vertical alignment".

Benefit: Teaching

Why?

Teaching in itself is a great technique for learning. You have to rephrase the content to match your own language, you hear yourself speak about the topic and you get questions pinpointing gaps in your understanding and/or explanation.

How?

• Do teach colleagues and friends about the various topics you've practiced.
• Write an educational article/blog post about what you've learned (you don't need to publish it to anyone to still get many of the benefits).
• Talk at a local test meetup and if there isn't one, arrange one.

Benefit: Peer

Why?

Working with Maria Kedemo and Martin Nilsson have allowed me to get feedback on the ideas I'm about to share, feedback on my interpretation of various topics and someone to speak with when I feel stuck. It has also allowed me to learn from their knowledge and experience of testing.

How?

• Speak with a colleague
• Join a local tester meetup
• Go to a test conference
• Join the active community on Twitter
• Try your own version of Transpection Tuesday (my post, Helena's post: 1, 2)
• More ideas...

Benefit: Observe testing

Why?

I've spent a significant amount of time both observing testers test (as a group), observed testers test (the actual testing done by an individual) and listened to testers speak about their testing. All three exposed me to new ideas and made me question my own approach. It's also interesting because you get to see a specific problem solved in many different ways which helps you understand what actually impacts the result; e.g. "what is the common denominator in these solutions, is there anything I can learn from that?" or "they all had different ways to setup but all ended up with the same solution, which setup worked best/most efficient and can I learn something from that?".

How?

• Pair testing
• Look at other testers' notes, reports etc.
• Do call for and attend debriefs no matter if you use the concept of test sessions or not
• Offer to review things
• Volunteer to mentor/coach another tester; this will enable you to observe another tester as well as get several of the other benefits mentioned in this post

Benefit: Consistency

Why?

To sit down and learn about various topics every day for over a year has definitely added some welcomed consistency to my self-education.

How?

• Create a routine, such as the already mentioned Transpection Tuesdays.
• Follow a curriculum
• Create your own skill development list

Benefit: Questioning basic assumptions

Why?

Explaining fundamental concepts is incredibly hard but rewarding! As an experienced tester I take quite a few things for granted and explaining concepts built on these assumptions to someone without experience lead to wonderful questions like "but why do we need testing at all", "what does it actually mean to test something", "why can testers find bugs if developers who know the code can't (as in why do bugs happen at all)?". Answering these questions without being able to rely on "experience based assumptions" has led to more than a few epiphanies (and a lot of frustration of course).

How?

• Talk testing with people having a different frame of reference (developers, management etc.)
• Talk testing with people who don't work in the industry; for instance try to explain what you do to a relative.
• Teach new testers at the company or teach e.g. developers in testing
• Talk testing with new, inexperienced testers

Benefit: Ask yourself "how do you train this skill"

Why?

Reading and listening is nice but sometimes you need to actually practice skills to be able to learn them. When teaching I've spent a fair amount of time trying to figure out exercises pinpointing a specific skill I want the students to practice or just exercises/projects in general helping students practice relevant testing skills. This experience help me now both when less experienced testers want help learning a skill, when I try to explain/teach something and when I try to teach myself something.

How?

• After e.g. a blog post, YouTube video or book; think about how you can incorporate the new concepts you've just learned about into your own work.
• Try various exercises and try to replicate various experiments yourself; such as: 1, 2, 3, 4; to help kickstart your brain.
• Whenever you're asked to explain something; try to come up with an exercise or experiment that helps demonstrating whatever you are to explain.

Benefit: Getting questions

Why?

I've already touched upon this but getting questions from the students on anything that's not clear to them is incredibly challenging but rewarding. It has helped me realize flaws in my own understanding, forced me to question my own assumptions and challenged me to find new ways to explain certain concepts in.

How?

• Explain concepts to others
• Ask for feedback
• Ask questions yourself; this both inspires others and help you ask questions "to yourself"
• When reading/watching e.g. a book, presentation (video) or article; pause periodically and ask yourself: "what did I just read/watch and what of that is unclear/seems strange to me?"

Benefit: Having the time and expectation to learn

Why?

When in the middle of deadlines, huge backlogs and conflicting priorities it's easy to forget learning. Having the explicit expectation to learn new things has been an interesting experience and I feel confident saying I leave the teacher assignment as a much more competent tester. Spending as much time as I did on learning is not possible in most working contexts but I think "expectation to learn" is the key concept here as it helps making it happen at all.

How?

• Ask your boss: "How much time am I expected (or at least allowed) to spend on education?"
• When doing backlog grooming (if you do this); add learning stories as dependencies e.g. "before we implement the new video player we need to learn a bit about streaming, video formats and performance testing related to streaming". If you end up never having time for these learning dependencies, try timeboxing them to make the expected time invested in learning more explicit.
• Remember learning is a fundamental part of testing.
• Differentiate between the learning that's necessary to solve your current task and learning with more long term, strategic value (e.g. learning more about testing in general, about web security, about test planning etc.). The "strategic learning" is often important to keep you and the company progressing but can easily be forgotten if put in the same "budget" as the task solving learning.

Final word

I removed several additional benefits I had initially included just to finally get this blog post published (it's been stuck in "draft" for over a year) so just to be clear: You can learn so much more about learning by teaching; this is just a mere introduction.

SWETish

It was quite a long time since I wrote about an event. This is mainly because "it's not new to learn new stuff" anymore so for me to write about an event it need to be pretty special. SWETish, a peer conference that just ended, was exactly that: Special.

I've tried to wrap my head around what made this different and it boils down to:

I've not had this many "epiphanies"/been surprised this many time at a test conference since... one of my first ever.

Next question is: Why? What made me get all those epiphanies?

Well, I've rewritten my explanation to that I don't know how many times now. It boils down to things I can't seem to describe well enough yet but I'll give it a shot in a separate post, probably after a Transpection Tuesday (I need your brain Helena).

So, let's skip to the good stuff: Content and lessons learned.

Credit

Before we start: Credit for the content below goes to all the participants of SWETish:

• Agnetha Bennstam
• Anders Elm
• Anna Elmsjö
• Björn Kinell
• Erik Brickarp
• Göran Bakken
• Johan Jonasson
• Kristian Randjelovic
• Morgan Filipsson
• Tim Jönsson

Talk 1: All Pairs by Morgan Filipsson

Morgan basically described a tool he thought was more or less obvious and most of us went like: "mind blown"... it felt a bit like Paul and Karen in the K-card story. It was just a simple non-mechanical combinatorial testing support tool made in Excel where you could decide the input values you would use in a test and the program would help you by showing how many valid pairs (as in "all pairs") you had not yet covered. I don't know if this is exactly what Hexawise, or some other tool, already does but to me it was ingenious.

But this is a peer conference so why stop there; open season time:

• Forming an unbiased opinion can be expensive and money does matter
• Getting many peoples biased opinions and comparing them can help sometimes
• Beware of decision fatigue when using exploratory testing
• Improving your test strategy or moving towards a more formal approach can delay the decision fatigue process
• Mind maps are cool but do have limitations
• Don't underestimate pen and paper
• Excel is a pretty bad ass testing tool
• Remember data can come in many shapes and forms and it often has complex relations to other data
• Sometimes the simplest of ideas are not obvious to others
• There are so many potential benefits with screen recording

Here's a like to the screen recording tool Morgan preferred by the way.

Talk 2: How we test by Anders Elm and Erik Brickarp

A bit unorthodox but a two person experience report at a peer conference. The background was I read Anders' abstract and realized we have a very similar test process at Verisure. So Anders described how they test at SICK, I described how we test at Verisure by focusing on the differences and at the end we shared our key lessons learned and challenges. Long story short: We had both started with the goal to implement SBTM (in my case credit should go to Maria Kedemo et al.) but in both companies this had diverged into something else. I described this new process to be much more similar to TBTM than SBTM.

I might talk more about this in the future but let's skip to the open season for now:

• How much of this diverging is due to strategic decisions and how much is due to laziness/lack of competence (valid question that requires more thinking on my part)?
• Internal demos after each completed user story and mob testing during these demos seemed awesome (done at SICK but not Verisure, where I work)
• We got an interesting question from Björn Kinell about "if you could magically change/correct whatever you wanted, what would that be". I don't want to share my answer but my advice: Ask yourself that question cause it can help when forming a vision for your testing.
• It's easy to forget/skip debriefs, test planning and testing not immediate related to a story in the sprint but be careful as these activities often provide quite a bit of value.
• Find activities that are "not testing but still testing" to easier get the developers involved. Examples: Add testability, "try" the product and support a tester when testing.
• Ask the question "how do we catch <very basic bug> before the product is handed over to testers?" to start a discussion in the team about testing and developer responsibility.
• Remember that small bugs that normally don't slip through development/testing can be a symptom of a much bigger problem like stress, change in culture or lack of support.
• Time spent improving charters is rarely a waste of time.
• SBTM "by the book" in a scrum team is not easy...
• If the timebox aspect (session) is removed you need to find new stopping mechanisms and/or heuristics to help you stop and reflect on whether or not to continue.
• Debriefs can be useful for many many reasons.

Lighningtalks

Johan Jonasson spoke about construct validity in metrics

• Johan referenced Cem's and Walter's great article: What Do They Measure and How Do We Know? I think that article explains the key points of Johan's talk.

Anna Elmsjö spoke about finding and making allies

• Candy solves most people problems (my wife approves)
• Finding allies among the developers is important to help you get attention to testing problems

Tim Jönsson spoke about the value to "knowing your audience" when reporting bugs

• There are more ways than an entry in a bug reporting tool to report bugs
• Do highlight helpful developer behavior/good code and show gratitude when developers help you
• Good lightning talks often focus on one thing and explains that one thing well!
• A compelling story helps your audience understand your message
• Testing is about psychology more that you might initially realize
• Be humble, be helpful, be useful... and be respectful.

Göran Bakken spoke about how integrity can get in the way of efficiency

• Refusing to do bad work is not always constructive
• Two different ways to approach a change is to focus on supporting values or to focus on methods.

Kristian Randjelovic spoke about how analogies

• There are many ways to use analogies to help colleagues with limited understanding of testing understand otherwise rather complex (testing) concepts.

A general lesson for me (even though I didn't present a lightning talk):

If I'm given 10 minutes for my talk and open season, I'll try to aim for a talk shorter than 4 minutes and only focus on one message.

Talk 3: Test documentation in the hands of a stylist by Agnetha Bennstam

Agnetha showed three different examples of how she had helped turn massive 200-page documents into lightweight, often visual, alternatives. It's hard to explain this in text but the running themes were figure out what is important, find the right level of detail and aggregate data in a way that's useful/enough for the receiver rather than show the full set of data.

Open season:

• Dare to try something different!
• You can add additional dimensions to a diagram by using the size of the dot you're plotting and color (see slide 55 in Andy Glover's presentation).
• Use feelings when you test e.g. "why do I feel bored" as indication that there might be something you should react to/take notice of (see Michel Bolton's presentation).
• Asking for people's gut feeling can be a good way to get their assessment without getting all the motives and data (makes people relax a bit).
• Sometimes asking for people's gut feeling can help them "dare to answer" so that you can start to figure out what is happening e.g.
  "How 's the quality of ...?"
  "I don't know..."
  "But what's your gut feeling?"
  "Well, so far it's not looking good"
  "Why is that...?"
• Gut feeling can sometimes mean "very useful and relevant information the person simply cannot/dare not articulate yet"
• ... but gut feeling can also mean, or more often be interpreted as, "I base this on absolutely nothing but I don't want to admit that"
• Beware of documentation existing only for the purpose of being ammunition/defense in a blame game
• A tool that could clearly visualize:
  How fresh the testing is (time and impact of changes since last tested)
  How important the testing is
  How well the testing went (typically quality)
  ... would be cool.
   

Other stuff

One more crazy thing about SWETish: All the talks and open seasons were amazing and that's despite the fact that the experience report all organizers voted as their favorite... was not delivered since Joel Rydén got sick!

Another cool detail. Anna Elmsjö added a nice twist to the K-cards she created for the conference: On each card there was a short helpful sentence/statement/word at the top explaining the card a bit. For instance sentences like "I would like to add..." on a yellow card or "This is the most important thing ever!" on a red card, most were funnier than that but the funny ones I saw don't translate well to English. To make it even cooler she had different statements for every single card... Johan, I expect the same for Let's Test 2017 ,)

Summary

THANK YOU! This was one of the best conferences I've attended, I already look forward to the next one!

... and I hope I can bring some of SWETish awesomeness to PEST, a peer conference I'll attend in just a couple of weeks (we seriously need to work on how we name the various peer conferences by the way...).

Next step

After a colleague pointed out I'm not 29 years old anymore I had to revisit my About page. While reading the rest of the text this sentence made me stop...

Next step is to spread the knowledge I've obtained and be a thinking practitioner who can hopefully do what James once did for me.

I wrote that sentence over 4 years ago, before any tester outside Linköping knew I existed. So I took a moment just to reflect.

"Spread the knowledge I've obtained"
I've been teaching students in Software Testing at a vocational university for 1,5 years, I've given numerous lectures, presentations and workshops on various topics at conferences (national and international), meetups and at my workplaces and I'm now test coach at Verisure, a role in which I'm basically paid to share the knowledge I've acquired so far. Finally I've, for most of the period, been a fairly active blogger.

"Be a thinking practitioner"
Transpection Tuesday, my blog, my peer conference appearances, my many dinner discussions with top notch testers (most recently Göran Bakken), my (former) activity on Twitter all add to the "thinking" part while my testing and test related experiments at Ericsson, Verisure and Zenterio, as well as my effort during Software Testing World Cup all add to the "practitioner" part.

Most important though: The two are not separate processes. The best example of this is probably Transpection Tuesday. During Transpection Tuesday, Helena and I often discuss a challenge one or both of us have, together we work out potential solutions or experiments to run, we go back to work to try these solutions/run the experiments and finally we share/ask for help to evaluate the results at a second Transpection Tuesday. Like I said, one process.

"Who can hopefully do what James once did for me"
After graduation I got emails from two former students, both made me feel I've accomplished exactly this...

... hmm, is it raining inside or why are my eyes moist all of a sudden...

On top of that other former students, testers and other colleagues (most recently a developer) have all helped me understand my efforts to inspire, guide and mentor have actually made a difference.

It's not without pride I say: In four years I've wildly exceeded my own expectations based on my memories of what I hoped to achieve 2012. Not just my expectations for the next four years but potentially for my whole career. Shit... I need a break.

... pause music...

What's my "next step"

Took quite a bit of thinking but here's my updated "next step", or mission if you prefer that:

I've presented to, spoken with, inspired, coached and mentored testers in testing and quality. I want to continue this but to a broader audience, in ways different from what has been done before and inspire others to present, speak, inspire, coach and mentor as well.

Clarification:
Broader audience refers to e.g. developers, students (not studying testing), managers etc.

If you happen to read this in few years, do remind me to report my status, please.

What's your "next step"

Enough about me, what's your next step?

The ideal testing project

To get to the good stuff quickly without much explanation of why and how this list exists, jump to the "How to use the list" chapter.

What?
This "exercise" is rather simple:
Describe an ideal testing project

So first off, there's a lot of ambiguity in that statement. Ideal? To who? In what context? Testing project? Means what? What are the boundaries? Valid in what context?

Exactly what this means is not so important, and I'll explain why in the next chapter. But to give you a good enough idea to understand this post: "Using the experience you have, list the actions/rituals/behaviors etc. you think would make for the best possible testing". That still leaves you with a ton of ambiguity but hopefully you agree that's not the bigger issue here when I explain...

Why?
I often find when I try to look at my current context and how to improve it, I limit my thinking. By trying to list "ideal attributes" I can look at each of them and say: "do we do this today, to what extent, is it ideal in our context and how would the ideal scenario look in my specific context". Those questions, for me, tend to break that mental limitation and it's, once again for me, especially useful to see those, individually insignificant changes, that together really make for a huge improvement.

How?
Feel free to skip to the next chapter if you don't care about how this list was created.

I've tried this "exercise" three times. First time was in a workshop at Securitas Direct/Verisure on testability. One question was: "how can we improve our testability" and my brilliant colleague Niklas Grönberg, suggested we, instead of thinking about what we had, should think about how we wish things were. It generated several great ideas and everyone, no matter of current context, could add to the list.

Soon after I suggested trying the same thing on a Transpection Tuesday with Helena Jeret Mäe but focus on the project in general. We spent roughly three hours adding to the list and while doing it sharing ideas on how this could be achieved in our respective contexts.

Finally I tried the idea as a group exercise at EAST. In this case 8-10 people shouted out ideas that were discussed/further described and then added to lists on a whiteboard. When we were done everyone got a few minutes to look at the lists (we filled two whiteboards) and in an open forum we shared our key takeaways. I'll add a few notes from that at the end.

To support/guide the thinking a bit I added a few categories both when running it as an exercise on EAST and with Helena. This also (I think) helped explain what I was looking for. You will see a similar setup in this post.

How to use the list?
This is not attributes of an ideal project, it's attributes that to someone seemed ideal in some, to that person, relevant context. Thus you can't just take an item from the list and say: "well how do we achieve this". Instead I suggest you look at an idea and ask yourself, as described before:

• Is this relevant to my context?
• Why/why not?
• Do we do this today?
• To what extent?
• How would this ideally work in my unique context?

Important: I do not agree with all the items below. But I wanted to add everything I picked up as it was mentioned by some skilled tester as ideal in a context (s)he could relate to and thus potentially helpful to someone reading this.

Also notice some of the items are in conflict with each other. Once again something might be valid in one context or aspect of a context while something else is true in a different context.

General

• Clear boundaries/limitations throughout the whole project
  Time, resources etc. also knowledge about why it's limited and why the limit is set to what it is
• Clear and well communicated goals and milestones
  Iteratively challenged throughout the whole project
• Knowledge about users and buyers
  In bigger companies the buyer and user can be two very different groups
• Access (throughout the whole project) to users and buyers
• Knowledge about stakeholder
• Knowledge about market (e.g. competitors and market windows)
• Manageable amount of changes and quickly informed about these
• No hidden agendas
• A product you...
  ... like to work with
  ... find meaningful
  ... believe in
  ... want
• Balanced work flow
• Time to experiment
  Including product, working methods, techniques and tools
• Being included and listened to
  E.g. invited to meetings
• See the product being used
  And see that it matters to the customer
• Low level of (boring) repetition
• Some "hype factor"
• Test ideas generated from day one
• The agreed level of quality and customer focus is not lowered throughout the project
• Everyone strives to achieve better results when agreed upon
  Includes everything from quality/scope to efficiency/cost/time
• One focus at a time (no multitasking)
• Testing being a respected and prioritized activity that everyone cares about

Start of project and general

• Involved when it's a problem to solve not a solution to implement
• Well communicated why (why it's a problem, why it's important)
  This includes being presented the real why, not the "official" why.
• Kick off to meet, bond and discuss
• Being listened to regarding testing issues and general risks we see
• Everyone genuinely agrees upon what to build and why
  Also includes for instance salespeople and maintenance/support

Team

• Motivated
• Mixed group of expertise/experience
• Skilled to do the job
• A willingness to get better, both individually and as a team
• Problems and concerns are shared
• Having fun together!
• Everyone is listened to
• Respect for each other
• Working well together
• Domain knowledge
• Clear, effective communication

Implementation discussions

• Quality characteristics are discussed and clearly prioritized.
  This is an iterative process
  Stakeholders, customers, testers, programmers should be involved
  Most important is to state what is not that prioritized
• All solutions should have a known "why"
• Testable paper mocks or similar
• Customer feedback on these mocks, communicated to the testers
• Testability and relevant quality characteristics issues should be discussed as part of each solution
• Always 3 options to consider (<- much debate to "at least", "roughly" or "exactly" three)
• Key areas and lower prioritized areas are discussed and agreed upon

Test planning and strategy

• Risks collected from testers, customers, stakeholders etc. discussed and prioritized
• Discussions on how to test with programmers involved
• Test process revisited and tweaked to fit current context
  E.g. what documentation, for who and why. When/how to involve programmers etc.
• Testers, programmers, customers and stakeholders read, understand and commit to the strategy
• Strategy is continuously revisited throughout the whole project

Testing / Developing

• Feedback on errors (email, popup), rather than scanning a log
  (could be achieved with a tool scanning a log of course)
• Being able to start a flow at any relevant position
  (e.g. in the middle of a setup wizard)
• Being able to extract any relevant information from the product
• Early first delivery and frequent deliveries throughout the whole project
• Attention on bugs, not just new functionality
• Programmers being aware of and taking relevant quality characteristics into account when developing
• High quality code delivered
  Helped by code reviews, static code analysis, TDD/BDD, quick manual sanity tests, (automated) smoketests, skilled programmers taking pride in delivering great quality code etc.
• Testers pairing up with programmers, customers, stakeholders when testing
• No distance between programmers and testers (and stakeholders)
• No delays from code written to code up and running in the test environment
• All target systems/platforms/versions used by customers available to testers
• Regularly time to test with and speak to customers
• Why should be well explained and motivated for every demanded administrational action
  E.g. documentation and reporting
• Iterative reflection on processes and tools that everyone is engage in, genuinely want to engage in and given time to act upon.
• Progress is celebrated!
• Everyone is having fun!
• Testers are not the only ones testing
  Stakeholders, programmers and customers test the product as well, both while pairing with testers and alone.
• Mistakes being embraced
  Admitting mistakes should never lead to shame, everyone naturally want to share mistakes done rather than keep them a secret.
• Lots of questions asked and people taking time to answer questions
• Amazing tools that support the desired process, not dictating it
• Short optimized feedback loops
  Includes communication
• Testers focusing on testing not checking
• No stupid measurements and KPIs
• Stakeholders accept relevant information, no extra translation necessary
  Much of this is probably better explained as both parties striving to understand each other and to learn speaking each other's language.
• Team! Not individuals
• Continuously challenging goals and milestones as more is learned about the product
• Stakeholders clearly explaining why they chose to ignore certain risks communicated by testers

Finishing up

• Customers blown away
• World peace and end of world hunger
• Time to reflect
  ... and result of reflection being used as input for the next project
• Clear finish
• Getting the time to feel proud
• Being shown appreciation from managers, customers and stakeholders
• Lessons learned shared between various projects (if projects are run in parallel with different testers)

Some reflections from the wrap up during EAST

As I said before we ended the exercise during EAST with sharing lessons and ideas to bring back to our respective contexts. One observation was most of them related to communication like ideas to improve communication or information someone should ask for. Another was very different kinds of lessons were described but everyone seemed to identify something relevant to their context that they might not had thought about before, or now thought about in a different way, which to me indicates that the list can be valuable to most testers.

Feel free to share your own additions as comments below, I will gladly add them to the list!

And to all of you who've added to this list already thank you! This is not my work, it's our work! Actually I even tried to shut up during EAST! (which went... okey I guess, at least compared to how it usually ends up)

Thank you for reading!

So when will you be done?

Project manager: How's it going?
Tester: Right now we see a lot of problems in the export functionality but import seems a lot more stable than last week
Project manager: Okey, so when will you be done?

The next time you get this question, answer:

I can be done tomorrow, or next week, or in September if you want, when I'm done (testing) is irrelevant, what's relevant is how confident you want to be with the product when it's released.

When are we "done"?
We don't really decide when we're done, that's up to the people delivering something for us to test (developers, designers, people responsible for builds etc.) and the people deciding when a product is ready for the market (let's call that person(s) a project manager in this post). How is that?

First of all we don't create quality, developers etc. do, we only make a quality assessment. That means we don't control when the "desired level of quality" is reached, that's once again up to developers etc.

Second of all we don't decide the "desired level of quality" itself, that's up to people with better knowledge of the market, internal business situation etc., the person(s) I call the project managers in this post.

And finally we don't know about the actual level of quality in a product, we only try to make an, as good as possible, assessment. This poses another problem as we have to try and communicate the level of quality we think the product is at and communicate this in a way so that the person actually deciding if the quality is good enough (once again, the project manager) can take an as good decision as possible. This is typically where risk comes in: "If we release now, here are the risks I see, it's your call if we can live with those or not" and the last part, if we can live with those or not, that could be translated to "if I'm done or not".

Problem, QA
Now there are a couple of problems.

First, why doesn't testers then decide if we should release or not? After all, we have the most detailed view of the system? Well, we call that quality assurance (QA) and there's several reason why I highly dislike that concept: One is we don't know the business well enough! There might be a market window we can't miss or that will be very costly to miss, we might need more money now to make crucial investments and so on. Long story short: Quality, no matter if we like it or not, is far from the only thing deciding when a product is ready to release. And also, I strongly believe QA could, or rather would, hamper our ability to stay critical and "test minded" as we would be biased to fulfill other goals (like the wish/pressure to release).

Problem, estimates
Second I know someone is saying: "But as a project manager I can't just have a huge question mark in my time plan, I need an estimate of when we think we can release!". Sure, in my team test is currently not estimated (we do have some days between code freeze and release but they are set, and same for every sprint), instead I just report if I see problems a long the way that I think jeopardize our ability to release when planned (risk). That's one way to do it.

At my previous work we did specific estimates for testing, nothing wrong in that, but it doesn't change anything. Once again, that's for someone to be able to make a plan. In the middle of everything though, the time is suddenly irrelevant and we can only hope our assessment of how long it will take developers etc. to reach a perceived level of quality where the project manager feels confident to release, is anywhere near reality.

... Oh, and test cases, or rather pass-fail/pass-total ratios, don't change any of this, it's only a way to pretend you have control over something you don't control.

Summary
Never ask a tester when (s)he will be done, ask about current risks and what consequences (s)he thinks come with those risks.

As a tester, don't provide a "time left", it's not up to you when you're done. But do help the person asking to figure out an estimate for themselves to when the product/feature might be ready by presenting known problems, how deep areas are covered and other relevant information.

Credit
This comes from a Transpection Tuesday with Helena Jeret-Mäe.

Clarification
I received a comment and want to add a clarification: At my current work, this is not a problem and has never been (at least I haven't come in contact with it)... but it has been earlier in my career.

Arguing for Exploratory Testing, part 2, Reuse

Intro
You can read a bit more about this series in the first post:
Arguing for Exploratory Testing, part 1, Traceability

The topic for our second Transpection Tuesday on "Arguing for Exploratory Testing" was Reuse.

We finished with two open questions:
Can we ensure we actually repeat the exact same test a second time?
How do you actually achieve reuse in exploratory testing (when it is desired)?

Reasons to reuse tests
First we tried to state reasons someone would want to reuse test cases:

• Save time during test design
• Functionality is changed and we want to rerun the full/part of the test scope
• We want to verify a (bug) fix

Preconceptions

Looking at reasons quickly led us to some preconceptions which became the topic for a big portion of the session:

• Effort = Value
• Equal execution = Equal value
• Our scope is (almost) complete
• Reuse = free testing
• A monkey can run a test case

Preconception: Effort = ValueSince we've invested so much time (as well as money and prestige) in writing test cases they must be worth more than a single execution.

• Even if presented with clear evidence we may reject it to defend out judgement
• We may overestimate what a test case is useful for (we want to get the most out of our work)
• It's my work, criticize it and you criticize me not the work! (common and unfortunate misconception)

It takes a lot of self-esteem to say: "Yeah I screwed up, could you help me?", especially in an environment where mistakes are not accepted. Notice many of the "so how can we make the most of this mistake" still communicates "so you made a mistake, now you'll have to suffer for it by telling us why you are a failure". It takes a lot of work to change this.

Preconception: Equal execution = Equal value

Let's say we execute the exact same steps in a scripted and an exploratory way, wouldn't that be two identical tests? We believe not.

1. Your goal differs. With test cases your goal is to finish as many test cases as possible (progress). That's how you measure "how much testing you were able to perform". In exploratory testing you are judged based on the information you provide thus you should be more incline to spend a few extra minutes observing/following something up even when it's not "part of your test".
2. Your focus differs. When you have a script you have to focus on following that script. In exploratory testing your goal is typically to find new leads to base the next test on. That means in one case your focus is on the product and in the other on an artifact. Think about the Invisible Gorilla experiment.
3. Scripts easier bias you not to observe. In a script you typically have verification steps e.g. "verify X=5". We believe this could bias you to not be as observant during the other steps: "this is just setup so nothing should happen that concerns me".

Preconception: Our scope is (almost) complete
We know a feature's boundaries (specifications, requirements) so when we set the scope for testing we can, and usually will, cover almost the entire feature.

• We can't know the boundaries of a feature:
• We will impact and use other components not "part of the feature" e.g. other code in the application, the operating system, surrounding applications, third party plugins, hardware, hardware states etc.
• We interpret planning documents differently, adding parts, discover things we couldn't had anticipated, correct mistakes or interpret something differently than intended by the author and/or interpreted by the tester.
• We can (almost) always tweak a test a little bit (e.g. change input data or timing). But testing all combinations (we recognize) is way too expensive. Also there are usually so many ways an application can be misused (intentionally or unintentionally) that even with a ton of creativity we can't figure out them all (ask any security expert .)

So our scope is basically a few small dots on a big canvas rather than a well colored map. But those dots are (hopefully) carefully selected to protect us from the greatest risks we can anticipate. Still, they are only dots.

As testers we easily support the preconception of full coverage by answering questions like "do we cover this feature now?", "is all testing done?" etc. with a simple "yes" or "almost". The more accurate answer would be "we cover the most important risks we identified limited by our current knowledge, time available and other constraints", but that answer is not very manager friendly which leads us to...

There is a general lack of knowledge and understanding of testing in most organizations. And we decided to stop there since that was a way too big question to tackle at the point we got there. But it's an important questions so please take a moment and think about it for yourself: How can you improve understanding and interest for test in your organization?

A final note. Since we only cover a small part, reusing a test scope will not help us catch the bugs we missed the first time. How big of a problem that is differs but repeat a few times and it may scale up in a nasty way.

Preconception: Reuse = free testing
We've already written the test case so wherever it's applicable (which should be self-explanatory) we can just paste it into out test scope and voí la! Free coverage!

The big issue here is the "self-explanatory" part. Problem is what fitted well in one feature might not do it in another even similar one. Even without needed tweaks we still have to figure out what the test case actually does, so that we know what we have covered with it and what we still need to cover in other ways.

This process is expensive, really expensive, so sure we save time not having to figure out the test and how to practically run it all over again but consider the time it takes to find the test case, analyse what it covers, analyse what it doesn't cover, analyse how it interacts with existing test cases, analyse if something has changed that impacts the test case compared to last time and so forth.

Preconception: A monkey can run a test case

• We all interpret things differently. Click can mean single click, double click, right click (already assuming the first two were left clicks), tab and use enter, middle button click, etc. Even a well written, simple test case can lead to different interpretations.
• One thing we're looking for is unexpected behavior and it's in the nature of "unexpected" to be something we can't plan for. Thus to get much use of a test case we need to handle the system well enough to investigate and identify when a behavior is unexpected or undesired.
• We do a ton more observations than we consciously think of. These observations takes practice, focus and skill. For example, when you boot your computer you would react to a lot more things than you would add in a "boot test". Examples: screen is blinking, smoke is coming out, lights in the room flickers, you hear strange mechanical sounds, all these should catch you attention but are unlikely written down.
  
  More skill and/or focus can lead to more valuable observations: The login screen looks different, memory calculations are wrong, it's slower than usual/expected, BIOS version is incorrect, the operating system's starting mode is wrong etc.
• When we don't fully understand something we tend to write it down less detailed (sucks to look stupid by writing down something incorrect and we're too lazy to investigate every detail we don't understand, it's easier to investigate as we get there).
• When we write a test case based on specifications, requirements and other "guesses" of how the end system will work even a flawless instruction will sometimes not correspond to how the system is actually working (including when working as desired). This of course requires the person executing to be able to correct the test case thus understand both the intention with the test case and how the system works.
• If we don't understand the system we may lose a lot of time setting up fully or partly irrelevant variables to the values stated in the instructions. The immediate comment is, if we have stated irrelevant variables in the test case we've failed. Consider then that the variable might be irrelevant to the test but mandatory to the system (e.g. you have to set a valid time server). Leave that out and the person executing once again needs to understand the system.

When is reuse actually beneficial?

• We have rewritten something from the ground up but want it to externally still work the same. Reuse could save time.
• We have some critical paths through the system that can't break.
• We need to quickly regression test a feature without having to dig in too deep in the feature itself.

But remember that the one executing still should understand the test and the system to ensure tweaks (using different input values, triggering different fault cases etc.) can be made and important observations are more likely to be made.

How can we achieve reuse in Exploratory Testing
Not covered much by this particular session but a few thoughts:

• Charters
• Debrief notes
• Test ideas
• Test plans

Try creating a report specifically used as a "feature summary" including valuable operational instructions, general test ideas, impacts, lessons, problems, tools, important details, testability etc. We did kind of this at my former company where we let the test plan continuously turn into an end report as we added lessons from our testing. This would not only help when retesting a similar feature but also as educational material or test plan input for instance. Important though is to stay concise, noise is a huge enemy! The number of readers of a document is inversely proportional to the number of pages in the document, you know .)

A few notes on test case storage
First off I love this post on having a big inventory of test cases by Kristoffer Nordström.

It's easy to think something you've already created is free, but there's no such thing. Having a large inventory to test cases costs in many different ways:

• Storage
• Noise (it's one more thing testers have to keep track of)
• Another tool/part of tool for testers to stay updated with / learn / understand
• For a test case to be fully reusable later it should be kept up to date. How many refactors all their old test cases as functionality is changed?
• ... if you do, that sounds really expensive.

Summary

Reuse has it's place but be careful!

Remember reuse means inheriting blind spots, has a cost and still requires the person "reusing" to know just as much about the feature, system and testing in general as if (s)he wasn't reusing old checks.

Take care, and I hope these Transpection Tuesday notes (even though somewhat messy) were helpful!

... and of course, thank you Helena!