What Rules? The Great Hustle of Exploratory Testing

The tournament will span for about 1 hour with each registrant competing as an individual throughout 3 different games requiring varying levels of skill, speed, and accuracy.   

The line above is the rules, in their entirety, for a darts tournament that was held earlier this year in Boston, during our Go-to-Market-Kickoff (GTMKO) here at SmartBear. This event brings in SmartBears from all over the world to learn, inspire, and in my case this year—to fight like hell to win an employee darts tournament against all odds. 

Those odds, to me, were very low. I hadn’t actually played darts in nearly 20 years, and I’d be competing against a large number of my far younger colleagues visiting from our office in Galway, Ireland.  

When I received the rules above in my inbox, they looked familiar. They reminded me of the vague sets of requirements that lack critical details that software testers are often tasked with using to “assure” the quality of a piece of software. (Note: “assure” is in quotes because that’s not actually what testers do or should even say they can do. It’s a long story.) 

Requirements, by their very nature, will always be incomplete. Exactly how incomplete is a bit subjective. Some testers may receive a set of requirements to test against and excitedly say, “Got it! I know exactly where to start and where to stop.” Others may sigh and wish, again, that they’d been invited to be a part of the requirements gathering process.  

And there are other testers who, to some degree, welcome brief, vague requirements, because it just gives them more room to go exploring. Knowing that what they’ll likely find outside of the straight and narrow rules of the road will be far more intriguing and thought-provoking than what they find by trying to stay on it. 

The insufficiency of rules and requirements

On the afternoon of the tournament, I pulled up the rules one last time to make sure I didn’t miss anything that would disqualify me or would lessen my chance of winning by me failing to note. Then, my English major/major grammar nerd brain went to work: 

“The tournament will span for about an hour” 
I have no clue what time the tournament will start or end.  

“Each registrant will compete as an individual” 
Makes sense. 

“…Throughout 3 different games” 
I really wish I knew what the games will be, but that’s okay. They’ll tell us at some point. Right? 

“…Requiring varying levels of skill, speed, and accuracy” 
Hold on. What? How could the tournament require “varying levels” of those three things to win? Will “poor or “low” levels of skill, speed, and accuracy be rewarded equally as those that are “great” or “high?” Maybe there’s a prize for the worst score? But how could low “speed” even be measured? Will someone get a prize for the fewest darts thrown? I’m slow! Maybe I’ll win! 

A quick—but brief—loss of faith

As someone who is often unreasonably competitive, even in competitions that I have little to no faith in being able to win, I grew restless, and almost a little nauseous as we got closer to leaving for the tournament. It was being held at an upscale darts establishment called Flight Club which was a little over a half-mile from the hotel where we were all staying during GTMKO. Given that it was in the single digits outside that evening, many of us opted for the complimentary shuttle to avoid dying of frostbite if we were to walk. Some did walk and did not die of frostbite, but I’m sure I would’ve. 

I believe it was the third time that the shuttle returned to our hotel that I was able to squeeze on board. By the time our van arrived, Flight Club was packed. Nearly every dartboard had large numbers of people already playing on them. Many bullseyes were hit, high-fives and chest bumps were doled out, and pint glasses were emptied. 

If the competition had already started, I was definitely behind. By the time I grabbed a pint glass of my own, ate a couple of hors d’oeuvres, and found a lonely, open dartboard in the corner of the room, an hour had passed.  

“Oh, well,” I thought. The night would still be a blast, and it felt kind of nice to not have to worry about trying to adhere to the tournament rules I couldn’t make sense of anyway. Plus, maybe I’d get that award for “lowest accuracy,” or “slowest pace of play,” like I still believed at the time might be a thing. 

Screw the rules

My Irish colleague, Frank, made his way over at some point and we decided to play a match against each other. Much to my surprise, this Irish individual was not humiliatingly better at darts than I was. That’s not to say he wasn’t good; he was! But, so was I! As an eternal down-player and excuse-provider of my own actual talents and skills, I was shocked that I wasn’t terrible.  

At some point during the impossibly loud evening, a Flight Club employee grabbed a microphone, and, while he tried his best to shout over the noise of hundreds of dart-flinging SmartBear employees, not much of what he said was understood by anyone, other than that the tournament was now starting. I, for a moment, thought about running up to him and asking where to sign up, how would scores be tabulated, which three games were we playing, and was there any way I could find out which of those, “Varying levels of skill, speed, and accuracy” would qualify for an award. I had no problem being vain enough to throw the competition by playing far worse than I was actually capable of if a prize of any actual monetary amount or bragging rights were on the line. 

I also remembered how much fun I was having by not caring about the rules. I thought about how poor or insufficient software requirements are not only “the norm” for many testers, they’re the last thing that would keep any tester who loves their job from the thing they love most—simply testing

So, Frank and I continued to play each other exactly as we had before the start of the tournament. He would win, and then I would win. I’d occasionally manage to win two straight, and then he would do the same. This went on for dozens of games. Eventually, some friends dropped by to join us for some group competitions, and while Frank and I played for different teams, we each continued to finish around first, second, or third place in each game. 

Reality check

The competition died down at some point over in our little corner, and some from our group left to chat with others at the party. I felt a tap on my shoulder. It was the employee from earlier who’d announced the start of the tournament. Even with him standing right next to me, it was almost impossible to hear him. 

“Just so you know, I’m about to announce the current leaders of the competition!” he said, excitedly. 

“Oh, OK. Sounds…good. This is a really cool spot. Thanks for having us.” 

“I just wanted to make sure you’ll be listening when I announce who’s currently leading.” 

“Um, I’ll try? I really couldn’t hear you at all the last time you were on the mic, but that’s okay, go for it.” 

“I’ll just show you now in case you can’t hear me.” 

I had no idea why he was so insistent on me knowing this information earlier than everyone else. He then held up a cocktail napkin that had the top 5 highest scores at that point in the competition. 

Frank and I were not only tied for first place, but we were also absolutely destroying every…other…player. 

“There’s NO way that’s right. How in the world do we have that many points?” I asked. 

It IS right. You get three points for every first-place finish, two for second place, and one point for third. I just looked at the numbers. They’re right. Anyway, I’m going to go announce it to everyone else; just wanted to let you know first! Great job! Keep it up and you’ll win this thing!” 

You know that scene in the movies that’s done so often, where someone gets some sort of life-changing news, and everything taking place around them goes silent and the camera dramatically zooms in on the person’s dumbstruck face?  

That’s exactly what happened at that moment. 

The entire bar went silent, a truly miraculous feat for a place that loud. I could barely make out the muffled, unintelligible shouts of the Flight Club employee, but I did see him emphatically pointed in our direction, I think I heard my name, and then I instantly snapped back to reality. I rushed to Frank. 

We’re WINNING. By…A LOT,” I said to Frank, not in disbelief, but in a fully coherent understanding of how that had happened. 

How’s that even remotely possible? We’re doing…okay…I would say, but there MUST be others here doing better than us. Some of these guys can really play.” Frank replied. 

“It doesn’t matter. It’s based on everyone’s total number of first, second, and third-place finishes. That’s IT. You and I are the only ones who’ve been playing with only two people to a game for almost the entire night. It’s not that we’ve played ‘better’ than everyone else; it’s that we’ve probably played 10 times as many GAMES as everyone else.” 

Don’t fall for “acceptance”

It suddenly made perfect sense to both of us. And so, we did what anyone with that information at that time would’ve done. We played as many games as humanly possible in the remaining time of the competition, and we continued to volley first and second-place finishes because those were the only places either of us could’ve finished. Even “last,” it turns out, was still worth two points. 

And, at the end of the night, we finished with the exact same number of points, miles ahead of every other competitor. 

Exploratory testers find themselves in these sorts of situations all the time. It’s not that they won’t play by the rules, it’s that they know there are entire worlds of possibilities and invaluable learnings outside of the rules. As the most curious and most-comfortable-amongst-unfamiliar-territory testers on your team, they have to be given the time to explore what others either won’t or would never think to.  

Rules are important. In sports, breaking them often comes with some sort of justifiable repercussion or punishment. But merely following the rules will hardly get an athlete or team rewarded for doing so. Likewise, in software, acceptance criteria is also important. Just remember that meeting acceptance criteria merely makes you “adequate.”  

Lots of companies are willing to be adequate, so let them. Explore what will make you exceptional. There may be a medal in it for you.