Monthly Archives: October 2009

My Attempt to Clarify My “Everything” statement

I got some comments on my post “Test Everything all the Time” — most notably people commenting that it’s impossible to test “everything”. I can’t agree more. The intention of the post was to make the point that we need to be able to test “everything we can” all the time. That is, you should be constantly iterating your automation, and not “waiting to run” the automation. Also, the point was to talk about how test design can solve all of your problems, not the automation tool. The tool is just an means to an end.

Automation allows us the ability to test wide swaths of functionality that we would not be able to do manually. It’s virtually impossible for a manual test team to adhere to the adage “Test Everything all the time”, they can’t do it.

Separately, a trend is occurring in our industry where testing is moving toward more and more code inspection and code execution strategies. I notice many more testers trying to determine quality by working with the code, rather than the finished product. I think this is a sound approach, in fact the sooner you can test the better, however for many in our industry code inspection is a skill that many of us don’t have. However, many of us do have test design creativity, and it’s here that when married to a good automation tool, can produce very reliable and consistent results.

As I noted in my blog post, you need to strive to have all of your tests done automatically, and that manual intervention is used only when necessary.

Giving an Atomic Bomb to a Caveman…

rman2739l - retouchThe challenges with any automation effort is to know your capability. I’ve seen too many automation efforts begin and end with a tool decision. Generally these tools are very complex pieces of software that do many more things then we would ever use in our normal everyday testing. It even adds more misery to the situation when we give this new tool to people who are entirely incapable of using and scaling the “newly” selected savior to our automation effort.

When I teach, I call this moment .. “it’s like giving and atomic bomb to a caveman”

Why do I call it this? well, because if I gave an atomic bomb to a caveman and then said to the caveman, “hey, I just gave you the most powerful weapon ever designed by man, and go use it against your enemies”… he would look at me and say, “thanks, but how do I use it.. “.. and I say.. “you’ll figure it out, just don’t be near it when it goes off..”

The caveman takes his spear and axe and hits the bomb, shakes it, kicks it around, and finally say to his clan mates.. “see…this thing doesn’t work.. I have no idea what that guy was talking about when he said it was the most powerful weapon, he’s full of it..”

That’s what happens to most automation tools that are not properly setup for your teams.

If you give a complex Integrated Development Environment (IDE) to tester who doesn’t or hasn’t programmed in years, you are just asking for trouble. However, this practice continues to this day, and what I’ve found it happens far more often then I think most companies would care to admit.

So how do you fix this problem?

Test Design…

Our solution to this problem was to develop Action Based Testing.

“Action Based Testing” is an attempt to give testers/biz analysts who don’t program a test design model that easily supports automation. When we talk about automation, we aren’t talking thousands of tests.. we are talking millions of tests. That’s the level you want to get to, since automation affords us the ability to geometrically scale our testing coverage. But you can only do this, if you have a test design framework that supports the scale and supports your testers. AND!! – if you have a tool that has adoption characteristics that support all members of the team, otherwise, you’re back to giving nukes out.

There are tools out there, cucumber, which is a context aware automation tool that is designed around the subject matter experts, that is the blackbox testers or business analysts. I’ve used this framework in addition to the framework (ABT) I helped develop at LogiGear.

The point of this post is to get you away from thinking tools are panacea’s and get down to thinking about test design and how the tools fit into this framework. I tend to use a lot of tools in my automation strategy, since I’m not one that believes in one size fits all.

Go out and choose a good automation friendly test design framework, learn it, get others involved with it and see how they interact with the new framework, and see if they are capable of applying the test design parameters you’ve setup for them inside the tool.

Good luck.. and don’t press the red button.