Category Archives: Test Process Improvement

Agile Retrospectives: The Preventative Medicine

3months

One of the features of using Agile methods is the opportunity for continuous improvement within a project. There are a number of improvement opportunities throughout a typical iteration or sprint─over the next few weeks I’m going to walk through a few, starting this week with the Retrospective. Retrospectives are one of the many tools in Scrum and other Agile methods that are absolute must-haves for continuous improvement.

2010 Global Testing Survey Results: Test Process & SDLC

Michael Hackett, Senior Vice President, LogiGear Corporation

Process

The objective of this survey and analysis is to gather information on the actual state-of-the-practice in software testing today. The questions originate from software development team assessments I executed over the years. A process assessment is an observation and questioning of how and what you and your team does.

Book Excerpt: The Agile Samurai

Jonathan Rasmusson

Author Jonathan Rasmusson explains in his latest book how to successfully set-up, execute and deliver Agile projects. Download the excerpt below for “Chapter 7: Estimation The Fine Art of Guessing.” To read his interview in last month’s issue, please click on “Spotlight Interview: Jonathan Rasmusson” to read his views on the best practices for test automation in Agile projects.

Same or Different? Retrospectives and Post-mortems

Michael Hackett

Let’s look at a few distinctions between the two process improvement practices that make all the difference in their usefulness for making projects and job situations better! An extreme way to look at the goals of these practices is: what makes your work easier (retrospective) versus what did someone else decide is best practice (post-mortem)?

First, look elsewhere for a “how-to.” There are many published materials on how to do a sprint retrospective and a post-mortem. We are looking at the differences in the mindset. To look at how post-mortems and retrospectives are different, let’s go back to the approach and foundation for these meetings. The following distinctions between the Agile and traditional software development mindset loom large in the mindset entering retrospectives and post-mortems.

Get High Performance Out of Your Testing Team

Michael Hackett

Testing is often looked upon by many as an unmanageable, unpredictable, unorganized practice with little structure. It is common to hear questions or complaints from development including:

  • What are test teams doing?
  • Testing takes too long
  • Testers have negative attitudes

Dr. Cem Kaner shares his experiences on software testing and essential skills needed to be software tester.

Dr. Cem Kaner – Director, Center for Software Testing Education & Research, Florida Institute of Technology

PC World Vietnam: What did you think of VISTACON 2010?

Dr. Kaner: I am very impressed that the event was very professionally organized and happy to meet my old colleagues to share and exchange more about our area of expertise. I was also very excited being one of the speakers at the conference and presented on some topics which focused on software testing as well as sharing some of my experiences from my research and work in this field.

PC World Vietnam: What do you believe are the most important skills required for a tester?

Dr. Kaner: Testers should have a good knowledge of the profession that they are working in; have testing skills and certain technology basics. For example, when testing banking or finance software, you should know about their trading activities, how they liquidate, and have a thorough grasp of some of the testing tools, testing plans and test automation methods. The technology basics means you should grasp some steps of the software development process, programming language skills, a knowledge of algorithms and operating systems. Besides that, other essential factors that can help you become a successful tester are having a good observant mind to easily recognize bugs during the testing process, the passion to apply the highest standards to the project, and finally, that you are always interested in new technology and new products.

PC World Vietnam:
Could you please share with us your experience of undertaking testing projects in different countries?

Dr. Kaner: I have had the opportunity to work and collaborate on projects in the US and India in the field of software testing. From my experience, you should focus on the test cases created during the period you are creating code. Parallel to that, you need to improve the testing process to achieve the desired quality.

It is important to find ways to solve problems by preventing bugs instead of finding bugs. Testers should talk and share their experiences of finding bugs to improve their professional skills. Also, the wages paid to a tester in each country varies. For instance, in the US, the cost of hiring a tester is very high, so most companies outsource their projects to another country to save costs. According to Global Services Media, last year Ho Chi Minh City ranked 5th in the list top 50 emerging global cities in outsourcing software. That is a good sign for the development of the Vietnamese software industry.

PC World Vietnam: Vietnamese students often tend to become programmers instead of testers, what do you think about this?

Dr. Kaner:
Actually, the hobbies and passions of each person will always be different. I myself have been rotated between the two positions of testing team manager and software programmers’ team manager. Some people find the testing field interesting, but others discover their own skills are more in line with programming. Those who prefer to focus on solving a problem or specific solutions may be appropriate as a programmer. But if you decide you want to become a tester, you should do your best to make this happen by studying hard and doing serious research.

PC World Vietnam: Thank you, Dr. Kaner

(This interview is condensed and translated from the PC World Vietnam Magazine – October 2010 Issue)

Computer Scientist – D. Richard Kuhn will provide some insights on how to become a software tester and shares his interest in combinatorial testing.

D. Richard Kuhn - Computer Scientist, National Institute of Standards & Technology

LogiGear: How did you get into software testing? What did you find interesting about it?

Mr. Kuhn: About 10 years ago Dolores Wallace and I were investigating the causes of software failures in medical devices, using 15 years of data from the FDA. About that time, Raghu Kacker, in NIST’s math division, introduced me to some work that his colleague at Bell Labs, Sid Dalal, had done on pairwise and interaction testing for software. The idea behind these methods is that some failures only occur as a result of interaction between some components. For example, a system may correctly produce error messages when file space is exhausted or user input exceeds some limit, but crashes only when these two conditions are both true at the same time. Pairwise testing has been used for a long time to catch this sort of problem.
I wondered if we could determine what proportion of the medical device failures were caused by the interaction of two or more parameters, and just how complex the interactions would be – in other words, how many failures were triggered by just one parameter value, and how many only happened when 2, 3, 4, etc. conditions were simultaneously true. We were surprised to find that no one had looked at this question empirically before. It turned out that all of the failures in the FDA reports appeared to involve four or fewer parameters. This was a very limited sample in one application domain, so I started looking at others and found a similar distribution. So far we have not seen a failure involving more than 6 parameters. This does not mean there aren’t any, but the evidence so far suggests that a small number of parameter values are involved in failures. The reason this is significant is that if we can test all 4-way to 6-way interactions, we are likely to catch almost all errors, although there are many caveats to that statement and we don’t want to oversell this.

LogiGear: You are working in very special field of software testing. It’s quite different to common software testing like: Unit tests, Automation testing, GUI Testing, etc…. So what kind of work are you doing? How did you pick those specific topics? Can you help explain this type of testing to our readers?

Mr. Kuhn: NIST’s mission is to develop better measurement and test methods, so Raghu and I proposed an internal project to build on the empirical findings discussed earlier. The first problem that had to be solved was to find ways of efficiently generating tests that cover all t-way combinations, for t=2, 3, 4, etc. This is a hard mathematical problem that has been studied for a century or more. Jeff (Yu) Lei had developed a very efficient algorithm to cover 2-way combinations as part of his dissertation at NC State, so we talked to him about extending the algorithm for t-way combinations. He did this successfully and we worked with him to incorporate the algorithm into a practical tool for testers, which is now being used at more than 300 companies.
The tool, ACTS, makes it easy for testers to enter parameter names and values for each, then generates test values that cover 2-way through 6-way combinations. For realistic applications, this can produce thousands of tests in some cases, so we have also worked on ways to automate the oracle problem for testing – how to determine the expected results for a particular set of input. This allows testing to be (almost) fully automated, so running hundreds or even thousands of tests can be done at a reasonable cost.

LogiGear: How can a college student prepare to go into software testing and become really good at it? What should he/she look for in teachers, courses, and methods?

Mr. Kuhn: The biggest challenge in software assurance is how to do it economically. Assurance methods used for life-critical software, such as in aviation, are too expensive for most applications, yet every year we become more dependent on software working correctly. One of the best ways to reduce cost, as in other fields, is to bring more science and technology to bear on the problem, so courses that focus more on the science than on managing testing will be important. Testing is only one part of assurance. An important area of research is how to integrate formal methods for specification and proof with testing, including combinatorial interaction testing.

LogiGear: What sort of graduate programs? Also, in your opinion, what are some of the more interesting research questions people are asking now and what do you think they’ll be researching in, say, 5 years?

Mr. Kuhn: Graduate programs with an established program in software engineering will be good choices for work in software assurance. In addition to how to integrate formal methods with combinatorial testing, important questions include: how to order or prioritize tests to find faults more quickly, how to identify the particular combination(s) that caused a failure, how to extend these methods to very large problems, and above all, determining the effectiveness of these methods in the real world. It would be nice if these problems were solved quickly, but I’m sure they will still be important in 5 years. Combinatorial testing has grown very rapidly in the past 10 years, from around 4 or 5 papers a year in the 1990s though 2002, to more than 50 per year recently, so it seems to be attracting a good deal of attention from researchers.

LogiGear: Lastly, who do you consider to be some of the leaders in this field and what are they doing?

Mr. Kuhn: In addition to Jeff Lei (U. Texas, Arlington) and Jim Lawrence (George Mason U.), who have been part of our team since the beginning, Charles Colbourn, at ASU, is the expert on t-way covering arrays and algorithms. We’re working with Renee Bryce (Utah State) and Sreedevi Sampath (U. of Maryland Baltimore County), who are looking at test prioritization. Myra Cohen (U. Nebraska-Lincoln) has done a good deal of work on both the theory and application of these methods. There are many other leaders including Sid Dalal, George Sherwood (from former AT&T, Bellcore, SAIC) , and Alan Hartman (from IBM)

A lot of people are now working on getting these methods into practice. Among those we are working with are Jim Higdon at Eglin Air Force Base and Justin Hunter of Hexawise. Turning research ideas into something that works in the real world can be more challenging than the research in some ways.

LogiGear: Thank you, Mr. Kuhn

Computer Scientist – D. Richard Kuhn will provide some insights on how to become a software tester and shares his interest in combinatorial testing.

D. Richard Kuhn - Computer Scientist, National Institute of Standards & Technology

LogiGear: How did you get into software testing? What did you find interesting about it?

Mr. Kuhn: About 10 years ago Dolores Wallace and I were investigating the causes of software failures in medical devices, using 15 years of data from the FDA. About that time, Raghu Kacker, in NIST’s math division, introduced me to some work that his colleague at Bell Labs, Sid Dalal, had done on pairwise and interaction testing for software. The idea behind these methods is that some failures only occur as a result of interaction between some components. For example, a system may correctly produce error messages when file space is exhausted or user input exceeds some limit, but crashes only when these two conditions are both true at the same time. Pairwise testing has been used for a long time to catch this sort of problem.
I wondered if we could determine what proportion of the medical device failures were caused by the interaction of two or more parameters, and just how complex the interactions would be – in other words, how many failures were triggered by just one parameter value, and how many only happened when 2, 3, 4, etc. conditions were simultaneously true. We were surprised to find that no one had looked at this question empirically before. It turned out that all of the failures in the FDA reports appeared to involve four or fewer parameters. This was a very limited sample in one application domain, so I started looking at others and found a similar distribution. So far we have not seen a failure involving more than 6 parameters. This does not mean there aren’t any, but the evidence so far suggests that a small number of parameter values are involved in failures. The reason this is significant is that if we can test all 4-way to 6-way interactions, we are likely to catch almost all errors, although there are many caveats to that statement and we don’t want to oversell this.

LogiGear: You are working in very special field of software testing. It’s quite different to common software testing like: Unit tests, Automation testing, GUI Testing, etc…. So what kind of work are you doing? How did you pick those specific topics? Can you help explain this type of testing to our readers?

Mr. Kuhn: NIST’s mission is to develop better measurement and test methods, so Raghu and I proposed an internal project to build on the empirical findings discussed earlier. The first problem that had to be solved was to find ways of efficiently generating tests that cover all t-way combinations, for t=2, 3, 4, etc. This is a hard mathematical problem that has been studied for a century or more. Jeff (Yu) Lei had developed a very efficient algorithm to cover 2-way combinations as part of his dissertation at NC State, so we talked to him about extending the algorithm for t-way combinations. He did this successfully and we worked with him to incorporate the algorithm into a practical tool for testers, which is now being used at more than 300 companies.
The tool, ACTS, makes it easy for testers to enter parameter names and values for each, then generates test values that cover 2-way through 6-way combinations. For realistic applications, this can produce thousands of tests in some cases, so we have also worked on ways to automate the oracle problem for testing – how to determine the expected results for a particular set of input. This allows testing to be (almost) fully automated, so running hundreds or even thousands of tests can be done at a reasonable cost.

LogiGear: How can a college student prepare to go into software testing and become really good at it? What should he/she look for in teachers, courses, and methods?

Mr. Kuhn: The biggest challenge in software assurance is how to do it economically. Assurance methods used for life-critical software, such as in aviation, are too expensive for most applications, yet every year we become more dependent on software working correctly. One of the best ways to reduce cost, as in other fields, is to bring more science and technology to bear on the problem, so courses that focus more on the science than on managing testing will be important. Testing is only one part of assurance. An important area of research is how to integrate formal methods for specification and proof with testing, including combinatorial interaction testing.

LogiGear: What sort of graduate programs? Also, in your opinion, what are some of the more interesting research questions people are asking now and what do you think they’ll be researching in, say, 5 years?

Mr. Kuhn: Graduate programs with an established program in software engineering will be good choices for work in software assurance. In addition to how to integrate formal methods with combinatorial testing, important questions include: how to order or prioritize tests to find faults more quickly, how to identify the particular combination(s) that caused a failure, how to extend these methods to very large problems, and above all, determining the effectiveness of these methods in the real world. It would be nice if these problems were solved quickly, but I’m sure they will still be important in 5 years. Combinatorial testing has grown very rapidly in the past 10 years, from around 4 or 5 papers a year in the 1990s though 2002, to more than 50 per year recently, so it seems to be attracting a good deal of attention from researchers.

LogiGear: Lastly, who do you consider to be some of the leaders in this field and what are they doing?

Mr. Kuhn: In addition to Jeff Lei (U. Texas, Arlington) and Jim Lawrence (George Mason U.), who have been part of our team since the beginning, Charles Colbourn, at ASU, is the expert on t-way covering arrays and algorithms. We’re working with Renee Bryce (Utah State) and Sreedevi Sampath (U. of Maryland Baltimore County), who are looking at test prioritization. Myra Cohen (U. Nebraska-Lincoln) has done a good deal of work on both the theory and application of these methods. There are many other leaders including Sid Dalal, George Sherwood (from former AT&T, Bellcore, SAIC) , and Alan Hartman (from IBM)

A lot of people are now working on getting these methods into practice. Among those we are working with are Jim Higdon at Eglin Air Force Base and Justin Hunter of Hexawise. Turning research ideas into something that works in the real world can be more challenging than the research in some ways.

LogiGear: Thank you, Mr. Kuhn

Centralize? Decentralize? Concentralize!

As I wrote in various articles, organization is one of the 3 key requisites for successful automated testing, the other two being test design and automation architecture.

A question coming up quite commonly in larger corporations and organizations is whether to centralize the test automation, either at division or corporate level, or whether to decentralize: distribute over the various groups and projects.

The question centralize/decentralize is probably one of the most common in organizational science, and numerous publications can be found on the topic. In my own experience as management I found that there is not one “good” solution. Both centralizing and decentralizing have advantages and disadvantages. The arguments for centralization are leveraging of knowledge, tools and environments. A strong central group can acquire the tools and equipment, and allocate them efficiently to various projects. They can also specialize in the various technical and non-technical aspect of testing and test automation, building knowledge, like testing and automation techniques, and create common libraries and in-house tools for various needs.

A decentralized organization of testing brings it closer to the project and the subject matter. The project manager will generally have a better grip on the activities, quality, timelines and costs involved. And it will generally be easier to leverage subject matter expertise when the testing are organized locally, even in some cases have the subject matter experts involved in test creation. However, it is more challenging to reach the same professional level, in particular for the automation part, and automation efforts can go lost after a project has finished, either by ineffective automation strategies, or simply because nobody is left to take care of the developed test-ware.

For automated testing however, I feel there is a good hybrid solution, which I like to nick-name “concentralize”. It would typically look something like this:

concentralize

In this solution a core group concerns with activities like:

  • know how, methods and techniques
  • technology, tools (both acquired and in-house)
  • common libraries, actions and interface definitions
  • environments, like server racks, virtual machines, etc
  • retaining and preserving developed testware

The core group supports testing teams that are local, meaning inside or close to projects and departments. Within the test teams there is distinct:

  • test development, that includes:
    • high level test design, and test development planning
    • design development of test modules
  • project level automation, as much as possible distinct from the test development. It will typically entail activities like project level actions, interface handling, and scripting (see the Action Based Testing method for more details how this can work)

The testing teams in turn work with, and answer to, the various stakeholders, for input and assessment of developed test modules and their execution results.

In addition to the formal organization I recommend to have at least some form of informal “networking”. This could take the form of “special interest groups” for fields like test design and automation techniques. These groups periodically gather, on a voluntary basis, to discuss the profession. Typically at each meeting one of the members will present a topic, like a project experience, a research item, etc. It can also make sense to invite outside speakers. Each group has a chairperson and a secretary. The organization should stimulate with facilities, food, cost coverage of external speakers etc. This is a very cost-friendly way to build the testing and test automation profession, thus benefiting all projects involved.

Automation Adoption

One of the basic challenges with test automation is adoption. I can’t tell you how many times I’ve cataloged licenses for a company and found out they already have many different automation software packages, none of which is being used. Traditionally I’ve been told that is because the tools don’t work and that the teams had a hard time implementing the automation.

Let’s take one thing off the table, there is NO tool problem. Most automation tools do the same thing, catalog objects, record and playback and advanced scripting abilities, so, please, if you trying to organize an automation project, don’t make it about a tool (I think I’m famous for saying this, since tools are only means to an end, they aren’t panaceas, in fact that’s the last thing they are).

Second, the problem is people, not tools. Most testers do not have the skill sets to program in one these monster packages like QTP. They are put in front of a complex IDE and expected to scale that tool with relative ease. Problem, most testers don’t know how to program or haven’t in so long that there skills are completely inadaquate to the task. My experience is that to get a line engineer to learn how to program effectively takes 18 months, I’m sure I have some aghast looks here, but to be “proficient” is a complete different level of understanding then just “‘grinding” through a software package and hoping your developing scalable automation.

Some of the new tools are trying to divide the effort between automation engineers and line test engineers. QTP has designed a keywork abstraction for their automation tooling. It’s quite complex, but I feel that they are going in the right direction. If you can pair up your subject matter experts with QC Component test automation experts, then you might have a scalable solution, but too many don’t go down this path.

What do I suggest?

  • Select a test design process
  • Organize the effort around mini-pilots
  • Don’t buy tools if you don’t need to, lease them at first
  • Understand your teams skill sets – this metric sets adoption

Good luck