Category Archives: Test Methods & Metrics

Agile Test Development with Action Based Testing – by Hans Buwalda

Agile methods are becoming more and more popular and successful for developing IT systems. Typical properties of an agile method, like Extreme Programming, are continuous user involvement and an emphasis on the testing role (‘Users’ may be the actual users of the system you are creating, customers, or business analysts who provide the requirements on behalf of the end-users). Testing is positioned in agile system development as a driver for system development, with tests often created before the functionality they are verifying.

In LogiGear we promote and support Action Based Testing (ABT), a very successful method for creating maintainable automated tests. In ABT the focus is on development of products called “test modules”, an activity that can be independent from system development. It has no specific preference on when in the life cycle those test modules are developed. That makes it easy to integrate the methodology into an agile life cycle. In this article I want to give some perspective on how this can best be achieved.

In my view there are the following ways to enter ABT into an agile project:

  1. Closely couple test development to system development
  2. Keep test development as a separate life cycle and use an agile approach to build the tests
  3. Hybrid approach: business-level tests are developed early; detailed, typically UI-specific tests are created in parallel with the development life cycle

Allow me to expand on each of these approaches:

  1. Closely coupling test development to system development would mean sticking to what most agile life cycles prescribe. The tests, in the ABT test modules, are created together with users, whenever the life cycle needs it. The advantage is that this is highly recognizable as an agile approach, and it allows for better interaction between users and developers, with tests serving as a means of communication.
  2. The alternative is to have a separate life cycle for developing the test modules. In my experience, this is the most common approach. This might be a good choice if there are experienced testers available who can work with the users to develop good and representative tests. Those tests are then input for the developers, who can still interact with the users, but have a finished product to work with: the test modules. This alternative is probably the more manageable one, since test development can have its own life cycle, and can be dealt with in an early stage when the users have room in their schedule. Another advantage here is that even if your software development cycle is following a traditional ‘waterfall’ approach, your test and automation development can still be ‘agile’.
  3. The hybrid approach is in my view the most attractive proposition. In the test development life cycle for business level tests, the users and test developers can concentrate on the business process and business specific issues, like how to calculate a monthly premium for a mortgage. Then when the developers are ready to define the user interaction with the system the users can create tests like “if I click on a customer, I see a list of his/her account balances”.

No matter which alternative is used, I would make the following recommendations:

  • Plan well, and make every effort to create good efficient tests, with a high level of re-use and automation
  • Make sure participants know what is expected of them, and have the experience and knowledge to deliver their part

Agile methods are a relatively new phenomenon in the software world, with great promise and many early success stories. Carefully fitting testing into the agile life cycle will improve both system development and test development, and lead to successful test automation.

Do Testers Have to Write Code?

Elisabeth Hendrickson

Do testers have to write code?

For years, whenever someone asked me if I thought testers had to know how to write code, I’ve responded: “Of course not.”

The way I see it, test automation is inherently a programming activity. Anyone tasked with automating tests should know how to program.

But not all testers are doing test automation.

Automation Testing Tools in Full View

Bogdan Bereza-Jarocinski

A trainer, quality specialist and owner of VictO (2004), Bereza-Jarocinski co-founded Swedish SAST (1995), Polish SJSI (2003), and Polish SPIN (2006). He is also founder and secretary general of ADPQB (2008). For more information on his academy, visit

In  order to make the right choices among tools, you must be able to classify them. Otherwise, any choice would be at best haphazard. Without functioning classification, you would not be able to understand new tools fast, nor come up with ideas of using, or creating new tools.

Four Fundamental Requirements of Successful Testing in the Cloud – Part III

Internet-based per-use service models are turning things upside down in the software development industry, prompting rapid expansion in the development of some products and measurable reduction in others. (Gartner, August 2008) This global transition toward computing “in the Cloud” introduces a whole new level of challenge when it comes to software testing.

Test design focused on expediting functional test automation

David W. Johnson

Test organizations continue to undergo rapid transformation as demands grow for testing efficiencies. Functional test automation is often seen as a way to increase the overall efficiency of functional and system tests. How can a test organization stage itself for functional test automation before an investment in test automation has even been made? Further, how can you continue to harvest the returns from your test design paradigm once the test automation investment has been made? In this article we will discuss the factors in selecting a test design paradigm that expedites functional test automation. We will recommend a test design paradigm and illustrate how this could be applied to both commercial and open-source automation solutions. Finally, we will discuss how to leverage the appropriate test design paradigm once automation has been implemented in both an agile (adaptive) and waterfall (predictive) system development lifecycle (SDLC).

Key Principles of Test Design

Hans Buwalda, CTO, LogiGear Corporation

Test design is the single biggest contributor to success in software testing. Not only can good test design result in good coverage, it is also a major contributor to efficiency. The principle of test design should be “lean and mean.” The tests should be of a manageable size and at the same time complete and aggressive enough to find bugs before a system or system update is released.

What is Combinatorial Testing and Why Should Testers Care?

Developers of large data-intensive software often notice an interesting — though not surprising — phenomenon: When usage of an application jumps dramatically, components that have operated for months without trouble suddenly develop previously undetected errors. For example, the application may have been installed on a different OS-hardware-DBMS-networking platform, or newly added customers may have account records with an oddball combination of values that have not occurred before. Some of these rare combinations trigger failures that have escaped previous testing and extensive use. Such failures are known as interaction failures, because they are only exposed when two or more input values interact to cause the program to reach an incorrect result.

Allpairs, Pairwise, Combinatorial Analysis

Last week I went to StarWest as a presenter and as a track chair to introduce speakers. Being a track chair is wonderful because you get to interface more closely with other speakers. Anyway…one of the speakers I introduced was Jon Bach. Jon is a good public speaker, and I was pleasantly surprised that he was doing a talk on the allpairs testing technique (also known as pairwise or combinatorial analysis). I wish Jon dedicated a little more time to the specifics of the technique during his talk and was generally more aware of available tools and information for folks to investigate further, but I think he successfully raised the general awareness and interest in pariwise testing as an effective testing technique among the audience.

Pairwise testing is one approach to solving the potential explosion in the number of tests when dealing with multiple parameters whose variables are semi-coupled or have some dependency on variable states of other parameters. For example, in the font dialog of MS Word there are 11 checkboxes for various effects such as superscript, strikethrough, emboss, etc. Obviously these effects have impact on how the characters in a particular font are displayed and can be used in multiple combinations such as Strikethrough + Subscript + Emboss. The total number of combinations of effects is the Cartesian product of the variables for each parameter, or 211 or 2048 in this example. This doesn’t include different font types, styles, etc. which also interdependent. So, you can see how the number of combinations increases rapidly especially as additional dependent parameters are included in the matrix.

The good news is the industry has a lot of evidence to suggest that most software defects occur from simple interactions between the variables of 2 parameters. So, from a risk based perspective where it may not be feasible to test all possible combinations how do we choose the combinations out of all the possibilities? Two common approaches include orthogonal arrays and combinatorial analysis.

But, true orthogonal arrays require that the number of variables is the same for all parameters. (Rarely true in software.) It is possible to create “mixed orthogonal arrays” where some combinations of variables will be tested more than once. For example, if we have 5 parameters and one parameter has 5 variables and the remains 4 parameters only have 3 variables each, we can see from the orthogonal array selector (available on FreeQuality website) the size of the orthogonal array is L25 (which basically means the test case will require 25 tests which is still significantly less than the total number of combinations of 405).

The other approach is combinatorial analysis (often referred to as pairwise or allpairs testing) because the approach most commonly used is to use a mathematical formula to reduce the total number of combinations in such a way that each variable for each parameter is tested with each variable from the other parameters at least once. In the above example, the number of tests would be reduced to 16. (Note: some tools will give slightly different results.) However, some tools (such as Microsoft’s PICT) also allow for more complex analysis of variable combinations such as triplets and n-wise coverage.

One problem that is hopefully not overlooked by testers using these tools is that some combinations of variables are simply not possible. For example, in the Effects group of the Font dialog it is impossible to check the Superscript checkbox and the Subscript checkbox simultaneously. Therefore, the tester either has to manually modify the output, or use a tool that allows constraints. Again, this is another situation where Microsoft’s free tool PICT excels. PICT uses a simply basic-like language for conditional and unconditional constraining of combinations of variables. PICT also allows weighting variables, seeding, output randomization, and negative testing.

I didn’t want this to be a PICT sales job, but alas my bias has influenced this post. So, I will conclude by pointing the readers to the Pairwise Testing website. My colleague Jacek Czerwonka has pulled together great resources on the technique of combinatorial analysis including a list of free and commercially available tools, and white papers supporting the value and practicality of this testing technique.

Article from Testing Mentor by BJ Rollison, Test Architect, Microsoft Inc

Combinatorial Software Testing

“Combinatorial testing can detect hard-to-find software faults more efficiently than manual test case selection methods.”

Developers of large data-intensive software often notice an interesting—though not surprising—phenomenon: When usage of an application jumps dramatically, components that have operated for months without trouble suddenly develop previously undetected errors. For example, newly added customers may have account records with an oddball combination of values that have not been seen before. Some of these rare combinations trigger faults that have escaped previous testing and extensive use. Alternatively, the application may have been installed on a different OS-hardware-DBMS-networking platform. Combinatorial testing can help detect problems like this early in the testing life cycle. The key insight underlying t-way combinatorial testing is that not every parameter contributes to every fault and many faults are caused by interactions between a relatively small number of parameters.


Suppose we want to demonstrate that a new software application works correctly on PCs that use the Windows or Linux operating systems, Intel or AMD processors, and the IPv4 or IPv6 protocols. This is a total of 2 × 2 × 2 = 8 possibilities but, as Table 1 shows, only four tests are required to test every component interacting with every other component at least once. In this most basic combinatorial method, known as pairwise testing, at least one of the four tests covers all possible pairs (t = 2) of values among the three parameters.

Note that while the set of four test cases tests for all pairs of possible values—for example, OS = Linux and protocol = IPv4—several combinations of three specific values are not tested—for example, OS = Windows, CPU = Intel, and protocol = IPv6.
Even though pairwise testing is not exhaustive, it is useful because it can check for simple, potentially problematic interactions with relatively few tests. The reduction in test set size from eight to four shown in Table 1 is not that impressive, but consider a larger example: a manufacturing automation system that has 20 controls, each with 10 possible settings—a total of 1020 combinations, which is far more than a software tester would be able to test in a lifetime. Surprisingly, we can check all pairs of these values with only 180 tests if they are carefully constructed.

Figure 1 shows the results of a 10-project empirical study conducted recently by Justin Hunter that compared the effectiveness of pairwise testing with manual test case selection methods.


The projects were conducted at six companies and tested commercial applications in development; in each project, two small teams of testers were asked to test the same application at the same time using different methods. One group of testers selected tests manually; they relied on “business as usual” methods such as developing tests based on functional and technical requirements and potential use cases mapped out on whiteboards. The other group used a combinatorial testing tool to identify pairwise tests. Test execution productivity was significantly higher in all of the projects for the testers using combinatorial methods, with test execution productivity more than doubling on average and more than tripling in three projects. The groups using pairwise testing also achieved the same or higher quality in all 10 projects; all of the defects identified by the teams using manual test case selection methods were identified by the teams using combinatorial methods. In five projects, the combinatorial teams found additional defects that had not been identifed by the teams using manual methods.

These proof-of-concept projects successfully demonstrated to the teams involved that manual methods of test case selection were not nearly as effective as pairwise combinatorial methods for finding the largest number of defects in the least amount of time.


Other empirical investigations have concluded that from 50 to 97 percent of software faults could be identified by pairwise combinatorial testing. However, what about the remaining faults? How many failures could be triggered only by an unusual interaction involving more than two parameters?

In a 1999 study of faults arising from rare conditions, the National Institute of Standards and Technology reviewed 15 years of medical device recall data to determine what types of testing could detect the reported faults (D.R. Wallace and D.R. Kuhn, “Failure Modes in Medical Device Software: An Analysis of 15 Years of Recall Data,” Int’l J. Reliability, Quality, and Safety Eng., Dec. 2001, pp. 351-371). The study found one case in which an error involved a four-way interaction among parameter values: demand dose = administered, days elapsed = 31, pump time = unchanged, and battery status = charged.

Pairwise combinatorial testing is unlikely to detect faults like this because it only guarantees that all pairs of parameter values will be tested. A particular four-way combination of values is statistically unlikely to occur in a test set that only ensures two-way combination coverage; to ensure thorough testing of complex applications, it is necessary to generate test suites for four-way or higher-degree interactions.

Investigations of other applications found similar distributions of fault-triggering conditions. Many faults were caused by a single parameter, a smaller proportion resulted from an interaction between two parameter values, and progressively fewer were triggered by three-, four-, five,- and six-way interactions.

Figure 2 summarizes these results. Thus far, a fault triggered by a seven-way interaction has not appeared.

With the Web server application, for example, roughly 40 percent of the failures were caused by a single value, such as a fie name exceeding a certain length; another 30 percent were triggered by the interaction of two parameters; and a cumulative total of almost 90 percent were triggered by three or fewer parameters.
While not conclusive, these results suggest that combinatorial methods can achieve a high level of thoroughness in software testing.

The key ingredient for this kind of testing is a covering array, a mathematical object that covers all t-way combinations of parameter values at least once. For the pairwise testing example in Table 1, t = 2, and it is relatively easy to generate tests that cover all pairs of parameter values. Generating covering arrays for complex interactions is much harder, but new algorithms make it possible to generate covering arrays orders of magnitude faster than previous algorithms, making up to six-way covering arrays tractable for many applications.

Figure 3 shows a covering array for all three-way interactions of 10 binary parameters in only 13 tests. Note that any three columns, selected in any order, contain all eight possible values of three parameters: 000,001,010,011, 100,101,110,111. Three-way interaction testing detected roughly 90 percent of bugs in all four of the empirical studies in Figure 2, but exhaustive testing of all possible combinations in Figure 3 would require 210 = 1,024 tests.


What are the pragmatic implications of being able to achieve 100 percent three-way coverage in 13 test cases on real-world software testing projects? Assuming that there are 10 defects in this hypothetical application and that 9 are identified through the 13 tests indicated, testing these 13 cases would find 71 times more
defects per test case [(9/13)/(10/1,024)] than testing exhaustively and uncovering all 10.

While the most basic form of combinatorial testing— pairwise—is well established, and adoption by software testing practitioners continues to increase, industry usage of these methods remains patchy at best. However, the additional training required is well worth the effort. Teams seeking to maximize testing thoroughness given tight time or resource constraints, and which currently rely on manual test case selection methods, should consider pairwise testing. When more time is available or more thorough testing is required, t-way testing for t > 2 is better. Practitioners who require very high quality software will find that covering arrays for higher-strength combinations can detect many hard to-find faults, and variability among detection rates appears to decrease as t increases.
Sophisticated new combinatorial testing algorithms packaged in user-friendly tools are now available to enable thorough testing with a manageable number of test cases and at lower cost, and make it practical for testers to develop empirical results on applications of this promising test method.

Article By

D. Richard Kuhn, Computer Scientist, NIST
Raghu Kacker, Mathematical Statistician, NIST
Yu Lei, Associate Professor, University of Texas at Arlington
Justin Hunter, CEO, Hexawise

Download here

Dr. Cem Kaner shares his experiences on software testing and essential skills needed to be software tester.

Dr. Cem Kaner – Director, Center for Software Testing Education & Research, Florida Institute of Technology

PC World Vietnam: What did you think of VISTACON 2010?

Dr. Kaner: I am very impressed that the event was very professionally organized and happy to meet my old colleagues to share and exchange more about our area of expertise. I was also very excited being one of the speakers at the conference and presented on some topics which focused on software testing as well as sharing some of my experiences from my research and work in this field.

PC World Vietnam: What do you believe are the most important skills required for a tester?

Dr. Kaner: Testers should have a good knowledge of the profession that they are working in; have testing skills and certain technology basics. For example, when testing banking or finance software, you should know about their trading activities, how they liquidate, and have a thorough grasp of some of the testing tools, testing plans and test automation methods. The technology basics means you should grasp some steps of the software development process, programming language skills, a knowledge of algorithms and operating systems. Besides that, other essential factors that can help you become a successful tester are having a good observant mind to easily recognize bugs during the testing process, the passion to apply the highest standards to the project, and finally, that you are always interested in new technology and new products.

PC World Vietnam:
Could you please share with us your experience of undertaking testing projects in different countries?

Dr. Kaner: I have had the opportunity to work and collaborate on projects in the US and India in the field of software testing. From my experience, you should focus on the test cases created during the period you are creating code. Parallel to that, you need to improve the testing process to achieve the desired quality.

It is important to find ways to solve problems by preventing bugs instead of finding bugs. Testers should talk and share their experiences of finding bugs to improve their professional skills. Also, the wages paid to a tester in each country varies. For instance, in the US, the cost of hiring a tester is very high, so most companies outsource their projects to another country to save costs. According to Global Services Media, last year Ho Chi Minh City ranked 5th in the list top 50 emerging global cities in outsourcing software. That is a good sign for the development of the Vietnamese software industry.

PC World Vietnam: Vietnamese students often tend to become programmers instead of testers, what do you think about this?

Dr. Kaner:
Actually, the hobbies and passions of each person will always be different. I myself have been rotated between the two positions of testing team manager and software programmers’ team manager. Some people find the testing field interesting, but others discover their own skills are more in line with programming. Those who prefer to focus on solving a problem or specific solutions may be appropriate as a programmer. But if you decide you want to become a tester, you should do your best to make this happen by studying hard and doing serious research.

PC World Vietnam: Thank you, Dr. Kaner

(This interview is condensed and translated from the PC World Vietnam Magazine – October 2010 Issue)