Category Archives: Interview

Spotlight Interview with Mark Levison

Mark Levison has over twenty years experience in the IT industry, working as a developer, manager, technical lead, architect, and consultant. He  discovered Agile in 2001 and is now a Certified Scrum Trainer and Agile Coach with Agile Pain Relief Consulting.

Levison has introduced Scrum, Lean and other Agile methods to a number of organizations and coaches from executive level to the individual developer and tester. Levison is also an Agile editor at InfoQ and has written dozens of articles on Agile topics and publishes a blog – Notes from a Tool User. Mark’s training benefits from his study and writing on the neuroscience of learning: Learning Best Approaches for Your Brain. To contact Mark Levison, please email him at mark@agilepainrelief.com.

Spotlight Interview with Jonathan Rasmusson

Author of The Agile Warrior, Rasmusson answers questions from LogiGear’s testing staff about test automation in agile projects.

An Interview with Computer Scientist – D. Richard Kuhn about applications development of combinatorial research.

D. Richard Kuhn - Computer Scientist, National Institute of Standards & Technology

LogiGear: How did you become interested in developing applications for combinatorial research? What led you to it personally, and what did you find fascinating about it?

Mr. Kuhn: About 12 years ago Dolores Wallace and I were investigating causes of software failures in medical devices. About that time, Raghu Kacker in our math division introduced me to some work on the design of experiments and testing, which had been done by a colleague of his at Bell Labs. The idea behind these methods was that some failures only occur as a result of an interaction between components. For example, a system may correctly produce error messages when space is exhausted or when input rate exceeds some limit, but crashes only when these two conditions are both true at the same time. Pairwise testing has been used for a long time to catch this sort of problem.

I wondered if we could determine what proportion of the medical device failures were caused by interactions, and just how complex the interactions would be – in other words, how many failures were triggered by just one parameter value, and how many only happened when 2, 3, 4, etc. conditions were simultaneously true. We were surprised to find that no one had looked at this question empirically before. It turned out that all of the failures in the FDA reports appeared to involve four or fewer parameters. This was a very limited sample in one application domain, so I started looking at others and found a similar distribution. So far we have not seen a failure involving more than 6 parameters. This does not mean there aren’t any, but the evidence so far suggests that a small number of parameter values are involved in failures. In general, most problems are caused by one or two input values, fewer by an interaction among three, and still fewer at each step as we increase interactions above three.

What’s interesting about this research is that it suggests we can significantly improve the efficiency and the effectiveness of software testing – if we can test all 4-way to 6-way interactions, we are likely to catch almost all errors, although there are a lot of caveats to that statement and we don’t want to oversell this. In particular, selection of test values is critical unless a variable has a small set of discrete values, but this problem exists for all test methods. With the Automated Combinatorial Testing for Software (ACTS) project and tool, we hope to make these methods practical for real-world testing.

LogiGear: What has been your own personal contribution to this research and development? What sort of problems are you attracted to and how have you been able to solve them?

Mr. Kuhn: The series of papers that I did on the number of parameters involved in interaction failures has helped establish an empirical basis for the work and seems to have encouraged research on combinatorial testing by others. I also developed a parallel algorithm for generating combinatorial tests, although, frankly, there hasn’t been much need for it since Jeff Lei’s IPOG algorithm in the ACTS tool is so fast. More recently I’ve worked out a way to apply combinatorial methods to testing event sequences – instead of n! Tests for n events, the number of tests is proportional to log n. This method was motivated by interoperability testing, and it has already been used successfully.

I’ve always been fascinated by large collections of data – whether there might be some interesting or useful properties hiding in a mass of numbers. This is one area where that interest seems to have paid off. Another is some work I did showing that there’s a hierarchy of fault classes for boolean expressions.

LogiGear: What do you think is the current state of this development? You’re working on developing an open-source testing tool. How practical is it now? What successes have you experienced with it and what does it need? Also, how would you like its development to proceed in the future -would you like to see it take off and be developed as a sort of ‘Linux OS’ of combinatorial testing? What would you tell those who seek to get involved in this work?

Mr. Kuhn: The ACTS tool development is being led by Jeff Lei at U. of Texas Arlington, who has been an integral part of this project since the beginning. It’s very robust and has been acquired by several hundred organizations, and so far the feedback is very positive. It appears to be easier to use and more efficient than other tools out there, but we want to integrate lessons learned and make it better for real-world projects. In particular, the constraint handling feature is being strengthened. This is what you need, for example, to prevent the generation of tests that specify Internet Explorer on a Linux system. It already has this ability, but we are learning new needs from users all the time.

I like your suggestion that ACTS could be like Linux, in that Linux has become both a freely available tool and the basis for successful commercial products and services. It would be great if ACTS followed a similar path. In fact we are already seeing commercial extensions of it.

LogiGear: And finally, where do you see your own contributions and questions heading in the future? Where do you see this group – this “movement” if you will – headed?

Mr. Kuhn: Two things I really want to work on are integrating combinatorial testing with formal specifications and model based testing, and investigating the combinatorial coverage of tests developed with other methods. Both of these efforts should help to make this kind of testing more practical and cost effective. We have a good start on both, but we need better automation and tool support. The evidence so far suggests that we can make testing more efficient, but we need real-world data to prove it. I also want to understand the relationship between test value selection and combinatorial testing – are combinatorial methods more or less sensitive than other test methods to the choice of input values? But our primary goal is to collect and analyze data on real-world projects.

LogiGear: Thank you, Mr. Kuhn.

Professor Janzen provides some insight to LogiGear Magazine on how to become a software tester

David S. Janzen – Associate Professor of Computer Science Department California Polytechnic State University, San Luis Obispo – homepage

LogiGear: How did you get into software testing and what do you find interesting about it?

Professor Janzen: The thing I enjoy most about computing is creating something that helps people. Since my first real job building telecommunications fraud detection systems, requirements and design were the most fun. Then when I heard about this thing called test-driven development, and something just clicked. Using unit test specification to do design made sense to me — plus, you get this great side effect of all these automated tests to make refactoring and maintenance easier. I guess I got pulled into software testing by way of software design.

LogiGear: What kind of work are you doing and how did you pick those specific testing topics?

Professor Janzen: My PhD research focused on how test-driven development (TDD) affects the internal, or design quality of software. I did a bunch of experiments with students and software professionals that provided some evidence about the benefits of TDD. The experiments are pretty straightforward:

Get two groups of essentially equivalent programmers, ask them to complete the same tasks, but have one group use the approach you are studying while the other uses the “traditional” approach. Then collect metrics and surveys to see how the two approaches varied.

Having become convinced that TDD is a great way to design and build software, my recent efforts have moved toward incorporating TDD into computing education.

I think TDD is taking the same path that objects took in the early ’90s. Folks in the industry started adopting objects so the broader academic community took notice. However, they mostly considered object-oriented programming to be an advanced concept, so it started to appear in graduate and upper-level courses. As educators became more comfortable with the approach, objects eventually made it down to first-year courses. I think the same is happening with TDD, so I am building tools and doing experiments to demonstrate that TDD can be taught to beginning programmers with great success. I call the approach Test-Driven Learning (TDL).

My current project is a web-based development environment for beginning programmers that will incorporate TDL-inspired labs. It will soon be available at http://web-ide.org.

LogiGear: How can a college student prepare to go into software testing and become really good at it and what should he or she look for in teachers, courses, and methods?

Professor Janzen: Software testing is such a great field to study. My students who are good at software testing often get the best job offers. Most undergraduate computer science and software engineering programs don’t have separate courses in software testing. The best route is to do well in your core programming courses, and then take as many software engineering courses as you can. Many software engineering courses involve team projects. Volunteer to be a software tester or quality assurance manager. Or, take the role of system integrator or build manager.

These roles will give you exposure to many of the automation tools that software testers use, and will help you start thinking about how to break software and not just how to build it.

LogiGear: What sort of graduate programs should college graduates consider? Also, in your opinion, what are some of the more interesting research questions people are asking now and what do you think they’ll be researching in 5 years?

Professor Janzen: There are two routes you might consider in graduate school. If you want to be involved in software development in companies and lead software teams, look at masters in software engineering programs. Many of these programs cater to software professionals who are already working in industry, by offering courses in the evenings or on weekends. If you are interested in more cutting-edge research, such as building new software testing tools, or developing new software testing methods, consider going to a traditional Computer Science PhD program. There are lots of smart researchers working on really interesting testing topics.

In five years? Well a lot of work seems to be focused on automatically generating automated tests, and also on tools for working with models. Look for some of the top software testing conferences and try to attend. Read software testing trade journals — there are plenty online, and many are still published in magazine format. Try to identify and follow interesting and cutting edge active researchers – and maybe if you’re lucky you’ll get a chance to meet these pioneers in person, which can be an exciting and thought provoking experience.

LogiGear: Lastly, who do you consider to be some of the leaders in this field and what are they doing in the field of software testing and development?

Professor Janzen: I am interested in some of the work being completed by Tao Xie and Laurie Williams at North Carolina State University. Tao is doing a lot of work with automated software testing and data mining. Laurie has working on static analysis tools, reliability, security, and mutation testing.

LogiGear: Thank you, Professor Janzen.

Hung Q. Nguyen Discusses Software Testing

The following is a transcript of a May 7, 2008 interview with Hung Q. Nguyen, founder and CEO of LogiGear Corporation and coauthor of the best selling textbook Testing Computer Software.

Interviewer: When it comes to software testing, what concerns or issues are you hearing from software developers?

Hung Q. Nguyen: The most pressing concern that I hear from companies that develop software is that they are not testing fast enough. With advancements in technology and development tools the rates for building software have become very fast. Development is outpacing software testing which is driving a huge demand for test automation. However, test automation, done incorrectly can be more of a hindrance.

Building Vietnam’s Leading Center For Software Testing

An Interview With Hung Nguyen, Founder and CEO of LogiGear Corporation

This interview was published by Saigon Entrepreneur Weekly (Issue 186, February 2, 2007), the most respected and widely read business magazine in Vietnam.

Hung Nguyen is a well known author of best selling books about software testing in the United States and is a widely recognized software testing industry thought leader. He has helped many major multinational companies with their software testing efforts. While Hung may be well known in the software testing industry, the industry sector is virtually unknown to many Vietnamese.