- Salice Thomas
- Follow Me
- June 29, 2024
-
Overview
Taguchi methods are statistical methods developed by Genichi Taguchi to improve the quality of manufactured goods. This methodology applied to manufacturing, product engineering, biotechnology and marketing. The goal is to obtain reproducible results and robust products, incurring the least possible expenses in material, cost and time. Here is where Taguchi Methods makes a difference, with the introduction of Orthogonal Arrays (OA) into the realm of experimental design The objective of this article is to simplify the techniques and present them in such a way that it is possible for software practitioners to apply them in their day-to-day work. We used the term Software Product Engineering to imply that the techniques are applicable in many points of a typical software development lifecycle. Following are the areas where Taguchi method can be used effectively.
- Configuration Testing
- User Interface Testing
- Performance Optimization
- Regression Testing
Definitions
Factors
A factor is a variable under study; an input that can be controlled. A factor can be either quantitative (measured in a continuous scale; example: memory utilization) or qualitative (measured in an integral scale example: radio button).
Levels
A level is a value that a factor can assume when used in an experiment. For example, memory utilization should be less than 1MB.
Taguchi defines a standard set of Orthogonal Arrays. Table 1 simplifies the choice of OA. Selecting an OA reduces to finding a row that accommodates the given number of factors and levels. Note that the final choice of array selection will be driven by engineering concerns like cost Vs coverage.
Using OAs is considered to be more superior than the traditional full factorial design method because
- It results in a smaller number of experiments.
- It is more efficient in handling a large number of input variables
- It helps determine the contribution of each input that influences quality of the product
- It allows easy interpretation of experiments
Applying Method
Here is the simplified procedure, tailored to the needs of software product engineering.
Step 1: Brainstorming
This is the first and most important step in experimental design as it promotes group participation, encourages creative thinking and generates many ideas in a short period of time. Following is expected out of the brainstorming session.
1. Define the objective: Experimental design can be conducted to meet different objectives like optimizing the output, studying the effect of variables, or reducing the number of test cases. Choose one that best suits the problem in hand.
2. Create a problem statement: Convert the engineering situation into an experimental design problem. This requires an intimate familiarity with the system under test, and some exposure to statistical design of experiments.
3. Identify the factors.
4. Identify the levels.
Best practice is to identify factors in such a way that the number of levels never exceeds 4 or 5 to keep the experiments simple. We can consider splitting an input into two factors or ignoring a few levels which are less significant where the number of levels is too large. Also the more prominent or probable level one come first.
Step 2: Designing the experiments
The experiment design involves the following:
Select an appropriate Orthogonal Array: select the appropriate orthogonal array from the standard set shown in the table 1.
Example: Consider the hypothetical case where we have three factors with two levels each. ThenL4 is the closest suitable orthogonal array since it accommodates our factor level combination. Table 2 gives an L4 orthogonal array.
Assign factors: Once we have an Orthogonal Array, the factors are assigned to the columns of the array, and the integers in the array are translated to the different levels the factors can take. Each row corresponds to a particular experiment. The unassigned columns can be deleted from the array.
Example: In our hypothetical case, we see that the L-4 experiment consists of four rows (experiments) and three columns (factors). That means we need 4 experiments with each row giving the level settings to be adopted for that experiment. In the first run, the three factors are set at level 1. In the second run, the first parameter is set at level 1 and the remaining two variables are set to level 2, and so on.
Step 3: Running the experiments
This corresponds to actually executing the tests as designed by the previous steps.
Testing Scenarios
Following are the areas where Taguchi method can be used effectively in software product engineering, but the list is by no means exhaustive.
Configuration Testing
Modern software products are expected to work in a variety of hardware platforms, a multitude of operating systems, and different versions of supporting libraries. Guaranteeing functionality in the different configurations, and the combinations there of, is a constant challenge faced by system testing teams. The Taguchi’s approach to Design of Experiments emerges as a logical answer to this problem. Not only does it reduces the number of test cases dramatically, but also it provides a reasonable guarantee regarding the combinations left out, owing to its strong statistical foundation. The problem statement in such a situation shall be formulated so that the main features, operating systems, hardware platforms, etc form the factors.
User Interface Testing
The problem is, even for the simplest of GUIs, the set of all possible inputs (the input domain) is too large to test exhaustively. Hence a tester needs to generate an effective set of test cases as a means of verifying the correct operation of the software. By designing the test cases (experiments) using the Taguchi approach, the input domain can be systematically covered as efficiently as possible. The factors in such a case will be the fields in the GUI, and possible user actions. Possible values in each field will need to be partitioned to different levels. For example a test filed in the data entry form can have three representative levels viz. null, valid and invalid.
Performance Tuning
In the product engineering, the more complex the system, the more will be the parameters to change, and the less predictable will be their influence on system performance. For example, a networking solution which has many tunable elements like the elementary packet size, delay between packets, size of send and receive buffers, period of sending cycle, choice of network card etc. It is difficult to arrive at an optimal value-set for all these factors, without a proven statistical technique like the Taguchi method of experimental design. In the case of the networking solution, the tunable elements will form the factors, and the values under consideration are the levels.
Regression Testing
Having a regression test suite is a must for the development of any non-trivial software product, which typically involves many build-test-fix cycles. Such a regression test suite is supposed to meet two objectives: They must be
(i) simple enough to be executed with minimal effort and
(ii) thorough enough to assume on successful completion that the product is stable.
Taguchi’s methods come to the rescue in such a case. Identifying the factors and levels in this case will be a restatement of the initial user needs (core functionality description) of the system.
Case Study – Configuration testing
Requirement overview
Consider an application to format a flash memory card. Application support two types of formatting say full format and quick format. The user can specify either a shipping format or a reference file with which to perform the formatting. The product is meant to run in different versions of Windows, upon different flash memory adapter hardware, and support cards from multiple vendors.
Problem Statement
The problem is to identify the minimal set of experiments to perform so as to statistically test all possible combinations of a formatting operation given below.
Following assumptions are made to simplify the problem
Only two card capacities (4 and 8 GB) are chosen, although other capacities also are supported.
An interrupt (user pressing cancel, or card being removed) can occur at any stage of the formatting operation, but such a distinction is not made.
Test case generation
The Orthogonal arrays suited for 10 factors with two levels are L-12, L-16 or L-32. We choose L-16 which is as given in Table 3. The columns A to O of the table represent the factors and the rows 1 to 16 represents the 16 experiments. The test cases designed using L-16 Orthogonal Array is given in Table 4.
Outcome
The number of test cases is reduced to 16, from the original, full factorial solution of 1024 (210) test cases.
Conclusion
The use of OAs offer a convenient middle path between cost and coverage requirements. As the case studies amply illustrates, the application of these methods yielded the following immediate benefits to practitioners:
- Reduced the number of test cases by many orders of magnitude.
- Cut down the time for test case design and test execution.
- Improved the confidence on product quality owing to the statistical basis of experiments.
- Brought more clarity into requirements, achieved by formulating a problem statement for experimentation.
- Forced the team to look at combinations they would have otherwise overlooked.
Standard Orthogonal Arrays
L4 Orthogonal Array
L8 Orthogonal Array
L9 Orthogonal Array
L12 Orthogonal Array
L16 Orthogonal Array
L18 Orthogonal Array
L27 Orthogonal Array
L36 Orthogonal Array
