Design of Experiments – Methods and Case Studies
• Dan Rand• Winona State University• ASQ Fellow• 5-time chair of the La Crosse / Winona
Section of ASQ• (Long) Past member of the ASQ
Hiawatha Section
Design of Experiments – Methods and Case Studies
• Tonight’s agenda– The basics of DoE– Principles of really efficient experiments– Really important practices in effective experiments– Basic principles of analysis and execution in a
catapult experiment – Case studies – in a wide variety of applications– Optimization with more than one response
variable– If Baseball was invented using DoE
Design of Experiments - Definition
• implementation of the scientific method. -design the collection of information about a phenomenon or process, analyze information, learn about relationships of important variables. - enables prediction of response variables. - economy and efficiency of data collection minimize usage of resources.
Advantages of DoE
• Process Optimization and Problem Solving with Least Resources for Most Information.
• Allows Decision Making with Defined Risks.• Customer Requirements --> Process
Specifications by Characterizing Relationships
• Determine effects of variables, interactions, and a math model
• DOE Is a Prevention Tool for Huge Leverage Early in Design
No DoE Poor Execution Poor Analysis
Why Industrial Experiment Fail
Poor Planning
Poor Design
Steps to a Good Experiment
• 1. Define the objective of the experiment.
• 2. Choose the right people for the team.• 3. Identify prior knowledge, then
important factors and responses to be studied.
• 4. Determine the measurement system
Steps to a Good Experiment
• 5. Design the matrix and data collection responsibilities for the experiment.
• 6. Conduct the experiment.• 7. Analyze experiment results and draw
conclusions.• 8. Verify the findings.• 9. Report and implement the results
An experiment using a catapult
• We wish to characterize the control factors for a catapult
• We have determined three potential factors:
1. Ball type
2. Arm length
3. Release angle
One Factor-at-a-Time Method
• Hypothesis test - T-test to determine the effect of each factor separately.
• test each factor at 2 levels. Plan 4 trials each at high and low levels of 3 factors
• 8 trials for 3 factors = 24 trials. • levels of other 2 factors?• Combine factor settings in only 8
total trials.
Factors and settings
Factors low highA ball type slotted solidB Arm length 10 12C Release angle 45 75
Factor settings for 8 trials
A B C1 2 3 4 5 6 7-1 -1 1 -1 1 1 -11 -1 -1 -1 -1 1 1-1 1 -1 -1 1 -1 11 1 1 -1 -1 -1 -1-1 -1 1 1 -1 -1 11 -1 -1 1 1 -1 -1-1 1 -1 1 -1 1 -11 1 1 1 1 1 1
Randomization
• The most important principle in designing experiments is to randomize selection of experimental units and order of trials.
• This averages out the effect of unknown differences in the population, and the effect of environmental variables that change over time, outside of our control.
From Design Expert:
Randomized by Design Expert
Factor 1 Factor 2 Factor 3
Std order Run order A:ball type B:Arm length C:Release angle
8 1 1 12 75
4 2 1 12 45
5 3 -1 10 75
7 4 -1 12 75
3 5 -1 12 45
6 6 1 10 75
1 7 -1 10 45
2 8 1 10 45
Experiment trials & resultsA B C
1 2 3 4 5 6 7 Result
-1 -1 1 -1 1 1 -1 76.5
1 -1 -1 -1 -1 1 1 78.5
-1 1 -1 -1 1 -1 1 87.75
1 1 1 -1 -1 -1 -1 89
-1 -1 1 1 -1 -1 1 81
1 -1 -1 1 1 -1 -1 77.5
-1 1 -1 1 -1 1 -1 79
1 1 1 1 1 1 1 77.5
A B AB C AC BC
contrast -1.75 19.75 1.25 -16.8 -8.25 -23.8 2.75
effect -0.4375 4.938 0.313 -4.19 -2.06 -5.94 0.688
Graph of significant effects
Detecting interactions between factors
• Two factors show an interaction in their effect on a response variable when the effect of one factor on the response depends on the level of another factor.
Interaction graph from Design Expert
Predicted distance based on calculated effects
• Distance = 80.84 + 4.94 * X2_arm_length – 4.19* X3_Release_angle – 2.06* X1*X3 - 5.94*X2*X3
• X2 = -1 at arm length of 10, = 1 at arm length of 12
• X3 = -1 at release angle of 45, = 1 at release angle of 75
Poorly executed experiments
• If we are sloppy with control of factor levels or lazy with randomization, special causes invade the experiment and the error term can get unacceptably large. As a result, significant effects of factors don’t appear to be so significant.
The Best and the Worst
• Knot Random Team and the String Quartet Team. Each team designed a 16-trial, highly efficient experiment with two levels for each factor to characterize their catapult’s capability and control factors.
Knot Random team results
Mean square error = 1268
Demonstration of capability for 6 shots with specifications 84 ± 4 inches , Cpk = .34
String quartet result
Mean square error = 362Demonstration of capability for 6 shots with specifications 72 ± 4 inches , Cpk=2.02
String Quartet Best Practices
• Randomized trials done in prescribed order
• Factor settings checked on all trials• Agreed to a specific process for releasing
the catapult arm• Landing point of the ball made a mark that
could be measured to ¼ inch• Catapult controls that were not varied as
factors were measured frequently
Knot Random – Knot best practices
• Trials done in convenient order to hurry through apparatus changes
• Factor settings left to wrong level from previous trial in at least one instance
• Each operator did his/her best to release the catapult arm in a repeatable fashion
• Inspector made a visual estimate of where ball had landed, measured to nearest ½ inch
• Catapult controls that were not varied as factors were ignored after initial process set-up
Multivariable testing (MVT) as DoE
• “Shelf Smarts,” Forbes, 5/12/03• DoE didn’t quite save Circuit City• 15 factors culled from 3000 employee
suggestions• Tested in 16 trials, 16 stores• Measured response = store revenue• Implemented changes led to 3% sales
rise
Census Bureau Experiment
• “Why do they send me a card telling me they’re going to send me a census form???”
• Dillman, D.A., Clark, J.R., Sinclair, M.D. (1995) “How pre-notice letters, stamped return envelopes and reminder postcards affect mail-back response rates for census questionnaires,” Survey Methodology, 21, 159-165
1992 Census Implementation Test
• Factors:– Pre-notice letter – yes/ no– SASE with census form – yes / no– Reminder postcard a few days after
census form – yes / no– Response = completed, mailed survey
response rate
Experiment results –net savings in the millions
letter envelope postcard Response rate
- - - 50%- - + 58%- + - 52.6%- + + 59.5%+ - - 56.4%+ - + 62.7%+ + - 59.8%+ + + 64.3%
Surface Mount Technology (SMT) experiment - problem solving in a
manufacturing environment
• 2 types of defects, probably related– Solder balls– Solder-on-gold
• Statistician invited in for a “quick fix” experiment
• High volume memory card product• Courtesy of Lally Marwah, Toronto, Canada
Problem in screening / reflow operations
Prep card
Solder paste screening
Component placement
Solder paste reflow
Clean card Inspect (T2)
Inspect (T1)
insert
8 potential process factors
• Clean stencil frequency: 1/1, 1/10• Panel washed: no, yes• Misregistration: 0, 10 ml• Paste height: 9ml, 12 ml• Time between screen/ reflow: .5, 4 hr• Reflow card spacing: 18 in, 36 in• Reflow pre-heat: cold, hot• Oven: A, B
Experiment design conditions
• Resources only permit 16 trials• Get efficiency from 2-level factors• Measure both types of defects • Introduce T1 inspection station for
counting defects• Same inspectors• Same quantity of cards per trial
Can we measure effects of 8 factors in 16 trials? Yes1 -1 -1 -1 -1 1 1 1 486
-1 1 -1 -1 1 -1 1 1 2211 1 -1 -1 1 1 -1 -1 314
-1 -1 1 -1 1 1 1 -1 6041 -1 1 -1 1 -1 -1 1 549
-1 1 1 -1 -1 1 -1 1 3541 1 1 -1 -1 -1 1 -1 502
-1 -1 -1 1 1 1 -1 1 2221 -1 -1 1 1 -1 1 -1 360
-1 1 -1 1 -1 1 1 -1 6491 1 -1 1 -1 -1 -1 1 418
-1 -1 1 1 -1 -1 1 1 13211 -1 1 1 -1 1 -1 -1 993
-1 1 1 1 1 -1 -1 -1 8931 1 1 1 1 1 1 1 840 response
A B C D E F G H factor5.5 -62 404 314 -109 5.5 136 -7.3
7 more columns contain all interactions
AB AC BC AD BD CD AE
CG DF DE CF CE BE DG
DH BG AG BH AH AF BF
EF EH FH EG FG GH CH
-16 -78 -157 124 38 196 25
• Each column contains confounded interactions
Normal plot for factor effects on solder ball defects
Normal plot- 15 mean effects
-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
-200 -100 0 100 200 300 400 500
Mean Effects
Z V
aria
te
CD
CD
Which confounded interaction is significant?
• AF, BE, CD, or GH ?• The main effects C and D are
significant, so engineering judgement tells us CD is the true significant interaction.
• C is misregistration• D is paste height
Conclusions from experiment
• Increased paste height (D+) acts together with misregistration to increase the area of paste outside of the pad, leading to solder balls of dislodged extra paste.
• Solder ball occurrence can be reduced by minimizing the surface area and mass of paste outside the pad.
Implemented solutions
• Reduce variability and increase accuracy in registration.
• Lowered solder ball rate by 77%• More complete solution:• Shrink paste stencil opening - pad
accommodates variability in registration.
The Power of Efficient Experiments
• More information from less resources• Thought process of experiment design
brings out: –potential factors – relevant measurements–attention to variability –discipline to experiment trials
Optimization – Back to the Catapult
• Optimize two responses for catapult• Hit a target distance• Minimize variability• Suppose the 8 trials in the catapult
experiment were each run with 3 replicates, and we used means and standard deviations of the 3
8 catapult runs with response = standard deviation (sdev)
A B C sdev
-1 -1 -1 1.6
1 -1 -1 2.5
-1 1 -1 1.5
1 1 -1 2.6
-1 -1 1 1.4
1 -1 1 2.4
-1 1 1 1.4
1 1 1 2.4
Slotted balls have less variability
Desirability - Combining Two Responses
70 75 80 85 90 95 1000
0.2
0.4
0.6
0.8
1
Desirability for distance
d
0 0.5 1 1.5 2 2.5 3 3.5 4 4.50
0.2
0.4
0.6
0.8
1
Desirability for std dev
std dev
Maximum Desirability
• Modeled response equation allows hitting the target distance of 84, d=1
• Best possible standard deviation according to model is 1.475
• d (for std dev) = (3-1.475)/(3-1) = .7625• D = SQRT(1*.7625) = .873
How about a little baseball?
• Questions?• Thank you• E-mail me at [email protected]• Find my slides at
http://course1.winona.edu/drand/web/