Description
Covers experiment planning, execution, analysis, and reporting
This single-source resource guides readers in planning and conducting credible experiments for engineering, science, industrial processes, agriculture, and business. The text takes experimenters all the way through conducting a high-impact experiment, from initial conception, through execution of the experiment, to a defensible final report. It prepares the reader to anticipate the choices faced during each stage.
Filled with real-world examples from engineering science and industry, Planning and Executing Credible Experiments: A Guidebook for Engineering, Science, Industrial Processes, Agriculture, and Business offers chapters that challenge experimenters at each stage of planning and execution and emphasizes uncertainty analysis as a design tool in addition to its role for reporting results. Tested over decades at Stanford University and internationally, the text employs two powerful, free, open-source software tools: GOSSET to optimize experiment design, and R for statistical computing and graphics. A website accompanies the text, providing additional resources and software downloads.
- A comprehensive guide to experiment planning, execution, and analysis
- Leads from initial conception, through the experiment’s launch, to final report
- Prepares the reader to anticipate the choices faced throughout an experiment
- Hones the motivating question
- Employs principles and techniques from Design of Experiments (DoE)
- Selects experiment designs to obtain the most information from fewer experimental runs
- Offers chapters that propose questions that an experimenter will need to ask and answer during each stage of planning and execution
- Demonstrates how uncertainty analysis guides and strengthens each stage
- Includes examples from real-life industrial experiments
- Accompanied by a website hosting open-source software
Planning and Executing Credible Experiments is an excellent resource for graduates and senior undergraduates—as well as professionals—across a wide variety of engineering disciplines.
Table of Contents
About the Authors xxi
Preface xxiii
Acknowledgments xxvii
About the Companion Website xxix
1 Choosing Credibility 1
1.1 The Responsibility of an Experimentalist 2
1.2 Losses of Credibility 2
1.3 Recovering Credibility 3
1.4 Starting with a Sharp Axe 3
1.5 A Systems View of Experimental Work 4
1.6 In Defense of Being a Generalist 5
2 The Nature of Experimental Work 7
2.1 Tested Guide of Strategy and Tactics 7
2.2 What Can Be Measured and What Cannot? 8
2.3 Beware Measuring Without Understanding: Warnings from History 12
2.4 How Does Experimental Work Differ from Theory and Analysis? 13
2.5 Uncertainty 23
2.6 Uncertainty Analysis 23
3 An Overview of Experiment Planning 27
3.1 Steps in an Experimental Plan 27
3.2 Iteration and Refinement 28
3.3 Risk Assessment/Risk Abatement 28
3.4 Questions to Guide Planning of an Experiment 29
4 Identifying the Motivating Question 31
4.1 The Prime Need 31
4.2 An Anchor and a Sieve 33
4.3 Identifying the Motivating Question Clarifies Thinking 33
4.4 Three Levels of Questions 35
4.5 Strong Inference 36
4.6 Agree on the Form of an Acceptable Answer 36
4.7 Specify the Allowable Uncertainty 37
4.8 Final Closure 37
5 Choosing the Approach 39
5.1 Laying Groundwork 39
5.2 Experiment Classifications 40
5.3 Real or Simplified Conditions? 43
5.4 Single-Sample or Multiple-Sample? 43
5.5 Statistical or Parametric Experiment Design? 45
5.6 Supportive or Refutative? 47
5.7 The Bottom Line 47
6 Mapping for Safety, Operation, and Results 51
6.1 Construct Multiple Maps to Illustrate and Guide Experiment Plan 51
6.2 Mapping Prior Work and Proposed Work 51
6.3 Mapping the Operable Domain of an Apparatus 53
6.4 Mapping in Operator's Coordinates 57
6.5 Mapping the Response Surface 59
7 Refreshing Statistics 65
7.1 Reviving Key Terms to Quantify Uncertainty 65
7.2 The Data Distribution Most Commonly Encountered The Normal Distribution for Samples of Infinite Size 71
7.3 Account for Small Samples: The t-Distribution 72
7.4 Construct Simple Models by Computer to Explain the Data 73
7.5 Gain Confidence and Skill at Statistical Modeling Via the R Language 77
7.6 Report Uncertainty 80
7.7 Decrease Uncertainty (Improve Credibility) by Isolating Distinct Groups 81
7.8 Original Data, Summary, and R 82
8 Exploring Statistical Design of Experiments 87
8.1 Always Seeking Wiser Strategies 87
8.2 Evolving from Novice Experiment Design 87
8.3 Two-Level and Three-Level Factorial Experiment Plans 88
8.4 A Three-Level, Three-Factor Design 89
8.5 The Plackett–Burman 12-Run Screening Design 93
8.6 Details About Analysis of Statistically Designed Experiments 95
8.7 Retrospect of Statistical Design Examples 101
8.8 Philosophy of Statistical Design 101
8.9 Statistical Design for Conditions That Challenge Factorial Designs 102
8.10 A Highly Recommended Tool for Statistical Design of Experiments 103
8.11 More Tools for Statistical Design of Experiments 103
8.12 Conclusion 103
9 Selecting the Data Points 107
9.1 The Three Categories of Data 107
9.2 Populating the Operating Volume 109
9.3 Example from Velocimetry 109
9.4 Organize the Data 112
9.5 Strategies to Select Next Data Points 114
9.6 Demonstrate Gosset for Selecting Data 120
9.7 Use Gosset to Analyze Results 133
9.8 Other Options and Features of Gosset 133
9.9 Using Gosset to Find Local Extrema in a Function of Several Variables 134
9.10 Summary 137
10 Analyzing Measurement Uncertainty 143
10.1 Clarifying Uncertainty Analysis 143
10.2 Definitions 153
10.3 The Sources and Types of Errors 156
10.4 The Basic Mathematics 170
10.5 Automating the Uncertainty Analysis 178
10.6 Single-Sample Uncertainty Analysis 181
11 Using Uncertainty Analysis in Planning and Execution 197
11.1 Using Uncertainty Analysis in Planning 197
11.2 Perform Uncertainty Analysis on the DREs 202
11.3 Using Uncertainty Analysis in Selecting Instruments 208
11.4 Using Uncertainty Analysis in Debugging an Experiment 209
11.5 Reporting the Uncertainties in an Experiment 213
11.6 Multiple-Sample Uncertainty Analysis 214
11.7 Coordinate with Uncertainty Analysis Standards 216
11.8 Describing the Overall Uncertainty in a Single Measurement 220
11.9 Additional Statistical Tools and Elements 222
12 Debugging an Experiment, Shakedown, and Validation 231
12.1 Introduction 231
12.2 Classes of Error 231
12.3 Using Time-Series Analysis in Debugging 232
12.4 Examples 232
12.5 Process Unsteadiness 234
12.6 The Effect of Time-Constant Mismatching 235
12.7 Using Uncertainty Analysis in Debugging an Experiment 236
12.8 Debugging the Experiment via the Data Interpretation Program 239
12.9 Situational Uncertainty 241
13 Trimming Uncertainty 243
13.1 Focusing on the Goal 243
13.2 A Motivating Question for Industrial Production 243
13.3 Plackett–Burman 12-Run Results and Motivating Question #3 245
13.4 PB 12-Run Results and Motivating Question #1 247
13.5 Uncertainty Analysis of Dual Predictive Model and Motivating Question #2 252
13.6 The PB 12-Run Results and Individual Machine Models 256
13.7 Final Answers to All Motivating Questions for the PB Example Experiment 263
13.8 Conclusions 265
14 Documenting the Experiment: Report Writing 269
14.1 The Logbook 269
14.2 Report Writing 269
14.3 International Organization for Standardization, ISO 9000 and other Standards 282
14.4 Never Forget. Always Remember 282
Appendix A: Distributing Variation and Pooled Variance 283
A.1 Inescapable Distributions 283
A.2 Other Common Distributions 286
A.3 Pooled Variance (Advanced Topic) 286
Appendix B: Illustrative Tables for Statistical Design 289
B.1 Useful Tables for Statistical Design of Experiments 289
B.2 The Plackett–Burman (PB) Screening Designs 289
Appendix C: Hand Analysis of a Two-Level Factorial Design 293
C.1 The General Two-Level Factorial Design 293
C.2 Estimating the Significance of the Apparent Factor Effects 298
C.3 Hand Analysis of a Plackett–Burman (PB) 12-Run Design 299
C.4 Illustrative Practice Example for the PB 12-Run Pattern 302
C.5 Answer Key: Compare Your Hand Calculations 303
C.6 Equations for Hand Calculations 305
Appendix D: Free Recommended Software 307
D.1 Instructions to Obtain the R Language for Statistics 307
D.2 Instructions to Obtain LibreOffice 308
D.3 Instructions to Obtain Gosset 308
D.4 Possible Use of RStudio 309
Index 311
