The concepts of Alpha and Beta errors have long been documented in the statistical literature. Often, the notion of significance has been more widely promoted than statistical Power. Under and over powered tests can easily lead data analysts to draw invalid conclusions. The problem is further complicated by the boom of ‘Big Data’ where rates of automation and data collection have increased exponentially. It is not uncommon for a data set to easily run into millions of rows and, potentially, thousands of columns. This talk will start by discussing the fundamental concepts of alpha and beta error in data analysis, as well as statistical power, while discussing some of the common pitfalls. As data collection and computing power grow at alarming rates, there is a risk for some to focus on the utilizing standard data analysis tools and be lured into making false conclusions. Newer analytic techniques have been developed to complement the wave of Big Data. Some of these newer methods will be briefly introduced as tools for modern day data analysts to consider in their quest for managing risk via proper data analysis.

1. OLEDWorks LLC©
Risk Management in Data Analysis:
Moving Beyond Alpha & Beta
David Lee
Director, Quality and Reliability
OLEDWorks LLC
dlee@oledworks.com
https://www.youtube.com/watch?v=OHwYpev2-4Q

2. OLEDWorks LLC©
OLEDWorks enables you to revel in possibility.
Design Freely.
Create Passionately.
Be Unlimited with Light.

3. OLEDWorks LLC©
What do people love about OLED lighting?

4. OLEDWorks LLC©
What’s an OLED?
OLEDs are electroluminescent electrical devices that can be tuned to emit a range
of visible light in response to an applied voltage/current
•There can be
anywhere up to ~20
layers in an OLED
device
•Each layer can
consist of a single
component or
mixtures of multiple
components
•The total thickness of
the organic layers is
on the order of 200-
300 nm (0.2-0.3 )

5. OLEDWorks LLC©
Hypothesis Testing, Power and
Sample Size Determination, Or…
• The idea for this talk originally came from CHRISTOPHER A. SEAMAN, JULIA E. SEAMAN AND
I. ELAINE ALLEN, “The Significance of Power”, Quality Progress, July 2015, pp. 51-53
• I’m sure that some people I work with believe I follow the ideology of the Dilbert cartoon
• BUT, it really boils down to understanding and managing risks by balancing the needs of the
study with the realities of time, budget, resources, competitive pressures, etc.

6. OLEDWorks LLC©
Comparing the Difference Between Two Groups
Take a hypothetical situation where you’re comparing two
groups
How do I quantify the risks associated with each situation?
• One of the objectives of today’s talk is to provide a
framework for people to quantify some of the trade-offs

7. OLEDWorks LLC©
Truth Table
We go out and run a test, collect the data and now need to
make a decision..
There are four possible outcomes based on the table above
Let’s look at a real life example before talking about stuff
like null hypothesis, alpha, beta and Power?
Truth (Unknown)
Decision

8. OLEDWorks LLC©
Judicial System Example: Criminal Trial
8
Jury’s Decision
Correct
Type II Error
(ββββ-Risk)
Type I Error
(αααα-Risk)
Correct
Not Guilty
Guilty
Actually
Innocent
Actually
Guilty
The Truth
Consequence: Innocent
Man Goes To Jail
Consequence: Criminal
Goes Free
HO: The man is innocent
H1: The man is not innocent
(i.e. Guilty)
Innocent Until Proven Guilty

9. OLEDWorks LLC©
For most of us that’s not an
everyday decision…
But these are terms
we, at least, hear
about in the
workplace
These terms can be
applied to any work
environment or field
of study!

10. OLEDWorks LLC©
I Always Start with Three Questions
1. What is the question you are trying to answer?
a) Formulate a hypothesis
b) What is it that you are trying test?
2. How much of a difference are you trying to detect?
a) Often the question gets turned around (or ignored):
I only have n samples, how much of a difference can I detect?
3. What is the response and how will it be measured?
a) Continuous, Pass/Fail, Proportions, etc.
b) What is the gage’s contribution to the measurement
system variability?
i. Don’t assume your response is without uncertainty

11. OLEDWorks LLC©
TESTS OF HYPOTHESES
As was mentioned earlier, sometimes we cannot survey or test all
persons or objects; therefore, we have to take a sample. From the
results of analysis from the sample data, we can predict the results from
the population. Some questions that one may want to answer are:
1) Are unmarried workers more likely to be absent from work than
married workers?
2) Are the sixth graders in a certain school significantly less skilled in their
mathematical abilities than the average student in the district?
3) In Fall 1996, did students in Math 163-01 score the same on the exam
as students in Math 163-02?
4) Is there any difference between the strengths of steel wire produced
by the XY Company and Bob’s Wire Company?
5) A hospital spokesperson claims that the average daily room charge for
a specific procedure is $622. Can we reject this claim?
11

12. OLEDWorks LLC©
WHAT IS A HYPOTHESIS?
Hypothesis: A statement about the value of a population parameter
developed for the purpose of testing.
Examples of hypotheses, or statements, made about a population
parameter are:
The mean monthly income from all sources for systems analysts is $3,625.
Twenty percent of all juvenile offenders ultimately are caught and sentenced to
prison.
Hypothesis testing: A procedure, based on sample evidence and probability
theory, used to determine whether the hypothesis is a reasonable
statement and should not be rejected, or is unreasonable and should be
rejected.
There is a five-step procedure for testing a hypothesis.
Now let's go through the five steps for gas price example.

13. OLEDWorks LLC©
5 STEPS IN THE HYPOTHESIS TESTING
PROCEDURE
1) State the null hypothesis and alternate
hypothesis
2) Select the appropriate test statistic and level of
confidence
3) State the decision rules
4) Compute the appropriate test statistic and make
the decision.
Compare the computed test statistic with critical value.
5) Interpret the results

14. OLEDWorks LLC©
State the Null and Alternative Hypotheses
The substantive hypothesis or research hypothesis (H1)
states the relationship in which we are really interested.
We always state research hypotheses in terms of
population parameters because we want to use sample
statistics to estimate population parameters. Our
research hypothesis was: H1: µgas prices > $2.86
The null hypothesis (H0) always contradicts the research
hypothesis, usually stating that there is no difference
between the population mean and some specified value.
Our null hypothesis was: H0: µY = $2.86

15. OLEDWorks LLC©
Select the appropriate test statistic
It is usually more convenient to standardize the sample mean and use a test
statistic based on the standardized distribution
The two most used statistics are the Normal and Student t distributions
where ܺഥ rom the data, μ is the mean of the original normal distribution or is
the hypothesized value, and σ is the standard deviation of original normal
distribution.
ࡴ࢕: ࣆࢍࢇ࢙ ࢖࢘࢏ࢉࢋ࢙ = $૛. ૡ૟ ࢝࢏࢚ࢎ ࢄഥ = $૛. ૢ૞, ࢔ = ૚૙૙ ࢙ࢇ࢓࢖࢒ࢋ࢙ and ࢙ത = $૙. ૚ૠ
ࢆ =
ࢄഥିࣆࣆࢍࢇ࢙ ࢖࢘࢏ࢉࢋ࢙
࣌ഥࢍࢇ࢙ ࢖࢘࢏ࢉࢋ࢙࢙
ࡺ
ൗ
=
૛.ૢ૞ି૛.ૡ૟
૙.૚ૠ
√૚૙૙ൗ
= 5.29
o
o
X-μ
Z =
σ
n

16. OLEDWorks LLC©
Type I (α) Errors and the Null
Hypothesis Distribution
• To understand Type I errors, we
consider the situation where the
null hypothesis is true
A traditional guideline for
choosing the level of significance
is as follows:
(a) the 0.10 level for consumer and
political polling,
(b) the 0.05 level for industrial
research projects, and
(c) the 0.01 level or lower for
quality assurance work.
Aerospace
Select the Significance Level and
State Decision Rules

17. OLEDWorks LLC©
Back to our Gas Prices Example
ࡴ࢕: ࣆࢍࢇ࢙ ࢖࢘࢏ࢉࢋ࢙ = $૛. ૡ૟ ࢝࢏࢚ࢎ ࢄഥ = $૛. ૢ૞, ࢔ = ૚૙૙ ࢙ࢇ࢓࢖࢒ࢋ࢙ and ࢙ത = $૙. ૚ૠ
ࢆ =
ࢄഥିࣆࣆࢍࢇ࢙ ࢖࢘࢏ࢉࢋ࢙
࣌ഥࢍࢇ࢙ ࢖࢘࢏ࢉࢋ࢙࢙
ࡺ
ൗ
=
૛.ૢ૞ି૛.ૡ૟
૙.૚ૠ
√૚૙૙ൗ
= 5.29
And
Type I or αααα Risk is set at 5% or 0.05
Then the decision limit is Z=1.95
And we would reject the null hypothesis that the
average price of gas is $2.86/gallon

18. OLEDWorks LLC©
What does this look like graphically

19. OLEDWorks LLC©
GPower Software
Free download at http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3

20. OLEDWorks LLC©
Decisions in Hypothesis Testing
The previous example seemed pretty
straightforward but also introduced a few new
concepts:
Type II or Beta Error
Statistical Power

21. OLEDWorks LLC©
Type I or α Risk Factors
α is used to indicate the degree of risk we are willing to
take in falsely rejecting the null hypothesis HO
• Sometimes called Manufacturers Risk
We use alpha as the threshold value (significance level)
to make our accept/reject decisions
• If p<α then reject the null hypothesis (some change did
occur)
• If p>α then fail to reject the null hypothesis (no change
occurred)
The practical implication of this risk is that we will take
action which will not yield any improvement
Typical values for α are 1%, 5% and 10%

22. OLEDWorks LLC©
Type II or β Risk Factors
β is associated with Type II Errors: Falsely accepting the null
hypothesis
• Sometimes called Consumer’s Risk
• The consequence of this type of error is likely to result in the
consumer receiving inferior product
1-β = Chance of detecting a specified change in the process if
the difference is actually there to detect
Called the “Power of the test”
The practical significance of this is that an opportunity for
improvement will be overlooked
• No significant differences will be detected upon testing
β errors can result in loss of market share
Typical values of β include 10% and 20%

23. OLEDWorks LLC©
Sample Size and Effect Size Codetermine Power
Some possible questions:
• How many samples (i.e. subjects) will I need for my study?
• I already know that I will only have access to n samples (subjects).
• Will this be enough?
• What is the sensitivity of my experiment?
• Answering these questions depends on understanding the relationship
between sample size, effect size (difference) and statistical power
23
Sample size n
β−Power 1
Effect Size d
+
+

24. OLEDWorks LLC©
What happens if we’re only trying
to determine a $0.05 difference?

25. OLEDWorks LLC©
SHOW JMP DEMO

26. OLEDWorks LLC©
Pitfalls of Under and Over Powered Studies
Very often underpowered tests fail to detect significant
difference (↑ Beta Error)
Only effective at accurately determining very large effect
sizes
Overpowered tests claim statistical significance even
when there’s no practical significance
Hyper sensitive tests; Everything’s significant
Increased and often unnecessary control schemes

27. OLEDWorks LLC©
Example of an Overpowered Test
The R2 and adjusted R2 are only ~3%,
which is pretty low.
Although, the p-value for the model is
very low which would lead you to believe
that it’s highly significant
This may be misleading because you have
so many degrees of freedom in the Error
term (df error = 1477)
The MS Error is essentially scaled by the
number of df’s by 1/sqrt (df error)
I call this a hyper sensitive model where
the analysis claims wildly significant results
but the graphical side doesn’t match up
The Parameter Est. predicts ~0.067
increase in Prod Rate per 1 unit increase in
FB3K but there’s still roughly a 5 unit swing
in rod Rates at nearly equal levels of FB 3K.

28. OLEDWorks LLC©
Production Rate
•This just your plot color coded
with the confidence intervals for
both the fitted observation and
individuals shaded
•Doesn’t look like there’s an effect
by Date
•The spread in Prod Rate at nearly
constant levels of FB 3000 is on
the order of the entire range of
all the data. It seems to
overwhelm the predicted rate at
a give level
•You'd like to see some separation
between, at least, the low and
high levels of FB3K

29. OLEDWorks LLC©
Extensions of Sample Size & Power
Software provides powerful
tools to asses the properties
of a design
See the strengths and
limitations of your existing
experimental design
You can asses the impact of lost
runs
Determine your design’s
ability to detect effects
associated with meaningful
changes in the response
Of course, you have to make
some assumptions

30. OLEDWorks LLC©
Example of Power Analysis to a DOE
This example
shows the
Power for a 3-
factor Box-
Behnken DOE
A prospective analysis requires
several assumptions, but their
sensitivities can also be tested
and trade-offs assessed

31. OLEDWorks LLC©
What Happens if We Lose Runs?
The power values for the actual design
are uniformly smaller than for the
intended design.
For Silica and Sulfur, the power of the
tests in the intended design is almost
twice the power in the actual design
For the Silica*Sulfur interaction, the
power of the test in the actual design is
0.231, compared to 0.672 in the
intended design.
The resulting design results in
substantial loss of power in comparison
with the intended design.

32. OLEDWorks LLC©
Comparing Designs
What if we wanted to run
an experiment to look at 6-
factors in a minimal
number of runs
Let’s compare two 13-run
experiments:
Plackett-Burhman
Definitive Screening Design
The PB design appears to
outperform is only
analyzing for Main Effects
What if you have active
interactions or quadratics?

33. OLEDWorks LLC©
Color Maps for PB and DSD Comparison
The Color Map on Correlations plots show that the PB
design aliases main effects with two-way interactions.
In contrast, the DSD does not alias main effects with
two-way interactions.

34. OLEDWorks LLC©
Power Analysis for PB and DSD
Comparison with Interactions
What if there were 3
active 2-fator
interactions?
In a the DSD Main Effects
and 2-Factor Interactions
are uncorrelated
Now, the DSD
outperforms the PB
Further ‘What-If’
analyses like this can be
conducted

35. OLEDWorks LLC©
As Data Collection Rates and Complexity
Increase Different Tools are Needed
The Boom of Big Data has
facilitated the need for
new analytic techniques
to asses variable
importance
As model complexity
increases interpretability
suffers
Moving away from
inferential statistics
toward predictive
analytics
The form of the model
can induce bias. It’s not
just simple least squares
anymore.
James, Witten, Hastie & Tibshirani, “An Introduction to Statistical Learning”, 2013

36. OLEDWorks LLC©
Managing Bias and Variance
Techniques for
estimating model
parameters and
variable importance
include:
• K-fold cross-
validation
• Training/Test Splits
• Bootstrapping
• Many, many more
• But, this really
should be another
talk…

37. OLEDWorks LLC©
Thank You & Questions

38. OLEDWorks LLC©
APPENDIX & MISCELLANEOUS

39. OLEDWorks LLC©
OLED Lighting

40. OLEDWorks LLC©
Light Sources and Luminaires

41. OLEDWorks LLC©
OLEDWorks – How we participate

42. OLEDWorks LLC©
OLEDWorks – What We Do
WE MAKE OLED LIGHT ENGINES
OLED material, formulation, process and quality/reliability experts
OLED lighting manufacturing innovation
Aachen: Make the world’s brightest panels, high volume capacity
Rochester: Disruptive low cost structure, amber, low volume, scalable
OLED collaboration and integration
Driver and electronics support, technical support, supplier collaboration
Department of Energy test site for new OLED technologies

43. OLEDWorks LLC©
Basic OLED Structure
5 – 10 V

44. OLEDWorks LLC©
Design of High Efficiency OLEDs

45. OLEDWorks LLC©
How Big is a Micron?
•Individual layers and deposition rates are measured in Nanometers (10-9 m) and
Angstroms/sec (10-10 m)

46. OLEDWorks LLC©
Definitive Screening Designs
2011 - Introduced by Jones and Nachtsheim
2013 – Jones and Nachtsheim show 2-level categorical factors
can be incorporated
2015 – Jones introduces concept of ‘Fake Factors’ at JMP
Discovery Summit to improve power and signal detection
2016 - Jones, B. and Nachtsheim, C.J. (2016), “Effective Model
Selection for Definitive Screening Designs,” Technometrics,
forthcoming
2016 – Two-Stage modeling implemented in JMP version 13

47. OLEDWorks LLC©
What is a Definitive Screening Design?
Inherently small designs with the minimum number of runs -> n=2m+1
If there are m=6 factors then the minimum DSD can be run in just 13 trials!
DSDs are three-level designs that are valuable for identifying main effects and second-order
interactions in a single experiment.
A minimum run-size DSD is capable of correctly identifying active terms with high probability if the
number of active effects is less than about half the number of runs and if the effects sizes exceed
twice the standard deviation
DSDs utilize a methodology called Effective Model Selection takes advantage of the unique structure
of definitive screening designs.
The Effective Model Selection for DSDs algorithm leverages the structure of DSDs and assumes strong effect
heredity to identify active second-order effects.
DSD’s can be augmented with properly selected extra runs to significantly increase the design’s ability
to detect second-order interactions.
These extra runs correspond to fictitious factors that are included in the design but not in the analysis
Jones and Nachtsheim* (2016) report power for detecting 2FI and quadratic effects is greatly
improved especially when there are fewer active Main Effect
*Jones, B. and Nachtsheim, C.J. (2016), “Effective Model Selection for Definitive Screening Designs,”
Technometrics, forthcoming

48. OLEDWorks LLC©
Introduction of extra runs via including fictitious
factors in the design of the experiment
In a DSD Main Effects are orthogonal to 2FI, two-stage
modeling splits the response into two new
components (i.e., responses)
Model occurs in two steps:
Y Main Effects
Y Second Order Effects
Assumes model heredity
What is Effective Model Selection?
Referred to as Two-Stage in v13

49. OLEDWorks LLC©
DSD Example: OLED Device
Experiment consisted of 6
Factors (A thru F)
conducted in 2 Blocks
This talk will discuss 6
responses
‘Typical’ 6-factor DSD
would have 2m+1=13 runs
This design incorporates 2
additional factors in the
set-up resulting in 4
additional runs and 1 run
for the Block effect totaling
18 runs.

50. OLEDWorks LLC©
Correlation & Power Comparison
JMP12: n=14 runs JMP 13: 2 Addt’l Factors, n=18 runs
• DSD is for 6 Factors run in 2 Blocks
• Power for detecting 2FIs and Quadratic effects is much higher for the two-
stage method especially when fewer MEs are active
Jones, B. and Nachtsheim, C.J. (2016), “Effective Model Selection for Definitive
Screening Designs,” Technometrics, forthcoming

51. OLEDWorks LLC©
Modeling of Y Main Effects
Each foldover pair sums to zero and the centerpoint estimates are zero

52. OLEDWorks LLC©
Modeling of Main Effects Continued
Since the foldover pairs sum to zero there are 8
independent values ( df = 8 )
There are 6 Factors; therefore, there are 8-6=2 df
for estimating variability
According to Jones*, an estimate of σ2 can be
obtained by summing the squared residuals and
dividing by the remaining degrees of freedom
Use this estimate to perform t tests on each
coefficient and determine significance
*Jones, B. (2015),JMP Discovery Summit

53. OLEDWorks LLC©
Identifying Significant 2nd Order Effects
Modeling of the 2nd order interaction in a DSD
assumes model heredity, so only interactions
and quadratic effects from significant main
effects are considered
JMP performs all subsets regression stopping
when the MSE of the ‘best’ model is not
significantly larger than the estimate of σ2

54. OLEDWorks LLC©
How it’s Done in JMP v13

55. OLEDWorks LLC©
DSD – Fit Least Squares
• Can’t save prediction
formula in the DSD Fit
platform
• Limited options for
Maximizing
Desirability
• Need to make and
run model, then you
can follow thru with
all of the options for
evaluating the model

56. OLEDWorks LLC©
Response Y2
The predicted model
didn’t seem to correlate
with the expected
physics of an OLED
device
Device experts and
optical models didn’t
seem to agree with the
DSD model
Alternate methods were
explored

57. OLEDWorks LLC©
Response Y2 – Alternate Modeling Strategies
Forward Stepwise w/ AICc Gen Reg w/ Elastic Net and AICc

58. OLEDWorks LLC©
Model Comparison
Stepwise seems to
outperform models
generated by the DSD as
well as Gen Reg
The difference seems to
boils down to how one
data point is handled
Inspection and
additional testing
showed no obvious issue
with this device
We had no reason to
believe there was an
issue with this point as it
didn’t exert extensive
leverage

59. OLEDWorks LLC©
DSD Conclusions
New in v13! The concept of introducing extra runs
via fictitious factors and the Effective Model
selection or Two-Stage method for constructing
and analyzing designs was demonstrated through
the optimization of an OLED device
Alternative methods such as Stepwise and
Generalized Regression were compared
Simulation studies were used to identify problem
areas and understand projected defect rates