Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Design of experiments (A/B Testing)

1 093 vues

Publié le

An introduction to Design of experiments (A/B Testing)- How to design statistical and efficient experiments.

Publié dans : Données & analyses
  • Identifiez-vous pour voir les commentaires

  • Soyez le premier à aimer ceci

Design of experiments (A/B Testing)

  1. 1. Design of Experiments Industry Applications and Usage https://www.quora.com/profile/Ratnakar-Pandey-RP https://www.linkedin.com/in/ratnakarpandey/
  2. 2. What is Design of Experiment In general usage, design of experiments (DOE) or experimental design is the design of any information-gathering exercises where variation is present, whether under the full control of the experimenter or not. Source: Wikipedia
  3. 3. DOE links causes to effects Experiment Causes or Independent Factors or Inputs Responses or Effects or Outputs Controlled Factors Un-Controlled Factors or Noise
  4. 4. Link causes and effect- many factors and their interaction can be evaluated at the same time. More efficient than change one factor at a time (OFAT approach) and hold other constants approach. Reduce Variance Confirm hypothesis Identify key factors Understand interaction among factors What if analysis and simulation Design products and offers Optimization What is DOE used for
  5. 5. Real world case studies of DOEs Use A/B testings, experimentation to deliver ● Right reward structure and difficulty levels of games to keep customers engaged ● Placement of ads to enhance revenue and minimize negative impact on customers ● Retention of customers ● Cross-sell and promotions strategies and track conversion Zynga, Rovios and other gaming companies Amelia Showalter, Director of Digital Analytics led Obama 2012 campaign and conducted hundreds of A/B Testing (or randomized desigend experiment to drive online funds collections of over $ 500MM. Net impact of Data anlytics and DOE estimated to be around $200MMM ● Email variations such as subject, and personalization ● Form lengths- short vs long ● Donation amount and history ● Website layout Obama 2012 Reelection Campaigns Use DOE for ● Product design. For example, does a 25% APR and 2% cash- back credit card sells more or 30% APR and 5% cash-back ● Promotions. Does 33% discount work better than buy 2 get 1 free ● Campaigns- Channels, Offers, Timing, Sequencing, Product combination to maximize responses ● Website design- placement of different sections of website, design, content etc. to get higher ranking, more customers Telecom, Financial, E Commerce and many other industries
  6. 6. Clear definition of problem or objective Constraints- time, cost etc. Factors- independent, dependent Levels of factors Type of design Model to be fitted Implementation plan Things to consider in DOE
  7. 7. Steps involved in DOE 1 Define Objectives 2 Setup Experiment 3 Stakeholders Buy-in 4 Conduct Experiment 5 Analyze Data 6 Implementation
  8. 8. Key technical concepts DOE establishes the cause and effect relationship between independent and dependent factors Dependent variables or Response variables are the outcome that DOE tries to impact Independent variables or Factors are the variables whose impact DOE tries to estimate Levels are the settings at which factors or independent variables are run during experiment Treatments are unique combinations of factors and levels. Each run of an experiment is a treatment Repeated treatments are called Replicates
  9. 9. Key technical concepts (cont.) Replication ● Repetition of certain or all experiment legs to build more confidence in the results ● Minimizes chance driven outcomes Example- 15% off on bills and 20% off on bills are repeated for 2 sets of 2 customer segments Randomization ● To create homogenous (similar or lookalike) groups for treatment ● Helps in removing the impact of unknown and uncontrolled factors on the experiment outcomes Example- customers are randomly selected for 15% off, 20% off offers to evaluate impact of the offers. Blocking ● Used to minimize the impact of noise factors or factors of no importance to the experiment. Example- Affect of 15% off and 20% off may be different on weekdays vs weekend. Days can be used as blocking, i.e, the same experiments are run on both weekdays and weekend. Randomized Blocked Design- customers are first divided in blocks and then randomized within each block.
  10. 10. Types of DOE Full Factorial ● Impact of all factors and their interaction can be investigated ● Most Robust but generally most expensive ● No. of treatments can be estimated by following formulae- LF when each factors has same levels. L is number of levels and F is number of factors. For example a DOE that involves two factors A & B at two levels ‘Low’ and ‘High’ will have 2^2 or 4 treatments. A B 1 1 1 -1 -1 -1 -1 1
  11. 11. Types of DOE (cont.) Full Factorial Similarly a three factors (A,B,C) at two levels (Low, High) will have 2^3 =8 treatments. If Factor A has ‘m’ levels and factor B has ‘n’ levels. For full factorial we need to run mxn treatments or experiments. For example- In an experiment involving following factors and levels. ● Customer Segment (Premium, Middle, Value) ● Products (Cards, Personal Loan, Car Loan) ● Term (Long, Short) ● Interest Rate (Low, Medium, High) we need to run 3x3x2x3= 54 experiments or treatment We can see clearly the size of the experiments in Full Factorial escalates very quickly as the number of factors and/or level increases. Hence it is not practical to run Full Factorial where levels and factors are high.
  12. 12. Types of DOE (cont.) Fractional Factorial (FFD) ● Only a portion (fraction) of the combinations are run and hence not all interactions are investigated ● Balanced Fractional Factorial Design helps answer main factor impact and some interaction impact as each factors is tested at the given levels same number of time. ● More economical and practical than Full Factorial designs for large number of factors and levels and hence more commonly used. Example- A Half Factorial Design with three factors (A,B,C) at two levels (Low, High) will have (2^3)/2 =4 treatments. These 4 experiments can be chosen in` more than several dozen possible ways. The best way to pick the 4 experiments is using the Balance Fractional Design.
  13. 13. Comparison of Full and FFD Full Factorial Fractional Factorial A B C A B CGetting Investigated Not getting investigated
  14. 14. Effect of main factors are confounded/aliased with interaction terms. For example, effect of main factors are confounded with 2 factors interactions. This is called Resolution 3 FFD. Some 2 factors interaction terms are aliased with other 2 factors interactions. Resolution 4 FFD has main effects confounded with 3 factors interactions or 2 factors interactions with other 2 factors interactions. Resolution 5 or higher doesn’t have any confounding between main effects and 2 factors interactions. Some main effects are aliased with 4 factors interactions and 2 factors interactions are aliased with 3 factor interactions. Shortcomings of FFD In general, the highest resolution fraction factorial design should be chosen as it has less confounding
  15. 15. Taguchi’s Orthogonal Arrays- Very efficient design to identify impact of main effect with only a few experiments. For example, L4 design can be used to identify impact of 3 main effects using only 4 runs. Custom Design- A good option when # of experiments are limited. Uses D- Optimality criteria to provide maximum information from experiment Response Surface Methods- Used primarily for optimization of factors settings to get desired response. Generally done towards the end of the experimentation Screening- Used generally in the beginning of the experiment to economically identify key main factors and lower order interactions. Other designs
  16. 16. SAS JMP Base SAS Matlab R DOE++ Minitab Etc. Tools for DOE