Data Analysis of Analytical Method Transfers
•Télécharger en tant que PPTX, PDF•
12 j'aime•6,026 vues
To provide the basis for a PDA task force discussion to arrive at a consensus of best industry practices for data analysis of method transfers. The discussion is also relevant to method validation activities.
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à Data Analysis of Analytical Method Transfers
Similaire à Data Analysis of Analytical Method Transfers (20)
Data Analysis of Analytical Method Transfers
- 1. Data Analysis of an Analytical Method Transfer to Two Labs Comparability by testing absolute tolerance limits, equivalence of means, or a combination of both.
- 2. Purpose of Presentation To provide the basis for a PDA task force discussion to arrive at a consensus of best industry practices for data analysis of method transfers. The discussion is also relevant to method validation activities.
- 3. Introduction:Mitigating Risks to Technology Transfer Comparisons of methods is discussed in USP <1010>, but comparison of laboratories is not. In USP and ICH, accuracy is defined only in context to unbiasedness (trueness). In ISO, accuracy combines the concept of unbiasedness and precision, a form of total error. Comparison of laboratories should be considered in regards to the phase of development of the drug product.
- 4. Introduction:Mitigating Risks to Technology Transfer Acceptable criteria should reflect the intended purpose of the method and control the risk of incorrectly accepting an unsuitable analytical method. For example, the bias of a stability indicating method, especially if the bias may indicate a different product expiry, should be considered as a higher risk.
- 5. Introduction:Mitigating Risks to Technology Transfer: Phased Approach Phased approach includes considerations of risk for the different stages of development: Pre-clinical through early Phase II Evolution of process development Evolution of analytical methods Late Phase II through Phase IV Note: Risks in context to the phased approach varies owing to drug product, within companies, and changes to guidance.
- 6. Introduction:Less Risk to Transfer Studies: When trending is unimportant. When equivalent results is not necessary. When deliberate changes are made to processes and analytical methods within early development stages, Phase I and early Phase II.
- 7. Data Analysis The presentation reviews data analysis by use of: Absolute tolerance limits that emphasize means and intermediate precision within qualification acceptance criteria. The equivalence of means by the two one-sided t-test (TOST) is demonstrated. The use of both is considered within the context of the phased approach.
- 8. Introduction:Transfer Study Design The analytical method was transferred to two laboratories. The originating lab and receiving labs, n=3 labs, each tested n=6 homogenous samples (reported results were the mean of two replicates), split between n=2 analysts.
- 9. Introduction:Transfer Study Design For comparing the results within this presentation, actual acceptance criteria are not used; instead the data analysis assesses the acceptance criteria that the data supports. An assumption of the study is that the true value of the homogenous test sample is 4.0 mg/mL.
- 10. Raw Data
- 11. Accuracy and Precision Analysis
- 12. Sample Suitability Based on Replicate Precision
- 17. Outlier Analysis
- 18. Equivalence of Means: TOST Requires predetermination of an acceptance interval for the lower and upper limits, this is called a practical difference threshold. The confidence intervals for the ratio of test/reference are often set based on convention (e.g., alpha = 0.10). Tests whether the measured bias is within the acceptance interval. Compared to the t-test, the null and alternative hypothesis are reversed. The type I error is the probability of erroneous acceptance of equivalence. Type II error is the probability of erroneous acceptance of nonequivalence.
- 19. Equivalence of Means: TOST A matched pairs study design is required for TOST, analysis by absolute tolerance limits does does not. Some forms of TOST allow for asymmetric study design TOST is discussed in USP <1010> but it is not appointed as to its appropriateness for evaluating equivalence for laboratories or methods. It is discussed in ICH in regards to bioequivalence testing. In USP and ICH, accuracy is defined only in context to unbiasedness (trueness). In ISO, accuracy combines the concept of unbiasedness and precision.
- 20. Assumption for Superiority of Equivalence of Means Testing “Reduces risk of wrongly concluding equivalence when in fact two laboratories or two methods are not equivalent.” Feng, Liang, Kinser, Newland, and Guilbaud. Anal BioanalChem (2006) 385:975-981.
- 21. TOST Analysis Using JMP JMP describes the TOST test as “Practical Difference” testing and warns that confirming no difference in means is impossible. This form of TOST demands that an interval around the hypothesized value is chosen. The test tries to show that the mean is not outside the interval. The key to success is that the desired control around the mean is successfully selected. JMP equivalence of means testing does not require a symmetric study design.
- 22. TOST Analysis Using JMP To perform a TOST test: Do a one-sided t-test that the mean is the low value of the interval, with an upper tail alternative. Do a one-sided t-test that the mean is the high value of the interval, with a lower tail alternative. If both tests are significant at some level α, then you can conclude that the mean is out-side the interval with probability less than or equal to α, the significant level. In other words, the mean is not significantly practically different from the hypothesized value, or, in still other words, the mean is practically equivalent to the hypothesized value. (Technically, the test works by a union intersection rule, whose description is beyond the scope of this book.) --JMP Start Statistics: A Guide to Statistics and Data Analysis Using JMP by John Sall, Lee Creighton, Ann Lehman
- 23. Jmp: Oneway Analysis of Results
- 24. Jmp : Oneway Analysis of Results
- 25. Jmp: Practical Equivalence Difference considered practically zero, “Specified Practical Difference Threshold)” = 0.4. The alpha level was set at 0.10.
- 26. Jmp: Practical Equivalence Difference considered practically zero, “Specified Practical Difference Threshold)” = 0.4. The alpha level was set at 0.10.
- 27. Jmp: Practical Equivalence Difference considered practically zero, “Specified Practical Difference Threshold)” = 0.4. The alpha level was set at 0.10.
- 28. Jmp: Practical Equivalence Difference considered practically zero, “Specified Practical Difference Threshold)” = 0.8. The alpha level was set at 0.20.
- 29. Jmp: Practical Equivalence Difference considered practically zero, “Specified Practical Difference Threshold)” = 0.8. The alpha level was set at 0.20.
- 30. Jmp: Practical Equivalence Difference considered practically zero, “Specified Practical Difference Threshold)” = 0.8. The alpha level was set at 0.20.
- 35. Optional Slides Slides that follow are for further discussions that are of interest to the task force for the PDA AMD TR.
- 36. Total Error and Phased Approach Total error approach should be considered to include intermediate precision , bias within laboratories, and equivalency of means. A phased approach for data analysis may consider the use of equivalency testing and statistical determination of the robustness of study design. Studies should provide greater probability of correctly concluding that two identical methods are equivalent for Phase III and even late Phase II development stages.
- 37. Equivalence of Means Analysis The Schuirmann’s TOST is the standard approach for bioequivalence testing. The acceptance criteria for bioequivalence testing may be consistent with comparability and tech transfer studies of potency/content and stability indicating assays. Equivalence testing is often considered as a total error approach.
- 38. Equivalence of Means Analysis Hoffman and Kringle (H&K) argue that “typical acceptance criteria for analytical method precision and accuracy are not chosen with regard to the concept of method suitability and are commonly based on ad-hoc rules” which they say is inadequate. H&K: “Although such ad-hoc approaches may meet regulatory requirements, they yield unknown and uncontrolled risks of rejecting suitable bioanalytical methods (producer risk) and accepting unsuitable bioanalytical methods (consumer risk). “ H&K: “Current criteria are based on observed estimates of bias and variability, rather than on the true method bias and variability.”
- 39. Introduction:Mitigating Risks to Technology Transfer Random and systematic variability should be addressed in context of the measurement process and the analyte measured. Statistical measures should address the direction and magnitude of the errors to include the mean and the standard deviation, or the expressions derived from them (e.g., %RSD). Estimated variability can be used to calculate confidence intervals for the mean and tolerance intervals to capture an estimate of the specified proportion of the individual measurements.
Notes de l'éditeur
- In addition to the consideration of different methods to ensure quality of transferred analytical methods, a phased approach is also considered.
- In addition to the consideration of different methods to ensure quality of transferred analytical methods, a phased approach is also considered.
- Accuracy is acknowledged as an estimate of the actual value due to random and systematic variability.ICH does not refer to TOST in regards to laboratory comparisons, it does in regards to comparability of methods and bioequivalence studies.
- Acceptable assay performance requires more than just accuracy (trueness or unbiasedness).
- Stronger designs could mean something as simple as larger n.Phased approach was discussed with Rajesh as a compromise in discussions of Chapter 6 sections that he is writing. The concept is homogenous with other opinions in industry, but needs further development.The level of confidence and power would be specified using available formulas that require considerable knowledge, skill, and abilities.
- The presentation reviews data analysis by use of:Absolute tolerance limits that emphasizes means within the qualification acceptance criteria as well as intermediate precision.The equivalence of means by the two one-sided t-test (TOST) is demonstrated.The use of both is considered within the context of the phased approach.
- Raw data is grouped by lab, result, and analyst with duplicate replicates per sample, mean of replicates, %RSD of replicates, and % nominal. The data looks acceptable if the accuracy acceptance criteria is 80 to 120% nominal, but not acceptable if the criteria is 90 to 110% nominal. The result for lab 1, result 1, of operator 1 would fail the tighter tolerances. The result for lab 3, result 6, of operator 6, is borderline, as is the result for lab 1, result 6, of analyst 2. If the study had the tighter criteria, it would have failed transfer and the results would have been investigated to determine the root cause.Confidence intervals may also be used to provide an estimate for the proportion of results that fall within the absolute tolerance limits. Setting the appropriate confidence interval are based on the intended purpose. For example, a stability indicating assay may require a 99% CI while a 90% CI may be acceptable for an impurity test.
- The table is the analysis of accuracy and intermediate-precision using simple statistics, but there is no equivalence of means testing.The data analysis breaks precision into neat compartments to assist in estimating which variables add imprecision: inter-replicate (sample suitability) intra-analyst inter-analyst (intra-laboratory) inter-laboratory
- The first step in data analysis is evaluating the validity of the data. Sample 1 of lab 1 and analyst 1 would not be acceptable if the transfer study criteria for accuracy was 90 to 110% nominal; this result would be a failed sample. Also of concern is the % nominal for lab 3, analyst 6, result no. 5.The replicate precision is enlightening as it is the closest analysis to repeatability that is available. All other precision analyses that deviates from this precision may used to estimate for error. For example, greater inter-analyst imprecision, compared to repeatability, may be attributed to different analyst abilities. The results for accuracy are acceptable for criteria of 80 to 120% nominal.Equivalence of means testing as in the TOST, is blind to the intra-analyst data as it is limited to inter-laboratory analysis. This method may not be suitable despite the potential to pass the equivalence testing acceptance criteria.
- Intra-analyst precisiondoes show a significant problem within an analyst. High %RSD for lab 1, analyst 1, result no. 1, is due to one higher reported result; precision is less than 5% for all analysts except for lab 1/analyst 1.
- There appears to be a bias of the means of the labs, despite equivalent precision.There is a difference between significant and relevant bias and this is a case when acceptable differences is determined by practical requirements rather than what would be determined by equivalence testing. Equivalence testing: Helpful to determine method capabilities, not whether differences are relevant.Helpful in providing the confidence of the proportion of results within the tolerance limits.
- Inter-laboratory precision and the grand mean indicate precision and accuracy, but it can be misleading as occasional outliers may indicate analysis of samples that is out-of-control due to unknown or unobserved variables. Confidence intervals is a more practical approach than the two one-sided t-test to providing an estimation of the confidence in the proportion of results that will fall within the absolute tolerance limits. Insufficient number of sample analyses may not provide enough information to observe outliers. Strategies for investigating OOS data is critical to providing quality of results. However, the use of OOS investigations is biased toward acceptable results as a Type II error is hidden.
- The circled data is of concern, depending on the acceptance criteria for accuracy. If the criteria for accuracy is 90 to 110% nominal, then the transfer should fail as all samples should be within the acceptance criteria for a study with so few n. The difference in labs is also of concern as their may be a significant bias between lab means. If the criteria is 80 to 120% nominal, then the data is acceptable for the analytical method’s intended use.If the acceptance criteria for accuracy and precision were 90 to 110% nominal and 10%RSD, the precision and accuracy data would indicate that the sub-group’s results are of significant concern and should be investigated. An outlier test would be used to help ferret out data of concern, but would not be used to provide a justification to pass the transfer. The error precedes intra-analyst precision, it was due to one replicate. The investigation begins with evaluating the documentation of that analyst and progresses from there.
- Grubbs' test is based on the assumption of normality, thus the analysis must be done within each sub-group as the entire data set is tri-modal. The sub-groups have an n of six.Grubbs' test detects one outlier at a time. Each outlier is removed from the data set and the test is applied in an iterative fashion until no further outliers are detected. Each iteration changes the probability of detecting outliers and the test should not typically be used for sample sizes of six or fewer as the possibility of finding outliers becomes increasingly possible. However, for the purposes of investigations, it is understood that outlier tests are subjective and thus, used cautiously. The intended use of the test is to discover potentially inappropriate results to find potential sources of error. In this example, the outlier test is used only as an investigation tool and therefore it is used appropriately as there is NO risk to the business or product quality.
- Absolute tolerance limits requires a risk-assessment and is equivalent to the “practical difference threshold.” Setting the other parameters of an equivalence test is far more difficult than setting the practical difference threshold. Determining the appropriate confidence intervals are not intuitive and are commonly mistakenly set by convention; they should be set based on suitability for the intended purpose and that is the problem. What proportion of samples can be outside the practical difference threshold? The confidence intervals must be set based on the intended purpose of the method? If the method is an impurity test with a specification far higher than that by the manufacturing process, then the confidence interval can be relatively liberal, while the confidence interval for a stability indicating method may need to be very conservative. However, it is impossible to avoid reporting some results outside the threshold.
- Hoffman and Kringle describe data in their publication as assumed to be normally distributed. TOST is a non-parametric test that requires no assumptions regarding the way in which the distributions of the groups of data may differ.They correctly identify the need for a symmetrical study design; symmetrical study design is not required in the absolute limits testing approach.Hoffman and Kringle state “our long run interpretation above is likely in agreement with the original intent (if not application) of the 4-6-15 rule.” Actually they are not even close, they indicate that if 4 of 6 controls must pass, one of each concentration (which they did not mention), then “at least 66.7% of the observed assay values (in the long run) are within 15% of the true value.” This simplistic argument again is based on a false assumption that makes the rest of their argument irrelevant. The authors miss the concept of “Design Space.” The criteria of 66.7% of the controls must pass is usually not related to the space between the design space and the operating space. The authors then have a significant discussion of irrelevant information such as normal distribution of the data.
- Statistical equivalence is not relevant; practical equivalence is relevant. Authors start with an erroneous assumption. However, this does not discount possible benefits of TOST.Authors emphasize their erroneous assumption with the following statement, “This test has the advantage of limiting the risk of erroneously accepting a new laboratory or new method as being unbiased (when it is actually biased) to very low degree.” This statement is clearly an unacceptable approach to comparability and equivalence testing! It would be erroneous to judge a reasonable difference as a significant difference and therefore failing a comparability or transfer study. (!!) This approach is far too conservative to be a sensible approach. Some amount of bias is expected and is acceptable, the amount that is acceptable defines the upper and lower acceptance intervals. Setting the acceptance level is equal to setting absolute limits. Equivalence of means testing provides an estimate for the proportion of means that can be expected to fall within the intervals. This is an estimate based on a small data set, at least in the example provided here, and is based on the same fundamentals for considering risk to regarding a method as suitable for its intended purpose. The proportion of results that fall within the absolute tolerance limits may be estimated by the simple use of confidence intervals that are based on the intended purpose. For example, a stability indicating assay may require a 99% CI while a 90% CI may be acceptable for an impurity test.
- The statement that “confirming no difference in means is impossible” is related to the test being subjective and that the statistical approach relies on data with inherent error, amongst additional reasons.The statement regarding the “key to success” seems quite similar to the many TOST advocate’s argument against the ad hoc approach required of absolute tolerance limits data analysis.Our methodology for determining risk and acceptable limits are the same and are based on business and product risks. TOST only adds the benefit of a confidence interval around the mean that is based on an additional limit that must be defined. The confidence interval should be set on risk too, not on convention as is often the case.
- The statement that “confirming no difference in means is impossible” is related to the test being subjective and that the statistical approach relies on data with inherent error, amongst additional reasons.The statement regarding the “key to success” seems quite similar to the many TOST advocate’s argument against the ad hoc approach required of absolute tolerance limits data analysis.Our methodology for determining risk and acceptable limits are the same and are based on business and product risks. TOST only adds the benefit of a confidence interval around the mean that is based on an additional limit that must be defined. The confidence interval should be set on risk too, not on convention as is often the case.
- The width of the diamonds shows the symmetry of the replicates per laboratory.The top and bottom tips of the diamonds show the 95% confidence intervals.The horizontal lines through the middles of the diamonds are the group means.The smaller horizontal lines near the tips are the overlap marks that indicate whether the group means are significantly different at the 95% confidence level. Laboratory 1 and 3’s overlap marks clearly do not overlap, indicating they are different.The confidence interval computation assumes that variances are equal across observations. Therefore, the height of the confidence interval (diamond) is proportional to the reciprocal of the square root of the number of observations in the group.
- The alpha level was set at 0.10.The grand mean is found in the “Summary of Fit” table. The group means are found in the “Means for Oneway ANOVA” table.The study design is partially described by the number of observations in the “Summary of Fit” and the “Means for Oneway ANOVA” table; the study is symmetrical with n=6 results per laboratory. The number of analysts is not reflected in this analysis.The analysis of variance indicates that the laboratory error and the unexplained error account for similar amounts of the total error, C. Total.The “Std Error” in the “Means for Oneway Anova” table shows that the estimates of the standard deviations for the group means is equal. That is because it is derived from the ratio of the root mean square error and the number of observations; this is a symmetric design. This is reflected in an equal difference between the upper and lower 95% confidence interval.
- Only two group means can be tested at a time.The input for “Difference considered practically zero” (Specified Practical Difference Threshold), was 0.4.The alpha level was set at 0.10.The differences between laboratories at the upper interval is clearly significantly practically zero.The difference between laboratories at the lower interval is significantly practically zero.
- The input for “Difference considered practically zero” (Specified Practical Difference Threshold), was 0.4.The alpha level was set at 0.10.The differences between laboratories at the upper interval is clearly significantly practically zero.The difference between laboratories at the lower interval does not show that the means are practically equivalent to 4.0 at the alpha = 0.10 significance level.Data will not be acceptable if samples are analyzed by both laboratories 1 and 3. This is an interesting problem, perhaps different inputs are acceptable. This is the same scenario as any tech transfer, the appropriate acceptance criteria with the correct assumptions are necessary for comparability of methods or laboratories. Some companies may be tolerant of using just one of the laboratories, but this is highly risky as the laboratory differences have not been investigated and resolved.
- The input for “Difference considered practically zero” (Specified Practical Difference Threshold), was 0.4The alpha level was set at 0.10.The differences between laboratories at the upper interval is clearly significantly practically zero.The difference between laboratories at the lower interval is significantly practically zero.Conclusion: Laboratories 1 and 3 are not practically equivalent, but both are equivalent to lab 2. A less strict acceptance criteria follows.
- This and the following two slides evaluates the data set using different acceptance criteria inputs for the practical difference threshold of (corresponds to ± 20% nominal) and the alpha level, 0.8.The differences between laboratories at the upper interval is clearly significantly practically zero.The difference between laboratories at the lower interval is clearly significantly practically zero.
- The differences between laboratories at the upper interval is clearly significantly practically zero.The difference between laboratories at the lower interval is clearly significantly practically zero.
- The differences between laboratories at the upper interval is clearly significantly practically zero.The difference between laboratories at the lower interval is clearly significantly practically zero.
- The differences between laboratories at the upper interval is clearly significantly practically zero.The difference between laboratories at the lower interval is clearly significantly practically zero.
- The required acceptance criteria dictate the appropriate design and sample size of an equivalence study (outside the scope of this presentation).
- Confidence intervals around the grand mean can be added, but with n=3 labs, it may not be appropriate and may be misleading. Confidence intervals within sub-group may be acceptable for the purposes of an investigation , but not as the basis of an acceptance criterion.
- In earlier phases, with less historic data and lesser optimized analytical methods, it is often appropriate to have wider acceptance criteria.
- The difficulty of appropriately defining the inputs of a TOST make it far likelier to result in inappropriate conclusions. Complex statistics in the hands of even experienced scientists will result in inappropriate conclusions. An experienced scientist may be able to intuitively understand the problem and reassess the data analysis tools used. The risk-to-reward of using equivalence of means testing is insufficient for most organizations and should always be used in conjunction with absolute limits testing. Absolute limits testing provides more information, more practical information, less confusion, and the lack of equivalence of means testing can be mitigated by using simple confidence intervals.
- Ad-hoc means tailored specifically to a task, which is a practical approach. OK, I am uncertain as to why that is a problem. It is not true that the practical approach does not consider acceptance criteria based on method suitability. This is a false assumption.Practical approach versus a statistical approach:Both approaches have the goal of acceptance criteria that provides for accepting analytical methods as suitable for their intended purpose. Both are based on historic data from analytical development, qualification, or validation results. Practical results are tailor made for the method’s intended purpose as is the equivalence of means testing. Practical approach considers intra- and intermediate-precision as well as bias of means. Total error is similar in absolute limits analysis as it is used to evaluate accuracy and precision and it is used to evaluate more sources of error. Equivalence testing considers just the bias and the confidence of the proportions of means within the tolerance limits, which is good data to know and not provided by absolute tolerance limits analysis. Acceptance criteria for determining the method’s suitability for it’s intended purpose are determined based on practical needs, not necessarily a method’s bias and variability; the method’s true performance characteristics may be insufficient.
- Ad-hoc means tailored specifically to a task, which is a practical approach. OK, I am uncertain as to why that is a problem. It is not true that the practical approach does not consider acceptance criteria based on method suitability. This is a false assumption.Practical approach versus a statistical approach:Both approaches have the goal of acceptance criteria that provides for accepting analytical methods as suitable for their intended purpose. Both are based on historic data from analytical development, qualification, or validation results. Practical results are tailor made for the method’s intended purpose as is the equivalence of means testing. Practical approach considers intra- and intermediate-precision as well as bias of means. Total error is similar in absolute limits analysis as it is used to evaluate accuracy and precision and it is used to evaluate more sources of error. Equivalence testing considers just the bias and the confidence of the proportions of means within the tolerance limits, which is good data to know and not provided by absolute tolerance limits analysis. Acceptance criteria for determining the method’s suitability for it’s intended purpose are determined based on practical needs, not necessarily a method’s bias and variability; the method’s true performance characteristics may be insufficient.
- There are assumptions for all tests and none can be violated, otherwise, another method may be required to evaluate the data.Normal distribution is not as important when the objective is to construct a confidence for the difference between means because of the central limit theorem. However, the normality of data of data used to construct confidence intervals