A presentation on my experience simulating large multiphase trails to predict time of completion based on observed patient status data

1. Signs of the Timings:
Predicting Time of Completion in
Multiphase Survival Trials
Dennis Sweitzer
Ali Falahati
Delaware Chapter of the ASA
September, 2006

2. Study Flowchart

3. The Protocols
Outcome:
• Time to Randomized Relapse
Open Label Phase
– Up to 36 weeks
– Patients must be stable for 12 weeks before randomization
– High withdrawal rate (30-70%)
– Assumed 50% randomize
Randomized Phase
– Up to 104 weeks
– High withdrawal rate
– Assumed 30% Relapse rate
– Trial could not end until last Patient randomized >28 weeks

4. Sensitivity to Relapse & Discontinuation rates (1)
Low Discontinuation
relative to Relapse
Cumulative Patient statuses
as trial progresses
100 Relapse ~Sep
Wrong assumptions, wait
longer

5. Sensitivity to Relapse & Discontinuation rates (2)
Higher event rates
deplete patient pool
Plan to stop enrollment as soon as
certain of reaching 100
~ July
Higher Discontinuation Rate,
Lower relapse Rate
Large delays
May never reach goal

6. Stopping Enrollment
Stopping Criteria
• At least 227 Relapses
• All patients still in Randomized Phase complete at least 28
weeks of treatment
Ideally:
• 227th Relapses occurs shortly after:
• All patients randomized >28 wks (Per Protocol)
• Randomization closed when:
• All enrolled patients randomize or discontinue
28 week Requirement later dropped (Protocol Amendment)
Presentation: use 200 Relapses

7. The Problems
Long Lead times
• Up to 36 weeks before randomization
• Plus 28 weeks Minimum randomization
Ideally: Stop enrollment 64 weeks before target
#Relapses
Must account for
• Enrollment D/C (30%-70%)
• Randomized D/C (D/C Rate ≈ Relapse Rate)
• Relapse Rates vary (Higher Relapse Rate Early)
• Competing Relapses (D/C vs Relapse)
• Sensitivity to rates (Close Rates High Variability)

8. Stopping Enrollment: Issues
Too Early: Too Late:
Fewer Patients  Higher certainty of
Fewer randomized reaching Goal
 Longer wait for Patients possibly in
target #Relapse. Open Label at End
May never reach Excessive #Relapses at
target #Relapses end of study
Ethics of Randomizing
Excess # pts

9. Many Management Questions
When do we stop enrollment while being sure of
eventually getting target # Relapses?
When can we stop randomization & ensure reaching the
target?
Whats the earliest and latest we can expect to reach the target?
When will all pts be randomized >28wks?
When can the trial be halted (required # Relapses & all pts randomized >28 wks)
Estimated Randomization Rate?
Estimated Relapse Rate?
How many active patients at the end?
How well does the outcome match our assumptions
etc
etc

10. Simulation Solution
• Make a stochastic model of the trial
• Monthly:
– Base model parameters on blinded data observed to date
– Incorporate assumptions where data insufficient
– Incorporate uncertainty of parameters
– Execute 1000’s of simulations of the trial
– Compute statistics from the collection of simulated trials
– Repeat with new data

11. Advantages
Transparency
• Modeling assumptions can be:
Specified -- Graphed -- Debated
Data Driven
• New Data updates the model
• Existing Active Patients are simulated to end
• Assumptions become less important as data
accumulates

12. Vision of Output
• Simulation reports varied according to changing
team needs (how many open label patients on June 1? When will we
reach 150 Relapses? How many randomized patients at time of 200th Relapse? If we stop randomizing on May 15, how many open
lable patiestin will there be?……………………………………………….

13. Stochastic Modeling Approach
1. Make a cartoon model of a patients
progress through the trial
2. What final outcomes are possible?
3. What could happen to the patient?
4. Identify States through which a patient
passes
5. Identify Random Processes which take
patients between states

14. Stochastic Model
Discontinued
Patients (Open
Label Phase) Discontinued
Patients
Open Label (Randomized
Enrolling Phase)
Patients Patients
Randomized
Patients
Relapses
Continuous Time Markov Chain
Markov States: the Bubbles Transitions: the Arrows
Transition Probabilities change with time in state

15. States
Discontinued
)
Patients (Open Label
Phase)
Discontinued Patients
Enrolling ) Open Label
)(Randomized Phase)
Patients Patients
Randomized Patients
)
) Relapses
2 Transitory Markov States:
Open Label Phase
 Randomized Phase
3 Terminal Markov States:
Discontinued from Open Label Phase
 Discontinued from Randomized Phase
 Randomized Relapses

16. Transition Processes
Discontinued Patients
)
) (Open Label Phase)
Discontinued Patients
(Randomized Phase)
Enrolling
Patients
Open Label
Patients )
Randomized Patients
)
Relapses
)
5 Random Transition Processes:
1. Trial Enrollment (Start Open Label)
2. Discontinuation from Open Label Phase
3. Randomization (from Open Label Phase)
4. Discontinuation from Randomized Phase
5. Randomized Relapses

17. Trial Enrollment
)
Enrolling Open Label
Patients Patients
For each simulated patient, generate a random length of
time since the last patient
• Pick an enrollment rate λ (Based on history & judgment)
• Assume: #pts/mo ~ Poisson process with mean λ
• Time between patients ~ Exponential(1/λ)
Can expand enrollment model to evaluate management
options:
• Incorporate mixture of site performances
• Adding/changing sites during the trial

18. Markov Process
) Discontinuation
Continuing
)
Relapse (or
Randomization)
Probability of
transitioning from
state i to state j
between times s and t

19. Aalen-Johansen estimator of Transition Probabilities
• Aalen-Johansen estimator of the transition probability matrices
For and
# obs. Direct transitions from states h to j, visits 1 to t
# pts in state h, just prior to visit t

20. Aalen-Johansen & Kaplan-Meier
• Generalization of Kaplan-Meier Estimation
to Non-homogeneous Markov Chains
• K-M Estimators easier:
– To program (already in SAS)
– To understand (Intuitive)
– To Explain (Familiar)

21. Models
• Enrollment: Poisson Process
• Open Label Phase: Competing Risk Model
0= Still in OL Phase
1= Randomized
2= Discontinue fr OL phase
• Ramdomized Phase: Competing Risk Model
0= Still in Rand Phase
1= Manic event
2= Depressed event
3= Discontinue fr Rand phase

22. Competing Risk Model
Mutually exclusive events
(e.g., Relapse vs Discontinuation, …)
2 Approaches (Pintilie, 2006)
• Jointly distributed Random Variables
• Latent failure times
– Assume both events eventually occur
– But we only observe the first
– Use only marginal distributions
– Assuming independence (between events)
– But cannot test for independence, if only observing 1st
– Independence:
Face validity & Simplest Assumption

23. Kaplan-Meier Simulation
• Assume event are independent
• Model Each process separately using Kaplan-
Meier Estimators
• Censor on other event, current time in trial
• Simulate each event separately
• Earliest of the 2 simulated processes is taken as
simulated outcome
• Caveat: Assumes Independent processes
Intuitive, easy to understand, easy to explain

24. Open Label Transitions
Discontinued Patients
) (Open Label Phase)
2 Competing Open Label
Patients
Processes: ) Randomized Patients
 Discontinuation Randomization
1. Generate Random Discontinuation time
2. Generate Random Randomization time
3. Use the earliest event

25. Randomized Phase Processes
Discontinued Patients (Randomized
) Phase)
Randomized Patients
Relapses
)
2 Competing Processes:
 Discontinuation * * Relapse
Choose event as previously described.
• Current Open Label Patients are simulated to
randomization or discontinuation
• If simulated randomization, then simulate
Randomized Discontinuation or Relapse

26. Generic Transition Process
Q: When to make the transition? State
State
A: First: estimate random transition "A" "B"
function
1. Generate K-M Survival Functions from data
(censoring on all other events)
2. Make assumptions about Survival beyond last event

 ?


27. Simulated Patients
State State (p, t)
"A" "B" p
Q: When to make
the transition? t
A: Second: Simulate Trials
For Each Simulated Trial:
• For each simulated patient within a trial
– Pick a random p∈(0,1)
– Interpolate t from the graph, so that (p, t) is on graph

28. Simulating Active Patients
State State
q (q, s)
"A" "B"
(q*p, t)
q*p
For each simulated trial s t
• For each observed patient within state “A” for time s
– Interpolate q∈(0,1) from the graph, so that (q, s) is on graph
– Pick a random p∈(0,1)
– Interpolate t from the graph, so that (q*p, t) is on graph

29. Incorporating Parameter Uncertainty
State State
"A" "B"
For each simulated trial
q
• Pick a random quantile r∈ r
(0,1)
• Simulate all patients using
the r%-tile confidence level t
of the Kaplan-Meier Curve
Simulates: combinations of high & low estimates of Event and D/
C Survival curves

30. Limitations
Requires representative data from all phases
• K-M estimates only through last event
• Assumptions must be made about hazard rate after last
available event(s)
– If assumptions correct, point estimates should be stable while
confidence intervals narrow
• Up to date data
– Special reporting of Relapses (faxes with follow up,
monitoring)
– IVRS, EDC, monitoring reports
• Heterogeneity:
– Earliest sites may not be representative of all sites
– Procedures may change (hopefully improve) over time
– Regional differences (standards of care, patient attitudes, etc)

31. Why Not a Parametric Model?
Trial Structure:
• Events tend to occur
on visits
 granularity
 ✭ Continuous
• Visits vary in
spacing
 ✭ Discrete
• Active Tx
 mixture model
Changing Hazard
over time
• Must make & defend
simplifying
assumptions

32. Diagnostic: Does It Fit?
Survival curves of:
Observed data vs. Simulated Data
(Censored Observed, Active OL Pts, Active Rand. Pts, Entirely simulated Pts)

33. Diagnostics
Plot K-M curves for each event, time in each phase
• Review assumptions (long term behavior)
• Identify data anomalies
• Identify simulation problems

34. Example: Regional Heterogeneity
Regional modeling (Trials A & B):
Parameters varied by region more than by trial
– Estimate parameters within regions
– Simulate patients with Trial and Region
– Summarize results by Trial
In addition to simulations which ignored region
Survival curves
followed 2
patterns by
region & trial

35. Reporting the Simulations
For each simulated Trial:
• Sort Patient Events by occurrence date (enrollment,
randomization, relapse, etc)
For each scenario
• Summarize over Event records which fit scenario
Examples:
• Summarize over all patients enrolled before a potential
enrollment cutoff date.
• … over all patients randomized before a cutoff date
• Summarize with and without a subset of sites

36. Changing Questions
• Early in Trial
– Are the protocol assumptions accurate?
– When to stop enrollment?
– Expected # patients (enrolled, randomized, etc)
Identify problems & Evaluate fixes
• Mid-Trial
– When to stop randomizing patients?
– Are the revised assumptions accurate?
– Were changes effective?
• Late Trial
– When will the last Relapse occur?
– How many patients will be active in various phases?
Plan for Closeout & Database lock

37. Early Trial
For each Enrollment Cutoff Date
• Summarize each trial for all patients enrolled
before that date
• Compute statistics over simulated trials
Some trial outcomes:
• Dates of: last Relapse; All patients randomized>28 week;
PP Completion (>target & >28 weeks)
• Event & patient counts at each of above milestones
• % of Simulations with ≥200, 190, 180,… Relapses at milestones
• # active patients (open label or randomized) at given dates &
milestones
Pick a cutoff date accordingly (e.g., minimize resource with
least risk of running late)

38. Trial Completion (1)
≥227 Relapses & All Active Patients Randomized 28wks
Earliest
Completion:
• 75% Certainty:
2 August Cutoff for
Nov 2006
Completion
• 90% Certainty:
1 Sep Cutoff for
Dec 2006
Completion

39. Trial Completion (2)
≥227 Relapses Events & All Active Patients Randomized 28wks
Earliest
Completion:
• 75% Certainty:
~1775 Enrolled for
Nov 2006
Completion
• 90% Certainty:
~1850 Enrolled for
Dec 2006
Completion

40. 29 Oct prediction from: 528 Enrolled, 272 Rand., & 58 Relapses
On 26 June: ~1950 Enrolled, 620 Randomized, 200 Relapses

41. Highly uncertain: 3 Relapses, 2 Randomized D/C, 73 Randomized

43. 1 Month Later: 16 Relapses, 7 Rand.D/Cs, 182 Randomized

44. Slowed Enrollment: 29 Relapses, 22 Rand. D/C, 272 Rand

45. Example: Will a trial end?
Study E:
• Endpoint: 300 type 1, 300 type 2 events
• Slower & Fewer than expected
• Simulation predicted:
– 10% chance of 300 of each
– 87% chance of 600 total
• Interim Analysis
– Supported by simulation
– 300 total expected April 25

46. Mid-Trial Output
For each Randomization Cutoff Date
• Summarize each trial for all patients
randomized before that date
• Compute statistics over simulated trials
• Generates same statistics per trial
• Summarize for each randomization cutoff date
Essentially, replace “enrollment” with
“randomization” & execute as before

47. • Conclusion: nothing to be gained by
randomizing beyond mid June

48. Late-Trial Output
Refine estimates of last Relapse, etc.
For Milestones & Calendar Dates
• Estimate #patients in each stage (e.g., how many
patients will be active at the end?)
Caveats:
• Corrected (or just collected) data may change
estimates
• Old, unreported Relapses may be discovered
Useful:
• Predict time between milestone to end
• Add prediction to best guess milestone

49. Bottom line: How accurate?
Not bad:
• Actual Date of 200th Relapse covered by
predicted 80% C.I.
• Width of C.I.s narrowed over time

53. Value Added
Early Refinement of Protocol Assumptions
• Protocol: 50% randomized, 30% Relapse rates
• Trial A: 33%, 37% Trial B: 55%, 41%
Early Identification of problems
Quick Response to problems
• Changed procedures to improve retention in Trial A
• Added sites to Trial B after delays in starting up sites
Better allocation of resources

54. Trials C, D
Mid trial: Regulators requested analysis of late
Relapses
• Enrollment had already ended for Trial C
– Enough patients to reach 88 late Relapses?
• Enrollment was still ongoing for Trial D
– Extend enrollment how long? Add sites?
• How would this affect time lines?

55. Data problem:
Unreported Relapses

57. Dirty Data Problems
• Some known Relapses are not usable due to missing data
(e.g., unknown randomization date)
• Corrected (or lately collected) data may change estimates
• Old, unreported Relapses may be discovered
• Data may be collected or corrected irregularly
– Separate data sources (e.g., Relapse log + IVRS)
– Drift & shift over time
– End of trial data clean up
Solutions:
• Estimate time between milestones
– Anchor to a known, early milestone
Future solutions:
• Estimate missing data effects
– Use time between Relapse occurrence and Relapse reporting
– Estimate number of missing Relapses from times in past

58. Some Feedback
• “… the simulations are very valuable
and the only way we have to plan our
timelines. As it has turned out, your
simulations seems to be pretty accurate
as we have increased the mood event
rate significantly … as predicted...”
• ... We would have been guessing and
spinning our wheels without them.”
• Could you simulate trials xyz & uvw?

59. Extra Slides

60. Diagnostics: Cohort Analysis
Cohorts:
• By month enrolled
• By month Randomized
Calculate randomization & Relapse rates
 Easy to understand
 Multiple Estimates which must be reconciled
 Doesn’t Provide Time to Relapses
 Useful reality check on Simulation
Point Estimates insufficient: need C.I.s

61. Cohorts by Randomization Month
Actual counts of patients by status by month

62. Cohorts by Month Randomized (Cumulative Count)
Height is # patients randomized in or before month

63. Cohorts by Month Randomized (Cumulative Percentage)
• >40% randomized eventually have Relapse
 <N/0.40 randomized to get N

64. Open Label Cohorts
(Current Status)

65. Open Label Cohorts (Cumulative Statuses)
• ~ 64% eventually Randomize & >40% Relapse Rate
•  # open label pt < N/(0.64*0.40)

66. Solutions?
Crude Relapse Rates of all Patients in Phase
Mixture of patients:
• Relapse & D/C rates change with exposure
• Mixture of Pt. Exposures changes with time
Cohorts: Track Relapses over Time
 Easy to understand
 Multiple Estimates which must be reconciled
 Doesn’t Provide Time to Relapses
 Useful reality check on Simulation
Point Estimates insufficient: need C.I.s

67. Clinical Trial Management
Planning Trials (future)
• Is a trial feasible?
• Sensitivity to assumptions?
• Costs: # Pts, # Pt-mos, #visits, #Sites, #Site-mos
Trial Execution (current)
• Anticipate delays
• No information on outcome
• Could be added to simulation
Program Planning (future)
• Replace “Trial Phases” with “Toll Gates”
• Enhance modeling of Trial Enrollment process

68. Example: Adding Site
Sites Discontinued
Patients
Randomized
Patients
New Relapses
Sites
• Drop OL Phase, expand enrollment process
• Simulate time to start up new site, pts/mo at a new
site, etc
• Report by #Additional Sites instead of cutoff dates