Presentation: Jodi Schneider and Sally Jackson, “Innovations in Reasoning About Health: The Case of the Randomized Clinical Trial.” 9th International Conference on Argumentation, International Society for the Society of Argumentation, Amsterdam, Netherlands, July 5, 2018
Abstract: Field-dependence in argumentation comes about through forms of inference invented by specialized fields. In recent work we introduced the concept of a "warranting device": (1) an inference license (2) invented for a specialized argumentative purpose and (3) backed by institutional, procedural, and material assurances of the dependability of conclusions generated by the device. Once established, fields employ such devices across many situations without further defense, even as the devices develop in response to newly-noticed problems.
Many new warranting devices have appeared over the past century to solve problems in reasoning about health and medicine, replacing and obsolescing earlier forms of medical reasoning. One such device is the Randomized Controlled Trial. This case study traces its historical evolution and discusses some current movements toward competing device types.
💕SONAM KUMAR💕Premium Call Girls Jaipur ↘️9257276172 ↙️One Night Stand With Lo...
Innovations in reasoning about health: the case of the Randomized Clinical Trial--ISSA 2018-07-05
1. Innovations in Reasoning About
Health: The Case of the
Randomized Clinical Trial
Jodi Schneider, Sally Jackson
University of Illinois at Urbana-Champaign
United States of America
ISSA, Amsterdam, 2018-07-05
2. Source for argument: Bada et al. (2015). Morphine versus clonidine for neonatal abstinence syndrome.
Claim: Clonidine may be a
favorable alternative to
morphine as a single-drug
therapy for NAS.
Data: Infants with NAS given
clonidine improved more
rapidly than those given
morphine and had equal
outcomes on other measures.
3. Claim: Clonidine may be a
favorable alternative to
morphine as a single-drug
therapy for NAS.
Data: Infants with NAS given
clonidine improved more
rapidly than those given
morphine and had equal
outcomes on other measures.
Source for argument: Bada et al. (2015). Morphine versus clonidine for neonatal abstinence syndrome.
Warrant: Randomized Clinical Trial
4. “Innovations” in reasoning and arguing
Our research project focuses on fields where innovation is occurring
rapidly (e.g., health):
• Search for novel ways of drawing and defending conclusions (new
warrants)
• Examination of how these novel warrants are established (and
sometimes dis-established) within fields
5. Warrants
• Warrants are inference rules governing how conclusions are drawn, not
statements from which conclusions are drawn.
• “The claim is not presented as following from the warrant; rather it is presented as following
from the grounds in accordance with the warrant.” (Hitchcock 2003, p. 71)
• Warrants do not typically appear in the argument.
• “Arguments instance inference rules, rather than include them as elements…” (Freeman
2011, p. 88; Hitchcock 2003 agrees)
• Warrants generate conclusions as well as justify them
• In law, relevant statute allows for “findings” of guilt or innocence (Toulmin, 1958, p. 99)
• In science, “standard equations” generate predictions (Toulmin, 1958, p. 121)
• They can be technical in multiple senses:
• Expert communities devise them as tools for conclusion-drawing
• They depend on “built” things (Jackson & Schneider 2018)
6. Warrants
• Warrants are inference rules governing how conclusions are drawn, not
statements from which conclusions are drawn.
• “The claim is not presented as following from the warrant; rather it is presented as following
from the grounds in accordance with the warrant.” (Hitchcock 2003, p. 71)
• Warrants do not typically appear in the argument.
• “Arguments instance inference rules, rather than include them as elements…” (Freeman
2011, p. 88; Hitchcock 2003 agrees)
• Warrants generate conclusions as well as justify them
• In law, relevant statute allows for “findings” of guilt or innocence (Toulmin, 1958, p. 99)
• In science, ”standard equations” generate predictions (Toulmin, 1958, p. 121)
• They can be technical in multiple senses:
• Expert communities devise them as tools for conclusion-drawing
• They depend on “built” things (Jackson & Schneider 2018)
7. Data Claim
Warranting rule
Dependable because
backed by:
Material
assurances
Procedural
assurances
Institutional
assurances
Source of figure: Jackson & Schneider 2018. Cochrane Review as a "Warranting Device" for Reasoning About Health.
Warranting Devices
8. Warranting Devices - Definition
(1) an inference license in Toulmin’s sense
(2) invented for a specialized argumentative purpose and
(3) backed by institutional, procedural, and material components that
provide assurances of the dependability of conclusions generated by
the device
Source: Jackson & Schneider 2018. Cochrane Review as a "Warranting Device" for Reasoning About Health.
9. Claim: Clonidine may be a
favorable alternative to
morphine as a single-drug
therapy for NAS.
Data: Infants with NAS given
clonidine improved more
rapidly than those given
morphine and had equal
outcomes on other measures.
Source for argument: Bada et al. (2015). Morphine versus clonidine for neonatal abstinence syndrome.
Warrant: Randomized Clinical Trial
material resources procedures institutions
10. patients & providers
recruited
Treatment B
Treatment A measurements
measurements
random allocation
blinding, other controls
monitoring
simple randomized clinical trial
which group did better?
statistical
comparison
11. simple randomized clinical trial
patients & providers
recruited
Treatment B
Treatment A measurements
measurements
random allocation
blinding, other controls
monitoring
which group did better?
statistical
comparison
protocol approved
15. Conclusion: Use CCTs to form beliefs
about medical treatments. [P27]
Goal: advancement of
knowledge on which good
treatment of patients depends
at population level [P27]
Circumstances: problems in prevailing
methods for forming beliefs about
treatment
• variability in patient response to
treatment [P6,7,9]
• inability to distinguish between small
effects and absent effects [P25]
• inability to identify cause [P9,22]
• inability to generalize [P9,10]
• bias in individual clinician judgment
[P13,22] and in patient judgment
[P22]
• a ‘literature’ composed of conflicting
true findings [P6]
Values: medical ethics (primacy
of patient welfare) [P3],
expertise [P1]
Means-End: CCTs correct weaknesses in
individual clinical judgments
• successful in Great Britain
[P21,22,25] and spreading [P14]
• trial design eliminates various
problems
• random allocation of patients
[P13]
• blinding [P21]
• exact specification of treatment
schedule [P17,18]
• standardized measurement
[P19]
• independent assessors [P21,23]
Reconstruction of Bradford Hill’s Warrant-Establishing Argument, adapted from the practical argument scheme presented in
I. Fairclough 2016: A Dialectical Profile for the Evaluation of Practical Arguments.
16. Circumstances: problems in prevailing methods
for forming beliefs about treatment
Means-End: CCTs correct weaknesses in
individual clinical judgments
Conclusion: Use CCTs to form beliefs
about medical treatments. [P27]
Imputed counterclaim: Reject CCTs
[P2]
Goal: advancement of knowledge on which good
treatment of patients depends at population level
Because the rationale for the action is faulty:
• primary goal must be best care for the
individual patient [P3]
• don’t need CCTs to know what works [P24,25]
• statisticians lack medical knowledge, count the
wrong things, care only about the group
[P1,3,4]
• measures can’t replace clinical judgment [P5]
• humans too variable to allow CCTs to generate
medical wisdom [P6]
Because the action itself is:
• impractical (large numbers [P7-11], co-
ordination [P18])
• unethical to experiment on patients [P26]
• disregard of primary clinical responsibility
• set treatment schedule not best [P17,18]
• wrong to reduce patients to numbers
• challenges authority of clinician [P1]
Adapted from the deliberation scheme presented in
I. Fairclough 2016: A Dialectical Profile for the Evaluation
of Practical Arguments.
17. Black boxing
• Warranting devices, once established within a field, may be employed
without further defense across many situations.
• Their initial establishment within a domain may be followed by an
open-ended process of reconfiguration in response to newly-noticed
problems.
• “Black box arguments are a constantly evolving technology for coming
to conclusions and making these conclusions broadly acceptable.”
(Jackson, 2008)
18. “Beyond RCT”
• Although RCT remains the gold standard for causal reasoning today,
new competitors can arise at any time.
19. A Summary of Our Argument
• Real argumentation serves practical purposes.
• Innovation in reasoning and arguing is driven by these practical
purposes.
• New inference rules have to be argued into existence; they can be
argued out just as well.
• Change in the stock of usable warrants is normal. New things are
invented and old things are abandoned.
20.
21. patients & providers
recruited
Treatment B
Treatment A measurements
measurements
random allocation
measurements
measurements
blinding, other controls
monitoring
protocol approved
randomized clinical trial with pretest and posttest measurements
did Group A improve?
did Group B improve?
which group did better?
22. Conclusion: Apply rule R
for belief formation in
domain D.
Goal: To form correct
beliefs in domain D.
Circumstances: Various
ways incorrect beliefs may
form in domain D.
Values: Professional ethics
in domain D, reason
domain D is studied, etc.
Means-End: Reasons for
believing that rule R
generates correct beliefs
and/or avoids incorrect
beliefs.
Editor's Notes
This is part of our ongoing project looking at how specialized fields are extending their repertoires of inference rules to improve their ability to draw and defend expert conclusions. The very fact that we innovate in reasoning all the time is a phenomenon that needs exploration.
Here’s the kind of argument we want to talk about.
This example is from a small pilot study that was used to build the rationale for a longitudinal multi-site trial that is not yet completed. (Protocol at https://clinicaltrials.gov/ct2/show/NCT03396588). But this kind of argument has become extremely common in reasoning about health and medicine.
Our study has to do with what warrants the move between the data and the claim.
In this example, the conclusion is warranted by the fact that these data were produced through a randomized clinical trial, considered the gold standard for evidence of treatment effects. That is, the logic of a randomized clinical trial warrants the conclusion. An important fact about this warrant is that it was invented not very long ago, and people were still arguing about whether to apply it in medical science as recently as the 1950’s.
------
Bada HS, Sithisarn T, Gibson J, Garlitz K, Caldwell R, Capilouto G, Li Y, Leggas M, Breheny P. Morphine versus clonidine for neonatal abstinence syndrome. Pediatrics. 2015 Feb;135(2):e383-91. doi: 10.1542/peds.2014-2377.
Argumentation theory needs to pay more attention to newly invented inference rules like this one. Any one of these innovations in reasoning has the same status as an already-cataloged scheme. That means we need scheme definitions for each one, we need to enumerate the critical questions for each one, and we need to show how people work with each one.
What we’ve been doing is searching out cases of new inventions and examining them.
We can think of these as new scheme types, but it’s convenient to think of them novel warrants in Toulmin’s sense, especially since we assume that as each one is proposed, it requires a warrant-establishing argument to promote its use.
Toulmin’s warrants still aren’t thought of the same way by everyone, but we think that a fairly mainstream understanding of warrants includes these three characteristics.
To the fairly mainstream understanding of warrants, we add something that we expect to have to defend: Sally and I have suggested that some warrants are technical objects that expert communities invent, apply, and evolve. And we’ve introduced the idea that a warrant may be bundled with technical components to produce what we call a warranting device.
What appears to a practitioner as a method for drawing conclusions involves procedural components, executed by an individual practitioner, but also other components that are delegated to institutions and to other expert communities. Delegated components can allow an arguer to build on “know-how” that an expert community accumulates and maintains over time.
This figure is from our recent paper in Argumentation paper, where we analyzed the Cochrane Review as a warranting device. Cochrane Reviews synthesize all available evidence about a health-related question. They allow inference from research literature to the best-justified treatment guidelines.
The three categories of assurance represent the distribution of responsibility among contributors to the growth of knowledge in the area.
We believe that these devices undergo significant technical improvement over time.
Here’s how we defined warranting device in that article.
Today we want to view Randomized Clinical Trial from this perspective, as an inference device of relatively recent invention. Let’s look again at an example of how Randomized Clinical Trial can be used as a warrant.
What we’re saying is that the Randomized Clinical Trial is another warranting device, an inference rule that comes packaged with specific assurances that its conclusions will be acceptable to other experts.
So we've seen what the argument looks like at its top level, but what does the experiment itself look like? The simplest acceptable design for a randomized clinical trial involves randomly dividing a pool of patients into two groups, giving the two groups two different treatments, and taking measurements afterwards in order to try to determine which group did better. At the end, the measurements from the two groups are analyzed statistically, which in current practice means computing group averages and testing them for statistical significance.
A moment ago we showed warranting devices as an inference rule with some special kinds of (non-propositional) backing: some institutional, some procedural, and some material. This diagram helps to clarify what we mean by each type of assurance. Procedural assurances that back the inference from an RCT include random allocation, blinding, well-defined treatment specifications, and consistent measurement. Many material resources are needed to conduct such an experiment, but the ones that make a difference to the dependability of the conclusion are things like access to appropriate patients and providers and statistical tools that are, at this point in time, implemented in standard computer packages.
Institutional assurances also affect the credibility of the result. While the logic of the RCT is quite easy to understand, conducting an RCT has become a highly regulated affair that makes it impossible for anyone acting outside complex institutional environments. For example, no one can enroll patients in an experiment of this kind without getting a protocol approved at multiple levels. For a decade or more, researchers have had to register their trials before beginning to recruit, and gradually it has become common for protocols to be published even before any results have been obtained (so that the community can know what things are being tried).
There’s a lot about this that would be counterintuitive from the clinician’s point of view: randomly allocate patients; blind the provider to the treatment; decide which group did better based on pre-defined measures. That’s why the randomized clinical trial did not spring fully formed into existence. Arguments had to be given for why the randomized clinical trial is an appropriate way of drawing conclusions about health.
Randomized Clinical Trial incorporates numerous components that are separate inventions in their own right. These inventions were motivated by countering problems in practice.
Together, these inventions make invention of the RCT possible.
===
Blinding – Wikipedia
Random allocation – Peirce and Jastrow, 1883-1884. Ian Hacking, "Telepathy: Origins of Randomization in Experimental Design," Isis 79, no. 3 (Sep., 1988): 427-451. https://doi.org/10.1086/354775
And The controlled clinical trial turns 100 years: Fibiger’s trial of serum treatment of diphtheria
Placebos – Meldrum p 751
Random allocation, using subgroups (1934) – Therapeutic Trials Committee of the Research Council on the serum treat- ment of lobar pneumonia (Lancet, 1934, 1, 290), cited in Bradford Hill’s Principles of Medical Statistics.
Published RCTs in Medicine – patulin trial on the common cold per Randomised controlled clinical trials
statistical significance testing – Wikipedia
confidence intervals – Wikipedia
Outcomes - Outcomes Research in Oncology: History, Conceptual Framework, and Trends in the Literature
Registration of trials - Clinical trial registration: a statement from the International Committee of Medical Journal Editors [Editorial]., Ann Intern Med, 2004, vol. 141 (pg. 477-8)
1966: NIH Office for Protection of Research Subjects created. Policies call for IRBs https://www.ctsi.ucla.edu/education/files/training/docs/research-ethics-wenger.pdf
The RCT has continued to evolve, with new practices and requirements bundled into its performance.
It took work for RCT to become established. One of the people doing that work was Austin Bradford Hill; we analyzed the published report of a speech he gave at Harvard Medical School.
So this is a part of our data, a source of insight into how the device became established as the "gold standard" for reasoning about medical treatments. Our method: we went paragraph by paragraph, searching for argumentatively relevant content. Most of the content fit well into a practical reasoning scheme such as the one offered by Isabella and Donald Fairclough. Although it was a single long-form argument, we found it necessary to import a dialectical framework, based on what Austin Bradford Hill treated as major counterarguments against his position.
In the Faircloughs’ work on practical reasoning in political discourse, the conclusion is always an action claim, such as a claim about what policy should be adopted. We’ve put the conclusion in a directive form, to highlight that until people start taking this action, they still haven’t agreed to it. Bradford Hill used the term “cooperative clinical trial” or CCT, but that phrase has now dropped out of use, and now we would refer to this as an RCT.
In addition to presenting the case for clinical trials (left), Bradford Hill also introduced and countered the case against clinical trials. Fairclough points out that in deliberative discussion a practical argument can come under attack both for the adequacy of the rationale offered for the action and for the acceptability of the action considered on its own. These aren’t equivalent attacks, but they can be completely separate bases for rejecting the proposed action. Bradford Hill alludes to a number of such attacks, possibly coming from multiple different attackers, and provides rebuttals to each one.
What we see in Bradford Hill is what it takes to establish the warrant prior to its use. By contrast, the argument that we opened with used this warrant without explicit defense, as most conclusions justified by Randomized Clinical Trials do.
Once Bradford Hill and others like him have explained how random allocation strengthens the conclusion, that need never be mentioned again. All that is needed is a mention that randomized allocation occurred in such-and-such a way.
Establishing a warrant means getting to this point where it can be used to defend individual conclusions without explicit defense. All of the backing elements are “boxed up” for repeated taken-for-granted use. Latour has described this process of black boxing generally, not with specific reference to inference rules, but with attention to the marked difference between the period in which the black box is being assembled and the period in which the black box is used by practitioners who feel no need to inspect its inner workings.
Once a warrant is established convincing and coherent way of drawing a conclusion, we may find ways in which it can go wrong, in which case we can revisit conclusions drawn from it in the past. So, black boxes can be re-opened, but they aren’t kept open all the time.
We’re currently examining the rise of n-of-1 experiments, which evaluate effects at the individual level, and aggregate effects rather than aggregating outcomes.
One reason for re-opening a black box is that the assumptions that are concealed inside become controversial and have to be excavated. One assumption buried inside the RCT is that our goal is to find the best treatment that can be given across the board. In the past few decades it has become clear that in many cases there is no single best treatment, bolstering the critique that the “primary goal must be best care for the individual patient” since the RCT cannot tell us the best treatment for an individual patient.
Rise of N-of-1 experiments and their aggregation.
Conducted on one individual person, but aggregated.
Learn from the distribution
Set up experimentally to create subgroups “at a distance”, facilitated by technology, often mobile technology
We are starting to see warrant-establishing arguments for claims based on n-of-1 data
Material components in Trialist: iPhone, Android, software
Expecting individuals to reliably follow the protocol
Change in circumstances: “ridiculously easy organizing” – Clay Shirky
In real argumentation, people are far less interested in whether their inferences are valid than in whether their inferences serve practical purposes (such as providing proper health care).
These practical purposes drive innovation in reasoning and argument: looking at warrant-establishing arguments through the lens of practical reasoning exposes why people invent new inference rules—what we want to do with what we can conclude from a given rule.
Innovation can be iterative, with a rule ‘boxed’ as it achieves stability or ‘unboxed’ as it encounters new challenges: a new inference rule can be argued into existence and argued out just as well.
How the rule develops depends on what people have noticed as vulnerabilities and objections.
The rule can develop different ways in different places (e.g., toward being able to generalize vs toward being able to address the particular case).
Rule emergence is highly sensitive to technical capabilities (from availability of inferential statistics in the rise of RCT to availability of online coordinating platforms and Big Data methods in the rise of N-of-1 experiments).
the design can be elaborated in all kinds of ways, including through the addition of pretest measurements. If these pretest measures are the same as the posttest measures, this design allows the researcher to evaluate how much each group changed, if at all, as well as to assess which group changed more.
While the logic of the RCT is quite easy to understand, conducting an RCT has become a highly regulated affair that makes it impossible for anyone acting outside complex institutional environments. For example, no one can enroll patients in an experiment of this kind without getting a protocol approved at multiple levels. For a decade or more, researchers have had to register their trials before beginning to recruit, and gradually it has become common for protocols to be published even before any results have been obtained (so the community can know what things are being tried).
Generalized form of a practical argument in support of a new inference rule
One thing to expect when you go looking for a warrant-establishing argument
We’ve analyzed two cases in detail, Cochrane Review (June 2018, Argumentation 32: 241-272.) and Randomized Controlled Trial
Sally Jackson will present a third case in the afternoon about modeling natural phenomena as computations.