By: Amy Ou
During a weekend off at my parents’ home, the subject of this chronic cough that I had developed for the entirety of the winter season came up. My mother, noticeably more concerned about it than I, asked: “Did you get a flu shot? Did you get your cough after you got your flu shot? You know this happened when you were little, right? I just don’t know about those flu shots, I think they have some bad side effects. Your dad and I were adamant about not getting flu shots this year.”
A nervous mixture of emotions churned in my stomach. Could it be? Could my highly-educated scientist/engineer parents be against vaccines? “We were talking to the retired couple next-door about it too. They agreed. They didn’t get it either.” Agck!
Anti-vaccine is quite the dirty word in the medical community. When the unvaccinated baby with high fevers rolled into the emergency room on my pediatrics rotation, the air in the ER was heavy with judgment. “This. This is what happens when you don’t vaccinate your kids.” Could it be that I myself was living in a community of anti-vaxxers? One that included my own parents? How could they, as intelligent as they are, not understand the overwhelming proof?
Hierarchies of Evidence
As I rambled to my parents about the insurmountable evidence of the safety of vaccines and the role of the FDA in ensuring that trials of safety and efficacy are appropriately conducted before mass vaccine campaigns are launched, I could tell that I was not making even a dent in their opinion. When I was a kid, my consecutive string of healthy winters was ruined by one particularly devilish cold. Unfortunately, it happened to occur the same year I got a flu shot for the first time. My parents’ brains kicked into gear and decided the two were related.
In 1979, the Canadian Task Force on Periodic Health Examination came up with the idea of ranking evidence by quality, with the pinnacle of the pyramid being evidence obtained from “at least one properly randomized controlled trial,” followed by “well-designed cohort or case-control analytic studies” [1]. At the bottom of the barrel was “opinions of respected authorities” [1]. The idea of hierarchies of evidence was adapted and popularized by many subsequent scientists.
The hierarchy seems intuitive after learning about it, but medical school was actually the first place I had heard of it. Concepts such as the placebo effect or association-versus-causation were ingrained in my head, but someone had to teach them to me first. It seemed obvious to me that the evidence from systematic reviews of double-blinded, randomized controlled trials of the influenza vaccine overruled my mother’s observation of one subject, but it’s not like my mother had sat through a whole course in clinical epidemiology like I had.
Arbiters of Evidence
Back in the emergency room, the nurse threaded a Foley catheter into the screaming baby. His mother was on the verge of tears, mortified at what was happening to her precious son. Did she wish this on her child when she made the decision to withhold vaccinations? Not likely.
It turns out the anti-vaccine movement was not born of the malicious intent of some layperson to harm whole generations of children. Vaccines have been controversial since 1796, when Edward Jenner first inoculated an eight-year-old boy with pus from a milkmaid’s cowpox lesion in the hope of preventing smallpox [2]. Historically, concern about the use of vaccines has been voiced by doctors and citizens alike, sometimes for good reason. For example, in the spring of 1955 a particular lot of polio vaccine actually contained active wild-type polio virus, causing 200 children to contract the disease [2].
The most recent surge in the anti-vaccine movement came from an article about an association between autism and the measles-mumps-rubella (MMR) vaccine written by a doctor. Yes, a retracted, but very Google-able and citable article. Andrew Wakefield, the author of the article, was eventually found to have falsified data. He has been repeatedly discredited, but to this day he maintains a substantial following of people who don’t believe in mainstream medicine [3]. And why not be skeptical? Germ theory was once out of the mainstream and it turned out to be correct [4].
We ask our patients to place their trust in us, to serve as arbiters of this vast quantity of evidence. But why? Simply because we went to medical school?
What We Want To Report
In searching for evidence, randomized controlled trials (RCTs) are often considered the gold standard in evaluating healthcare interventions. The CONSORT Group, a self-described “international and eclectic group, comprising trialists, methodologists and medical journal editors,” publish a statement, updated every few years, with guidelines for the reporting of RCTs [5]. They recognize that RCTs are only as good as their design, and provide a checklist of things that make for a high-quality trial. Without these features, RCT findings may be no better than those of observational trials.
In a review of all RCTs indexed in PubMed in December 2006, surprisingly few studies followed those guidelines [6]. Out of 616 studies identified, 47% didn’t define the primary outcome, 66% failed to report the method of random sequence generation, and 75% didn’t report how allocation was concealed (and of the 25% that did report it, 50% used envelopes, an extremely fallible form of concealment). Forty-four percent of the papers they looked at were published in journals that endorse CONSORT. And this was an improvement over the results from December 2000.
What We Want To Believe
If RCTs are the gold standard, systematic reviews and meta-analyses are the platinum standard. They similarly have a group, PRISMA (previously known as QUOROM), which publishes statements on what to include in systematic reviews and meta-analyses [7]. They too have found inconsistencies in the reporting of systematic reviews, with one-third of studies not revealing funding sources, one-third not even reporting what electronic databases were searched, and only 23% assessing for publication bias, a very insidious problem in scientific research [8].
A 1991 paper in the Lancet gives a demonstration of this problem of publication bias. A retrospective survey was conducted of research projects approved by the Central Oxford Research Ethics Committee between 1984 and 1987 [9]. The authors identified 285 studies whose data had been analyzed at the time of their investigation. Of those, 73% had been published or presented. Among the published/presented studies, 68% had statistically significant results, compared to only 29% of those neither published nor presented. A study that had been published or presented had an odds ratio of 4.54 of being statistically significant over being null. There is a whole sector of evidence that meta-analyses are not picking up.
One test that attempts to evaluate the existence of publication bias in meta-analyses is the funnel plot, which graphs trial sample size against effect size, with the expected result being that smaller studies will be scattered widely at the bottom of the graph with a narrowing spread as the studies get larger [10]. Without bias, the graph is expected to be symmetrical. A review of reviews from the Cochrane Database of Systematic Reviews demonstrated that, of reviews that included enough trials to be analyzed by the funnel method, a whopping 48% suggested the presence of publication bias [11].
We place a lot of trust in our evidence, but how do we know we are not just selectively choosing evidence to support what we want to believe?
Observations
My understanding of evidence-based medicine in shambles, I return to my parents’ refusal of the flu vaccine, and the ER mother’s refusal of all childhood vaccines. My parents, having made it to April, escaped the illness we tried to protect them from. The ER baby’s mother was perhaps not so lucky, although I never found out the child’s final diagnosis. They both took a risk based on their own judgment and their own internal weighing of the odds of morbidity. Were they presented with all the evidence? Does the evidence even mean anything?
No experiment is capable of producing absolute truth. Statistical testing is a game of chance that’s subject to interpretation. When we tell patients to get vaccinated, we are not dispensing fact so much as probability. When they refuse, they are simply weighing their own impression of the odds.
I’m certainly not throwing my evidence out the window, but I’m carrying with me a big block of salt. As far as my parents go, a large Cochrane review (with only a brief line on publication bias) was performed in 2014 of 116 data sets comparing the influenza vaccine to placebo or no intervention [12]. As it turns out, in healthy adults, 71 people would need to be vaccinated to prevent one case of influenza, with no effect on working days lost or hospitalization. A separate 2010 Cochrane review of the influenza vaccine in the elderly failed to reach any clear conclusions [13]. Was my cough from the vaccine? Probably not. Were my parents and their neighbors taking significant risks by refusing the vaccine? Maybe not as much as I thought they were.
As for the ER baby, if the mother’s choice to not vaccinate her baby came from the general fear that the MMR vaccine causes autism, a 2012 Cochrane review (with no mention of publication bias) shows no such evidence [14]. It does, however, point out that certain strains of the vaccine (strains still in use in some countries) have been shown to have an association with aseptic meningitis, and that there is an increased risk of febrile seizures in children following general receipt of the vaccine. Does the very real risk of hospitalization or death from measles, mumps, and rubella outweigh the minute risk of the side effects? I strongly believe so. And is there a civic duty to protect ourselves and our fellow citizens from these contagious diseases? Definitely. But are parents within reason to question our insistence to vaccinate? Maybe more so than we think.
To convince my parents or anybody else’s parents, I can’t just shove my beliefs down their throats. I have to consider my evidence more critically, and more importantly, to give some credence to the beliefs my patients hold, in order to facilitate learning on both ends.
Amy Ou is a medical student at NYU Langone School of Medicine Class of 2017
Peer Reviewed by Michael Tanner, MD, Executive Editor, Clinical Correlations
References
- The periodic health examination. Canadian Task Force on the Periodic Health Examination. Can Med Assoc J. 1979;121(9):1193-1254.
- Stern AM, Markel H. The history of vaccines and immunization: familiar patterns, new challenges. Health Aff (Millwood). 2005;24(3):611-621.
- Deer B. How the case against the MMR vaccine was fixed. BMJ. 2011;342:c5347.
- Britt LD. The death of an American President and the birth of an organization: the American Surgical Association and its legacy. Ann Surg. 2013;258(3):377-384.
- Schulz KF, Altman DG, Moher D, Group C. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332.
- Hopewell S, Dutton S, Yu LM, Chan AW, Altman DG. The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ. 2010;340:c723.
- Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264-269,W64.
- Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78.
- Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet. 1991;337(8746):867-872.
- Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629-634.
- Sutton AJ, Duval SJ, Tweedie RL, Abrams KR, Jones DR. Empirical assessment of effect of publication bias on meta-analyses. BMJ. 2000;320(7249):1574-1577.
- Jefferson T, Di Pietrantonj C, Rivetti A, Bawazeer GA, Al-Ansary LA, Ferroni E. Vaccines for preventing influenza in healthy adults. Cochrane Database Syst Rev. 2014;3:CD001269.
- Jefferson T, Di Pietrantonj C, Al-Ansary LA, Ferroni E, Thorning S, Thomas RE. Vaccines for preventing influenza in the elderly. Cochrane Database Syst Rev. 2010(2):CD004876.
- Demicheli V, Rivetti A, Debalini MG, Di Pietrantonj C. Vaccines for measles, mumps and rubella in children. Cochrane Database Syst Rev. 2012;2:CD004407.