I’ve been researching the effectiveness of flu vaccines lately, since everybody keeps asking me if I’m going to get one. I sometimes ask, “Are they effective?” and the response is always “Well, I got one last year and I didn’t get the flu.” After reading “How to read articles about health and healthcare”, I began to wonder if this flu vaccination thing smells right.
We hear endless news stories quoting doctors:
State Health Commissioner Dr. Karen Remley, a Virginia pediatrician, recommends the shot as basic prevention. She’s concerned about what it can cause.
“Flu is one of the leading causes of death in our country,” she says. “Over 36,000 people die every year in our country and it’s preventable by getting the flu shot.”
([cbn.com])[1]
We could really save 36,000 people’s lives if they only got the flu shot? That’s remarkable. What worries me when I hear news organizations quoting people who aren’t experts in the research they’re purporting to cover in their “story” is that everybody buys into it without thinking. “Look at me, I’m a trained physician with a white coat and stethoscope around my neck—I went to school and you can trust me!”
I worry in the back of my mind that the bulk of “science” out there has come from bad scientists. Odds are, it has. Odds are that most of the research out there was done by mediocre scientists or scientists who are looking to get published, and so they’ll do research trying to fit the data to existing studies, because that’s an easy way to get published: find existing research, copy it as best you can making sure the results match, submit to journals. “We can confirm that flu vaccination cuts the risk of death to those who received it in half, compared to those who didn’t.”
This reminds me of another recent article that turned the “lactic acid in muscles is bad for you” science upside down. True or false: When you work out, lactic acid builds up in your muscles. If too much builds up, you start getting fatigued.
It used to be true, but now it’s false, it turns out. Lactic acid is fuel your muscles use to create more energy. Who knew? (nytimes.com)
At any rate, the flu thing began to bother me in the same way.
The first thing I did was looked at studies or meta-studies of flu vaccine effectiveness: what is the measured benefit of flu vaccination in the real-world? We all know that polio and other (non-morphing) diseases have been virtually eradicated because of vaccination. This should work for flu, too, right?
Not entirely. The plain language summary for a 2007 metastudy concluded:
There is not enough evidence to decide whether routine vaccination to prevent influenza in healthy adults is effective.
The review of trials found vaccinations against influenza avoided 80% of cases at best (in those confirmed by laboratory tests, and using vaccines directed against circulating strains), but only 50% when the vaccine did not match, and 30% against influenza-like illness, in healthy adults. It did not change the number of people needing to go to hospital or take time off work.
([wiley.com])[2] (emphasis mine)
So, it’s effective against the precisely matching strain. How often does that occur? Here’s the CDC’s answer (read the second section carefully):
How is influenza vaccine effectiveness measured?
Vaccine efficacy and effectiveness studies use various endpoints or outcomes, which influence how we interpret the results. These endpoints may include the prevention of medically attended acute respiratory illness (MAARI), prevention of laboratory-confirmed influenza virus illness or hospitalization, prevention of influenza-like illness (ILI, such as illness with fever and cough or sore throat), or influenza-associated hospitalizations or deaths.
Studies that use outcomes such as an influenza laboratory-confirmed outcome provide the most specific estimates of the impact of the vaccine in preventing influenza. The more non-specific the outcome being measured (e.g., all pneumonia hospitalizations or influenza-like illness that include many illnesses not caused by the influenza virus), the lower the estimates of vaccine effectiveness. For example, a study by Bridges et al. (JAMA 2000) among healthy adults found that the inactivated influenza was 86% effective against laboratory-confirmed influenza, but only 10% effectiveness against all respiratory illnesses in the same population and year.
In other words, while 86% of the time you’ll avoid the strain you were vaccinated against, only 10% of the time you’ll avoid the flu. What’s really happening is this: there are dozens, possibly scores of viruses out there every year. Some are flu strains, most are not. Many of these viruses cause the same flu-like symptoms (fever, aches, chills, cough, sore throat, etc.) and your odds of resisting all of these in any given flu season with a flu vaccination is 1 in 10.
That’s not terribly effective.
This month’s The Atlantic came out with an interesting article that goes even further, citing a variety of reputable researchers who claim that all of the studies indicating that people who get the flu shot are half as likely to die as those who don’t get the flu shot are probably the effect of cohort bias (i.e., typically only health-minded people elect to get the flu shot; this group is already half as likely to die as those who aren’t health-minded).
Here’s one lone voice in the wilderness:
Nonetheless, in 2004, Jackson and three colleagues set out to determine whether the mortality difference between the vaccinated and the unvaccinated might be caused by a phenomenon known as the “healthy user effect.” They hypothesized that on average, people who get vaccinated are simply healthier than those who don’t, and thus less liable to die over the short term. People who don’t get vaccinated may be bedridden or otherwise too sick to go get a shot. They may also be more likely to succumb to flu or any other illness, because they are generally older and sicker. To test their thesis, Jackson and her colleagues combed through eight years of medical data on more than 72,000 people 65 and older. They looked at who got flu shots and who didn’t. Then they examined which group’s members were more likely to die of any cause when it was not flu season.
Jackson’s findings showed that outside of flu season, the baseline risk of death among people who did not get vaccinated was approximately 60 percent higher than among those who did, lending support to the hypothesis that on average, healthy people chose to get the vaccine, while the “frail elderly” didn’t or couldn’t. In fact, the healthy-user effect explained the entire benefit that other researchers were attributing to flu vaccine, suggesting that the vaccine itself might not reduce mortality at all. Jackson’s papers “are beautiful,” says Lone Simonsen, who is a professor of global health at George Washington University, in Washington, D.C., and an internationally recognized expert in influenza and vaccine epidemiology. “They are classic studies in epidemiology, they are so carefully done.”
The results were also so unexpected that many experts simply refused to believe them. Jackson’s papers were turned down for publication in the top-ranked medical journals. One flu expert who reviewed her studies for the Journal of the American Medical Association wrote, “To accept these results would be to say that the earth is flat!” When the papers were finally published in 2006, in the less prominent International Journal of Epidemiology, they were largely ignored by doctors and public-health officials. “The answer I got,” says Jackson, “was not the right answer.”
There’s tons more in this article I won’t spoil for you. The point is, that most scientists stink at doing real science. I didn’t really want to say that or believe it. I want to believe that the men and women in lab coats and protective eyewear are serious-minded people, who have turned their back on wealth and prestige for the greater good of research and knowledge.
But I’m beginning to see that they’re kind of like everybody else in the world: you have a tiny group of hard-core awesome passionate people who are willing to sacrifice their social standing for the sake of the truth, and a vast herd of group-thinkers smart enough to play the game but not smart enough to realize that the real game is one you play against yourself, and where everybody else wins because you play.