Yet another climate post, but like my previous ones, this isn’t really about climate, it’s about science and it’s current state. Everyone by now has read about the leaked emails and source code (source followup here) last week. It’s a big deal.
There was a NY Times post today about a scientist who addresses skeptics. She listed three approaches a scientist can take when confronted by skeptics:
The Climate Research Unit went for #2, the scientist in the article believes that #3 is the best way (I agree), and this got me thinking about where skeptics come from in the first place, and how to deal with them.
One of the reasons I become skeptical about a some scientific and medical research is because the authors hide the data and they won’t make their methods known. I’m not a scientist by training, but I have some background in critical thinking and statistics. I’m not a stupid person (deliberately, anyway). If you show me the numbers and tell me how you interpret them, I can better judge your results.
Which leads to point 2:
“Skeptics are willfully ignorant of the scientific method and religiously cling to unfounded beliefs in the face of contrary data.”
Ok, you see my point? I’ll spell it out for the above mentioned skeptic: not all skeptics are created equally. Smart scientists know this, of course. And smart skeptics know that not all scientists are created equally.
Scientists who follow methods 1 and 2 above are not helping the cause of science. Whenever we use non-scientific tools (such as “consensus,” which is just large scale appeal to authority), we’re not helping the intelligent-but-ignorant skeptic join us, and intelligent people know the difference between a good argument and a bad one.
I understand the frustration when dealing with the hopelessly stupid skeptic—these people are never going to change their minds no matter what evidence you present to them; they should just be ignored.
But please don’t ignore the honest skeptic, the fellow who is simply begging for the straight evidence. A good scientist is also a good communicator, who can take the data and explain it clearly to another intelligent and honest seeker of understanding.
If you have good data, why not share it? If you have iffy data, for the sake of good science, share it too, and we’ll all be wiser.
I’ve been researching the effectiveness of flu vaccines lately, since everybody keeps asking me if I’m going to get one. I sometimes ask, “Are they effective?” and the response is always “Well, I got one last year and I didn’t get the flu.” After reading “How to read articles about health and healthcare”, I began to wonder if this flu vaccination thing smells right.
We hear endless news stories quoting doctors:
State Health Commissioner Dr. Karen Remley, a Virginia pediatrician, recommends the shot as basic prevention. She’s concerned about what it can cause.
“Flu is one of the leading causes of death in our country,” she says. “Over 36,000 people die every year in our country and it’s preventable by getting the flu shot.”
We could really save 36,000 people’s lives if they only got the flu shot? That’s remarkable. What worries me when I hear news organizations quoting people who aren’t experts in the research they’re purporting to cover in their “story” is that everybody buys into it without thinking. “Look at me, I’m a trained physician with a white coat and stethoscope around my neck—I went to school and you can trust me!”
I worry in the back of my mind that the bulk of “science” out there has come from bad scientists. Odds are, it has. Odds are that most of the research out there was done by mediocre scientists or scientists who are looking to get published, and so they’ll do research trying to fit the data to existing studies, because that’s an easy way to get published: find existing research, copy it as best you can making sure the results match, submit to journals. “We can confirm that flu vaccination cuts the risk of death to those who received it in half, compared to those who didn’t.”
This reminds me of another recent article that turned the “lactic acid in muscles is bad for you” science upside down. True or false: When you work out, lactic acid builds up in your muscles. If too much builds up, you start getting fatigued.
It used to be true, but now it’s false, it turns out. Lactic acid is fuel your muscles use to create more energy. Who knew? (nytimes.com)
At any rate, the flu thing began to bother me in the same way.
The first thing I did was looked at studies or meta-studies of flu vaccine effectiveness: what is the measured benefit of flu vaccination in the real-world? We all know that polio and other (non-morphing) diseases have been virtually eradicated because of vaccination. This should work for flu, too, right?
Not entirely. The plain language summary for a 2007 metastudy concluded:
There is not enough evidence to decide whether routine vaccination to prevent influenza in healthy adults is effective.
The review of trials found vaccinations against influenza avoided 80% of cases at best (in those confirmed by laboratory tests, and using vaccines directed against circulating strains), but only 50% when the vaccine did not match, and 30% against influenza-like illness, in healthy adults. It did not change the number of people needing to go to hospital or take time off work.
([wiley.com]) (emphasis mine)
So, it’s effective against the precisely matching strain. How often does that occur? Here’s the CDC’s answer (read the second section carefully):
How is influenza vaccine effectiveness measured?
Vaccine efficacy and effectiveness studies use various endpoints or outcomes, which influence how we interpret the results. These endpoints may include the prevention of medically attended acute respiratory illness (MAARI), prevention of laboratory-confirmed influenza virus illness or hospitalization, prevention of influenza-like illness (ILI, such as illness with fever and cough or sore throat), or influenza-associated hospitalizations or deaths.
Studies that use outcomes such as an influenza laboratory-confirmed outcome provide the most specific estimates of the impact of the vaccine in preventing influenza. The more non-specific the outcome being measured (e.g., all pneumonia hospitalizations or influenza-like illness that include many illnesses not caused by the influenza virus), the lower the estimates of vaccine effectiveness. For example, a study by Bridges et al. (JAMA 2000) among healthy adults found that the inactivated influenza was 86% effective against laboratory-confirmed influenza, but only 10% effectiveness against all respiratory illnesses in the same population and year.
In other words, while 86% of the time you’ll avoid the strain you were vaccinated against, only 10% of the time you’ll avoid the flu. What’s really happening is this: there are dozens, possibly scores of viruses out there every year. Some are flu strains, most are not. Many of these viruses cause the same flu-like symptoms (fever, aches, chills, cough, sore throat, etc.) and your odds of resisting all of these in any given flu season with a flu vaccination is 1 in 10.
That’s not terribly effective.
This month’s The Atlantic came out with an interesting article that goes even further, citing a variety of reputable researchers who claim that all of the studies indicating that people who get the flu shot are half as likely to die as those who don’t get the flu shot are probably the effect of cohort bias (i.e., typically only health-minded people elect to get the flu shot; this group is already half as likely to die as those who aren’t health-minded).
Here’s one lone voice in the wilderness:
Nonetheless, in 2004, Jackson and three colleagues set out to determine whether the mortality difference between the vaccinated and the unvaccinated might be caused by a phenomenon known as the “healthy user effect.” They hypothesized that on average, people who get vaccinated are simply healthier than those who don’t, and thus less liable to die over the short term. People who don’t get vaccinated may be bedridden or otherwise too sick to go get a shot. They may also be more likely to succumb to flu or any other illness, because they are generally older and sicker. To test their thesis, Jackson and her colleagues combed through eight years of medical data on more than 72,000 people 65 and older. They looked at who got flu shots and who didn’t. Then they examined which group’s members were more likely to die of any cause when it was not flu season.
Jackson’s findings showed that outside of flu season, the baseline risk of death among people who did not get vaccinated was approximately 60 percent higher than among those who did, lending support to the hypothesis that on average, healthy people chose to get the vaccine, while the “frail elderly” didn’t or couldn’t. In fact, the healthy-user effect explained the entire benefit that other researchers were attributing to flu vaccine, suggesting that the vaccine itself might not reduce mortality at all. Jackson’s papers “are beautiful,” says Lone Simonsen, who is a professor of global health at George Washington University, in Washington, D.C., and an internationally recognized expert in influenza and vaccine epidemiology. “They are classic studies in epidemiology, they are so carefully done.”
The results were also so unexpected that many experts simply refused to believe them. Jackson’s papers were turned down for publication in the top-ranked medical journals. One flu expert who reviewed her studies for the Journal of the American Medical Association wrote, “To accept these results would be to say that the earth is flat!” When the papers were finally published in 2006, in the less prominent International Journal of Epidemiology, they were largely ignored by doctors and public-health officials. “The answer I got,” says Jackson, “was not the right answer.”
There’s tons more in this article I won’t spoil for you. The point is, that most scientists stink at doing real science. I didn’t really want to say that or believe it. I want to believe that the men and women in lab coats and protective eyewear are serious-minded people, who have turned their back on wealth and prestige for the greater good of research and knowledge.
But I’m beginning to see that they’re kind of like everybody else in the world: you have a tiny group of hard-core awesome passionate people who are willing to sacrifice their social standing for the sake of the truth, and a vast herd of group-thinkers smart enough to play the game but not smart enough to realize that the real game is one you play against yourself, and where everybody else wins because you play.
I’ve been saying it for a while: students learn better with qualified, caring teachers and quality teaching materials than anything else, and One-Laptop-Per-Child is an expensive distraction from real education.
I think everyone who has been educated knows this instinctively. We can all point to a mentor (a teacher, a parent, a caring friend) who guided us, encouraged us, and cared enough about us to help us learn what is worth learning and what is not worth learning.
Giving a child a laptop does neither of these things. Some argue that simply having access to information will be the child’s ticket to a bright future, but now that the results are in, we can finally stop beating this dead horse.
You will always find exceptions: highly motivated students will learn no matter what you do to them, and giving them access to a computer will certainly give them an opportunity to get the information they want faster; but for the majority of students, giving them a computer will do little good, and will often do much harm, to their education.
The Internet is not a discriminator of information, and this is its chief problem. A classically-based education, as an example, imposes a hierarchy on knowledge. It says, “These things are more important than those things.” And when I say “it” I really mean a classically trained teacher guiding a student through a classical curriculum.
The Internet can’t do that for you by itself. You still need a guide, a mentor of some kind to help you know what’s worth learning. Autodidacts learn this early on and quickly learn which voices in life they want to trust (generally by way of books and other reading material), but the rest of us simply don’t know where to go.
And while we’re on the topic of online learning, did you hear of the recent meta-study by the US Department of Education? Everybody in the tech world is feeling validated, but they shouldn’t. It claims:
The meta-analysis found that, on average, students in online learning conditions performed better than those receiving face-to-face instruction.
Read the study and notice that it’s a meta-study, largely sampling the results of self-selecting, uncontrolled learning experiements. Highly-motivated students, the kind who opt to take online courses, will always perform better than unmotivated students, no matter what medium they’re using.
The abstract concludes:
Analysts noted that these blended conditions often included additional learning time and instructional elements not received by students in control conditions. This finding suggests that the positive effects associated with blended learning should not be attributed to the media, per se. An unexpected finding was the small number of rigorous published studies contrasting online and face-to-face learning conditions for K–12 students. In light of this small corpus, caution is required in generalizing to the K–12 population because the results are derived for the most part from studies in other settings (e.g., medical training, higher education).
So don’t take this as a validation for K-12 computer learning. Additional studies over OLPC and K-12 computer learning aren’t encouraging:
Under the auspices of the No Child Left Behind Act, the U.S. Department of Education and Mathematica Policy Research recently completed a rigorous, two-year test of reading and math software (using programs that had at least some nonexperimental evidence that they “worked”) in dozens of school districts nationwide. With one minor exception, the studies found that children using the software scored no better than peers who did not have access to the software.
Economists Ofer Malamud and Cristian Pop-Eleches studied a program in Romania that distributed discount vouchers for the purchase of home computers to low-income families. When they compared the families that used the vouchers to acquire computers with families that were just above the income cut-off to receive the vouchers, they found that computers had a negative effect on students’ grades and educational goals.
(Leigh) Linden (an economist at Columbia University) also led one of the few experimental studies to show a positive impact from the use of computers — a project in India that provided computers and education software to schools and randomly assigned some schools to use the software during school hours and others to encourage computer use after hours. This study found that using computers during school hours-—essentially substituting computers for teachers—actually hurt learning, while using them after hours as a supplement to traditional classroom teaching had dramatic positive effects on the weakest students. Even this outcome doesn’t really support the OLPC mission, though; the software evaluated is very much in the “drill and practice” model that Negroponte has explicitly derided.
I won’t spoil all the fun of this article, but there are far, far better and less expensive ways to improve education than giving a child a laptop. In developing countries the list includes deworming children, providing tutors, and creating more private schools.
In the United States, computers in classrooms and schools should be removed and with the money not spent on technology, hire additional teachers or give the existing teachers a raise. I’m certain we’d get more of what we’re really looking for.
Remember a few weeks ago when I griped about global warming? What bothered me wasn’t that we have fringe groups, conspiracy theorists, and “deniers.” These folks will always be with us in everything (sometimes they’re even right).
What bothered me was that there seems to be a legitimate case against the predominant view and few scientists in the mainstream are addressing it directly and scientifically. Almost nobody from the scientific “consensus” community are taking these people on, rather they’re using name-calling and ostracism to maintain their position. It also bothers me that scientists feel they have to stoop to “consensus” and PR smear campaigns to make their point.
Consensus is fine—great—for some things, but not in science:
If the brightest minds on Wall Street got suckered by group-think into believing house prices would never fall, what other policies founded on consensus wisdom could be waiting to come unraveled? Global warming, you say? You mean it might be harder to model climate change 20 years ahead than house prices 5 years ahead? Surely not—how could so many climatologists be wrong?
What’s wrong with consensuses is not the establishment of a majority view, which is necessary and legitimate, but the silencing of skeptics. “We still have whole domains we can’t talk about,” Dr. Bouchard said, referring to the psychology of differences between races and sexes.
I would argue that there are even more things than psychology that aren’t being talked about, including climate change, because of this fear of consensus. Politicians and business-types are the kinds of people who look for consensus of opinion—that’s because they don’t actually know or do anything. They mobilize opinion to get other people to actually think or do. But scientists are different—they’re doers. Science is for observing, gathering, analyzing and interpreting data with confidence intervals. Maybe come up with a theory. Scientific disagreements should be about the data and its interpretation, not posturing and raspberry-blowing.
Scientists are of course entitled to have opinions as much as the next op-ed writer, but a scientist is expected to have some rigor and discipline when engaging in and refuting arguments. More importantly, a scientist is expected to be a thinker capable of ignoring the ridicule of his or her peers to get to the truth of a matter.
An article in the New York Times recently wrote about how Groupthink might be the culprit of our financial crisis (read: “collapse”):
In his classic 1972 book, Groupthink, Irving L. Janis, the Yale psychologist, explained how panels of experts could make colossal mistakes. People on these panels, he said, are forever worrying about their personal relevance and effectiveness, and feel that if they deviate too far from the consensus, they will not be given a serious role. They self-censor personal doubts about the emerging group consensus if they cannot express these doubts in a formal way that conforms with apparent assumptions held by the group.
When scientists bond together and form a consensus, science itself is imperiled. Because scientists are human and care about what their peers think, they take a big risk of stifling creative, scientific thought. Science is, at its core, the opposite of what we achieve by consensus. Science needs naysayers and village idiots every bit as much as the respectable, well-groomed, mainstream thinkers.
Paul Graham wrote an essay called “What you can’t say” that puts it even better:
Of course, we’re not just looking for things we can’t say. We’re looking for things we can’t say that are true, or at least have enough chance of being true that the question should remain open. But many of the things people get in trouble for saying probably do make it over this second, lower threshold. No one gets in trouble for saying that 2 + 2 is 5, or that people in Pittsburgh are ten feet tall. Such obviously false statements might be treated as jokes, or at worst as evidence of insanity, but they are not likely to make anyone mad. The statements that make people mad are the ones they worry might be believed. I suspect the statements that make people maddest are those they worry might be true.
Another way of putting it is Bob Sutton’s “strong opinions, weakly held.” Whatever we call it, it’s something more scientists need to learn apparently. We’re not in high school anymore, and any scientist who puts popularity and peer acceptance ahead of their scientific rigor, even in their spare time, is doing more harm than good.
In a recent lecture, Paul Krugman, winner of the Nobel prize in economics in 2008, argued that much of the past 30 years of macroeconomics was “spectacularly useless at best, and positively harmful at worst.” Barry Eichengreen, a prominent American economic historian, says the crisis has “cast into doubt much of what we thought we knew about economics.”
Alright, maybe not all of economics is bunk, but I think a lot of macroeconomics was simply invented by economists to justify their own positions. Even economist John Kenneth Galbraith (lousy Keynesian) said:
Economics is extremely useful as a form of employment for economists.
Create an academic field around a small idea and pretty soon your experts in that field will be finding new ways to justify themselves. It’s true in every field. We do this in the name of science, of course, but the trouble is that there aren’t enough people in the world to say “Hey, your entire field of study is stupid and wrong. You’re making things worse.”
Well, I’m going to say it here: Macroeconomists, you’re making it worse. Stop pretending you have answers. You don’t. I don’t either, but I’m not pretending. Stop talking to policy-makers. Tell them that you don’t know and nobody else does either. Tell them that maybe we ought to consider getting out of debt as a country, to spend less than we make, as a lark, you know, to see what happens. Then please find another area of study.
Lest my readers think I’m a “denialist”, I post here a link to a great article about interpreting the numbers on global warming.
But while these recent posts have ostensibly been about climate change science, they’ve really been about the nature of science itself. For example, I’m still troubled by the heavy political language. Here we have someone skilled in statistical analysis using the same kind of browbeating against any dissent that Paul Krugman uses in his recent article:
And as I watched the deniers make their arguments, I couldn’t help thinking that I was watching a form of treason — treason against the planet.
Treason? That’s pretty strong language (but this is Prince Krugman—he can say whatever he wants and never be wrong, even as a non-scientist). I think all sensible people realize that things are warming up (even if there are local variations from year to year). The long-term indicators we can measure say that the globe is warming. We can accept that (though apparently some can’t—that’s fine too).
What is hard to stomach is the assertion that the near-universal scientific consensus about what is causing the warming to this this fantastically complex system we call Earth implies that there can be no more inquiry into the matter.
It is not science’s job to quash dissent through consensus. It is science’s job to welcome honest differences and to acknowledge places where we still lack data. A good scientist does the research, crunches the numbers, and gives the confidence intervals. Let the Op Ed writers push the policy.
Whenever you adopt science as your religion, you should be prepared to deal with the atheists, agnostics, and general skeptics with grace, civility, and respect. Those who fail to do this do not deserve the honorable title of scientist.
The world is full of plenty of things to believe and plenty to doubt, and a real scientist withholds judgement on both sides, knowing that the jury is always out in the face of new evidence.
Scientists who have signed up to turn climate change science into climate change dogma by ridiculing or ostracizing any nay-sayers are giving all true scientists a black eye. Science is about open and honest inquiry. Leave the political jockeying to lesser minds.