Here’s a New Yorker article about the shortcomings of the scientific method. Basically it looks at the idea that heaps of scientific ideas are greatly exaggerated by early results without adequate testing and as experiments continue the results become less impressive.
“The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws…
“Once I realized that selective reporting is everywhere in science, I got quite depressed,” Palmer told me. “As a researcher, you’re always aware that there might be some nonrandom patterns, but I had no idea how widespread it is.” In a recent review article, Palmer summarized the impact of selective reporting on his field: “We cannot escape the troubling conclusion that some—perhaps many—cherished generalities are at best exaggerated in their biological significance and at worst a collective illusion nurtured by strong a-priori beliefs often repeated.”
Such anomalies demonstrate the slipperiness of empiricism. Although many scientific ideas generate conflicting results and suffer from falling effect sizes, they continue to get cited in the textbooks and drive standard medical practice. Why? Because these ideas seem true. Because they make sense. Because we can’t bear to let them go. And this is why the decline effect is so troubling. Not because it reveals the human fallibility of science, in which data are tweaked and beliefs shape perceptions.”
So what does this all mean. Obviously science has great explanatory power. And can largely be trusted, over time, to give us an improved understanding of the world we live in, and how it works. But we’ve got to keep remembering that science is a human tool used by stupid people and subject to mistakes, and external pressures like the need to publish in order to secure research funding. There’s a really interesting quote at the end of the article that dovetails nicely into this thread on the Friendly Atheist, where the problem with holus bolus acceptance of scientific naturalism is demonstrated once again (particularly see my question in the comments). Arguments to best explanation are great. And necessary. But we’ve got to keep a bit of epistemic humility and perspective and remember that one generation’s best explanation is the next generation’s $2 archaic science textbook on sale in the nostalgia section of the op shop.
“We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.”
