Lehrer's getting a raw deal, but...

Jonah Lehrer is getting a lot of flak from the skeptic community over an article in the New Yorker he wrote about the "decline effect", whereby some statistically significant correlations tend to "wear off" over time as more and more attempts are made to replicate them.

It's an interesting problem, and for the most part he gave a thoughtful examination of the issues and their causes (the primary cause, IMO, being simple publication bias). However, he was harshly criticized -- and rightly so -- for the final paragraph of the original article in which he seems to flirt with a destructive form of post-modernism, a rejection of the very nature of objective fact. The worry is that he has given more fodder for science deniers of all stripes to ignore the overwhelming evidence in favor of evolution, climate change, etc. and against most forms of alternative medicine, psychic ability, etc.

I felt his first follow-up (sorry, I can't find the link at the moment) clarified his position sufficiently, and I sort of think the continued criticism he is receiving is overblown. I do want to highlight one rhetorical question from his most recent follow-up, however, because I think this question is very answerable. The context here is that he is discussing how the theory in evolutionary biology of "fluctuating asymmetry" initially got several false positives in numerous studies, before the preponderance of evidence turned against it.

This raises the obvious problem: If false results can get replicated, then how do we demarcate science from pseudoscience?

In principle, there is a very simple answer to this question, though of course applying it is often quite messy in practice: Bayes' theorem.

I'm a bit too busy at work right now to give even a brief treatment of the Bayesian mode of thought, but the Wikipedia article does a nice job, so read that if you are unfamiliar. As it applies to Lehrer's present question, we can (usually) demarcate science from pseudoscience by taking a guess at the prior probability.

Fluctuating asymmetry falls in the "science" camp, despite the fact that it ultimately turned out to be false, and even despite the fact that it was prematurely elevated to the status of "true" based on an (in retrospect) insufficient number of positive results, because it was a priori plausible. Homeopathy, on the contrary, falls in the "psuedoscience" camp, despite the fact that some trials of homeopathy have achieved positive results, because it is completely implausible.

As I said, this can get messy in practice. How would one classify early trials of acupuncture, when plausibility could be argued either way? And how much undue resistance was there to the theory of continental drift (now solidified in the more complete theory of plate tectonics) because most scientists at the time found the idea of moving continents to be a priori implausible?

But we already knew science is messy, and I think that overall Lehrer's original article and follow-ups have helped illuminate one of those messy aspects to a lay audience. I must reiterate that I agree with those who found the concluding paragraph of the original article to be severely misguided -- and I think in this most recent reply, he also missed an opportunity for a teaching moment by not digging deeper into his self-posed question. The question about demarcation does have an answer, one that is relatively simple in principle, if not in practice.