Debunking Academics

Mar 24, 2010 Alun Digital Academia, Featured, Science 1 Comment Archaeoastronomy, Publishing, Skepticism

Duty Calls by xkcd

I have some sym­pathy with altern­at­ive archae­olo­gists when it comes to debunk­ing. It’s com­mon to see blog­gers debunk­ing their work, but not so much other aca­dem­ics. One reason for that could be that aca­dem­ics, doing their work as a pro­fes­sional job in spe­cial­ist domains aren’t likely to make as many mis­takes as an ama­teur with a the­ory that cov­ers a couple of thou­sand years and the entire globe. But that can only be half the story. Some blog­gers don’t cri­ti­cise other aca­dem­ics at all. Wouldn’t it be a bit odd that aca­dem­ics never make any mis­takes? What should you do when they do?

A couple of months ago, I read an odd paper, we’ll call it Paper A, for reas­ons that might become clear below. Author A made a very simple and basic math­em­at­ical error. Some­thing a bit like mix­ing up a plus and a minus sign and con­clud­ing that the Great Pyr­amid was a hole around 150 metres deep. It wasn’t that bad, but the author thought the con­clu­sions flew in the face of everything known about a site. Still, the math­em­at­ics were con­clus­ive, so he had to go with it. There were more errors, but basic­ally the paper was given one big shove in the wrong dir­ec­tion, and the very intel­li­gent and cre­at­ive author tried to inter­pret the evid­ence to fit the math­em­at­ical cer­tainty. It was pub­lished in Journal A. How do you debunk that?

What I’ve done is sub­mit­ted a paper of my own point­ing out the error. Rather than shred the paper to bits, I’ve shown how any­one can make the mis­take of assum­ing a math­em­at­ical cer­tainty. The example I give is an idea I had that, after sev­eral months, I worked out was a Bad Idea — even if it looked con­vin­cing. I ima­gine I’ll annoy Author A, but I’ve tried to take the sting out of the rebut­tal. It’ll get a brief men­tion here if it gets pub­lished, and I’ll be able to host it on an insti­tu­tional repos­it­ory, or pos­sibly the uned­ited ver­sion on arXiv. I decided to sub­mit the rebut­tal as a paper and not a blog post here because the claim appeared in Journal A, so that’s the appro­pri­ate venue to dis­pute it in. Because the rebut­tal is under peer-review I’m hid­ing the name and so on to keep it anonym­ous. Sadly it’s easy to keep anonym­ous because it’s not made any pub­lic splash. This is a shame. It was a clever piece of think­ing and had a sexy con­clu­sion. If it had been sound then it would have deserved a lot more pub­lic attention.

The reason I bring it up today is that I’ve read a much worse paper today. Paper A had one big mis­take and the smal­ler ones ten­ded to fol­low from that. Paper B has at least two and I sus­pect three or more BIG errors. One is that the author has renamed a site. It makes it dif­fi­cult to track back the prior work on the site, and the bib­li­o­graphy doesn’t men­tion it. I wouldn’t blame the peer-reviewer if he thought no ser­i­ous work had been done on the site before this paper. Another prob­lem is the sci­entific method used in the invest­ig­a­tion. Have you ever laid on your back and made animal shapes from the clouds? If have you have, does that make you a zoolo­gist? If you think that’s a bit of a leap, you might have trouble with this paper.

Paper B gives me a prob­lem. I wrote the rebut­tal of Paper A, because it was close to the sort of thing I do. It is fairly well sugar-coated and hope­fully any­one read­ing it won’t simply assume author A is an idiot. Paper B is fur­ther from what I do, but still around one of my fields. It’s a lot worse.

So now I’m won­der­ing if I should be writ­ing a rebut­tal of Paper B, given that my rebut­tal of author A’s work wasn’t per­sonal and Paper A was not as bad. I think I can write some­thing ori­ginal rebut­ting Paper B, but I can also forsee drag­ging myself into a series of boil­er­plate neg­at­ive papers. It’s not my idea of fun. I think I can pull a mit­ig­at­ing factor out of it, with some effort. An altern­at­ive is to stick it up as a research blog­ging post. It that won’t be read by many people who read the ori­ginal paper and it’s giv­ing away some­thing that with a little more effort could be a paper on the CV.

Per­haps I think about it a dif­fer­ent way. Author A was worth my tal­ent, because Author A had some­thing intel­li­gent to say, even if it was fun­da­ment­ally flawed. Author B in con­trast could be a waste of time. You should insert a five-minute gap here, because while I’m writ­ing this some­thing else has occurred to me. Author B might be a waste of time, but Audi­ence B isn’t. Audi­ence B could have some inter­est­ing people in it. Per­haps a rebut­tal, if I can get the tone right, could be a way of net­work­ing with audi­ence B.

I haven’t decided where I’m going with Paper B yet. If I do write a paper, then I’ll still put up a sum­mary of the prob­lem if there’s no OA option.