I have some sympathy with alternative archaeologists when it comes to debunking. It’s common to see bloggers debunking their work, but not so much other academics. One reason for that could be that academics, doing their work as a professional job in specialist domains aren’t likely to make as many mistakes as an amateur with a theory that covers a couple of thousand years and the entire globe. But that can only be half the story. Some bloggers don’t criticise other academics at all. Wouldn’t it be a bit odd that academics never make any mistakes? What should you do when they do?
A couple of months ago, I read an odd paper, we’ll call it Paper A, for reasons that might become clear below. Author A made a very simple and basic mathematical error. Something a bit like mixing up a plus and a minus sign and concluding that the Great Pyramid was a hole around 150 metres deep. It wasn’t that bad, but the author thought the conclusions flew in the face of everything known about a site. Still, the mathematics were conclusive, so he had to go with it. There were more errors, but basically the paper was given one big shove in the wrong direction, and the very intelligent and creative author tried to interpret the evidence to fit the mathematical certainty. It was published in Journal A. How do you debunk that?
What I’ve done is submitted a paper of my own pointing out the error. Rather than shred the paper to bits, I’ve shown how anyone can make the mistake of assuming a mathematical certainty. The example I give is an idea I had that, after several months, I worked out was a Bad Idea — even if it looked convincing. I imagine I’ll annoy Author A, but I’ve tried to take the sting out of the rebuttal. It’ll get a brief mention here if it gets published, and I’ll be able to host it on an institutional repository, or possibly the unedited version on arXiv. I decided to submit the rebuttal as a paper and not a blog post here because the claim appeared in Journal A, so that’s the appropriate venue to dispute it in. Because the rebuttal is under peer-review I’m hiding the name and so on to keep it anonymous. Sadly it’s easy to keep anonymous because it’s not made any public splash. This is a shame. It was a clever piece of thinking and had a sexy conclusion. If it had been sound then it would have deserved a lot more public attention.
The reason I bring it up today is that I’ve read a much worse paper today. Paper A had one big mistake and the smaller ones tended to follow from that. Paper B has at least two and I suspect three or more BIG errors. One is that the author has renamed a site. It makes it difficult to track back the prior work on the site, and the bibliography doesn’t mention it. I wouldn’t blame the peer-reviewer if he thought no serious work had been done on the site before this paper. Another problem is the scientific method used in the investigation. Have you ever laid on your back and made animal shapes from the clouds? If have you have, does that make you a zoologist? If you think that’s a bit of a leap, you might have trouble with this paper.
Paper B gives me a problem. I wrote the rebuttal of Paper A, because it was close to the sort of thing I do. It is fairly well sugar-coated and hopefully anyone reading it won’t simply assume author A is an idiot. Paper B is further from what I do, but still around one of my fields. It’s a lot worse.
So now I’m wondering if I should be writing a rebuttal of Paper B, given that my rebuttal of author A’s work wasn’t personal and Paper A was not as bad. I think I can write something original rebutting Paper B, but I can also forsee dragging myself into a series of boilerplate negative papers. It’s not my idea of fun. I think I can pull a mitigating factor out of it, with some effort. An alternative is to stick it up as a research blogging post. It that won’t be read by many people who read the original paper and it’s giving away something that with a little more effort could be a paper on the CV.
Perhaps I think about it a different way. Author A was worth my talent, because Author A had something intelligent to say, even if it was fundamentally flawed. Author B in contrast could be a waste of time. You should insert a five-minute gap here, because while I’m writing this something else has occurred to me. Author B might be a waste of time, but Audience B isn’t. Audience B could have some interesting people in it. Perhaps a rebuttal, if I can get the tone right, could be a way of networking with audience B.
I haven’t decided where I’m going with Paper B yet. If I do write a paper, then I’ll still put up a summary of the problem if there’s no OA option.