At the start I should make clear that I am not an expert and I can’t say for certain exactly what Impact is. The reason I’m writing this is that sadly no one else can say what Impact is either, though HEFCE has a go. That’s a problem, because it’s going to play a big part in the assessment of universities in 2012.
Periodically universities in the UK are assessed. In the past this has been done by the RAE, the Research Assessment Exercise. The RAE came in for a lot of criticism. The rules would change and often decisions about the the assessment would work were not taken till after the universities had made their submissions. Many people argued that the RAE was expensive and unreliable. What’s happened is that the government have listened to the criticisms and replaced the RAE with the REF, the Research Excellence Framework. This differs from the RAE in some respects, but it’s not exactly clear how, keeping the element of uncertainty. On the plus side some features like the expense have been retained. The big mystery though is Impact.
How do you measure how good a department’s academic research is? You could examine that research by looking at a sample of academic papers, and that’s 65% of the framework. But 25% of the assessment will be on Impact in non-academic contexts.
The obvious example is Economic Impact, and this is where some people have an idea of what Impact is. You might research magnetic monopoles, but if you find one, what’s the point? If however you patent the process for making them and exploit it, you can show this research has a clear economic payback. The catch is your department has to exploit it. You can’t claim there’s an economic if a hundred companies independently exploit it, because it’s their work as innovators that’s giving your research economic impact rather than your own efforts. You can’t claim for other people’s work, which sounds sensible. You might have spotted a flaw in that, but it gets better.
Some fields clearly have no obvious economic impact. You can’t really package number theory as a product and a new interpretation of an Aristophanes’ play which sheds light on Athenian democracy in the Peloponnesian War is going to be hard to sell as an industrial process. So to help historians and archaeologists there’s Cultural Impact. As far as I, or anyone else, can tell Cultural Impact is generated the same way as Economic Impact. So let’s take the Antikythera Mechanism as an example and see what Cultural Impact it’s had.
Work on the mechanism has featured in every major quality newspaper around the planet. That adds up to a total Cultural Impact of zero. That’s because it’s the newspapers that exploited that to make the stories, not the research team. I don’t know how much visits to the National Museum in Athens have gone up by, but it’s safe to bet there’s more visitors to see the mechanism. Luckily I don’t have to know the figures, because the Cultural Impact would be zero. It would be the National Museum that’s putting on the display, not the research group. The research has clearly put the mechanism much more in the public consciousness that it has been. But we don’t need to work out a way of measuring that but we don’t need to anyway because the Cultural Impact of that is zero. A popular book might be an example of Impact. They’re working on one, and there’s another already out, Decoding the Heavens. This book, which exists as a direct consequence of their work, won’t count for Impact because Jo Marchant wrote it.
You may need to turn off your irony meter. I needed an example of excellent research that everyone knows about [PDF]. There’s hardly anything better known in recent archaeological research than the Antikythera Mechanism. That’s why I used it, but — because it’s a thing I’m doing — they can’t claim Cultural Impact for it. If I’d used my own bit of research that no one had heard of then I could use this blog post as an example of cultural impact, because clearly the only impact it would have had would be traceable to me.
This painfully illustrates why the measurement of Cultural Impact is broken. Defining what Culture is is a major project. One common factor though is that culture is shared. I don’t know of anyone arguing for a culture of One. So anything which genuinely has cultural impact as you or I would understand it must have escaped from its creator. If you want a new word in the Oxford English Dictionary one of the basic rules is that it must be in use beyond its original source. If you want your work to have cultural impact it must be used in ways that you had no influence over. This is the exact opposite to the REF’s notion of Impact.
It’s not just a nonsense idea. If we take it seriously then it actively promotes departments who take an ivory bunker mentality. Let’s imagine I find King Arthur’s Crown. If I want to score Impact I keep the details of the research to myself for my own popular book. I make sure all the photos of the object taken are assigned copyright to me, so no one shows the crown without my permission. If I put the crown on a table with no explanation for an audience of 100 people that’s more Impactful than if I let the British Museum put on an extravaganza putting the crown in its medieval context. This is all about locking down IP so that I own it, and the public who likely funded the research don’t.
Sadly this applies to other forms of Impact. Take the work at CERN. That’s produced the World Wide Web. What is the Economic Impact of the web, which handles millions of pounds of trade across the world every hour? Zero. Because other people are exploiting it. If it were a licenced technology that only major companies could afford then it would have a massive impact even if most people couldn’t use it.
It seems that there’s two options. One is that the REF’s notion of Impact is borked and totally unsuitable for the task it’s given. The other is that Impact is fine and genuinely transformative research which everyone can use in their own way is a Bad Thing that we should not encourage. If this is about making supporting excellent research that has genuine cultural impact, then the current Impact idea would be exactly the wrong way to go about it.Google+