REF & ʇɔɐdɯı lɐɹnʇlnɔ

Dec 4, 2009 Alun Digital Academia, Politics, Science, The Past 2 Comments Academia, Impact, REF

A photo of Atlantis. It has more Impact if you can’t see it.

At the start I should make clear that I am not an expert and I can’t say for cer­tain exactly what Impact is. The reason I’m writ­ing this is that sadly no one else can say what Impact is either, though HEFCE has a go. That’s a prob­lem, because it’s going to play a big part in the assess­ment of uni­ver­sit­ies in 2012.

Peri­od­ic­ally uni­ver­sit­ies in the UK are assessed. In the past this has been done by the RAE, the Research Assess­ment Exer­cise. The RAE came in for a lot of cri­ti­cism. The rules would change and often decisions about the the assess­ment would work were not taken till after the uni­ver­sit­ies had made their sub­mis­sions. Many people argued that the RAE was expens­ive and unre­li­able. What’s happened is that the gov­ern­ment have listened to the cri­ti­cisms and replaced the RAE with the REF, the Research Excel­lence Frame­work. This dif­fers from the RAE in some respects, but it’s not exactly clear how, keep­ing the ele­ment of uncer­tainty. On the plus side some fea­tures like the expense have been retained. The big mys­tery though is Impact.

How do you meas­ure how good a department’s aca­demic research is? You could exam­ine that research by look­ing at a sample of aca­demic papers, and that’s 65% of the frame­work. But 25% of the assess­ment will be on Impact in non-academic contexts.

The obvi­ous example is Eco­nomic Impact, and this is where some people have an idea of what Impact is. You might research mag­netic mono­poles, but if you find one, what’s the point? If how­ever you pat­ent the pro­cess for mak­ing them and exploit it, you can show this research has a clear eco­nomic pay­back. The catch is your depart­ment has to exploit it. You can’t claim there’s an eco­nomic if a hun­dred com­pan­ies inde­pend­ently exploit it, because it’s their work as innov­at­ors that’s giv­ing your research eco­nomic impact rather than your own efforts. You can’t claim for other people’s work, which sounds sens­ible. You might have spot­ted a flaw in that, but it gets better.

Some fields clearly have no obvi­ous eco­nomic impact. You can’t really pack­age num­ber the­ory as a product and a new inter­pret­a­tion of an Aris­to­phanes’ play which sheds light on Athenian demo­cracy in the Pelo­ponnesian War is going to be hard to sell as an indus­trial pro­cess. So to help his­tor­i­ans and archae­olo­gists there’s Cul­tural Impact. As far as I, or any­one else, can tell Cul­tural Impact is gen­er­ated the same way as Eco­nomic Impact. So let’s take the Anti­kythera Mech­an­ism as an example and see what Cul­tural Impact it’s had.

Work on the mech­an­ism has fea­tured in every major qual­ity news­pa­per around the planet. That adds up to a total Cul­tural Impact of zero. That’s because it’s the news­pa­pers that exploited that to make the stor­ies, not the research team. I don’t know how much vis­its to the National Museum in Athens have gone up by, but it’s safe to bet there’s more vis­it­ors to see the mech­an­ism. Luck­ily I don’t have to know the fig­ures, because the Cul­tural Impact would be zero. It would be the National Museum that’s put­ting on the dis­play, not the research group. The research has clearly put the mech­an­ism much more in the pub­lic con­scious­ness that it has been. But we don’t need to work out a way of meas­ur­ing that but we don’t need to any­way because the Cul­tural Impact of that is zero. A pop­u­lar book might be an example of Impact. They’re work­ing on one, and there’s another already out, Decod­ing the Heav­ens. This book, which exists as a dir­ect con­sequence of their work, won’t count for Impact because Jo Marchant wrote it.

You may need to turn off your irony meter. I needed an example of excel­lent research that every­one knows about [PDF]. There’s hardly any­thing bet­ter known in recent archae­olo­gical research than the Anti­kythera Mech­an­ism. That’s why I used it, but — because it’s a thing I’m doing — they can’t claim Cul­tural Impact for it. If I’d used my own bit of research that no one had heard of then I could use this blog post as an example of cul­tural impact, because clearly the only impact it would have had would be trace­able to me.

This pain­fully illus­trates why the meas­ure­ment of Cul­tural Impact is broken. Defin­ing what Cul­ture is is a major pro­ject. One com­mon factor though is that cul­ture is shared. I don’t know of any­one arguing for a cul­ture of One. So any­thing which genu­inely has cul­tural impact as you or I would under­stand it must have escaped from its cre­ator. If you want a new word in the Oxford Eng­lish Dic­tion­ary one of the basic rules is that it must be in use bey­ond its ori­ginal source. If you want your work to have cul­tural impact it must be used in ways that you had no influ­ence over. This is the exact oppos­ite to the REF’s notion of Impact.

It’s not just a non­sense idea. If we take it ser­i­ously then it act­ively pro­motes depart­ments who take an ivory bunker men­tal­ity. Let’s ima­gine I find King Arthur’s Crown. If I want to score Impact I keep the details of the research to myself for my own pop­u­lar book. I make sure all the pho­tos of the object taken are assigned copy­right to me, so no one shows the crown without my per­mis­sion. If I put the crown on a table with no explan­a­tion for an audi­ence of 100 people that’s more Impact­ful than if I let the Brit­ish Museum put on an extra­vag­anza put­ting the crown in its medi­eval con­text. This is all about lock­ing down IP so that I own it, and the pub­lic who likely fun­ded the research don’t.

Sadly this applies to other forms of Impact. Take the work at CERN. That’s pro­duced the World Wide Web. What is the Eco­nomic Impact of the web, which handles mil­lions of pounds of trade across the world every hour? Zero. Because other people are exploit­ing it. If it were a licenced tech­no­logy that only major com­pan­ies could afford then it would have a massive impact even if most people couldn’t use it.

It seems that there’s two options. One is that the REF’s notion of Impact is borked and totally unsuit­able for the task it’s given. The other is that Impact is fine and genu­inely trans­form­at­ive research which every­one can use in their own way is a Bad Thing that we should not encour­age. If this is about mak­ing sup­port­ing excel­lent research that has genu­ine cul­tural impact, then the cur­rent Impact idea would be exactly the wrong way to go about it.