Thursday, 9 May 2013

The academic backlog

Photo from http://www.pa-legion.com

Here’s an interesting question to ask any scientist: If you were to receive no more research funding, and just focus on writing up the data you have, how long would it take? The answer tends to go up with seniority, but a typical answer is 3 to 5 years.
I don’t have any hard data on this – just my own experience and that of colleagues – and I suspect it varies from discipline to discipline. But my impression is that people generally agree that the academic backlog is a real phenomenon, but they disagree on whether it matters.
One view is that completed but unpublished research is not important, because there’s a kind of “survival of the fittest” of results. You focus on the most interesting and novel findings, and forget about the rest. It’s true that we’ve all done failed studies with inconclusive results, and it would be foolish trying to turn such sow’s ears into silk purses.  But I suspect there’s a large swathe of research that doesn’t fall into that category, but still never gets written up.  Is that right, given the time and money that have been expended in gathering data? Indeed, in clinical fields, it’s not only researchers putting effort into the research – there are also human participants who typically volunteer for studies on the assumption that the research will be published.
I’m not talking about research that fails to get published because it’s rejected by journal editors, but rather about studies that don’t get to the point of being written up for publication. Interest in this topic has been stimulated by Ben Goldacre’s book Bad Pharma, which has highlighted the numerous clinical trials that go unreported – often because they have negative results. In that case the concern is that findings are suppressed because they conflict with the financial interests of those doing the research, and the Alltrials campaign is doing a sterling job to tackle that issue. But beyond the field of clinical trials, there’s still a backlog, even for those of us working in areas where financial interests are not an issue.

It’s worth pausing to consider why this is so. I think it’s all to do with the incentive structure of academia. If you want to make your way in the scientific world, there are two important things you have to do: get grant funding and publish papers. This creates an optimisation problem, because both of these activities take time, and time is in short supply for the average academic.  It’s impossible to say how long it takes to write a paper, because it will depend on the complexity of the data, and will vary from one subject area to the next, but it’s not something that should be rushed. A good scientist checks everything thoroughly, thinks hard about alternative interpretations of results, and relates findings to the existing research literature. But if you take too much time, you’re at risk of being seen as unproductive, especially if you aren’t bringing in grant income. So you have to apply for grants, and having done so, you have then to do the research that you said you’d do. You may also be under pressure to apply for grants to keep your research group going, or to fund your own salary.

When I started in research, a junior person would be happy to have one grant, but that was before the REF. Nowadays  heads of department  will encourage their staff to apply for numerous grants, and it’s commonplace for senior investigators have several active grants, with estimates of around 1-2 hours per week spent on each one. Of course, time isn’t neatly divided up, and it’s more likely that the investigator will get the project up and running and then delegate it to junior staff, then putting in additional hours at the end of the project when it’s time to analyse and write up the data. The bulk of the day-to-day work will be done by postdocs or graduate students, and it can be a good training opportunity for them. All the same, it’s often the case that the amount of time specified by senior investigators is absurdly unrealistic. Yet this approach is encouraged: I doubt anyone ever questions a senior investigator’s time commitment when evaluating a grant, few funding bodies check whether you’ve done what you said you’d do, and even if they do, I’ve never heard of a funder demanding that a previous project be written up before they’ll consider a new funding application.

I don’t think the research community is particularly happy about this: many people have a sense of guilt at the backlog, but they feel they have no option. So the current system creates stress as well as inefficiency and waste. I’m not sure what the solution is, but I think this is something that research funders should start thinking about. We need to change the incentives to allow people time to think. I don’t believe anyone goes into science because they want to become rich and famous: we go into it because we are excited by ideas and want to discover new things. But just as bankers seem to get into a spiral of greed whereby they want higher and higher bonuses, it’s easy to get swept up in the need to prove yourself by getting more and more grants, and to lose sight of the whole purpose of the exercise – which should be to do good, thoughtful science.  We won’t get the right people staying in the field if we value people solely in terms of research income, rather than in terms of whether they use that income efficiently and effectively.

21 comments:

  1. There are checks and balances with respect to how the money get spent, and especially if you apply for another grant in the same area. I always thought that grants don't get given unless you show that you've been publishing consistently in good journals. Have I really been that blind?

    ReplyDelete
    Replies
    1. The point is not about whether you publish or not, but about whether good stuff is left unpublished because of pressure to get on with the next thing. I get funded and I have decent publications but I still have a backlog of unpublished studies that would take a few years to clear.

      Delete
  2. Charlie Wilson @crewilson9 May 2013 at 13:55

    It would probably take me more than 2 years to get everything out of the data I currently have in hand, and I'm relatively early in my career.

    Another factor for me has been moving between different projects on short postdoc-ey or fellowship-ey projects. Each time I get only 2 years. Each time I try to publish to show that I did the project. Each time I have to move on to be paid. Each time I add to the backlog the stuff that would bear fruit with more work but needs the time I don't have because, somehow, I need to have a salary.

    So yes, it's the incentive structure, but it's also the career structure that promotes this backlog. If someone would fund my salary for 2 years to catch up (i.e. analyse and write, no experiments), I think I could do some really good science. Chance would be a fine thing....

    ReplyDelete
    Replies
    1. I agree - it is not a sensible structure. In the 1980s, when I started as an independent researcher, the MRC had a senior research fellowship scheme that gave you 5 yr renewable for 5 years. I did the core work that got me established and everything I did got written up. Almost immediately after I got this fellowship, the scheme was changed to make the time periods shorter (3 yr + 3 yr, and I'm not sure what now). I guess the funders have the difficult choice of deciding whether to support more people for shorter periods, or fewer people for longer. The latter is risky and could be wasteful if you support the wrong people. But all i can say is that I needed the stability of the initial 5 years to really get to grips with what I was investigating. I made mistakes and learned from them. And I was not distracted by having to apply for other grants.

      Delete
  3. Personally I cannot see why grants are taken into account at all. They are inputs and its output that matters or if you want to be technical it should be Value Added (output - inputs). Say researchers A and B have grants of £1m and £2m respectively and they independently come up with the same discovery. A is a better researcher because he did the same work more efficiently but typical research assesments would wrongly give B more credit. My experience (& in case you haven't guessed, I am an economist) is that scientists just cannot grasp this point such is the fetish about getting grants.
    Kevin Denny

    ReplyDelete
    Replies
    1. I could not agree more. I have made similar arguments in previous blogposts, and drawn attention to the case of Nobel Laureate Daniel Kahneman as an example of someone who would be regarded as underperforming if evaluated by current REF criteria, because his research did not require expensive equipment and could easily be done by running experiments on samples of students- this kind of work could be done with minimal funding.
      The reason for the fetish, of course, is because the universities benefit financially from grant income. We are explicitly enjoined to apply for grants, preferably big ones, to boost our departmental income.

      Delete
    2. Kevin,
      One problem is that the researchers and their peers have a grant-getting fetish. Another problem is that Deans and their superiors have a "how much money our scientists brought in" fetish. The first problem can easily be fixed by coming to grips with your economic analysis. The second problem is much more difficult to fix. From the perspective of a Dean or higher, income brought in is a very real and tangible positive, regardless of output. Look how it helps your school's ratings! Look how it pays for renovations, staff, and an array of temporary positions!

      If ranking systems could adopt something like your system, e.g. surface level REF minus money spent to gain that REF, it would be interesting to see the fall out. (P.S. Given that I am in the U.S., I hope that last part was coherent.)

      Delete
  4. As I have often said "the incentive structure of academia" seems to have been designed to encourage shallow, short-term, work, and to encourage dishonesty (usually minor but sometimes fraud).

    That is no way to run a ship.

    ReplyDelete
  5. V interesting post. Actually, some of us in Ireland now have plenty of time to catch up on that backlog, because funding for research into fundamental areas of science has pretty much disappeared. In a way, I'm glad to step out of the proposal application loop, but god knows what the future holds...
    Cormac

    ReplyDelete
    Replies
    1. Well, we all might be in this position soon, so perhaps we will have time to tackle the backlogs.
      You comment also gives me the chance to clarify that my post is not be a call for a reduction in research funding. (I realise you are not saying that, but I was worried it could be misconstrued that way). Rather, I would like to see a more equitable distribution of research funding - it would be great if there were, for instance, the kind of longer-term support offered to promising junior researchers that would allow them to get stuck into a problem - see comment above by crewilson and my reply.
      When I see senior figures with numerous big grants, I start to see parallels with fat cat bankers, hogging all the resources. Of course, there are excellent senior investigators who use their grant income to support junior people and foster their careers. But there are some who just seem to treat the funding as a status symbol and try to accumulate as much as they can. That's why I think we should start looking at how much time investigators are devoting to research projects, and whether it is realistic - if they want the money but won't or can't put in the time, then I think this is a problem.

      Delete
    2. There are some very interesting points being made. Too many to make full comments but a few thoughts occur to me. There is certainly a "fat cat" phenomenon in grant getting. It seems clear to me that as soon as one gets into the big-grant loop there is a tendency for the money to keep coming as one appears at the right committee meetings and becomes well-known to the grant awarding bodies. One also gets to judge the grants of potential competitors. Even if one is utterly honest in this process ( as one would hope were the case), in general people will not in general want to antagonise you. All these things contribute to a self-perpetuating system.
      I am a brain imager and my wife is a psychologist. On paper I do better as my grants are bigger but that is only because the research itself is intrinsically more expensive. My wife deals with work that is much more about intervention in the community. The journals that she publishes in also have lower impact factors than "typical" imaging journals. It would be foolish in the extreme to say that her work is less important, it simply doesn't do as well on the metrics. One could argue strongly that her research has a greater practical impact than mine.
      What would be interesting is to compute a productivity index such as citations per unit of grant income rather than just counting grant income alone. However, whenever I have suggested this grant committee meetings it seems to produce expressions of shock and fear. Maybe big grants just don't produce as much as they should.

      On balance, looking back at my career, the most interesting time was when my colleagues and I were doing work on tiny amounts of money we had scraped together. We were simply interested in seeing what the technology could tell us about the brain. Happy days.

      Delete
  6. Here's an experiment in backlog - I was retired two years ago with a year's notice and am in the throes of writing the last paper. So 3-3.5 years for me.

    ReplyDelete
  7. If we all pulled out, took apart and wrote up all the filing cabinet papers it would be a full-time job. Multiple research grants are baffling.

    Kevin Denny's post is provocative and persuasive but overlooks two things: it is not the scientists who fetishise the Brobdignagian grants -apart from those who enjoy bragging about these things- but their employers. The old RAE used income as an index of research quality. Second, the larger grant, with FEC, means that the University would get a larger cut of the booty and therefore would more greatly benefit from (in this argument) the less efficiently completed research.

    ReplyDelete
    Replies
    1. Yup, absolutely.
      I've tried arguing that the system should change to value a high output/input ratio as argued by Anon economist above - but I don't expect anyone will take this seriously:
      http://deevybee.blogspot.co.uk/2010/08/how-our-current-reward-structures-have.html

      Delete
  8. Great post! Did you catch my blog special on the year of scandals in psychology for 2011? I think some of the messages complements this post nicely: http://fixingpsychology.blogspot.com/2011/12/christmas-special-year-of-scandals-in.html

    ReplyDelete
  9. Thanks for the pointer. I enjoyed your post. Interesting to see you have similar concerns across the pond. Your comment "we should be incredibly suspicious of people who appear to be impossibly productive" is spot on.

    ReplyDelete
  10. Just chiming in with the other posters to say i've just left my PhD and can already see the backlog building. I think Charlie Wilson's comment is spot on. How are we every suppose to get full value out of our data when we have to constantly redefine ourselves to ensure steady employment? I often joke that I will stay a post-doc forever but I really think we should push for permanent research only faculty positions that come with built in funding. I'm probably dreaming but I've heard success stories from colleagues at University of Toronto.

    ReplyDelete
  11. Interesting post. Of course another reason that research findings can languish unpublished, is the certain knowledge that if submitted the paper will be returned with a raft of requests for additional experiments, with no-one or no funding available to do the work. This is another unfortunate consequence of our culture of short-term funding contracts - tax payer loses out because publicly funded research never gets into the public domain, post-doc/student loses out because they need papers to progress, and PI loses out because they need papers too. It's a lose-lose-lose situation.

    ReplyDelete
  12. I think the REF is exacerbating this problem. We have been told implicitly that 1* and 2* papers are indicative of "worthless" (i.e. unfundable) research, which means these papers are not factored into the assessment of an individual (or at least they are weighted far less). This massively reduces the incentive to bother publishing them. The REF focus on quality over quantity is laudable in one way but it does encourage waste.

    ReplyDelete
    Replies
    1. There's even worse than that: some institutions (not mine fortunately) calculate researchers' mean impact factor (not even mean citations!). That's a dreadful incentive: better not publish a paper than one that would have to go into a journal whose IF is below your current mean... yuk!

      Delete
  13. Again you are spot on Dorothy!
    I have recently come to just the same conclusion. I still have two grants running, but then I have vowed to stop applying for more and clear the backlog (which I also estimate to be 3-5 years).
    I have also realised that even when (if) my backlog is cleared, I could spend a lifetime clearing the backlog of others. Many people have collected beautiful data but have only partly exploited them. They are only too happy if you have good hypotheses to test on their data and if you therefore help them make the most out of it. Of course, I also know that sometimes new hypotheses really require new experiments.

    ReplyDelete