Thursday 10 October 2013

On the need for responsible reporting of research to the media

This was one of the first tweets I saw when I woke up this morning :


In response, a parent of two girls with autism tweeted "gutted to read this. B's statement has been final for 1 yr but no therapy has been done. we're still waiting."

I was really angry. A parent who is waiting for therapy for a child has many reasons to be upset. But the study described on the BBC Website did NOT identify a 'critical window'. It was not about autism and not about intervention.

I was aware of the study because I'd been asked by the Science Media Centre to comment on an embargoed version a couple of days ago.

These requests for commentary on embargoed papers always occur very late in the day, which makes it difficult to give a thorough appraisal. But I felt I'd got the gist: the researchers had recruited 108 children aged between 1 and 6 years and done scans to look at the development of white matter in the brain. They also gave children a well-known test of cognitive development, the Mullen scales, which assesses language, visual and fine motor skills. It's not clear where the children came from, but their scores on the Mullen scales were pretty average, and as far as I can tell, none of them had any developmental disorders.

The researchers were particularly interested in lateralisation: the tendency to have more white matter on one side of the brain than the other. Left-sided lateralisation of white matter in some brain regions is well-established in adults but there's been debate as to whether this is something that develops early in life, or whether it is present from birth. In the introduction, the authors state that this lateralisation is strongly heritable, but although that's often claimed, the evidence doesn't support it (Bishop, 2013). A preponderance of white matter in the left hemisphere is of interest because in most people, the left side of the brain is strongly involved in language processing.

The authors estimated lateralisation in numerous regions of the left and right brain using a measure termed the myelin water fraction. Myelin is a fatty sheath that develops around the axons of cells in the brain, leading to improved efficiency of neural transmission. Myelination is a well-established phenomenon in brain development.

The main findings I took away from the paper were (a) myelin is asymmetrically distributed in the brains of young children, with many regions showing greater myelin density in the left than the right; (b) although the amount of myelin increases with age, the extent of lateralisation is stable from 1 to 6 years. This is an important finding.

The authors, however, put most focus on another aspect of the study: the relationship between myelin lateralisation and language level. Overall, there was no relationship with asymmetry of a temporal-occipital region that overlapped with the arcuate fasciculus, a fibre tract important for language that previously had given rather inconsistent results (see Bishop, 2013). However, looking at a total of eight brain regions and four cognitive measures, they found two regions where leftward asymmetry was related to language or visual measures, and one where rightward asymmetry was related to expressive and receptive language.

Their primary emphasis, however, was on another finding, that there were interactions between age and lateralisation, so that, for instance, left-sided lateralisation of myelin in a region encompassing caudate/thalamus and frontal cortex only became correlated with language level in older children. I found it hard to know how much confidence to place in this result: the authors stated that they corrected for multiple comparisons using false discovery rate, but if, as seems the case, they looked at both main effects and interaction terms in 32 statistical analyses, then some of these findings could be chance.

Be that as it may, it is an odd result. Remember that this was a cross-sectional study and that on no index was there an age effect on lateralisation. So it does not show that changes in language ability - which are substantial over this age range - are driven by changes in lateralisation of myelin. So what do the authors say? Well, in the paper, they conclude "The data presented here are cross sectional, longitudinal analysis will allow us to confirm these findings; however, the changing interaction between ability and myelin may be mediated by progressive functional specialization in these connected cortical regions, which itself is partly mediated by environmental influences" (p. 16175). But this is pure speculation: they have not measured functional specialisation, and, as they appear to recognise, without longitudinal data, it is premature to interpret their results as indicating change with age.

If you've followed me so far, you may be wondering when I'm going to get on to the bit about intervention for autism and critical periods. Well, there's no data in this paper on that topic. So why did the BBC publish an account of the paper likely to cause dismay and alarm in parents of children with language and communication problems? The answer is because King's College London put out a press release about this study that contained at least as much speculation as fact. We are told that the study "reveals a particular window, from 2 years to the age of 4, during which environmental influence on language development may be greatest." It doesn't do anything of the kind. They say: "the findings help explain why, in a bilingual environment, very young typically developing children are better capable of becoming fluent in both languages; and why interventions for neurodevelopmental disorders where language is impaired, such as autism, may be much more successful if implemented at a very young age. " Poppycock.

A few months ago the same press office put out a similarly misleading press release about another study, quoting the principal researcher as stating: “Now we understand that this is how we learn new words, our concern is that children will have less vocabulary as much of their interaction is via screen, text and email rather than using their external prosthetic memory. This research reinforces the need for us to maintain the oral tradition of talking to our children.” As I noted elsewhere, the study was not about children, computers or word learning.

I can see that there is a problem for researchers doing studies of structural brain development. It can be hard to excite the general public about the results unless you talk about potential implications. It is frankly irresponsible, though, to go so far beyond your data that the headline is based on the speculation rather than the findings.

I am tired of researchers trying to make their studies relevant by dragging in potential applications to autism, schizophrenia, or dyslexia, when they haven't done any research on clinical groups. They need to remember that there are real people out there whose everyday life is affected by these conditions, and that neither they nor the media can easily discriminate what a study actually found from speculations about its implications. It is the duty of researchers and press officers to be crystal clear about that distinction to avoid causing confusion and distress.

POSTSCRIPT
11/10/13: Dr O'Muircheartaigh has commented below to absolve the KCL Press Office of any responsibility for the content of their press release. I apologise for assuming that they were involved in decisions about how to publicise this research and have reworded parts of this blogpost to remove that implication.


References 

Bishop, D. V. M. (2013). Cerebral asymmetry and language development: Cause, correlate, or consequence? Science, 340 (6138) DOI: 10.1126/science.1230531

O'Muircheartaigh, J., Dean, D. C., Dirks, H., Waskiewicz, N., Lehman, K., Jerskey, B. A., & Deoni, S. C. L. (2013). Interactions between white matter asymmetry and language during neurodevelopment. Journal of Neuroscience, 33(41), 16170-16177. doi: 10.1523/jneurosci.1463-13.2013

 

8 comments:

  1. I agree this is bad reporting. I also know that is is *very* tempting when you are asked for a press release to add extra interpretation to a study - because if you don't mention developmental disorders or something scary, then the press will just ignore the study. And getting on the BBC is a great CV point, even if the story is rubbish. I've refused to play this game in the past, and I'm sure that has reduced my lab's press cover.

    ReplyDelete
  2. The sad thing is, fear mongering can be destructive to the parent-child relationship. Don't ask how I know...but until I got over the "fix my kid" stage, it was difficult to develop a loving, accepting response.

    Now, I did give my son hours of therapy at an early age out of fear, 15 years ago,that it would lead to full blown autism if I didnt. That was the push at the time. How so many adult autistics were raised without this info in the past, makes me wonder.

    But had I "let him come to me" in his own time and developmental readiness I'm not sure he might have done as well with a far less anxious mother.

    ReplyDelete
  3. As the first author and probable cause of the confusion I think I should reply. If there is fault, it’s mine for not choosing my words more carefully and not that of the press office.

    You are absolutely correct, there is a large amount of pressure on researchers to communicate their science, from funders and universities (I think it’s even a factor in the REF), and increasingly the journals themselves. As you’ve acknowledged in this blog in the past, this is a good and bad thing.

    It seems odd that you link this release to another from an entirely different group, irrespective of university. The press offices are using quotes, not reinterpreting. The researchers consent to a given press release.

    In any communications, I did spend quite a bit of time emphasising that this research involved typically developing children. However, the purpose of the lab is to longitudinally track brain development so as to have a normative dataset specifically to investigate, again longitudinally, developmental disorders. It is here that I (underlined) failed to emphasise strongly enough that this is future research.


    With regards to the speculation in the paper itself, in fairness, that was in the discussion. In context, it was fairly unambiguously speculation and is not part of the press release. It was written with respect to a long preceding paragraph. It is not written in the conclusion, which is below for completeness:

    “Using a large cross-sectional sample of infants and toddlers and a novel in vivo approach to the quantification of water myelin content, we demonstrate that, while cerebral asymmetry may be established and constrained at an early age, this asymmetry is related to language ability in children undergoing a crucial period of neuroanatomical development. Importantly, the coupling between asymmetrical structure and function changes during development and this flux in relations between anatomy and language stabilizes around the age of 4 years”

    (Jonathan O'Muircheartaigh)

    ReplyDelete
  4. Dear Dr O'Muircheartaigh,

    Thank you for providing such a clear account of events. I appreciate your frankness and I will add a note to the blogpost pointing readers to your comment.

    It was clearly wrong of me to assume that the Press Office was at fault, and I apologise to them, but it was a conclusion I came to having two such similarly problematic reports from this press office in a relatively short space of time. The fact that the other story concerned a different research group just increased my interpretation that the press office must be pressing researchers to add some speculation about relevance in order to make the story newsworthy. In my experience, most researchers don't want their work misrepresented and are cautious about going beyond the data - particularly where the media are involved.

    My own advice on this topic is that researchers in neuroscience should obey one simple ground rule when engaging with the media. Do not mention the words autism, schizophrenia, dyslexia unless your study has involved these clinical groups, and do not talk about implications for intervention or screening unless you have done studies on those topics.

    ReplyDelete
  5. Given that media reporting of research is almost uniformly bad with or without the help of an eager press office or lab under pressure, why do we persist in popularization efforts through the press? Blogs, direct outreach through education, YouTube, etc. Wouldn't those be better ways to go about things. Lazy journalists, venal headline writers and pressured researchers seem to make up a potent mix of confusion, disinformation and panic mongering that can do little to popularize science and research in any beneficial way (other than satisfying myopic funders).

    I normally ignore any neurobabble reporting in the press but this was retweeted by Steven Pinker so I clicked on the link expecting the worst. Given the vast amount of controversy around the critical period I had no doubt that no single paper could support a statement such as:

    "The brain has a critical window for language development between the ages of two and four, brain scans suggest."

    That is the summary of the article. And the only hint that this might be controversial is "it is too early to be confident about functional implications of the findings" by @deevybee herself. But what non-expert reader can interpret that as 'any talk about critical periods based on these results is irresponsible twaddle'?

    Perhaps experts should be commenting not on the original findings but on the actual reporting before it goes out. My expertise is on the functional side (linguistics) so my first steps were to this blog to see if I can confirm my suspicions. And as always, I will be sending others here when they ask about this.

    But the real lesson is: most research like this is not news. Long-term replication and falsification efforts leading to interventions is something worth reporting. But that does not happen on specific dates with embargoed press releases. So, I ask again, isn't it the whole popularisation-through-media paradigm that is broken? I am in favour of the opposite of Cartesian obscurantism but isn't this worse than writing scholarship in Latin to protect it from the masses?

    ReplyDelete
    Replies
    1. I largely agree with you. It's hard to think of good instances of accurate and responsible reporting of new research in the media, and I think there is just a mismatch between the incremental progress in science and the need for a 'breakthrough' by journalists. I blogged about this some years ago: http://deevybee.blogspot.co.uk/2010/06/journalists-and-scientific-breakthrough.html
      I think science is better suited to features than to news: there's plenty of interesting stuff out there, but it is seldom the result of one new paper. It would make more sense to have researchers work with journalists to produce articles that gave more of an overview of a developing area.

      Delete
    2. Dear Mr Lukeš and Dr Bishop,

      You've highlighted an interesting case here. I'm pleased that the author of the study commented clarifying a point that is lost in these cases: Press releases are written by university-employed staff. They would/should get fired if they spread misinformation under the name of the university or the researcher, without approval.

      Having said that I also share the frustration that churning stories from press releases is a really bad practice, and sadly too widespread. As a scientist-turned journalist I have had the chance to look at the industry from both sides. And the conclusion I come to is rather different now, than when I was in academia.

      There are journalists who do genuine work by looking at the actual papers and talking to relevant researchers. Most of the time they also get a good audience too.

      But it is unfair to expect every news organisation to live up to that standard. It's all to easy to think journalists are lazy and only care about a good story, even if facts need to be made up.

      The problems you highlight about journalism are true of science too. Peer review lets through a lot more mistakes than most scientists think (see: http://econ.st/1a0Y0wO). There are proposals that instead post-publication review should be the norm.

      I think the same should apply to journalism. If you think that a story is done badly, criticise it in the comments and don't promote it. Instead point out to your social network a place where the story had been covered correctly.

      Alternatively, take the time to write yourself. There are limitations here, of course. Dr Bishop does a fantastic job and has a wide reach, but not all do. That is why places like The Conversation (http://theconversation.com/uk), where I'm science editor, have sprung up. Here academics can work with professional editors to write for the public and then the editors take on the responsibility of promoting the "truth".

      All this is to say that the problem is real and many people recognise it, but that should not be used against popularisation of science which when done correctly has had a lot of positive impact on the society. The pressure from the government and funding bodies to communicate is based on that evidence. And instead of worrying about why scientists should bother, the question should be how best can the scientists address these malpractices.

      Delete
  6. One of these days we are going to wake up and realize that there is nothing inherently more interesting about neurobiology than about digestive biology. Personally, I find both fascinating, but I don't see why the general public would be particularly interested in either. As you say, neuroscientists are under tremendous pressure to make their research sound more exciting than it actually is. It isn't fair on them, and it isn't fair on the public that gets mislead.

    ReplyDelete