Monday, 21 July 2014

Percentages, quasi statistics and bad arguments


© www.CartoonStock.com

Percentages have been much in the news lately. First, we have a PLOS One paper by John Ioannidis and his colleagues which noted that less than one per cent of all publishing scientists in the period from 1996 to 2011 published something in each and every year of this 16-year period.

Then there was have a trailer for a wonderfully silly forthcoming film, Lucy, in which Scarlett Johansson suffers from a drug overdose that leads her to learn Chinese in an hour and develop an uncanny ability to make men fall over by merely pouting at them. Morgan Freeman plays a top neuroscientist who explains that whereas the rest of us use a mere ten per cent of our brain capacity, Johansson's character has access to a full hundred per cent of her brain.

And today I've just read an opinion piece in Prospect Magazine by the usually clear-thinking philosopher, A. C. Grayling, which states: Neuropsychology tells us that more than ninety per cent of mental computation happens below the level of awareness.

Examples like these can be used to demonstrate just how careful you need to be when interpreting percentages. There are two issues. For a start, a percentage is uninterpretable unless you know the origin of the denominator (i.e., the total number of cases that the percentage is based on).  I'm sure the paper by Ioannidis and colleagues is competently conducted, but the result seems far less surprising when you realise that the 'less than one per cent' figure was obtained using a denominator based on all authors mentioned on all papers during the target period. As Ioannidis et al noted, this will include a miscellaneous bunch of people, including those who are unsuccessful at gaining research funding or in getting papers published, those taking career breaks, people who are trainees or research assistants, those working in disciplines where it is normal to publish infrequently, and those who fit in  research activity around clinical responsibilities. Presumably it also includes those who have died, retired, or left the field in the study period.

So if you are someone who publishes regularly, and are feeling smug at your rarity value, you might want to rethink. In fact, given the heterogeneity of the group on whom the numerator is based, I'm not sure what conclusions to draw from this paper. Ioannidis et al noted that those who publish frequently also get cited more frequently – even after taking into account number of publications and concluded that the stability and continuity of the publishing scientific workforce may have important implications for the efficiency of science. But what one should actually do with this information is unclear. The authors suggest that one option is to give more opportunities to younger scientists so that they can join the elite group who publish regularly. However, I suspect that's not how the study will be interpreted: instead, we'll have university administrators adding 'continuity of publishing record' to their set of metrics for recruiting new staff, leading to even more extreme pressure to publish quickly, rather than taking time to reflect on results. A dismal thought indeed.

The other two examples that I cited are worse still. It's not that they have a misleading denominator: as far as one can tell, they don't have a denominator at all.  In effect, they are quasi-statistics. Since the publication of the Lucy trailer, neuroscientists have stepped up to argue that of course we use much more than ten per cent of our brains, and to note that the origin of this mythical statistic is hard to locate (see, for instance here and here). I'd argue there's an even bigger problem – the statement can't be evaluated as accurate or inaccurate without defining what scale is being adopted to quantify 'brain use'. Does it refer to cells, neural networks, white matter, grey matter, or brain regions? Are we only 'using' these if there is measurable activity? And is that activity measured by neural oscillations, synaptic firing, a haemodynamic response or something else?

In a similar vein, in the absence of any supporting reference for the Grayling quote, it remains opaque to me how you'd measure 'mental computation' and then subdivide it into the conscious and the unconscious. Sure, he's right that our brains carry out many computations of which we have no explicit awareness. Language is a classic case – I assume most readers would have no difficulty turning a sentence like You wanted to eat the apples that she gave you into a negative form (You didn't want to eat the apples that she gave you) or a question (Did you want to eat the apples that she gave you?) but unless you are a linguist, you will have difficulty explaining how you did this. I don't take issue with Grayling's main point, but I am surprised that an expert philosopher should introduce a precise number into the argument, when it can readily be shown to be meaningless.

The main point here is that we are readily impressed by numbers. A percentage seems to imply that there is a body of evidence on which a statement is based. But we need to treat percentages with suspicion; unless we can identify the numerator and denominator from which they are calculated, they are likely to just be fancy ways of trying to persuade us into giving more weight to an argument than it deserves.

Saturday, 12 July 2014

Should Vice-Chancellors earn more than the Prime Minister?

©CartoonStock.com

In my previous post about the university as big business, I wrote about the dangers that arise when an institution focuses only on its finances, and fails to foster its key resource - the academic staff. An obsession with the bottom line can lead to staff being treated as disposable commodities, to be hired and fired as convenient. But if a university wants to become renowned for its stellar teaching and research record, it needs to attract and retain academics who have a sense of loyalty to the institution, and will be proud to act as ambassadors for it. The ethos of the institution as a community of scholars requires that there is some continuity, and that members have a sense of common purpose. That's not going to happen if jobs are insecure and people are judged according to grant income rather than academic excellence. As senior management at King's College London (KCL) may be starting to realise, reputation is all-important and should never be compromised in seeking financial solutions. If you lose your reputation as a benign institution that cares for its staff and students, you'll make the financial situation worse rather than better, because you will be shunned by the staff and students you wish to attract.

Of course, universities need money to function, and a good university administration will keep a careful eye on finances. But the troubles start when income becomes an end in itself rather than a means to an end. There was a great piece by Richard Horton in the Lancet this week that said it all far better than I could, noting the chilling effect of the current obsession with league tables and amounts of grant income.

My focus here, though, is on an additional trend that is symptomatic of all that is wrong: the tendency for top university executives to be paid enormous salaries. The arguments used to justify these sums are similar to those used by bankers defending their bonuses. First, the job is demanding and involves responsibility for complex budgets and large numbers of people. Second, you won't be able to attract the right people unless you pay them this much. Neither point seems valid to me. The job of a vice-chancellor is complicated, but so is the job of the Prime Minister, whose salary is £142K. Unless you have an expensive drug habit or a penchant for yachts, you can live a very comfortable life on such an income - I speak as one who earns a very good salary well below that level.  The point was made forcibly by a group of four academics who put in a joint job application for the post of vice-chancellor at the University of Alberta, noting that the salary of Can$400K was sufficient to divide up and still give each of them a decent income.

There are various reasons why someone would demand more than £142K. Maybe a few of them do have an expensive drug habit or a penchant for yachts. It's more likely, though, that their sense of self-worth is tied up with how much they are paid, and their competitive nature means they need to demonstrate that they are better than their contemporaries. They'd be quite happy with £142K if everyone else earned £130K, but once they hear of someone else earning £150K, the iron enters into the soul. Or, like Scrooge, they may get satisfaction from seeing the money building up, even though they don't actually need it. The point is that none of these characteristics is an attractive personality trait, and I would question whether we should regard them as essential criteria when selecting a vice chancellor.

The arguments against paying vice-chancellors such huge sums are clear. First, the money could be better spent: why give an inflated salary to a vice-chancellor to squirrel away in the stock market, when you could use it to address the university's core mission: e.g., for student welfare, scholarships, fellowships or academic posts? There are also interesting psychological considerations. As discussed in a recent piece by Anne Manne, being ultra-rich alters how you see your fellow human beings. You no longer need to do any of the ordinary things that other mortals do, such as worrying about whether you can pay the mortgage, traveling on public transport, or saving for a holiday. You start to see yourself as a different kind of person with a sense of entitlement. It's a dangerous mindset for someone who has to make decisions that impact large numbers of employees.

The senior management at KCL have completely lost the confidence of their staff and students by their recent actions. They maintain that the redundancies that they plan are tough but essential: the only solution after considering all other options. Well, here's an idea that I first mooted in the Comments section of the Times Higher opinion piece that I wrote on the Kings College debacle.  They could voluntarily settle for lower salaries themselves. According to figures from the Times Higher, Rick Trainor, the Vice Chancellor earned a package worth £321K in 2013, compared to an average salary of £50K for non-professorial academic staff. Cutting the VC salary to £142K could provide funds that could save the jobs of three more junior academics*. We don't know how much the Vice-Principals and other senior academics earn - by coincidence, it emerged this week that someone had put in a freedom-of-information request, and KCL was very reluctant to reveal the figures. It seems likely, though that if they were willing to take a cut, a few more posts could be rescued. Overall, only a small proportion of the planned redundancies would be saved by such a step, but it would be a gesture that would demonstrate to staff that their senior management were sincere in taking every possible measure to save jobs, and that they placed the good of the institution above their own egos. And it might also set a precedent for other Universities, and reverse the trend for Vice-Chancellors to create a wedge between themselves and their academic staff by behaving like bankers when negotiating salary deals.

*In this case, it would show remarkable generosity of spirit if my advice was followed, since Trainor is leaving in a few weeks, and so it would be his successor - who no doubt has already negotiated a substantial salary deal - who would be affected.

Tuesday, 8 July 2014

Bishopblog catalogue (updated 8th July 2014)

Source: http://www.weblogcartoons.com/2008/11/23/ideas/

Those of you who follow this blog may have noticed a lack of thematic coherence. I write about whatever is exercising my mind at the time, which can range from technical aspects of statistics to the design of bathroom taps. I decided it might be helpful to introduce a bit of order into this chaotic melange, so here is a catalogue of posts by topic.

Language impairment, dyslexia and related disorders
The common childhood disorders that have been left out in the cold (1 Dec 2010) What's in a name? (18 Dec 2010) Neuroprognosis in dyslexia (22 Dec 2010) Where commercial and clinical interests collide: Auditory processing disorder (6 Mar 2011) Auditory processing disorder (30 Mar 2011) Special educational needs: will they be met by the Green paper proposals? (9 Apr 2011) Is poor parenting really to blame for children's school problems? (3 Jun 2011) Early intervention: what's not to like? (1 Sep 2011) Lies, damned lies and spin (15 Oct 2011) A message to the world (31 Oct 2011) Vitamins, genes and language (13 Nov 2011) Neuroscientific interventions for dyslexia: red flags (24 Feb 2012) Phonics screening: sense and sensibility (3 Apr 2012) What Chomsky doesn't get about child language (3 Sept 2012) Data from the phonics screen (1 Oct 2012) Auditory processing disorder: schisms and skirmishes (27 Oct 2012) High-impact journals (Action video games and dyslexia: critique) (10 Mar 2013) Overhyped genetic findings: the case of dyslexia (16 Jun 2013) The arcuate fasciculus and word learning (11 Aug 2013) Changing children's brains (17 Aug 2013) Raising awareness of language learning impairments (26 Sep 2013) Good and bad news on the phonics screen (5 Oct 2013) What is educational neuroscience? ( 25 Jan 2014) Parent talk and child language ( 17 Feb 2014) My thoughts on the dyslexia debate ( 20 Mar 2014)

Autism
Autism diagnosis in cultural context (16 May 2011) Are our ‘gold standard’ autism diagnostic instruments fit for purpose? (30 May 2011) How common is autism? (7 Jun 2011) Autism and hypersystematising parents (21 Jun 2011) An open letter to Baroness Susan Greenfield (4 Aug 2011) Susan Greenfield and autistic spectrum disorder: was she misrepresented? (12 Aug 2011) Psychoanalytic treatment for autism: Interviews with French analysts (23 Jan 2012) The ‘autism epidemic’ and diagnostic substitution (4 Jun 2012) How wishful thinking is damaging Peta's cause (9 June 2014)

Developmental disorders/paediatrics
The hidden cost of neglected tropical diseases (25 Nov 2010) The National Children's Study: a view from across the pond (25 Jun 2011) The kids are all right in daycare (14 Sep 2011) Moderate drinking in pregnancy: toxic or benign? (21 Nov 2012) Changing the landscape of psychiatric research (11 May 2014)

Genetics
Where does the myth of a gene for things like intelligence come from? (9 Sep 2010) Genes for optimism, dyslexia and obesity and other mythical beasts (10 Sep 2010) The X and Y of sex differences (11 May 2011) Review of How Genes Influence Behaviour (5 Jun 2011) Getting genetic effect sizes in perspective (20 Apr 2012) Moderate drinking in pregnancy: toxic or benign? (21 Nov 2012) Genes, brains and lateralisation (22 Dec 2012) Genetic variation and neuroimaging (11 Jan 2013) Have we become slower and dumber? (15 May 2013) Overhyped genetic findings: the case of dyslexia (16 Jun 2013)

Neuroscience
Neuroprognosis in dyslexia (22 Dec 2010) Brain scans show that… (11 Jun 2011)  Time for neuroimaging (and PNAS) to clean up its act (5 Mar 2012) Neuronal migration in language learning impairments (2 May 2012) Sharing of MRI datasets (6 May 2012) Genetic variation and neuroimaging (1 Jan 2013) The arcuate fasciculus and word learning (11 Aug 2013) Changing children's brains (17 Aug 2013) What is educational neuroscience? ( 25 Jan 2014) Changing the landscape of psychiatric research (11 May 2014)

Statistics
Book review: biography of Richard Doll (5 Jun 2010) Book review: the Invisible Gorilla (30 Jun 2010) The difference between p < .05 and a screening test (23 Jul 2010) Three ways to improve cognitive test scores without intervention (14 Aug 2010) A short nerdy post about the use of percentiles (13 Apr 2011) The joys of inventing data (5 Oct 2011) Getting genetic effect sizes in perspective (20 Apr 2012) Causal models of developmental disorders: the perils of correlational data (24 Jun 2012) Data from the phonics screen (1 Oct 2012)Moderate drinking in pregnancy: toxic or benign? (1 Nov 2012) Flaky chocolate and the New England Journal of Medicine (13 Nov 2012) Interpreting unexpected significant results (7 June 2013) Data analysis: Ten tips I wish I'd known earlier (18 Apr 2014) Data sharing: exciting but scary (26 May 2014)

Journalism/science communication
Orwellian prize for scientific misrepresentation (1 Jun 2010) Journalists and the 'scientific breakthrough' (13 Jun 2010) Science journal editors: a taxonomy (28 Sep 2010) Orwellian prize for journalistic misrepresentation: an update (29 Jan 2011) Academic publishing: why isn't psychology like physics? (26 Feb 2011) Scientific communication: the Comment option (25 May 2011) Accentuate the negative (26 Oct 2011) Publishers, psychological tests and greed (30 Dec 2011) Time for academics to withdraw free labour (7 Jan 2012) Novelty, interest and replicability (19 Jan 2012) 2011 Orwellian Prize for Journalistic Misrepresentation (29 Jan 2012) Time for neuroimaging (and PNAS) to clean up its act (5 Mar 2012) Communicating science in the age of the internet (13 Jul 2012) How to bury your academic writing (26 Aug 2012) High-impact journals: where newsworthiness trumps methodology (10 Mar 2013) Blogging as post-publication peer review (21 Mar 2013) A short rant about numbered journal references (5 Apr 2013) Schizophrenia and child abuse in the media (26 May 2013) Why we need pre-registration (6 Jul 2013) On the need for responsible reporting of research (10 Oct 2013) A New Year's letter to academic publishers (4 Jan 2014)

Social Media
A gentle introduction to Twitter for the apprehensive academic (14 Jun 2011) Your Twitter Profile: The Importance of Not Being Earnest (19 Nov 2011) Will I still be tweeting in 2013? (2 Jan 2012) Blogging in the service of science (10 Mar 2012) Blogging as post-publication peer review (21 Mar 2013) The impact of blogging on reputation ( 27 Dec 2013) WeSpeechies: A meeting point on Twitter (12 Apr 2014)

Academic life
An exciting day in the life of a scientist (24 Jun 2010) How our current reward structures have distorted and damaged science (6 Aug 2010) The challenge for science: speech by Colin Blakemore (14 Oct 2010) When ethics regulations have unethical consequences (14 Dec 2010) A day working from home (23 Dec 2010) Should we ration research grant applications? (8 Jan 2011) The one hour lecture (11 Mar 2011) The expansion of research regulators (20 Mar 2011) Should we ever fight lies with lies? (19 Jun 2011) How to survive in psychological research (13 Jul 2011) So you want to be a research assistant? (25 Aug 2011) NHS research ethics procedures: a modern-day Circumlocution Office (18 Dec 2011) The REF: a monster that sucks time and money from academic institutions (20 Mar 2012) The ultimate email auto-response (12 Apr 2012) Well, this should be easy…. (21 May 2012) Journal impact factors and REF2014 (19 Jan 2013)  An alternative to REF2014 (26 Jan 2013) Postgraduate education: time for a rethink (9 Feb 2013) High-impact journals: where newsworthiness trumps methodology (10 Mar 2013) Ten things that can sink a grant proposal (19 Mar 2013)Blogging as post-publication peer review (21 Mar 2013) The academic backlog (9 May 2013) Research fraud: More scrutiny by administrators is not the answer (17 Jun 2013) Discussion meeting vs conference: in praise of slower science (21 Jun 2013) Why we need pre-registration (6 Jul 2013) Evaluate, evaluate, evaluate (12 Sep 2013) High time to revise the PhD thesis format (9 Oct 2013) The Matthew effect and REF2014 (15 Oct 2013) Pressures against cumulative research (9 Jan 2014) Why does so much research go unpublished? (12 Jan 2014) The University as big business: the case of King's College London (18 June 2014)  

Celebrity scientists/quackery
Three ways to improve cognitive test scores without intervention (14 Aug 2010) What does it take to become a Fellow of the RSM? (24 Jul 2011) An open letter to Baroness Susan Greenfield (4 Aug 2011) Susan Greenfield and autistic spectrum disorder: was she misrepresented? (12 Aug 2011) How to become a celebrity scientific expert (12 Sep 2011) The kids are all right in daycare (14 Sep 2011)  The weird world of US ethics regulation (25 Nov 2011) Pioneering treatment or quackery? How to decide (4 Dec 2011) Psychoanalytic treatment for autism: Interviews with French analysts (23 Jan 2012) Neuroscientific interventions for dyslexia: red flags (24 Feb 2012)

Women
Academic mobbing in cyberspace (30 May 2010) What works for women: some useful links (12 Jan 2011) The burqua ban: what's a liberal response (21 Apr 2011) C'mon sisters! Speak out! (28 Mar 2012) Psychology: where are all the men? (5 Nov 2012) Men! what you can do to improve the lot of women ( 25 Feb 2014) Should Rennard be reinstated? (1 June 2014)

Politics and Religion
Lies, damned lies and spin (15 Oct 2011) A letter to Nick Clegg from an ex liberal democrat (11 Mar 2012) BBC's 'extensive coverage' of the NHS bill (9 Apr 2012) Schoolgirls' health put at risk by Catholic view on vaccination (30 Jun 2012) A letter to Boris Johnson (30 Nov 2013) How the government spins a crisis (floods) (1 Jan 2014)

Humour and miscellaneous Orwellian prize for scientific misrepresentation (1 Jun 2010) An exciting day in the life of a scientist (24 Jun 2010) Science journal editors: a taxonomy (28 Sep 2010) Parasites, pangolins and peer review (26 Nov 2010) A day working from home (23 Dec 2010) The one hour lecture (11 Mar 2011) The expansion of research regulators (20 Mar 2011) Scientific communication: the Comment option (25 May 2011) How to survive in psychological research (13 Jul 2011) Your Twitter Profile: The Importance of Not Being Earnest (19 Nov 2011) 2011 Orwellian Prize for Journalistic Misrepresentation (29 Jan 2012) The ultimate email auto-response (12 Apr 2012) Well, this should be easy…. (21 May 2012) The bewildering bathroom challenge (19 Jul 2012) Are Starbucks hiding their profits on the planet Vulcan? (15 Nov 2012) Forget the Tower of Hanoi (11 Apr 2013) How do you communicate with a communications company? ( 30 Mar 2014)

Wednesday, 18 June 2014

The University as big business:

The case of King's College London

 


© www.CartoonStock.com

King's College London is in the news for all the wrong reasons. In a document full of weasel words ('restructuring', 'consultation exercise'), staff in the schools of medicine and biomedical sciences, and the Institute of Psychiatry were informed last month that 120 of them were at risk of redundancy. The document was supposed to be confidential but was leaked to David Colquhoun who has posted a link to it on his blog.  This isn't the first time KCL has been in the news for its 'robust' management style. A mere four years ago, a similar though smaller purge was carried out at the Institute of Psychiatry, together with a major divestment in Humanities at KCL.

Any tale of redundancies on such a scale is a human tragedy, whether it be in a car factory or a University. But the two cases are not entirely parallel. For a car factory, the goal of the business is to make a profit. A sensible employer will try to maintain a cheerful and committed workforce, but ultimately they may be sacrificed if it proves possible to cut costs by, for instance, getting machines to do jobs that were previously done by people. The fact that a University is adopting that approach – sacking its academic staff to improve its bottom line – is an intellectual as well as a human tragedy. It shows how far we have moved towards the identification of universities with businesses.

Traditionally, a university was regarded as an institution whose primary function was the furtherance of learning and knowledge. Money was needed to maintain the infrastructure and pay the staff, but the money was a means to an end, not an end in itself. However, it seems that this quaint notion is now rejected in favour of a model of a university whose success is measured in terms of its income, not in terms of its intellectual capital.

The opening paragraph of the 'consultation document' is particularly telling: "King’s has built a reputation for excellence and has established itself as a world class university. Our success has been built on growing research volumes in key areas, improving research quality, developing our resources and offering quality teaching to attract the best students in an increasingly competitive environment." Note there is no mention of the academic staff of the institution. They are needed, of course, to "grow research volumes" (ugh!), just as factory workers are needed to manufacture cars. But they aren't apparently seen as a key feature of a successful academic institution. Note too the emphasis is on increasing the amount of research rather than research quality.

The most chilling feature of the document is the list of criteria that will be used to determine which staff are 'at risk'.  You are safe if you play a key role in teaching, or if you have grant income that exceeds a specified amount, dependent on your level of seniority.
What's wrong with this? Well, here are four points just for starters:

1. KCL management justifies its actions as key for "maintaining and improving our position as one of the world’s leading institutions". Sorry, I just don't get it. You don't improve your position by shedding staff, creating a culture of fear, and deterring research superstars from applying for positions in your institution in future.

2. The 'restructuring' treats individual scientists as islands. The Institute of Psychiatry has over the years built up a rich research community, where there are opportunities for people to bounce ideas off each other and bring complementary skills to tackling difficult problems. Making individuals redundant won't just remove an expense from the KCL balance sheet – it will also affect the colleagues of those who are sacked. 

3. As I've argued previously, the use of research income as a proxy measure of research excellence distorts and damages science. It provides incentives for researchers to get grants for the sake of it – the more numerous and more expensive the better. We end up with a situation where there is terrific waste because everyone has a massive backlog of unpublished work.
 
4. I suspect that part of the motivation behind the "restructuring" is in the hope that new buildings and infrastructure might reverse the poor showing of KCL in recent league tables of student satisfaction. If so, the move has backfired spectacularly. The student body at KCL has started a petition against the sackings, which has drawn attention to the issue worldwide.I urge readers to sign it.

Management at KCL just doesn't seem to get a very basic fact about running a university: Its academic staff are vital for the university's goal of achieving academic excellence. They need to be fostered, not bullied. One feels that if KCL were falling behind in a boat race, they'd respond by throwing out some of the rowers.