Friday, 24 February 2012

Neuroscientific interventions for dyslexia: red flags

I’m often asked for my views about interventions for dyslexia and related disorders. In recent years there has been a proliferation of interventions offered on the web, many of which claim to treat the brain basis of dyslexia. In theory, this seems a great idea; rather than slogging away at teaching children to read, fix the underlying brain problem. If your child is struggling at school, it can be very tempting to try something that claims to re-organise or stimulate the brain. The problem, though, is sorting the wheat from the chaff. There's no regulation of educational interventions and it can be hard for parents to judge whether it is worth investing time and money in a new approach.
My aim here is to provide some objective criteria that can be used. First, there is scientific evaluation: does the intervention have a plausible basis, and how has it been tested? Where claims are made about changing the brain, are they based on solid neuroscientific research? Second, there are red flags, some of which I listed in a previous post on ‘Pioneering treatment or quackery?” Here I've gathered these together so that there is a ready checklist that can be applied when a new intervention surfaces.

1. Who is behind the treatment and what are their credentials?
What you should look for here are relevant qualifications, particularly a higher degree (preferably doctorate) from an academic institution with a good reputation. Red flags are:
  • No information about who is involved ▶#1 
  • Intervention developed by someone with no academic credentials ▶#2 
  • Citation of spurious credentials; affiliation with organisations that have very lax membership criteria, e.g., Royal Society of Medicine ▶#3 
  • A lack of publications in peer-reviewed journals. Publications only in books counts as a red flag, because there is no quality control. ▶#4
It can be hard for a lay person to evaluate point #3, because some people cite qualifications that sound impressive but have no credibility. Academics in the field, however, can quickly identify whether a string of letters is indicative of prestige, or whether they are a smokescreen for lack of formal qualifications.
As far as #4 is concerned, relevant information can be obtained checking an author against a database such as Web of Science. However, access to such databases is largely restricted to academic institutions. Google Scholar is widely available, though its results are not restricted to peer-reviewed literature.

2. Is there a credible scientific basis to the treatment?
This is often difficult for a lay person to evaluate. Google Scholar may be helpful in tracking down articles that discuss the background to the intervention. Ideally, one is looking for a review by someone who is independent of those who developed it and who has good academic credentials. If no relevant journal articles are found on Google Scholar this is a red flag ▶#5. If a journal article is found, try to find whether the journal is a mainstream peer-reviewed publication.

3. Who is the intervention recommended for?

It is implausible that the whole gamut of neurodevelopmental problems has a single underlying cause, and it is unlikely that they will all respond to the same intervention. If an intervention claims to be effective for a host of diverse disorders, then this is a red flag ▶#6.

4. Is there evidence from controlled trials that the intervention is effective?
If there is such evidence, the main website for the intervention should describe it and provide links to the sources. No mention of controlled trials ▶#7, and heavy reliance on testimonials ▶#8 are both red flags. Chldren's progress should be measured on standardized and reliable psychometric tests, i.e. measures that have been developed for this purpose where normal range performance has been established. Failure to provide such information is another red flag ▶#9. It is not uncommon to find reference to trials with no controls, i.e. children’s progress is monitored before and after the intervention, and improvements are described. This is not adequate evidence of efficacy, for reasons I have covered in detail here: essentially, improvement in test scores can arise because of practice on the tests, maturation, statistical variation or expectation effects. If scores from before and after treatment are presented as evidence for efficacy, with no reference to control data, this is another red flag ▶#10, because it indicates that the practitioners do not understand the basic requirements of treatment evaluation.
If the evidence comes solely from children tested by people with a commercial interest, there may even be malpractice, with scores massaged to look better than they are. When there were complaints about an US intervention, Learning RX, ex-employees claimed that they had been encouraged to alter children's test scores to make their progress look better than it was (see comment from 6th Dec 2009). One hopes this is not common, but it is important to be alert to the possibility and to ensure those administering psychological tests are appropriately qualified, and if necessary get an independent assessment.
The strongest evidence for effectiveness comes from randomised controlled trials, which adopt stringent methods that have become the norm in clinical medicine. Where several trials have been conducted, then it is possible to combine the findings in a systematic review, which uses rigorous standards to evaluate evidence to avoid bias that can arise if there is ‘cherrypicking’ of studies. This level of evidence is very rare in behavioural interventions for neurodevelopmental disorders because the studies are expensive and time-consuming to do.

5. What is the attitude of those promoting the intervention to conventional approaches?
The question that an advocate for a new treatment has to answer is, if this is such a good thing, why hasn’t it been picked up by mainstream practitioners?
An answer that implies some kind of conspiracy by the mainstream to suppress a new development is a red flag ▶#11. This kind of argument is widely used by alternative medicine practitioners who maintain that others have vested interests (e.g. payments from pharmaceutical companies). This doesn’t hold water: basically, if a treatment is effective, then it makes no financial sense to reject it, given that people will pay good money for something that works.
Another red flag ▶#12 is if the new intervention is promoted alongside other alternative medicine methods that do not have good supportive evidence. This suggests that the practitioners do not take an evidence-based approach.


6. Are the costs transparent and reasonable?
Lack of information about costs on the website is a red flag ▶#13, especially if you can only get information by phoning (hence allowing the practitioner to adopt a hard sell approach). Is there any provision for a refund if the intervention is ineffective? If someone tells you their treatment has a 90% success rate, then they should be willing to give you your money back if it doesn't work. Another red flag is if you are asked to sign up in advance for a long-term treatment plan ▶#14. For example, in the case of the Dore programme, there were instances of families tied into credit agreements and forced to pay even if they don’t continue with the intervention.  

I’ll illustrate by applying the criteria to Sensory Activation Solutions. This is just one example of neuroscientific interventions on offer on the web. I've singled it out because I was recently asked my opinion after a new SAS Centre opened in Milton Keynes this month.
1. Who is behind the treatment and what are their credentials?
The SAS website states Sensory Activation Solutions (SAS) is the 'brainwave' of Steven Michaëlis and Kaśka Gozdek-Michaëlis and is the culmination of over 30 years of study and work relating to how we learn and how we can be more effective in life. I tried various approaches to search terms but was not able to find any publications by either person on Google Scholar. This is worrying: one would expect 30 years of study to yield some peer-reviewed papers. 
The biography of Steven Michaëlis does not mention any academic qualifications. He has a background in sound processing and computer technologies and has trained as a group counsellor. The website states that: Kaśka Gozdek-Michaëlis is an inter-faith, cross-cultural lecturer, writer, psychotherapist and life-coach with over 25 years experience. She gained a Master Degree in Oriental Studies at the prestigious University of Warsaw, Poland. She is the author of two books in Polish, 'Develop your genius mind' and 'Super-possibilities of your mind'.
Overall, the originators of the treatment are up-front about their background and do not hide behind spurious qualifications. However, neither of them appears to have any training in brain science or neurodevelopmental disorders, and their methods have not been subject to peer review. Two red flags:▶#2 ▶#4 

2. Is there a credible scientific basis to the treatment?
There were no entries in Google Scholar for "Sensory activation solutions", so I read the section on The science behind the SAS programs. This provided a quite complex story, about how playing sounds through headphones "activates the auditory processing centres in the brain... leading to less sensory overload, faster understanding, better verbal expression and improved reading and writing." It is a truism that playing sounds to people will activate auditory centres of the brain: that's what hearing is. The key question is whether the sounds used by SAS do anything special. There are numerous components to the SAS package, including use of vision, touch, taste, smell and proprioception "to reduce sensory overload." Sensory overload is a problem for some children, notably a subset of those with autistic spectrum disorder. But it's not generally viewed as problematic for children with dyslexia. It's also claimed that by presenting different auditory stimuli to the left and right ears, the SAS method can promote right-ear dominance and inter-hemispheric integration. In a video presentation, Michaëlis explains this aspect of the theory further, picking up on some old ideas about cerebral lateralisation, interhemispheric communication and rapid auditory processing. To those who don't know the literature, this will sound convincing, but his account is oversimplistic, and makes leaps from theory to intervention with no evidence. For example, with current methods of imaging it would be possible to test whether SAS stimuli alter children's cerebral lateralisation, but there's no indication of any studies investigating this. Overall, the account of the brain bases of dyslexia is out of line with contemporary neuroscientific research. One red flag: ▶#5

3. Who is the intervention recommended for?
SAS is described as appropriate for attention deficit disorder,  hyper-activity,  dyslexia, dyscalculia, hearing and speech disorders,  stammering,  autism,  Asperger's Syndrome,  Down Syndrome,  global developmental delay, Cerebral Palsy, eating disorders, sleeping disorders. In the video it is also recommended for acquired aphasia. One red flag: ▶#6

4. Is there evidence from controlled trials that the intervention is effective?
The "research" section of the website cites descriptive statistics only, largely based on parent satisfaction indices. There is no evidence that psychometrically sound measures were used to evaluate progress. 
There is a small scientific literature on Auditory Integration training (AIT), which has many features in common with aspects of the SAS package; most  studies focussed on autism, where there is little evidence of efficacy (Sinha et al, 2006). The American Speech-Language-Hearing Association concluded a review of AIT thus: Despite approximately one decade of practice in this country, this method has not met scientific standards for efficacy and safety that would justify its inclusion as a mainstream treatment for these disorders. Four red flags: ▶#7 ▶#8 ▶#9 ▶#10

 5. What is the attitude of those promoting the intervention to conventional approaches?
The 'resources' section of the website contained a wealth of information about other kinds of intervention, both mainstream and alternative. 

 6. Are the costs transparent and reasonable? 
The website was quite complicated to navigate, and I may have missed something, but I could not find any information about costs of treatment, only a phone number. It's not possible therefore to say if costs are reasonable. It seems unlikely that clients would be tied in to long-term contracts, as treatment duration seems quite short, lasting weeks rather than months. One red flag: ▶#13

Overall, you can see that SAS earns nine red flags on my evaluation scale.

I suspect no intervention is perfect, and if you have a child who is struggling at school you may want to go ahead and try an intervention regardless of red flags. My goal here is not to stop people trying new interventions, but to ensure that they do so with their eyes open. If practitioners make claims about changing the brain, then they can expect to have those claims scrutinised by neuroscientists. The list of red flags is intended to help people make informed decisions: it may also serve the purpose of indicating to practitioners what they need to do to win confidence of the scientific community.  

Reference
Sinha, Y., Silove, N., Wheeler, D., & Williams, K. (2006). Auditory integration training and other sound therapies for autism spectrum disorders: a systematic review Archives of Disease in Childhood, 91 (12), 1018-1022 DOI: 10.1136/adc.2006.094649

P.S. 6th March 2013: Here are some additional tips for spotting bad science more generally:

 

15 comments:

  1. I have read this article and find it very perplexing. I have so may mixed thoughts and feelings about the content and its focus.

    Do you honestly think you are going to find many if any interventions that meet the criteria you have stated to look for, and show little if any Red Flags?

    Seriously you say "If no relevant journal articles are found on Google Scholar this is a red flag ▶#5"

    Again another example of what i would consider in some ways an elitist view on things. There are many people out there working hard to come up with interventions, ideas, ways to work with and things to help those with say Dyslexia for example.

    People do not NEED essentially to have an academic background to be effective in a review or statement of its effectiveness.

    How can you make such a claim?

    What about parents who have found something effective for the child? What about children themselves who have said...This has been great for me?
    What about recommendations from other people who have nil monetary interest in the intervention or the fact of making a recommendation?
    Are you saying all those people would be RED Flags...because they are non academic?
    People who have real and sufficient reason to make a positive statement?

    I am glad that your last block of info was...I suspect no intervention is perfect, and if you have a child who is struggling at school you may want to go ahead and try an intervention regardless of red flags.

    I agree 100% you could forget almost 85% of your red flags and go with common sense, positive honest feedback on the intervention, how much you sense this person has a passion for helping, and last but not least How Hard this person may have had to have worked to over come the false and often misguided and restrictive barriers of the so called Scientific Community.

    Possibly your readers should be pointed back to one other of your articles namely this one: How to become a celebrity scientific expert! @ http://deevybee.blogspot.com.au/2011/09/how-to-become-celebrity-scientific.html

    I feel like you have just knocked almost 98% of the "People" out there who are working hard trying to help, with little resources or help, but massive amounts of passion and desire to get things improved and better than what the situation is right now.

    They are doing that despite all the money and resources that so called Scientific Experts have available and have applied to the problems.
    Lots of talk about Dyslexia from academics, but over the last 20 years very little effective results really for the high standing you are putting them in and there worth.

    There are still massive problems in our schools and teaching, there are still many many parents who gain little help or understanding, there are still many Dyslexics who do not know what to do or where to go...from what I can see and its not from behind my Rose Colored Glasses, a massive amount of the positive things being done for Dyslexics this last ten or twenty years has in fact come from those very same people you would be giving the red flags out to.

    Thank you for an interesting Article.
    David Dell
    Dyslexia Thinking Positive @ dd3.tv

    ReplyDelete
  2. Here is a red flag you missed. When success is measured by the increase in score compared to a base line test and the second test is the same base line test. I have actually read information on brain changing interventions where the increase in scores was greater by the people who did not actually try the intervention and yet the data was posted as part of the claim of success.I believe it was for the computer screen follow the light type of program.

    I sell a dyslexia intervention, See Right Dyslexia Glasses, that would get some of your red flags but I feel I have 2 redeeming factors.

    I only market to visual dyslexics that can describe visual problems that make reading difficult and clearly state that those without visual problems can not expect to benefit from a visual intervention.

    I offer a money back guarantee for any reason.

    Feel free to comment about whether or not my approach reasonably counters other red flags.

    For information about my little niche of visual dyslexia and See Right Dyslexia Glasses visit dyslexiaglasses.com .

    I had not considered the lack of a posted price before as being a red flag but I see your point and agree .

    ReplyDelete
  3. I completely agree with David Dell. As a mum who has struggled to get her child recognised as dyslexic at school - even though her dad is also dyslexic and they're 'two peas in a pod' - I think that parents have no choice but to try some of these things.

    My daughter has had Irlen Lenses since she was 5 (she's now 8 1/2) and we've seen a huge improvement in her since she's had them. We went for this option as soon as we could because she was telling us that 'the words were moving' and my husband was one of the first people in the country to get them (as his special needs teacher and author of multi-sensory literacy programme Beat Dyslexia became one of the first diagnosticians in the UK). He immediately began to be able to pass exams, read books for the first time and went on go get a degree and become a successful businessperson.

    We had to wait until she was 7 to get her officially diagnosed, and the woman who diagnosed her suggested 'sound therapy' which sounds similar to the programme that you reviewed. She been doing this now for almost a year, and is just about to complete the 'course' and again we have seen a great improvement (and she's enjoyed listening to music whilst relaxing in bed for 10 minutes every night).

    She's also followed Beat Dyslexia at school and tried EasyRead at home - both basic multi-sensory literacy intervention programmes delivered in different ways. Both were fun to do and boosted her confidence and self-esteem and highlighted areas where she was struggling and needed help.

    My philosophy in all of this is that I want to do everything I can for my child before she gets disillusioned, switched off and damaged by an education system that doesn't seem to recognise her needs and cater for her......and I blog about my experiences in order to try and help other parents who are going through the same things.......

    Lisa Holmes
    Blessedarethecheesebrains.com

    ReplyDelete
  4. Thank you to David, John and Lisa for putting some different points of view.
    Let me try to explain further where I am coming from.
    I am most definitely not saying that people who offer interventions with ‘red flags’ are all unprincipled snake-oil merchants. That would be both silly and offensive. I’m well aware that many people are offering these alternative interventions from the best of motives, and that many people are happy with them.
    I’m also not saying that you should only go for scientifically proven interventions. If that were the case, it would rule out most of what is done in the mainstream - such as the nice retired teacher up the road who is offering extra maths tuition - as well as what is on the fringes.
    My concern is with interventions that offer a pseudoscientific rationale for what they are doing, and try to impress people by claiming to understand how to fix brain problems. Most of these practitioners will argue that rather than helping a child to read, for instance, by teaching reading, you can do something else to alter the brain and fix the problem more directly. It would be lovely it it were true, but I know of no evidence that this works. And evidence is what I am about.
    Why aren’t I satisfied with positive endorsements from satisfied customers? Well, I realise that for many people, that is enough, and for those people I am not going to change their minds. But I’m all too well aware that these are a seriously unreliable guide to what works. There are loads of people who will cheerfully endorse homeopathy, crystal healing, or various nutritional regimes for cancer. Many perfectly intelligent people will decide they will try these, especially if they don’t think conventional methods have anything to offer them. One of the things you learn to do as a researcher, though, is NOT to trust your instincts, because you know how they can mislead you. It may seem very perverse, but there are numerous situations where the hard evidence indicates that you are kidding yourself. There’s a whole branch of psychology dealing with how we deceive ourselves, and I’ve been heavily influenced by it. See for instance, The Invisible Gorilla and Other Ways our Intuitions Deceive Us, by Chabris and Simons.
    And sadly, amidst all the genuine practitioners who want to help people, there are some snake-oil merchants who see desperate parents as a commercial opportunity. So what I am saying to people is be careful. Don’t believe all that people tell you, especially if they tell you they have a special method for changing the brain that will fix your child’s problems. Do a bit of research. If you do decide to go for a commercially available intervention, do it with your eyes open.

    ReplyDelete
  5. Another great article Dorothy, thank you.

    Here in the United States, there are a number of these heavily-promoted but supported only by testimonials "brain fixes" (Brain Balance, Learning Rx and more) .

    There is another rule of thumb, developed by Hurt, Arnold, and Lofthouse

    "In guiding patients, clinicians can apply the SECS versus RUDE rule: treatments that are Safe, Easy, Cheap, and Sensible (SECS) require less evidence than those that are Risky, Unrealistic, Difficult, or Expensive (RUDE). Two"

    ReplyDelete
  6. "In guiding patients, clinicians can apply the SECS versus RUDE rule: treatments that are Safe, Easy, Cheap, and Sensible (SECS) require less evidence than those that are Risky, Unrealistic, Difficult, or Expensive (RUDE). Two"

    I think this is very wise, Liz.

    Using the book from the library, Behavioral Intervention by C. Maurice, I gave my son $60,000 ABA therapy (language therapy only) for free, gave him eASE audio therapy for the cost of a new disc player and headphones. I believe both were very helpful, only one was given regard by scientific journals. The Orton-Gillingham method of multisensory teaching for LD kids doesn't seem to be studied at all on PubMed, but it is used prolifically for kids who can't make the grade via typical public educational methods.

    People are searching out answers just to be able to DO SOMETHING, do you understand, Dr. Bishop? As long as it doesn't cost an arm or a leg (as ABA does when not covered by insurance), why not try it?

    ReplyDelete
  7. "The Orton-Gillingham method of multisensory teaching for LD kids doesn't seem to be studied at all on PubMed, but it is used prolifically for kids who can't make the grade via typical public educational methods."

    I always advise parents to look into evidence-based methodologies in regard to educating their children. O-G generally incorporates the components of reading instruction that the National Reading Panal found to be effective, so I think it's safe to say that the approach has an evidence base. It's a little old and stodgy for me, but that's a story for another time. One more caveat, the "multisensory" piece that O-G practitioners love to talk about hasn't been proven. The International Dyslexia Association is hoping to look into this. My hunch is that it will not end up being a strong contributor to outcomes. We'll see.

    ReplyDelete
  8. Thank you, Dr. Shapiro. I would love to, in time, see the data regarding your hunch!

    And to Dr. Bishop...Thanks for including this "I suspect no intervention is perfect, and if you have a child who is struggling at school you may want to go ahead and try an intervention regardless of red flags."

    I was one of those who had to do -- something!

    ReplyDelete
  9. I still recommend evidence-based interventions. Just make sure that it is offered at the right intensity, for the right duration, and that the individual who is implementing the intervention has proper credentials, lots of experience, a good track record, and most important, that he or she monitors progress responsibly.

    ReplyDelete
  10. i've been online all evening trying to find how much sas charges, does anyone know? Also i've not found any feed back other than that linked to their site, i'd be interested to hear from anyone who has used them

    ReplyDelete
    Replies
    1. Home programme prices can be found here: http://uk.sascentre.com/page/sasathome and are between £ 119.95 and £ 199.95.

      Delete
  11. This is unbalanced. You should also award flags (green flags?) for things that make a treatment more plausible.

    ReplyDelete
  12. A response straight from the horse's mouth.

    I only just found this blog post on-line, nearly a year after its publication. A pitty that we were not informed about this or asked to comment at the time, but there we go!

    I set up Sensory Activation Solutions four years ago and still am very much involved in leading the team that develops the methodology. These are my comments on your red flags:

    1. Who is behind the treatment and what are their credentials?
    Yes, I like to be honest and up-front and certainly do not like to hide behind spurious qualifications. Please also take into account everyone involved in the development of a treatment. The Chief Executive of a hospital is not necessarily a brilliant surgeon and may even be unable to cut a roast chicken, but I would hope the surgeons he employs can. All 18 staff members and their qualifications are clearly detailed on our website - see: http://uk.sascentre.com/page/people.

    2. Is there a credible scientific basis to the treatment?
    "Overall, the account of the brain bases of dyslexia is out of line with contemporary neuroscientific research." I recommend you first read the research by Dirk J. Bakker as referenced on our Dyslexia page as this provides a very solid scientific basis - see: http://uk.sascentre.com/page/dyslexia

    3. Who is the intervention recommended for?
    "It is implausible that the whole gamut of neurodevelopmental problems has a single underlying cause, and it is unlikely that they will all respond to the same intervention. If an intervention claims to be effective for a host of diverse disorders, then this is a red flag." A single underlying cause can be irrelevant as aspirin, for instance, treats a wide range of conditions with many underlying causes. Your red flag, not mine.

    4. Is there evidence from controlled trials that the intervention is effective?
    Admittedly there is not yet much independent scientific research into the SAS method, however, we publish all available research to date and are actively promoting more trials. All details are published on our website - see: http://uk.sascentre.com/page/research. Furthermore we publish detailed Client Feedback, including contra-indications, here: http://uk.sascentre.com/page/statistics. As our method is quite different from AIT, we do not refer to AIT research anywhere.

    5. What is the attitude of those promoting the intervention to conventional approaches?
    The 'resources' section of the website contained a wealth of information about other kinds of intervention, both mainstream and alternative. See: http://uk.sascentre.com/page/links

    6. Are the costs transparent and reasonable?
    The costs of home programmes is clearly spelled out on this page: http://uk.sascentre.com/page/sasathome and starting at £ 8.33 per half-hour programme will be considered very reasonable by most. Centre based costs vary from country to country and from practitioner to practitioner and are therefore not published on-line.

    In fairness to the writer, we do update our website regularly and some information that is available now may not have been available at the time of writing.

    I am all in favour of more scientific research into new, alternative or complementary treatments. Keep things in perspective, however. Aspirin is a widely used medication considered safe under most circumstances. Should we worry therefore that there still is research in progress on its effects and is described as an "emerging science" on this Bayer website http://aspirinresearch.com/, that also states "Aspirin truly is a wonder drug" – does that warrant a red flag maybe ?

    Overall I quite agree that potential clients need to question and proceed with caution and like the post-script “... it may also serve the purpose of indicating to practitioners what they need to do to win confidence of the scientific community.”

    Comments or direct responses to me always welcome.

    Steven Michaelis – stevenm@sascentre.com.

    ReplyDelete
  13. On the credentials of teachers of dyslexic children, Jane Gaudie is head of the Chiltern Tutorial School for Dyslexics in Hampshire and has vast experience in her field.
    See her report on a recently completed experiment which should have caused more of a stir than it did. I wonder why?

    http://community.tes.co.uk/perceptual_learning_in_action/b/weblog/archive/2013/07/10/dyslexia-defeated.aspx

    ReplyDelete