2014 Reading Recovery study- can we believe the results?

This is the hub of the site and the place to post queries, start discussions and join in the conversation!
User avatar
Susan Godsland
Posts: 45
Joined: Sun May 24, 2015 1:32 pm

2014 Reading Recovery study- can we believe the results?

Post by Susan Godsland »

I was alerted to this* 2014 Reading Recovery (RR) study by a tweet from Prof. Dan Willingham
V. positive effects (d = .69) of Reading Recovery: 1st results from huge trial
*Year One Results from the Multisite Randomized Evaluation of the i3 Scale-Up of Reading Recovery
American Educational Research Journal: http://aer.sagepub.com/content/52/3/547.abstract?etoc

Here’s what Prof Diane McGuinness has to say about Reading Recovery ‘research’
Several years ago, a letter was sent to members of the U.S. Congress with 31 signatures of the top researchers in the field of reading urging Congress to suspend support for RR because independent research showed the method had no effect. It is extremely costly to implement, re teacher training, tutoring time, and materials. Not only this, but RR “research” is notorious for misrepresenting the data. In a recent publication by the Institute of Education, the same problems appear. 1. Nearly half of the children from the 145 strong “RR-tutoring group” were dropped from the study at post-testing, while the control group remained intact. (Barely a mention of this, and no attempt to solve the problem this creates.) 2. The RR group received individual tutoring, the control group got none. One could go on. The published paper bears the hallmarks of a bona fide “scientific” journal, until a closer inspection reveals it is published by Reading Recovery. No chance for an impartial peer review process here
http://www.publications.parliament.uk/p ... me1302.htm

I contacted Prof. James Chapman (IFERI committee member) for advice. He said that he was aware of the AERJ RR research and had discussed it with a fellow academic, Prof. Bill Tunmer.

James very kindly sent me the following 4 points re. this latest RR study:
Our problems with the study can be summarized in 4 points.

1. First, consider the control group. Clearly, RR is better than doing nothing, and according to footnote 4 (pp. 30-31), about a fourth of the control group children in the AERJ study received no intervention. The rest appeared to have received some small group intervention instruction, but no systematic data on the amount or type of instruction are reported. Clearly, one-to-one instruction in general is better than no instruction or small group instruction. A more important question is whether RR is superior to other forms of one-to-one intervention, both in terms of cost and outcomes (see this recent article by Lorraine Hammond who also makes this point): https://theconversation.com/there-are-m ... very-39574
A recent meta-analysis by Slavin et al. (2011) showed that the magnitude of different reading interventions was positively related to the amount of explicit instruction in phonics that was included in the intervention. Not surprisingly RR, which includes limited explicit, systematic instruction in alphabetic coding skills, came out last. Related to this point is research indicating that RR is more effective for children from code-oriented classrooms than from whole language classrooms (Center et al., 2001). Slavin et al. also reported that the effect size for RR (.24 I think) was similar to that for volunteer and largely untrained tutors.

2. As indicated on page 9 of the AERJ article, only 52.4% of the treatment children completed RR with 22.4% referred on. That is, more than one fifth of the struggling readers didn’t benefit much from RR, which is consistent with what others have reported. This percentage may be an underestimate, as another 20% of the children in the treatment sample didn’t complete RR for a variety of vaguely stated reasons. These findings are consistent with our analyses of annual RR monitoring data reported in New Zealand indicating that RR is not effective for those struggling readers who need the most help (i.e., who are most at risk of reading failure). In a chapter on RR in a soon to be released book on “Excellence and Equity in Literacy Education: The Case of New Zealand” (London: Palgrave Macmillan, June/July, 2015) we argue that the effectiveness of RR interacts with where children are located on the developmental progression from prereader to skilled reader. Our research clearly indicates that RR is not effective for those children at the lower end of the developmental continuum who need much more explicit instruction in phonemic awareness and alphabetic coding skills than what is typically provided in the standard RR lesson. Louisa Moats underscored this point in an article recently reported in the Melbourne Age newspaper:
http://www.theage.com.au/victoria/readi ... m8m9e.html

3. Providing support for the view expressed in the preceding paragraph is very strong evidence from research reported in New Zealand and Australia and the AERJ article indicating that the lowest performing readers (i.e., those most in need of extra support) are excluded from selection for RR. This issue is discussed at length in the AERJ article (see pp. 23- 26, 28), where it is explicitly stated that many schools “preferred to reserve Reading Recovery slots for students they regarded as more likely to benefit from the intervention” (p. 25). This widely adopted practice strongly suggests that the effectiveness of RR interacts with the extent to which children at the lower end of the developmental progression of reading acquisition are excluded from entering the program. In view of this consideration it is highly likely that the effect sizes reported in the evaluation of RR would have been significantly lower if the RR Standards and Guidelines (RRCNA, 2009) had been followed. These guidelines require schools to “use only the OS [i.e., scores from Clay’s Observational Survey] to select the lowest achieving first-grade students and to serve the lowest scorers first” (p. 24 of AERJ article). Given the rigidity of RR, the only way school systems can make it work is to exclude the lowest performing children, the ones for whom more systematic instruction in phonological skills is needed to enable them to achieve progress. However, given the theoretical underpinnings of the program, such instruction will never be available unless the program is modified. But then they will argue, as they did, that such modified RR programs are no longer RR.

4. The AERJ article states that RR “enables students to catch up to their peers and sustain achievement at grade level into the future” (p. 3). However, there is simply no convincing evidence in support of this claim. In the RR chapter of our upcoming book, we discuss three recent large-scale studies carried out in New Zealand indicating very limited positive maintenance effects for children who had participated in RR two to four years earlier. Slavin et al. (2011, p. 19) discusses studies carried out in the UK and US that report similar findings. We argue that the main reason for the failure to sustain positive outcomes from RR (among the successful/discontinued students) is straight forward: the theoretical underpinnings of the RR approach to reading intervention (i.e., the constructivist, multiple cues approach) were shown to be incorrect by the scientific community three decades ago.
James Chapman and Bill Tunmer have a book out soon. A full discussion of RR issues will be presented in one chapter.
http://www.palgrave.com/page/detail/?K=9781137415561
User avatar
Susan Godsland
Posts: 45
Joined: Sun May 24, 2015 1:32 pm

Re: 2014 Reading Recovery study- can we believe the results?

Post by Susan Godsland »

Renowned educational bloggers, Greg Ashman and 'Horatio Speaks', have also recently written about the Reading Recovery research:

Is Reading Recovery like Stone Soup?
https://gregashman.wordpress.com/2015/0 ... tone-soup/
Researchers from the Universities of Delaware and Pennsylvania have written a paper describing a large, multi-site, randomly controlled trial of Reading Recovery. The effect size is impressive: 0.69 when compared to a control group of eligible students. This is above Hattie’s effect size threshold of 0.40 and so suggests that we should pay attention. As a proponent of evidence-based education, you may think it perverse of me to question such a result.

It’s not
No stone unturned
https://horatiospeaks.wordpress.com/201 ... -unturned/
Just because Reading Recovery has a track record of inflating claims, and a poor record of solving reading problems in the long term, does that mean that everything about Reading Recovery is bad? It makes good sense to identify and give extra help to children who are falling behind, and who could argue with sessions which focus on “phonemic awareness, phonics, vocabulary, fluency, comprehension and composition”?
User avatar
Debbie_Hepplewhite
Posts: 2498
Joined: Sat May 23, 2015 4:42 pm

Re: 2014 Reading Recovery study- can we believe the results?

Post by Debbie_Hepplewhite »

There are grave, long-standing, international worries about the efficacy of the Reading Recovery intervention programme and its promotion and funding by various governments, universities and businesses.

'Reading Recovery' is now a hugely influential, entrenched, establishment, internationally-used intervention programme.

Historically, investment in Reading Recovery might have been understandable - in the days when Reading Recovery was a serious attempt to provide intervention for struggling readers that we now know to be 'instructional casualties' from the 'whole word' and 'whole language' eras in the teaching profession.

Reading Recovery, in effect, is a 'whole language' intervention programme that arose in a 'whole language' era.

Decades of research findings have shown us that the 'whole language' approach with its lack of explicit, systematic synthetic phonics teaching has led to many 'instructional casualties'.

We should have moved on from promoting and funding a 'whole language' intervention programme, but the sheer scale of Reading Recovery's entrenchment and funding internationally raises the question as to whether it is even possible to hold to account those in authority, across the world, who persist in maintaining the status quo.

Arguably, this is a travesty of enormous proportions.

Steadfastly, various researchers, and others, have endeavoured to unpick the flaws in the research papers based on Reading Recovery - and to raise the alarm with governments, and others, that keep funding and supporting its entrenchment to the possible detriment of learners themselves.

The pedagogy of Reading Recovery is also, arguably, serving to mis-train the teaching profession on a huge, international scale.

We now need to bring together the papers and literature which describe the flaws and misleading conclusions in the various Reading Recovery papers and say 'enough' - it is no longer accountable to persist with a funded whole language intervention when taking into account the body of research findings which has long since discredited the 'multi-cueing reading strategies'.

Recently, I opened the UK Reading Reform Foundation conference (March 2015) with a talk that raises this issue about contradictory policy and guidance to teachers in the context of England. This may complement the various technical reviews of researchers such as James Chapman, Bill Tunmer and Kevin Wheldall - and various rational bloggers such as Greg Ashman and 'Horatio':

https://www.youtube.com/watch?v=e5QT8kE ... e=youtu.be

It's nearly an hour long - so you may need a cup of coffee and patience to get to the conclusions at the end! ;)
User avatar
Debbie_Hepplewhite
Posts: 2498
Joined: Sat May 23, 2015 4:42 pm

Re: 2014 Reading Recovery study- can we believe the results?

Post by Debbie_Hepplewhite »

Here is more information about the 'Science and Technology Select Committee inquiry' which I touch upon in my RRF talk mentioned in the previous email for anyone who is interested:


http://www.phonicsinternational.com/for ... .php?t=586
Geraldine Carter
Posts: 1
Joined: Wed May 27, 2015 10:44 am

Re: 2014 Reading Recovery study- can we believe the results?

Post by Geraldine Carter »

Horatio Speaks has flagged up these interesting statistics from 3RsPlus:
Astonishing statistic in the comments on recent post re Reading Recovery. https://horatiospeaks.wordpress.com/201 ... omment-246
User avatar
Debbie_Hepplewhite
Posts: 2498
Joined: Sat May 23, 2015 4:42 pm

Re: 2014 Reading Recovery study- can we believe the results?

Post by Debbie_Hepplewhite »

I think it is relevant to get some parents' responses to this issue of Reading Recovery's continued promotion in schools.

In England, some parents are more knowledgeable about Systematic Synthetic Phonics and the differences compared to the multi-cueing reading strategies which underpin the Reading Recovery programme.

This old thread from the UK Reading Reform Foundation message forum provides a picture of several parents' frustrations and upsets when their children's teachers and schools persist with Reading Recovery and multi-cueing reading strategies.

http://rrf.org.uk/messageforum/viewtopic.php?f=1&t=6056

So, how do ordinary parents hold their children's schools to account for persisting with Reading Recovery and multi-cueing reading strategies?

This question remains to this day.

And, of course, this is not just about England, Reading Recovery and other multi-cueing programmes and practices dominate in the USA, Australia and New Zealand and other English-speaking countries.
User avatar
Debbie_Hepplewhite
Posts: 2498
Joined: Sat May 23, 2015 4:42 pm

Re: 2014 Reading Recovery study- can we believe the results?

Post by Debbie_Hepplewhite »

When it comes to the intervention programme, Reading Recovery, I often use the words 'entrenched, establishment programme' and I speculate whether it is SO entrenched that, as a world, we shall never be able to stop its use and proliferation, despite the review of its research, and despite its cost and despite the fact that there are much better, cheaper, research-informed programmes and practices in existence.

This is such a very serious issue, however, that we just have to keep on drawing attention to this issue and attempting to hold those people to account who keep on with its use, and funding its use, and mis-training and mis-guiding the world's teaching profession regardless.

So, if you want to get an some idea of the 'scale' of Reading Recovery - take a look at this:

http://oncampus.osu.edu/46-million-gran ... -recovery/
User avatar
Debbie_Hepplewhite
Posts: 2498
Joined: Sat May 23, 2015 4:42 pm

Re: 2014 Reading Recovery study- can we believe the results?

Post by Debbie_Hepplewhite »

Greg Ashman writes another good post to which I've added my comments about the context of the Education Endowment Foundation in England:

'The truth about teaching methods'

https://gregashman.wordpress.com/2015/0 ... omment-268

Please note that Greg includes a description of Reading Recovery's teaching method - see here:
But it is also possible to poorly design an RCT by varying a whole bunch of factors at once. I recently wrote about such an RCT evaluating a scale-up of Reading Recovery. In this case, the differences between the control and intervention groups were multiple and it is impossible to tell whether it is the specific Reading Recovery practices that caused the effect.

In my post on the research, I asked if other studies had been conducted on Reading Recovery that were better controlled. One person linked me to this paper where Reading Recovery was compared (amongst other conditions) to a strange version of direct instruction where the students hardly did any reading. If you have access, it is worth reading the full paper, particularly for its description of the Reading Recovery teaching method:

In this example, Dana is reading Nick’s Glasses (Cachemaille, 1982), an 8-page illustrated book about a boy who cannot find his glasses because he is wearing them. The text on page 6 says, “‘Have you looked behind the TV?’ said Peter.”

Dana read, “Have you looked under the….” She hesitated, glanced at the picture (which did not provide the needed information), and searched the line of print. Then she started over, ” ‘Have you looked behind the TV?’ said Peter.”

At the end of the page, her teacher quickly said, “I like the way you were checking carefully on that page. Show me the tricky part.” Dana pointed to the word behind, saying, “It had a b.” “Yes,” said the teacher, “Under would have made sense. He could have looked under the TV, but that word couldn’t be under. I also like the way you checked the picture, but that didn’t help enough, did it? You were really smart to use the first letter; read it again fast and be sure that it makes sense.”

Dana read the page again fluently, saying, “That’s right.” In this example, the teacher was pointing out to Dana how she effectively used several different sources of information simultaneously to monitor her own reading.

This seems like a poor method. Encouraging students to guess words from the pictures is problematic because it won’t help them to read books that don’t have lots of pictures in them. Phonics should be a first resort not a half-hearted last resort when guessing fails. In this instance, phonics was only employed in relation to the first letter of the word rather than for the decoding of the whole word.

This makes me even more skeptical that the recent positive result from an RCT was due to the specific Reading Recovery methods.
User avatar
Debbie_Hepplewhite
Posts: 2498
Joined: Sat May 23, 2015 4:42 pm

Re: 2014 Reading Recovery study- can we believe the results?

Post by Debbie_Hepplewhite »

Martin Kozloff wrote this summary of the folly of Reading Recovery 11 years ago.

What is so very telling is that Reading Recovery is still going strong - internationally - to this day.

It seems that there is nothing to stop the juggernaut - no amount of scientific findings and overt challenge to the Reading Recovery approach - and so teachers across the world continue to be mistrained and children continue to be mistaught:

http://www.iferi.org/wp-content/uploads ... covery.pdf
Reading Recovery: Anatomy of Folly

Martin Kozloff

2004

There are substantial problems with Reading Recovery that do not warrant its strong advocacy. Moreover, there are better (more effective and far less expensive) ways to accomplish the objective of trying to ensure that all students become proficient readers. With that in mind, here's an overview.
User avatar
Debbie_Hepplewhite
Posts: 2498
Joined: Sat May 23, 2015 4:42 pm

Re: 2014 Reading Recovery study- can we believe the results?

Post by Debbie_Hepplewhite »

Maggie Downie writes about the promotion and funding of Reading Recovery in England at the same time as the independent national 'Rose Review' was taking place. This was 2005 - 2006.

Teachers were clearly getting very mixed messages indeed as to 'what' approach to take for reading instruction - bear in mind that Reading Recovery to this day is based at England's 'Institute of Education' - firmly entrenched:

http://www.rrf.org.uk/archive.php?n_ID= ... eNumber=60
Reading Recovery (RR) is the trade mark of a remedial reading programme developed by Marie Clay in New Zealand in the 1970s. It is a whole language programme, developed in response to falling levels of literacy engendered by whole language teaching, and has a primary emphasis on reading for meaning. It has eschewed, or marginalised, the teaching of systematic, explicit phonics as part of learning to read. There has been a substantial amount of criticism, by the international reading research community, of the RR programme and the claims made for its effectiveness, e.g. Grossen, Coulter & Ruggles, Tunmer and Chapman and others [1]. Critics point out that RR cannot produce any independent research to back its claims.

Explicit phonics instruction

Nancy Salvato, in the US, reports [2] that Marie Clay, in 2001, refused to modify RR to include explicit, systematic phonics instruction in order to make it eligible for Reading First funding in the US. Instead, she stated that she would rewrite the description of RR’s components in such a way as to bring it into compliance with the Reading First legislation.
Conclusion:
The DfES has abandoned its original plan to ‘kitemark’ high quality phonics programmes, relying instead on the expertise of practitioners to apply the criteria and choose an appropriate programme. However, we can see no reason why the DfES should be ignoring its own criteria and actively promoting interventions which contravene them. The DfES refuses to name or promote high quality phonics programmes for initial reading instruction; it should maintain consistency by similarly refraining from naming or promoting intervention programmes. If it feels that it must alert teachers to intervention programmes it should confine its guidance to programmes in which the key element is high quality phonics and ensure that no other part of the programme contains elements which have been superseded by the Rose Report Findings.
My understanding is that the Government in England was somehow committed to funding Reading Recovery until 2014 despite Rose's recommendations being accepted by Government.

To this day, what are teachers in England to think?

They certainly still believe in multi-cueing reading strategies as I describe in my talk for the Reading Reform Foundation conference:

https://www.youtube.com/watch?v=e5QT8kE ... e=youtu.be
Post Reply