The Education Endowment Foundation comes under more criticism

This is the hub of the site and the place to post queries, start discussions and join in the conversation!
Post Reply
User avatar
Debbie_Hepplewhite
Posts: 2498
Joined: Sat May 23, 2015 4:42 pm

The Education Endowment Foundation comes under more criticism

Post by Debbie_Hepplewhite »

A fortune is being ploughed into the Education Endowment Foundation in England with the avowed intention:
“The EEF has so far funded 127 evaluations in 7200 schools across England to find the most effective approaches and interventions for improving pupil attainment”

Sir Kevan Collins, Chief Executive

What does the EEF do?

The Education Endowment Foundation (EEF) is an independent grant-making charity dedicated to breaking the link between family income and educational achievement, ensuring that children from all backgrounds can fulfil their potential and make the most of their talents.

We aim to raise the attainment of children facing disadvantage by identifying and funding promising educational innovations that address the needs of disadvantaged children in primary and secondary schools in England.

We share and promote the use of evidence in schools by providing independent and accessible information through the Sutton Trust-EEF Teaching and Learning Toolkit, and the Early Years Toolkit summarising educational research from the UK and around the world. Both Toolkits provide guidance for teachers and school leaders on how best to use their resources to improve the attainment of all pupils.

Our range of publications are an invaluable resource for school leaders and policy makers to inform decision making in schools.
https://educationendowmentfoundation.org.uk

The core questions include whether the findings of the EEF justify the cost, and whether the research projects are truly valid in the first place. Do the projects and their findings really take the education profession any further forwards?

A further core question is whether the money could be better spent 'for improving pupil attainment'.

Another question might be whether the research projects are, in some instances at least, simply trying to discover what we already know from previous research projects - like hamsters running on their wheel - getting nowhere!

Who, or which body, evaluates the work of the EEF to decide on its value for money and effectiveness?

A number of popular bloggers and tweeters are beginning to question at least some of the summaries of the EEF and, at this rate, and at some point, it may well be that the EEF needs to be investigated and held to account as to whether their projects are truly worthwhile and worth the money.

And could the work of the EEF, with all its corporate image and clout, actually detract from existing research findings and deflect the work of schools away from more important research findings and more important research projects?

Having been personally alarmed by the very weak and flawed description of 'phonics' of the EEF, and being aware of some bloggers' concerns, I've started this thread to build up a record.
User avatar
Debbie_Hepplewhite
Posts: 2498
Joined: Sat May 23, 2015 4:42 pm

Re: The Education Endowment Foundation comes under more criticism

Post by Debbie_Hepplewhite »

The 'Horatio Speaks' blog written by James Murphy has for some time raised questions about the summaries of the Education Endowment Foundation - here is one example:
Vested Interests?

OCTOBER 17, 2015 / HORATIO SPEAKS
https://horatiospeaks.wordpress.com/201 ... interests/
Apparently looking for two months’ additional progress per month of intervention is ‘absurd’. When researchers focus their expectations of what is ‘normal’ on statistical experience, and write about findings in this light, they do students a grave disservice. No amount of crying ‘vested interests’ makes a three-month gain ‘effective’ for struggling readers. Schools need to look beyond the bland headlines when researching interventions. The Education Endowment Foundation needs to think carefully about whether or not their current approach is providing the service to schools for which they are funded.
Note that within the postings of James, he links to other people raising questions about the summaries and practices of the Education Endowment Foundation.
User avatar
Debbie_Hepplewhite
Posts: 2498
Joined: Sat May 23, 2015 4:42 pm

Re: The Education Endowment Foundation comes under more criticism

Post by Debbie_Hepplewhite »

Here is James Murphy again, this time noting a possible:
A Convergence of Interests?

MAY 10, 2014 / HORATIO SPEAKS
https://horatiospeaks.wordpress.com/201 ... interests/
Consider, for example, the Education Endowment Foundation. The government has invested considerable funds via the EEF to run randomized controlled trials. One of the RCT evaluations recently released by the EEF was for a programme called Switch-On Reading. What is not apparent in the headline, but appears later in the report, is that the programme is in fact a repackaging of Reading Recovery, which is now being aimed at students at the transition between Key Stages 2 and 3.
Is there a pattern here? It seems obvious to me. Is it a conspiracy of some kind? A conspiracy is entirely unnecessary to bring about a favourable slant in a report on a weak set of data. But I do think there may be a convergence of interests here, where research findings of ‘the right sort’ can be presented in one way in the headlines, while the fine print says something rather different. It is all about who writes the reports and who they are aligned with. For example – and this is not an aspersion on character, but a question about sympathies between organisations – a researcher from the Institute for Evidence in Education was seconded to the Education Endowment Foundation last year – a researcher who described himself at the National Literacy Trust conference in January as ‘Reading Recovery trained’. Is there a connection between this and the mismatch of headline and detail in the EEF report? I can only ask the question, but the patterns I have seen suggest that strong links between research groups can lead to some strange outcomes.
Do read the whole piece - it is very thought-provoking indeed.
User avatar
Debbie_Hepplewhite
Posts: 2498
Joined: Sat May 23, 2015 4:42 pm

Re: The Education Endowment Foundation comes under more criticism

Post by Debbie_Hepplewhite »

Jim Thornton notes this about the Education Endowment Foundation summary on:
Teaching Philosophy in Primary Schools

JULY 14, 2015
https://ripe-tomato.org/2015/07/14/teac ... y-schools/
Still no tests of statistical significance (the lead author, Stephen Gorard, has some sort of principled objection to them) but their lack does not stop him concluding: “The results in tables 5 and 7 are unlikely to be due to chance.” On this basis the report’s first Key Conclusion, the primary finding of the trial, states:

“There is evidence that P4C had a positive impact on Key Stage 2 attainment. Overall, pupils using the approach made approximately two additional months’ progress on reading and maths.”

This sentence, plastered all over the Education Endowment Foundation website and press releases led predictably to the breathless headlines.

But it’s wrong. The triallists pre-specified two primary outcomes but only reported one, which showed no difference. They pre-specified seven secondary outcomes which showed no differences either. However when they altered their analysis plan after seeing the data they noticed that two of the secondary outcomes showed a tiny shift in mean change scores favouring the intervention. The effect size was about 10% of a standard deviation, and less than half the participants had the relevant scores measured, but who cares! Without any tests of statistical significance they declared that it was unlikely to have occurred by chance!

In an email to me Stephen Gorard wrote that he had no axe to grind. His research group Research Design & Evaluation (click here) had nothing to do with SAPERE or P4C; they had just been commissioned to evaluate the programme. He likened RD&E to a “taxi for hire”. Indeed so. Taxis get you where you want to go. RD&E gets you the results you want.
User avatar
Debbie_Hepplewhite
Posts: 2498
Joined: Sat May 23, 2015 4:42 pm

Re: The Education Endowment Foundation comes under more criticism

Post by Debbie_Hepplewhite »

And here is another post about the summary of the philosophy research via Andew Old's blog - a guest post by mingles:
Does teaching philosophy to children improve their reading, writing and mathematics achievement?

(Guest post by @mjinglis)

July 14, 2015
To introduce the post, Andrew writes:
I’ve been getting a bit concerned that the EEF’s evaluations of educational methods, which were meant to help provide a more solid evidence base for teaching, are actually leading to the same sort of unreliable research and hype that we have seen all too often in educational research. The following guest post is by Matthew Inglis (@mjinglis) who kindly offered to comment on a big problem with the recent, widely-reported study showing the effectiveness of Philosophy for Children (P4C).
https://teachingbattleground.wordpress. ... -mjinglis/

mjinglis concludes:
So what can we conclude from this study? Very little. Given the pre-test scores, if the P4C intervention had no effect whatsoever on reading, writing or mathematics, then this pattern of data is exactly what we would expect to see.

What is most curious about this incident is that this obvious account of the data was not presented as a possible (let alone a highly probable) explanation in the final report, or in any of the EEF press releases about the study. Instead, the Director of the EEF was quoted as saying “It’s absolutely brilliant that today’s results give us evidence of [P4C]’s positive impact on primary pupils’ maths and reading results”, and Stephen Gorard remarked that “these philosophy sessions can have a positive impact on pupils’ maths, reading and perhaps their writing skills.” Neither of these claims is justified.

That such weak evidence can result in a national newspaper reporting that the “best way to boost children’s maths scores” is to “teach them philosophy” should be of concern to everyone who cares about education research and its use in schools. The EEF ought to pause and reflect on the effectiveness of their peer review system and on whether they include sufficient caveats in their press releases.
User avatar
Debbie_Hepplewhite
Posts: 2498
Joined: Sat May 23, 2015 4:42 pm

Re: The Education Endowment Foundation comes under more criticism

Post by Debbie_Hepplewhite »

David Didau, via his 'The Learning Spy' blog, comments on the EEF review of written marking here:

http://www.learningspy.co.uk/assessment ... n-marking/
A marked decline? The EEF’s review of the evidence on written marking

May 18, 2016
David comments:
The only one of these statements that can reasonably be concluded from the flimsy research base the review’s authors unearthed is the finding that awarding grades seems to undermine the effects of written feedback. All the rest is speculation at best and unexamined, biased assumption at worst.

Let’s consider each claim in turn.
This drew the attention of personnel at the EEF and there was an immediate response to David's post, here:

https://educationendowmentfoundation.or ... -feedback/

The EEF concludes with this comment:
3. Thanks for reading

If the researchers and schools are to have a productive relationship – and at the EEF we firmly believe that they can do – then communication between them must be open and honest. It’s extremely helpful to get feedback – positive and negative – on our work and we look forward to receiving applications for trialling different marking approaches from teachers when the next funding round opens in June.
Perhaps, then, we should see a response from the EEF regarding the criticism of research projects which resemble Reading Recovery in all but name - see James Murphy's post above 'A Convergence of Interests'!
User avatar
Debbie_Hepplewhite
Posts: 2498
Joined: Sat May 23, 2015 4:42 pm

Re: The Education Endowment Foundation comes under more criticism

Post by Debbie_Hepplewhite »

Further, the EEF did respond to my criticism of their description of 'phonics' when I tweeted about my worries, but this was through private email exchanges and amounted to 'passing on my feedback' along with links to some phonics studies - but no further development that I'm aware of.

When I just checked out the EEF description I was concerned about, it is still there as per the original EEF description for 'phonics', see:

https://educationendowmentfoundation.or ... it/phonics

This is my (informal) review of the EEF description of phonics which I provide on another thread (what the EEF writes is within the quote marks in blue, what I say is in red:
Phonics

Phonics is an approach to teaching reading, and some aspects of writing, by developing learners’ phonemic awareness. This involves the skills of hearing, identifying and using phonemes or sound patterns in English. The aim is to systematically teach learners the relationship between these sounds and the written spelling patterns, or graphemes, which represent them. Phonics emphasises the skills of decoding new words by sounding them out and combining or ‘blending’ the sound-spelling patterns.

This is a very weak and inadequate description of the phonics in relation to what phonics provision should 'look like' in schools in England.
How effective is it?

Phonics approaches have been consistently found to be effective in supporting younger readers to master the basics of reading, with an average impact of an additional four months’ progress. Research suggests that phonics is particularly beneficial for younger learners (4-7 year olds) as they begin to read.

The statement above makes no sense. What does the figure 'four months' progress' actually mean - using what assessment/s and relative to what? Compare this 'four months' progress' statement with the progress made according to Dr Marlynne Grant's report of two studies with pupils of various profiles. Go to pages 6, 7 and 8 in the report below and look at the children's chronological ages compared to their reading ages (word level) and their spelling ages, using standardised tests, at the end of Reception, Year One and Year Two. I suggest that an analysis of results from phonics provision in the North East should take into account the findings from following a truly systematic synthetic phonics programme such as Dr Marlynne Grant's shows here:

http://www.rrf.org.uk/pdf/Grant%20Follo ... 202014.pdf

Teaching phonics is more effective on average than other approaches to early reading (such as whole language or alphabetic approaches), though it should be emphasised that effective phonics techniques are usually embedded in a rich literacy environment for early readers and are only one part of a successful literacy strategy.

What is meant by the 'alphabetic' approach? I query this because synthetic phonics as it is known in England is about teaching the 'alphabetic code'. This statement is therefore not clear.
For older readers who are still struggling to develop reading skills, phonics approaches may be less successful than other approaches such as Reading comprehension strategies and Meta-cognition and self-regulation. The difference may indicate that children aged 10 or above who have not succeeded using phonics approaches previously require a different approach, or that these students have other difficulties related to vocabulary and comprehension which phonics does not target.

This statement above is very misleading. It does not matter what is the age of the learner, if they have a weakness in alphabetic code knowledge and the blending (synthesising) skill, then that is the gap that must be addressed for the sake of their life-long literacy, learning, job prospects and self-esteem.

When people encounter an unknown word in reading material (unknown because they don't recognise the word and also it is not in their spoken language), then the only way to come up with a pronunciation for the new word is some form of phonics - that is, translating the letters, letter groups, and/or word chunks into sounds. The person can possibly deduce the meaning of the unknown word when it is presented in context, but without a pronunciation, that word cannot be added to the person's oral vocabulary. Thus, phonics knowledge is not an either/or scenario. If children have reached the age of 10 and they are struggling with their reading, then the teacher needs to assess whether their stumbling block is language comprehension (spoken language) or technical ability to lift new words off the page, or a combination of both (the Simple View of Reading diagram is very helpful for illustrating the relationship between the technical skills for reading - what ARE the words? and the language comprehension skills - what do the words MEAN?)


The Simple View of Reading and the Simple View of Writing:

http://www.phonicsinternational.com/The ... _model.pdf

Far too many teachers believe that 'phonics doesn't suit some children' or 'phonics hasn't worked so far, let's try something different' and this is a fundamentally flawed understanding about the processes involved in teaching reading and learning to read. It may well be that the 10+ year old pupil needs more intensive practice of reading, or that they need more opportunity for repetition to build up fluency and confidence - but it is not a case of 'who have not succeeded using phonics approaches'.

The EEF is misleading to state, 'these students have other difficulties related to vocabulary and comprehension which phonics does not target' because phonics provision should target vocabulary and comprehension -but a close look at schools in the North East would discover whether phonics provision fails to target vocabulary and comprehension or not. Bring on the report of findings in the schools.

For children over 10 'who have not succeeded using phonics approaches previously, it is misleading to suggest that they 'require a different approach' (meaning, 'not' phonics). Further, 'phonics approaches' is a very inadequate way to describe phonics provision and implies that it is adequate to consider all 'phonics approaches' under the same broad brush stroke. This is not true. We are in the era of identifying what it means to provide phonics in a rigorous, systematic, explicit and content-rich way. This is why the EEF is entirely misguided to consider that the job of research is over for investigating phonics programmes and their variations.

I observe phonics lessons as part of my work, and I can describe first hand that phonics programmes and phonics provision do not look at all the same from school to school or even from class to class in the same school.



Qualified teachers tend to get better results when delivering phonics interventions (up to twice the effectiveness of other staff), indicating that expertise is a key component of successful teaching of early reading.
So, again, what is the picture of phonics provision - for first-time and for intervention - in the schools in the North East of England? I agree that 'expertise is a key component of successful teaching of early reading' but, what is the situation in the schools themselves, where is the report of observations? What needs to be addressed in the schools? Does first-time teaching and phonics training need to be addressed for example before rolling out specific interventions as the answer to weak literacy?

How secure is the evidence?

Overall, the evidence base related to phonics is very secure. There have been a number of studies, reviews and

Several robust studies of phonics programmes in English have been published in recent years. The findings show that phonics programmes can be effective in English schools, but also underline the importance of high quality implementation. Recent evaluations of Switch-on Reading, a programme involving phonics components delivered by teaching assistants, and Fresh Start, showed that both had an average impact of three additional months’ progress. However two other programmes, both targeting struggling, older readers, did not improve reading outcomes.

The interventions should be focused on children's needs - and if the mindset of the EEF is that children older than 10 have not succeeded with 'phonics approaches', then this is very worrying if that is what the children, or some of the children, do indeed need - regardless of age.

People in my field are approached by Secondary school personnel all the time regarding learners of 11+ who are still in great need of high-quality phonics teaching and content. How can it be that such a huge, corporate research organisation such as the EEF is writing such weak statements about phonics, referring to 'four months' progress' without any explanation as to what this means, and suggesting that phonics is OK for early reading but that beyond 10, children need something else rather than the phonics approaches which they did not succeed with earlier?

What are the costs?

Overall, the costs are estimated as very low. The costs associated with teaching phonics arise from the need for specific resources and professional training. Evidence suggests that the effectiveness of phonics is related to the pupil's stage of reading development, so it is also important that teachers have professional development in effective assessment as well as in the use of particular phonic techniques and materials
.

The costs for children are very high if those who need the most rigorous phonics provision are not receiving it. If a region in England is looking particularly weak compared to others, then the first port of call would be to examine the phonics provision for reading and spelling instruction of the mainstream teaching and also of the existing intervention provision.

It may well be that the region provides high-quality, content-rich phonics provision in every school, and maybe the children are up to speed with their knowledge of the most complex alphabetic code in the world and their blending skills for reading and oral segmenting skills for spelling - such that they now need 'something else'.

If this is the case, where is the report to illustrate this transparently?

4
What should I consider?

Before you implement this strategy in your learning environment, consider the following:

Phonics can be an important component in the development of early reading skills, particularly for children from disadvantaged backgrounds. However, it is also important that children are successful in making progress in all aspects of reading including vocabulary development, comprehension and spelling, which should be taught separately and explicitly.

THIS IS A TOTALLY FLAWED STATEMENT. It's not that phonics 'can' be an important component in the development of early reading skills, phonics IS an extremely important component in the development of early reading skills.

Furthermore, if the EEF personnel were knowledgeable, they would know that a high-quality systematic synthetic phonics programme includes vocabulary development, comprehension and spelling. This does not preclude teachers from teaching vocabulary explicitly, and spelling additionally, outside of the main phonics lessons - and language comprehension should be part of every lesson, every day across the curriculum anyway.

It is plain wrong for the EEF to state, however, that the components of 'vocabulary development, comprehension and spelling....should be taught separately and explicitly' as if they are not part of phonics provision because they should be!

The teaching of phonics should be explicit and systematic to support children in making connections between the sound patterns they hear in words and the way that these words are written.

The teaching of phonics should be matched to children’s current level of skill in terms of their phonemic awareness and their knowledge of letter sounds and patterns (graphemes).


Yes - but the 'teaching and learning cycle' for phonics provision, certainly when provided through a reputable systematic synthetic phonics programme, should include much more than the EEF description above.

Phonics improves the accuracy of the child's reading but not the comprehension. How are you planning on developing wider literacy skills such as comprehension?



This statement above is flawed. Yes, phonics does improve the accuracy of the child's reading, and that in itself unlocks the child's language comprehension. Without being able to decode the sentence, 'the dog is black', the child cannot comprehend the sentence. If the child understands about 'dogs' and colours, then the relationship for reading is a combination of both technical decoding and existing language comprehension.

Of course teachers need to address children's language comprehension all the time, and how books 'work', but the whole tenor of this description of phonics and its relationship with language comprehension and spelling by the EEF is inadequate and misleading.


I have heard nothing further from the EEF and their, arguably, flawed description of phonics worryingly remains.

This very description of 'phonics' by the EEF gave some authoritative people in Australia the grounds to diminish the importance of schools in Australia implementing the 'systematic synthetic phonics' teaching principles. The EEF's weak and flawed description of phonics was quoted as an argument AGAINST the need for systematic synthetic phonics!

And here is what IFERI committee member, Gordon Askew, has to say about older learners who are not yet up to speed with reading. This directly challenges the EEF comments about 'older readers':

http://ssphonix.blogspot.co.uk/2014/12/ ... etely.html

Monday, 29 December 2014

And now for something completely different

I am often asked about the role of phonics in 'catch up'. Some learners are in KS2, KS3 or beyond and, sadly, have not yet got very far at all with mastering basic reading. Teachers and parents understandably want to know how best to help them, to start them on the reading journey, or at very least to enable them to become functional readers.

One of the pronouncements I hear most frequently in respect of these learners generally goes along the lines of: 'They have been doing phonics for years and it hasn't worked for them. Now they need to try a different approach,' or 'Phonics does't work for everyone. These kids obviously need something else.'

Unfortunately such thinking is a massive red herring, and can have disastrous results, depriving learners of the very teaching they most desperately need to achieve the desired 'catch up'.

There are two strong reasons for saying this.
User avatar
Debbie_Hepplewhite
Posts: 2498
Joined: Sat May 23, 2015 4:42 pm

Re: The Education Endowment Foundation comes under more criticism

Post by Debbie_Hepplewhite »

These comments from John Hattie are very interesting and relate to the work of the Education Endowment Foundation:
Leave research to the academics, John Hattie tells teachers
https://www.tes.com/news/school-news/br ... s-teachers
Teachers should avoid becoming researchers in their classrooms and leave the job to academics, according to one of education’s most influential professors.

Attempts to make teaching more research-based have taken off in England recently, with the success of the grass-roots researchED organisation and the UK government's funding of the Education Endowment Foundation (EEF).

But Professor John Hattie, director of the Melbourne Education Research Institute and one of the world’s most widely quoted education academics, is concerned that the movement should not lead to teachers trying to conduct their own research.

“Researching is a particular skill,” he told TES. “Some of us took years to gain that skill. Asking teachers to be researchers? They are not.

“I want to put the emphasis on teachers as evaluators of their impact. Be skilled at that. Whereas the whole research side, leave that to the academics.”

The New Zealand-born academic told TES he thought there was a danger that schools were trying to become too theoretical.

“They are more obsessed about how they ride a bike than whether they can ride a bike well,” he said.

But Kevan Collins chief executive of the EEF, which is currently working with Professor Hattie, said he did not want to get “hung up on divisions”.

“What we need to do is build the bridges and encourage a dialogue, because researchers need teachers and obviously teachers need researchers,” he said. “We have seen what happens when those two worlds don’t orbit together – basically nobody gets better.

Professor Hattie’s comments come just a fortnight come after another globally respected education academic, Professor Dylan Wiliam, warned that the idea of teaching as a “research-based profession” was “never going to happen”.

He wrote in TES that it was unrealistic to for teachers to ever expect to find credible research showing what action would lead to better outcomes for their own particular group of pupils.

Professor Hattie, whose 2008 milestone “meta-analysis” of 80,000 separate education studies is considered by some as a guide to what works in teaching, also argues that teachers need to concentrate more on the results of their actions on pupils and “ask about the worth and significance and merit of what they are doing”.

“Research often is about the causality, about how well you have implemented something,” he said. “I am much more interested about the impact.”

He was speaking after giving a talk organised by Pearson in London, in which he suggested that every school should appoint specialists in interpreting data.

“I think it’s totally unreasonable to ask teachers to be experts in everything,” Professor Hattie said, explaining that education authorities in New South Wales, Australia, were already putting the idea into practice.

“I don’t have any time for making teachers researchers,” he added. “We have got no evidence that action researchers make any difference to the quality of teaching.”

Dr Collins said: “Our studies now involve one in five schools in England and they work because teachers, teaching assistants and heads are willing to become active participants in the research question.

“Just because they not then crunching the numbers does not mean they are not involved. This should be seen as a collaboration.”

“Although it is right [that] primary research has to be rigorous and well established and it is hard to do that in one school, I actually don’t think that means that teachers shouldn’t be using the literacy and sensibility of research to inform their practice.”
Post Reply