A campaign by parents to keep their children off school as a protest against SATs prompted a Twitter discussion about the pros and cons of standardised tests. One teacher claimed that they’re important because they hold schools to account. I think that’s a misuse of standardised tests. First, because test results are a poor proxy measure of teaching quality. Second, good teaching (and hard work on the part of the student) are necessary but not sufficient conditions for good test performance. Third, using test results to hold schools to account overlooks the natural variation inherent in large populations.
test results as a measure of teaching quality
Tests such as the National Curriculum Tests (commonly known as SATs) GCSEs and A levels sample students’ recall and understanding of a particular body of knowledge – the KS2 curriculum, GCSE/A level course. The knowledge is sampled because testing the student’s knowledge of all the material in the course would be very time consuming and unwieldy. In other words, test results are a proxy for the student’s knowledge of the course material.
But the course material itself is a proxy for all that’s known about a particular topic. KS2 students learn basic principles about how atoms and molecules behave, GCSE and A level students learn about atomic theory in more detail, but Chemistry undergraduates complain that they have to then unlearn much of what they were taught earlier because it was the simplified version. So test results are actually a second order proxy for the student’s knowledge of a particular topic.
Then factors other than the student’s knowledge impact on test results. The student might be unwell on the day of the test, or might have slept badly the night before. In the months before the test they might have been absent from school for weeks with glandular fever or their parents might have split up. In other words, test results are affected by factors other than teaching and learning; factors beyond the control of either the school or the student. In other words, test results are a weak proxy for both the quality of teaching and the student’s knowledge.
good teaching and hard work are necessary but not sufficient for good test performance
There’s an asymmetry between the causes of high and low test results. It’s difficult to get a high test score without hard work on the part of the student and good teaching on the part of the school. But there are many reasons why a student might get a low score despite hard work and good teaching.
That’s at the individual level. Similarly at the school level it’s safe to conclude that a school with consistently good results in national tests is doing its job properly, but it’s not safe to conclude that a school that doesn’t get consistently good results isn’t.
The education system has been plagued over the years by two false assumptions about student potential. Either that all students have the potential to get good test scores and that good teaching is the key determining factor, or that students from certain demographic groups won’t get good test scores however well they’re taught. In reality it’s more complicated than that, of course. Students from leafy suburbs are more likely to do well in tests for many reasons; even if they are taught badly, they have access to resources that can sometimes compensate for that. Students from the kind of housing estate that motivates Iain Duncan Smith are at a higher risk of adverse life events scuppering their chances of getting good test results no matter how good the teaching at their school. And the older they get, the more adverse life events they are likely to encounter.
So, test results are a pretty good first order proxy for a student’s knowledge of course material. They are a not-so-good second order proxy for a student’s knowledge of the topic the course material represents. And only a weak proxy for quality of teaching.
life is just one damn thing after another*
Those in favour of standardised testing often cite cases of particular schools in deprived areas that have achieved amazing outcomes against the odds. Every child can read by the age of six, or is fluent in French, or whatever. The implication is that if one school can do it, all schools can. In principle, that’s true. In principle, all head teachers can be visionaries, all teachers can be excellent and all families can buy in to what the school wants to achieve.
But in practice life doesn’t work like that. Head teachers get sick, senior staff have to work part-time because of family commitments, local housing is unaffordable making recruitment a nightmare, or for many families school is just one more thing they can’t quite keep up with.
On top of that, human beings are biological organisms. Like all populations of biological organisms we show considerable variation due to our genes, our environment and interactions between the two. It might be possible to improve test performance across the education system, but there are limits to the improvement that’s possible. Clean water and good sanitation increase life expectancy, but life expectancy doesn’t go on increasing indefinitely once communities have access to clean water and sanitation. Expecting more than 50% of children in primary schools to perform above average simply shows a poor grasp of natural variation – and statistics.
standardised testing: what is it good for?
Standardised testing in primary schools makes sense. It samples children’s knowledge of key material. It allows schools to benchmark attainment. Standardised testing as a performance measure can alert schools to problems that are impacting on children’s learning.
However, the reasons for differences in students’ performance in standardised tests are many and varied. Performance will not improve unless the reasons for poor performance are addressed. Sometimes those reasons are complex and not within the schools’ remit. To address them local families might need better public services, better jobs or better housing – arguably not the core responsibility of a school. Poor teaching might not be involved at all.
However, successive governments haven’t used test results simply as broad indicators of whether a school is on track or whether there are problems that need to be addressed (not necessarily by the school), but as a proxy for teaching quality. Test results have been used to set performance targets and determine funding, regardless of whether schools can control the factors involved.
This shows a poor understanding of performance management§, and it’s hardly surprising that the huge amounts of money and incessant policy changes thrown at the education system over recent decades have had little impact on the quality of education of the population as a whole.
First posted 3 April 2016 here.
*A quotation attributed to Elbert Hubbard, an American writer who died when the Lusitania was sunk in 1915.
§ The best book I’ve read on performance management is a slim volume by Donald Wheeler called Understanding variation: The key to managing chaos. A clearly written, step-by-step guide to figuring out if the variation you’ve spotted is within natural limits or not. Lots of references to things like iron smelting and lumber yards, but still very relevant to schools.