Sean Harford HMI National Director, Education
Correspondence regarding mathematics research review
Thank you for your letter of 12 July 2021.
Thank you for your recent correspondence regarding Ofsted’s mathematics research review. This letter responds to your specific concerns about that review. We want to assure you that we take all correspondence about our work seriously and have looked carefully at your full complaint. We note that your concerns cover the following areas:
§ the form and function of Ofsted’s mathematics research review does not conform with typical university protocols for literature/research reviews
§ (related to the above area of concern) individual citations and the extent to which they are matched to the claims made within the mathematics research review
§ the inclusion of research where small numbers of pupils/children are involved in the studies
§ the appropriateness of including international research/studies
§ the appropriateness of including research conducted more than ten years ago
§ a possible mismatch between the age groups of study subjects in research and age groups referred to in the body of the mathematics research review.
Regarding concerns about the overall form and function of Ofsted’s mathematics research review:
In your correspondence, you state that most teachers’ understanding of literature reviews is that they start with literature and a thesis is developed from contrasting viewpoints. Related to this, you explain that, instead of including, listing, discussing and summarising sources for and against concepts, Ofsted’s mathematics research review starts with a thesis and then seeks supporting literature. You state that because of the use of supporting sources, this may make the review’s evidence base appear stronger than it is.
In our published paper ‘Principles behind Ofsted’s research reviews and subject reports’ we outline the form and function of the mathematics research review. We did not set out to write conventional literature reviews. This was our reason for calling these documents ‘research reviews’ – because the term ‘research review’ is not generally used in any fixed way and it is simply descriptive in this context.
In terms of justifying conclusions drawn, the research review process acknowledges that research in the field of mathematics education is contested and contestable. We are not aiming to summarise the totality of educational literature; that would clearly be impossible. We have used several criteria to act as filters so that we can select the most relevant evidence. These are explained further in our principles paper and summarised in our ‘Education inspection framework: overview of research’. Within these parameters our process did include considering sources with contrasting viewpoints. However, our research review format does not outline these deliberations. This is because including this explanation would make the research review less accessible. One imperative for bodies such as Ofsted is for our work to be accessible and useful for the widest possible audience. We believe that our research review format provides that accessibility. There is a delicate balance between accessibility and transparency. This is the reason why we wrote a separate principles paper to explain our approach. We understand that a misapprehension has arisen around the use of some references and further explanation might have avoided this.
The process of putting together the review included theorisation of the conception of quality of mathematics education emerging from the findings. The review has gone through a thorough checking process, in line with the protocols established and shared in the principles paper.
Regarding concerns about individual citations not matching claims made in the review:
In your correspondence, you state that a large proportion of sources do not match the claims in the research review. We have sampled 35 papers from the index of the citations you identify. We have not found any examples of non-matched citations among the sample. Our analysis shows that, it is possible that these concerns may also have arisen as a result of some misinterpretation of the form and function of the mathematics research review. We are confident that the references we cited inform and support the statements made in the research review. However, the research review format does not allow more detailed explanation of the way each reference is relevant to related content. This is because the more detailed explanations that a formal literature review format allows would considerably reduce the accessibility of the piece. Below we have shared our response to selection of the sources you have queried.
Example 1, footnote 18: P Sahlberg, ‘PISA in Finland: an educational miracle or an obstacle to change?’, in ‘Center for Educational Policy Studies Journal’, Volume 3, Issue 1, 2011, pages 119 to 140
Commentary from AMET:
“The Finn and Sahlgren sources for footnote 18 match the point made but the Sahlberg source does not. Sahlberg gives an overview of Finland's performance that does not support the points made in the other two sources. i.e. The others claim that previous traditional schooling was responsible for Finland's good results, but this overview of international comparisons show that they were just average previously (except in reading where they've been consistently high) and became top after the reforms.”
Commentary from Ofsted:
The source is one of three sources cited as part of a paragraph in the following introductory section of the research review:
“It is also important to consider that high attainment and proficiency of older pupils may be due to historical curricular and pedagogical approaches, rather than the educational approaches of that time. Finland is a good example of an education system where success in the OECD’s Programme for International Student Assessment (PISA) is thought to be the result of historical approaches18.”
This paragraph illustrates the extent to which observations reveal the outcome of historical (rather than immediate) input and approaches. While the Sahlberg source does discuss the general role and impact of PISA results on the Finnish educational system, the following section of the Sahlberg source, in which ‘progress’ can also be attributed to ‘success’, is referenced because it informs the claim that attainment may be due to historical approaches:
“It is noteworthy that student achievement in Finland also consistently demonstrates progress according to PISA data, unlike several education superpowers. It is important to note that any effects that teaching may have on these results in given education systems have been influenced primarily by education policies and reforms implemented in the 1990s, not by the most recent education reforms.”
Based on the above, the citation does inform and support our findings.
Example 2, footnote 26: PA Alexander, ‘The development of expertise: the journey from acclimation to proficiency’, in ‘Educational Researcher’, Volume 32, Issue 8, 2003, pages 10 to 14
Commentary from AMET:
“greater focus on motivation and problem solving p.12 ‘...students must be encouraged to modify and combine strategies in ways that fit them and the problems at hand. Third, even though knowledge and strategies remain keys to expertise, my colleagues and I have found that individuals’ investment in their learning and development is equally critical. (e.g. Alexander & Murphy, 1998). We have determined that interest, especially individual interest, is tied to students’ knowledge and strategic efforts. If the educational experience is too narrowly focused on the acquisition of domain-specific knowledge, without regard to motivational forces, we may be stressing one aspect of expertise to the detriment of others. Thus, schools can do much to nurture emerging competence by allowing students to pursue topics and tasks of interest and by immersing them in meaningful learning experiences that are fertile ground for the growth of enduring interest.’ It does say that in acclimation students will need some explicit instruction. The article focused on four subjects but this did not include mathematics.”
Commentary from Ofsted:
In our research review we state that:
“The mathematics curriculum is the product of careful selection, sequencing and linking of declarative, procedural and conditional knowledge. Pupils need to systematically acquire core mathematical facts, concepts, methods and strategies to be able to experience success when problem-solving and in order to become proficient mathematicians26.”
Your colleagues note the section in which the author asserts that individual interest is tied to students’ knowledge and strategic efforts and then goes on to make the claim that learners should choose a path of interest. However, on the same page the following finding is shared: ‘knowledge and strategy changes in competent learners are linked to increases in individuals’ personal interest’. This finding suggests that the causal relationship might also work the other way. This finding is particularly important given that the conception of quality is intended to be relevant to schools and the education of very young children, many of whom do not have the requisite knowledge to know and choose all the best possible paths of interest available to them. It is useful to note at this point that the paper in question is cited in the research review section about the importance of (knowing) curriculum content. On page 11 of Alexander’s paper it states ‘…the ability to apply strategies efficiently and effectively is linked to individuals’ base of domain specific knowledge’. The statement is, therefore, about the importance of knowing curriculum content, hence the inclusion of the source paper, but this does not mean it is implying that pedagogical aspects are not important.
Regarding the comment that the paper focused on four subjects but did not include mathematics, we found the author notes that the themes of the paper (for example, the value of domain-specific knowledge) have been upheld in subject areas beyond the four stated in the abstract (social studies, astrophysics, human biology/immunology, educational psychology, special education, history, technology, music therapy and physical education). Based on these observations, we agreed with the author’s generalisation that the path from novice to expertise (proficiency) should feature the acquisition of domain-specific knowledge. We have surmised that these conclusions should also apply to mathematics education. The paper was included because, together with other citations, it demonstrated the importance of content acquisition. This citation also provided a useful insight into the importance of ‘strategies’ needed as part of the content acquisition process.
Based on the above, the citation does inform and support our findings and the associated messages in the review.
Example 3, footnote 28: J Bransford and others, ‘How people learn: brain, mind, experience and school’, National Academy Press, 2004
Comment from AMET
“This section is not about sequencing at all but about expert recognition of key features and patterns cf novice. Pages 32 to 36”
Comment from Ofsted
The source, which is one of two, informs the following section of the research review:
“Careful sequencing of content, instruction and rehearsal can also show pupils new and consistent patterns of useful information. These then form the basis of further concepts, rules and principles that pupils can store in their long-term memory28.”
This paper was informative because it describes the way in which depth and breadth of domain-specific knowledge, when stored in long-term memory, enables the proficient mathematician to ‘see’ more than the novice. The recommendation, in the source paper, at the end of the section entitled ‘Meaningful patterns of information’ was particularly pertinent to the claims we made in the research review. The paper states:
“Research on expertise suggests the importance of providing students with learning experiences that specifically enhance their abilities to recognise meaningful patterns of information.”
The recommendation and the information in this section of the chapter demonstrate the additional benefits that may arise from careful sequencing of content, including that which enables pupils to see new and consistent patterns of useful information. The other paper which informs this claim and which is also attached to footnote 28, gives more description about what these ‘learning experiences’ might look like. Specifically, the paper mentions the way that ‘concomitant variation’ can be planned into massed and distributed practice. This type of variation can help pupils to confirm causal relations, enable the abstraction of concepts and then facilitate generalised understanding. The two papers inform the claim made in the research review.
Based on the above, the citation does inform and support our findings.
Example 4, footnote 31: ‘Learning with understanding: 7 principles’, in ‘Learning and understanding: improving advanced study of maths and science in U.S. high schools’, National Research Council, 2002
Comment from AMET
“The paper does say prior knowledge is important but does not match the statement here. It specifically says disconnected facts are pointless.”
Comment from Ofsted
The source, which is one of three sources that are cited in footnote 31, informs a statement in a section of the review describing the importance of early and thorough emphasis on core content:
“Foundational knowledge, particularly proficiency in number, gives pupils the ability to progress through the curriculum at increasing rates later on 31.”
We did not find that the paper stated that ‘disconnected facts are pointless’. Instead, on page 199, the paper states that ‘knowing many disconnected facts is not enough’ and goes on to describe the way in which expert knowledge (which includes ‘facts’) is ‘conditionalised to specify the context in which it is applicable’. It is also useful to note that the research review does not claim that pupils should learn ‘disconnected facts’. We found that this particular source, although more general in its interpretations about the importance of prior knowledge, offers up a number of mathematics-specific insights. For example, on page 121:
‘To be successful in advanced study of science or mathematics, students must have acquired a sufficient knowledge base that includes concepts, factual content and relevant procedures on which to build’.
The suggestion here is that without foundational knowledge, pupils would struggle to make progress, particularly if they had to re-learn foundational knowledge before being able to move on. The use of the word ‘proficiency’ in the research review speaks to the nature of knowing, understanding and being able to apply, rather than simply knowing ‘disconnected facts’.
Based on the above, the citation does inform and support our findings.
Example 5, footnote 94: National Mathematics Advisory Panel, ‘Foundations for success: the final report of the National Mathematics’
Comment from AMET:
“It did not say that fluency must come before problem solving.”
Comment from Ofsted:
The source, which is one of three sources that are cited for footnote 94, informs a statement in a section of the review about the importance of a curricular approach to sequencing conditional knowledge. The following is the statement in the review that the footnote is attached to:
“Pupils need to be fluent with the relevant facts and methods before being expected to learn how to apply them to problem-solving conditions 94.”
Fluency, here, refers to the combination of speed and accuracy of recall and deployment of mathematics facts and methods. The paper explores the necessity of learning component knowledge prior to learning composite skills. The following extract from page XIX sums up some of the findings from the paper and informs our claim about needing a degree of familiarity with useful facts and methods before being expected to use them to solve more complex problems:
“…taken together, conceptual understanding of mathematical problems, fluent execution of procedures and fast access to number combinations jointly support effective and efficient problem solving”.
The paper suggests there are prerequisites for problem-solving success which include pupils’ knowledge and use of facts and methods. This message is repeated in the other two papers which inform the claim at this point: Decker and Robert’s paper specifically stated ‘basic calculation skill was a significant predictor of math problem solving’ (your letter to us stated this source was an incomplete match because it did not look exclusively at calculation skills, but also at other variables) and Zhang et al’s paper also notes the importance of ‘math fact fluency’ (among other factors) for problem-solving and, in the main body of the paper, discusses these factors further, including the importance of knowing ‘arithmetic procedures’. The three papers each contribute to the claim made in the review at this point.
Regarding a concern about the inclusion of sources for research studies where small numbers of pupils are involved:
In your correspondence, you have highlighted a concern that the review doesn’t discuss, in the text, where sources have small numbers of study subjects. We appreciate your acknowledgement that we have not reduced the review to simply what can be measured by randomised-controlled trials and have avoided dismissing other types of knowledge. Similarly, we are aware that different kinds of evidence will have different weights when used to inform and support claims. Regarding footnote 166 that you highlighted as an example of this concern, we do appreciate that this study standing alone is less robust. However, we do not think the claim that pupils who have mastered mathematical components are less likely to engage in disruptive behaviours is a controversial statement. This is given the weight of evidence already provided in the review around the importance of securing fluency in mathematics to make effective progress. The totality of the Gilbertson et al. paper – which includes the prior studies they reference and the methodology they replicate – provides further compelling evidence to this effect, despite the small number of pupils involved in the research.
Regarding a concern about the appropriateness of including international research/studies:
In your correspondence to us, you highlighted that there were many more citations from US sources than UK sources. Related to this, you highlighted that the US does not perform well internationally and that their education system is different to ours.
Our search strategy was not limited to any country and was guided by the search strategy set out in our principles paper. It is the case, that we were limited by what we could access in English, but overall, we used research from a range of countries. We would hope that countries who perform better than England would still see our good quality educational research as robust and useful, irrespective of our international ranking. Equally, where research in the US evidences effective practice, for example that concerned with the attainment and progress of disadvantaged children and children with SEND, that does not mean it is put into practice widely in US schools, from which pupils are drawn for international comparisons, such as the OECD’s PISA tests.
Regarding concerns about the use of sources that are more than ten years old:
In your correspondence, you highlighted that over 50% of sources were more than ten years old. Related to this, you stated that as the national curriculum is less than ten years old, use of these sources would result in discussing non-existent practices.
Our search strategy focused primarily on research conducted since 2010. However, where there were seminal works or we could not find good quality literature on a specific research question, we went further back. Good quality being defined by our principles paper.
Regarding concerns that age groups described in sources do not exactly match age groups described in the content of the research review:
In your correspondence to us, you highlighted the use of studies for specific year/age groups used to generalise for wider age groups beyond those in the source. You pointed out a paper used for footnote 46 as an example of this type of misuse. You stated that this US source, which you identified included six- to eight-year-olds as study subjects, was cited for a paragraph in the research review that included the phrase ‘start of academic journey’. Related to this, you pointed out AMET’s view that the start of an academic journey in England was ‘nursery or reception classes’, i.e. three- to five-year-olds.
The review acknowledged that different education systems do not use the same age groups or nomenclature for different stages of education as England. However, where claims are made about phases of education, alignment/overlap has been sought. The example you referred to relates to a statement about the ‘start of the academic journey’ as a period where ‘maths anxiety’ can manifest. For some, the ‘start of the academic journey’ can be at any point in their lives. However, the source paper was for a study of children in grades one to two in the US (mostly six- to seven-year-olds) and there is not much disparity between the source age group and the age group being referred to in the research review. This is because the research review states that the conception of quality outlined is applicable to reception year onwards (not nursery stages) and the fact that maths anxiety in US grade one is likely to have some of its origins in kindergarten, which is approximate to reception year and is, in many states, compulsory.
Regarding correspondence from authors of cited studies:
With regard to your correspondence highlighting to us that authors of cited studies have contacted your organisation to say that their research has been misused, we have received some correspondence from individual authors. This includes authors you have suggested had claimed they were incorrectly cited. Please be assured that all correspondence from individual authors has been/will be acknowledged and responded to on a case-by-case basis.
We would like to take this opportunity to thank you for writing to us. We share your enthusiasm for mathematics and hope that this response demonstrates how seriously we take your and others’ concerns, and that it provides more clarity around how we have considered research in the review.
We look forward to working with teachers and organisations in the future as we continue our work.
Sean Harford HMI
National Director, Education