Lost in Translation?: Investigating A Perceived English–French Achievement Gap on EQAO Tests

Theo J. O'Connell

University of Toronto
Research Advisor: Dr. Meï-Lan Mamode

Introduction

The Education Quality and Accountability Office (EQAO) is an agency of the Government of Ontario, Canada, established in 1996 to provide longitudinal data on student achievement within the province. This is achieved through the annual creation, distribution, and grading of four major standardized tests: two Assessments of Reading, Writing and Mathematics for Grades 3 and 6, the Grade 9 Assessment of Mathematics, and the Ontario Secondary School Literacy Test, typically occurring in Grade 10. These tests occur ubiquitously across Ontario’s four publicly funded school systems, two of which operate in English and two in French (see Appendix A). Having taken all four EQAO tests throughout my elementary and secondary education, I was interested in evaluating the organization’s ability to meet its benchmarks for an ‘intervention assessment’ project in the University of Toronto’s Munk One Program. In reviewing EQAO annual reports, I noticed an apparent performance gap among students in English- and French-language schools from 2016–2019, wherein French-language schools reported higher EQAO test scores on average (see Appendix B). Eager to investigate this topic further—especially given its absence from much existing scholarly literature on the EQAO—I pursued the present research project, which I am excited to begin work on this summer.

To produce a foundational analysis of the English–French EQAO disparity, I intend to combine multilevel modelling, content analysis, and semi-structured interviews. Using data from the EQAO itself and the Financial Accountability Office of Ontario, I will both determine the significance of the disparity and identify the more pertinent predictors of EQAO test performance within the province. If possible, I will also analyze longitudinal EQAO data to examine how this disparity may have evolved over the past two decades. To complement these quantitative insights, I will then examine bilingual EQAO guidelines tailored toward teachers and school administrators, highlighting any significant differences between the English and French versions. Finally, anecdotal data gained from interviews with relevant experts and/or stakeholders will help contextualize the findings of my research. In focusing on this topic, I bring my prior knowledge of the EQAO's policies and practices to a far-reaching yet under-researched topic with broad implications for Ontario education.

Research Questions

With this research project, I will first assess the significance of the apparent English–French performance gap on EQAO tests. Then, I aim to determine how it has endured or changed over time, to what extent, and as a result of what factors. In doing this, I will be guided by the following three research questions:

• Was there a significant achievement gap among students in English- and French-language schools in Ontario on EQAO tests for the recent 2022–2023 assessment period?
• What factors may contribute to this disparity, and to what extent?
• How has this trend evolved over time from 2003–2019 and 2021–2022 (whereas the EQAO was cancelled from 2019–2021)?

Methodology

Outlined below is the proposed methodology for answering each of my three research questions.

• Was there a significant achievement gap among students in English- and French-language schools in Ontario on EQAO tests for the recent 2022–2023 assessment period?

To address this research question, I will collect data from the official EQAO website for each of Ontario's public school boards that have sufficient data available. This information includes but is not limited to students' access to technology/Internet at home, their time spent in Canada, and the total enrolment of each board. To complement this information, I will refer to a 2022 data set created by the Financial Accountability Office of Ontario (FAO), which includes each board’s EQAO pass rates in addition to factors such as remoteness, i.e., distance from a major urban centre, and dispersion, i.e., the distance of a given board’s schools from each other. Data will be coded to indicate which data set they were retrieved from. Some school boards on the EQAO website lack data due to their small sample size, which I will also make note of. Finally, I will graph the data and determine the statistical significance of the divide using SPSS.

• What factors may contribute to this disparity, and to what extent?

This question benefits from a mixed-methods approach involving multilevel modelling, virtual interviews, and content analysis. Referring to the two chosen data sets, I will first use multilevel regression analysis to examine the effect of individual variables on student achievement for English- and French-language schools separately. Second, I plan to conduct semi-structured interviews over Zoom with relevant stakeholders. Of particular interest are interviewees with expertise or experience relating to the EQAO and the French-language education system: this may include EQAO representatives, faculty of the Ontario Institute for Studies in Education, school board trustees, and teachers or school administrators. Each interviewee will be given a $50 honorarium for their time. I will aim for as many as eight interviews over the course of the six-week research period, which I plan to schedule beforehand. With the permission of each interviewee, I will have each interview automatically transcribed using Zoom Live Transcription. I will then code the transcripts according to relevant and recurring themes, allowing me to more effectively integrate their insights into my final research paper. Proceeding with the content analysis, I will review bilingual EQAO guidelines, working alongside Dr. Mamode to code web pages and documents so as to identify any significant differences in their English and French versions. Guidelines will be selected according to their relevance to teachers and school administrators.

• How has this trend evolved over time from 2003–2019 and 2021–2022 (whereas the EQAO was cancelled from 2019–2021)?

Answering this research question requires access to 17 years of EQAO data: in particular, I seek access to student achievement results aggregated at the school, board, and provincial levels for all four EQAO assessments. With this data, I will graph trends in student achievement using board-specific data for each school board in Ontario, making a distinction between English- and French-language boards. I will then assess the statistical significance of the English–French performance gap, both by year and cumulatively.

Timeline

• Before the research period: submitting EQAO data request, contacting potential interviewees, scheduling interviews for within the research period (weeks 4–6), receiving guidance with multilevel modelling
• Week 1 (June 17–23): working on RQ1: data collection, visualization, & interpretation
• Week 2 (June 24–July 1): working on RQ2: multilevel regression analysis & interpretation of findings
• Week 3 (July 1–7): working on RQ3, formulating interview questions based on quantitative insights from weeks prior
• Week 4 (July 8–14): working on RQ2: content analysis; conducting, transcribing, & coding first round of interviews
• Week 5 (July 15–21): conducting, transcribing, & coding second round of interviews
• Week 6 (July 22–28): conducting, transcribing, & coding third round of interviews; finalizing conclusion & implications

Research Objectives

With this research project, I work toward bridging a gap in EQAO-related research, providing a foundation for further analysis of the apparent English–French achievement gap in Ontario. While unaffiliated with the EQAO, I pursue this study in accordance with the organization's goals of promoting accountability in education and better understanding the myriad factors that influence student achievement. Indeed, the EQAO as an organization exists in part to keep administrative actors informed on trends in student achievement, thereby allowing them to more effectively meet students’ needs. By lending further context to the EQAO’s raw data, researchers therefore assist in this process. Examining the relationships among relevant variables similarly informs existing studies while serving as a stepping stone for future research. Given that the EQAO compares test results against international benchmarks while coordinating Ontario’s participation in large-scale assessments, my research findings may be applicable beyond a provincial context. As such, this study has the potential to identify pertinent obstacles to equity within bilingual education systems and elucidate strategies to minimize them.

Appendix A: Number of school boards, schools, and students in Ontario by school system and language of instruction

As shown in the above table, Ontario comprises four publicly funded school systems: English Public, English Catholic, French Public, and French Catholic. This table, which I created for my prior research project on the EQAO, highlights a significant size difference between English- and French-language systems, wherein there are approximately 17 times as many students enrolled in the former as in the latter. Data on the number of school boards and students enrolled are from the FAO's aforementioned 2022 data set, while data on the number of individual schools are from a Government of Ontario data set released in 2023. The FAO measures enrolment through Average Daily Enrolment, measured in turn by individual schools on count dates in October and March of each year.

Appendix B: Visualization of an apparent English–French performance gap on EQAO tests: Grades 3 & 6, 2016–2019

The above graphs highlight the particular component of the English–French disparity that motivated me to pursue this research project. Data were obtained from the EQAO’s annual report for 2019–2020, the final assessment period before the temporary cancellation of the organization’s K–12 testing program due to COVID-19. Because data for Grade 9 students were only available separately for the Academic and Applied streams (discontinued as of September 2022), this information was not included in the graphs. Also note that the provincial standard (as determined by the Ministry of Education) is a Level 3, for which the minimum bound is a 70%.