Study: Common school ratings biased, often Inaccurate

This story was originally published by Chalkbeat. Sign up for their newsletters at ckbe.at/newsletters”.

by Matt Barnum, Chalkbeat

Jan 24, 2022 2:57pm EST

There is a powerful, intuitive idea that has long shaped perceptions of American education: the schools with the highest test scores are the best schools. It seeps into everything from conversations among parents to real estate websites.

But this thinking is profoundly misleading, according to new research. Schools with the top scores aren’t necessarily the ones that actually help students learn more. On top of that, rating schools based on overall scores unfairly penalizes those serving more students of color, potentially exacerbating racial segregation.

“If you just go by published accountability ratings, you will be guided to the schools that have the most white and Asian students,” said Josh Angrist, a professor at MIT who recently won the Nobel Prize in economics. “You’re not actually being guided to the best schools.”

This conclusion isn’t new in education research, but the paper, by Angrist and three other economists, is among the first to rigorously examine the validity of the measures used by the popular third-party rating site GreatSchools. The study comes after reporting by Chalkbeat in 2019 showed that GreatSchools ratings effectively steer families towards schools serving more affluent, white, and Asian students.

Using data from middle schools in Denver and New York City, the researchers compare the components of GreatSchools’ ratings to their own statistically sophisticated measurements of school performance. (The researchers look only at test scores, mirroring how GreatSchools rates middle schools.)

They find that the part of a school’s 1-10 rating that is based on students’ proficiency on state exams is particularly off base, and is strongly skewed in ways that hurt schools with more students of color.

Another part of the rating, the academic “growth” score, is much less biased and a strong gauge of how much a school contributes to student learning, the researchers conclude.

Angrist says their point is not to pick on GreatSchools but to show that the test score metrics widely used by states, media outlets, and other third-party sites are fraught with bias. U.S. News and World Report faced similar criticism last year after rolling out its own rankings of elementary and middle schools across the country.

“We think of [GreatSchools] as emblematic of what’s going on in the domain of school ratings,” Angrist said.

The new research was partially funded by the Walton Family Foundation, which is also a funder of both Chalkbeat and GreatSchools.

Jon Deane, the CEO of GreatSchools, said the organization welcomes this type of study. “This is the work we do: to consistently try to provide the best information to families,” he said. “When we see a signal that there may be better information, we get excited about that.” Deane also noted that the research is limited by its focus on middle schools in just two cities.

Still, he said the work validated GreatSchools’ recent shift to place more weight on academic growth, which — when available — is now the biggest factor in the site’s ratings of elementary and middle schools.

Asked why GreatSchools continues to use proficiency as a significant component of their school ratings, in light of this and other research, Deane said, “We believe combining multiple measures is really important to share a richer picture of what matters most.”

The researchers say that some of the differences in proficiency rates across schools reflect factors outside of schools’ control that affect students’ test scores. Those include poverty, food insecurity, and environmental hazards, like lead poisoning, that students of color are more likely to experience.

Of course, school ratings are only one factor in how parents choose schools. Another recent study suggests that school demographics themselves make a big difference.

In a survey-based experiment, White, Asian, and Hispanic parents in New York City were all much less likely to say they would send their children to a high school with mostly Black students compared to one with fewer Black students. White and Asian parents were also less likely to select a predominantly Hispanic school than a racially mixed or mostly white school.

This was true even when parents were provided with information showing that the school’s graduation rate and safety were similar to other schools with a different racial makeup. These survey results closely mirror families’ actual decisions in New York City’s high school application process.

“People often use these ratings as a way to justify the preferences they already have coming into making school choices,” said Chantal Hailey, a University of Texas at Austin professor who conducted the study. “They already have preconceived notions of which schools are quality schools based upon the demographics.”

Still, a third recent study offers some reason to believe that putting more emphasis on progress would encourage less racially segregated schools. Presented with growth data on nearby schools, white and high-income families were more likely to say they would choose one with more students of color and low-income students.

“One of the chief virtues of measuring and reporting growth is that it upends the conventional wisdom that the most effective schools are almost always the Whitest and most affluent,” wrote researchers David Houston of George Mason University and Jeffrey Henig of Teachers College, Columbia University.

Any effort to use growth scores to point families to high-quality schools is going to face a major roadblock in the years to come, though. State testing was canceled in 2020, and many fewer students took the tests than usual in 2021. That means it will be challenging — although some say not impossible — to calculate accurate school growth scores.

Currently, GreatSchools ratings are largely based on tests taken in spring 2019. Deane says the organization is still figuring out how to use data from state tests taken last year.

Chalkbeat is a nonprofit news site covering educational change in public schools.

https://www.chalkbeat.org/2022/1/24/22899133/great-schools-ratings-bias-economists-research