Dallas jumped to No. 5 in the ranking of 27 large school districts across the country.
The analysis was based on test scores from 2017. A headline from The Dallas Morning News at the time announced that the results show "Dallas, Fort Worth lagging behind big city peers."
But, comparing cities using the data ignores large differences in their student bodies. "It's very difficult to compare states or even school districts, because they serve great demographics with students with different levels of needs," said Kristin Blagg, a researcher at the Urban Institute and co-author of the report.
In other words, using average scores from the National Assessment of Educational Progress to pit districts against one another is a lot like comparing apples and oranges.
In an attempt to level the playing field, Urban Institute researchers recalculated each district's scores after controlling for demographic differences such as race, income level and English language proficiency. Dallas' average scores rose 14 points, the largest adjustment of any district.
Fort Worth also saw a substantial improvement, rising to 10th in the rankings. Austin's school district, which had a much smaller adjustment, remained near the top at third. They're joined in the top 10 by Houston, Chicago, Boston, Charlotte and three Florida districts.
"It does do us a disservice when these broad generalizations are made, without closer comparisons.” — Michael Hinojosa
The standardized test is given to students across the nation to compare student proficiency in subjects like math, reading and science between states. But large school districts, like Dallas and Fort Worth, have opted to be ranked as well.
DISD Superintendent Michael Hinojosa has fought back against the perception that Dallas students are failing the exams. In a 2019 Dallas Morning News article, in which the paper characterized Dallas and Fort Worth as faring poorly on the latest NAEP, Hinojosa is quoted as saying, "It does do us a disservice when these broad generalizations are made, without closer comparisons.”
Whether this study provided Hinojosa with a measure of vindication is unclear. The district did not respond to a request for comment.
But district trustee Miguel Solis said that the report offered a more nuanced approach to the NAEP data, and one that gives the district the credit it deserves. "When you take into consideration demographics, we have to be growing kids — who are so far behind — exponentially better than other districts," he said.
"So once you put on the lens of socioeconomic inequity, we are doing a significantly better job getting those kids who are behind, up to where they need to be."
He credits this to reforms like merit-based pay for teachers, increasing investing in poorly performing schools and the expansion of early child education — reforms that policymakers are trying to replicate across the state.
DISD is made up primarily of minority students, who typically fare far worse on standardized exams. Nearly half of students in the district have limited English proficiency, and 87% are economically disadvantaged. The persistent gap in scores between rich and poor, as well as white and minority students, is a nationwide problem.
The Urban Institute has argued that demographic adjustments should be made by the federal government when they release the scores, so that journalists and policymakers can put them in the proper context.
"Politicians and advocates from across the political spectrum point to high or improved NAEP scores as evidence supporting their preferred policies, ignoring researchers who argue that these scores rarely provide credible evidence that policies are working or failing," wrote Matthew Chingos, the director of the Institute's Center on Education Data and Policy, in a 2015 report.
Still, the complexities of interpreting NAEP results haven't stopped academics from trying. The scores have been trending downward for the last decade, following rapid gains in the 2000s.
The trend defies a definitive explanation. Theories include shifting demographics, a decline in funding in the wake of the recession and the end of No Child Left Behind, which aimed to hold districts accountable to narrowing academic performance gaps.
"My forecast is a mixed bag," said Solis when asked whether he was optimistic in the face of falling scores. He said that the district is rolling out a new test to hold itself accountable to national standards, but he pointed out that curriculum and funding decisions are largely out of the district's control.
"I do think that there is a willingness from our state, and certainly there has been a willingness by our district, to think radically different about what we're doing."