DPS students made huge gains over 11 years. Was it controversial reforms or something else?
Denver Ite
There is a debate brewing in academia about Denver Public Schools’ controversial era of reform and whether a CU Denver study overestimated the success of the strategy.
A new review of that study says its conclusions about student gains made in the DPS reform are exaggerated and don’t definitively link improved outcomes to the changes. But others defend the study, saying the review fails to offer any other plausible alternative explanation for such large improvements in student achievement.
The issue is significant, as large urban districts grapple with how to boost academic achievement. Much of what Denver tried from 2008 to 2019 has since been dismantled in the district and, nationally, the focus on test-based accountability has diminished.
Denver’s reform strategy embraced a so-called portfolio model, with a mix of neighborhood, charter and innovation schools. Denver opened 65 new schools and closed, replaced and restarted 35 others.
The model relied on school choice and competition, closing low-performing schools, creating new schools, empowering educators, paying teachers more to work in high-poverty schools and holding everyone accountable for test results. It also created a student-based funding model that allocates dollars based on need.
What the CU Denver study found:
In September, researchers from CU Denver’s School of Public Affairs released a study showing that DPS reforms from 2008-19 led to huge gains for students. The study found students got the equivalent of nine to 14 months of additional learning per year, and graduation rates jumped from 43 percent to 71 percent.
Parker Baxter, director of the Center for Education Policy at CU Denver, compared 40,000 student outcomes in Denver and 11 surrounding districts during the reform era. The study showed the reforms benefited all student groups, including low-income and special education students, making it “one of the most successful education reform strategies in U.S. history,” according to Baxter.
He also specifically studied students who left a school that closed, entered a new school, or were enrolled in a “turnaround” school, where the school is completely overhauled. The first two categories saw test gains, though more modest than the districtwide gains.
Another view emerges
A new review published by CU Boulder’s National Education Policy Center, states that while the study provided evidence of academic gains, attributing those gains to the reforms is overstated and not substantiated by the data.
“Its conclusions are exaggerated in both magnitude and certainty,” the review stated. The gains that the study found are two to three times bigger than what other studies have shown for students that get individual intensive intervention. The review suggests other factors beyond the portfolio reform were at play in explaining results.
The reviewer, Robert Shand, an assistant professor at American University, said to carry out a study of Denver’s big and broad reforms over such a long period of time is incredibly difficult to do in research, especially “to be able to precisely identify what was happening and what caused what,” he said.
Shand found Baxter’s method of summing test score gains over multiple years without regard for “well-documented” fading-out effects over time makes gains look bigger than they are.
Graduation rates increased dramatically in Denver, but they also increased in many other parts of the state.
“There were gains in Denver and I think yes, they are worth looking at and celebrating, but they are not nearly as large relative to the rest of the state as the report makes it out to be,” said Shand.
Other factors at play?
Shand argued the study downplayed or ignored other factors that could have played a role, such as funding, reduced class sizes, changes in senior leadership or curriculum changes.
While all districts in Colorado were losing millions of dollars in state funding over those years, Shand asserts that Denver’s funding relative to other districts didn’t go down as much because DPS had funding from foundations and philanthropists.
“How can we be sure that it is the reform that’s changing (scores) and not other things?”
But the review itself doesn’t present evidence that average per-pupil funding was substantially different in Denver from other places.
Douglas Harris, a professor at Tulane University with extensive experience studying school reforms, said based on prior research, the effects of school funding alone could not come close to explaining the massive improvements documented in the Denver study.
He said that decisions like reducing class size could also be linked to the move to the new model, “driven by portfolio decision-making.” A key aspect of Denver’s model allows schools to make decisions about leadership and curriculum, he said.
In contrast, Shand said attributing all the gains to a broad “portfolio” strategy isn’t helpful to other cities looking to implement change. He said the study didn’t explain how each component of the reform (school choice, accountability, school closures, school autonomy, school interventions) directly affected student learning.
A long and lasting argument
Shand argues that the study “does not grapple with” the full range of negative effects and inequities of the reforms. For example, while school choice and the reforms might benefit some students, it can heighten disparities between families and schools. Some families can afford transportation to a school out of the neighborhood, others can’t. Some larger charter school networks may benefit from outside private funding while a neighborhood school doesn’t.
Many observers would describe the reform years as turbulent, marked by turmoil and turnover at the school level, disruptions and community pushback.
“It does not address the political and social tensions that accompany such reforms,” the report reads.
Ultimately, Tulane’s Harris, who is an affiliate of CU Boulder’s NEPC, said the review doesn’t substantively change the general conclusion that the reforms increased student test scores quite substantially. He said Shand’s reviews are more summaries of the difficulties that naturally arise when studying large systemic reforms.
“Those types of studies are, by nature, harder to understand because they are multifaceted. But it makes sense to start understanding the effects in the way the [original] study approached it,” he said.
The review states that the report provides evidence that “something” is happening in Denver worthy of study.