It’s rare for researchers to test a promising treatment on only a portion of students in real schools to see if it’s effective. Educators typically loathe the idea of denying anyone an opportunity to learn, which is what happens in a randomized controlled trial. Even rarer in education is a replication of this kind of experiment to see if something works in a different setting. So a recent study caught my attention because it not only copied a New York City program for community college students in three different Ohio communities but it also studied 1,500 low-income Ohio students for three years to see how many graduated when only half of them received the treatment.
The treatment was a multifaceted and expensive program that included not only free tuition and books for three years, from 2015 to 2018, but also a $50 a month stipend. Students were required to attend college full time, participate in a first-year seminar and meet frequently with an adviser. Extra academic tutoring and career counseling were also part of the program.
The results were good: 35 percent of the treated group succeeded in getting a two-year associate degree within three years, nearly double the graduation rate of 19 percent for untreated students who only had access to the usual services on campus.
Related: Worried about enrollment and judged on success, some colleges boost support
A 35 percent graduation rate after three years might seem pitifully low and there’s still much room for improvement. But the 16 percentage point difference in graduation rates between the treated and untreated students is unusually high in rigorous experiments like this where researchers factored in the outcomes of all students who were assigned for treatment but didn’t comply with the program’s rules or even enroll in college at the start. When the original New York program, called the Accelerated Study in Associate Programs (ASAP) was studied at the City University of New York (CUNY), researchers calculated an 18 percentage point difference in graduation rates between treated and untreated students. The New York researchers believed it to be one of the largest improvements in graduation rates ever posted in a randomized controlled trial in higher education. The Ohio result was statistically similar.
“People thought maybe it was just a New York thing but this study shows you can actually take this and do it somewhere else,” said Colleen Sommo, a senior associate at the research organization MDRC, which conducted the January 2020 evaluation. (The evaluation and the Ohio version of the ASAP program were partly supported by several philanthropies that are also among the many funders of the Hechinger Report: Arnold Ventures, the Bill & Melinda Gates Foundation, the ECMC Foundation, the Joyce Foundation and the Lumina Foundation.)
There were substantial differences between the New York and Ohio students. The New York students were more than 40 percent Hispanic and only 10 percent white. Those demographic ratios flipped in the opposite direction for Ohio. (Both had the same share of black students, about a third.) More importantly, Ohio’s community college population tended to be older. More than 30 percent of the Ohio students were age 24 or older compared with less than 20 percent of the New York students. More than a quarter of the Ohio students were parents with their own children compared to only 13 percent in New York.
It was unclear at the start of the study if Ohio’s nontraditional students with many responsibilities would be able to juggle full-time school along with part-time work. Two-thirds of the Ohio students worked an average of 30 hours a week. To accommodate busy schedules, Ohio students were offered online tutoring instead of in-person tutoring on campus. Another tweak from the New York model, where students were given free subway passes, was to give Ohio students gift cards for gas or groceries.
It’s too early to tell how much the Ohio program helped students get a degree that they wouldn’t have gotten otherwise or if it’s mostly helping students get their degrees faster. In a six-year follow-up study of the New York students, published in 2019, an additional 19 percent of the untreated students eventually got their two-year associate degrees. The graduation gap between the treated and untreated students shrunk from 18 percentage points after three years to just 10 percentage points after six years.
None of the many aspects of the program are particularly new or innovative. But when colleges have separately tested just one or two of the program’s pieces, such as advising and tutoring, they didn’t generate such large graduation improvements, MDRC’s Sommo explained.
“I think what’s working here is the combination of multiple things and doing it over the long term, not just one or two semesters,” said Sommo. “Community college students have multiple needs, barriers and challenges…There was a lot of hope that if you could get students on the right track in their first year, they’ll be fine. But that’s not what the evidence points to. The evidence is that you have to be able to provide a holistic set of supports and support them through to the finish line.”
Related: These colleges turn low-income students into middle-class earners — but how?
Providing many services is expensive, of course. MDRC calculated that it cost the three Ohio community colleges $5,500 extra per student to run the program over three years. (Because the students are low-income, their tuition is mostly covered by federal and state grants, such as $6,345 Pell Grants, and the colleges don’t have to subsidize that themselves.) But the costs to the colleges can balloon to $8,000 per student if you also count indirect costs because the program requires students to take more credits and causes the colleges to add courses and hire more instructors.
The state of Ohio awards more funds to community colleges as their students reach certain milestones and graduate, offsetting a large portion of the cost. But the program doesn’t pay for itself; the colleges still incur $4,500 per student themselves without philanthropic grants. Despite the proven boost to graduation rates, two of the Ohio community colleges that tried the program — Cincinnati State Technical and Community College and Cuyahoga Community College — eliminated it after the study ended. Only Lorain County Community College continues to operate its program, which is currently called Students Accelerating in Learning program (SAIL).
Meanwhile, CUNY has cut some costs from its $14,000 program by eliminating the first-year seminar and has expanded it to 25,000 students. Westchester Community College, which is part of the State University of New York (SUNY) system, and the colleges in the San Mateo County Community College System in California, such as Skyline College, have launched pilot programs. In January 2020, two colleges in West Virginia — West Virginia University Parkersburg and Blue Ridge Community and Technical College — received $4 million from Arnold Ventures to try the program there.
But with the coronavirus pandemic, prospects for further expansion are now slim. “My personal opinion is that there’s really strong evidence right now that if you can afford to do this as a college you should,” said Sommo. “Unfortunately, we’re now at a point that all the colleges are broke.”
This story about CUNY’s ASAP program was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for the Hechinger newsletter.