A machine-scored writing exercise introduced this year as part of the admissions process for five of Philadelphia’s top public high schools is being misused, an education professor says.
Joshua Wilson, an associate professor at the University of Delaware who studies automated essay scoring, said the writing tool is meant to identify struggling learners and inform classroom instruction, not make high-stakes decisions about students’ futures.
“No one has done research on whether it can be used to make placement decisions,” Wilson said about the writing tool, MIWrite, which is a product made by Measurement Inc., based in Durham, N.C. Using it this way is a “mistake,” he said.
He is scheduled to speak at Thursday’s Board of Education meeting.
In response to Wilson’s criticism, district spokesperson Monica Lewis issued a statement to Chalkbeat saying that the district had been “assured by the vendor that using this tool to score essays as part of the application process would be appropriate.”
Measurement, Inc. did not immediately respond to an email request for comment.
Lewis said that one of the five schools, Parkway Center City Middle College, had for several years been using a computer-graded writing sample for admissions with “positive feedback.”
This year the school district overhauled its selective admissions process in an effort to improve access for traditionally underserved students. Instead of giving individual school leaders final say over who is admitted, all students who meet the qualifications are entered into a lottery.
Each school can still set standards for grades, attendance and behavior. But missing the cutoff on the machine-scored writing sample can eliminate a student from the lottery.
Concern about the writing test is only one of several issues being raised about the new process. A group of parents plan to demonstrate outside district headquarters Thursday before the board meeting, and several parents and students registered to speak at the meeting on this topic.
Black and Latino enrollment at the district’s two most coveted and rigorously selective schools, Central High and Masterman Laboratory and Demonstration School, has declined in recent years, an issue highlighted during the national protests that followed a police officer killing George Floyd last year. Alumni, students and some faculty at the two schools demanded that the district act. Black enrollment at both schools is below 20%, in a district where more than half of the students are Black.
The goal of the new system, officials said, is to address racial disparities in admissions and open opportunities to more students by attempting to minimize any bias that could occur as school leaders shaped their next class.
“We worked hard to avoid data points that involved human assessment,” said Karyn Lynch, the district’s chief of student support services, when she announced the new process in October, on the day that the application window for fall 2022 admissions opened.
As part of the new system, the district also gave preference to students living in five ZIP codes where students are underrepresented at top schools.
Wilson, the professor, called the effort to make the system more equitable “a noble goal,” but said the question was “how to get the action to match the intention with a more valid way of doing it.”
Besides Central and Masterman, the other three schools requiring the writing test are Academy at Palumbo, Parkway Center City, and Carver High School of Engineering and Science. Carver and Parkway Center City are majority Black, while Palumbo is one-third Black and one-third Asian American.
Before the change, students were admitted to the five schools based on grades, attendance, behavior, and a writing sample.
Standardized test scores were a big factor before the pandemic, but they haven’t been administered since spring 2019. Officials eliminated their use for this cycle, but have not said whether they will resume using them in the future. Standardized test results typically correlate with socioeconomic status, putting underserved Black and Latino students at a disadvantage.
The MIWrite test used this year gives students a prompt, mostly on topics that don’t require detailed background knowledge. Questions might include “What is your favorite subject and why” or “If you could go back in history and meet someone, who would that be?” Students have 90 minutes to write an essay, which is immediately scored on a scale of six to 30.
To qualify for Palumbo, Carver and Parkway Center City, the student needs a score of 17. For Central and Masterman, the score is 22, according to the district.
Wilson raised concerns about how the district determined the cutoff scores on the MIWrite test for admission. “Where’s the research that validates the cut scores?” he said. “I think Philadelphia has the burden of showing the proof that these cut scores are appropriate.”
Wilson said that 17 “is just below the median, and probably not that high a bar.” In a 90-minute sample, he said, “it would probably be achievable for most kids unless they really lack basic skills.”
Wilson also noted that the instrument doesn’t score for content. “I can write a beautifully written response off-topic and get a good score,” he said.
Students take the computerized exercise in class, and immediately learn their score. Students are taking the test now; the testing window is between Nov. 29 and Dec. 18.
Timothy Boyle, principal of Science Leadership Academy Middle School, said that 43 of his students took the test last week, and almost all scored above 17. Based on what he knew about the students, he said he wondered what the scores actually revealed about them.
One straight-A student scored a 21, one point shy of the score that would make them eligible for Central or Masterman. Another student, with a 1.5 grade point average, scored a 20, putting them above the cutoff for three of the five schools, though they might be ineligible based on other factors. “How does this result in more fairness and more opportunity?” Boyle asked.
The scoring system is based on an algorithm derived from what trained human raters produced based on the six-trait writing model, which includes rubrics for ideas, organization, sentence fluency, word choice, and style conventions. Each trait can receive a score between 1 and 5, adding to a maximum score of 30.
Wilson said the main use and benefit of machine-scored tests is to encourage teachers to assign more writing because it helps them to pinpoint student weaknesses and guide improvements quickly. “Scoring and evaluating writing is so time consuming,” he said.
Boyle also applauds the effort to make the system more equitable. But he said if district leaders suspect bias on the part of its educators as a reason for the disparities in the demographics at its most selective schools, it needs to tackle that issue directly.
“They say they don’t know if this [new process] is going to work,” he said. “It’s like they are throwing stuff at the wall, and confusing action for strategy.”
The high school selection process includes three tiers — the five “criteria-based” ones at the top, “citywide admission” schools with a particular theme or focus, and those with catchment areas that have special programs.
In the statement, made in response to Chalkbeat’s questions, Lewis said that this year, the district received more high school applications compared to last, suggesting that the effort to broaden access is working. She said 15,382 students — from both inside and outside the district — submitted 62,591 applications (each student can choose five schools), which amounts to “365 more students participating in the process than last year.”
Lynch and Superintendent William Hite, who will end his 10-year tenure in August, have said that the intent is to continually refine the process.
“We know that change can be difficult and we also know that the time for equity-focused change is now,” the district’s statement said. “This year’s changes are a starting point” that will lead to “further enhancements…as we move toward becoming a stronger and more equitable school district.”
This story will be updated.