Computer problems that plagued the ISTEP+ testing process had no measurable effect on students’ scores, education officials say, citing an independent report.
State Superintendent Glenda Ritz and the Indiana Department of Education released the results of a report commissioned to examine the impact of computer interruptions on this year’s tests.
“There is considerable evidence that the interruptions had no negative impact on student scores for the vast majority of students,” the report concluded. “Students who were interrupted had somewhat larger gains across years than those who were not interrupted.” The report conceded that the latter finding was a surprise.
DOE hired Dr. Richard Hill of the National Center for the Improvement of Educational Assessment to review the results and look for discrepancies in the scores. The report goes into great detail about computer problems and the number of student tests affected, but the overall conclusion was that computer interruptions did not have a significant impact and that the true overall effect “could not be ascertained.”
Indiana has been transitioning from pencil-and-paper testing to online testing since 2009. This year, about 95 percent of students took the test via computer, an increase from 71 percent in the previous year. The department said 79,442 students—about one-sixth of the total student population—experienced disruptions. There were a total of 117,379 disruptions because some students were interrupted more than once.
The majority of the problems happened during sessions 1 and 2 of the test, when students were taking the math portion. Far fewer disruptions were reported during the English/language arts (ELA) portion.
The issue traces back to April 29, when students first experienced problems while taking the computer portions of the test. Around 10:30 a.m., a memory issue with servers from contractor CTB/McGraw-Hill caused interruptions. Engineers worked to isolate the problem and fix it, but Ritz ended up extending the statewide testing window by two days.
On the second day of testing, a different problem on CTB/McGraw Hill’s end caused another disruption. Students experienced the same difficulties as they did the day before, but the glitch was more widespread. CTB switched to a “disaster recovery site”—essentially, a backup server—that had its own problems. Some students couldn’t access their tests or their previous responses when they logged back into the testing system. The results weren’t lost, but they were inaccessible to students and schools.
Ritz told schools to finish the portion of the test they were working on and then called for schools to suspend online testing. Upon resumption of the tests the following day, schools were told to test at 50 percent capacity and Ritz extended the statewide testing window by another three days to May 17.
When testing resumed May 1 at reduced capacity, schools reported no further widespread problems. As a precaution, districts still tested at half capacity to ease the strain on the system. This continued until May 6, when schools resumed testing at full capacity. ISTEP+ testing was completed across the state on May 17.
On May 24, the Department of Education gave schools a list of students affected by testing disruptions. Schools were to check the accuracy of the list and add any students who experienced problems with the test. The department also requested the qualifications of three national companies experienced in validating test results. Of those, the department selected the National Center for the Improvement of Education Assessment, which compiled the final report.
According to the report, the actions of Ritz, teachers, administrators, students and parents minimized the potential impact of testing interruptions on ISTEP+ scores. The report also concluded that “the average negative statewide impact on scores was not measurable.”
“First, I want to acknowledge the extraordinary efforts of Indiana students, parents, teachers, administrators and the employees of the Department of Education,” Ritz said. “Because of their dedication and hard work, the impact of these interruptions was limited. However, let me be clear, the problems with the ISTEP+ contractor were absolutely unacceptable.”
Ritz said even though the statewide average score was not affected, the problems still took a toll on students. In his report, Hill said, “We cannot know definitively how students would have scored this spring if the interruptions had not happened.”
Ritz said she’s given schools flexibility on how the test scores will affect teacher evaluations and compensation. She said CTB/McGraw-Hill has been instructed to conduct more rigorous stress tests on its servers to make sure the problem doesn’t happen again.
The department said the most plausible conclusion was that efforts by administrators, teachers and students managed to overcome any problems caused by the testing interruption, minimizing the impact. The department said the test took significantly more effort to complete this year because of the issues.
The department outlined three changes that could have affected scores:
- The new policy to retain students in grade 3 because of unsatisfactory scores on the IREAD test
- The switch from pencil-and-paper to online administration for many schools
- The interruptions affecting the online administration
The state said the retention policy had more impact on changes to the scores for grades 3 and 4. The switch from pencil-and-paper hasn’t had much of an impact on test scores as the department transitioned students, though the impact could’ve been more pronounced in 2013, as more students made the change.
The state admitted that it was possible that the testing disruptions had an impact on individual students because the numbers were examined as a whole. It’s possible that some testing patterns could show that individual students were affected.
“The data strongly suggest, that the vast majority of students scored as well as they would have had the interruptions never happened,” the report concluded.