Computer Gets It Wrong in Medical Admission Test

Computer Gets It Wrong in Medical Admission Test
New York Times
By KAREN W. ARENSON
Published: January 30, 2007

When Daniel Sonshine, a senior at Brown University, took the Medical College Admission Test on Saturday, he was asked to read a passage on robotic fish in the verbal reasoning part of the exam. Then he was presented with a series of questions about songbirds.

This was not a trick question; it was an error.

“I was completely distraught,” Mr. Sonshine said yesterday. “I was struggling to stay focused, but I was not focusing.” He was probably not alone. About 800 students who took the exam, known as the MCAT, encountered the mistake.

Another kind of problem surfaced in the College Board’s SAT exams, also given this past weekend. At least one student in South Korea had a part of the test before it was taken by 326,000 other students, according to the Educational Testing Service, which handles security for the exam. Raymond Nicosia, privacy protection officer for the testing service, said it was working with investigators in South Korea to find out “what the student had and who else had it.”

Mr. Nicosia said scores for students who had prior information about the test would be canceled. At this point, he added, “all the information we have is that it is a localized situation to Seoul, South Korea.”

Robert F. Jones, a senior vice president at the Association of American Medical Colleges, which oversees the MCAT, generally viewed as the most stressful of the admissions exams, said the error on that test was “something we regret.” “No more than 800” test-takers of about 2,500 were affected, he said.

About 50,000 people take the test each year, some more than once. Mr. Jones added that the error appeared to be “a test publishing problem,” but that he did not yet know how it had come about.

Last weekend was the first time the MCAT was administered only by computer rather than by paper and pencil. The medical college association announced in 2005 that it would move to an entirely computer-based format beginning in 2007, and that it would work with Thomson Prometric to administer the test at its computer testing centers.

Jodi Katz, a spokeswoman for Thomson Prometric, declined to comment on the error.
Mr. Jones, of the association, said he expected that students with the bad test would get their results within 30 days, like other students, because their scores could be extrapolated from the rest of their responses.

“It will not affect their scores,” he said. But Mr. Sonshine said he thought the painful experience could affect how he had done, and his admission to medical school.

“We are going to get screwed a little bit,” said Mr. Sonshine, who was taking the test a second time because he was not satisfied with the score he received on one section of the exam when he took it last summer.

He said he spent three weeks of his Christmas vacation studying for the exam, from 9 a.m. to 5 p.m. daily, and took five computerized practice tests. When he got to the exam on Saturday, in New Jersey, he whizzed through the first section, on the physical sciences, he said. But his confidence evaporated as he began the second section, verbal reasoning.

Using a technique he had learned from a test coaching company, he looked for a passage to read that seemed easy, to gain momentum. Unfortunately, the one he chose was the one on robotic fish.

“I read through and took some notes,” he said. “Then I went to the questions. ‘The male warbler cries for the female warbler when...?’ I’m starting to freak out.”

When he saw that the next question was also about warblers, he said, he started scrolling frantically through the other sections to see if they were misaligned, too. He then started trying to answer what he could.

“Seven minutes were already gone,” he said. “Every fiber of my being is telling me to go outside and say something. At the same time, the clock is ticking, and I haven’t answered a question yet. I struggled through the rest.”

Mr. Jones said that after some students raised questions during the exam, all the Thomson Prometric testing centers were notified and were advised to tell students to ignore the problem section and not to worry. But Mr. Sonshine said he had not been told about the problem.

Some critics of standardized testing said yesterday that while there were advantages to computerized testing, they were not surprised to hear of problems with the MCAT.
“Every time a test has been computerized, there have been huge glitches,” said Robert A. Schaeffer, public education director for FairTest, an advocacy organization that opposes widespread standardized testing.

Mr. Schaeffer said the SAT problem grew out of the College Board’s practice of reusing its tests. John S. Katzman, chief executive and founder of the Princeton Review, a test coaching company, said the problems with the SAT last weekend were a reminder that security and operational problems “happen even with paper and pencil.”

Mr. Jones said pilot tests last year, including one in August in which about 3,000 students took the computerized version of the MCAT, “went very smoothly.”