Who Scores the CMT and CAPT? Temps, That's Who

In England, the word '"sweatshop" was derived from "sweater," and was defined as an "employer who exacts monotonous work for very low wages."

Those conditions are exactly the type where "readers" employed by Measurement Incorporated work, scoring many Connecticut districts' CMT and CAPT exams, including Weston's. Results of the exams are used to determine whether a school district is meeting minimum educational standards; "failing" scores can result in a loss of government funding.

"If everything is true, or if even part of it is true, it's nothing short of a scandal," Weston's Assistant Superintendent Thomas Scarice told Patch. "It's horrifying to think we've aligned our scores" with a process that reveals little about what students actually learn.

Measurement Incorporated, headquartered in Durham, N.C., operates several scoring facilities around the nation and has been relatively scandal-free. Repeated written and verbal requests for comment were returned undeliverable or ignored.

On its website, the reader application is easily downloadable and the pay is $10.70 per hour, which can be increased to over $11 if the reader progresses satisfactorily. According to their website, "highly qualified" readers may have a college degree in any field.

The website citypages.com spoke with several former employees of companies such as NCS Pearson, Data Recognition Corporation and Questar Assessment. For-profit scoring companies hire thousands of workers every year for the purpose of reading through student essays and assigning the essays a score.

Measurement Incorporated has partnered with NCS Pearson, Data Recognition Corporation and Questar Assessment in the past.

There are several well-documented cases of scoring errors that have resulted in lawsuits against the corporations, and many former readers and managers have come forward, whistleblower-style, to open a previously-closed door on what many assume is a legitimate scoring process.

At a Minnesota scoring facility, an education official from an unidentified state visited a NCS Pearson site and informed the workers that "too many ones and twos" were being issued to students in her state, and instructed the company to give more "three" scores. The workers dutifully complied, but did not return to re-score the exams that had been completed. Instead, the workers simply began issuing more threes.

"The U.S. is somewhat on its own for using tests for the explicit purpose of accountability," Scarice noted. "Other nations use tests to improve teaching and learning."

Scarice also said that the "standards movement" that began in the late 1980s is diametrically opposed to the theory behind the CMT and CAPT scoring, because companies such as Measurement Incorporated rely upon the bell curve. In a bell curve, there will always be outliers that don't meet the standard, and it's impossible to tell from the scores if those at the bottom actually learned the appropriate material properly. 

Unfortunately, there seems to be little room for readers to deviate from the scoring "rubric," and many former readers complain that students who clearly knew and understood the material were getting low scores. 

In a phenomenon known as "scorer drift," readers, over time, begin to stray from the established rubric. When that occurs, readers say, supervisors confront the reader and insist that the score be changed to match the rubric.

Written essays appear on a screen and are often difficult to read, and scorers average reading roughly 200 essays per day, with one short, unpaid break. Essays are reviewed for scoring in a matter of seconds. One scorer reported that she had forgotten her glasses one day yet continued to score, because she was desperate for work.

The same employee, who worked for Questar Assessment, said that readers were comprised of young, inexperienced college graduates who couldn't find paying jobs in their fields, desperate people who were facing foreclosure and who were working two or three jobs, and retired teachers, many of whom complained vigorously about the scoring rubric.

second grade teacher noted that "this is the type of situation that occurs when you have a large distance between policymakers and those that implement policy."

"One of the cornerstones of good teaching practice is the appropriate use of assessment," he continued, adding that there may not be as much of a link between student retention and CMT and CAPT scores "as we've been asked to believe."

Scarice points to the scoring scandal as a motivating factor for the , which hopes to measure what students have actually learned through applied-knowledge assessment.

McNeill is a strong supporter of AIM.

"[AIM] offers a very attractive alternative to this situation," he said. "It puts the responsibility for teaching, learning, and assessment into the hands of those with a vested interest."

It allows us to say, with certainty, "this is what our children have learned, this is what they can do, and here is my evidence,'" he said.