The National Center for Education Statistics (NCES), which administers the National Assessment of Educational Progress (NAEP), also known as The Nation’s Report Card, awarded four grand prizes and recognized four runner-up teams in its first automated scoring challenge. The winners used advanced natural language processing methods that promise to reduce scoring costs while maintaining accuracy similar to human scoring. Those honored described their technical approaches prior to scoring and met requirements for transparency, interpretability, and fairness.
“The winning approaches represent current best practices in natural language processing and demonstrate evidence of similar reliability to human scoring with certain types of items,” said Peggy G. Carr, NCES Commissioner. “All of the winning teams conducted fairness analyses showing that their models were not biased due to demographics or family background. These results suggest a promising path for NAEP to use automated scoring in the near future.”
The challenge had two parts: item-specific, where competitors created a different model for each item, and generic, where teams applied a model created from different item responses. Challenge winners include respondents from different sectors, including assessment companies, university researchers, and a student team.