Examining the exam

New metrics for improving the performance of multiple-choice questions

Abstract

In any unit, the credible assessment of learning outcomes is vital, and needs constant vigilance. Any misalignment between what is defined in the unit of study outline, what is taught in class, and what is assessed, has many negative consequences. Firstly, misalignment in assessment undermines tertiary standards and complicates transition into subsequent units. Secondly, misalignment in assessment weakens students’ sense of agency and belonging, which can affect ongoing academic confidence and performance.

Our study focuses on the evaluation of multiple-choice questions (MCQs) in a final exam, using a case study from a large cohort (N=1234) in a first-year data science unit. First: we use multiple known metrics to gauge the quality of the exam questions, including the discrimination index (DI), difficulty index (DIFF), distractor efficiency (DE), and the Cronbach’s alpha (CA). We discuss the strengths and weaknesses of each metric, and how together they can highlight questions that may indicate a potential gap in constructive alignment in the curriculum. Second: we propose new metrics to more deeply analyse each question, including a targeted distractor analysis, which reveals how distractors (the wrong multiple-choice answers) are functioning in practise, both across the cohort, and across the cohort divided into grades (P, CR, DI, HD). In total, our methods allow a detailed, transparent analysis of the performance of each multiple-choice question, which leads to concrete strategies empowering educators to improve a multiple-choice question bank for its next iteration.