Research Bite #20: Three Options Are Optimal for Multiple-Choice Items
By Michael C. Rodriguez
Hello, and welcome to the latest Research Bite!
Each week, I will use my favourite AI tool, NotebookLM, to summarise a research paper that has influenced my teaching, coaching, and thinking about education*.
Before we dig in, you might like to check out:
Video insights from the world’s leading experts I have interviewed
Write-ups of everything I have learned from watching 1000s of lessons
Title: Three Options Are Optimal for Multiple-Choice Items: A Meta-Analysis of 80 Years of Research
Authors: Michael C. Rodriguez
Access the original paper here
Listen to a deep-dive podcast:
Paper summary
This meta-analysis synthesizes 80 years of research on multiple-choice item writing, focusing on the optimal number of response options. The study uses meta-analytic techniques to examine how altering the number of options impacts item difficulty, discrimination, test reliability, and validity. Results indicate that three options are generally optimal, maximizing test efficiency and potentially improving content coverage. However, the method of distractor removal significantly impacts reliability, with random removal proving detrimental. The findings suggest a revision of existing item-writing guidelines to prioritize three well-constructed options over a larger number of less effective distractors.
What are the key implications for teachers in the classroom?
Based on a meta-analysis of 80 years of research, the optimal number of options for multiple-choice (MC) items is three. This finding has several implications for teachers in the classroom:
Efficiency in test development: Teachers can save time by creating only two plausible distractors instead of three or four, without compromising the quality of the test.
Improved content coverage: With three-option items, teachers can administer more items in the same testing time, potentially improving content coverage. The inclusion of more high-quality items can also improve test score reliability, which can provide additional validity-related evidence.
Item Analysis: Teachers should be aware that item analysis data is tentative and sample specific and should be used carefully. From an IRT perspective, the elimination of distractors makes items easier, which is irrelevant. In addition, in a standard setting context, item difficulty plays a significant role in popular methods. In a classroom context, grading standards may need adjustment if tests become easier due to the use of fewer options.
Test Validity: Validity evidence should be gathered to ensure the quality of test item development, and item-writing guidelines could serve as a checklist. Test developers should provide a rationale for the number of options used in MC items.
Reduced test-taking time: Students may complete three-option tests more quickly than tests with four or five options, which can be beneficial in timed testing situations. However, on highly speeded tests, lower-ability students may resort to blind guessing, which may have a detrimental effect on validity.
Focus on distractor quality: Instead of focusing on creating more distractors, teachers should focus on creating high-quality, plausible distractors. The quality of the distractors, not the number of them, is what guards against awarding undeserved credit.
Option deletion methods: When reducing the number of options, randomly deleting distractors can be more detrimental to reliability than deleting ineffective distractors. Teachers should consider which distractors to eliminate thoughtfully.
Use of 3-option items: Based on the meta-analysis, teachers should use three options for multiple choice questions (one correct option and two plausible distractors), because using more options does not significantly improve item and test score statistics and often results in implausible distractors.
In summary, using three-option multiple-choice items can help teachers create more efficient, reliable, and valid assessments, as long as they focus on creating high-quality items and thoughtfully consider how to eliminate options when needed.
Quote
Three options are optimal for MC items in most settings
What are the implications of this research for you?
What questions does it make you want to ask?
Let me know in the comments below
🏃🏻♂️Before you go, have you… 🏃🏻♂️
… read my other research paper summaries?
… watched the video insights from the world’s leading experts I have interviewed?
… read my write-ups of everything I have learned from watching 1000s of lessons?
… checked my new 90-minute online CPD series?
*Note from Craig Barton’s legal team: Research Bites is in no way, shape, or form influenced by Peps Mccrea’s Evidence Snacks.