BLOG ARTICLE

Proving That Multiple Choice Assessments Suck

Published by
Stephane Rivard
June 21, 2023
SUMMARY
Whether you love multiple choice on the evaluator’s side or hate multiple choice on the respondent’s side (typically that is how the cookie crumbles), it’s obvious that the format does have its advantages. The same can be said for open-ended questions, but is that proven? Spoiler alert: Yes.
Table of Contents

    It was a warm afternoon in Kansas and the year was 1914, the first time a senator would be required to be elected by popular vote, and while everyone was preparing for the November election, Frederick J. Kelly was busy at home creating the first-ever multiple choice questions. Ok, we don’t know if it went exactly like that, but we can tell you that both democratic processes and multiple-choice questions have flourished since then.

    Multiple-choice questions provide a convenient and perceivably objective way to evaluate performance. This question format has become so popular that when you type “multiple choice questions” into Google, you’ll get more than five billion results.

    Before multiple choice, questions were simply open-ended. An open-ended format allows for elaborate questions to be asked and meaningful answers to be given when needed. While this format is fundamental to human communication, it doesn’t scale very well in classrooms or professional settings, where a talent acquisition specialist may have a high-volume hiring mandate.

    Whether you love multiple choice on the evaluator’s side or hate multiple choice on the respondent’s side (typically that is how the cookie crumbles), it’s obvious that the format does have its advantages. The same can be said for open-ended questions, but is that proven? Spoiler alert: Yes.

    Let’s take a closer look at these question formats in a professional setting before proving that multiple-choice questions are by and large sucky.

    Four Critical Flaws With Multiple Choice in Pre-Hire Assessments

    As someone who may currently use multiple choice questions in your candidate assessments, you may be asking yourself why are they hating? Well, it’s one thing to report on research findings (which we will get to!), but it’s another to remark that there is a growing body of evidence against multiple choice as a practice!

    Fails to Measure Deep Thinking

    Multiple choice can measure a person’s understanding of a subject, yes, but it cannot measure deep thinking skills, says curriculum director at TeachThought, Terry Heick, in the Washington Post. He criticizes the certainty of the answer format, which is an either/or binary scenario that ignores how humans connect with ideas and information. The format of the question is a flawed way of teaching, to begin with. Even Frederick J. Kelly himself, said “These tests are too crude to be used, and should be abandoned.”

    From an academic perspective, some facilitators are speaking out about multiple choice. Devrim Ozdemir, the Director of Student Success and Assessment at the University of North Carolina at Charlotte, writes that multiple-choice can only evaluate memory recall, not higher-order thinking activities like “analysis, synthesis, creation and evaluation”. So why bother using multiple choice at all? Ozdemir says it is likely out of convenience to the instructors and evaluators.

    Questions Can Be Easily Misinterpreted

    Multiple-choice questions can be misinterpreted if they try to be too complex, which can yield unreliable responses. One Forbes author even writes how questions are often intentionally formulated to be confusing to the respondent:

    “There tends to be a lot of questions that could be straightforward, but instead, they’re intentionally worded as trick questions. So, they test the applicant’s reading comprehension skills more than the supposed certification topic knowledge. Even native speakers can easily miss the tricky part under the stress of a timed exam situation. And if English is the applicant’s second language, then even the time extension for non-native speakers may not help much.”

    Faculty Focus explains “the clarity of multiple-choice questions is easily and regularly compromised—with negatives or too much material in the stem, for example.” They suggest that anyone using multiple choice assessment do an item analysis to determine if high scorers are all missing one question or another. If so, they reason that likely it is the question that is flawed and should be removed from the assessment.  In an open-ended setting, the respondent would have the opportunity to explain themselves.

    The Answers Can Be Compromised

    It’s plain logic that if there is only one right answer to a question, that answer can be known and shared. If there isn’t just one answer to a question, then it’s much harder to cheat. 

    In a multiple choice setting it’s much easier for respondents to research past exams to get the answer. This isn’t true for open-ended questions whereby a thoughtful and original response has to be provided on the spot and explained.

    Scientists Don’t Like Multiple-Choice

    There are over 250 different hiring assessment vendors that hiring managers can choose from in the market today, and it must feel overwhelming to choose one. Many are built on binary question frameworks, with multiple-choice questions used. Some of these tools even claim to be using AI with this format, but from a scientist’s point of view that’s an oxymoron. Multiple-choice-question assessments, or those with one right answer, are not evaluating candidates with AI, and if they say they are, they're lying. From this point of view, you can think of the question format as a litmus test for the presence of true AI in a tool. Our Head of Research and Development explains this in the following excerpt from our recent article:

    “Many communication assessments are closed-ended, which flattens the candidate's abilities and opens the door for cheating. Using a dynamic AI system allows hiring teams to construct assessments with open-ended questions, where the candidate needs to react to a given scenario in their own words in writing or speaking. 
    A multiple-choice format is not AI. This is just straightforward true or false. So the only way that an assessment can truly be AI-driven is if you have open-ended questions. Instead of measuring one right or wrong response, AI measures the way things are said, how they are said, grammar, fluency, how relevant the answer is to the question that was asked, whether the candidate showed empathy to the customer, used proper language, and more. It’s even better if the open-ended questions actually mimic real on-the-job scenarios.” 

    If multiple choice is so bad, why are people still using it? 

    Scaling Open-Ended Questions 

    It’s obvious that evaluators are addicted to the convenience of multiple choice, even with these flaws. Yet, just because something is good for the instructor doesn’t mean it’s good for the student. Daniil Baturan, CTO and a co-founder of VyOS, shares this sentiment on Forbes and thinks multiple-choice questions are flawed, and the only reason they’re so prevalent is that “They were cheap to administer on a large scale.” Devrim Ozdemir says time-savings around grading make large classrooms more manageable for instructors.

    While this may be true, AI has made the large-scale evaluation of open-ended questions possible now too. Natural language processing research has the potential to save hiring managers a lot more time than multiple-choice questions can!

    With the ability to evaluate open-ended questions at scale, hiring managers can essentially replace any high-volume interviewing requirements that previously relied on multiple choice. A view of how candidates scored in their open-ended question evaluation, means that hiring managers can get an in-depth understanding of each candidate at scale without interviewing them because open-ended questions can reveal so much more about a person. Hiring managers can quickly identify their top 1% and save interviews for just these people, or even send candidates straight through to training (like this team now does!).

    With all their known flaws, multiple-choice questions may be leading your hiring efforts down a bad path. And if open-ended questions can be distributed and evaluated at scale, then teams can feel confident doing volume hiring with this question format to get better results. If you’re still feeling like a non-believer, just look at the data. 

    Research Reveals Open-Ended Questions Are Better Than Multiple Choice for Hiring

    Our data scientists agreed that there were many pros and cons to using multiple-choice questions to evaluate a person’s abilities. They even listed a few of their own including:

    Multiple-Choice Pros

    • Easy to answer
    • Can ask a greater number of questions
    • Easy to score 

    Multiple-Choice Cons

    • Time-consuming to create
    • The data is only quantitative
    • Measurement is limited and lacks detail
    • There is a high chance of guessing and random responses

    With an understanding of multiple-choice limitations, our researchers wanted to understand if they could actually prove that open-ended questions are a better choice for candidate evaluation. Padma Pelluri, the Senior Psychometrician at HiringBranch and a Behavioural Assessment Specialist with more than ten years of experience found that when looking at data from the company’s past assessments, multiple-choice questions were unreliable compared to open-ended questions. Pelluri explains:

    “A majority of multiple choice questions display low positive correlations with the factors that they measure.”

    What does that mean? It means that multiple-choice can only measure surface-level skills and not deep-order skills, which evaluators may need to look at in order to properly assess a candidate. Consider Bloom’s taxonomy below, where different levels of thinking are organized into a hierarchy:

    Under this analysis, multiple choice questions could reveal a person’s abilities near the bottom of the pyramid but would fail to test a candidate for anything above Understand. Open-ended question evaluation would be needed to move further up the pyramid. 

    Pelluri found that evaluating with open-ended questions also led to higher reliability scores. She reported that “[open-ended] questions display correlations above 0.6, which indicates a high relatedness and correlation.” As you can see below, open-ended questions are four-times more reliable than multiple-choice. 

    Reliability wasn’t the only issue that emerged. The study also revealed a difference between multiple-choice questions and open-ended questions in terms of the average item difficulty index and in the item-factor correlations. 

    Another big theme was consistency. In high volume assessments, as the term suggests, there is a large number of candidates taking a given assessment. Therefore the chances of guess work in the assessment are also high, and random patterns can emerge from the data, which may result in unfit candidates passing the assessment. Alternatively, open-ended questions measure exactly what is needed in a consistent and robust manner.

    *This internal Reliability Index is based on Crohnbach's Alpha and was applied to sample data from past HiringBranch assessments for the purposes of this research. 

    Going Beyond Multiple Choice

    In order for teams to move beyond the limited choices that multiple-choice assessments offer them, they need to understand what the viable alternatives are. Open-ended-question assessments that can be evaluated at scale are changing the game for hiring teams. To summarize the pros and cons check out our downloadable battle card below!

    Download the full battle card here.

    Image Credits

    Feature Image: Unsplash/Thomas Park

    Image 1: Via Valamis

    Read More Blogs
    View all
    Read report
    Read report
    Read report
    Read report
    Read report
    Read report
    Read report
    High-Volume Recruitment
    Contact Center
    Best Practices

    Getting Through the Contact Center’s High-Volume Season with Ease

    Read report
    Read report
    Read report
    Read report
    Read report
    Read report
    Read report
    Read report
    Read report
    Read report
    Contact Center
    High-Volume Recruitment

    Addressing Call Center Turnover Rate

    Read report
    Contact Center
    High-Volume Recruitment
    Best Practices

    Average Attrition Rates and How To Improve Them in High-Volume Roles

    Read report
    Read report
    Read report
    Read report
    Read report
    Talent Acquisition
    Artificial Intelligence

    6 Quick-Read Stats About AI and Talent Acquisition

    Read report
    Assessments
    Talent Acquisition

    Candidates Speak Up About Assessments

    Read report
    Artificial Intelligence
    Digital Transformation

    Adding AI to Your HR Tech Stack: Build, Buy or Partner?

    Read report
    Talent Acquisition
    Best Practices

    Untangling Quality of Hire

    Read report
    Read report
    Events
    High-Volume Recruitment
    Artificial Intelligence

    Top 7 Moments in Our Summer Podcast Tour

    Read report
    Talent Acquisition
    High-Volume Recruitment

    The History of Interviews According to Kevin Wheeler

    Read report
    Read report
    Read report
    Read report
    Read report
    Read report
    Artificial Intelligence
    High-Volume Recruitment
    Digital Transformation

    How is Artificial Intelligence Changing the Recruiting Process?

    Read report
    Read report
    High-Volume Recruitment
    Assessments
    Digital Transformation

    The No-Interview Hiring Model is Changing Talent Acquisition Forever

    Read report
    Read report
    Read report
    Artificial Intelligence
    Contact Center
    Customer Service
    Digital Transformation

    Do Your Call Center Agents Accurately Represent Your Brand Personality?

    Read report
    Assessments
    High-Volume Recruitment

    The Essential Guide To Pre-Hiring Assessments

    Read report
    Assessments
    Customer Service
    Digital Transformation
    High-Volume Recruitment

    How To Hire For Great Communication Skills Without Interviewing?

    Read report
    Artificial Intelligence
    Contact Center
    Customer Service
    Digital Transformation

    Are Bots The Key To Providing Outstanding Customer Support?

    Read report
    Assessments
    Contact Center
    High-Volume Recruitment

    Finding Candidates in the Digital Age with New Hire Assessments

    Read report
    Read report
    Read report
    Contact Center
    Digital Transformation
    High-Volume Recruitment

    How Do Virtual Contract Recruiters Help Companies Hire Fast And Accurately?

    Read report
    Contact Center
    Customer Service
    High-Volume Recruitment

    A Guide To Hiring And Managing Frontline Employees

    Read report
    Artificial Intelligence
    Contact Center
    Digital Transformation
    High-Volume Recruitment

    How To Scale Recruitment With A Virtual Recruiter: Accurate, Quick, and Performance-Based

    Read report
    Read report
    Read report
    Contact Center
    Customer Service
    High-Volume Recruitment
    Sales

    How To Guide: 3 Essential Steps For Hiring A Quality Frontline Employee

    Read report
    Artificial Intelligence
    Contact Center
    High-Volume Recruitment

    5 Ideas To Transform Your Contact Center

    Read report
    Read report
    Read report
    Contact Center
    Customer Service
    High-Volume Recruitment

    How To Hire Top Talent Remotely?

    Read report
    Read report
    Contact Center
    Customer Service
    High-Volume Recruitment

    How To Tackle 7 Common Contact Center Issues

    Read report
    Assessments
    Digital Transformation

    The Modern Workplace Demands Soft Skills

    Read report
    Contact Center
    Customer Service
    Digital Transformation
    High-Volume Recruitment

    Top 5 Things You Need to Know About Customer Support

    Read report
    Read report
    Read report
    Read report
    Artificial Intelligence
    Digital Transformation
    High-Volume Recruitment

    Case Study - The advantages of using AI for assessing voice and chat candidates

    Read report
    Read report
    Read report
    Read report
    Read report
    Artificial Intelligence
    Assessments
    Contact Center
    Digital Transformation
    High-Volume Recruitment

    High-volume recruitment is time-heavy and expensive - 3 tips to help

    Read report
    Read report
    Digital Transformation
    High-Volume Recruitment

    How to interview 20,000 people a year

    Read report
    Read report

    Does SHL guarantee a performance improvement?

    We do. Gone are the days of guessing skills in an interview. Rely on technology to do it for you with 99% accuracy and boost your hire quality, guaranteed.

    Some of our happy customers

    Try an assessment today!

    Get a sample assessment sent to your inbox and experience the HiringBranch difference yourself.
    Thank you for requesting a sample HiringBranch assessment
    We are verifying your information and will send you a link to test drive HiringBranch soon!

    If you are a candidate (and not an employer), kindly note that you will not receive a sample assessment. We invite you to visit our Help Center for candidate related questions and answers.
    Oops! Something went wrong while submitting the form.

    Try an assessment today!

    Get a sample assessment sent to your inbox and experience the HiringBranch difference yourself.
    Thank you for requesting a sample HiringBranch assessment
    We are verifying your information and will send you a link to test drive HiringBranch soon!

    If you are a candidate (and not an employer), kindly note that you will not receive a sample assessment. We invite you to visit our Help Center for candidate related questions and answers.
    Oops! Something went wrong while submitting the form.
    Test drive a HiringBranch assessment

    Try an assessment today

    Get a sample assessment sent to your inbox and experience the HiringBranch difference yourself.
    Thank you for requesting a sample HiringBranch assessment
    We are verifying your information and will send you a link to test drive HiringBranch soon!

    If you are a candidate (and not an employer), kindly note that you will not receive a sample assessment. We invite you to visit our Help Center for candidate related questions and answers.
    Oops! Something went wrong while submitting the form.