Do you want candidates who can pass a language test? Or candidates who can communicate with customers?

Summary

Language assessments are a part of every contact center recruiter’s toolkit. Whether it’s checking grammar or spoken fluency, the goal has been to ensure that an applicant has a basic proficiency level in English or another language.

As the industry automates, and customer expectations increase, the job of the customer service agent is becoming more complex. Is the standardized language test still the right tool to use? This paper looks at the differences between standardized language tests and more sophisticated communication assessments.

Three significant differences are found. The standardized language tests are lacking in communication skills, comprehension skills as well as accuracy in spoken fluency. This results in a different ranking of candidates. The paper provides examples of how a standardized language test can result in ‘false positives’ where the wrong candidate is passed, as well as ‘false negatives’ where a good candidate is overlooked. The paper also shows how a more sophisticated communication assessment can more accurately select for frontline customer service job performance.

‍

Problem

Almost every contact center agent working today has passed a language test of one kind or another. But how well do those tests work? How often have language test results correlated with the candidates’ actual performance? Over one billion USD is spent by corporations on language testing every year. This represents a significant cost for Talent Acquisition departments. Are they getting value for the dollar?

Research shows that in offshore centers (outside North America and Europe), up to 20% of agents do not have the required language and communication skills to do their job effectively. ¹

And across the industry, customer satisfaction levels are reported to have declined in 2020. According to data compiled by the CFI group, this year’s reading is the lowest score since the report was first issued and it represents the uphill climb many companies face in satisfying increasingly demanding consumers.

"In the world of self-service, customers now reserve only the toughest problems for the contact center agents," says Sheri Petras, CEO of CFI Group. "Many simple questions are now handled by the customer on the company website, saving the complicated, frustrating questions for contact center agents. Organizations must implement the systems and processes needed to provide effective online self serve tools to customers, while also ensuring that customers easily can reach an effective and empowered agent when needed.

"One of the biggest challenges facing the recruiting industry is to both fill seats AND provide an accurate screen. Even one poor hire who lacks language skills (a ‘false positive’) and has to be either trained, shifted into another position, or let go, represents a significant cost for an organization.

Similarly, in a competitive job market, no employer wants to pass over quality candidates who may fail a standard language test for the wrong reasons.

So why are the standard industry language tests selecting the wrong candidates?

Analysis & Solution

We looked closely at an industry-leading standardized language test and compared it with a language assessment designed specifically for frontline contact center employees. We found three significant differences between the two types of assessments.

The following tests were compared:
‍
1. HiringBranch Pre-Hire Communication Assessment English (HB): The test consists of Reading, Listening, Writing and Speaking for Contact Centers
Duration: 45 minutes
2. Industry-leading standardized test for Speaking and Writing (ST): There are 2 tests - Speaking (15 minutes) and Writing (35 minutes)
Total Duration: 50 minutes

A sample of non-native English speakers took both the HB and ST tests and the results were compared. The results for Speaking, Writing, and Final Score for each test were correlated. While there was some correlation in Writing, there were lower correlations for Speaking and Final Score, which means that the tests selected different candidates for passing. The main reasons are outlined below.

‍

Communication Skills

Standardized Language Tests (ST) focus on language proficiency only whereas HiringBranch (HB) adds another layer on top of that in which communication skills are assessed. Thus, HB does not only check how the candidate says things but also what they say. This is done based on real scenarios that imitate the candidate’s anticipated work, which enables HB to assess the candidates that are fit for the intended environment.

HB’s test not only checks language proficiency but also assesses the candidate’s real-world communication skills, like paying attention to details, acknowledging the customer’s concern, their ability to perform under pressure, using positive language, etc. To see how the incorporation of communication skills into a test affects its validity, let’s look at the following data:

Correlation Coefficient between Communication Skills and Test Scores:

^{Table 1 - The Pearson correlation coefficient between communication skills and test scores}

Table 1 shows that there is a high correlation between the candidate’s communication skills and their test results, not only in comparison to HB but also in comparison to ST. Thus, communication skills are highly efficient in predicting both the candidate’s linguistic ability and their chances to succeed in their future work.

‍

To demonstrate this, let’s look at a few specific outliers: For the following question, the candidate needs to show that they understood the customer’s issue as well as to assure the customer that they are going to help them.

The following response, of a candidate named Sam, is quite good. Sam rephrases the customer’s issue in a concise way that shows that Sam understood it. Then, Sam describes how the feature should work. Moreover, the sentence “It would help us resolve your issue better” conveys Sam’s commitment to helping the customer.

Now, let’s consider the following response to the same question written by candidate Mary:

Mary repeats the word “issue” twice but does not state what the issue is. Thus, we cannot know whether Mary understood the issue or not. The response includes a general statement that shows Mary’s willingness to help, but when Mary started writing the actual solution, Mary ran out of time and did not complete the task.

In this case, then, not only did Mary not fulfill the requirements of the task, but Mary was also unable to complete it within the four-minute deadline. Thus, from the perspective of skills like working under pressure and attention to detail, two crucial skills in Mary’s future work, Mary was unable to deliver.

Sam’s communication skills are better than Mary’s. Nonetheless, Mary was ranked quite high in the Standard’s test while she failed HB’s test. The opposite is the case with Sam.

^{Table 2 - Mary vs. Sam’s rank in percentile (10th percentile means only10% of candidates have lower scores.)}

Similar differences in the ranking are found with other scenarios, both in writing questions and in speaking questions. In many cases, the standardized language test produces a different ranking than the HB

Spoken fluency

The second issue that accounts for the imperfect correlation between HB and ST speaking tests (and by extension between the final scores of both tests) is the reliability of the spoken fluency measurements in each of the tests.

To enable a fair comparison, the results were manually scored for spoken fluency and pronunciation separately by two people. Their manual scores were then averaged to better represent the candidate’s spoken fluency.

Correlation Coefficient between Fluency and Test Scores

^{Table 4 - The standard test’s correlation coefficient between the tests’ speaking scores and manually scored spoken fluency}

As can be seen from Table 5, HB speaking test results correlate better with the candidates’ spoken fluency.

Fluency is an important component of ‘intelligibility’, or the ability for a listener to easily understand a speaker without having to ask for clarifications. It is separate from accent or pronunciation and is a key factor in how smoothly a spoken conversation can be conducted.

The HB assessment uses a variety of metrics that together score on intelligibility. The ST assessments take a more traditional approach. This leads to differences in how the

Let’s look at all the example candidates and how they we reranked.

Differences in Candidate Rankings 
90th percentile is high, 10th is low

^{Table 6 - Mary, Kristine, Oliver and Sam rankings in percentile}

As can be seen from Table 5, HB speaking test results correlate better with the candidates’ spoken fluency.

Fluency is an important component of ‘intelligibility’, or the ability for a listener to easily understand a speaker without having to ask for clarifications. It is separate from accent or pronunciation and is a key factor in how smoothly a spoken conversation can be conducted.

The HB assessment uses a variety of metrics that together score on intelligibility. The ST assessments take a more traditional approach. This leads to differences in how the candidates are ranked in the speaking results.

Further discussion

Why would the addition of communication skills and comprehension matter?

We know from research and surveys that specific communication skills correlate directly with agent performance, such as customer satisfaction, average contact handle time, reopen rate and customer satisfaction.

Industry surveys show that the top 3 skills contact center trainers look for (in addition to spoken and written fluency)are the ability to listen carefully and acknowledge, pay attention to detail and to reassure a customer that their needs will be looked after.<sup>2

‍</sup>Recent consumer behaviour research conducted on ‘concrete language’ or attention to detail, as well as listening or ‘attending to’ a customer, shows that even small changes in words used by agents have positive outcomes.

The field data suggest that increasing linguistic concreteness by one standard deviation improves customer satisfaction by 9% and actual spending by at least 13%. ³

This data reinforces the appropriateness of the HB assessment in targeting the real-world skill requirements of frontline customer service employees.

Conclusions

Despite the high correlation between candidates’ scores in HB and ST tests, we have shown three major differences between the two tests:

1. While both HB and ST evaluate linguistic proficiency, HB’s test adds a layer of assessment that accounts for the candidate’s communication skills. This layer contributes dramatically to the prediction of the candidate’s chances to succeed in their prospective work.
2. HB achieves more reliable results in scoring candidates’ spoken fluency.
3. HB is the only test that analyses the candidates’ reading and listening abilities, two crucial components in the candidates’ future work.

The multiple layers of assessment and test components enable HB to easily spot false positives, thus addressing one of the biggest challenges of the recruiting industry by saving time and money.

The HB assessment also addresses false negatives and ensures that the right candidates are not passed over.

Contact centers looking to hire for customer service excellence can consider a wider range of testing than what is available from standard language tests. Specifically, assessments that evaluate communication, intelligibility and comprehension skills will produce different results thana standard language test and the results will be more accurate for performance in the contact center work environment.