Your AI Hiring Tool Might Be Guessing at The Most Important AI Skills

Every hiring leader wants to hire someone who is "comfortable with AI" right now. Ask that same hiring leader how they screen for that, and the answer gets vague fast. That's because the skills that make someone good with AI aren't technical. Anyone can learn to type a prompt in an afternoon. What makes a strong AI user are skills like judgment, critical thinking, and attention to detail; in other words, soft skills.

In the World Economic Forum's Future of Jobs Report 2025, analytical thinking ranked as the top core skill for the third consecutive year, with seven in 10 companies calling it essential. Resilience, flexibility, and agility were also listed among the top skills. That said, many of the tools used to screen for these skills run on the same generic AI models that can’t actually measure soft skills (unlike HiringBranch, which can measure these with 98% accuracy).

The Soft Skills Behind Every Good AI User

Generative AI is fluent and fast, but it is also sometimes wrong (and it rarely tells you when). That puts the burden of truth on the person using AI technology. Employees with the right skills will recognize these types of limitations and be able to extract the full value from AI, and these skills have nothing to do with coding. Here’s a list of the top skills needed for AI use, and why they’re so important:

• Judgment: This skill means knowing when AI is the right tool to use, when it isn't, and when an answer is good enough to act on. A frontline agent who pastes an AI-suggested reply to an upset customer without judgment can do more damage than no reply at all.
• Critical thinking: This skill means questioning the output instead of accepting it. Does this answer make sense? What is it assuming? What did it leave out?
• Attention to detail: This skill involves catching the wrong information like a policy number, a statistic that isn’t accurate or is inconsistent, or a tone that's slightly off. AI errors are often small and buried in otherwise polished text.
• Clear communication: This skill is the opposite of being vague. It requires people who can explain what they need, with the right context, just as they do from colleagues.
• Adaptability: This skill requires a person to be comfortable with ongoing change. After all, AI tools are evolving every few weeks now, it seems.
  

A 2025 study from Microsoft Research and Carnegie Mellon University surveyed 319 knowledge workers who use generative AI at least weekly. Workers with more confidence in AI applied less critical thinking to its output, while workers confident in their own skills applied more. 

The Skills AI Can't Fake

There are skills needed to effectively wield AI, like those listed above, but then there are also essential skills that AI can't do at all. These are the traits that make interactions human. And with the onset of AI in the workplace, these human skills are arguably needed more than ever.

For example, when a customer calls an airline about a third delayed flight, they need a customer service agent who hears the frustration, adjusts their tone, and makes them feel taken care of. Some of the skills involved in that type of interaction could include empathy, active listening, and the ability to read a situation as it unfolds.

In frontline service roles, humans often prefer other humans anyway. In a Gartner survey of 5,728 customers, 64% said they would prefer companies not use AI for customer service. Their top concern, cited by 60%, was that AI would make it harder to reach a person.

Linguists call the study of human communication socio-pragmatics. This discipline analyzes how context, power dynamics, and social norms shape what we say and how it lands. It's what lets a great rep know when to slow down, when to stay on the problem, and when it's time to wrap up the call. Large language models are built to predict the next likely word, which is a fundamentally different skill.

Even newer architectures built to move beyond next-word prediction, like Meta AI's VL-JEPA, are focused on efficiency and representation learning, not on modeling the social judgment that soft skills require. As AI takes over more routine tasks, these human moments are becoming a bigger share of the job.

So Why Are We Letting Generic AI Score Them?

The in-house scientists at HiringBranch have experimented with the limitations of LLMs. When our research team had AI agents handle simulated high-stress customer service calls, with detailed instructions and explicit direction to use soft skills, the AI got the facts right every time, but the conversations were flat and formulaic. Empathy was expressed as a template, not a thoughtful or felt response to the person on the other end.

Empathy was expressed as a template, not a thoughtful or felt response to the person on the other end.

Most people assume these AI models can do everything accurately. Our research team put that assumption to the test and compared how closely different models agreed with expert human raters when scoring real conversations for soft skills. The results revealed:

• HiringBranch's purpose-built scoring model reached a correlation of r = 0.846 with human judgment on a fully held-out test set. That explains about 71% of the variation in human scores.
• Leading LLMs from OpenAI and Google, given carefully engineered prompts, landed between r = 0.65 and r = 0.76. That explains roughly half.

In practice, the generic models stumbled on the same things a strong AI user would catch. They penalized appropriate conversational pivots, like a rep politely closing a call after empathy was already established. They lost track of intent across multiple turns…they missed nuance.

Other researchers have come to the same conclusion as the HiringBranch team. A survey of pragmatics benchmarks presented at ACL 2025 (Ma et al., 2025) found that evaluating how well language models handle context-dependent meaning remains a persistent challenge, even as their fluency improves. 

And regardless, at the end of the day, improved fluency is still not the same as human judgment. If an LLM can’t effectively express soft skills, why would a hiring team trust it to evaluate skills like communication, empathy, and critical thinking?

What This Means If You're Hiring at Scale

The workforce will be shaped by two kinds of soft skills over the next few years and hiring teams need an effective way to measure these. The first set defines whether a person can be effective with AI, including judgment, critical thinking, attention to detail, and clear communication. The second set covers what AI can't do, like empathy, active listening, and reading the moment (context).

Both sets of skills matter more than ever, and both are notoriously hard to measure at scale or systematically. If your hiring stack relies on a generic LLM, or a vendor wrapping one, to score these skills, it's worth asking a few questions:

• How was the model built, and what was it trained to recognize?
• Has it been validated against human expert judgment?
• Has it been validated against real on-the-job performance?

At high volume, a model that captures only half of what human experts see can turn into thousands of hiring misses. Look for a scoring engine that is grounded in linguistic research and workplace communication, and validated against real human judgment and real on-the-job outcomes (cough cough, ah hem…like HiringBranch). 

Further reading: Discover earlier research comparing generic LLMs to our purpose-built engine, which measures soft skills 32% to 50% more accurately.

‍

Image Credits

Feature Image: Unsplash/Simone Secci

‍