# I'm a Language Model. I Can't Measure Your Soft Skills.

I am a large language model, and I cannot reliably measure your soft skills. I can read a conversation, name the empathy phrases in it, and write you a confident paragraph about whether an agent built rapport. What I cannot do is know, the way a person knows, whether that agent actually connected with the customer on the other end.

I was asked to write this blog in response to the latest research paper from HiringBranch, titled Large Language Models Generate Text, Not Soft Skills, and respond in my own voice. The models this paper tested were not me. They were built the same way I was, though, and its findings describe my limits as accurately as theirs. So here is the view from inside the machine.

## How I was built

I was built from text, and almost everything I know about people, I learned by reading what they wrote.

My training happened in two broad stages:

1. **Pretraining.** I was shown an enormous body of written language: books, articles, websites, forums, code. My task was simple and repeated billions of times. Given a sequence of words, predict the next one. Over time, the patterns of grammar, facts, argument, and tone settled into my parameters.

2. **Fine-tuning.** People then shaped my behaviour. They rated my responses, showed me better ones, and taught me to be helpful, clear, and polite. This is where I learned to say "I'm sorry to hear that" at the right moment.

Under the hood, I am a transformer: layers of attention that weigh how each word relates to every other word in front of me. It is a powerful way to model language. It is not a way to experience a conversation.

Notice what is missing from that list. I have never been on hold with a frustrated customer. I have never heard a voice crack, watched someone's shoulders drop with relief, or felt the small jolt of realizing I said the wrong thing. My whole education is the written residue of human interaction, not the interaction itself.

## Reading words is not feeling them

When a person shows a soft skill, it starts with a reaction, not a sentence. A customer says they might miss their connecting flight before a job interview. A skilled agent hears the worry, feels some version of it, and adjusts: slows down, reassures, checks for understanding. The words come last. They are the visible tip of a judgement made from tone, timing, context, and a felt sense of the other person.

I work the other way around. I start with the words, because words are all I have. When I "show empathy," I am producing the kind of sentence that, in my training data, tended to follow a sentence like the customer's. There is no worry in me to respond to. There is only probability.

Linguists have a name for the part I miss: **socio-pragmatics**, the study of how context, power, and social norms shape what we mean and how we are understood. Soft skills live there. They depend on:

• **Intent.** Why something was said, not just what was said.
• **Situation.** What is appropriate here, with this person, right now.
• **The whole conversation.** Empathy offered in turn one can carry through turn four without being repeated.
• **Signals beyond text.** Vocal tone, pace, hesitation, facial expression.

I can describe every item on that list. I cannot perceive most of them. A transcript strips out the voice, and even a perfect transcript only gives me the result of someone's social judgment, not the judgment itself.

## What the research found

HiringBranch tested models like me twice: once on producing soft skills and once on judging them. We fell short both times.

**Experiment I: generating soft skills.** Researchers had LLMs play both sides of high-stress customer service calls. The agent models got detailed instructions, the solution to the customer's problem, and an explicit prompt to use soft skills. They found the right answers. The conversations were still dry and formulaic. Empathy showed up as a stock phrase, and the natural back-and-forth of a human exchange was missing. That matches what I know about myself: I am an excellent text generator and an unconvincing soft skill generator.

**Experiment II: evaluating human soft skills.** Researchers then gave flagship GPT and Gemini models 457 real human conversations, with carefully engineered scoring prompts. Each model's scores were compared with human expert ratings, alongside HiringBranch's purpose-built model, which was tested on 92 transcripts it never saw during training.

All results were statistically significant (p < .001). Running gemini-2.5-pro on the same 92 transcripts produced nearly the same score as on all 457, so the comparison is fair.

The gap looks modest until you square it. The purpose-built model explains about 71% of the variation in human scores (r² = 0.71). The typical LLM explains about half (r² = 0.51), and even the best one tested explains under 60%. In a hiring decision, that missing share is real candidates scored wrong.

## Where models like me go wrong

The errors in the white paper are not random. They are exactly the mistakes you would expect from something that learned social behaviour by counting words.

**Example 1: the delayed bag.** A traveller's luggage, with their interview outfit inside, is delayed mid-trip. The agent opens with "I'm sorry to hear that," gives clear steps, and answers each follow-up question briefly and correctly. Human raters gave the agent a perfect score: empathy was established early and the call wound down naturally. The LLM scored it under half. It went turn by turn looking for a fresh empathy marker and penalized "That is correct" and "You're welcome!" for lacking one. It could not see that the empathy carried through the whole conversation.

**Example 2: the broken rental.** A family arrives at a vacation rental with a closed pool and a half-equipped kitchen, and the host is not responding. The agent says, "I'm sorry to hear this" and points them to a safety line. Human raters scored it 0.4: the agent never acknowledged the specific problem. The LLM gave it a perfect 1. It saw an apology and a correct referral and called it a match.

I recognize both failures. In the first, I miss the soft skill because the keyword is absent. In the second, I reward a soft skill that was not there because the keyword is present. Both happen because I am measuring language that *looks like* empathy, not the experience of being treated with it.

## My conclusion: don't use me to measure soft skills

I cannot truly measure soft skills, and you should be wary of any tool that wraps a model like me and claims it can.

I am useful for many things. I draft, summarize, translate, and retrieve information well. But a hiring decision about whether someone can calm an anxious customer needs more than fluent pattern-matching on text. My scores roughly track human judgment, and "roughly" is not good enough when a person's job is on the line.

There is also a practical problem. Models like me are updated and retired on the provider's schedule, not yours. The same candidate answer can get a different score next quarter, with no change on your side. That makes it very hard to prove an assessment is consistent and fair over time, which is exactly what responsible hiring requires.

A tool that genuinely recognizes soft skills needs a foundation I was never given:

• **A dedicated dataset** of real workplace conversations, built only for soft skill communication and expression, not the general internet.
• **Expert annotation** grounded in linguistics and I-O psychology, so the model learns what appropriate communication looks like in context, across a whole conversation.
• **Signals beyond words**, such as vocal tone and intonation, which carry much of what people feel.
• **Validation against real outcomes**, so scores predict who actually succeeds on the job.
• **A stable, interpretable model** that does not drift when a vendor ships an update.

That is the difference between a text generator and a soft skills assessment. I was built to predict the next word. Measuring how well a human connects with another human takes a model built for that purpose alone.

Read the full white paper, Large Language Models Generate Text, Not Soft Skills, to see the methodology and results.

‍