How Talkume's AI Estimates Your Speaking Level — and What It Can't Do

When you press "Check language level" on a Talkume link, about a minute later you get a level from A1 to C2, a breakdown by skill, a few comments and a transcript. This post explains what happens in that minute, why it's built the way it is, and — just as important — what it can't tell you.

Why not just ask one AI for a level?

The obvious approach is to give a recording to one AI model and ask "what level is this?" We tried that first. It rated almost everyone as an intermediate speaker, native speakers included, and it sometimes "found" grammar mistakes that were never said. A single model asked for a single label tends to play it safe.

So the evaluation is split into steps that each do one job, and the final level is calculated from them rather than guessed in one go.

Step 1: Listen and write down exactly what was said

A speech model listens to the recording and writes a word-for-word transcript — including the mistakes, false starts and filler words. That last part matters: most transcription tools quietly correct grammar, which is helpful for meeting notes and useless for judging a language level. It also notes the language, the pace, pauses, and any words that were hard to make out.

Step 2: Measure what can be measured

For English, French, German, Italian and Spanish, a separate speech recognizer analyses the same recording and produces numbers instead of opinions:

• Clarity — how much of the speech it could recognize with confidence. Clear pronunciation scores well; an accent only lowers this when it makes words genuinely hard to recognize.
• Speaking speed — words per second while actually speaking, with pauses taken out. The bands are adjusted per language, because German, for example, packs more meaning into fewer, longer words than Spanish.
• Words between pauses — longer runs usually mean more fluent speech.

This step also double-checks the language. If the first model wrote down a different language than the one actually spoken — it can happen with a strong accent — the recording is listened to again in the right language before anything is rated.

Step 3: Two independent examiners rate five skills

Two AI models from different developers each read the transcript, the listening notes and the measurements, and rate five skills separately — the same kinds of criteria human examiners use for the CEFR:

• Vocabulary — how wide and precise the language is.
• Grammar — how reliably basic and complex structures are used.
• Fluency — how smoothly the speaker keeps going.
• Coherence — how well ideas are linked and organized.
• Pronunciation — how easy the speaker is to understand.

Each examiner has to quote the transcript when it points out a mistake, so comments are tied to things you actually said. Each one rates the recording twice, because the same model can swing by a level between two runs; four ratings average out those swings.

Step 4: Combine the ratings into one level

The four ratings are averaged skill by skill. The overall level is then the middle value of the five skills, so one unusually strong or weak skill can't drag the result up or down on its own. One extra rule: the overall level can't be more than one step above the vocabulary rating, because very limited vocabulary caps how well anyone can really communicate, however clear their pronunciation.

How we checked it

Before launch, the evaluation was tested on recordings that human experts had already scored, on native speakers with different accents (it shouldn't punish an Indian or Australian English accent, for example), and on test recordings with typical learner mistakes and accents in all five measured languages. The results followed the expert scores closely, though not perfectly — which is why it's presented as an estimate.

What it can't do

• It's not a certified test. It can't replace IELTS, TOEFL, DELF or Goethe exams, and it doesn't test reading, writing or listening.
• A short sample shows less than you know. Thirty seconds of simple sentences can't show C1 vocabulary even if you have it. Results from short recordings are labelled as such.
• Reading a script can flatter the result. A polished text read aloud sounds better than how most people speak spontaneously, and the AI can't always tell the difference.
• Strong accents and noise can cause slips. If a word is misheard, the grammar rating is based on what was heard, not what was meant.
• Other languages get a simpler check. Beyond the five measured languages, the level is rated from the transcript and listening notes only.

Getting a fair estimate

Speak freely rather than reading, aim for 45 to 90 seconds so there's enough to judge, keep it to your own voice and one language, and talk at your normal pace. For recording itself — mic distance, rooms, takes — see how to record a voice introduction. And if you're unsure what the levels themselves mean, CEFR levels explained covers that.

The evaluation only runs when someone signed in asks for it, and the result is saved to that link — so it isn't run again for that link, and anyone you share it with sees the same result.