The three things people mean by "speaker labels"
| Task | Question it answers | Output |
|---|---|---|
| Transcription | What was said? | Words, with timestamps |
| Diarisation | Who spoke when? | Segments grouped into anonymous speakers |
| Speaker recognition | Which known person is this? | A name, matched to a voice profile |
Diarisation produces "Speaker 1" and "Speaker 2", not "Marta" and "Alex". Attaching real names is a separate step, usually done by matching against the meeting’s participant list rather than by voice, which is why the labels are often right in a two-person call and drift in a six-person one.
How it works
The audio is cut into short windows, typically a second or two. Each window is turned into an embedding, a vector meant to capture the characteristics of the voice rather than the content of the speech. Those embeddings are clustered, and each cluster becomes a speaker. Finally the boundaries are refined, because the initial cut rarely lands where a speaker actually changed.
The number of speakers is usually not known in advance, so the clustering has to decide it. This is the part that goes wrong: given a noisy call, the same person can end up split across two clusters, or two people with similar voices merged into one.
How accurate is it?
The standard measure is diarisation error rate, which adds up the time attributed to the wrong speaker, missed speech, and false alarms, as a proportion of total speech time. Lower is better. On clean, well-recorded meeting audio, current systems land in the low-to-mid teens as a percentage; on the AMI meeting corpus, a standard benchmark with real overlapping conversation, published figures for widely used open models sit around 11%.
What makes it worse
- Crosstalk. Two voices in the same window produce an embedding that belongs to neither, and overlapping speech is where most of the error lives.
- Laptop speakers instead of headsets, because the microphone picks up the other participants and the system has to separate a voice from its own echo.
- Short turns. "Yeah" is a third of a second, which is not much to characterise a voice with.
- Similar voices. Two people with the same vocal range in the same room is the classic merge case.
- Low bandwidth, which strips exactly the frequency detail the embeddings depend on.
What to do about it
Very little, at the software layer. The practical fixes are the ones a meeting organiser controls: headsets, one person speaking at a time, and names said out loud early. If a transcript matters, as it does for a hiring panel or a client scope discussion, being able to correct a speaker label once and have it apply to the whole meeting is worth more than another point of benchmark accuracy.