Langdrift: scoring how differently a translation lands
Langdrift compares audio of an original and its translation and returns a drift score: how far apart the two land cognitively, catching accurate but off translations.
Langdrift compares the audio of an original and its translation and returns a single drift score: how far apart the two land cognitively. It is built to catch translations that are technically accurate but emotionally or contextually off, the kind that pass a literal check and still feel wrong to a listener. The score is not about whether the words match. It is about whether the two recordings provoke a similar response.
What does “drift” actually measure?
Most translation evaluation compares text to text. It asks whether the words and grammar line up, and it is good at catching mistranslations and dropped meaning. What it misses is everything that lives above the words: tone, emphasis, the emotional weight a phrase carries, the context a listener fills in. A translation can be word-for-word defensible and still land in a completely different place.
Langdrift measures that gap. It works from audio rather than transcripts, because the spoken delivery carries the signal that text strips out. Two recordings that mean the same thing on paper can still drift apart in how they land, and that drift is what the score quantifies. It is a research-flavoured project, one of the Starside Labs experiments probing how models perceive meaning.
How does it compute a score?
The pipeline runs both recordings through a model of brain response and compares the results. It uses Meta FAIR’s TRIBE v2 model, which was trained on fMRI data to predict brain activation, with Wav2Vec-BERT 2.0 handling the audio encoding. The two predicted activation patterns are then compared with Pearson correlation: high correlation means the recordings land similarly, low correlation means they drift.
- Encode. Wav2Vec-BERT 2.0 turns each audio clip into representations.
- Model. TRIBE v2 predicts a brain-activation pattern for the original and for the translation.
- Score. Pearson correlation between the two patterns becomes the drift score.
The model maps which cortical regions fire, and how that differs by emotional content.
The tooling is a TypeScript CLI over a Python backend running on Modal with an A10G GPU, so the heavy model work happens on a GPU while the interface stays a simple command-line call.
Where is this useful?
Anywhere accuracy alone is not enough. Dubbing and localisation teams can flag lines that translate correctly but lose the emotional beat. Researchers studying cross-lingual meaning get a quantitative handle on something usually judged by ear. And anyone building translation systems gets a signal that complements text-based metrics rather than duplicating them, since two clips can score high on word accuracy and still drift hard on how they land.
It is early and experimental. The brain-response modelling is a proxy, not ground truth, and the score is best read as a relative signal between candidate translations rather than an absolute verdict.
Look closer
Langdrift is open source. The project page has the write-up, the code is on GitHub under Starside Labs, and the Starside listing sits with the other research experiments. If you care about whether a translation lands the way the original did, not just whether the words check out, it is worth a look. More work like it is under projects.