← back to the demo
GNM Studio
A human head that lip-syncs from audio, running entirely in your browser.
Nothing here is pre-rendered and nothing is computed on a server. The face
is generated vertex by vertex in JavaScript, and a small neural network
listens to the audio and decides what the mouth should do — about 5 ms of
work for every 90 ms of sound.
The three parts
1. The head
Google's Generic Neutral Model is a human head expressed
as maths rather than a fixed mesh: 17 821 vertices, 253 components that
change who the face is, and 383 that change what it is doing.
Adding those components together produces a face. Doing it every frame is what
lets the head talk, blink and change identity live.
2. The listener
openlips-s is a neural network with 2.37 million
parameters — small enough to download in a couple of seconds. It reads the
audio as a spectrogram and outputs the 383 numbers that pose the face, 100
times a second.
It has about five seconds of memory. That matters more than it sounds:
the shape your mouth makes for a sound depends on the sounds around it, so a
system that judges each instant in isolation always looks twitchy.
3. The face rig
Speech alone is not a performance. On top of it: an emotion layer that adds
rather than replaces, blinking with natural irregular timing, eye gaze where
the lids follow the eyes, sticky lips that resist parting and peel from the
centre outward, strand-based beard and eyebrows bound to the skin so they
deform with it, and screen-space ambient occlusion.
How it learned
By imitation. NVIDIA's Audio2Face is far larger — 159 MB, needs a
dedicated GPU, cannot run in a browser. We ran it over 10.79 hours of
freely-licensed speech, recorded exactly what it did with the face,
and trained the small network to reproduce that.
The mistake that mattered most.
The first pipeline translated the teacher's output
into 52 standard face controls, then into 14 mouth shapes. That second step
threw away so much that reconstructing the face from it was
worse than guessing the average face. Smiles,
lip rolls and upper-lip motion were destroyed outright. Every model trained
that way was faithfully learning a target that had already lost the point.
Skipping both translations — mapping the teacher's
geometry straight onto the head — cut the error by 7×, and is the
single change that made this work.
Does it actually work?
Measured on speakers the model never heard during training, in millimetres
of real error on the mouth — not on whether labels agree:
| Driver | Mouth error | Movement |
Lip closures | Lag |
|---|
| Audio2Face (the teacher) | 0 mm |
100% | 1.00 | 0 ms |
| A motionless face | 2.29 mm |
0% | 0.00 | — |
| Hand-written rules | 3.00 mm |
81.8% | 0.28 | 53 ms |
| openlips-s |
1.15 mm |
89.9% |
0.69 |
10 ms |
Worth noticing: the hand-written rules have higher
error than a face that never moves at all. They move convincingly but not
correctly — which is exactly why movement and timing are reported next to
accuracy instead of beneath it. A model that scores well by sitting still is
the easiest trap in this field to fall into.
Honest limitations
- Still about twice as jittery as the teacher it learned from.
- Trained on read speech. Singing, shouting and heavy accents are outside
what it has seen.
- Tongue and teeth are approximate — the teacher is not precise there
either.
- Not yet measured on a low-powered laptop or phone.
Credits
GNM head model © Google, Apache-2.0 — used with
attribution and gratitude.
NVIDIA Audio2Face-3D was the teacher for distillation. Its
weights are not included or redistributed.
Speech corpora: Common Voice (CC0), LibriSpeech (CC-BY),
Multilingual LibriSpeech (CC-BY), VoxPopuli (CC0).
Voices in the demo are ElevenLabs.
Building on this, or feeding it to an AI? There is a
machine-readable summary at llms.txt with the
architecture, training details and measured numbers.
← back to the demo