# GNM2Lips

> A photoreal-topology human head that lip-syncs from audio entirely in the
> browser. A 2.37 M-parameter causal neural network turns microphone or file
> audio into 383 GNM expression coefficients at ~5 ms per 90 ms of audio, with
> no server-side inference.

Live demo: https://web-production-4c327.up.railway.app

## What this is

Three things stacked:

1. **GNM head evaluation in JavaScript.** Google's Generic Neutral Model (GNM,
   Apache-2.0) ships as a linear basis: 17 821 vertices, 253 identity
   components and 383 expression components. `web/gnmlive.js` evaluates
   identity -> expression -> visemes per frame in plain JS, so the mesh is
   generated live rather than played back.

2. **openlips-s** — a causal temporal convolutional network mapping log-mel
   audio features to those 383 expression coefficients. 2 369 663 parameters,
   6 residual blocks, dilations 1-32, 5.09 s receptive field, 30 ms lookahead.
   Exported to ONNX and run with ONNX Runtime Web (WASM).

3. **Face rig extras** that make it read as a face rather than a talking mask:
   additive emotion layer, blink, eye gaze with eyelid follow, sticky lips,
   strand-based beard and eyebrows, and GTAO screen-space ambient occlusion.

## How the model was trained

Knowledge distillation from NVIDIA Audio2Face-3D (regression variant), which
outputs GEOMETRY (61 520 vertices), not blendshape weights.

- **Corpus**: 10.79 h of CC0/CC-BY speech (Common Voice, LibriSpeech, MLS,
  VoxPopuli), labelled at 18.5x realtime on one RTX 2060 SUPER.
- **Target**: A2F geometry is transferred onto GNM's vertices by nearest
  neighbour after a uniform scale fit (median correspondence error 1.58% of
  head size), then projected onto GNM's own 383-component expression basis.
  The whole chain folds into one 140x383 matrix.
- **Why not ARKit**: the original pipeline went A2F geometry -> 52 ARKit ->
  14 visemes. Measured on 20 223 frames the second step has relative residual
  3.08 — worse than predicting the mean — and destroys mouthSmile (4.2),
  mouthRollUpper (16.1) and mouthUpperUp (13.8-22.4). The direct path scores
  0.42. This single change is what made the model usable.
- **Loss**: L1 + 2.0 velocity + 0.8 acceleration. No closure or lip-critical
  terms, because the target is basis coordinates, not blendshape weights.
- **Post-training**: a dispersion gain of 1.20 is folded into the output head.
  L1 against a one-to-many mapping converges on a conditional median, which
  under-moves; the model reached within 0.42 mm of lip contact on closures it
  missed. The gain restores articulation 76.7% -> 89.9% and closure rate
  12.1% -> 15.6% against the teacher's 15.2%.

## Measured results

Held-out, 12 of 80 speakers never seen in training, measured on rendered mouth
geometry in millimetres (not on label agreement):

| driver | vertex err | articulation | closure F1 | closure rate | lag |
|---|---|---|---|---|---|
| Audio2Face teacher | 0 mm | 100% | 1.000 | 15.2% | 0 ms |
| motionless face | 2.290 mm | 0% | 0.000 | 0% | — |
| DSP heuristic | 3.004 mm | 81.8% | 0.277 | 24.7% | 53 ms |
| **openlips-s** | **1.150 mm** | **89.9%** | **0.691** | **15.6%** | **10 ms** |

The hand-written DSP heuristic has HIGHER vertex error than a motionless face.
It moves convincingly but not correctly, which is why articulation and jitter
are reported alongside error rather than after it.

## Architecture notes that matter

- **Causality is enforced, not assumed.** `nn.GroupNorm` on [B,C,T] normalises
  over channels AND time, leaking the end of the utterance into every frame; it
  is replaced by a channels-only `ChannelNorm`. Lookahead is a fixed output
  shift, never a non-causal convolution, and streaming output is asserted equal
  to full-sequence output to 2e-6.
- **Feature parity.** The log-mel front end exists twice, in Python and JS, and
  is cross-checked to 5e-7 against a 1e-3 bar. Train/inference feature mismatch
  is the most common cause of a model that works offline and fails in a browser.
- **Playback, not just inference.** Web Audio delivers ~85 ms blocks, so each
  inference returns ~9 frames. Publishing only the last one animated the face
  at 11 Hz under a model fluent at 100 Hz. Frames go into a playout ring read
  back on the wall clock: 49 updates/sec.

## Payload

First load 52.7 MB, down from 125.6 MB:

- expression basis 41.0 -> 6.3 MB (lossless repack; 85% of the dense array was
  zeros, and the runtime already walked per-component ranges)
- baked teacher track 27.4 -> 6.5 MB (f16 binary instead of JSON text)
- model 9.4 -> 2.5 MB (int8; 0.037 mm from fp32 on mouth vertices, identical
  closure F1)

## Repository map

- `web/poc.html` — the demo: UI, drivers, rig, groom, lighting
- `web/gnmlive.js` — GNM basis evaluator (identity, expression, visemes)
- `web/hair.js` — strand hair bound by (triangle, barycentric) so it deforms
- `web/audio2face.js` — the DSP heuristic baseline
- `training/model/net.py` — the network
- `training/model/train_direct.py` — training against GNM coefficients
- `training/model/calibrate_gain.py` — dispersion gain calibration
- `training/data/teacher_direct.py` — A2F -> GNM transfer matrix
- `training/eval/metrics_direct.py` — geometry metrics in millimetres
- `training/runtime/openlips.js` — browser inference + playout ring
- `deploy/Caddyfile` — static host plus a TTS proxy that injects the API key
  server-side so it never reaches the browser

## Licensing

- Project code: Apache-2.0.
- GNM head model data: (c) Google, Apache-2.0. Redistribution of derived files
  is permitted under §2; §4 conditions (licence copy, modification notices,
  NOTICE) are NOT yet fully satisfied in this repository.
- NVIDIA Audio2Face-3D was used as a distillation teacher. Its weights are not
  included or redistributed. NVIDIA Open Model License §2.4 disclaims ownership
  of model outputs.
- Bundled audio clips are ElevenLabs stock-voice output; terms depend on the
  generating account's plan.
- `assets/skin_*.webp` have undocumented provenance and are deliberately NOT
  deployed.

## Known gaps

- openlips-s remains ~2x jitterier than the teacher (J/A 0.276 vs 0.118).
- Trained on read speech at 16 kHz; TTS and spontaneous speech are out of
  domain.
- Browser performance has not been measured on a low-end machine.
- The identity basis (26 MB) still loads eagerly despite being deferred in the
  loader.