A talking avatar you can run on a laptop. Type or speak, Grok answers out loud, and the mouth follows the voice in the browser — no GPU lipsync model, no cloud viseme service.
The lipsync stack is intentionally small: Grok TTS already returns character timestamps, those characters map to a 15-shape viseme set, and the browser’s Web Audio analyser only supplies jaw energy. That is enough for 2D Rive mascots and 3D GLB heads.
You ──text or mic──► Grok chat
│
▼
Grok TTS (with_timestamps)
│
┌────────────┼────────────┐
▼ ▼ ▼
MP3 audio visemes amplitude
│ │ │
└────────────┼────────────┘
▼
LEWIN avatar
(Rive or Three.js)
Most talking-head demos ship a neural lipsync network, a forced aligner, or a paid viseme API. LEWIN does not need those on the hot path.
- Timestamps, not a model. Grok TTS
with_timestampsalready says when each character is spoken.server/visemes.mjsmaps English, Russian, and Chinese (via pinyin) onto the Oculus / wawa viseme set:PP FF TH DD kk CH SS nn RR aa E I O U sil. - One analyser node. wawa-lipsync is a few hundred lines of Web Audio FFT. It is used for amplitude (how open the jaw is), not for guessing phonemes.
- Hybrid player.
HybridPlayerclocks visemes againstaudio.currentTimeand reads volume onrequestAnimationFrame. No WASM lipsync runtime, no extra network hop. - Tiny mouth alphabet. 2D Rive files expose one input per viseme; 3D GLBs use morph targets of the same names. Switching avatar does not change the pipeline.
- Optional extras stay optional. Montreal Forced Aligner can refine Mandarin phones. Local MLX speech-to-text can drive the mic. Neither is required to chat and lipsync.
- Grok LLM replies, spoken with Grok TTS
- English, Russian, and Chinese viseme mapping, including spoken forms of digit runs
- 2D Rive mascots and 3D Ready Player Me / VRoid / AvatarSDK heads
- Voice input with barge-in (interrupt the avatar by talking)
- Background presets and a compact chat panel
You need Node 20+, Yarn 1, and an xAI API key.
git clone https://github.com/robofighter/lewin-ai-agent.git
cd lewin-ai-agent
cp .env.example .envPut your key in .env:
XAI_API_KEY=xai-...
Then:
yarn
yarn devOpen http://localhost:5173. That starts the API on 127.0.0.1:8000 and the Vite UI together.
| Command | What it does |
|---|---|
yarn dev |
API + UI |
yarn server |
API only |
yarn demo |
UI only (expects the API) |
yarn test |
viseme / STT / VAD unit tests |
Typing a message is enough. Live microphone needs the optional STT setup below.
Local STT uses mlx-audio and mlx-community/nemotron-3.5-asr-streaming-0.6b-8bit. Skip this on non-Apple machines — chat and lipsync still work.
python3 -m venv server/.venv-stt
server/.venv-stt/bin/pip install -r server/stt-requirements.txtUncomment in .env:
STT_PYTHON=server/.venv-stt/bin/python
Restart yarn dev, then press Live. ffmpeg on your PATH helps convert the mic clip to WAV.
If Montreal Forced Aligner is installed, Chinese replies can swap character visemes for phone-level timing. Leave MFA_ENABLED=0 unless you have MFA; the pinyin fallback is the default.
| Variable | Default | Purpose |
|---|---|---|
XAI_API_KEY |
— | xAI API key (required) |
GROK_MODEL |
grok-4.20-0309-non-reasoning |
Chat model |
GROK_VOICE |
eve |
TTS voice |
GROK_TTS_LANGUAGE |
auto |
TTS language (ru, en, zh, or auto) |
PORT |
8000 |
API port |
MFA_ENABLED |
0 |
Mandarin MFA aligner |
STT_PYTHON |
python3 |
Interpreter for the optional STT worker |
server/ Grok chat, TTS, visemes, optional STT
packages/wawa-lipsync Browser FFT lipsync (amplitude)
examples/lipsync-demo React UI, Rive + Three.js avatars
Maintained by @robofighter.
LEWIN builds on wawa-lipsync by Wassim Samad (MIT), via NoelClay/wawa-lipsync. The viseme alphabet is the Oculus LipSync set. Chat and speech come from xAI Grok.