TaoMate-HD
Open research
ALIBABA TAOLIVE AIGC TEAM
Progressive-Resolution Autoregressive
Audio-Video Generation
High-resolution video and sound, generated together. Coarse-to-fine chunks carry audiovisual context forward as the dialogue evolves.
Progressive denoising
One generative model
Long-form demonstrations
Code & preview weights
Our paper is on arXiv, with the first release of the inference code and preview weights.
01 / GENERATED WORLDS
Conversations, performances, and imagined worlds.
A selection of audio-video generation across scenes and styles.
Muted autoplay · Hover for sound · Expand for fullscreen The cityscape is presented without audio.
02 / BEYOND THE SHORT CLIP
Four scenes. One hour each.
Jump through a full-length generated sequence.
03 / THE METHOD
Each chunk follows one joint audiovisual trajectory: two steps establish coarse video structure at 480p, then a final step refines detail at 768p. Audio is generated throughout.
Early steps work on a smaller video grid, reserving high-resolution computation for final refinement. A coupled-noise lift preserves the signal and noise as video moves to the finer grid. Audio keeps the same latent geometry throughout.
Each stage estimates a clean audiovisual endpoint before the remaining stages finish. It supplies native-grid context to the same stage of the next chunk, so coarse generation can overlap with earlier chunks’ high-resolution refinement. An initial visual anchor and two recent audiovisual chunks carry context forward.
The model learns from its own progressive rollouts. The generator advances through overlapping dialogue windows, the learned fake score keeps the initial extended window, and the frozen teacher score uses the original source-transcript prefix. Both scores supervise the generated sequence on a shared high-resolution canvas.
When the text advances from A + B to B + C, the shared span B connects the old and new dialogue. Before switching text, each stage commits its pending clean prediction under the window that generated it, then carries its history and temporal positions forward. Retained audio tells the model how far speech has progressed, without an online speech recognizer or an explicit text pointer. Speech-completion training teaches it to finish the remaining words and transition to natural nonspeech behavior.
04 / RESEARCH RESULTS
Coarse-to-fine generation retains fine visual detail with less high-resolution work. Continuation training improves how faithfully speech follows the dialogue.
Over direct 768p generation with matched BF16/SDPA execution, including audio-video decoding.
Stage overlap, LOW-stage fusion, KV reuse, selective W8A8, attention kernels, and codec scheduling vs. serial Progressive on eight H20 GPUs.
Continuation training improves transcript accuracy: 30.75% → 19.57% CER.
| Method | Temporal ↑ | Long-range ↑ | Visual realism ↑ | Audio aesthetics ↑ | CER (%) ↓ |
|---|---|---|---|---|---|
| OmniForcing | 0.786 | 0.278 | 4.85 | 3.571 | — |
| Lynn | 0.367 | 0.778 | 4.30 | 4.207 | 86.70 |
| Lynn-Flash | 0.457 | 0.765 | 5.00 | 4.326 | 88.37 |
| TaoMate | 0.843 | 0.983 | 4.65 | 3.889 | 103.74 |
| TaoMate-HD Serial | 0.913 | 0.930 | 5.00 | 4.407 | 19.57 |
| TaoMate-HD Accelerated | 0.913 | 0.928 | 5.00 | 4.367 | 21.28 |
Quality results from Table 1 on 100 sequences lasting 1–10 minutes. Temporal measures frame-difference stability; Long-range measures color/brightness continuity. CER measures transcript accuracy.
05 / BUILT TO BE SHARED
Explore the demos, run inference, and build with TaoMate-HD.
Inference code and preview weights are available below.
@misc{gan2026taomatehdprogressiveresolutionautoregressiveaudiovideo,
title={TaoMate-HD: Progressive-Resolution Autoregressive Audio-Video Generation},
author={Qijun Gan and Chenwei Zhang and Xinyang Li and Meiguang Jin and Junfeng Ma and Shen Qiu},
year={2026},
eprint={2611.03394},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2611.03394}
}