Anchor-Guided Memory Bridging Evolving and Reference States for Real-Time Audio-Video Digital Human Generation
1Taobao & Tmall Group of Alibaba 2Nanjing University
Anchor-Guided Persistent Memory
An anchor-guided persistent-memory framework for few-step joint audio-video generation — stable appearance across prompt transitions, strong audio-visual synchronization, and real-time streaming inference.
Real-time long-form digital-human generation relies on causal models to extend audio-visual content while preserving subject appearance and audio-video synchronization across successive segments. A bounded cache retains local motion and phonetic context but discards older evidence, whereas attending to the complete generated history is computationally expensive and can propagate accumulated errors. We present TaoMate, an anchor-guided persistent-memory framework for few-step joint audio-video generation. The framework preserves an immutable visual anchor, compresses completed video and audio blocks into fixed-capacity dynamic states, and retrieves those states through modality-specific residual attention without extending the active cache. A reference-aware modulation method additionally conditions video features on dynamic and anchor appearance statistics. Anchor-preserving causal-context distillation varies rollout horizon, prefix provenance, and cache-history reliability while keeping the immutable visual anchor unperturbed. By separating persistent memory from stage-local denoising dependencies, TaoMate further admits stage-parallel execution across blocks, accelerating autoregressive inference without pipeline-specific retraining. We evaluate long-form video continuations with appearance, temporal, synchronization, facial, and speech diagnostics. Results show that TaoMate preserves stable appearance across prompt-conditioned segments and strong audio-visual synchronization under autoregressive generation.
Three complementary designs separate persistent memory from stage-local denoising.
An immutable visual anchor is coupled with adaptive audio-video states beyond the active KV cache. Modality-specific residual attention retrieves structural and acoustic history, while reference-aware FiLM reconciles dynamic appearance with the anchor.
Causal-context distillation with mixed rollout horizons and student-generated prefixes models accumulated causal errors, while perturbing only non-anchor history keeps the persistent visual reference intact for robust recursive generation.
Successive denoising stages are assigned to different devices, letting causally ready blocks advance concurrently after pipeline fill — retraining-free real-time long-form generation on a 22B-parameter backbone.
One uninterrupted thirty-minute documentary narration — stable appearance, scene, and voice from beginning to end.
Five- to six-minute continuous generations with stable appearance and audio-video synchronization.
Ten-second clips with stable appearance and lip-sync.
Comparison on the bilingual long-form benchmark against recent audio-video and avatar generators.
Best values in bold; our method is highlighted. Higher is better except Desync and ColorΔ.
| Method | Params | LipSync ↑ | Desync ↓ | Align. ↑ | Express. ↑ | Long Cons. ↑ | ColorΔ ↓ | DiT FPS ↑ |
|---|---|---|---|---|---|---|---|---|
| OmniAvatar | 1.3B | 4.794 | – | – | – | 0.6467 | 0.01566 | 0.948 |
| StableAvatar | 1.3B | 3.246 | – | – | – | 0.9242 | 0.00895 | 0.352 |
| Wan2.2-S2V-14B | 14B | 4.898 | – | – | – | 0.8413 | 0.00814 | 0.079 |
| LiveAvatar | 14B | 5.396 | – | – | – | 0.9619 | 0.00541 | 4.529 |
| Hallo3 | 5B | 4.855 | – | – | – | 0.7947 | 0.01383 | 0.150 |
| LTX-2.3 (chunked) | 22B | 2.349 | 0.050 | 4.35 | 3.60 | 0.3745 | 0.04387 | 0.262 |
| JavisDiT | 2.1B | 0.904 | 1.025 | 5.00 | 3.90 | 0.3545 | 0.07159 | 0.437 |
| MOVA | 32B/18B | 3.860 | 0.060 | 5.00 | 3.75 | 0.7129 | 0.02825 | 0.128 |
| OVI | 11B | 3.493 | 0.135 | 4.60 | 3.70 | 0.8252 | 0.01797 | 0.231 |
| TaoMate (Ours) | 22.1B | 5.773 | 0.000 | 5.00 | 4.00 | 0.9520 | 0.00483 | 16.32 |
Params denotes denoising-generator size, with MOVA shown as total/active parameters. DiT FPS excludes data loading, VAE decoding, and post-processing. Dashes denote unavailable measurements.
@misc{taomate2026,
title={TaoMate: Anchor-Guided Memory Bridging Evolving and Reference States for Real-Time Audio-Video Digital Human Generation},
author={Qijun Gan and Chenwei Zhang and Meiguang Jin and Junfeng Ma and Qiu Shen},
year={2026},
eprint={2607.24359},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2607.24359},
}