跳到正文
arXiv · virtual production· Xiaoyu Zhan·精选AI 评分57

UMo:面向实时语音驱动数字人的统一稀疏运动建模架构

UMo: Unified Sparse Motion Modeling for Real-Time Co-Speech Avatars

AI 导读

Xiaoyu Zhan 等作者提出 UMo,一种面向实时语音驱动数字人的统一稀疏运动建模架构,在统一框架内处理文本、音频和运动 token。该方法结合空间稀疏的 Mixture-of-Experts 框架与以关键帧为中心的时间稀疏设计,实现实时稠密重建,可同时生成面部表情与手势动画,并采用多阶段训练与针对性音频增强提升声学多样性和语义一致性。

推荐理由

论文提出统一稀疏运动建模架构,在低延迟约束下兼顾面部表情与手势的实时生成质量。

正文 · 原文

Authors:Xiaoyu Zhan, Xinyu Fu, Chenghao Yang, Xiaohong Zhang, Dongjie Fu, Pengcheng Fang, Tengjiao Sun, Xiaohao Cai, Hansung Kim, Yuanqi Li, Jie Guo, Yanwen Guo

View PDF HTML (experimental)

Abstract:Speech-driven gestures and facial animations are fundamental to expressive digital avatars in games, virtual production, and interactive media. However, existing methods are either limited to a single modality for audio motion alignment, failing to fully utilize the potential of massive human motion data, or are constrained by the representation ability and throughput of multimodal models, which makes it difficult to achieve high-quality motion generation or real-time performance. We present UMo, a unified sparse motion modeling architecture for real-time co-speech avatars, which processes text, audio, and motion tokens within a unified formulation. Leveraging a spatially sparse Mixture-of-Experts framework and a temporally sparse, keyframe-centric design, UMo efficiently performs real-time dense reconstruction, enabling temporally coherent and high-fidelity animation generation for both facial expressions and gestures. Furthermore, we implement a multi-stage training strategy with targeted audio augmentation to enhance acoustic diversity and semantic consistency. Consequently, UMo preserves fine-grained speech-motion alignment even under strict latency constraints. Extensive quantitative and qualitative evaluations show that UMo achieves better output quality under low latency and real-time performance constraints, offering a practical solution for high-fidelity real-time co-speech avatars.
Subjects: Graphics (cs.GR); Computer Vision and Pattern Recognition (cs.CV); Sound (cs.SD)
Cite as: arXiv:2605.14731 [cs.GR]
  (or arXiv:2605.14731v1 [cs.GR] for this version)
  https://doi.org/10.48550/arXiv.2605.14731

arXiv-issued DOI via DataCite

Submission history

From: Xiaoyu Zhan [view email]
[v1] Thu, 14 May 2026 11:56:03 UTC (9,125 KB)

来源:arXiv · virtual production · arxiv.org