liveavatar.github.io
A real-time streaming framework that generates interactive avatar video with a diffusion model.
Visit website (opens in a new tab)The Live Avatar project introduces a groundbreaking algorithm-system framework designed for real-time, streaming interactive avatar video generation. Powered by a robust 14-billion-parameter diffusion model, this technology delivers video output at an impressive 20 frames per second (FPS) using 5 H800 GPUs with just 4-step sampling. A standout feature of Live Avatar is its support for Block-wise Autoregressive processing, allowing for an astonishing streaming video generation capability of up to 10,000+ seconds in length.
The real-time and streaming capabilities of Live Avatar facilitate engaging interactions, enabling users to conduct natural face-to-face conversations via a microphone and camera. They receive immediate visual feedback as their avatars respond in real-time. An integration with Qwen3-Omni further enhances this, allowing for fully interactive dialogue between avatars.
Real-time streaming interactions necessitate a model that can generate frames faster than playback speed while supporting continuous streaming. Live Avatar achieves this through several advanced techniques:
Distribution Matching Distillation: This technique transforms the larger bidirectional multi-step video diffusion model into a more efficient 4-step streaming model.
Timestep-forcing Pipeline Parallelism (TPP): This novel paradigm decouples the sequential denoising stages across multiple devices, enabling linear speedup relative to the number of devices. This effective scaling facilitates high-speed generation, achieving an 84× FPS improvement over the baseline.
Traditional talking-avatar systems often experience degradation during long-duration autoregressive generation, leading to issues like identity drift and color variations. The project identifies three main internal phenomena contributing to these limitations:
Strategies Addressing Long-Horizon Failures:
Rolling RoPE: By dynamically updating the RoPE (Relative Positional Encoding) of the sink frame, this strategy helps stabilize long-term identity.
Adaptive Attention Sink (AAS): This approach replaces the original reference frame with a generated frame, disrupting persistent factors that contribute to distribution drift.
History Corrupt: By introducing noise into the KV-cache, this technique simulates inference errors, encouraging the model to reference both stable details from the sink and motion data from historical frames.
The integration of these techniques significantly boosts the Live Avatar's capability to deliver infinite-length streaming without compromising quality or identity integrity, achieving streaming durations exceeding 10,000 seconds.
Live Avatar can produce stunningly realistic portrait videos that demonstrate the capability of the underlying model. These videos can effectively capture live interactions, providing an immersive experience for users.
In addition to realistic representations, the technology also excels at generating cartoon-style videos. This feature can appeal to creative industries and entertainment, where animated avatars are often preferred.
The system has shown a remarkable ability to produce long-form video content seamlessly, further enhancing its versatility in applications. This includes use cases in gaming, virtual meetings, and interactive storytelling.
The performances and innovations presented by Live Avatar stand out notably against existing methods in the realm of avatar generation. This project addresses many limitations observed in previous technologies and advances the capabilities of avatar-based interactions significantly.
The Live Avatar project represents an important step in the field of interactive media and avatar generation. By leveraging advanced technologies such as real-time streaming and long-duration generation without quality loss, it opens up a realm of possibilities for various applications. These include interactive online experiences, gaming, video conferencing, and virtual interactions, making it valuable for individuals in industries ranging from entertainment to education and beyond.
This innovative framework not only enhances the user experience through lifelike interactions but also sets a benchmark for future developments in avatar technology, promising a new era of communication and engagement in a plethora of digital platforms.
Ready to try it? Visit liveavatar.github.io (opens in a new tab)
