Runway 发布 57B 视频扩散 transformer 实时推理研究，四张 H100 跑通 24 fps 对话式视频

Every conversation has a rhythm. One person stops talking, the other starts and that back-and-forth pace sets the tone. We interrupt each other, talk over each other and bounce from topic to topic, staying on track the whole time. To be convincing — to cross the uncanny valley — a virtual character has to keep that same rhythm.

Runway Characters can turn a single image into a live character you can talk to, and behind it is a roughly 57B-parameter autoregressive video diffusion transformer. At 24 frames per second (fps), that model has to generate a new four-frame chunk of 704x1088 video every 166.7ms, for one user at a time, for as long as the conversation lasts.

In May, we first shared how we built Characters. In our new published research, Runway Characters: Serving a 57B Video Diffusion Transformer for Real-Time Conversation, we cover inference and the system that makes the rhythm behind this possible in production. We’ll give a high-level look at each below. The paper covers all of them in depth.

Key findings:

• Characters launched on eight H100s per session. On four, the launch software took 240.9ms per diffusion step, well over the 166.7ms we require. Our optimizations cut it to 159.3ms (1.51x faster, 24.5 fps sustained), so each session now runs in real time on four H100s. A fused communication kernel takes it to 147.4ms (1.63x).
• In the original step, 29% of GPU time went to reprocessing a mostly unchanged cache in each of 240 attention calls, while attention itself took 6%. Doing that work once, as each frame is written, cut cache upkeep from 69.7ms to 6.2ms per step.
• Fusing the tensor-parallel all-reduce into the FP8 matrix multiplication hides 54-68% of its time. At our production shape the fused call takes 153.8µs against 208.2µs, and the advantage grows with more GPUs.
• Characters runs on an autoregressive model, which makes pixel comparisons relatively meaningless. We measured each change against the variation between repeated runs. Across 51 scored clips, mean lip-sync differences were at most 0.03, the same order as run-to-run variation, and four 10-minute sessions showed the same.
• Over seven weeks of telemetry, the worker averaged above 23.5 fps in 99.5% of sessions that streamed for at least a second. After a rollback and fix, the fused kernel has run with 0.08% of sessions at 23.5 fps or less.

A Clock That Never Stops

Traditionally generative video works on the model’s schedule. You write a prompt, wait and get a finished clip. But a character has to listen, respond and react as fast as the person on the other side, with no natural stopping point and no chance to catch up later.

That makes speed a hard requirement: a video character in conversation can never fall behind. At 24 fps, Runway Characters has to generate a new four-frame chunk of video every 166.7ms for as long as the conversation lasts. The player holds only a few chunks in reserve, and because each frame carries its slice of the audio, a late chunk freezes the face and stalls the voice at the same time. The video model also shares the loop with a voice agent, so any time generation adds is time the user spends waiting for a reply.

Most serving systems get their efficiency from batching, sharing GPUs across many users’ requests. Characters can’t do that. Each session is a private, one-on-one stream, and at 24 fps it needs 2.24 PFLOP/s of useful work, more than a single H100’s dense FP8 peak. The matrix multiplications already ran near two-thirds of that peak, so adding a second session to the batch would nearly double the step. Every millisecond we saved had to come from inside the step itself.

Where the Time Went

Each step denoises one latent frame in four passes through a 60-block transformer, which adds up to 240 attention calls. Each call reads its own cache of 5,788 slots, and only 757 of them are new. In the original implementation, every call redid work over the whole cache: re-applying rotary position embeddings (RoPE) to every cached key, re-quantizing the cache to FP8 and shifting the window by copying it. That upkeep took 69.7ms of a 239.2ms step.

Do It Once, at Write Time

To fix this, we apply RoPE and FP8 quantization to each new frame as it enters the cache, using per-head scales calibrated once per session. We store the cache as a ring, so the window advances without copying. Cache upkeep fell to 6.2ms per step

Positions in our model reset every 42 steps, so cached keys can’t keep their rotation forever. At each reset we apply a single delta rotation to the cached window instead of re-rotating everything on every read. It shows up as one slower step in 42, at 263ms, which the playout buffer absorbs.

The ring only works if RoPE is applied at write. Each new frame overwrites the oldest slot, so once the ring wraps, slot order no longer matches time order. The original code assigned positions by slot order, so combined with the ring it would have silently given cached frames the wrong positions. A test now pins that combination as invalid.

Cumulative waterfall of diffusion time per step. Each optimization and its effect, in milliseconds; lower is better.

Fold Communication Into Compute

With the cache work gone, the next largest cost after the matrix multiplications was the tensor-parallel all-reduce that follows every block, 240 times per step. Swapping in a faster collective saved just 0.4ms, because the reduction still couldn’t start until the whole matrix multiplication finished.

So we fused the two. We extended the fused GEMM and all-reduce kernel from ParallelKittens to FP8 and to odd GPU counts. Each output tile is reduced in the NVLink switch as soon as its partial results land, while later tiles are still being computed. At our production shape the fused call takes 153.8µs against 208.2µs, hiding 54-68% of the all-reduce.

Without the fused kernel, our optimizations took the diffusion step to 159.3ms, sustaining 24.5 fps on four H100s. With it, the step reaches 147.4ms.

Give the Decoder its Own GPU

The decoder that turns latents into video frames takes 114ms per step. On a diffusion GPU it would add that time to every step, since the tensor-parallel group waits for its slowest member. On its own GPU it decodes the previous chunk while the next is denoised, and the step period exceeds diffusion time by only 1.6ms.

We tested the alternative on B200s. A GPU shared between decoding and diffusion sat at its 1,000 W power cap, which slowed its diffusion kernels by 18-26%.

One session on four H100s, with decoding of step i−1 and playback of step i−2 overlapping the denoising of step i.

Judging Quality When Pixels Can’t

Three of these optimizations change the model’s numerics: the FP8 cache, the fused pre-attention kernels and the fused all-reduce. The usual check is to compare output pixels against a reference. In an autoregressive model that comparison stops meaning anything within seconds, because any numeric difference, even GPU nondeterminism, sends the video down a different trajectory. After about 10s, two runs of the same production configuration differ in PSNR as much as any two configurations do.

So we judged each change on the task instead. We generated our 53-clip evaluation set, built from 50 synthetic reference images across photoreal, cartoon, claymation, fantasy and non-human styles, with every configuration. We also ran production four times to measure how much lip-sync scores move between identical runs.

Across the 51 clips SyncNet scores in every run, mean differences were at most 0.03 and every 95% interval fell within ±0.08. The four production runs alone spanned 0.024 in LSE-D and 0.018 in LSE-C. Four 10-minute sessions alternating speaking and listening showed the same pattern, with no difference growing over the session beyond what a rerun produces.

These results bound the changes rather than prove equivalence. SyncNet is weaker on stylized characters, with intervals two to five times as wide, and four long sessions can’t rule out a slow trend.

What Broke in Production

The fused kernel went into production in August 2026 and ran cleanly for weeks. Then, after a PyTorch and NCCL upgrade, workers started running out of memory due to a lack of input variety in our benchmarks.

A session’s first forward pass has a length that depends on its prompt, and the kernel kept a buffer for every token count it saw, so workers bound new multicast buffers for each new prompt length. Benchmarks with fixed prompts never triggered it. Every optimization in the system sits behind a switch that keeps the plain code path, so rolling back to NCCL took a configuration change.

We now fuse only the steady-state shape and let first forward passes use NCCL. After a 15-hour canary in which every measured session averaged 24.0 fps or more, we re-enabled the kernel on October 8, 2026. In the 15 days that followed, 0.7% of sessions of at least a second averaged 24 fps or less, against 7-23% in each of the five weeks before.

Two other bugs passed every eager-mode test and failed only under compilation. In one, the compiler legally moved one call’s read past the next call’s write to a shared buffer, leaving 96% of elements wrong. Every kernel test now chains calls under torch.compile with CUDA graphs.

Our worst bug was a single .item() call per step, added to monitor FP8 scales. It took a step from 168.5ms to 278ms by stopping the CPU from running ahead of the GPU, adding more time than all of our cache work put together had saved.

Where a Conversational Turn Goes

Conversation has its own rhythm. In a study across 10 languages, the gap between one speaker finishing and the next starting averaged 208ms. A commercial voice agent on its own averages 1.30s on the same measure.

We traced 51 turns across 12 live sessions. The median turn, from the end of the user’s speech to the character’s voice at the client, took 2.13s. The voice agent accounted for 1.42s of it. Our pipeline, from the agent’s first audio to the first speaking frame, took 0.49s, and delivery to the browser added about 190ms.

Per-stage median turn latency. Where the time goes from the end of user speech to avatar audio at the client.

Blackwell and Beyond Four GPUs

Production runs in real time on four H100s at 159.3ms per step, and on three B200s at 154.3ms. But not every H100 optimization carries over. The FP8 cache saves 17.0ms on H100 but costs 2.4 to 6.7ms on B200, where our build has no FP8 attention kernel that takes per-head scales. The fused kernel matters most where decoding is off the critical path, adding up to 1.20x end to end at higher GPU counts.

More GPUs speed up the model, but not the decoder. On eight H100s, diffusion drops to 118.2ms per step, but the single GPU that turns results into video frames can’t keep pace, so each step still takes 128.7ms (31 fps).

Build Real-Time With Us

We believe real-time video is where generative media is headed, and that serving it is a research problem in its own right. The full paper covers the kernel design, quality evaluation and production measurements. If you want to work on low-latency systems and GPU kernels, we’re hiring.