好文集ChatGPT Plus¥119

A LoRA is all you need to make a speech LLM full-duplex

文章目录

TL;DR: you don't need to train a full-duplex voice model from scratch. Take a speech LLM that already exists, freeze it, train about 0.3% extra parameters for 30 hours on one GPU, and it learns to listen while it talks. Code and weights are open.

A few months ago I shared ConversationalVoice, our pipeline for full-duplex training data. This is the model side of that story.

Full-duplex voice is having a moment. Block, Nuance and Fish Audio all launched models in the last few weeks, and they sound great. The usual way to get there is to train on a huge amount of conversational speech. We're a small team with a fixed budget, so that path was closed and we had to find a cheaper one. I think the cheaper one turned out to be more interesting.

1. An LLM already knows what to say

Think about what a half-duplex speech LLM can already do. It understands speech, it reasons, and it knows what to say. The one thing it can't do is talk and listen at the same time, so it waits for you to finish.

That makes full duplex a small missing skill. Everything about language is already there, and what's left to learn is timing: when to speak, when to stay quiet, and when to say "mm-hm".

文章配图

What timing looks like: the agent says "mm-hm" while the user is still talking, and stops when the user cuts in.

2. So we made timing part of the vocabulary

We took Qwen3-Omni, a 30B open model, and froze it. Then we gave it 17 new tokens for things like "pause" and "yield", and trained a LoRA plus a few small layers so it could use them. That's 97M parameters, about 0.3% of the model, and it took about 30 hours on a single B200.

文章配图

Every 80 ms the model emits one token. It can be a word, a pause, or a signal to take the floor or give it up. So deciding when to talk becomes next-token prediction, which is the thing LLMs are best at. There's no separate turn detector anywhere, and a talker turns the tokens into a voice.

文章配图

Grey is frozen, orange is trained.

3. It works, and it stays smart

On Full-Duplex-Bench, it's the open model least likely to cut in when you pause to think (0.17 vs 0.58 for the next best).

文章配图

And it keeps its intelligence. We ran the same idea on a second, smaller model (Fun-Audio-Chat, 8B, this time with full fine-tuning), where we could compare directly against the half-duplex original. It scores 61% vs 66% on groundedness in long conversations, and 90% vs 100% on spoken logic puzzles. The Qwen version leads the open full-duplex models at 53%, where the next best is 35%.

文章配图

4. Why it matters

If turning a smart LLM into a full-duplex one is this cheap, then full duplex can be a thin layer on top of whatever the smartest model is, and every new LLM can get it almost for free.

And once a model listens while it talks, it also thinks while you talk. A text model starts thinking after you hit send. A full-duplex model gets the whole time you're speaking. I think that's the part people haven't priced in yet.

It's not perfect yet. It still sounds unnatural sometimes, mostly because of our training data, and there are recipe ideas we didn't get to try due to limited time and resources.

We're sharing everything because a small team can go surprisingly deep on a problem, and I'd love to compare notes with anyone working on voice or interaction models.

Demos and write-up: Qwen3-Omni version and Fun-Audio-Chat version

Weights on Hugging Face: Qwen3-Omni version and Fun-Audio-Chat version

Code: Qwen3-Omni version and Fun-Audio-Chat version

全文完
查看原文