Nuwa · v0.2 · in development
Speech in, speech out, on a machine you own
Nuwa wires a language model, a speech recogniser and a voice-cloning synthesiser into a single spoken loop. You talk; about a second after you stop, it starts reading the answer back in whichever voice you built.
Every stage runs on the local GPU by default. A hosted model can be attached per agent when you actually want one — but nothing forces you to send your audio anywhere.
The web client. Chinese interface today; English is being filled in.
The loop
What happens when you say something
Most of the work in a voice assistant is not the models. It is the seam between them — knowing when the user has finished talking, deciding when to start speaking, and getting out of the way when they interrupt. That seam is where most of this project's effort went.
-
You stop talking and it notices
An energy-based detector watches the microphone. Roughly 1.1 seconds of silence ends the recording on its own. No push-to-talk, though the button still works if you prefer it.
-
Transcription runs locally, then sends itself
Audio goes to the recogniser on the same machine. The result is not dropped into a text box for you to confirm — it is submitted immediately, because confirming every turn defeats the point.
-
The reply is spoken before it is finished
Tokens stream back over SSE. Completed sentences are split off and sent to synthesis as they arrive, so playback starts while the model is still writing the rest.
-
You can talk over it
The microphone keeps listening during playback. Start speaking and it stops mid sentence, clears the queued audio and reopens the mic for the next turn.
What runs it
The parts, and what they actually are
Engines are swappable because each one runs out of process behind a stable interface. Changing a model or a GPU vendor does not change the API the clients talk to.
Served by Ollama, currently gemma4:e4b. Responses stream over
server-sent events. An OpenAI-compatible endpoint can be pointed at instead, per
agent.
Paraformer-Large by default, which handles Mandarin better than the alternatives on this hardware. Whisper, GLM-ASR and Qwen3-ASR are selectable at runtime.
GLM-TTS clones a voice from a short reference clip and supports per-segment emotion across long passages. The weights are 8.5 GB, which is the main reason INT8 quantisation is on the list. CosyVoice3, Qwen3-TTS and OpenVoice also work.
One Rust service on Axum 0.8. It owns the scheduler: named pipelines such as
voice_reply and text_chat_stream, a semaphore bounding
concurrent GPU work, a cancel endpoint, and partial results preserved when a single
step in a pipeline fails.
React 19 and Vite, with session history in IndexedDB so a reload does not lose the conversation. Playwright covers the end-to-end paths.
CUDA on an RTX 5070 is the path that gets exercised daily. ROCm on an RX 9070 XT
runs but still has HIP compatibility gaps. NUWA_GPU_BACKEND forces a
choice, including CPU.
A sidecar bridges a personal messaging account to the same agents, relaying text, voice notes, images and video, so the assistant is reachable when you are away from the desk.
Status
Where v0.2 actually stands
Written down honestly, because a status table that says everything is finished is not worth reading.
About
Why it runs locally
An assistant that hears you, understands you and answers in a chosen voice needs all three of those streams. Sending them to someone else's servers is the normal arrangement, and it is a lot to hand over for a convenience. The question this project is trying to answer is whether the same experience holds up on one workstation with a consumer GPU.
So far it mostly does. Latency is the honest weak point, and the roadmap reflects that: chunked audio streaming and a quantised synthesiser are the next two things, both aimed at the gap between you finishing a sentence and hearing a reply.
Not everyone has a spare GPU, so a hosted tier is the other thing being built: the same loop, with synthesis and recognition run on rented GPUs instead of yours, and a managed model standing in where self-hosting one is not worth the trouble. Local stays the default. Hosted is for people who want to hear it work before buying a card.
Nuwa is built by Awesomelog, started in 2026. The source is private for now while some of the training material is cleaned up. If you want to see it, ask.