Limits & concurrency

What is capped, what is not, and how to size your traffic.

There is no rate limit today

No per-key request or concurrency limit. Requests are not rejected for pacing, and no x-ratelimit-* headers are returned.

Measured: 40 simultaneous WebSocket callers all succeeded, median first audio 554 ms, slowest 954 ms, no errors. And 1000 HTTP calls at 25 concurrent: 100% success, TTFA p50 120 ms, p90 232 ms.

That is not a licence to fire without limit. Latency rises with concurrency — about 220 ms median at 4 in flight, 550 ms at 40 — because the GPU batches more work. Past a point you are trading everyone’s latency for your own throughput.

Size your in-flight count to the latency you need, not to a limit that will stop you.

Limits may be introduced for abusive traffic. If they are, they will arrive as 429 with retry-after, and this page will say so.

Close abandoned requests properly

If you stop reading part-way — the user interrupts, a timeout, a crashed process — close the connection cleanly rather than dropping it.

Node

const ac = new AbortController();
const res = await fetch(URL, { signal: ac.signal, ... });
// user interrupted
ac.abort();

A half-abandoned request holds a GPU slot until it times out — so it costs everyone, not just you.

Text length

There is no practical limit. Send a full paragraph in one request and you get one continuous audio stream.

Do not split text into sentences yourself for the HTTP endpoint. One request per paragraph streams better than several stitched together.

On the WebSocket it is the opposite — send each sentence as its own text frame with the same context_id, as fast as your LLM produces them.