> ## Documentation Index
> Fetch the complete documentation index at: https://docs.apiyi.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Realtime Voice (WebSocket)
> Four realtime voice models, one wss endpoint, one APIYI key: bidirectional streaming audio, barge-in, two turn-detection modes, function calling and image input. Private beta — includes a full field-by-field comparison of the two protocols and a zero-cost text-only self-test path.
## Overview
Realtime models run over a **long-lived WebSocket connection**: audio streams in, audio streams out, and the model can be interrupted mid-sentence — no "record, upload, wait, play back" cycle. The difference from stitching ASR + a text model + TTS is that this is end-to-end: the model hears tone, pauses and emotion directly, and speaks directly. Latency lands in the sub-second range.
APIYI currently offers **4 models across 2 protocols**, sharing one endpoint and one key:
* `gpt-realtime-2.1` / `gpt-realtime-2.1-mini` — OpenAI Realtime GA protocol
* `qwen3.5-omni-plus-realtime` / `qwen3.5-omni-flash-realtime` — Alibaba Cloud Model Studio protocol
**Status: private beta / integration in progress.** Realtime voice is in limited supply and **not yet open for self-service** — you need to contact us to be enabled. Upstream protocol and behavior may still change during the beta; everything in the "Known Limitations" section below is measured, and will be updated as upstream changes. Please do not ship to production without a fallback path. If you plan to integrate, or need higher concurrency, reach us via [WeCom support](https://work.weixin.qq.com/kfid/kfc9adfd5810ece25ec) or email [hi@apiyi.com](mailto:hi@apiyi.com) / [feedback@apiyi.com](mailto:feedback@apiyi.com).
**🎤 Highlights**: bidirectional streaming audio over one connection, **barge-in at any time**, `server_vad` and `semantic_vad` turn detection, full function-calling round trip (including result injection), image input, and `usage` broken out per modality. All of the above verified on all four models (2026-08-24, UTC+8).
**One thing to remember first**: the 4 models use **two different request protocols**, with different field names and different event names. **Changing only the `model` parameter without changing the request body will not work** — this is by far the most common integration failure. The differences amount to 6 fields and 3 event names, all listed under "Protocol Comparison" below.
Contact WeCom support with your account and expected concurrency, and we will enable the beta group on your key.
Key creation, base URL, billing modes and other general conventions.
Create keys, select groups and set quotas.
Inspect token usage and actual charges per call in the console.
This page is long. Three sections are required reading: **Protocol Comparison** (read before switching models), **Start With Text** (verify the whole chain without a microphone), and **Known Limitations** (four measured differences that affect client code).
## Let an AI Agent Do the Integration
If you develop with Codex / Claude Code / Cursor, copy the prompt below to it. It will first fetch the plain-text version of this page (append `.md` to any docs URL), then write code for your stack — the **two field families**, the sample-rate red line, cancel semantics and idle disconnects are all baked into the requirements.
Help me integrate / troubleshoot APIYI Realtime voice (bidirectional streaming over WebSocket) in this project.
Read the docs before writing code: fetch [https://docs.apiyi.com/en/api-capabilities/realtime/overview.md](https://docs.apiyi.com/en/api-capabilities/realtime/overview.md) for the plain-text version of this page, focusing on the "Protocol Comparison" and "Known Limitations" sections.
Requirements:
1. This is a **WebSocket long-lived connection**, not an HTTP request. The endpoint is `wss://api.apiyi.com/v1/realtime?model=` and auth is the `Authorization: Bearer ` header. **Do not write it as an HTTP POST, and do not try to build `/v1/audio/speech` or `/v1/audio/transcriptions` URLs** — those are a different API.
2. **Before writing any field, determine which protocol family the model belongs to.** `gpt-realtime-2.1` and `gpt-realtime-2.1-mini` use the Realtime GA protocol; `qwen3.5-omni-plus-realtime` and `qwen3.5-omni-flash-realtime` use the Alibaba Cloud Model Studio protocol. Only the endpoint and auth are shared — request bodies and event names differ throughout: output modality `modalities` vs `output_modalities`; voice at top level `voice` vs `audio.output.voice`; `input_audio_format` vs `audio.input.format`; top-level `turn_detection` vs `audio.input.turn_detection`; `input_audio_transcription` vs `audio.input.transcription`. Event names: `response.text.delta` vs `response.output_text.delta`, `response.audio.delta` vs `response.output_audio.delta`. **Write these as two config templates, do not scatter if-branches through the code.**
3. Audio format red line: always PCM signed 16-bit, mono, Base64-encoded into `input_audio_buffer.append`. **The sample rate differs between the two** — Model Studio uses 16000, Realtime GA requires at least 24000 and rejects 16000 with `integer_below_min_value`. Resample on the client; do not expect the server to fix it.
4. Do not block on `response.done` to close out a barge-in. After sending `response.cancel`, **the two Model Studio models currently do not deliver `response.done`** (reproduced consistently in testing). Use `response.output_item.done` as the end-of-turn signal and add a 5-second timeout as a backstop; the two Realtime GA models behave correctly, but the same logic works for both.
5. Long connections need keepalive and reconnect. **Model Studio drops idle connections after 300 seconds, and WebSocket-level ping/pong does not count as activity** — it will not extend that timer. Either send an application-level event periodically while idle, or accept the disconnect and reconnect automatically. Realtime GA sessions carry `expires_at` (measured at roughly 30 minutes from connect) and also need reconnection. **After reconnecting you must replay `session.update` and any required context**, otherwise the new session runs with defaults.
6. Pin the voice in the **first `session.update`**. Once a session has produced audio output, changing the voice fails with `cannot_update_voice`. Open a new session to switch voices.
7. Read the key from the `APIYI_API_KEY` environment variable; never hardcode it and never commit it. **Do not connect from the frontend** — write a backend relay that holds the key and forwards audio frames.
8. Run a text-only smoke test before touching the microphone: set `output_modalities` to text only, send one `input_text`, and confirm you receive text deltas and the `usage` object inside `response.done`. Then move to audio. When you are done, actually run one call against each protocol family and paste both `usage` objects back to me.
| Requirement | Pitfall it avoids |
| ------------------------------------------- | ----------------------------------------------------------------------------------- |
| Identify the protocol family first | Changing only `model` lets the handshake pass but `session.update` is then rejected |
| Sample rate pinned per family | Sending 16 kHz to Realtime GA fails with `integer_below_min_value` |
| Output-modality field was renamed | Writing `modalities` on the GA protocol is simply an unknown field |
| Event names changed too | Listening for `response.text.delta` on the GA protocol never fires |
| Close out on `output_item.done` | Model Studio omits `response.done` after cancel; waiting for it hangs the turn |
| Keepalive needed, ping does not count | Assuming a heartbeat prevents the 300-second idle disconnect |
| Voice pinned in the first frame | Changing it after audio has been produced fails with `cannot_update_voice` |
| Backend relay, no direct browser connection | Connecting from the browser hands the key to every visitor |
## Why APIYI for Realtime Voice
Same `wss` endpoint, same auth. Switching models means changing the `model` parameter and the matching field template — no second vendor account to maintain.
Reach `api.apiyi.com` from mainland data centers, home broadband or overseas nodes. No upstream vendor account, identity verification or prepayment required.
Field comparison, event-name comparison, sample-rate limits and four measured limitations are all documented here so you do not have to rediscover them.
Verify handshake, auth, fields, tool wiring and concurrency without a microphone — audio tiers cost an order of magnitude more than text, so this is real money saved during integration.
Handshake p50 0.65–1.08 s and first text delta p50 0.54–0.95 s at 20 concurrent sessions, with the test conditions and date stated under Technical Specs.
Beta users get a direct WeCom channel for integration questions, concurrency increases and upstream behavior changes.
## Core Capabilities
Audio streams out as it is generated; the client can send `response.cancel` at any time. The session survives and context is preserved. Verified on all four models.
`server_vad` splits on silence duration, `semantic_vad` splits on intent (better at ignoring filler words like "uh-huh"). Both verified on all four models.
The model triggers a tool, the client executes it, `function_call_output` injects the result, and the model continues speaking. Verified end to end on all four models.
Send images mid-session for the model to read; `usage` returns text / audio / image tokens separately so cost can be attributed. Verified on all four models.
## Supported Models
| Model | Protocol family | Default voice | Input sample rate | Prompt caching | Positioning |
| ----------------------------- | --------------- | ------------- | ----------------- | -------------- | ---------------------------------------------- |
| `gpt-realtime-2.1` | Realtime GA | `marin` | ≥ 24 kHz | ✅ Supported | Flagship; strongest multilingual and reasoning |
| `gpt-realtime-2.1-mini` | Realtime GA | `marin` | ≥ 24 kHz | ✅ Supported | Cost-effective; ample for everyday dialogue |
| `qwen3.5-omni-plus-realtime` | Model Studio | `Tina` | 16 kHz | ⏸ Not observed | Flagship for Chinese-language scenarios |
| `qwen3.5-omni-flash-realtime` | Model Studio | `Tina` | 16 kHz | ⏸ Not observed | Cost-effective for Chinese-language scenarios |
Output audio is **PCM signed 16-bit / mono / 24 kHz** on all four models.
The two protocol families **share only the endpoint and the auth scheme**. Request fields and server event names both differ. When you switch models you must switch field templates too — see "Protocol Comparison" below.
## Pricing
**Pricing in one sentence**: billed per token, and **audio costs an order of magnitude more than text** (for `gpt-realtime-2.1`, audio input \$32 vs text input \$4; audio output \$64 vs text output \$24). Run text-only during integration and switch to audio once the chain is verified — see "Start With Text" below.
The tables below are the **vendors' official list prices**, in USD per 1M tokens. **Actual charges on APIYI are whatever the [call logs](https://api.apiyi.com/log) show**; the [recharge bonus](/en/faq/recharge-promotions) lowers the effective cost further.
### Realtime GA protocol
| Model | Text input | Text output | Cached read | Image input | Audio input | Audio output |
| ----------------------- | ---------- | ----------- | ----------- | ----------- | ----------- | ------------ |
| `gpt-realtime-2.1` | \$4 | \$24 | \$0.4 | \$5 | \$32 | \$64 |
| `gpt-realtime-2.1-mini` | \$0.6 | \$2.4 | \$0.06 | \$0.8 | \$10 | \$30 |
### Model Studio protocol
The billing dimensions differ: image input is folded into the text tier, and output is split into "text only" and "text + audio" (only the audio portion is charged at the latter rate).
| Model | Text / image input | Audio input | Text output | Text + audio output |
| ----------------------------- | ------------------ | ----------- | ----------- | ------------------- |
| `qwen3.5-omni-plus-realtime` | \$1.38 | \$11 | \$8.25 | \$41.26 |
| `qwen3.5-omni-flash-realtime` | \$0.45 | \$3.71 | \$2.75 | \$14.71 |
**Beta note**: Realtime voice is in limited supply and billing is still being aligned with upstream. If your actual charges deviate noticeably from the tables above, please contact support so we can reconcile. Pricing may change with vendor policy and supply. This capability is offered to **secure supply and serve customers**, not as a profit-driven listing.
## Access Group
**How to get enabled during the beta**: self-service group selection is not available yet; access is granted on request. Contact [WeCom support](https://work.weixin.qq.com/kfid/kfc9adfd5810ece25ec) with your account, use case and expected concurrency, and we will enable the beta group on your key and share the current caveats. General availability will be announced in the [changelog](/en/changelog); no key or code changes will be needed then.
## Technical Specs
| Dimension | Value |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| **Transport** | WebSocket (`wss://`), bidirectional streaming over one connection |
| **Auth** | `Authorization: Bearer ` header |
| **Event format** | Compatible with the OpenAI Realtime event model (client events / server events) |
| **Input audio** | PCM signed 16-bit, mono, Base64. **Model Studio 16 kHz; Realtime GA ≥ 24 kHz** |
| **Output audio** | PCM signed 16-bit, mono, 24 kHz |
| **Output modalities** | Text / audio (text-only is allowed) |
| **Turn detection** | `server_vad`, `semantic_vad`, or disabled for manual `commit` |
| **Function calling** | Supported, including `function_call_output` result injection |
| **Image input** | Supported (syntax differs per family, see below) |
| **Session lifetime** | Realtime GA: session carries `expires_at`, measured at roughly 30 minutes from connect. Model Studio: measured idle disconnect at 300 seconds |
### Measured latency and concurrency
Measured 2026-08-24 (UTC+8) over the public `api.apiyi.com` path, 20 concurrent sessions × 2 models, single-turn text-only exchange:
| Metric | Measured |
| -------------------- | ----------------------------------------------------- |
| WebSocket handshake | p50 0.65–1.08 s |
| First text delta | p50 0.54–0.95 s |
| Full single turn | p50 \< 1 s |
| Session success rate | 99.6% (one handshake failure, recovered on reconnect) |
These are point-in-time measurements at a specific concurrency level and are not a performance commitment. **No availability SLA is offered during the beta** — implement reconnection and graceful degradation on the client.
## Endpoint
| Endpoint | Purpose | Auth |
| ---------------------------------------------------- | ----------------------------- | ----------------------------- |
| `wss://api.apiyi.com/v1/realtime?model=` | Open a realtime voice session | `Authorization: Bearer ` |
All four models share this endpoint; the `model` query parameter selects which one you reach.
**On connecting from the browser**: this endpoint also accepts auth via the `Sec-WebSocket-Protocol` subprotocol (`realtime, openai-insecure-api-key., openai-beta.realtime-v1`), so a browser `WebSocket` can connect directly — but that **hands your key to the browser**, where any visitor can read it from the network panel. **Use it for local verification only.** In production, write a backend relay: the backend holds the key and opens the connection to APIYI, while the frontend talks only to your own service.
## ⚠️ Protocol Comparison (read before switching models)
The two families share the endpoint, the auth scheme and the overall event flow. The differences are concentrated in the `session.update` field structure and a few server event names.
### Request field comparison
| Purpose | Model Studio protocol | Realtime GA protocol |
| ------------------- | ----------------------------------- | --------------------------------------------------------- |
| Output modality | `modalities: ["text","audio"]` | `output_modalities: ["audio"]` |
| Voice | `voice` (top level) | `audio.output.voice` |
| Speed | Not supported | `audio.output.speed` (0.7 / 1.0 / 1.5 measured as linear) |
| Input audio format | `input_audio_format: "pcm"`, 16 kHz | `audio.input.format: {"type":"audio/pcm","rate":24000}` |
| Output audio format | `output_audio_format: "pcm"` | `audio.output.format: {"type":"audio/pcm","rate":24000}` |
| Turn detection | `turn_detection` (top level) | `audio.input.turn_detection` |
| Input transcription | `input_audio_transcription` | `audio.input.transcription` |
### Server event comparison
| Content | Model Studio protocol | Realtime GA protocol |
| ---------------------- | --------------------------------- | ---------------------------------------- |
| Text delta | `response.text.delta` | `response.output_text.delta` |
| Audio delta | `response.audio.delta` | `response.output_audio.delta` |
| Audio transcript delta | `response.audio_transcript.delta` | `response.output_audio_transcript.delta` |
All other events — `session.created`, `session.updated`, `conversation.item.create`, `input_audio_buffer.append`, `input_audio_buffer.commit`, `response.create`, `response.cancel`, `response.done` — are named identically in both.
### Two minimal session.update payloads
The same thing written twice; copy directly. **Model Studio protocol**:
```json theme={null}
{
"type": "session.update",
"session": {
"modalities": ["text", "audio"],
"voice": "Ethan",
"input_audio_format": "pcm",
"output_audio_format": "pcm",
"input_audio_transcription": { "model": "qwen3-asr-flash-realtime" },
"turn_detection": { "type": "semantic_vad" }
}
}
```
**Realtime GA protocol**:
```json theme={null}
{
"type": "session.update",
"session": {
"type": "realtime",
"output_modalities": ["audio"],
"audio": {
"input": {
"format": { "type": "audio/pcm", "rate": 24000 },
"transcription": { "model": "whisper-1" },
"turn_detection": { "type": "semantic_vad" }
},
"output": {
"format": { "type": "audio/pcm", "rate": 24000 },
"voice": "alloy",
"speed": 1.0
}
}
}
}
```
**Sample rate is a hard constraint**: `audio.input.format.rate` on the Realtime GA protocol must be **≥ 24000**; sending 16000 fails immediately with `integer_below_min_value: Expected a value >= 24000`. The Model Studio protocol requires 16 kHz input. Resample on the client.
## Start With Text: What the Text Channel Is For, and a Three-Step Self-Test
An audio pipeline involves microphone capture, resampling, chunking and turn detection. Any broken link shows up as "nothing happens", which is hard to diagnose. So **do not start with a microphone**.
### Text is the control plane, not a fallback input
In a realtime voice model, text is not "another way to send input" — it is **the entire control channel other than the audio stream**:
| Combination | Typical use |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Text → control plane | `instructions` system prompts, `function_call_output` tool results, retrieved context — all text, none reachable via microphone |
| Text → audio | Effectively a **TTS with full conversational context**: let a regular model think it through, then have the realtime model say it. Good for announcements and prompts |
| Text → text | **The cheapest debugging channel.** Plain text chat is better served by a regular chat model; its value here is verifying the chain at no cost |
### Three-step self-test
Set `output_modalities` to text only, disable turn detection, and send one `input_text`. That alone verifies the handshake, the key and group, **whether you picked the right field template**, whether `session.update` took effect, whether tools inject correctly, whether multi-turn context holds, and how concurrency behaves. **No audio tokens are produced at all.**
Use a fixed local audio file instead of a microphone, feeding it into `input_audio_buffer.append` in 100 ms chunks. This decouples the **audio pipeline** (format, sample rate, chunking, `commit`, VAD triggering) from your business logic and makes it repeatable — the same file should produce the same result twice.
Once the first two steps pass, only capture and playback remain. If something breaks now, the search space is already small.
**No test audio handy?** On macOS, built-in tools generate a compliant file in one line:
```bash theme={null}
say -v Samantha -o /tmp/ask.aiff "What is the weather in Beijing today? Answer in one sentence."
# Realtime GA protocol uses 24000
afconvert -f WAVE -d LEI16@24000 -c 1 /tmp/ask.aiff ask_24k.wav
# Model Studio protocol uses 16000
afconvert -f WAVE -d LEI16@16000 -c 1 /tmp/ask.aiff ask_16k.wav
```
Picking the wrong sample rate is the most common failure in step 2 — the two protocols differ, do not mix them up.
### A runnable text smoke test
Depends only on `websockets` (`pip install websockets`). Flip one variable to switch protocols:
```python theme={null}
import asyncio, json, os, websockets
FAMILY = "ga" # ga = gpt-realtime-2.1 series; omni = qwen3.5-omni series
MODEL = "gpt-realtime-2.1" if FAMILY == "ga" else "qwen3.5-omni-plus-realtime"
URL = f"wss://api.apiyi.com/v1/realtime?model={MODEL}"
HEADERS = {"Authorization": "Bearer " + os.environ["APIYI_API_KEY"]}
# The two protocols diverge only here: session structure and the text-delta event name.
SESSION = ({"type": "realtime", "output_modalities": ["text"],
"audio": {"input": {"turn_detection": None}}}
if FAMILY == "ga" else
{"modalities": ["text"], "turn_detection": None})
TEXT_DELTA = "response.output_text.delta" if FAMILY == "ga" else "response.text.delta"
async def main():
async with websockets.connect(URL, additional_headers=HEADERS) as ws:
while json.loads(await ws.recv())["type"] != "session.created":
pass
await ws.send(json.dumps({"type": "session.update", "session": SESSION}))
await ws.send(json.dumps({"type": "conversation.item.create", "item": {
"type": "message", "role": "user",
"content": [{"type": "input_text", "text": "Explain WebSocket in one sentence."}]}}))
await ws.send(json.dumps({"type": "response.create"}))
while True:
e = json.loads(await ws.recv())
if e["type"] == TEXT_DELTA:
print(e["delta"], end="", flush=True)
elif e["type"] == "response.done":
print("\n\nusage =", json.dumps(e["response"]["usage"]))
return
elif e["type"] == "error":
print("\nERROR:", json.dumps(e))
return
asyncio.run(main())
```
If this runs, the endpoint, key, group and field template are all correct — move on to step 2.
## Session Features: Voice, Turn Detection, Tools, Images
### Voice
| Item | Model Studio protocol | Realtime GA protocol |
| ---------------- | ------------------------------- | ----------------------------------------------------------------- |
| Default voice | `Tina` | `marin` |
| Verified working | `Tina`, `Ethan` and others | `alloy`, `marin`, `cedar`, `shimmer`, `verse` |
| Speed control | Not supported | `audio.output.speed`; duration scales linearly at 0.7 / 1.0 / 1.5 |
| Invalid voice | `Voice 'xxx' is not supported.` | `invalid_value` |
**Pin the voice in the first `session.update`.** Once a session has produced audio output, changing the voice fails with `cannot_update_voice` — this applies to both protocols. Open a new session to switch voices. Also, on the Model Studio protocol **do not send an empty string as the voice**; it falls back to an unsupported voice and returns 400. Simply omit the field if you do not need to set it.
### Turn detection: server\_vad and semantic\_vad
* `server_vad` — splits on silence duration, with straightforward parameters (`threshold`, `silence_duration_ms`, `prefix_padding_ms`).
* `semantic_vad` — splits on conversational intent, ignoring filler words and meaningless background noise. More robust in multi-speaker environments.
* You can also disable turn detection (`null` or `none`) and run in **manual mode**: send `input_audio_buffer.commit` yourself, then `response.create`. This suits push-to-talk interfaces where the UI controls turns.
In VAD mode you **must keep streaming**. After the speech ends, keep pushing a short stretch of silence (2 seconds is enough in testing) so the server can detect end-of-speech. If you push only the voiced part and then stop, `speech_stopped` never fires and no response is generated.
### Function calling
Event order: the model emits a `response.output_item.done` of type `function_call` (carrying `call_id` and `arguments`) → the client executes it → the result is injected → another `response.create` lets the model continue.
```json theme={null}
{
"type": "conversation.item.create",
"item": {
"type": "function_call_output",
"call_id": "call_xxx",
"output": "{\"city\":\"Beijing\",\"weather\":\"light rain\",\"temp_c\":21}"
}
}
```
The full loop is verified on all four models — after injection the model correctly restates what the tool returned.
### Image input
**Realtime GA protocol**: put `input_image` directly into the message; the value can be a data URI.
```json theme={null}
{
"type": "conversation.item.create",
"item": {
"type": "message",
"role": "user",
"content": [
{ "type": "input_image", "image_url": "data:image/jpeg;base64,..." },
{ "type": "input_text", "text": "What does the image say?" }
]
}
}
```
**Model Studio protocol**: images are treated as **video frames**, so audio must be appended first, otherwise you get `Error append image before append audio.`. In testing, the working approach is to interleave `input_image_buffer.append` into the `input_audio_buffer.append` stream at roughly one frame per second.
## Known Limitations (beta)
All four items below are measured, and all of them affect client code. Read this before integrating.
| Behavior | Affected family | Client workaround |
| --------------------------------------------------------------------------------------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `response.done` is not delivered after `response.cancel`, so the turn never closes out | Model Studio (reproduced in all 6 test runs) | Use `response.output_item.done` as the end-of-turn signal, plus a 5-second timeout. The session itself is unaffected and conversation continues normally after the interruption |
| Idle connections dropped at 300 seconds; WebSocket ping/pong **does not count as activity** | Model Studio | Send an application-level event periodically while idle, or accept the disconnect and reconnect automatically; replay `session.update` after reconnecting |
| Voice cannot be changed once the session has produced audio; fails with `cannot_update_voice` | Both families | Pin `voice` in the first `session.update`; open a new session to switch |
| Input-transcription completion event is not delivered in manual `commit` mode | The `flash` model on Model Studio | Switch to `server_vad` / `semantic_vad`, where transcription works normally. The conversation itself is unaffected — the model understands and answers the audio correctly |
These behaviors may change as upstream evolves during the beta; this page will be kept current. If you hit something not listed here, please report it via [WeCom support](https://work.weixin.qq.com/kfid/kfc9adfd5810ece25ec) or [feedback@apiyi.com](mailto:feedback@apiyi.com), including the timestamp and `session.id` so we can trace it.
## Best Practices
Write the two `session.update` payloads as two config constants selected by model name, rather than scattering if-branches. This is the part most likely to break during maintenance six months later.
Set `output_modalities`, `voice`, `speed`, `turn_detection` and `transcription` in the very first `session.update`. Voice especially — once audio has been produced it is too late.
Run the text smoke test on this page to confirm endpoint, key, group and field template are all correct, then move to audio. Audio tiers cost an order of magnitude more than text, so this saves most of your integration budget.
PCM signed 16-bit, mono; 16 kHz for Model Studio, ≥ 24 kHz for Realtime GA. Do not expect server-side correction — a wrong format usually shows up as silence rather than an explicit error.
Do not wait on `response.done` alone. This approach is correct on both families and avoids hanging a turn when the user interrupts.
Watch the 300-second idle limit on Model Studio and `expires_at` on Realtime GA. **After reconnecting, replay `session.update` and any required context**, otherwise the new session runs with defaults.
Keep the key on the backend and have the frontend talk only to your own service. Direct browser connections work technically but expose the key.
## Errors and Retries
| Symptom | Meaning | What to do |
| --------------------------------------------------- | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| Handshake returns 401 | Invalid key, or the `Authorization` header was not sent | Check the key itself, whether you used `https://` by mistake, and whether the header is present |
| Handshake returns 503 with no available channel | The key is not enabled for the beta group, or the model name is wrong | Contact support to confirm the group is enabled; verify the `model` parameter |
| Handshake returns 400 | The `model` query parameter is missing | The endpoint must include `?model=` |
| `invalid_request_error` on an unknown field | **Wrong protocol family** | Switch to the field template for that model using the comparison tables above |
| `integer_below_min_value` | Input sample rate below 24000 on Realtime GA | Resample to 24 kHz or higher on the client |
| `cannot_update_voice` | Voice changed after the session produced audio | Pin the voice in the first frame; open a new session to switch |
| `Error append image before append audio.` | An image was appended before any audio on Model Studio | Append audio via `input_audio_buffer.append` first, then image frames |
| WebSocket 1006 / 1011 | Network instability or an upstream disconnect | Reconnect with exponential backoff (1 s / 4 s / 16 s) and replay `session.update` |
| Connection drops after roughly 5 minutes of silence | The Model Studio idle limit | See the keepalive approach under Known Limitations |
Troubleshooting tip: record the `event_id` of each event and the `session.id` of the session, and include them when reporting an issue — it shortens diagnosis considerably. Note also that **Realtime GA error objects carry `code` and `param`** (naming the exact field and its accepted values), while Model Studio error messages are coarser. Validate field syntax on the former first when debugging.
## FAQ
Interactive playgrounds are driven by OpenAPI specs, which describe a single request and a single response over HTTP. Realtime is dozens of event types flowing in both directions over one long-lived connection, which does not map onto that model. The alternative is the text smoke test in the "Start With Text" section — a few dozen lines, no microphone, and it confirms the chain works.
**No.** The endpoint and auth are the same, but request fields and event names belong to two protocols. At minimum you must change: `modalities` ↔ `output_modalities`, `voice` ↔ `audio.output.voice`, `input_audio_format` ↔ `audio.input.format`, `turn_detection` ↔ `audio.input.turn_detection`, `input_audio_transcription` ↔ `audio.input.transcription`, plus the event names `response.text.delta` ↔ `response.output_text.delta` and `response.audio.delta` ↔ `response.output_audio.delta`. See the Protocol Comparison section for the full mapping.
Check five things in order: 1. the scheme is `wss://`, not `https://`; 2. the endpoint includes `?model=`; 3. the `Authorization: Bearer ` header is present; 4. the key is enabled for the beta group (503 with "no available channel" if not); 5. no reverse proxy in between is stripping the `Upgrade` header — this is a common issue when relaying through your own gateway.
Technically yes — the endpoint accepts auth via the `Sec-WebSocket-Protocol` subprotocol, so a browser `WebSocket` can connect directly. But that **hands your key to the browser**, where any visitor can read it from the network panel, so it is **only suitable for local verification**. In production write a backend relay: the backend holds the key and opens the connection to APIYI, and the frontend talks only to your own service.
The Realtime GA protocol requires an input sample rate of **at least 24000**; 16000 returns `integer_below_min_value`. The correct form is `"audio": {"input": {"format": {"type": "audio/pcm", "rate": 24000}}}`. The two Model Studio models require 16 kHz instead — the two are not interchangeable.
Follow the three-step self-test in the "Start With Text" section: verify the chain over text first (no audio tokens produced), then replay a local wav file to verify the audio pipeline, and only then connect a live microphone. Test audio can be generated in one line with the macOS built-ins `say` and `afconvert` — the commands are in that section.
This is a known behavior of the two Model Studio models (reproduced in all 6 test runs): after an interruption you receive `response.text.done`, `response.content_part.done` and `response.output_item.done`, but `response.done` is not delivered. **Use `response.output_item.done` as the end-of-turn signal and add a timeout as a backstop.** The session itself is unaffected and the conversation continues normally. The two Realtime GA models behave correctly here.
The Model Studio protocol drops connections after **300 seconds of inactivity**, and **WebSocket-level ping/pong does not count as activity** — a heartbeat will not extend that timer. Either send an application-level event periodically while idle (a `session.update`, for instance), or accept the disconnect and reconnect automatically. Remember to replay `session.update` and any required context after reconnecting.
On the Realtime GA protocol the `session.created` event carries `expires_at`, measured at roughly 30 minutes from connect, after which you need to reconnect. On the Model Studio protocol the constraint we mainly observed is the 300-second idle disconnect. Design long conversations on the assumption that sessions expire, and plan how context carries across sessions.
Set the voice in `session.update`: top-level `voice` on Model Studio, `audio.output.voice` on Realtime GA. **Once the session has produced audio output the voice can no longer be changed** — this applies to both protocols and returns `cannot_update_voice`. Pin it in the first frame and open a new session to switch. Also, do not send an empty string as the voice on Model Studio; it returns 400.
The `flash` model on Model Studio does not deliver the transcription completion event in manual `commit` mode (reproduced consistently across runs); the `plus` model does, and both work in VAD mode. **Switch to `server_vad` or `semantic_vad`.** Testing shows the transcript text lands in an undocumented field on the delta events in this case, but that field may change at any time and **should not be relied upon**. Note this only affects displaying what the user said in your UI — the conversation is unaffected, and the model understands and answers the audio correctly.
All four models support image input, but the syntax differs. On Realtime GA you place `input_image` directly in the message. On Model Studio images are treated as **video frames**, so audio must be appended before any image, which is what triggers that error. In testing the working approach is to interleave image frames into the audio stream at roughly one frame per second.
The two Realtime GA models support it and it applies automatically — in testing the second turn within a session already hit, with a value in `usage.input_token_details.cached_tokens`. No cache hits were observed on the two Model Studio models.
The `usage` object on `response.done` reports tokens per modality (text / audio / image, separately for input and output), so cost can be attributed. Audio tiers are substantially higher than text, which is why text-only is recommended during integration. **For actual charges, refer to the [call logs](https://api.apiyi.com/log).**
## Related Documentation
Key creation, base URL, billing modes and other general conventions.
Create keys, select groups and set quotas.
Regular chat models — a better fit for text-only conversation.
Live pricing, endpoints and groups for every model on the platform.
Lowers your effective cost further.
Contact WeCom support with your account and expected concurrency.
Realtime voice is currently in **private beta**. Every measured result on this page is dated 2026-08-24 (UTC+8) and will be updated as upstream changes. If you plan to integrate, hit something this page does not cover, or need higher concurrency, reach us at [hi@apiyi.com](mailto:hi@apiyi.com) / [feedback@apiyi.com](mailto:feedback@apiyi.com).