Short answer
- Streaming vs non-streaming is entirely decided by your own code—the
streamfield in the request body. Same key, same model, same endpoint: if it flips back and forth, your client code (or the SDK / framework wrapping it) is doing the flipping. The gateway never switches it randomly. - Both modes return identical final content and are billed identically. The only differences are when you get the text and how you parse it.
- How to choose: a human is watching the screen → streaming; a program consumes the result (JSON parsing, batch jobs, tool calls) → non-streaming.
The differences at a glance
Why do my requests flip between streaming and non-streaming?
This is the most common question, and the answer is: something on your side is changing it. Work down this list — one of them almost always matches:1. `stream` is a variable or a config value in your code
1. `stream` is a variable or a config value in your code
stream=config.get("stream", False) or stream=is_web_request. Different entry points reach the same function with different values, and the logs look like the mode is flipping at random.How to check: print the request body you actually send and look at the stream field.2. Different SDKs and frameworks have different defaults
2. Different SDKs and frameworks have different defaults
- OpenAI SDK
chat.completions.create(): non-streaming by default client.chat.completions.stream()orwith_streaming_response: streaming- Wrappers like LangChain / LlamaIndex: depends on whether you call
invokeorstream, and whether you passedstreaming=Truewhen constructing the model object - Desktop clients, agent tools, workflow platforms: usually expose a “streaming output” toggle in settings, with varying defaults
4. A middlebox flattened the stream
4. A middlebox flattened the stream
stream: true, but Nginx, a corporate gateway, or some proxy buffered the response — the server sent it chunk by chunk, the proxy held it and released it all at once, and it feels non-streaming.How to check: test once bypassing the proxy; turn buffering off on Nginx (proxy_buffering off;). Note that in this case the console log still shows is_stream = true, because the gateway genuinely streamed it out.Choosing by scenario
Use streaming
- Chat UIs and support bots — users need immediate feedback
- IDE plugins / coding assistants (Claude Code, Cursor, etc.)
- Long-form generation (long articles, long translations, large code blocks)
- Long reasoning-model tasks — at least you can see progress
- Anywhere the user can hit “stop” mid-generation
Use non-streaming
- Structured output: you need the whole JSON for
json.loads() - Parsing function-calling / tool-call arguments
- Batch processing, offline jobs, scheduled tasks
- Backend flows where only the final result matters and nobody is waiting
- Quick verification, debugging, writing test cases
Integration effort: the same task, both ways
- Python non-streaming
- Python streaming
- Node.js streaming
- cURL side by side
/v1/messages) uses a different streaming protocol: Anthropic’s named-event SSE (message_start / content_block_delta / message_delta and friends), not OpenAI’s uniform data: chunks, and usage is split across the message_start and message_delta events. Full parsing guide: Claude native format: streaming and non-streaming responses.Billing and usage: identical either way
Two traps aroundusage:
- Streaming does not return usage by default. On OpenAI-compatible endpoints you must pass
stream_options: {"include_usage": true}; the usage then arrives in the final chunk (whosechoicesarray is empty — check before indexing). This is verified working on several models on APIYI. - Don’t reconcile your bill against the
usageechoed by the API, especially the cache-related fields. The echoed values do not always match what was actually billed; whether a cache hit occurred is determined by the “cache billing details” in the console log. See Cache billing explained.
Six common misconceptions
1. Streaming prevents timeouts
1. Streaming prevents timeouts
gemini-3.1-pro-preview, gpt-5.6-sol, gpt-5.5-pro, etc.) can emit nothing at all during the thinking phase, which trips your client’s read timeout just the same.The right fix is per-scenario timeout values — see How to avoid API timeouts.2. Streaming is faster
2. Streaming is faster
3. Streaming is cheaper, or only bills what you received
3. Streaming is cheaper, or only bills what you received
4. Every model and endpoint supports streaming
4. Every model and endpoint supports streaming
stream or reject it.A few models have extra restrictions on certain parameter combinations under streaming. When unsure, get the call working non-streaming first, then add stream: true.5. Non-streaming is more reliable
5. Non-streaming is more reliable
- Non-streaming risks: the connection is silent for the whole generation, so proxies, CDNs, and corporate gateways may drop it on idle timeout. With very large response bodies (base64 image output easily reaches tens of MB) you can also hit a stalled terminator — see Requests that finish transferring but never return and Log shows completed but the client gets nothing.
- Streaming risks: unfriendly to middleboxes that do not support SSE or that force buffering; client parsing is more complex and easy to get subtly wrong.
api-cf.apiyi.com (the CDN endpoint) has an approximately 100-second request ceiling that affects both modes. For long requests use api.apiyi.com or vip.apiyi.com — see Base URL configuration guide.6. You can't get the complete answer from a stream
6. You can't get the complete answer from a stream
delta.content in order gives you exactly the non-streaming message.content.If the assembled text looks incomplete, check three things: whether you ignored finish_reason, whether you exited the loop before receiving data: [DONE], and whether a middlebox truncated the response.Streaming not working? Four steps
Confirm the request body really contains stream: true
Test directly with curl -N
Check middlebox buffering
proxy_buffering off; on Nginx. Corporate gateways and security appliances may scan text/event-stream as a whole payload — ask your network admin to allow it through.Review your parsing logic
:, and stop at data: [DONE]. The final chunk carrying usage has an empty choices array — don’t index into it.