Skip to main content

Short answer

Three sentences:
  1. Streaming vs non-streaming is entirely decided by your own code—the stream field in the request body. Same key, same model, same endpoint: if it flips back and forth, your client code (or the SDK / framework wrapping it) is doing the flipping. The gateway never switches it randomly.
  2. Both modes return identical final content and are billed identically. The only differences are when you get the text and how you parse it.
  3. How to choose: a human is watching the screen → streaming; a program consumes the result (JSON parsing, batch jobs, tool calls) → non-streaming.

The differences at a glance

Why do my requests flip between streaming and non-streaming?

This is the most common question, and the answer is: something on your side is changing it. Work down this list — one of them almost always matches:
The classic case: stream=config.get("stream", False) or stream=is_web_request. Different entry points reach the same function with different values, and the logs look like the mode is flipping at random.How to check: print the request body you actually send and look at the stream field.
The same business logic behaves differently depending on the client:
  • OpenAI SDK chat.completions.create(): non-streaming by default
  • client.chat.completions.stream() or with_streaming_response: streaming
  • Wrappers like LangChain / LlamaIndex: depends on whether you call invoke or stream, and whether you passed streaming=True when constructing the model object
  • Desktop clients, agent tools, workflow platforms: usually expose a “streaming output” toggle in settings, with varying defaults
How to check: confirm which entry point actually issued the call.
A single key used by both a web chat UI (streaming) and a nightly batch job (non-streaming) produces logs that look random when viewed together.How to check: create separate tokens per use case — the logs then separate themselves. See Token management.
You really did send stream: true, but Nginx, a corporate gateway, or some proxy buffered the response — the server sent it chunk by chunk, the proxy held it and released it all at once, and it feels non-streaming.How to check: test once bypassing the proxy; turn buffering off on Nginx (proxy_buffering off;). Note that in this case the console log still shows is_stream = true, because the gateway genuinely streamed it out.
To confirm what a specific call actually did: check the is_stream field in the console log, or pull it in bulk with the Log Query API. That is the source of truth — far more reliable than impressions.

Choosing by scenario

Use streaming

  • Chat UIs and support bots — users need immediate feedback
  • IDE plugins / coding assistants (Claude Code, Cursor, etc.)
  • Long-form generation (long articles, long translations, large code blocks)
  • Long reasoning-model tasks — at least you can see progress
  • Anywhere the user can hit “stop” mid-generation

Use non-streaming

  • Structured output: you need the whole JSON for json.loads()
  • Parsing function-calling / tool-call arguments
  • Batch processing, offline jobs, scheduled tasks
  • Backend flows where only the final result matters and nobody is waiting
  • Quick verification, debugging, writing test cases
A few special cases:

Integration effort: the same task, both ways

Claude’s native format (/v1/messages) uses a different streaming protocol: Anthropic’s named-event SSE (message_start / content_block_delta / message_delta and friends), not OpenAI’s uniform data: chunks, and usage is split across the message_start and message_delta events. Full parsing guide: Claude native format: streaming and non-streaming responses.

Billing and usage: identical either way

Streaming is neither cheaper nor more expensive. Billing is per token and has nothing to do with how the bytes are transported.Disconnecting midway is still billed—after you hit Ctrl+C or your client times out, the upstream generation still runs to completion and the request is charged normally. So “cut the stream off early to save money” does not work.
Two traps around usage:
  1. Streaming does not return usage by default. On OpenAI-compatible endpoints you must pass stream_options: {"include_usage": true}; the usage then arrives in the final chunk (whose choices array is empty — check before indexing). This is verified working on several models on APIYI.
  2. Don’t reconcile your bill against the usage echoed by the API, especially the cache-related fields. The echoed values do not always match what was actually billed; whether a cache hit occurred is determined by the “cache billing details” in the console log. See Cache billing explained.
Either way, the console log records the token counts, latency, and billing for every call — transport mode makes no difference. Field meanings: Understanding log billing details.

Six common misconceptions

It does not. Streaming only makes the first token arrive early. It does not shorten total generation time, and it does not guarantee a steady flow of data.Reasoning models (gemini-3.1-pro-preview, gpt-5.6-sol, gpt-5.5-pro, etc.) can emit nothing at all during the thinking phase, which trips your client’s read timeout just the same.The right fix is per-scenario timeout values — see How to avoid API timeouts.
The first byte is faster; the total is not. For the same model and prompt, streaming and non-streaming finish in roughly the same time.Streaming buys you perceived speed: the user sees movement within a second instead of staring at a spinner for 30. If nobody is watching the screen, that value is zero.
No. See “Billing and usage” above: identical billing, and disconnecting midway is still charged.
No. Text chat models generally do; image generation, embedding, and rerank endpoints have no streaming concept and will either ignore stream or reject it.A few models have extra restrictions on certain parameter combinations under streaming. When unsure, get the call working non-streaming first, then add stream: true.
Both have their failure modes.
  • Non-streaming risks: the connection is silent for the whole generation, so proxies, CDNs, and corporate gateways may drop it on idle timeout. With very large response bodies (base64 image output easily reaches tens of MB) you can also hit a stalled terminator — see Requests that finish transferring but never return and Log shows completed but the client gets nothing.
  • Streaming risks: unfriendly to middleboxes that do not support SSE or that force buffering; client parsing is more complex and easy to get subtly wrong.
Also note: api-cf.apiyi.com (the CDN endpoint) has an approximately 100-second request ceiling that affects both modes. For long requests use api.apiyi.com or vip.apiyi.com — see Base URL configuration guide.
You can — you just assemble it yourself. Concatenating every chunk’s delta.content in order gives you exactly the non-streaming message.content.If the assembled text looks incomplete, check three things: whether you ignored finish_reason, whether you exited the loop before receiving data: [DONE], and whether a middlebox truncated the response.

Streaming not working? Four steps

1

Confirm the request body really contains stream: true

Print the JSON you actually send. With wrapper libraries, “I thought I passed it” and “it was passed” are often different things.
2

Test directly with curl -N

Bypass your own code and any proxy using the command in the “cURL side by side” tab above. If curl shows chunks arriving progressively, the server side is fine and the problem is in your client or a middlebox.
3

Check middlebox buffering

Add proxy_buffering off; on Nginx. Corporate gateways and security appliances may scan text/event-stream as a whole payload — ask your network admin to allow it through.
4

Review your parsing logic

Read SSE line by line, skip blank lines and comment lines starting with :, and stop at data: [DONE]. The final chunk carrying usage has an empty choices array — don’t index into it.
If you get this far without an answer, contact support with the request_id — the console log shows directly whether that call was handled as a stream, plus its total latency and time to first byte.

How to avoid API timeouts

Timeout values by scenario, and why streaming doesn’t save you

Base URL configuration guide

Differences between endpoints, and the CDN node’s 100-second ceiling

Log shows completed but no response

The classic large non-streaming response problem, with segment timing

Claude streaming and non-streaming

Parsing Anthropic’s native named-event SSE protocol

Text generation API

Full parameter list and call examples

Understanding log billing details

What each console log field means, including is_stream