Rate limits and concurrency
Control request volume and handle 429 responses.
Two different limits
RPM counts requests over time; concurrency counts requests running at once. Long streams occupy concurrency slots. More client threads do not increase server capacity.
Your allowance
Limits depend on account, group, model and service configuration. An unpublished limit is not unlimited. Contact support before scheduling bulk work or requesting additional capacity. No unconfirmed production limit is published here.
Handle 429
Respect Retry-After. Without it, use exponential backoff with jitter and a bounded retry count. Limit client concurrency. Do not continuously retry content refusals, invalid keys or insufficient balance.
Timeouts and cancellation
Set an appropriate read timeout for long tasks. Consume streams incrementally and close the connection when the user cancels. Replaying a request after receiving content may create duplicate output and extra usage.
Still have a question?
Support details have not been published.