The gateway limits how fast a key and an organization can send requests. The limits protect the service and keep one noisy client from starving the others.
| Limit | Default | Applies to | On breach |
|---|---|---|---|
| Requests per minute, per key | 60 | Each API key. You can set a different value on the key. | 429 rate_limited with Retry-After |
| Requests per minute, per organization | 600 | All keys of the organization together. | 429 rate_limited with Retry-After |
| Concurrent requests, per organization | 20 | Completion requests running at the same time (a stream counts until it ends). | 429 rate_limited, Retry-After: 1 |
| Failed authentication attempts, per IP address | 30 per minute | Requests that present a missing, malformed or unknown key. | 429 rate_limited with Retry-After |
| Request body | 2 MiB | Each request. | 413 request_too_large |
| Request time | 120 seconds | Non-streaming requests. | 504 upstream_timeout, not charged |
| Stream length | 10 minutes | Streaming requests. | The stream ends with an error event and is billed for what was produced |
The defaults can be changed by the operator, and a key can carry its own per-minute limit, so read the Retry-After header instead of hard-coding the numbers above.
How the minute limits work
Each limit is a token bucket. The bucket holds as many requests as the per-minute limit (a burst of that size is allowed) and refills continuously at one sixtieth of the limit per second. A request takes one token. With the default key limit of 60, you can send up to 60 requests at once and then roughly one per second.
The limits are enforced per gateway process, and they reset when the process restarts.
Handling 429
A 429 carries Retry-After, the number of seconds to wait. The body tells the cases apart:
rate_limited: slow down, then retry. Concurrency breaches can clear in about a second.key_cap_exceeded: the key hit its monthly spending cap. Retrying before the next month does not help; raise the cap or use another key.
curl -sS -o /dev/null -w "%{http_code}" https://api.ticonai.com/v1/models \
-H "Authorization: Bearer $AIP_API_KEY"The gateway does not send x-ratelimit-* headers. Use Retry-After, and add exponential backoff with jitter when many clients run together. See Best practices.
Related
- A request that is rate limited is rejected before the wallet is touched, so it costs nothing.
- Spending limits are separate from rate limits: see the monthly cap in API keys and the hold rules in Billing and wallet.