Rate limits explained

Published 2026-10-15 · Updated 2026-10-18

“Rate limit” is a pile of different counters wearing one name. Knowing which one you hit decides whether to wait a second, a minute, or a day.

The usual counters

• RPM, requests per minute. How many calls you can start. Easy to hit with parallel workers; easy to fix with a queue.
• TPM, tokens per minute. The real constraint on long prompts and long outputs. A single 100K-context request can eat an entire low-tier budget.
• RPD, requests per day. Common on free tiers (Gemini and Groq both use daily caps). Hit it at 9 AM and you’re done until reset.
• Credit / quota. Money or tokens remaining on the key. 402 errors and “insufficient balance” responses mean this counter, not the per-minute ones.

Reading the headers

Most providers return limit headers, x-ratelimit-remaining-requests, x-ratelimit-reset-*, retry-after. Log them. retry-after on a 429 is the provider literally telling you when to try again; honor it instead of picking a backoff number.

A sane retry pattern

import time, random

def call_with_retry(fn, attempts=5):
    for i in range(attempts):
        try:
            return fn()
        except Exception as e:
            if "429" not in str(e) or i == attempts - 1:
                raise
            time.sleep(min(60, 2 ** i) + random.random())

Exponential backoff with jitter. The jitter matters, ten workers retrying on the same beat just hit the limit again in sync.

Practical fixes, ordered

1. Queue requests instead of firing them all at once.
2. Batch inputs into fewer, larger prompts when TPM is the binding constraint.
3. Rotate keys across the pool when you’re working through a list of shared keys, each has its own counters.
4. Drop to a smaller model for the bulk and reserve the expensive one for what needs it.
5. Cache responses. Half of most workloads are duplicate prompts.

The counter you can’t fix by waiting is quota, that’s when a key is done and you move to the next one.