Rate Limits and Usage Caps - Why You Hit Them and How to Cope

Sooner or later you will see a message saying you have reached a limit, or an API call will fail with a "too many requests" error. It feels like a malfunction, but it is a deliberate control. This guide explains the two main kinds of limits and gives you practical ways to live within them.

In plain English

Running these models costs real computing power, and providers share it among many customers. Limits stop one person or program from using more than their share and protect the service for everyone. Think of a shared printer with a queue: you can print a lot, just not all at once.

Two different things

1. Usage caps in apps (chat products). Consumer and team apps typically limit how much you can use in a period, often depending on your plan, the model and how heavy your requests are. Exact allowances vary and change, so look at your plan's current help page rather than relying on remembered numbers. Longer conversations, large attachments and the most capable models generally use up allowances faster.

2. Rate limits on APIs. Developers calling a model from code face limits on how many requests and how much text they can send per minute, and sometimes on total spend. Limits commonly grow as your account history and spending grow, and many providers let you view them in a dashboard or request an increase.

What counts toward the limit

• The number of requests.
• The amount of text, measured in tokens, in both what you send and what you get back.
• Re-sending a long conversation each turn, which uses more with every message.
• Large files and images.
• Extra features such as long reasoning or tool use, which add tokens.

If you hit a cap in a chat app

1. Read the message. It usually says when you can continue.
2. Start a new chat for new topics. Long conversations carry all their earlier text forward on every turn.
3. Trim the inputs. Paste only the relevant section, not the whole document.
4. Pick the right model. A lighter model for simple tasks saves the heavier one for hard problems, where the app offers a choice.
5. Combine requests. One well-written prompt beats ten small follow-ups.
6. Check your plan. Compare it with your actual usage, and consider whether a team or business offering fits better.

If you hit a rate limit in code

Handle it as a normal event, not a crash.

• Retry with exponential backoff: wait a short time, then longer after each failure, with a bit of random jitter so many clients do not retry in lockstep.
• Respect hints the service sends, such as a header saying how long to wait. Check the provider's documentation for the exact names.
• Cap retries, then fail with a clear message.
• Queue work and process it at a steady pace instead of firing everything at once.

import random, time

def call_with_backoff(fn, tries=5):
    for attempt in range(tries):
        try:
            return fn()
        except RateLimitError:          # use your library's real error class
            time.sleep((2 ** attempt) + random.random())
    raise RuntimeError("Still rate limited after retries")

Using fewer tokens

• Shorten prompts and remove repeated boilerplate.
• Send only relevant context. Retrieval helps here, see our RAG guides.
• Cap the maximum output length.
• Summarize older conversation turns instead of resending them all.
• Cache results you will need again, and check whether your provider offers prompt caching for repeated prefixes.
• For work that can wait, look for a batch option, since some providers process bulk jobs asynchronously on different terms.
• Use smaller models for simple steps, and larger ones for hard steps.

Planning for scale

• Estimate: requests per minute times tokens per request, then compare with your limits.
• Add monitoring and alerts for errors, spend and latency.
• Set budgets and spending caps where offered.
• Spread work over time rather than in bursts.
• Keep a fallback: a second model or provider for graceful degradation.

Technical notes

• Limits are usually enforced on several dimensions at once, such as requests per minute and tokens per minute, so you can hit one before the other.
• Rate-limit errors are normally retryable. Authentication and invalid request errors are not. Do not retry those.
• Your library may already retry automatically. Check before adding your own, so retries do not multiply.
• Log rate-limit events with timestamps to see patterns and size your needs.
• Concurrency control, such as a semaphore around calls, prevents self-inflicted bursts.

Next steps

• Look up your current plan's or account's limits on the provider's official page.
• Add exponential backoff with jitter to any code calling an API.
• Start a fresh chat for each new topic this week and note the difference.
• Read our guides on managing long conversations and on controlling AI costs.
• Set a spending alert if your provider offers one.