What Is a Context Window in AI? Limits and Costs

A context window in an LLM is the hard ceiling on total tokens evaluated in a single inference pass, including input tokens, output tokens, and control tokens. It defines the absolute working memory available to the model during an API call rather than just the volume of text you can submit in your prompt.1

Understanding this distinction is critical for developers managing production budgets. While context sizes have scaled to 1M tokens across frontier architectures, physical hardware bottlenecks and asymmetric output limits change how multi-turn pipelines must be designed.2

What a context window actually measures in an LLM

In transformer models, self-attention computes vectors of weights for each token in a sequence of text, where each weight represents how relevant that token is to others in the sequence. The size of the context window determines exactly how many token relationships can be tracked at one time.3

Because every request consumes part of this shared budget, developers tracking Claude token pricing must account for the full scope of their calls. In documentation from OpenAI, the context window includes all input tokens, output tokens, and control tokens. When an LLM generates a response, each generated output token occupies space within that same context boundary alongside system instructions and conversation history.1

Total context window capacity versus max output tokens

A common mistake when designing agent pipelines is assuming that a model with an advertised context window can generate completions equal to that entire length. In practice, generation limits are often capped well below the total input boundary.2

For example, Claude Fable 5.1, Claude Opus 5.5, Claude Sonnet 5.5, and Claude Haiku 5.5 all provide a 1M-token context window with a max output limit of 128K tokens. By contrast, Google built Gemini 4 Argon architecture to support long-decode generation up to 1M tokens, enabling generation sequences that match the entire breadth of its context capacity.24

Hardware bottlenecks and attention degradation inside long windows

Expanding context capacity is bounded by physical computing constraints. Three architectural and hardware constraints limit context window sizes: transformers, GPU memory, and KV cache memory growth.3

During inference, GPUs move data around in fast memory (SRAM) with limited capacity and slow memory (HBM) with a higher capacity. Attention to tokens constantly shuffles between the two types, and the bottleneck on the GPU is its memory bandwidth. As conversation depth grows, storing the key-value states of prior tokens in the KV cache requires massive memory throughput.3

Beyond memory bandwidth, long contexts suffer from retrieval failure points. Language model performance can degrade significantly when accessing relevant information placed in the middle of long contexts compared to the beginning or end. This 'lost in the middle' phenomenon means that feeding an entire documentation corpus into an open prompt often underperforms compared to structured retrieval pipelines.5

Comparing context limits and API costs across frontier models

Operating inside multi-hundred-thousand-token windows carries stark cost trade-offs. Standard pricing scales steeply on high-reasoning tiers, though lighter models provide massive margins for ingestion.2467

From our data: For 1 million input plus 1 million output tokens, Anthropic Claude Haiku 5.5 costs $0.60 and OpenAI gpt-6-astra costs $60, 100 times as much. Standard rates on earlier tiers also demonstrate this spread, with Claude Opus 5 priced at $5 input and $25 output per million tokens, while Claude Sonnet 5 matched Sonnet 5.5 at $2 input and $10 output.7

How prompt caching alters large-context economics

Because loading large prompt prefixes repeatedly overloads GPU memory bandwidth, providers use prompt caching to reuse calculated KV cache states across requests. Understanding cache mechanics directly impacts the bottom line of multi-agent workflows.38

Anthropic prompt caching provides a default 5-minute lifetime measured from the start of the request that writes or reads the cache entry. Pricing multipliers charge 1.25 times the base input price for 5-minute cache writes and 2 times base input for 1-hour writes. However, as detailed in our guide to Claude prompt caching, cache hits and refreshes on Claude Opus 5.5 and Claude Sonnet 5.5 are charged at 0.05 times the base input token price.8

Competitors implement different billing structures. GPT-6 Astra standard processing rates for context lengths under 272K charge $1.00 per million cached input tokens against its $10.00 base input rate. Gemini 4 Argon standard paid pricing through November 30, 2026, charges $0.10 per million tokens for context caching, relying solely on implicit caching without support for explicit user writes. Meanwhile, Gemini 3.8 Flash charges an hourly context caching storage price of $0.50 per 1,000,000 tokens through December 31, 2026.46

When building applications requiring large contexts, decouple context capacity from output needs. If a workflow demands massive context reuse across multi-turn queries, select architectures supporting aggressive prompt caching discounts like Gemini 4 Argon or Claude Sonnet 5.5, and place mission-critical facts at the immediate beginning or end of your prompts to prevent retrieval degradation.458