Long-context AI is expensive mainly because standard transformer attention compares every token with every other token. Double the context and the number of pairs to score roughly quadruples; make it ten times longer and the pairs grow a hundredfold. Subquadratic attention is the line of research that tries to bend that curve by letting each query attend to only a chosen subset of keys. This guide explains the mechanics, what two new arXiv preprints submitted on October 30, 2026 actually claim, and how developers can control long context cost today. Facts were checked against the papers and the official Anthropic and OpenAI pricing documentation as of November 3, 2026.

Why long context is expensive: attention complexity is quadratic

A large language model processes text as tokens. In each attention layer, every token produces a query that is scored against the keys of the tokens it can see, and those scores decide how the corresponding values are mixed. With n tokens there are about n times n scores to compute. The FlashAttention paper states the problem plainly in its abstract: transformers are slow and memory-hungry on long sequences because the time and memory complexity of self-attention are quadratic in sequence length (arXiv 2205.14135).

The table is a conceptual calculation. A real model also contains feed-forward layers and other components whose cost grows linearly with length, so total cost does not rise exactly with the square. Still, once the context reaches hundreds of thousands or millions of tokens, attention dominates. The LSA authors describe full attention above one million tokens as prohibitively expensive (arXiv 2610.42147).

Prefill, decode and the KV cache

It helps to split inference into two phases. Prefill reads the whole prompt at once; it is the wait you notice before the first word appears after pasting a long document. Decode generates the answer one token at a time, and each new token attends to everything before it. To avoid recomputing, servers keep the keys and values in a KV cache. The longer the context, the more GPU memory that cache occupies, which reduces how many requests one GPU can serve at the same time. For a provider, fewer concurrent users per GPU is the real cost of long context, and that cost eventually shows up in pricing.

What is subquadratic attention? Sparse and hierarchical indexers

Subquadratic attention is an umbrella term for attention methods whose cost grows more slowly than the square of the context length. The most common route is sparse attention: instead of scoring every key, each query computes exact attention over a small set of keys that look relevant. The catch is the indexer that chooses those keys. If the indexer itself scans every key, the method is still quadratic underneath. The ZHA paper groups earlier work into two camps: sliding-window and linear attention variants with a fixed-size state, which give up some expressiveness, and sparse methods whose indexers scale quadratically, which only deliver constant-factor speedups (arXiv 2610.42112).

The intuition behind a hierarchical indexer is a library catalog. Rather than checking every book, you walk from section to shelf to row, discarding most of the building along the way. Keys are grouped into summaries; a query scores the top-level summaries first and only descends into promising branches. The selected keys then receive exact attention. Both new papers share this structure, though they build the hierarchy differently.

What the two new papers claim: LSA and ZHA

Latent Sparse Attention (LSA)

LSA clusters keys hierarchically in a learned latent space, and queries score the hierarchy of clusters instead of every key. The authors say this speeds up prefill, decode and training at long context. They propose Distilled-LSA, which can be added to pre-trained models through distillation, and Native-LSA, trained from scratch on next-token loss alone. According to the abstract, at 10M tokens D-LSA speeds up one attention layer by 22.7x in prefill and 4.6x in decode compared with GLM-5.3-FLASH's original sparse attention, and by 72.4x in prefill and 42.5x in decode compared with full attention in QWEN-3.8-27B, while keeping similar accuracy across a range of tasks. N-LSA is reported to match full attention accuracy with a 1.6x per-layer training speedup at 128k tokens (arXiv 2610.42147).

Zyphra Hierarchical Attention (ZHA)

ZHA builds a tree of recursively compressed key summaries over the KV cache. Future queries traverse it from the top down to fetch a sparse subset of keys. By growing the tree's depth as the sequence grows, the per-query cost, including routing and the final attention step, rises sublinearly, so overall attention scales subquadratically but still superlinearly. The authors report that ZHA can be applied to existing pretrained models with a short KL distillation stage over a few billion tokens, that it maintains performance on long-context retrieval, reasoning, math and agentic coding evaluations, and that it stays robust with trees up to six levels deep. Their performance model suggests large advantages over competing methods beyond one million tokens and potential to extend effective context to hundreds of millions or billions of tokens (arXiv 2610.42112).

To follow how new models and papers are being discussed, the theddari AI Telegram trends page and the blog trends page collect mentions in one place. Buzz is not the same as validated performance, so keep reading the original abstracts and evaluation scope alongside any headline.

Page Reference

Trendstheddari.com›TrendsTelegram · AILive posts from AI channels

Page Reference

Trendstheddari.com›TrendsBlog PostsSummarized Naver blog posts

How long context cost shows up in API pricing

API customers do not pay for attention operations directly; they pay tokens times the per-token rate. That makes the bill look linear, but providers fold the quadratic compute and memory burden into their pricing policies. Based on the official documentation checked on November 3, 2026, policies fall into two broad patterns: a flat rate across the window, or a higher rate once a request crosses a threshold.

Anthropic notes that for Haiku 5.5 the prompt length includes cache reads and cache writes, and a request over the threshold pays the higher price even if part of it is a cache hit (Claude Pricing). OpenAI applies the GPT-5.4 surcharge across standard, batch and flex tiers (GPT-5.4 docs). In other words, two models with similar context windows can produce very different bills near the threshold. Prices change often, so recheck the current tables before you budget.

Practical ways to cut long context cost today

You do not have to wait for new attention methods to reach production models. Several levers are already in your hands.

1. 1

   Know each model's threshold

   Check whether your model has a long-context price break, such as 100,000 tokens or 272K tokens. If many requests land just above it, trimming a little context to stay below can change the effective rate significantly.

2. 2

   Cache repeated prefixes

   Sending the same system prompt, reference document or chat history every time means paying to process it every time. On Anthropic, a cache hit costs 0.1x the base input rate (lower on some newer models) and a five-minute cache write costs 1.25x, so a single reuse already pays off. Caches expire, so plan for rewrites.

3. 3

   Batch work that can wait

   Anthropic's Batch API discounts both input and output by 50 percent, which suits bulk summarization or offline analysis. OpenAI's GPT-5.4 docs also list batch endpoint support.

4. 4

   Put only what matters in the prompt

   Use retrieval to pull relevant passages instead of whole documents, and replace old conversation turns with summaries. Fewer input tokens lower both cost and time to first token.

5. 5

   Measure quality with a small eval set

   After trimming context or switching models, compare answers on a few dozen real questions from your workload. Record the results so you know savings did not come at the expense of accuracy.

Quality risks and how to read the research

The core risk of sparse attention is simple: keys that are not selected are left out. If the indexer misses the key holding the decisive fact, the model may answer as if it never saw it. Rare but critical details, such as one exception clause in a long contract or a single config line in a large codebase, are the most exposed. Read any claim of similar accuracy together with the tasks, models and lengths where it was measured.

• Check the unit of the speedup: LSA's 22.7x and 72.4x figures are for one attention layer, not whole-model latency.
• Check the baseline: speedups over an existing sparse attention and over full attention are different comparisons.
• Check how it was applied: distilled variants may behave subtly differently from the original model.
• Check for replication: treat results as provisional until code, weights or independent reproductions appear.
• Check for deployment: a paper is not an announcement of lower API prices or a specific product change.

In short, long context costs more because attention compares every pair of tokens, and subquadratic attention tries to shrink that structural cost by using hierarchical indexers to compare fewer pairs. Until that research is validated and deployed, the reliable way to manage long context cost is to know your model's price thresholds, cache repeated prefixes, batch non-urgent work, keep prompts lean, and measure quality on your own tasks.