Lounge
Connecting…
Connecting…
View all

KV Cache Exposure: Cloudflare GLM-5.3 Leak, What to Check

Cloudflare says 155 Workers AI GLM-5.3 responses held other customers' data. How the KV cache exposure happened and what API users should check.

AILatest7 min read58 viewsPublished Updated
Contents8 sections
  1. What happened in the Cloudflare GLM-5.3 KV cache exposure
  2. What is a KV cache, and why could data cross between users?
  3. How to check whether your app was affected
  4. AI inference security habits worth adopting now
  5. If you self-host SGLang: the official recommendations
  6. What the disclosure does and does not tell you
  7. Frequently asked questions
  8. Sources and references

Key takeaways

  • On October 26, 2026, Cloudflare disclosed that 155 Workers AI GLM-5.3 responses contained other customers' request data between August 27 and September 9, 2026.
  • Every affected response answered a prompt longer than 681,280 tokens. Cloudflare found no evidence of deliberate exploitation and says no other Workers AI model was affected.
  • Cause: a prefill KV cache pool of 681,280 tokens, smaller than the 1,048,576-token context window, plus an SGLang bug that let rejected requests proceed without an error.
  • If you used GLM-5.3 in that window, start with your own logs: model ID, timestamp, and prompt_tokens, then review any stored responses for content unrelated to your request.
  • Teams running their own SGLang clusters should upgrade to v0.5.22 or later and check KV pool size against context length on every node.

A KV cache exposure happens when an LLM serving engine generates a response while reading GPU memory that still holds another request's data, so fragments of someone else's prompt or output end up in your answer. On October 26, 2026, Cloudflare disclosed exactly that for its Workers AI GLM-5.3 model (@cf/zai-org/glm-5.3): between August 27 and September 9, 2026, 155 responses contained data from other customers' requests. This guide, current as of October 2026, sticks to what Cloudflare and the SGLang maintainers have published and turns it into a practical checklist for developers who call hosted AI APIs.

All times below are UTC. Cloudflare has not published a per-customer list of affected or source requests, so anything beyond the official figures is labeled as uncertain.

What happened in the Cloudflare GLM-5.3 KV cache exposure

Security researcher Gadi Evron of Knostic reported through Cloudflare's HackerOne program on September 8, 2026 that very long prompts sent to GLM-5.3 sometimes returned text from other users' sessions. Cloudflare reproduced the leak with two independent accounts the next day, set the model to private within about 40 minutes of declaring an incident, and restored it the same evening after fixing both the configuration and the inference engine.

Time (UTC)Event
2026-08-27GLM-5.3 launches with a prefill KV pool smaller than its context window; exposure present from launch
2026-09-08 23:11Report received via HackerOne
2026-09-09 13:22Security incident declared; leak reproduced with two accounts within 25 minutes
2026-09-09 14:00Model set to private; requests return HTTP 400
2026-09-09 14:52All prefill nodes moved to corrected configuration
2026-09-09 19:40Model restored with corrected config plus engine patch
2026-10-26Coordinated disclosure with the researcher and the SGLang team at RadixArk

Cloudflare counts 155 affected responses, all to prompts longer than 681,280 tokens. After excluding its own and the researcher's tests, the matching requests looked like ordinary long-context use in timing and volume, and Cloudflare says it found no evidence of deliberate exploitation. No other Workers AI model was affected. While it was happening, the public status page described it only as increased GLM 5.3 errors, from 14:25 to 20:09 UTC on September 9.

What is a KV cache, and why could data cross between users?

What is a KV cache, and why could data cross between users?

When a transformer model generates text, it stores the key and value vectors for every token it has already processed so it does not recompute them for each new token. That store is the KV cache. On Workers AI, Cloudflare explains, the cache lives in a shared pool of fixed-size GPU pages. Each request gets pages when admitted and hands them back when finished. Pages are not zeroed between requests for performance reasons; the next owner simply overwrites them. Tenant isolation therefore depends entirely on the engine tracking who owns which page.

Two problems lined up

  1. A configuration mismatch. GLM-5.3 was deployed with a 1,048,576-token context window, but the KV pool on the prefill GPUs only fit 681,280 tokens. Cloudflare knew about the gap before launch, expected the engine to reject anything that did not fit, and its long-context tests checked HTTP status codes rather than output quality.
  2. An SGLang bug in disaggregated mode. Workers AI splits prompt processing (prefill) and token generation (decode) across separate GPU pools. An over-length request was marked as rejected and swapped for a one-token placeholder, but the admission queues only checked whether a request had finished, not whether it had been rejected, so it kept moving.
  3. The result. The decode node reserved pages for the full long prompt, yet only the first page was freshly written. The rest still held earlier users' prompts and partial generations. The model attended over them and the client got HTTP 200 with a fluent reply instead of HTTP 400.

The uncomfortable detail is that nothing errored. The reported prompt_tokens matched the long prompt and the text read naturally. The only tell Cloudflare mentions is that the reply arrived faster than a prompt that size should allow. Monitoring that watches status codes alone would not have caught it.

How to check whether your app was affected

Cloudflare's post does not include a customer action list, so your own logs are the practical way to narrow things down.

  1. 1

    Confirm model and dates

    Search application logs or AI Gateway records for calls to @cf/zai-org/glm-5.3 between August 27 and September 9, 2026 (UTC). If you never called that model, this incident did not touch your traffic directly.

  2. 2

    Filter for very long prompts

    Within that window, pull requests whose prompt_tokens exceeded 681,280. Per Cloudflare, every response that received foreign data falls in this group.

  3. 3

    Review stored outputs

    If you kept those responses, look for names, code, or documents unrelated to your input. Treat anything that looks like another party's data as such: quarantine it and do not feed it into other systems or training sets.

  4. 4

    Assess whether you were a source

    Leaked fragments came from requests made shortly before. If you sent API keys, passwords, or personal data to GLM-5.3 in that window, rotating those credentials and asking your privacy or legal team whether any notification duty applies is the cautious path.

  5. 5

    Ask through official channels

    For account-level confirmation, contact Cloudflare support. Report new security findings through Cloudflare's HackerOne program, as the disclosure requests.

AI inference security habits worth adopting now

The incident shows that a provider promise not to store prompts does not cover every risk: isolation can fail while data sits in memory during processing. Cloudflare's Workers AI data usage page (updated October 16, 2026) says standard stateless inference inputs and outputs are not retained or used for training, but that content is stored if you turn on AI Gateway logging or caching or write prompts to your own logs.

Minimize prompts

Why it matters
Whatever you send is what can leak
Example
Redact keys, IDs, and card numbers before the call

Anomaly checks on output

Why it matters
This leak came back as a clean HTTP 200
Example
Flag replies with entities absent from the input

Lean logging

Why it matters
Your logs become a sensitive data store
Example
Log model, time, and token counts, not raw text

Record model IDs

Why it matters
Disclosures are scoped by model and date
Example
Store model ID and timestamp per request

Fallback routing

Why it matters
A model can be pulled offline suddenly
Example
Configure a secondary model path

Security incidents like this spread fast through developer channels. To see how AI-focused Telegram channels are discussing it, check the AI Telegram trends page, and use the blog trends dashboard to track write-ups from engineering blogs.

If you self-host SGLang: the official recommendations

SGLang is the open-source inference engine Cloudflare uses. The SGLang team's write-up notes the bug only appeared in disaggregated prefill/decode serving; single-node deployments returned an error as intended. The same post covers a separate race-condition family in which late RDMA writes from an aborted request could land in another request's KV cache.

  • Upgrade. Cloudflare strongly recommends that all operators depending on SGLang move to v0.5.22 or later. The incident fixes are #36988 (v0.5.20) and #38935 (v0.5.21).
  • Compare KV pool to context length. After any change to weights, quantization, parallelism, or memory fraction, read the pool size from each prefill and decode node's startup log. If it is smaller, lower the context length. Do not rely on a proxy to enforce the limit.
  • Enable deferred KV release on both roles and cap Mooncake transfer batch sizes.
  • Turn on corruption detection with --disaggregation-enable-kv-checksum and the KV canary where your model supports them, and confirm in logs that requests are actually verified.
  • Test outputs, not just status codes. Cloudflare called validating only response codes and performance the critical gap in its testing.

The SGLang maintainers say race-type corruption leaves no error or log line, so operators have no reliable way to tell whether exposure already occurred. After upgrading, they recommend watching the deferred-release metrics added in v0.5.22 to confirm the protection is working.

What the disclosure does and does not tell you

Confirmed: one model (GLM-5.3), one window (August 27 to September 9, 2026), 155 responses, a trigger condition of prompts over 681,280 tokens, no evidence of deliberate exploitation, and the fixes. After remediation, the prefill KV pool holds about 2.04 million tokens, roughly twice the context window.

Not disclosed: what kind of data leaked, whether it included personal data, and which customers' requests were the source. That makes both "my data is definitely safe" and "massive breach" unsupported claims. Multi-tenant GPU serving keeps costs down, but its isolation is only as good as the engine's bookkeeping. For sensitive workloads, the controls you own, such as minimal prompts, output checks, and lean logs, are the most reliable place to start.

Frequently asked questions

Was my data leaked if I only sent short prompts to GLM-5.3?
Your responses were not among the affected ones: Cloudflare says all 155 answered prompts longer than 681,280 tokens. The other direction is less clear. The leaked material came from other customers' requests to the same model made shortly before, so a short prompt you sent during August 27 to September 9, 2026 could in principle have surfaced in someone else's response. Cloudflare's post does not list which requests were the source, so you cannot rule this out from public information alone.
Did this affect other Cloudflare models or other AI providers?
Cloudflare confirmed no other Workers AI model was affected, because its other disaggregated deployments have KV pools larger than their context length, so no request could reach the faulty rejection path. Other providers are outside the scope of this disclosure. However, the SGLang team says a related family of KV cache lifetime bugs existed in prefill/decode disaggregated setups, so anyone self-hosting that architecture should audit it separately.
Is a KV cache leak the same as prompt injection?
No. Prompt injection is an attack that smuggles instructions into a model's input. This incident was a memory-management fault in the serving engine: an ordinary long request was enough to make the decoder read cache pages left behind by earlier requests. Content filters and model safety settings cannot stop that; only fixes to the infrastructure configuration and engine can.
Does Cloudflare store my Workers AI prompts?
According to Cloudflare's Workers AI data usage page (updated October 16, 2026), standard inference requests that do not use a stateful feature are not retained after the response is delivered, and customer content is not used for training. Storage does happen if you enable AI Gateway logging or caching, write prompts to application logs, or use storage services. The GLM-5.3 issue involved short-lived GPU memory, not a stored database.

Sources and references

  1. 1Cloudflare Blog: Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang(opens in a new tab)blog.cloudflare.com
  2. 2LMSYS Org: SGLang security update: improving request isolation(opens in a new tab)www.lmsys.org
  3. 3Cloudflare Status: Cloudflare Workers AI increased errors(opens in a new tab)www.cloudflarestatus.com
  4. 4Cloudflare Docs: Workers AI Data usage(opens in a new tab)developers.cloudflare.com