Cloudflare on October 26, 2026 published an incident report disclosing a cross-request KV cache exposure in the GLM-5.3 model on its Workers AI platform, under which 155 responses to very long prompts contained data from other customers’ sessions.
Security researcher Gadi Evron of Knostic reported the tenant-isolation issue to Cloudflare on September 8, 2026 through the company’s HackerOne program. The exposure affected the model, served as @cf/zai-org/glm-5.3, starting August 27, 2026, the day GLM 5.3 launched on Workers AI. Cloudflare remediated the issue and restored the model on September 9, 2026, and said it confirmed that no other model hosted on Workers AI was affected.
All 155 affected responses went to prompts longer than 681,280 tokens. Cloudflare said that, after excluding its own testing and the researcher’s requests, the requests it identified matched ordinary long-context use in timing and volume rather than repeated exploitation of the bug, and that it found no evidence of deliberate exploitation.
The Two Issues Behind the Exposure
GLM is a text generation model from Z.ai built for agentic coding workflows, one of many models available through Workers AI. Cloudflare runs GLM 5.3 with SGLang, an open-source inference engine, in a disaggregated configuration: prefill GPUs process the prompt and produce its KV cache, then hand that cache over remote direct memory access (RDMA), via Mooncake, to decode GPUs that generate the response. The KV cache is a shared pool of fixed-size pages; pages are not zeroed between requests, so the engine is responsible for tracking ownership of each page.
The report traces the exposure to two issues. The first was a misconfiguration: Cloudflare deployed GLM 5.3 with a 1,048,576-token context window, but the KV cache pool on its prefill GPUs held only 681,280 tokens. Cloudflare said it was aware of the mismatch before launch but expected the engine to return an error for any prompt it could not process, and its long-context tests returned successful responses, so it did not believe the mismatch was an issue. The company said its testing validated HTTP response codes and engine performance at long contexts without validating model output and coherence, and that it now checks KV pool size against context length directly before any disaggregated deployment change ships.
The second was an SGLang bug in the handling of over-length requests in disaggregated prefill and decode mode. A prompt that did not fit the pool was internally marked as rejected and swapped for a one-token placeholder, but prefill still generated KV cache for the placeholder and signaled that the transfer was complete. The decode node then generated a response while attending over stale cache pages that still held data from earlier requests, and the client received an HTTP 200 response instead of HTTP 400.
The SGLang team states in its own coordinated post that the single-node rejection path returned an error instead, so the issue occurred only in disaggregated serving. The bug had been reported to SGLang by another user as issue #30233 on July 6, 2026, which Cloudflare said it noticed while investigating.
Incident Response and Remediation
Cloudflare began verifying the report at 08:00 UTC on September 9, 2026 and declared a security incident at 13:22 after its Product Security team reproduced the leak using two independent accounts within 25 minutes. It set GLM 5.3 to private at 14:00, identified the root cause at 14:35, rolled all prefill nodes to the corrected configuration by 14:52, rebuilt the full deployment with a Cloudflare-authored engine patch by 19:04, and restored the model to public access at 19:40. The Cloudflare status page recorded the Workers AI incident as starting at 10:25 AM EDT on September 9, 2026, identified at 11:02 AM EDT, and resolved at 4:09 PM EDT.
The configuration fix matched the GLM 5.2 setup, freeing enough GPU memory that the prefill KV pool now holds about 2.04 million tokens, roughly twice the 1,048,576-token context length, so a prompt that fits the context window always fits the pool. The engine patch retires a request carrying a pending rejection at the disaggregation admission queue, with its original error, before any bootstrap, allocation, or KV transfer happens. Cloudflare validated both changes against the researcher’s reproduction script and its own canary-marker tests before restoring the model.
Upstream Fixes and Ongoing Work
Cloudflare proposed its fix upstream as SGLang pull request #38704, and the maintainers merged #38935, which rejects the affected requests at admission following Cloudflare’s approach; a related fix, #36988, had been merged on September 6, 2026 for a related failure path. The patch shipped in SGLang v0.5.20, with follow-ups in v0.5.21, released October 2, 2026, and v0.5.22, released October 19, 2026, and Cloudflare recommends all SGLang operators update to v0.5.22 or later. The SGLang team states that v0.5.21 protects Mooncake and NIXL deployments by default for every failure kind, that MORI deployments need a build in v0.5.22, and that the fix line was not backported to earlier release branches.
The SGLang team reported that across 21 runs on two H100 GPUs, throughput deltas with the protections were within ±2%, with every 95% confidence interval spanning zero, and that a separate 16-run sweep on two H200 GPUs measured the cost of enabling deferred KV release by default at −0.38%, with a confidence interval of −2.12% to +1.36% that it described as not significant. Cloudflare met the SGLang maintainers at RadixArk on September 10, 2026, rolled additional hardening to all of its production SGLang deployments between September 11 and 19, 2026, and upstream merges covering the admission fix, per-transfer checksums, abort-acknowledgment preservation, and default-on deferred KV release ran from September 16 to October 12, 2026.
For other operators running SGLang in disaggregated mode, Cloudflare’s report recommends measuring the KV pool against context length on every prefill and decode node after any change to weights, quantization, parallelism, or memory fraction; upgrading to a build that includes both admission fixes; running with deferred KV release on both roles; enabling corruption detection where the model supports it; and configuring load balancers so a client disconnect does not abort a request mid-transfer.
Cloudflare said it had previously built a prototype design that tags every physical KV page and checks the mapping before decode reads the cache, and that it cost under 1% in throughput and tail latency in tests; the company tested the prototype on GLM 5.2 in mid-August 2026 but disabled it shortly afterward due to reliability issues. Cloudflare is working with the SGLang team on a production version built into the engine, with the per-transfer checksum in pull request #39500 as a first step, and said it is directing its formal-methods experience toward modeling and proving the safety of an RDMA-based KV cache transfer protocol using Mooncake in its production SGLang deployment.
The SGLang team’s post acknowledges Cloudflare engineers Kevin Flansburg and Kastan Day for their work on the fixes and lists the coordinated disclosure by SGLang and Cloudflare on October 26, 2026 as the final entry in its disclosure timeline. The researcher’s own report, which the SGLang team said will describe the discovery and external validation, is forthcoming.
Credit: Source link
