Meta engineering team introduced ZGateway, a proxy tier that now sits between client applications and ZippyDB, the Meta’s most widely used key value store. ZippyDB backs product metadata, counters, and configuration at billions of operations per second. ZGateway started as a fix for connection sprawl across more than a million client hosts and grew into the home for batching, admission control, caching, and failover.
Why ZippyDB Needed a Proxy
Under direct access, every ZippyDB client connected to every database host it needed. A single client could touch tens of thousands of shards across hundreds of thousands of hosts, so both a typical client and a typical database host carried tens of thousands of TLS connections. Each idle connection consumed memory, CPU, and a file descriptor on both ends, and inbound counts grew with every client cohort. Reconnection storms caused crashes from file descriptor exhaustion and OOMs; in one incident a routing bug made every client open a connection per shard and the fleet fell into a reboot loop. Client-side fixes were impractical because hundreds of teams own the client fleet.
What is ZGateway
ZGateway is a stateless proxy tier between ZippyDB clients and the ZServer database fleet. Per Meta, it handles more than 1 billion operations per second and carries about 40% of ZippyDB traffic, projected to pass 60%, at about 6% computational overhead for an average use case.
It runs as regional tiers discovered through ServiceRouter, Meta’s service mesh, in two flavors: a pure proxy and a read-through cache. The engine is Meta’s thick C++ ZippyDB client, so ZGateway is effectively a ZippyDB client run as a managed service.
A client sends requests over a sticky connection to a regional ZGateway host, which terminates TLS, authorizes against the use case’s ACLs, applies per-tenant admission control and shaping, resolves the shard, checks the local cache on caching tiers, batches the request with other in-flight work for that shard, and forwards it to the correct replicas. Responses are demultiplexed back with per-use-case metrics, traces, and quota usage recorded. TLS stays in the Thrift/ServiceRouter stack and replica selection stays in the embedded client.
The Fan-In and Fan-Out Math
Meta models the fleet as balls thrown into bins: with B shards and H hosts, a host is hit with probability . With mock figures of 20 regions, 500,000 database hosts, 30,000 proxy hosts, 1,000,000 clients, and 50,000 shards per client, per-host connection counts collapse by roughly 97 to 98% and total persistent connections drop about 19x. The deeper win is scaling: direct-access fan-in grows linearly with clients, while ZGateway fan-in reduces to roughly regions times shard density per host, independent of both fleets.
Capabilities That Followed
- Safe migration: Configuration flags scoped per service and shard prefix provide a percentage ramp, a region filter, and a global kill switch.
- Discriminant Load Shedding (DLS): Requests map to per-tenant buckets split by priority that drain round-robin, so a flooding tenant only fills its own bucket. In a controlled overload above 90% CPU across roughly 1,350 tenant buckets, only 6 noisy neighbors shed load, the rest executed 99.9% of requests with zero rejections, goodput held near 97 to 98%, and the machinery cost about 8% of CPU.
- Read caching: Cache tiers serve hot reads in-process, take a per-key fill lock on misses, and stay fresh via change-data-capture events under a bounded-staleness contract.
- Load balancing: Tiers mix roughly 26-core to 126-core hosts, so a control-plane balancer nudges each host’s ServiceRouter weight opposite to its recent CPU load.
- Cross-region resilience: Global routing, mega-regions, and rings let a saturated regional tier fail over to healthy capacity nearby.
- Transactions: Client-side bookkeeping moved into the gateway, consolidated in nine phases to 100% of transaction traffic with no reliability regression.
Key Takeaways
- ZGateway handles over 1B ops/sec and carries about 40% of ZippyDB traffic at roughly 6% overhead.
- The proxy turns database fan-in from linear in client count into a bounded number Meta controls.
- Cross-client batching and coalescing kill hot-key stampedes and retired fragile client libraries.
- DLS isolated 6 noisy tenants out of about 1,350 under 90%+ CPU with 97 to 98% goodput.
- Not deployable outside Meta; the value is in the patterns, not a package.
Check out the Meta Engineering blog post and the X announcement. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
Michal Sutter is a data science professional with a Master of Science in Data Science from the University of Padova. With a solid foundation in statistical analysis, machine learning, and data engineering, Michal excels at transforming complex datasets into actionable insights.
Credit: Source link
