In 2026, LangSmith, Langfuse, Opik and Braintrust all added a gateway in front of the model providers. The main reason is control: a tracing tool can only record a model call, while a gateway can stop one.
Two Tools That Did Separate Jobs
For the past two years, many teams ran large language model (LLM) applications with two separate tools. A gateway sat between the application and the model providers. It held the provider API keys, translated between provider APIs and retried failed calls. LiteLLM, an open-source proxy, became a common choice for this job. Beside the application sat a tracing tool such as LangSmith or Langfuse, which recorded each model call, tool call and retrieval step so that engineers could see what an agent had done.
The split made sense, because a gateway has to be fast and support every provider’s API, while a tracing tool needs a good interface, storage for large payloads and an SDK that wraps application code. Langfuse still documents the two-tool setup, with a LiteLLM Proxy integration that sends every call through the proxy and into Langfuse. In 2026, the tracing vendors began building the gateway themselves.
Four Launches in Six Months
LangChain announced the LangSmith LLM Gateway in private beta in May 2026 and opened it to public beta on 30 July, and has since made it part of LangSmith’s bring-your-own-cloud deployments as well. Braintrust deprecated its earlier AI proxy and pointed users to a new gateway, which is in preview and free until general availability. Comet released the Opik LLM Gateway as a beta endpoint that accepts OpenAI-format requests. On 4 November, Langfuse opened its AI Gateway in public beta, both on Langfuse Cloud and for self-hosted deployments. The gateway is MIT-licensed and written in Rust.
MLflow, which started as a machine learning lifecycle tool rather than a tracer, has had an AI gateway for longer, with budget alerts and limits.
Gateways have been around for years, and the risks of running one are well documented. The new gateways come from companies whose business was recording model calls, and those companies now want every call to pass through them first.
Why a Budget Needs the Request Path
A tracing tool records a model call after the provider has answered and the tokens have been paid for. That record is enough to draw a spend chart, but the tracing tool cannot stop the spending. Enforcing a team’s budget, blocking a model or removing a secret from a prompt all mean checking the request before it reaches the provider. Only the component in the request path can do that.
LangChain makes this argument in its launch post. It describes teams combining a separate gateway, a guardrails product and an observability tool, and says that preventing problems means enforcing policies at the request layer. Its gateway answers a call that would exceed a budget with HTTP status 402 instead of forwarding it to the provider.
One Entry Point Records Every Call
The second reason is coverage. An SDK records only the code paths that someone instrumented. Calls outside those paths are where agent failures tend to go unnoticed. A script on a laptop, a notebook or a coding agent using a copied API key leaves no trace. A gateway that holds the provider keys sees every call, because there is no other route to the provider.
Langfuse builds its gateway around this idea. According to its gateway documentation, each gateway key belongs to a user or a service account and can carry metadata such as a team or cost centre. Only the gateway holds the provider credential, and administrators approve which models each credential may call. LangSmith publishes setup pages for routing Claude Code, Codex and Gemini CLI through its gateway. These agents call models from a developer’s machine, outside any application’s tracing code.
What a Tracer’s Gateway Adds
The tracing vendors also make the reverse argument: a standalone gateway is not enough on its own. Langfuse’s launch post says a standalone gateway can only count tokens and cost, while its own gateway makes each call part of a full trace. Langfuse’s gateway accepts a W3C traceparent header, so a call through it joins the trace that the application has already started. By default it records usage and cost, not prompt and response content.
This is the strongest technical argument for building the gateway into the tracer. A standalone gateway sees a flat list of completions. A tracer already has the agent’s tool calls and retrieval steps from its SDK, and it can place each gateway call inside that structure.
There is also a commercial reading, which the vendors do not state themselves. Whoever runs the gateway holds the provider keys and the spend report, and the spend report is what finance and platform teams read. That makes a gateway harder to replace than a trace viewer. LangSmith and Langfuse already use that position differently: LangSmith sells Gateway Credits, which let customers call hosted models such as Kimi without their own provider key, while Langfuse’s FAQ says it does not resell tokens.
What the Beta Gateways Cannot Do Yet
These gateways are young, and the vendors document their own gaps. Langfuse’s gateway supports only OpenAI and Anthropic today, with Amazon Bedrock, Google Vertex AI and Google AI Studio on its roadmap. Fallbacks between providers and hard spend caps per team, user or key are planned but not shipped, and for now the gateway offers a spend dashboard and alerts. Its documentation does not describe rate limits or response caching.
The Langfuse gateway does not convert between API formats, so an OpenAI client can call only OpenAI models. The self-hosted version calls official provider URLs only, which rules out Azure OpenAI, Bedrock or an internal proxy for now. Comet describes the Opik gateway as designed for development and testing. Braintrust has not announced pricing for its gateway.
LangSmith’s gateway already does more than the others. Its documentation lists fallbacks, spend limits, rate limits and data policies as current features, and it accepts the OpenAI Chat Completions, OpenAI Responses and Anthropic Messages formats. Fallbacks and rate limits were listed as future work in May, so they arrived during the beta. The product as a whole is still labelled beta.
Load testing shows a different kind of gap. A self-hosted benchmark of the Langfuse gateway, LiteLLM and AcruxCore found that the Langfuse gateway added less latency per call than LiteLLM. With its default Docker settings, however, it failed 739 of 1,000 simultaneous streams, because its process kept Docker’s default limit of 1,024 open files. One line in Langfuse’s Compose file fixed that.
By comparison with these betas, LiteLLM, a standalone gateway for years, supports more than 100 providers and already includes virtual keys, spend tracking, guardrails, load balancing and an admin dashboard. LiteLLM is also moving its request path from Python to Rust. Its Rust AI Gateway is in beta, and the LiteLLM team expects Anthropic-style /v1/messages requests to run end to end in Rust by 1 December, with about 20 times less latency. A few open-source platforms, such as MLflow and AcruxCore, already ship a gateway and tracing in one product.
The Case for Keeping Them Separate
Merging the gateway into the tracer has costs, and the largest is coupling. With a separate gateway, a team changes tracing vendors by changing a logging callback. When the tracer is also the gateway, every service’s base URL, keys and model identifiers change too. Langfuse, for instance, writes model identifiers in the form credential:model.
The tracer also becomes a runtime dependency. If a tracing backend goes down, the team loses traces. If a gateway goes down, model calls fail. A tracing vendor that runs the gateway now needs the uptime of critical infrastructure.
Security vendors want the same place in the request path. Palo Alto Networks completed its acquisition of Portkey, a standalone gateway, in May 2026 and made it the AI gateway of its Prisma AIRS platform. A security team may want the gateway under its own control rather than inside an observability tool. Gateways have also become a target for attackers, as the LiteLLM supply-chain breach in March showed.
How Teams Can Choose
Teams can start by listing what has to happen before a model call. If they only need visibility, a tracer beside the request path is enough and adds no runtime dependency. If they also need budgets, key policies or model approval, they need a gateway of some kind.
Provider coverage comes next. A team should write down every provider and API format it calls in production, including Azure OpenAI and Bedrock, and compare that list with each gateway’s current support rather than its roadmap.
Teams that already run LiteLLM have little reason to move until a beta gateway offers something they lack. The main thing a tracer’s gateway adds is a gateway call placed inside the agent trace, and that has to be worth a migration. Teams starting fresh can keep the gateway as its own layer even when one vendor supplies both. Keeping the base URL and keys in one configuration module means a later move is a configuration change rather than a code change.
Most of these gateways are labelled beta or preview, so the list of gaps will shrink quickly. The open question is who will own the request path inside a typical company: the observability team, the platform team that runs a standalone gateway, or the security team. This year, LangSmith and Langfuse built gateways of their own and Palo Alto Networks bought one, yet in many stacks the request path still runs through LiteLLM.
Credit: Source link
