DEV Community: Anthony Max

My first open-source service reached its first 20 stars on GitHub 💎

Anthony Max — Mon, 02 Nov 2026 05:19:39 +0000

Hello everyone Today, I’d like to share a small but joyful piece of news. My project on GitHub has reached its first 20 stars ☆.

I’ve been working on it since July, and for the most part, I don’t have much time for development and promotion due to work. In any case, I’d like to share this small achievement with you. I think it’s important to talk about it openly when you’re doing something in Open‑Source.

For a long time, I didn’t understand how to make it useful for other people, but I guess I should write to other people about it — those whose projects don’t even have 5 stars.

By the way, what’s noteworthy is that now there are also 10 projects there 💎, which can’t but please the users.

I’ve written about it many times, but if anyone wants to get acquainted with EmbedCatalog, here’s the repository:

https://github.com/embedcatalog/embedcatalog

Maybe you can suggest something to do in the issue, or you can create something cool in the code yourself!

Best article

Anthony Max — Wed, 28 Oct 2026 07:25:30 +0000

Compares top tools by routing and licensing limits

Anthony Max

Oct 27

5 Best Open-source LLM Gateways 💎

#ai #webdev #programming #opensource

12 min read

[Boost]

Anthony Max — Tue, 27 Oct 2026 14:29:51 +0000

Compares top tools by routing and licensing limits

Anthony Max

Oct 27

5 Best Open-source LLM Gateways 💎

#ai #webdev #programming #opensource

12 min read

5 Best Open-source LLM Gateways 💎

Anthony Max — Tue, 27 Oct 2026 13:27:11 +0000

TL;DR

An open-source LLM gateway gives applications one place to connect to model providers. Depending on the project, it can also handle routing, retries, budgets, access controls, caching and request logs.
The five projects here take different approaches: Bifrost, Kong, Agent Router, Apache APISIX and LiteLLM.
Bifrost is our pick for teams that want a self-hosted gateway with model, MCP and agent traffic controls. Its commercial Edge product can also route AI traffic from employee devices.
Open source does not mean every feature is free or self-hostable. Check each project's license and feature boundaries before adopting it.

A few years ago, many teams adopted gateways to centralize provider keys and billing. Those needs have not gone away, but teams now also have to think about model choice, reliability, usage limits and the growing number of tools connected to agents.

The projects in this list are not interchangeable. Some focus on model routing, while others build AI features on top of a broader API or service-mesh stack. I compared five open-source LLM gateways, their deployment models and the features available without a paid plan.

What an open-source LLM gateway does

At a basic level, an open-source LLM gateway is a reverse proxy for model requests. An application sends traffic to a shared endpoint, and the gateway handles provider-specific formats, routing and retries across services such as OpenAI, Anthropic, AWS Bedrock and Google Vertex AI.

Some projects add budgets, access controls, audit trails, caching or MCP support. Others focus on getting requests to the right model with low operational overhead. The right choice depends on which of those jobs you need the gateway to do.

How to choose an open-source LLM gateway

Provider coverage. Confirm the providers and API formats your application needs today, then check how new providers are added.
Reliability and latency. Look for retries, fallbacks and load balancing. Treat benchmark numbers as a starting point.
Deployment and license. Check the license for the gateway itself, and distinguish open-source components from paid hosting, support or enterprise features.
Operations. Consider how the project handles secrets, budgets, access controls, logs and upgrades. A gateway still needs someone to run it.
MCP and agent support. If your applications call tools, check whether the gateway routes or governs MCP traffic, and how identity and permissions work.

The five gateways at a glance

Here is the short version. Bifrost and LiteLLM are model-focused gateways, Kong brings AI traffic into a broader API platform, Agent Router is built around Envoy Gateway and Kubernetes, and Apache APISIX combines AI plugins with a general-purpose API gateway. These five open-source LLM gateways differ in deployment and feature boundaries.

Capability	Bifrost	Kong AI Gateway	Agent Router	Apache APISIX	LiteLLM
Form factor	Standalone Go gateway	AI features in Kong Gateway	Envoy Gateway control plane and CLI	API gateway with AI plugins	Python SDK and proxy
Model coverage	25+ providers, 10,000+ models	Major hosted and self-hosted APIs	Hosted and self-hosted providers	OpenAI, Anthropic, Bedrock and more	100+ providers
Routing and reliability	Retries, fallbacks, load balancing	Routing, load balancing, plugins	Central routing, quotas and failover	Multi-provider proxy, balancing and retries	Routing, retries, fallbacks
MCP support	Native MCP gateway	AI MCP Proxy plugins	AI and MCP traffic	OpenAPI-to-MCP plugin	MCP gateway
Deployment	Self-hosted; Enterprise adds private options	Self-hosted or Konnect	Kubernetes or standalone CLI	Self-hosted	Self-hosted proxy
A good fit when	You want one gateway for models and agents	Kong already runs your APIs	Your platform is built on Envoy	You want an open-source API and AI gateway	Your stack is Python-first

The sections below look at what each project does well, where it fits and what to check before adopting it.

1. Bifrost: A gateway for models, MCP and agents

Bifrost is an open-source LLM gateway for enterprise AI, licensed under Apache 2.0 and written in Go. It provides one OpenAI-compatible interface to more than 10,000 models across 25+ providers. With its drop-in setup, an existing app can usually switch over by changing its base URL. Bifrost ships as a single binary and can run on a VM, in Kubernetes, in a private cloud or on a disconnected network.

Core capabilities

Performance is one of Bifrost's main selling points. Its published benchmarks report about 11 microseconds of overhead at 5,000 requests per second on a single t3.xlarge, with a 100% success rate. In a separate AWS EC2 comparison with 500 concurrent users over 60 seconds, Bifrost handled 424 requests per second, while LiteLLM handled 44.84. The reported P99 latencies were 1.68 seconds and 90.72 seconds, respectively.

For governance, virtual keys let teams set budgets and rate limits by key, team, customer or model. The Enterprise tier connects to identity providers such as Okta, Microsoft Entra, Keycloak, Zitadel, Auth0 and Google Workspace, as well as other OIDC providers. It also adds SCIM 2.0 directory sync, detailed role-based access control, data access controls and reusable access profiles. Routing supports weighted load balancing, retries with backoff and automatic fallbacks. Semantic caching, backed by Redis, Qdrant, Weaviate or Pinecone, can return stored answers for similar prompts.

// Detect dark theme var iframe = document.getElementById('tweet-2024540241844224013-789'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2024540241844224013&theme=dark" }

The MCP gateway controls which MCP servers and tools agents can use. Access can be filtered at the client, request and virtual-key levels, with per-user OAuth so each person connects using their own account. Code Mode takes a different approach to large tool sets: instead of loading every tool definition into the model context, the model writes a short script to coordinate the tools it needs. Bifrost reports that this can reduce input token use by up to 92.8%. In a 508-tool test, the reported cost fell from $377 to $29, with all tasks passing. The MCP gateway write-up has the full results.

Bifrost Edge extends those controls to AI tools running on employee devices. Available for macOS, Windows and Linux, it routes traffic from desktop chat apps, browser-based AI, terminal coding agents and their MCP servers through the Bifrost gateway after a one-time SSO sign-in. Edge routes AI traffic only and does not inspect other device activity. Gateway virtual keys, budgets and guardrails can then apply to those tools, including content safety, PII and secret detection on requests and responses, as well as integrations with AWS Bedrock Guardrails, Azure Content Safety, Google Model Armor, CrowdStrike AIDR and Check Point Guard. Administrators can allow or block apps and MCP servers centrally. Rules are enforced on the device. Rollout can use existing device management tools such as Jamf, Intune, Kandji, Omnissa Workspace ONE or JumpCloud.

For regulated environments, the Enterprise tier adds clustering, adaptive load balancing with circuit breakers, signed audit logs, and integrations with AWS Secrets Manager, GCP Secret Manager and HashiCorp Vault. It also supports in-VPC, on-premises and air-gapped deployments, where prompts and responses can stay within your infrastructure. Bifrost lists SOC 2 Type II, ISO 27001, HIPAA and GDPR among its compliance credentials, and says Fortune 500 companies use the product in production across several industries.

Where it fits

There are a few things to check before committing. Identity-based governance, clustering, adaptive load balancing, guardrail and vault integrations, and in-VPC deployment are part of the paid Enterprise tier, not the open-source core. Edge also requires an Enterprise gateway and is priced per endpoint. Since it is still in early access, a limited pilot makes sense before a wider rollout.

Best for: Teams that need to manage model, MCP and agent traffic from one gateway, especially when they have strict deployment or governance requirements. Bifrost supports self-hosted, VPC-isolated, on-premises and air-gapped environments.

💎 Check out the Bifrost repository ☆

2. Kong AI Gateway: AI features in an established API platform

Kong AI Gateway adds AI-specific features to Kong's established API gateway. If your APIs already run on Kong, you can use the same control plane, plugin model and operational setup for model traffic. The AI features include provider routing, token-based rate limits, prompt guard plugins, semantic caching and semantic routing across models.

Core capabilities

Kong added MCP support in Gateway 3.12 in October 2025. Its AI MCP Proxy plugin can expose REST APIs as MCP tools or sit in front of an MCP server. Access controls apply both when clients discover tools and when they invoke them. A companion OAuth2 plugin handles authentication for MCP traffic.

Where it fits

The main trade-offs are licensing and operational scope. Kong's documentation lists the MCP plugins as part of its AI Gateway Enterprise offering, and labels the OAuth2 plugin as a tech preview. If Kong is not already in your stack, deploying the full API gateway platform just for model traffic may be more than you need.

Best for: Organizations that already use Kong and want to manage AI traffic through their existing control plane.

3. Agent Router: Envoy-based routing for Kubernetes

Agent Router, formerly Envoy AI Gateway, is an Apache 2.0 project for managing access to hosted models, self-hosted inference and MCP servers. It provides an OpenAI-compatible API and is built around Envoy Gateway. The repository has moved to theagentrouter, but project documentation says the API resources, API group, CLI name and image paths remain unchanged.

Core capabilities

The project describes a two-tier pattern: one gateway handles shared authentication, routing and global limits, while another can manage traffic to self-hosted inference. It also offers a standalone CLI for local use, which makes it possible to try the API before deploying the Kubernetes components.

Where it fits

The trade-off is its platform focus. Teams already operating Envoy Gateway and Kubernetes have a clear path to evaluate it. For a small application that just needs a lightweight proxy, a simpler gateway may involve less setup. Its documentation and release notes are the best places to check the current status of specific providers and features.

Best for: Platform teams using Envoy Gateway or Kubernetes that want an open-source control plane for model and tool traffic.

4. Apache APISIX: Open-source AI plugins in an API gateway

Core capabilities

Apache APISIX is an Apache Software Foundation project licensed under Apache 2.0. Its open-source API gateway includes AI plugins for connecting to model providers, routing requests across instances, balancing traffic, retrying failures and limiting usage by tokens. The ai-proxy plugin supports documented providers and OpenAI-compatible APIs, while ai-proxy-multi adds multi-instance load balancing and fallback behavior. The ai-rate-limiting plugin applies token quotas.

APISIX also has an openapi-to-mcp plugin that exposes operations from an OpenAPI document as MCP tools. This is a useful contrast with Kong: APISIX's AI plugins are part of its open-source project, while Kong documents its MCP Proxy plugins as Enterprise features.

Where it fits

Because APISIX is a full API gateway, teams adopting it for AI traffic also take on its broader gateway configuration and operations. Its plugin model suits teams that want to manage API and model traffic together, with the AI capabilities available in the open-source project.

Best for: Platform teams that want an open-source API and AI gateway with multi-provider routing, token limits and MCP support.

5. LiteLLM: Broad provider support and a Python-first workflow

Core capabilities

LiteLLM is a common starting point for teams building with language models. It combines an MIT-licensed Python SDK with a self-hosted proxy that supports more than 100 providers, virtual keys, budgets, routing, fallbacks, guardrails and an MCP gateway. Semantic caching supports Qdrant, Valkey and Redis. Its documentation recommends using that feature for single-shot prompts rather than multi-turn agent traffic. A paid enterprise license adds SCIM, audit logs and some guardrail integrations. Admin UI SSO is free for up to five users.

Trade-offs

Two considerations are worth a closer look. In Bifrost's published benchmarks, LiteLLM handled 44.84 requests per second with a P99 latency of 90.72 seconds in an AWS EC2 test with 500 concurrent users over 60 seconds. Bifrost handled 424 requests per second with a P99 latency of 1.68 seconds in the same comparison. These are vendor-published results; performance varies by workload.

There was also a supply-chain incident. Malicious PyPI versions 1.82.7 and 1.82.8 were published on March 24, 2026; LiteLLM says a maintainer's PyPI account may have been compromised, according to its security update. The affected versions were removed. The incident involved a compromised package release rather than a vulnerability in LiteLLM's gateway code, and highlights the value of pinning dependencies and mirroring packages internally.

Best for: Python-focused teams with moderate traffic that value a broad provider ecosystem and can operate and secure a self-hosted service.

Deployment and licensing compared

This table is a starting point, not a substitute for checking the current license and feature matrix. Several projects keep parts of their enterprise functionality outside the open-source gateway.

Control	Bifrost	Kong AI Gateway	Agent Router	Apache APISIX	LiteLLM
License	Apache 2.0 core; Enterprise features are paid	Apache 2.0 Kong Gateway; some AI plugins are paid	Apache 2.0	Apache 2.0	MIT core; some enterprise features are paid
Where it runs	Self-hosted; private deployment options in Enterprise	Self-hosted, hybrid or Konnect	Kubernetes or standalone CLI	Self-hosted	Self-hosted proxy
Main trade-off	Some governance and deployment features require Enterprise	Advanced AI features may require a license	Best suited to Envoy and Kubernetes stacks	Broader API gateway to configure and operate	Python-based operations and security upkeep

Which AI Gateway Is Right for Your Team

If you are choosing a gateway for production, start with the constraints that matter most to your team. Bifrost provides multi-provider routing, agent workflows and private deployment options. Its published benchmark reports 11 µs of overhead.

LiteLLM is a natural option for Python-centric teams. Kong fits organizations already running Kong, while Agent Router suits teams built around Envoy Gateway and Kubernetes. Apache APISIX adds AI capabilities to an open-source API gateway. The best choice is the one that fits your existing platform and the amount of gateway operations your team is ready to own.

Frequently asked questions

What is an open-source LLM gateway?

An open-source LLM gateway sits between an application and one or more model providers. Depending on the project, it can provide a shared API, route requests, retry failures, apply usage limits and log traffic. Check the license and feature list because commercial hosting or enterprise controls may be separate.

How is an LLM gateway different from an API gateway?

A traditional API gateway focuses on routing and authentication for general API traffic. An LLM gateway adds features specific to model requests, such as provider routing, token-based limits, model fallbacks and usage reporting. Some API gateways now offer AI plugins as well, so the distinction depends on the product.

Should we self-host or use a managed gateway?

Self-hosting gives your team more control over prompts, keys and logs, but also means operating the service. A managed gateway can be quicker to adopt, though traffic passes through the vendor's network. The right choice depends on your security requirements and the operational support available to your team.

Do LLM gateways support MCP?

MCP support varies. Bifrost has a native MCP gateway, Kong offers AI MCP Proxy plugins, Agent Router handles AI and MCP traffic, Apache APISIX provides an openapi-to-mcp plugin, and LiteLLM includes MCP support in its proxy. Compare authentication and tool-level permissions rather than treating support as a yes-or-no feature.

How much latency does an LLM gateway add?

The answer depends on the gateway, workload and enabled features. Bifrost reports about 11 microseconds of overhead at 5,000 requests per second on a single t3.xlarge in its published benchmarks. That is small next to the time many models take to respond, but added latency can matter when an agent makes a long sequence of calls.

Can we move off LiteLLM without rewriting code?

Often, yes. Both products expose OpenAI-compatible APIs, and Bifrost provides drop-in adapters for the LiteLLM SDK. Teams using the LiteLLM proxy can also migrate models, keys, teams and users. See the LiteLLM migration guide for the process and its details.

Can an LLM gateway govern employees using ChatGPT or Claude desktop apps?

Only when that traffic passes through the gateway. A standard gateway typically sees requests that developers have configured to use it. Bifrost Edge is designed to route traffic from desktop apps, browsers and coding agents through Bifrost so gateway policies can apply.

Is Bifrost open source, and what does the Enterprise tier add?

Bifrost's core is open source under Apache 2.0 and can be self-hosted for free. The Enterprise tier adds high availability, identity-based governance, guardrail and vault integrations, signed audit logs, and in-VPC or air-gapped deployment. It includes a 14-day free trial. Bifrost Edge requires an Enterprise gateway and is priced per endpoint.

Conclusion

There is no single gateway that is right for every stack. Choose based on the providers you use, how you plan to deploy, and whether you want a dedicated model proxy or AI features inside an existing API platform. Check which features are included in each open-source project, and test each of the five gateways with your own workload before committing. If Bifrost is on your shortlist, its documentation and demo cover its deployment and Enterprise options.

Resources:

Bifrost GitHub: https://github.com/maximhq/bifrost
Bifrost Docs: https://docs.getbifrost.ai
Bifrost CLI: npx -y @maximhq/bifrost-cli

Thanks for reading this article! ❤️

I'd love to hear your thoughts on this in the comments!

💎 Check out the Bifrost repository ☆

How to promote your open-source project 💎

Anthony Max — Sat, 24 Oct 2026 20:01:15 +0000

TL;DR

Hello! Today we’ll be discussing a rather important topic that rarely comes up in conversations. Let’s talk about marketing.

However, not in the usual sense, when people show ads for some kind of food on TV, but in the sense of another product — open‑source code.

Yes, it may seem simple to you, but it’s not. In this article, I will try to share a few points that I’ve understood over the 3 years of working on code.

Well, let’s get started! 🏎️

🔎 Understanding the target audience of the product

The most basic and simplest thing possible. Why does anyone need your product at all? Who should use it in their work? And many similar questions.

Understand that just because you found something interesting to do rather than playing games or watching shows doesn’t mean it’s as important to others as it is to you. This only means that you’ve created a very slim chance that someone will find your project useful.

Today, with AI, your products are no longer needed by anyone, because there are so many of them that people are already sick of yet another open‑source project that will simply stop being supported until the author runs out of money or finds a new project to develop.

Understanding this, you will see that people don’t register in the app or buy a subscription from you not because they’re mean, but simply because your project doesn’t matter. So, for the project to matter at all, you need to create a unique value proposition. The reason why people will choose you and not anyone else. Understanding this will make it easier for you to promote your product.

👀 Thanks! What's next?

You’ve created a product — some mobile app or SaaS — and you want people to use it. You have an idea of how to promote it, but you don’t know where.

That’s already a good question. However, no matter how many articles I read on this topic, everyone starts describing social media, like how great it is to promote your products there. Thank you very much, as if we didn’t know! In 2026, it’s probably not obvious :|

Also, I’ve read equally silly advice about how you need to make a good README file, as if that’s not obvious? As if you, who are reading this article, are completely stupid.

I won’t do that; I’ll just say that it might work.

📋 Lists

These are various lists where people post projects on different topics. For example, the best projects in JavaScript, and so on.

Let’s say there’s ProductHunt:

My experience has shown that the third place there will give you 10–15 stars and that’s it. More emails from people who are advertising their services.

I came up with something, no, not another ProductHunt clone :). It’s the same list, but its distinguishing feature is the embed.

I came up with something, no, not another ProductHunt clone :). It’s the same list, but its distinctive feature is the embed. I’ve created a platform where you can share what you’ve created and show it to everyone on your project.

With profiles, followers, and much more. The cool thing is that you won’t need to pay anyone to get to the top spot on ProductHunt. On the platform, you’ll be able to create your own embed and share it with others — not as an achievement, but as your own creative work.

https://embedcatalog.com

🌐 Subreddit

I hate Reddit, and if I had my way, I’d never be on this social network, but the fact is that the community doesn’t care about my opinion or anyone else’s.

There are many different subreddits. There’s programming, SaaS, and others. When I create a new repository, I’ll post it on coolgithubprojects once or twice, but no more than that.

Read the Reddit rules, don’t post links on new accounts, and post rarely, only responding to comments. This platform bans accounts for everything. For an image, for several identical texts in a row, and, of course, for a link.

𝕏 Twitter

The most effective method of all existing ones. Apart from money, this is the only platform for real marketing of your project. On par with YouTube, and even better.

// Detect dark theme var iframe = document.getElementById('tweet-2111806952439443772-566'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2111806952439443772&theme=dark" }

But there’s the most obvious drawback: you have to spend a lot, a lot of free time on it. Literally every day, you have to sit and come up with something. If you don’t have the money to pay bloggers $500 per post, then there’s no other option on this platform.

💻 HackerNews

There’s a tiny, literally negligible chance that your app, which no one needs, might suddenly catch other people’s interest, get a lot of upvotes and comments, and you’ll end up with hundreds of stars on the repository.

But it seems to me that the chances of winning the jackpot in a lottery are greater than the chances of getting more than 100 upvotes with AI. This is impossible, mainly without money or people who have a karma of around 100,000.

✒️ Articles

If you’re reading (dev to, daily dev, etc.) this, it means you’ve been drawn into promoting my project platform through the article. I don’t think there’s anything to explain within the article itself.

I can only say from my own experience that if you don’t like writing articles, don’t even try. A seemingly simple article can take from several hours to several days. No one is saying you’ll be writing a book, but it’s a very challenging format, and because people don’t like reading, your efforts will be in vain. It’s much easier to watch a silly video on TikTok, so keep that in mind.

✅️ Conclusion

Marketing is not coding. It doesn’t have algorithms (which we’re discussing), and it doesn’t have any secrets. It’s about daily work and the talent of understanding your audience. You can make 100 posts on Reddit and get 10 stars, or you can upload a video to TikTok or YouTube Shorts and get 1,000 stars in a day. No one knows what will work, but if you don’t create content, your product will remain unwanted by anyone.

If you’ve read this far, I hope I’ve helped you in some way.

Also, I want you to give it a star. I didn’t write this for nothing :)

💎 Star the EmbedCatalog repository ☆

Thanks for your attention! ❤️

Cool article

Anthony Max — Tue, 13 Oct 2026 16:34:35 +0000

Anthony Max

Oct 13

Best Enterprise AI Gateways

#ai #webdev #programming #opensource

121

15 min read

[Boost]

Anthony Max — Tue, 13 Oct 2026 07:45:34 +0000

Anthony Max

Oct 13

Best Enterprise AI Gateways

#ai #webdev #programming #opensource

121

15 min read

Best Enterprise AI Gateways

Anthony Max — Tue, 13 Oct 2026 06:49:33 +0000

TL;DR

An enterprise AI gateway sits between your applications (and, increasingly, your agents and employees) and every model provider. It handles routing, failover, budgets, access control, guardrails and audit logs in one place, so you stop rebuilding them in every codebase.
Judge an enterprise AI gateway on five things: overhead under real load, governance depth, where it can be deployed, MCP support, and how much of your actual AI traffic it can see.
This guide compares the five leading enterprise AI gateways: Bifrost, Kong AI Gateway, Azure API Management, Portkey and LiteLLM.
Bifrost is the strongest all-round enterprise AI gateway for teams that want a dedicated, self-hostable gateway with MCP governance and, through Bifrost Edge, coverage of AI tools running on employee laptops. The other four each win in a narrower situation, covered below.
Two things changed this year and belong in any procurement conversation: Palo Alto Networks acquired Portkey, and LiteLLM had a PyPI supply chain incident in March.

A couple of years ago, a gateway conversation meant one thing: stop pasting provider keys into config files and get a single invoice. That's still part of the job. But platform teams now get harder questions. Which MCP servers can our agents reach? Who approved them? Why did one team's spend triple in a week? And what is the sales team pasting into the desktop chat app nobody configured?

An enterprise AI gateway is where those answers live, or at least where they should live. The catch is that the category has stretched. Some products are purpose-built LLM gateways, some are classic API gateways with AI policies added on top, and one just got bought by a security vendor. Picking between them on a feature checklist gets you a spreadsheet full of green ticks and no real answer.

Disclosure: Bifrost is built by Maxim AI, which also publishes this blog. Weigh the ranking with that in mind. Each section below says plainly where the alternatives are the better call.

What an enterprise AI gateway actually does

At its core, an AI gateway is a reverse proxy for model traffic. Your apps talk to one OpenAI-compatible endpoint, and the gateway translates, routes and retries across providers like OpenAI, Anthropic, AWS Bedrock and Google Vertex AI.

What turns an AI gateway into an enterprise AI gateway is everything around that proxy: identity (SSO and roles), budgets per team or customer, audit trails you can hand to a compliance officer, deployment inside your own network, and policy that applies the same way whether the caller is a service, an autonomous agent or a person. If you want the longer version of that argument, the LLM Gateway Buyer's Guide walks through the capability matrix in detail.

What to check before you shortlist an enterprise AI gateway

Overhead under load. Every request passes through the gateway, so its processing time is added to every model call. It's invisible on a single chat completion and very visible when an agent chains forty calls.
Governance depth. Per-team budgets, rate limits, role-based access and audit logs should be enforced at the infrastructure layer, not left to each app team's good intentions.
Where it runs. Regulated teams often need in-VPC, on-prem or air-gapped deployment. A gateway that only exists as a hosted service is a non-starter for them.
MCP. If agents call tools, the gateway needs to control which servers and tools they can reach, and with whose credentials.
Guardrails. PII redaction, secrets detection and prompt injection checks need to run before data reaches a model, with a clear choice between block, redact and log.
Coverage. A gateway only governs traffic that reaches it. Ask what happens to the AI tools people install themselves.

Enterprise AI gateways at a glance

If you only have a minute, this is the table to read. Bifrost is the go-to enterprise AI gateway because it covers model calls, MCP tool calls and agent traffic in one gateway, runs entirely inside your own infrastructure, and is the only one here that also reaches the AI tools running on employee laptops, through Bifrost Edge.

Capability	Bifrost (our pick)	Kong AI Gateway	Azure API Management	Portkey	LiteLLM
Form factor	Standalone gateway in Go, single binary	Plugins on Kong Gateway or Konnect	Policies on Azure API Management	Gateway plus hosted platform	Python SDK and proxy
Model coverage	10,000+ models, 25+ providers	Major hosted and self-hosted providers	Azure OpenAI plus other registered backends	1,600+ models	100+ providers
Failover and balancing	Weighted routing, retries, fallbacks, adaptive balancing and circuit breakers (Enterprise)	Load balancing and semantic routing	Backend load balancing	Retries, fallbacks, load balancing	Routing and fallbacks
Semantic caching	Yes (Redis, Qdrant, Weaviate, Pinecone)	Yes (plugin)	Yes (policies with Redis)	Select Enterprise plans	Yes (Qdrant, Valkey, Redis)
MCP governance	Native MCP gateway, tool filtering, per-user OAuth, Code Mode (up to 92.8% fewer input tokens)	AI MCP Proxy and OAuth2 plugins	MCP servers inside API Management	MCP Gateway	MCP gateway
Guardrails	Content safety, PII and secrets detection, plus AWS Bedrock Guardrails, Azure Content Safety, Google Model Armor, CrowdStrike AIDR and more (Enterprise)	Prompt guard plugins	Content safety policy	Built-in guardrails with PII detection	Integrations, some need a license
AI on laptops (desktop, browser, coding agents)	Yes, with Bifrost Edge	Routed traffic only	Routed traffic only	Routed traffic only	Routed traffic only
Our verdict	Best all-round enterprise choice	Best if Kong already runs your APIs	Best for Azure-only estates	Best managed option, especially for Palo Alto shops	Best for Python-first teams

The sections below take each gateway in turn, starting with the one we'd pick.

1. Bifrost

Bifrost is an open-source (Apache 2.0) enterprise AI gateway written in Go. It gives you one OpenAI-compatible interface to 10,000+ models across 25+ providers, and the drop-in setup means moving an existing app over is a single base URL change. It ships as one binary, so it runs on a VM, Kubernetes, private cloud or a fully disconnected network, and most teams are live the same day.

Speed is the headline. Bifrost's published benchmarks show about 11 microseconds of overhead at 5,000 requests per second on a single t3.xlarge, with a 100% success rate. In the head-to-head run on AWS EC2 (500 concurrent users, 60-second runs), Bifrost sustained 424 requests per second against 44.84 for LiteLLM, with a P99 latency of 1.68 seconds versus 90.72 seconds.

On the governance side, virtual keys carry budgets and rate limits that can be set per key, team, customer or model. In the Enterprise tier, governance runs off the identity provider you already use (Okta, Microsoft Entra, Keycloak, Zitadel, Auth0, Google Workspace or any OIDC provider), with SCIM 2.0 directory sync, fine-grained RBAC and data access control, and reusable access profiles so platform teams set the rules once. Routing combines weighted load balancing, retries with backoff and automatic fallbacks to another provider, and semantic caching serves stored answers to similar prompts, backed by Redis, Qdrant, Weaviate or Pinecone.

// Detect dark theme var iframe = document.getElementById('tweet-2024540241844224013-932'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2024540241844224013&theme=dark" }

The MCP gateway is where it pulls ahead of most of this list. It governs which MCP servers and tools agents can use, with filtering at the client, request and virtual key level, and per-user OAuth so each person connects under their own account. Its Code Mode has the model write a short script to orchestrate tools instead of loading every tool definition into context, which cuts input token use by up to 92.8% across large tool sets. In a 508-tool run, Code Mode took the cost from $377 to $29 with a 100% task pass rate. The full numbers are in the MCP gateway cost and access control write-up.

Then there's Bifrost Edge, which the other gateways here simply don't have an answer for. Edge runs on macOS, Windows and Linux and routes traffic from desktop chat apps, browser AI, terminal coding agents and the MCP servers they connect to through your Bifrost gateway, after a one-time SSO sign-in. Only AI traffic is routed, and nothing else on the device is inspected. The same virtual keys, budgets and guardrails apply to tools nobody wired up on purpose, with content safety, PII and secrets detection on requests and responses, plus connections to AWS Bedrock Guardrails, Azure Content Safety, Google Model Armor, CrowdStrike AIDR, Check Point Guard and others. Admins can allow or block apps and MCP servers centrally, and a denial is enforced on the device, not just flagged in a dashboard. Rollout goes through the device management you already run: Jamf, Intune, Kandji, Omnissa Workspace ONE or JumpCloud.

For regulated teams, the enterprise tier adds clustering (a peer-to-peer cluster with no leader, so an outage one node sees is acted on by all of them), adaptive load balancing with circuit breakers, signed audit logs that can't be altered, secret management through AWS Secrets Manager, GCP Secret Manager or HashiCorp Vault, and in-VPC, on-prem or air-gapped deployment where no prompts or responses leave your infrastructure. Bifrost is SOC 2 Type II, ISO 27001, HIPAA and GDPR compliant, and multiple Fortune 500 companies run it in production across financial services, healthcare, technology, pharmaceuticals and defense.

Where to be careful: the enterprise depth (identity-based governance, clustering, adaptive load balancing, guardrail integrations, vault integrations and in-VPC deployment) sits in the paid Enterprise tier, not the open-source core. Edge requires the Enterprise gateway and is priced per endpoint, so budget for it separately, and it's still offered through early access, so pilot it on a subset of devices before a fleet-wide push.

Best for: Bifrost is built for enterprises running mission-critical AI workloads that require best-in-class performance, scalability, and reliability. It serves as a centralized AI gateway to route, govern, and secure all AI traffic across models and environments with ultra low latency. Bifrost unifies LLM gateway, MCP gateway, and Agents gateway capabilities into a single platform.

Designed for regulated industries and strict enterprise requirements, it supports air-gapped deployments, VPC isolation, and on-prem infrastructure. It provides full control over data, access, and execution, along with robust security, policy enforcement, and governance capabilities.

2. Kong AI Gateway

Kong AI Gateway is what happens when a mature API gateway learns about LLMs. If Kong already fronts your APIs, the pitch writes itself: same control plane, same plugin model, same team on call. The AI layer adds provider routing, token-based rate limiting, prompt guard plugins, semantic caching and semantic routing across models.

MCP arrived with Gateway 3.12 in October 2025. The AI MCP Proxy plugin can expose existing REST APIs as MCP tools or sit in front of an MCP server, with ACLs applied to both tool discovery and tool invocation, and a companion OAuth2 plugin handles authentication for MCP traffic.

The trade-offs are practical ones. Kong's docs list the MCP plugins as part of its AI Gateway Enterprise offering, and the OAuth2 plugin was still labelled tech preview there at the time of writing. And if Kong isn't already in your stack, standing up a full API gateway platform just for LLM traffic is a lot of machinery for the job.

Best for: organizations already standardized on Kong that want AI traffic on the control plane they already run.

3. Azure API Management

For Azure-first companies this is the path of least resistance. Microsoft's AI gateway capabilities aren't a separate product, they're a set of policies on top of API Management. A token limit policy enforces per-consumer quotas (and can count prompt tokens up front so oversized requests never reach the backend), a metrics policy sends token usage to Application Insights, and semantic caching runs on Azure Managed Redis or another RediSearch-compatible cache. LLM backends can be load balanced, and Entra ID handles authentication.

MCP support is real too. You can expose APIs as MCP servers and, per the June 2026 release notes, package them into products to control who gets what.

The costs are portability and tier. Capabilities vary by API Management service tier, everything is configured through policy XML, and the whole design assumes you live in Azure. Multi-cloud or on-prem estates will feel the friction.

Best for: Azure-centric enterprises with an existing API Management investment and security teams that already trust Entra and Azure Monitor.

4. Portkey (now Prisma AIRS AI Gateway)

Portkey made its name as the developer-friendly managed gateway: one API for 1,600+ models, retries, fallbacks, load balancing, guardrails with PII detection, budgets, RBAC and detailed logs. It added an MCP Gateway in January 2026, the core gateway is open source, and deployment options run from SaaS to private cloud, VPC and fully self-hosted. Enterprise plans bring SAML 2.0 SSO and audit logs.

The big news is ownership. Palo Alto Networks announced it would acquire Portkey and closed the deal on May 29, 2026, and the product now appears as the Prisma AIRS AI Gateway. For existing Palo Alto customers that's a neat consolidation, with the gateway sitting next to runtime security and identity tooling. For everyone else it's a fair moment to ask about roadmap and packaging, even though Palo Alto said existing and new Portkey customers will continue to be supported. One more detail: per Portkey's caching docs, semantic caching is limited to select Enterprise plans.

Best for: teams that want a managed, quick-to-adopt gateway, especially those already in the Palo Alto Networks ecosystem.

5. LiteLLM

LiteLLM is where a lot of teams start, for good reason. It's an MIT-licensed Python SDK plus a self-hosted proxy with 100+ providers, virtual keys, budgets, routing and fallbacks, guardrails and an MCP gateway. Semantic caching works with Qdrant, Valkey or Redis, though its docs recommend keeping it to single-shot prompts and away from multi-turn agent traffic. A paid enterprise license adds SCIM, audit logs and some guardrail integrations, and Admin UI SSO is free for up to five users.

Two things to weigh. First, performance: in Bifrost's published benchmarks (AWS EC2, 500 concurrent users, 60-second runs), LiteLLM sustained 44.84 requests per second with a P99 latency of 90.72 seconds, against 424 requests per second and 1.68 seconds for Bifrost. Second, supply chain hygiene. On March 24, 2026, versions 1.82.7 and 1.82.8 were published to PyPI carrying credential-stealing malware after attackers got hold of publishing credentials through a compromised CI/CD pipeline, as NetSPI's write-up describes. The bad versions were pulled. It wasn't a flaw in how the gateway works, and it could have hit any popular package. But a gateway holds every provider key you own, so pin versions and mirror packages internally whichever product you choose.

Best for: Python-heavy teams with moderate traffic that value ecosystem breadth and have the capacity to run and harden a self-hosted service.

Also worth a look

Cloudflare AI Gateway is a solid managed option for teams already on Cloudflare, with caching, rate limiting, dynamic routing, guardrails and DLP integration. It runs on Cloudflare's network, though, so it's not an option if you need self-hosted or in-VPC. OpenRouter suits individual developers and small teams who want broad model access under one bill better than it suits governance-heavy enterprise rollouts.

Enterprise controls and deployment compared

This second table covers what security reviews and procurement tend to ask about when they evaluate an enterprise AI gateway: identity, audit, where the gateway runs and how it's licensed. It's where Bifrost's open-source core with an enterprise tier on top makes the most difference for regulated buyers.

Control	Bifrost (our pick)	Kong AI Gateway	Azure API Management	Portkey	LiteLLM
SSO and roles	Okta, Entra, Keycloak, Zitadel, Auth0, Google Workspace or any OIDC provider, with SCIM 2.0, RBAC and data access control (Enterprise)	Kong's existing consumer and identity model	Entra ID with Azure RBAC	SAML 2.0 SSO and RBAC (Enterprise)	Admin UI SSO free for 5 users, license beyond; SCIM with license
Audit trail	Signed, tamper-proof audit logs with export to S3, GCS and BigQuery (Enterprise)	Kong logging plugins and Konnect analytics	Azure Monitor and Application Insights	Access rules and audit logs (Enterprise)	Audit logs with license
Where it runs	Single binary: self-hosted, in-VPC, on-prem, air-gapped, clustered	Self-hosted, hybrid or Konnect	Azure	SaaS, private cloud, VPC or self-hosted	Self-hosted (Docker, Kubernetes)
Licensing	Apache 2.0 core plus Enterprise tier (14-day free trial); Edge priced per endpoint	Open-source core, advanced AI plugins (including MCP) need a license	Azure service, capabilities vary by tier	Open-source gateway plus enterprise plans	MIT plus enterprise license key
Main trade-off	Depth sits in the paid tier; Edge needs Enterprise and is in early access	Needs Kong in place; MCP OAuth2 in tech preview	Azure-centric, policy XML	Roadmap now tied to Palo Alto	Python overhead and supply chain hygiene

Which AI Gateway Is Right for Your Team

For teams running enterprise AI workloads with multi-provider routing, agent workflows, regulated industry compliance, or high-throughput applications, the open-source Bifrost enterprise AI gateway addresses requirements that Python-based and lighter managed gateways tend to struggle with at scale. The 11 µs overhead, native MCP gateway, hierarchical virtual key governance, and support for air-gapped and VPC deployments make it the strongest production fit where those factors matter.

LiteLLM fits teams with Python-centric stacks and moderate traffic volumes that prioritize ecosystem familiarity. Kong AI Gateway fits organizations that have standardized on Kong for general API management. Azure API Management fits Azure-first estates where API Management is already approved, and Portkey fits teams that want a managed gateway, especially those already in the Palo Alto Networks ecosystem. Cloudflare AI Gateway and OpenRouter fit teams with simpler requirements: managed infrastructure, lighter governance needs, or early-stage workloads that have not yet encountered the scale at which a self-hosted, compiled-language gateway becomes necessary. Teams comparing options across this range will find the LLM Gateway Buyer's Guide a useful reference before finalizing a decision.

Frequently asked questions

What is an enterprise AI gateway?

It's a control layer between AI consumers (applications, agents and employees) and model providers. It routes and retries requests across models, enforces budgets and access rules, applies guardrails, and records everything for audit, usually through one OpenAI-compatible endpoint.

How is an enterprise AI gateway different from an API gateway?

A classic API gateway counts requests and checks auth. An AI gateway understands tokens, models and prompts: token-based budgets, failover across model vendors, semantic caching keyed on meaning, and per-model cost attribution. The line is blurrier than it used to be, since Kong and Azure API Management have added AI policies on top of their API gateways, while products like Bifrost treat those capabilities as native.

Should we self-host or use a managed enterprise AI gateway?

Self-hosting keeps prompts, keys and logs inside your own boundary and gives you control, in exchange for running the service. Managed options start faster but send traffic through a vendor's network. Regulated and high-volume teams usually end up self-hosted or in-VPC.

Do enterprise AI gateways support MCP?

All five now offer something. Bifrost has a native MCP gateway, Kong uses licensed plugins, Azure API Management exposes and packages MCP servers, Portkey has an MCP Gateway, and LiteLLM added one to its proxy. The differences are in authentication, tool-level access control and token cost, so test those rather than ticking a box.

How much latency does an enterprise AI gateway add?

It depends on the runtime and which features are switched on. Bifrost adds about 11 microseconds of overhead at 5,000 RPS on a single t3.xlarge, as shown in its published benchmarks. Against model latency measured in seconds, that's noise for a single call, but overhead compounds when agents chain many calls.

Can we move off LiteLLM without rewriting code?

Usually yes, since both expose OpenAI-compatible APIs. Bifrost ships drop-in adapters for the LiteLLM SDK, and teams on a LiteLLM proxy can migrate models, keys, teams and users across automatically. The LiteLLM migration guide covers the cutover.

Can an enterprise AI gateway govern employees using ChatGPT or Claude desktop apps?

Only if that traffic reaches it. Standard gateways cover what developers deliberately route through them. Bifrost Edge closes that gap by routing desktop, browser and coding-agent traffic through the gateway so existing policies apply.

Is Bifrost open source, and what does the Enterprise tier add?

The core is open source under Apache 2.0 and free to self-host. The Enterprise tier adds high availability, identity-based governance, guardrail integrations, signed audit logs, vault integrations and in-VPC or air-gapped deployment, with a free 14-day trial. Bifrost Edge requires the Enterprise gateway and is priced per endpoint.

Conclusion

The five gateways in this comparison cover different points on the tradeoff curve between operational simplicity and production depth. For teams where AI is a core infrastructure concern, Bifrost provides the performance baseline, governance model, and enterprise deployment options that production requirements demand. To see how Bifrost fits your infrastructure, book a demo with the Bifrost team.

Resources:

Bifrost GitHub: https://github.com/maximhq/bifrost
Bifrost Docs: https://docs.getbifrost.ai
Bifrost CLI: npx -y @maximhq/bifrost-cli

Thanks for reading this article! ❤️

I'd love to hear your thoughts on this mode in the comments!

💎 Check out the Bifrost repository ☆

Must read

Anthony Max — Sun, 04 Oct 2026 10:08:43 +0000

Includes video and contributor perks

Anthony Max

Oct 3

EmbedCatalog is participating in Hacktoberfest 2026

#webdev #javascript #programming #opensource

145

2 min read

EmbedCatalog is participating in Hacktoberfest 2026

Anthony Max — Sat, 03 Oct 2026 21:53:00 +0000

Hi everyone! A little late, perhaps, but in this article, I’d like to share that my new project is participating in Hacktoberfest.

I have been participating with projects for three years now, and this year was no exception. Today, I would like to talk about this in more detail.

Well, let's get started! 🏎

🔎 About Hacktoberfest

First of all, I would like to briefly explain what kind of festival this is. Hacktoberfest is a month-long celebration of open source throughout October. It’s free, and it’s for everyone, wherever you are in your open-source AI or software creating journey.

It is worth noting that this year, in-person events are also being held in many cities around the world.

💎 My new project

It is called EmbedCatalog, and its purpose is to create an embed that you can add to your website, README, and so on.

Over the course of a month, the core functionality required for the catalog was implemented. Features such as user registration, project statistics, custom embeds, and much more were added.

🎥 Video announcement

I had the idea to create something new for this festival, so I recorded a short video that you can watch here:

// Detect dark theme var iframe = document.getElementById('tweet-2106363636063392079-497'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2106363636063392079&theme=dark" }

The text was written in a couple of minutes, but it turned out pretty well, I think :)

👀 What else was done?

I also created a small page where I listed all the issues in detail and added the contributors' avatars.

I’ve created 10 issues that you can check out and help with by writing code.

💬 Feedback

I would really like to hear your opinion on this festival. It seems like a pretty interesting format - a chance for people to work on open-source projects and make new connections within the developer community.

🔗 Links

Repository: https://github.com/embedcatalog/embedcatalog
Issues: https://github.com/embedcatalog/embedcatalog/issues

Thank you all very much for reading the article! ❤️

Also, if you liked the project, I would appreciate it if you starred the repository.

[Boost]

Anthony Max — Sun, 27 Sep 2026 06:46:59 +0000

Anthony Max

Sep 24

Qodo vs CodeRabbit vs Cubic: Which One Enforces Rules Instead of Just Suggesting Them? 🔥

#ai #webdev #programming #opensource

10 min read

[Boost]

Anthony Max — Fri, 25 Sep 2026 14:09:27 +0000

Anthony Max

Sep 24

Qodo vs CodeRabbit vs Cubic: Which One Enforces Rules Instead of Just Suggesting Them? 🔥

#ai #webdev #programming #opensource

10 min read