Airia enables enterprises to track AI spending, optimize costs, and enforce governance at runtime ATLANTA, Nov. 02, 2026 (GLOBE NEWSWIRE) -- Airia, an enterprise AI security and governance platform, today announced that it has been named a Market Shaper as of October 2026 in the Gartner® Emerging Market Quadrant for AI FinOps Tools – Startup Vendors. As enterprises deploy more AI models and autonomous agents, unpredictable token consumption, variable model pricing, and increasingly complex workflows make costs harder to manage.

Cut Python container sizes by 80%+ automatically using AST dependency segregation & verified multi-stage Dockerfiles. Docker & Rancher Desktop Extension with FinOps.

CloudKeeper, an AI and cloud cost optimization partner, has announced the launch of Full Stack FinOps for AI, a platform designed to help businesses track AI-related expenditure across models, applications,…

The finance lead forwards you the model provider invoice with one line highlighted. Spend tripled in a month. Revenue did not. When you dig in, a single tenant on the starter plan is running a nightly script that sends the same 40-page contract through your "summarize" endpoint 600 times, every request goes to the most expensive model because that was the default in the SDK call, and nothing in your stack knew which tenant any of those tokens belonged to. I wrote this with help from AI tools and checked every config, code path and claim myself. I run Loop Origin, where we build custom web applications for B2B SaaS teams, and this is the final part of the series: the gateway layer that turns LLM calls from an unbounded cost into a metered, budgeted product feature. > **📚 Series Roadmap — Production AI & SaaS Systems Engineering** > > * **Part 1:** Building Production MCP Servers & Tool-Calling AI Agents: Sandboxing, Schemas & Approval Gates > * **Part 2:** Zero-Downtime Database Migrations in Laravel & Node.js SaaS: Expand-Contract & Lock Timeouts > * **Part 3:** Multi-Tier Caching & Stampede Protection with Redis/Valkey: Singleflight Locks & SWR > * **Part 4:** SaaS Usage-Based Billing & Metering Architecture: Event Ingestion, Rollups & Stripe Invoicing > * **Part 5:** Production Observability on a Budget: OpenTelemetry Traces, Structured Logs & Tail Sampling > * **Part 6 (This Post):** LLM Gateway & FinOps Architecture for B2B SaaS: Semantic Caching, Routing & Tenant Quotas > ## What is an LLM gateway and how does it control AI costs in a B2B SaaS? **Direct answer:** An LLM gateway is one internal service that every model call passes through. It identifies the tenant and feature, checks a token budget before the call, serves exact or semantic cache hits, routes each task to the cheapest model that meets its quality bar, and records actual token usage afterwards so costs land on the right tenant's ledger. ## Why per-service SDK calls stop scaling In most SaaS codebases, the first AI feature calls the provider SDK directly from a controller. The second feature copies that code. By the fifth, you have five places that pick a model, five retry policies, no shared cache, and API keys in three services. Nobody can answer the two questions finance will ask: _which tenant spent this_ and _what would it cost on a cheaper model_. A gateway centralizes four decisions that should never be made inside product code: 1. **Who is paying?** Tenant, plan and feature on every request. 2. **May they spend this?** Budget and rate checks before tokens are bought. 3. **Have we answered this before?** Exact and semantic caching. 4. **Which model is good enough?** Routing by task class, with fallbacks. ## The architecture Laravel API ──┐ Node workers ─┼─► LLM Gateway ──► quota check (Redis) ──► cache (exact → semantic) MCP agents ──┘ │ │ miss │ ▼ │ router: task class → model tier │ │ └──── usage events (Part 4 meter) ◄── provider A / B / self-hosted The gateway can be an open-source proxy you configure or a small service you write. Either way, product code sends a _task_ , not a model name: `{ tenant_id, feature: "contract_summary", input }`. ## Step 1: Budgets before tokens Reserve an estimated token cost before the call, then settle the real number afterwards. A Lua script keeps the check-and-reserve atomic in Redis or Valkey: -- KEYS[1] = budget:{tenant}:{yyyy-mm} ARGV[1] = estimate ARGV[2] = monthly limit local used = tonumber(redis.call('GET', KEYS[1]) or '0') if used + tonumber(ARGV[1]) > tonumber(ARGV[2]) then return -1 end redis.call('INCRBY', KEYS[1], ARGV[1]) redis.call('EXPIRE', KEYS[1], 40 * 24 * 3600) return used + tonumber(ARGV[1]) // gateway/budget.ts export async function reserve(tenant: Tenant, estimate: number) { const key = `budget:${tenant.id}:${monthKey()}`; const res = await redis.eval(RESERVE_LUA, 1, key, estimate, tenant.plan.monthlyTokens); if (res === -1) throw new QuotaExceeded(tenant.id, tenant.plan.name); } export async function settle(tenant: Tenant, estimate: number, actual: number) { await redis.incrby(`budget:${tenant.id}:${monthKey()}`, actual - estimate); await meter.emit({ tenant: tenant.id, metric: "llm_tokens", value: actual }); // Part 4 pipeline } Estimate input tokens with the provider's tokenizer and add the request's `max_tokens` cap for output. Overestimating slightly is fine because settlement corrects it; what matters is that the 600-request nightly loop hits a wall instead of the invoice. ## Step 2: Cache in two layers **Exact cache** first: hash the normalized request (model tier, system prompt version, input, temperature) and look it up. It is cheap and has zero quality risk. In my experience it catches more than teams expect, because batch jobs and retries resend identical inputs. **Semantic cache** second: embed the input, search a vector index scoped to the same tenant and feature, and return the stored answer only above a strict similarity threshold. const hit = await vectors.search({ namespace: `${tenant.id}:${feature}:${promptVersion}`, vector: await embed(input), topK: 1, }); if (hit && hit.score >= FEATURE_THRESHOLDS[feature]) return hit.payload; // e.g. 0.95 Three rules keep semantic caching from becoming a data-leak or wrong-answer machine: * **Always namespace by tenant.** A cross-tenant cache hit returns one customer's data to another. * **Include the prompt version.** Changing the system prompt must invalidate old answers. * **Only cache stable tasks.** FAQ answers and classification, yes. Anything personalized or time-sensitive, no. _AI-generated image._ ## Step 3: Route by task class, not by habit Most SaaS AI features are not equally hard. Classify them once and map each class to a tier: Task class | Example feature | Model tier | Fallback | Cacheable ---|---|---|---|--- Extraction / classification | Tag a support ticket | Small, fast model | Second small model | Exact + semantic Summarization | Contract summary | Mid-tier model | Small model with shorter output | Exact Reasoning / generation | Draft a proposal | Frontier model | Mid-tier model + warning flag | Rarely Agentic tool use (MCP) | Multi-step workflow | Frontier model with tool support | Halt and ask human | No Embeddings | Search, semantic cache | Embedding model | Queue and retry | N/A # gateway/routes.yaml contract_summary: primary: { provider: provider_a, model: mid-tier, max_tokens: 800 } fallback: { provider: provider_b, model: small, max_tokens: 500 } timeout_ms: 20000 plans: starter: { tier_override: small } ticket_classify: primary: { provider: self_hosted, model: small } fallback: { provider: provider_a, model: small } The `plans` override is where FinOps meets pricing: starter tenants get the cheaper tier, enterprise tenants get the better one, and the difference is visible on your pricing page instead of hidden in your margin. Validate a cheaper route before switching: replay a sample of real (anonymized) inputs through both tiers and score the outputs with your own rubric. Route changes without an eval are guesses. ## Step 4: Fallbacks that do not double the bill Retries are where LLM costs quietly multiply. As Part 3 covered for caches, a retry storm turns one outage into many. In the gateway: * Retry only on timeouts and 429/5xx, never on 4xx validation errors. * Use a circuit breaker per provider so a failing provider is skipped for a cool-down window. * Count tokens from _every_ attempt against the tenant budget. A failed call that streamed half an answer still cost money. ## Step 5: Make cost observable Reuse the OpenTelemetry setup from Part 5. Each gateway call becomes a span with low-cardinality attributes: span.setAttributes({ "llm.feature": feature, "llm.route": route.name, "llm.model_tier": route.tier, "llm.cache": cacheResult, // "exact" | "semantic" | "miss" "llm.tokens.input": usage.input, "llm.tokens.output": usage.output, "tenant.plan": tenant.plan.name, // plan, not tenant ID, to keep metrics cheap }); Generate metrics from these spans for cost per feature, cache hit rate and fallback rate. Keep the tenant ID in the usage events and the billing ledger, not in metric labels, so dashboards stay affordable. ## Before and after Metric | Direct SDK calls | Behind the gateway ---|---|--- Who spent it | Unknown | Tenant + feature on every call Runaway script | Runs all night | Blocked at the monthly budget Repeated inputs | Paid every time | Exact cache hit Model choice | Whatever the SDK default was | Per task class and plan Provider outage | Feature down | Fallback route + circuit breaker Billing | Absorbed in margin | Metered and invoiceable (Part 4) ## Diagnostic checklist 1. Does every model call in the codebase go through the gateway? Search for direct SDK imports. 2. Does each request carry tenant, plan and feature? 3. Is the budget check atomic and run _before_ the provider call? 4. Are tokens from failed and retried attempts counted? 5. Is the semantic cache namespaced by tenant, feature and prompt version? 6. Does every route have a timeout, a fallback and a circuit breaker? 7. Was each cheaper route validated with an eval on real inputs? 8. Do usage events reach the metering pipeline and reconcile with the provider invoice? 9. Are prompts and outputs kept out of span attributes and logs? 10. Is there an alert when any tenant passes 80% of budget? ## FAQ ### Should I build a gateway or use an open-source one? Start with an existing open-source LLM proxy if it supports your providers, budgets and logging. Write your own thin service when you need tenant-aware quotas tied to your billing data, which most B2B SaaS products eventually do. ### Does a gateway add latency? A budget check in Redis and an exact cache lookup are usually a few milliseconds. Semantic cache adds an embedding call, so only enable it for features where hit rates justify it. ### What similarity threshold should semantic caching use? There is no universal number. Start strict, around 0.95 cosine similarity, log near-misses, and review them by hand before loosening it per feature. ### How do I estimate tokens before the call? Count input tokens with the provider's tokenizer library and add the `max_tokens` cap for output. Settle with the real usage numbers the provider returns. ### Can I pass LLM costs through to customers? Yes, if your metering is accurate. Emit settled token usage into the same pipeline as your other usage events and price it per plan, with clear limits shown in the product. ### How does this fit with MCP agents from Part 1? Agents make many calls per task, so they need the gateway most. Give each agent run its own budget reservation and stop the run, with a human approval gate, when it exceeds it. That closes Production AI & SaaS Systems Engineering. Across six parts we sandboxed MCP agents, migrated databases without downtime, protected caches from stampedes, metered usage for billing, made traces affordable and, here, put a budget and a router in front of every model call. The common thread: put one deliberate control point in front of every expensive or risky operation, measure what passes through it, and let product code ask for outcomes instead of hard-coding infrastructure choices. Questions about your own setup are welcome in the comments. ### Follow & Connect * **Website:** Loop Origin * **LinkedIn:** Loop Origin * **X:** Muhammad Ali Mughal (@AliMughal887) * **GitHub:** Loop Origin (@LoopOrigin)

A few months ago I was performing a cloud cost analysis for a project. One of the things that stood...

Learn how a missing ec2:CreateTags permission in AWS EC2 Image Builder left EBS snapshots untagged, blocked lifecycle cleanup, and drove up cloud costs.

This post presents a five-stage operating model for managing Amazon Bedrock cost: attribute every dollar to an owner, make every request observable, enforce budgets, optimize platform defaults, and operate a review cadence. Written for FinOps and platform engineering teams.

Learn how to use AWS Cost Explorer to analyse cloud spending, identify cost drivers, understand usage patterns, forecast expenses, and make informed decisions to optimise AWS costs.

Amazon Bedrockのコストを端から端まで管理する「FinOpsとプラットフォームチームのためのプレイブック」を解説。コスト配分やログ監視、予算の強制停止など5つのステップを詳しく紹介します。

(TrendHunter.com) CloudKeeper is introducing unified ai cost management through Full Stack FinOps for AI, a platform that helps businesses understand spending across their artificial intelligence operations. Rather...

Pip: Welcome to Cloudy with a Chance of Licensing — where the forecast is partly cloudy with a strong chance of your Azure bill going up. Mara: rich_gibbons has been tracking some significant changes across Azure pricing, reservation policy, and Copilot Studio this episode, alongside Microsoft's latest financial results. There's a lot that touches FinOps and ITAM teams directly. Pip: