Architecting a Self-Hosted Hybrid LLM Gateway: Dynamic Cost Routing, Semantic Caching, and Usage Billing with Node.js, Ollama, and Next.js

Introduction: The Month the Token Bill Beat Payroll

A founder I worked with last year opened their cloud invoice and found that LLM tokens had quietly overtaken their engineering payroll. Nobody had shipped anything reckless. They had simply built a handful of LangGraph agents, each of which called a frontier model for every step, including steps like "classify this message as a greeting" and "extract the date from this sentence." Multiply that by a few thousand customers and the meter spins fast.

The fix was not "use AI less." The fix was putting a brain between the agents and the models: a gateway that knows what each request costs, what it actually needs, and whether someone already asked it five minutes ago.

In this guide we will build that brain. By the end you will have a blueprint for a self-hosted hybrid LLM gateway that:

• Proxies all agent traffic through Node.js and Express.js behind Nginx
• Routes dynamically between local Ollama models and cloud APIs based on query complexity and live latency
• Uses semantic vector caching in PostgreSQL (pgvector) to eliminate redundant calls
• Writes granular token telemetry to MongoDB
• Streams live cost analytics to a Next.js + TypeScript + Redux Toolkit + shadcn/ui dashboard via Socket.io
• Enforces JWT/OAuth multi-tenant rate limiting
• Bills usage through Stripe, Razorpay, and Cashfree webhooks
• Ships with zero-downtime Docker deployments on AWS Linux

This is a long one. Grab a coffee, and let's treat AI spend like the infrastructure problem it actually is.

Why a Gateway, and Why Self-Hosted?

Managed gateways exist, and they are fine for many teams. But there are three situations where building your own wins:

1. You run local models. Routing between Ollama on your own GPUs and cloud APIs requires knowing your fleet's real-time queue depth, which a third-party gateway cannot see.
2. You resell AI. If you charge customers per token, your billing ledger is a revenue system. You want it in your database, with your audit trail.
3. Data residency. Many enterprise contracts in 2026 require that prompts never leave a specific region or VPC. A self-hosted gateway makes that a configuration choice instead of a legal negotiation.

Think of the gateway as the AI FinOps control plane. FinOps for cloud compute taught us that visibility comes before optimization. The same is true for tokens: you cannot route intelligently, cache aggressively, or bill fairly until every request flows through one place that records what happened.

High-Level Architecture

Here is the shape of the system, from the outside in:

• Clients: LangGraph and LangChain agents, internal services, and customer apps. They speak the OpenAI-compatible Chat Completions API, so the gateway is a drop-in base URL change.
• Nginx: TLS termination, connection limits, request size limits, and blue-green upstream switching during deploys.
• Gateway (Node.js + Express): Authentication, rate limiting, complexity scoring, routing, semantic cache lookups, streaming proxy, and telemetry emission.
• Model backends: An Ollama pool for local models and one or more cloud providers.
• PostgreSQL + pgvector: Tenants, API keys, pricing tables, and the semantic cache.
• MongoDB: Append-only token telemetry (time-series collection).
• Redis: Rate-limit buckets, latency stats, and Socket.io pub/sub across gateway replicas.
• Billing worker: Aggregates telemetry and pushes usage to Stripe, Razorpay, or Cashfree; receives their webhooks.
• Dashboard (Next.js): Live cost and latency analytics over Socket.io.

The Life of a Single Request

It helps to walk one request through the system before writing code:

1. An agent sends POST /v1/chat/completions with a tenant JWT.
2. Nginx terminates TLS and forwards to a healthy gateway container.
3. The gateway verifies the JWT, loads the tenant's plan, and checks the token bucket in Redis.
4. The prompt is embedded locally and the semantic cache is queried. On a hit, the cached answer is returned and a zero-cost telemetry event is recorded.
5. On a miss, the router scores complexity, checks live latency for the local pool, and picks a backend.
6. The response streams back to the client while the gateway counts tokens.
7. When the stream ends, the gateway writes telemetry to MongoDB, stores the answer in the cache (if eligible), and publishes a cost event to the dashboard.

Every step is cheap. That is the whole point: the gateway should cost milliseconds and save dollars.

Step 1: Nginx as the Front Door

LLM responses stream for a long time, so the classic Nginx defaults (buffering on, 60-second timeouts) will hurt you. Here is a production-oriented config:

# /etc/nginx/conf.d/llm-gateway.conf
upstream gateway_active {
    # Swapped between blue and green ports during deploys
    include /etc/nginx/upstreams/active.conf;
    keepalive 64;
}

limit_conn_zone $binary_remote_addr zone=perip:10m;

server {
    listen 443 ssl;
    http2 on;
    server_name llm.example.com;

    ssl_certificate     /etc/letsencrypt/live/llm.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/llm.example.com/privkey.pem;

    client_max_body_size 2m;
    limit_conn perip 50;

    location /v1/ {
        proxy_pass http://gateway_active;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header X-Request-Id $request_id;
        proxy_set_header X-Real-IP $remote_addr;

        # Critical for token streaming
        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 300s;
        proxy_send_timeout 300s;
    }

    location /socket.io/ {
        proxy_pass http://gateway_active;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 3600s;
    }
}

The active.conf include is a one-line file like server 127.0.0.1:4001;. We will flip it during zero-downtime deploys later.

Step 2: The Express Gateway Skeleton

Keep the gateway lean. Every millisecond here is paid on every request.

// src/server.ts
import express from "express";
import helmet from "helmet";
import { createServer } from "node:http";
import { authenticate } from "./auth";
import { rateLimit } from "./rateLimit";
import { chatCompletions } from "./routes/chat";
import { attachRealtime } from "./realtime";

const app = express();
app.disable("x-powered-by");
app.use(helmet());
app.use(express.json({ limit: "2mb" }));

app.get("/healthz", (_req, res) => res.json({ ok: true }));

app.post("/v1/chat/completions", authenticate, rateLimit, chatCompletions);

const server = createServer(app);
attachRealtime(server);

const port = Number(process.env.PORT ?? 4001);
server.listen(port, () => console.log(`gateway listening on ${port}`));

// Graceful shutdown lets in-flight streams finish during deploys
process.on("SIGTERM", () => {
  server.close(() => process.exit(0));
  setTimeout(() => process.exit(1), 30_000).unref();
});

That SIGTERM handler matters more than it looks. Without it, every deploy chops off streaming responses mid-sentence, and your agents retry, and you pay twice.

Step 3: Multi-Tenant Authentication and Rate Limiting

JWT and OAuth

Most gateways need two kinds of callers: humans using the dashboard (OAuth via your identity provider) and machines calling the API (long-lived API keys exchanged for short-lived JWTs). Either way, by the time a request reaches the router, it should carry a verified tenantId, plan, and scopes.

// src/auth.ts
import { jwtVerify, createRemoteJWKSet } from "jose";
import type { Request, Response, NextFunction } from "express";

const JWKS = createRemoteJWKSet(new URL(process.env.OAUTH_JWKS_URL!));

export async function authenticate(req: Request, res: Response, next: NextFunction) {
  const token = req.headers.authorization?.replace(/^Bearer\s+/i, "");
  if (!token) return res.status(401).json({ error: "missing_token" });

  try {
    const { payload } = await jwtVerify(token, JWKS, {
      issuer: process.env.OAUTH_ISSUER,
      audience: "llm-gateway",
    });
    req.tenant = {
      id: String(payload.tenant_id),
      plan: String(payload.plan ?? "free"),
      scopes: String(payload.scope ?? "").split(" "),
    };
    next();
  } catch {
    res.status(401).json({ error: "invalid_token" });
  }
}

Using a remote JWKS means you can rotate signing keys at the identity provider without redeploying the gateway.

Token-Aware Rate Limiting

Request-per-minute limits are almost useless for LLMs. One request can be 50 tokens or 50,000. Rate limit on estimated tokens, with a Redis-backed token bucket so limits hold across all gateway replicas.

// src/rateLimit.ts
import Redis from "ioredis";
import type { Request, Response, NextFunction } from "express";
import { estimateTokens } from "./tokens";

const redis = new Redis(process.env.REDIS_URL!);

const PLAN_LIMITS: Record<string, { capacity: number; refillPerSec: number }> = {
  free: { capacity: 20_000, refillPerSec: 50 },
  pro: { capacity: 200_000, refillPerSec: 800 },
  enterprise: { capacity: 2_000_000, refillPerSec: 8_000 },
};

// Atomic token bucket in Lua so concurrent replicas never double-spend
const BUCKET_LUA = `
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local cost = tonumber(ARGV[4])
local state = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(state[1]) or capacity
local ts = tonumber(state[2]) or now
tokens = math.min(capacity, tokens + (now - ts) * refill)
if tokens < cost then
  redis.call('HMSET', key, 'tokens', tokens, 'ts', now)
  redis.call('EXPIRE', key, 3600)
  return -1
end
tokens = tokens - cost
redis.call('HMSET', key, 'tokens', tokens, 'ts', now)
redis.call('EXPIRE', key, 3600)
return tokens
`;

export async function rateLimit(req: Request, res: Response, next: NextFunction) {
  const limits = PLAN_LIMITS[req.tenant.plan] ?? PLAN_LIMITS.free;
  const cost = estimateTokens(req.body.messages) + (req.body.max_tokens ?? 512);
  const remaining = await redis.eval(
    BUCKET_LUA, 1, `rl:${req.tenant.id}`,
    limits.capacity, limits.refillPerSec, Date.now() / 1000, cost
  );

  if (Number(remaining) < 0) {
    res.setHeader("Retry-After", "5");
    return res.status(429).json({ error: "token_budget_exceeded" });
  }
  res.setHeader("X-Token-Budget-Remaining", String(remaining));
  next();
}

Charging max_tokens up front is deliberately pessimistic. You can refund the difference after the response completes if you want to be generous.

Step 4: Dynamic Cost Routing

This is the heart of the gateway. The question for every request: is this worth a frontier model, or will a local 8B model do?

Scoring Complexity Cheaply

You might be tempted to ask an LLM to classify each prompt's difficulty. Don't, at least not as the first pass. You would be paying a model to decide whether to pay a model. Start with fast heuristics, and only add a tiny local classifier once you have telemetry to train it on.

Signals that work well in practice:

• Prompt length in tokens (long contexts favor larger context windows)
• Tool definitions present (local models are improving at function calling, but still less reliable)
• Reasoning cues like "prove", "step by step", "analyze the trade-offs", "refactor"
• Agent node hint sent by LangGraph as a header, such as classifier, extractor, or planner
• Response format (strict JSON schema output raises the bar)
• Tenant policy (some customers pay for "always premium")

// src/router/complexity.ts
import type { ChatRequest } from "../types";

const REASONING_CUES = /\b(prove|step[- ]by[- ]step|trade-?offs?|architect|refactor|derive|debug)\b/i;
const CHEAP_NODES = new Set(["classifier", "extractor", "router", "summarizer"]);

export function scoreComplexity(req: ChatRequest, nodeHint?: string): number {
  const text = req.messages.map((m) => m.content).join("\n");
  let score = 0;

  score += Math.min(text.length / 4000, 3);            // up to 3 points for length
  if (req.tools?.length) score += 2;
  if (REASONING_CUES.test(text)) score += 2;
  if (req.response_format?.type === "json_schema") score += 1;
  if (nodeHint && CHEAP_NODES.has(nodeHint)) score -= 3;
  if (nodeHint === "planner") score += 2;

  return Math.max(0, score);
}

Latency-Aware Decisions

A local model that is cheap but stuck behind a 20-request queue is not cheap anymore; it is slow, and slow agents burn engineer time and customer patience. Track an exponentially weighted moving average (EWMA) of time-to-first-token per backend in Redis and let it override the complexity decision.

// src/router/latency.ts
import Redis from "ioredis";
const redis = new Redis(process.env.REDIS_URL!);
const ALPHA = 0.2;

export async function recordTtft(backend: string, ms: number) {
  const key = `lat:${backend}`;
  const prev = Number(await redis.get(key)) || ms;
  await redis.set(key, String(ALPHA * ms + (1 - ALPHA) * prev), "EX", 600);
}

export async function getTtft(backend: string): Promise<number> {
  return Number(await redis.get(`lat:${backend}`)) || 0;
}

The Router

// src/router/index.ts
import { scoreComplexity } from "./complexity";
import { getTtft } from "./latency";
import { breaker } from "./breaker";

export type Route = { backend: "ollama" | "cloud"; model: string; reason: string };

const LOCAL_TTFT_BUDGET_MS = Number(process.env.LOCAL_TTFT_BUDGET_MS ?? 1500);
const COMPLEXITY_THRESHOLD = Number(process.env.COMPLEXITY_THRESHOLD ?? 3);

export async function chooseRoute(req, tenant, nodeHint?: string): Promise<Route> {
  if (tenant.policy === "premium_only") {
    return { backend: "cloud", model: "gpt-premium", reason: "tenant_policy" };
  }

  const score = scoreComplexity(req, nodeHint);
  const localTtft = await getTtft("ollama");
  const localHealthy = breaker.isClosed("ollama");

  if (score < COMPLEXITY_THRESHOLD && localHealthy && localTtft < LOCAL_TTFT_BUDGET_MS) {
    return { backend: "ollama", model: "llama3.1:8b", reason: `score=${score.toFixed(1)}` };
  }

  if (!localHealthy || localTtft >= LOCAL_TTFT_BUDGET_MS) {
    return { backend: "cloud", model: "gpt-mini", reason: "local_saturated" };
  }

  return { backend: "cloud", model: "gpt-premium", reason: `score=${score.toFixed(1)}` };
}

Notice the middle tier. When the local pool is saturated, we do not jump straight to the most expensive model; we spill to a cheaper cloud model. That one branch saved more money than any other line of code in the systems I have built this way.

Always return the reason. Logging why a route was chosen is what lets you tune thresholds with evidence instead of vibes.

Circuit Breakers

If Ollama starts throwing errors (GPU out of memory is the classic), stop sending it traffic for a cool-down window. A simple breaker that opens after five consecutive failures and half-opens after 30 seconds is enough to keep one bad node from taking down your agent fleet.

Step 5: Semantic Caching with PostgreSQL and pgvector

Exact-match caches are nearly useless for natural language. "What's our refund policy?" and "how do refunds work here?" are different strings and the same question. A semantic cache embeds the prompt, finds the nearest previously answered prompt, and returns its answer if the similarity is high enough.

Schema

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE semantic_cache (
  id               BIGSERIAL PRIMARY KEY,
  tenant_id        TEXT        NOT NULL,
  model_family     TEXT        NOT NULL,
  system_hash      TEXT        NOT NULL,
  prompt_text      TEXT        NOT NULL,
  embedding        vector(768) NOT NULL,
  response         JSONB       NOT NULL,
  prompt_tokens    INT         NOT NULL,
  completion_tokens INT        NOT NULL,
  hit_count        INT         NOT NULL DEFAULT 0,
  created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
  expires_at       TIMESTAMPTZ NOT NULL
);

CREATE INDEX semantic_cache_hnsw
  ON semantic_cache USING hnsw (embedding vector_cosine_ops);

CREATE INDEX semantic_cache_scope
  ON semantic_cache (tenant_id, model_family, system_hash);

Three scoping columns do the safety work:

• tenant_id prevents cross-customer leakage. This is non-negotiable.
• model_family stops a cheap local answer from being served to a request that was routed to a premium model.
• system_hash ensures that two agents with different system prompts never share answers, even if the user message is identical.

Embedding Locally

Do not pay a cloud API to embed every prompt; that defeats the purpose. Ollama runs embedding models like nomic-embed-text (768 dimensions) comfortably on CPU.

// src/cache/embed.ts
export async function embed(text: string): Promise<number[]> {
  const res = await fetch(`${process.env.OLLAMA_URL}/api/embed`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ model: "nomic-embed-text", input: text }),
  });
  if (!res.ok) throw new Error(`embed failed: ${res.status}`);
  const data = await res.json();
  return data.embeddings[0];
}

Lookup and Store

// src/cache/semanticCache.ts
import { Pool } from "pg";
import { createHash } from "node:crypto";
import { embed } from "./embed";

const pool = new Pool({ connectionString: process.env.DATABASE_URL });
const SIMILARITY_THRESHOLD = 0.95;

const toVector = (v: number[]) => `[${v.join(",")}]`;
export const hashSystem = (s = "") => createHash("sha256").update(s).digest("hex");

export async function lookup(tenantId: string, family: string, system: string, prompt: string) {
  const vector = await embed(prompt);
  const { rows } = await pool.query(
    `SELECT id, response, 1 - (embedding <=> $1::vector) AS similarity
       FROM semantic_cache
      WHERE tenant_id = $2 AND model_family = $3 AND system_hash = $4
        AND expires_at > now()
      ORDER BY embedding <=> $1::vector
      LIMIT 1`,
    [toVector(vector), tenantId, family, hashSystem(system)]
  );

  const best = rows[0];
  if (best && best.similarity >= SIMILARITY_THRESHOLD) {
    pool.query("UPDATE semantic_cache SET hit_count = hit_count + 1 WHERE id = $1", [best.id]);
    return { hit: true as const, response: best.response, similarity: best.similarity, vector };
  }
  return { hit: false as const, vector };
}

export async function store(entry: {
  tenantId: string; family: string; system: string; prompt: string;
  vector: number[]; response: unknown; promptTokens: number; completionTokens: number;
  ttlHours?: number;
}) {
  await pool.query(
    `INSERT INTO semantic_cache
       (tenant_id, model_family, system_hash, prompt_text, embedding, response,
        prompt_tokens, completion_tokens, expires_at)
     VALUES ($1,$2,$3,$4,$5::vector,$6,$7,$8, now() + make_interval(hours => $9))`,
    [entry.tenantId, entry.family, hashSystem(entry.system), entry.prompt,
     toVector(entry.vector), entry.response, entry.promptTokens,
     entry.completionTokens, entry.ttlHours ?? 24]
  );
}

We return the vector from lookup so we can reuse it in store and avoid embedding twice.

Picking the Threshold

There is no universal magic number. A threshold of 0.95 is a conservative starting point; FAQ-style support bots can often go lower, while code-generation workloads should go higher or skip caching entirely. The practical method: log every lookup's best similarity score for a week without serving hits, sample pairs in each score band, and have a human mark them as "same question" or "different question". Then choose the lowest band where nearly every pair is a true match.

What Not to Cache

• Requests with temperature above a small value (the caller asked for variety)
• Requests that include tool results or retrieved documents (the context changes)
• Prompts containing detected personal data
• Anything with time words like "today", "latest", or "right now"

Step 6: Streaming Proxy with Live Token Counting

Agents expect Server-Sent Events. The gateway must forward chunks as they arrive, not buffer the whole answer. Because Ollama exposes an OpenAI-compatible endpoint at /v1/chat/completions, both backends can share one streaming path.

// src/routes/chat.ts
import type { Request, Response } from "express";
import { chooseRoute } from "../router";
import { lookup, store } from "../cache/semanticCache";
import { recordTtft } from "../router/latency";
import { emitUsage } from "../telemetry";
import { BACKENDS } from "../backends";

export async function chatCompletions(req: Request, res: Response) {
  const body = req.body;
  const tenant = req.tenant;
  const system = body.messages.find((m) => m.role === "system")?.content ?? "";
  const userPrompt = body.messages.filter((m) => m.role === "user").at(-1)?.content ?? "";
  const route = await chooseRoute(body, tenant, req.header("x-agent-node"));
  const cacheable = (body.temperature ?? 0) <= 0.2 && !body.tools?.length;

  let vector: number[] | undefined;
  if (cacheable) {
    const cached = await lookup(tenant.id, route.model, system, userPrompt);
    vector = cached.vector;
    if (cached.hit) {
      await emitUsage({ tenantId: tenant.id, route, cacheHit: true, promptTokens: 0, completionTokens: 0 });
      return res.json(cached.response);
    }
  }

  const backend = BACKENDS[route.backend];
  const started = Date.now();
  const upstream = await fetch(`${backend.baseUrl}/v1/chat/completions`, {
    method: "POST",
    headers: { "Content-Type": "application/json", ...backend.headers },
    body: JSON.stringify({
      ...body,
      model: route.model,
      stream: true,
      stream_options: { include_usage: true },
    }),
  });

  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("X-Gateway-Route", `${route.backend}:${route.model}`);

  let firstChunk = true;
  let fullText = "";
  let usage = { prompt_tokens: 0, completion_tokens: 0 };
  const decoder = new TextDecoder();

  for await (const chunk of upstream.body as any) {
    if (firstChunk) {
      firstChunk = false;
      recordTtft(route.backend, Date.now() - started);
    }
    res.write(chunk);

    for (const line of decoder.decode(chunk, { stream: true }).split("\n")) {
      if (!line.startsWith("data: ") || line.includes("[DONE]")) continue;
      try {
        const evt = JSON.parse(line.slice(6));
        fullText += evt.choices?.[0]?.delta?.content ?? "";
        if (evt.usage) usage = evt.usage;
      } catch { /* partial line, ignore */ }
    }
  }
  res.end();

  await emitUsage({
    tenantId: tenant.id, route, cacheHit: false,
    promptTokens: usage.prompt_tokens, completionTokens: usage.completion_tokens,
    latencyMs: Date.now() - started,
  });

  if (cacheable && vector && fullText) {
    store({
      tenantId: tenant.id, family: route.model, system, prompt: userPrompt, vector,
      response: { choices: [{ message: { role: "assistant", content: fullText } }] },
      promptTokens: usage.prompt_tokens, completionTokens: usage.completion_tokens,
    }).catch(() => {});
  }
}

In production you will want a proper SSE line parser that handles lines split across chunks, but the structure above is the important part: forward first, count as you go, persist after. The client never waits for your bookkeeping.

Step 7: Token Telemetry in MongoDB

Every completed request produces one immutable usage event. MongoDB time-series collections are a strong fit: they compress well and make "cost per tenant per hour" aggregations fast.

// src/telemetry.ts
import { MongoClient } from "mongodb";
import { publishCost } from "./realtime";
import { priceFor } from "./pricing";

const client = new MongoClient(process.env.MONGO_URL!);
const db = client.db("llm_gateway");

export async function initTelemetry() {
  const exists = await db.listCollections({ name: "usage_events" }).hasNext();
  if (!exists) {
    await db.createCollection("usage_events", {
      timeseries: { timeField: "ts", metaField: "meta", granularity: "seconds" },
      expireAfterSeconds: 60 * 60 * 24 * 400,
    });
  }
}

export async function emitUsage(e) {
  const costUsd = e.cacheHit ? 0 : priceFor(e.route.model, e.promptTokens, e.completionTokens);
  const doc = {
    ts: new Date(),
    meta: { tenantId: e.tenantId, backend: e.route.backend, model: e.route.model },
    promptTokens: e.promptTokens,
    completionTokens: e.completionTokens,
    costUsd,
    cacheHit: e.cacheHit,
    routeReason: e.route.reason,
    latencyMs: e.latencyMs ?? 0,
  };
  await db.collection("usage_events").insertOne(doc);
  publishCost(doc);
}

Pricing Local Models Honestly

A common mistake is recording Ollama calls at zero cost. Your GPUs are not free. Compute an effective cost per million tokens from your instance price divided by measured throughput, and record that. It keeps your routing decisions honest and lets the dashboard show true savings rather than a flattering illusion.

// src/pricing.ts
const PRICE_PER_MTOK: Record<string, { in: number; out: number }> = {
  "llama3.1:8b": { in: 0.04, out: 0.06 },   // amortized GPU cost (example values)
  "gpt-mini":    { in: 0.15, out: 0.60 },   // illustrative, load from config
  "gpt-premium": { in: 2.50, out: 10.00 },  // illustrative, load from config
};

export function priceFor(model: string, inTok: number, outTok: number) {
  const p = PRICE_PER_MTOK[model] ?? { in: 0, out: 0 };
  return (inTok * p.in + outTok * p.out) / 1_000_000;
}

Load real prices from a config table that you update when providers change rates. Hard-coded prices go stale faster than you think.

Step 8: Streaming Live Costs to the Dashboard

Server Side: Socket.io with Tenant Rooms

// src/realtime.ts
import { Server } from "socket.io";
import { createAdapter } from "@socket.io/redis-adapter";
import Redis from "ioredis";
import { verifyDashboardToken } from "./auth";

let io: Server;

export function attachRealtime(httpServer) {
  const pub = new Redis(process.env.REDIS_URL!);
  const sub = pub.duplicate();
  io = new Server(httpServer, { cors: { origin: process.env.DASHBOARD_ORIGIN } });
  io.adapter(createAdapter(pub, sub));

  io.use(async (socket, next) => {
    try {
      const user = await verifyDashboardToken(socket.handshake.auth.token);
      socket.data.tenantId = user.tenantId;
      next();
    } catch {
      next(new Error("unauthorized"));
    }
  });

  io.on("connection", (socket) => socket.join(`tenant:${socket.data.tenantId}`));
}

export function publishCost(doc) {
  io?.to(`tenant:${doc.meta.tenantId}`).emit("usage", doc);
  io?.to("admin").emit("usage", doc);
}

The Redis adapter is what makes this work with multiple gateway replicas: an event emitted on replica A reaches a dashboard connected to replica B.

Client Side: Redux Toolkit Slice

// dashboard/store/usageSlice.ts
import { createSlice, type PayloadAction } from "@reduxjs/toolkit";

export type UsageEvent = {
  ts: string;
  meta: { tenantId: string; backend: "ollama" | "cloud"; model: string };
  promptTokens: number;
  completionTokens: number;
  costUsd: number;
  cacheHit: boolean;
  routeReason: string;
};

type UsageState = { events: UsageEvent[]; spendToday: number; hits: number; total: number };
const initialState: UsageState = { events: [], spendToday: 0, hits: 0, total: 0 };

const usageSlice = createSlice({
  name: "usage",
  initialState,
  reducers: {
    usageReceived(state, action: PayloadAction<UsageEvent>) {
      state.events.unshift(action.payload);
      if (state.events.length > 500) state.events.pop();
      state.spendToday += action.payload.costUsd;
      state.total += 1;
      if (action.payload.cacheHit) state.hits += 1;
    },
  },
});

export const { usageReceived } = usageSlice.actions;
export default usageSlice.reducer;

Capping the in-memory list at 500 events keeps the browser responsive. Historical charts should come from an API backed by a MongoDB aggregation, not from the live stream.

Wiring the Socket into React

// dashboard/components/LiveSpendCard.tsx
"use client";

import { useEffect } from "react";
import { io } from "socket.io-client";
import { useDispatch, useSelector } from "react-redux";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Badge } from "@/components/ui/badge";
import { usageReceived } from "@/store/usageSlice";
import type { RootState } from "@/store";

export function LiveSpendCard({ token }: { token: string }) {
  const dispatch = useDispatch();
  const { spendToday, hits, total } = useSelector((s: RootState) => s.usage);

  useEffect(() => {
    const socket = io(process.env.NEXT_PUBLIC_GATEWAY_URL!, { auth: { token } });
    socket.on("usage", (evt) => dispatch(usageReceived(evt)));
    return () => {
      socket.disconnect();
    };
  }, [token, dispatch]);

  const hitRate = total ? Math.round((hits / total) * 100) : 0;

  return (
    <Card>
      <CardHeader className="flex flex-row items-center justify-between">
        <CardTitle>Spend today</CardTitle>
        <Badge variant="secondary">{hitRate}% cache hits</Badge>
      </CardHeader>
      <CardContent>
        <p className="text-3xl font-semibold tabular-nums">
          ${spendToday.toFixed(4)}
        </p>
        <p className="text-sm text-muted-foreground">{total} requests this session</p>
      </CardContent>
    </Card>
  );
}

Pair this card with a shadcn/ui chart showing cost by model and a table of recent routing decisions with their routeReason. The routing table is the feature engineers love most: it explains, request by request, why the bill looks the way it does.

Step 9: Usage-Based Billing with Stripe, Razorpay, and Cashfree

The Golden Rule: The Ledger Is the Source of Truth

Never bill from in-memory counters, and never let a payment provider be the source of truth for usage. Bill from the MongoDB telemetry ledger, aggregated by a scheduled worker. If a webhook fails or a provider is down, you can always recompute.

// billing/aggregate.ts
export async function hourlyUsage(db, from: Date, to: Date) {
  return db.collection("usage_events").aggregate([
    { $match: { ts: { $gte: from, $lt: to }, cacheHit: false } },
    { $group: {
        _id: "$meta.tenantId",
        tokens: { $sum: { $add: ["$promptTokens", "$completionTokens"] } },
        costUsd: { $sum: "$costUsd" },
    } },
  ]).toArray();
}

Stripe: Billing Meters

Stripe's current metered billing model uses Billing Meters: you report meter events, and Stripe attaches them to the customer's subscription price.

// billing/stripe.ts
import Stripe from "stripe";
const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!);

export async function reportToStripe(tenant, tokens: number, windowId: string) {
  await stripe.billing.meterEvents.create({
    event_name: "llm_tokens",
    payload: { stripe_customer_id: tenant.stripeCustomerId, value: String(tokens) },
    identifier: `${tenant.id}:${windowId}`, // idempotency across retries
  });
}

The identifier field is your safety net. If the worker crashes and retries the same hour, Stripe deduplicates.

Razorpay and Cashfree: Verify, Then Trust

For Indian customers, Razorpay and Cashfree are common choices. A frequent pattern is to compute the period's overage from your ledger, then attach it as a subscription add-on or a one-off charge before the invoice is generated. The part that trips most teams up is webhook verification, so here is a shared receiver:

// billing/webhooks.ts
import express from "express";
import { createHmac, timingSafeEqual } from "node:crypto";
import { markPaid, alreadyProcessed, recordProcessed } from "./ledger";

export const webhooks = express.Router();

// Raw body is required: signatures are computed over the exact bytes
webhooks.use(express.raw({ type: "application/json" }));

function safeEqual(a: string, b: string) {
  const ab = Buffer.from(a), bb = Buffer.from(b);
  return ab.length === bb.length && timingSafeEqual(ab, bb);
}

webhooks.post("/razorpay", async (req, res) => {
  const expected = createHmac("sha256", process.env.RAZORPAY_WEBHOOK_SECRET!)
    .update(req.body).digest("hex");
  if (!safeEqual(expected, req.header("x-razorpay-signature") ?? "")) {
    return res.sendStatus(400);
  }
  const eventId = req.header("x-razorpay-event-id")!;
  if (await alreadyProcessed("razorpay", eventId)) return res.sendStatus(200);

  const event = JSON.parse(req.body.toString());
  if (event.event === "invoice.paid") await markPaid("razorpay", event.payload);
  await recordProcessed("razorpay", eventId);
  res.sendStatus(200);
});

webhooks.post("/cashfree", async (req, res) => {
  const ts = req.header("x-webhook-timestamp") ?? "";
  const expected = createHmac("sha256", process.env.CASHFREE_SECRET_KEY!)
    .update(ts + req.body.toString()).digest("base64");
  if (!safeEqual(expected, req.header("x-webhook-signature") ?? "")) {
    return res.sendStatus(400);
  }
  const event = JSON.parse(req.body.toString());
  const eventId = `${event.type}:${event.data?.order?.order_id ?? ts}`;
  if (await alreadyProcessed("cashfree", eventId)) return res.sendStatus(200);

  if (event.type === "PAYMENT_SUCCESS_WEBHOOK") await markPaid("cashfree", event.data);
  await recordProcessed("cashfree", eventId);
  res.sendStatus(200);
});

Stripe gets the same treatment using stripe.webhooks.constructEvent with the raw body. Mount this router before your global express.json() middleware, or the raw bytes will already be parsed and every signature check will fail. That bug has cost more teams an afternoon than I can count.

When a payment fails, flip the tenant's plan to a restricted tier in PostgreSQL. The rate limiter picks it up on the next request, and you have usage enforcement without a single extra service.

Step 10: Plugging In LangGraph and LangChain Agents

Because the gateway speaks the OpenAI-compatible API, agents need only a new base URL and an optional node hint header.

// agent/model.ts
import { ChatOpenAI } from "@langchain/openai";

export const makeModel = (node: string, tenantJwt: string) =>
  new ChatOpenAI({
    model: "auto",              // the gateway decides
    apiKey: tenantJwt,
    configuration: {
      baseURL: "https://llm.example.com/v1",
      defaultHeaders: { "x-agent-node": node },
    },
    temperature: 0,
  });

// In your LangGraph graph:
// const classifier = makeModel("classifier", jwt);
// const planner    = makeModel("planner", jwt);

This is where the savings compound. In a typical agent, most graph nodes are routing, classification, or extraction steps. With the node hint, those land on local models automatically, while the planner keeps its frontier model.

Step 11: Zero-Downtime Docker CI/CD on AWS Linux

The Container

# Dockerfile
FROM node:22-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build && npm prune --omit=dev

FROM node:22-alpine
WORKDIR /app
ENV NODE_ENV=production
COPY --from=build /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
USER node
EXPOSE 4001
HEALTHCHECK --interval=10s --timeout=3s CMD wget -qO- http://localhost:4001/healthz || exit 1
CMD ["node", "dist/server.js"]

Blue-Green on a Single Host

On an Amazon Linux 2023 EC2 instance, run two slots: blue on port 4001 and green on port 4002. A deploy starts the idle slot, waits for its health check, flips Nginx, then drains the old slot.

#!/usr/bin/env bash
# deploy.sh
set -euo pipefail
IMAGE="$1"
ACTIVE=$(grep -o '400[12]' /etc/nginx/upstreams/active.conf)
if [ "$ACTIVE" = "4001" ]; then NEW=4002; SLOT=green; else NEW=4001; SLOT=blue; fi

docker pull "$IMAGE"
docker rm -f "gateway-$SLOT" 2>/dev/null || true
docker run -d --name "gateway-$SLOT" --env-file /etc/llm-gateway.env \
  -e PORT="$NEW" -p "127.0.0.1:$NEW:$NEW" --restart unless-stopped "$IMAGE"

for i in $(seq 1 30); do
  curl -fs "http://127.0.0.1:$NEW/healthz" && break
  sleep 2
done
curl -fs "http://127.0.0.1:$NEW/healthz" || { echo "new slot unhealthy"; exit 1; }

echo "server 127.0.0.1:$NEW;" | sudo tee /etc/nginx/upstreams/active.conf
sudo nginx -t && sudo systemctl reload nginx

# Old slot receives SIGTERM and finishes in-flight streams
OLD=$([ "$SLOT" = "green" ] && echo blue || echo green)
docker stop -t 35 "gateway-$OLD" || true

nginx reload keeps existing connections alive on old workers, and our SIGTERM handler gives streaming responses up to 30 seconds to finish. Users never see a dropped response.

GitHub Actions Pipeline

# .github/workflows/deploy.yml
name: deploy-gateway
on:
  push:
    branches: [main]

permissions:
  id-token: write
  contents: read

jobs:
  build-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm ci && npm test
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE }}
          aws-region: ap-south-1
      - uses: aws-actions/amazon-ecr-login@v2
        id: ecr
      - run: |
          IMAGE=${{ steps.ecr.outputs.registry }}/llm-gateway:${{ github.sha }}
          docker build -t $IMAGE .
          docker push $IMAGE
          echo "IMAGE=$IMAGE" >> $GITHUB_ENV
      - run: |
          aws ssm send-command \
            --document-name AWS-RunShellScript \
            --targets Key=tag:Role,Values=llm-gateway \
            --parameters commands="/opt/gateway/deploy.sh $IMAGE"

Using OIDC role assumption and SSM Run Command means no long-lived AWS keys and no SSH port open to the internet. For larger fleets, the same pattern maps cleanly to ECS with rolling deployments behind an Application Load Balancer.

A Real-World Example: What the Numbers Can Look Like

To make this concrete, here is an illustrative profile based on a pattern I see often: a B2B support SaaS running a LangGraph agent with five nodes (intent classifier, entity extractor, retriever, answer writer, and a quality checker).

• Before the gateway: every node called a premium cloud model. Roughly 80% of calls were classification or extraction.
• After routing: classifier, extractor, and quality checker moved to a local 8B model. Only the answer writer stayed on a premium model, and it spilled to a mini model during local saturation.
• After caching: support questions repeat a lot. With a tenant-scoped cache at a 0.95 threshold, a meaningful share of answer-writer calls were served from cache within the first weeks.

The combined effect in setups like this commonly lands in the range of a 50 to 75 percent reduction in token spend. Your mileage will vary with workload repetitiveness and local GPU capacity, which is exactly why the telemetry dashboard exists: it tells you your real number instead of mine.

Best Practices

• Make the gateway OpenAI-compatible. It turns adoption into a one-line config change for every team.
• Log the routing reason on every request. Threshold tuning without it is guesswork.
• Price local inference honestly. Amortize GPU cost into an effective per-token rate.
• Scope the cache three ways: tenant, model family, and system prompt hash.
• Calibrate thresholds with shadow mode. Record similarity scores before serving hits.
• Rate limit on tokens, not requests. Charge max_tokens up front and refund later.
• Bill from an append-only ledger. Providers are destinations, not sources of truth.
• Make every webhook idempotent. Providers retry, and so will your workers.
• Handle SIGTERM gracefully. Streaming responses must survive deploys.
• Keep secrets out of images. Use environment files managed by SSM Parameter Store or Secrets Manager.

Common Mistakes

• Using a cloud LLM to classify complexity. You pay to decide whether to pay. Use heuristics first.
• A global semantic cache. One missing WHERE tenant_id = ... clause is a data breach.
• Caching tool-augmented responses. The answer depended on context that changes.
• Buffering streams in Nginx. Leave proxy_buffering on and your agents feel frozen.
• Recording local models at zero cost. It inflates "savings" and hides capacity problems.
• Parsing webhooks as JSON before verifying. Signatures are over raw bytes; parsed bodies never match.
• Counting tokens from your own tokenizer only. Prefer the provider's reported usage and fall back to estimates.
• Restarting containers in place. That is "some-downtime deployment" with extra steps.

🚀 Pro Tips

• Add a "cheap retry" lane. If a local model returns invalid JSON for a schema-constrained request, retry once on a cloud mini model before failing. You get local economics with cloud reliability.
• Train a tiny router later. After a month of telemetry, you have labeled data: prompts, routes, and whether the local answer needed a retry. A small classifier trained on that beats hand-tuned heuristics.
• Warm the cache deliberately. For support bots, pre-generate answers to your top 200 FAQ phrasings at off-peak hours.
• Expose cost headers. Return X-Gateway-Cost-USD on every response so agent developers see the price of their prompts in their own logs.
• Set per-tenant daily spend caps. A runaway agent loop at 3 AM should hit a ceiling, not your credit card.
• Keep Ollama models resident. Use keep_alive so models stay loaded in GPU memory; cold loads wreck time-to-first-token.
• Version your pricing table. Store the price version on each usage event so historical invoices stay reproducible after rate changes.

📌 Key Takeaways

• A hybrid gateway that routes easy prompts to local Ollama models and hard prompts to cloud APIs is the single highest-leverage cost control in an AI stack.
• Semantic caching with pgvector catches paraphrased repeats, but only safe when scoped by tenant, model, and system prompt.
• Token telemetry in a MongoDB time-series collection gives you both a live dashboard and an auditable billing ledger.
• Socket.io with a Redis adapter delivers real-time cost visibility across multiple gateway replicas.
• Stripe Billing Meters, Razorpay, and Cashfree can all sit behind one provider-agnostic ledger with idempotent, signature-verified webhooks.
• Blue-green Docker deploys behind Nginx, plus graceful SIGTERM handling, keep streaming responses alive through every release.

Conclusion

LLM costs are not a mystery; they are just unmanaged. Once every request flows through a gateway that knows who sent it, how hard it is, whether it has been answered before, and what it really costs, the bill stops being a monthly surprise and becomes a number you can engineer.

The architecture in this guide is deliberately boring in the best sense. Express, Nginx, PostgreSQL, MongoDB, Redis, and Docker are tools your team already understands. The intelligence lives in a few small, well-placed decisions: a complexity score, a latency budget, a similarity threshold, and an idempotency key. Start with routing and telemetry, add the semantic cache once you can measure its hit rate, and wire up billing when you are ready to turn your AI features into revenue instead of overhead.

Build the control plane first. The savings follow.

References

• Ollama OpenAI compatibility documentation
• Ollama API reference: embeddings
• pgvector: open-source vector similarity search for Postgres
• LangGraph documentation
• LangChain JS: ChatOpenAI integration
• MongoDB time series collections
• Socket.io Redis adapter
• Redux Toolkit documentation
• shadcn/ui components
• Stripe usage-based billing with Billing Meters
• Razorpay webhook validation
• Cashfree Payments webhooks
• NGINX proxy module reference
• Docker multi-stage builds
• GitHub Actions: configuring OpenID Connect in AWS
• FinOps Foundation: FinOps for AI