Module: LLM Gateway

Unified LLM gateway with pluggable routing strategies, response caching, and cost tracking. Ships with Bedrock (Converse + prompt caching, IRSA auth, the default), Anthropic, OpenAI, and Groq providers, gateway-level provider fallback, six routing strategies (static, round-robin, latency, cost, adaptive, linucb), three caching modes (hash, sliding-TTL, none), token counting via js-tiktoken, and cost anomaly detection using z-score analysis.

Name	`module-llm-gateway`
Version	`0.1.0`
Category	composable-modules
License	Apache-2.0
Persona	engineering
Tags	llm, gateway, routing, caching, cost-tracking, typescript
Source	`templates/module-llm-gateway`

Render it

Three front doors, one catalog. Pick whichever suits the caller.

# CLI
npx @nanohype/sdk render module-llm-gateway --out ./my-app

# SDK
import { LocalSource, renderTemplate } from "@nanohype/sdk";
const result = await renderTemplate(source, "module-llm-gateway", variables);

# MCP — from an agent
get_template({ name: "module-llm-gateway" })

Prerequisites

Tool	Version	Why
`node`	`>=24`	Node.js runtime for the LLM gateway module

Variables

Required

ProjectName string: Kebab-case project name, used as package name and directory Must be lowercase kebab-case starting with a letter.

Optional

Description string — defaults to Unified LLM gateway with routing, fallback, and cost tracking: Short project description for package.json and README
DefaultProvider string — defaults to bedrock: Default LLM provider (bedrock, anthropic, openai, groq, or a custom name)
RoutingStrategy string — defaults to adaptive: Default routing strategy (static, round-robin, latency, cost, adaptive)

What it produces

52 files. Placeholder names such as __APP_NAME__ are what the renderer substitutes into.

.env.example
.gitignore
biome.json
package.json
README.md
src/gateway/__tests__/bedrock-cache.test.ts
src/gateway/__tests__/caching.test.ts
src/gateway/__tests__/circuit-breaker.test.ts
src/gateway/__tests__/cost.test.ts
src/gateway/__tests__/gateway.test.ts
src/gateway/__tests__/icc.test.ts
src/gateway/__tests__/registry.test.ts
src/gateway/__tests__/routing.test.ts
src/gateway/__tests__/tokens.test.ts
src/gateway/bootstrap.ts
src/gateway/caching/hash.ts
src/gateway/caching/index.ts
src/gateway/caching/key.ts
src/gateway/caching/none.ts
src/gateway/caching/registry.ts
src/gateway/caching/sliding-ttl.ts
src/gateway/caching/types.ts
src/gateway/cost/anomaly.ts
src/gateway/cost/pricing.ts
src/gateway/cost/tracker.ts
src/gateway/diagnostics/icc.ts
src/gateway/index.ts
src/gateway/metrics.ts
src/gateway/providers/anthropic.ts
src/gateway/providers/bedrock-cache.ts
src/gateway/providers/bedrock.ts
src/gateway/providers/circuit-breaker.ts
src/gateway/providers/groq.ts
src/gateway/providers/index.ts
src/gateway/providers/mock.ts
src/gateway/providers/openai.ts
src/gateway/providers/registry.ts
src/gateway/providers/types.ts
src/gateway/routing/adaptive.ts
src/gateway/routing/cost.ts
src/gateway/routing/index.ts
src/gateway/routing/latency.ts
src/gateway/routing/linucb.ts
src/gateway/routing/registry.ts
src/gateway/routing/round-robin.ts
src/gateway/routing/static.ts
src/gateway/routing/types.ts
src/gateway/tokens/counter.ts
src/gateway/types.ts
tsconfig.build.json
tsconfig.json
vitest.config.ts

Composes with

Pairs with ts-service
Pairs with agentic-loop
Pairs with rag-pipeline
Nests inside monorepo

Composites that use it

AI Platform — Full AI platform with HTTP service, LLM gateway, vector store, data pipeline, auth, billing, and monitoring.
Cost-Optimized AI Stack — Minimize LLM costs with intelligent routing, response caching, and cost monitoring.
Platform Tenant — One-shot scaffolder for a k8s-native nanohype Platform tenant: the k8s-app-tenant chart (which vendors tenant-chart-base and ships per-env values, an ArgoCD ApplicationSet entry, and the Platform + BudgetPolicy CRs) plus the optional AI building blocks as local packages — the LLM gateway and the vector store.