Eval Harness

Scaffolds a standalone TypeScript evaluation framework for testing LLM outputs. Loads eval suites from YAML files, runs each case against an LLM provider, applies composable assertions (string matching, regex, JSON schema, token limits, custom predicates), and reports results to the console or as structured JSON. Works with any LLM system: a bedrock provider calls Bedrock under the IAM role the run assumes, and a gateway provider calls the Platform ModelGateway from inside the cluster.

Name	`eval-harness`
Version	`0.1.0`
Category	ai-systems
License	Apache-2.0
Persona	engineering
Tags	typescript, eval, testing, llm, ai, bedrock, model-gateway
Source	`templates/eval-harness`

Render it

Three front doors, one catalog. Pick whichever suits the caller.

# CLI
npx @nanohype/sdk render eval-harness --out ./my-app

# SDK
import { LocalSource, renderTemplate } from "@nanohype/sdk";
const result = await renderTemplate(source, "eval-harness", variables);

# MCP — from an agent
get_template({ name: "eval-harness" })

Prerequisites

Tool	Version	Why
`node`	`>=26`	Node.js runtime for TypeScript execution

Variables

Required

ProjectName string: Kebab-case project name, used in package.json and directory Must be lowercase kebab-case starting with a letter.

Optional

Description string — defaults to LLM evaluation harness: Short project description for package.json and README
LlmProvider string — defaults to bedrock: Default LLM provider name: bedrock for a run on a CI runner or a workstation, under an IAM role the run assumes (GitHub OIDC federation on a runner, an AWS profile on a workstation); gateway for a run inside the cluster, through the Platform ModelGateway with no credential; or any custom registered provider
PlatformName string — defaults to ${ProjectName}: Name of the Platform CR whose ModelGateway routes a run inside the cluster grades. The gateway provider defaults MODEL_GATEWAY_ENDPOINT to the Service the eks-agent-platform operator creates for it, <PlatformName>-gateway in namespace tenants-<PlatformName>
IncludeCi bool — defaults to true: Include GitHub Actions CI workflow for automated eval runs

What it produces

33 files. Placeholder names such as __APP_NAME__ are what the renderer substitutes into.

.env.example
.github/workflows/evals.yml
.gitignore
bin/run-evals.ts
biome.json
package.json
README.md
src/__tests__/assertions-registry.test.ts
src/__tests__/assertions.test.ts
src/__tests__/bedrock.test.ts
src/__tests__/corpus.test.ts
src/__tests__/gateway.test.ts
src/__tests__/suite.test.ts
src/assertions.ts
src/bootstrap.ts
src/case.ts
src/corpus.ts
src/providers/bedrock.ts
src/providers/gateway.ts
src/providers/index.ts
src/providers/mock.ts
src/providers/registry.ts
src/providers/types.ts
src/reporters/console.ts
src/reporters/json.ts
src/resilience/__tests__/circuit-breaker.test.ts
src/resilience/circuit-breaker.ts
src/runner.ts
src/suite.ts
suites/adversarial.yaml
suites/example.yaml
tsconfig.json
vitest.config.ts

Composes with

Pairs with agentic-loop
Pairs with rag-pipeline
Pairs with mcp-server-ts
Nests inside monorepo

Composites that use it

Agent Team — Multi-agent system with a central orchestrator, specialized agents for research and writing, an evaluation harness, and MCP tool server.
AI Chatbot — Full-stack AI chatbot with agentic loop, HTTP service, authentication, evaluation harness, and a Platform tenant chart whose ModelGateway is the agent's model path..
AI Web Application — Full-stack web application with Next.js frontend, RAG pipeline, authentication, database, and a Platform tenant chart whose ModelGateway is the app's model path..
Document Intelligence — Document search and question-answering system with RAG pipeline, HTTP service, file storage, database, and a Platform tenant chart whose ModelGateway serves the pipeline's generation and embeddings routes..
Enterprise AI Infrastructure — Full enterprise AI stack with agents, tool servers, safety guardrails, evaluation, prompt management, observability, and a Platform tenant chart whose ModelGateway is the stack's model path..
AI Evaluation Suite — Standalone evaluation infrastructure with test harness, versioned prompts, and guardrails validation.
Go Microservice — Go HTTP service with observability and deployment.
Internal Tool — CLI tool or browser extension backed by MCP servers for internal team use..
MCP Toolkit — MCP server with evaluation harness, prompt library, and deployment.
Multi-Agent System — Multi-agent architecture with A2A protocol peers, MCP tool servers, and an orchestrating agent.
AI Proof of Concept — Minimal AI agent with evaluation harness and the Platform tenant chart whose ModelGateway is its model path.
RAG-Powered Agent — AI agent that uses retrieval-augmented generation as a tool.
Safe AI Agent — AI agent with safety guardrails, evaluation harness, prompt management, and the Platform tenant chart whose ModelGateway is its model path.