Eval Harness

Scaffolds a standalone TypeScript evaluation framework for testing LLM outputs. Loads eval suites from YAML files, runs each case against an LLM provider, applies composable assertions (string matching, regex, JSON schema, token limits, custom predicates), and reports results to the console or as structured JSON. Works with any LLM system — Anthropic and OpenAI providers included out of the box.

Name	`eval-harness`
Version	`0.1.0`
Category	ai-systems
License	Apache-2.0
Persona	engineering
Tags	typescript, eval, testing, llm, ai
Source	`templates/eval-harness`

Render it

Three front doors, one catalog. Pick whichever suits the caller.

# CLI
npx @nanohype/sdk render eval-harness --out ./my-app

# SDK
import { LocalSource, renderTemplate } from "@nanohype/sdk";
const result = await renderTemplate(source, "eval-harness", variables);

# MCP — from an agent
get_template({ name: "eval-harness" })

Prerequisites

Tool	Version	Why
`node`	`>=22`	Node.js runtime for TypeScript execution

Variables

Required

ProjectName string: Kebab-case project name, used in package.json and directory Must be lowercase kebab-case starting with a letter.

Optional

Description string — defaults to LLM evaluation harness: Short project description for package.json and README
LlmProvider string — defaults to anthropic: Default LLM provider name (built-in: anthropic, openai — or any custom registered provider)
IncludeCi bool — defaults to true: Include GitHub Actions CI workflow for automated eval runs

What it produces

28 files. Placeholder names such as __APP_NAME__ are what the renderer substitutes into.

.env.example
.github/workflows/evals.yml
.gitignore
bin/run-evals.ts
biome.json
package.json
README.md
src/__tests__/assertions-registry.test.ts
src/__tests__/assertions.test.ts
src/__tests__/suite.test.ts
src/assertions.ts
src/bootstrap.ts
src/case.ts
src/providers/anthropic.ts
src/providers/index.ts
src/providers/mock.ts
src/providers/openai.ts
src/providers/registry.ts
src/providers/types.ts
src/reporters/console.ts
src/reporters/json.ts
src/resilience/__tests__/circuit-breaker.test.ts
src/resilience/circuit-breaker.ts
src/runner.ts
src/suite.ts
suites/example.yaml
tsconfig.json
vitest.config.ts

Composes with

Pairs with agentic-loop
Pairs with rag-pipeline
Pairs with mcp-server-ts
Nests inside monorepo

Composites that use it

Agent Team — Multi-agent system with a central orchestrator, specialized agents for research and writing, an evaluation harness, and MCP tool server.
AI Chatbot — Full-stack AI chatbot with agentic loop, HTTP service, authentication, evaluation harness, and deployment..
AI Web Application — Full-stack web application with Next.js frontend, RAG pipeline, authentication, database, and deployment..
Document Intelligence — Document search and question-answering system with RAG pipeline, HTTP service, file storage, database, and deployment..
Enterprise AI Infrastructure — Full enterprise AI stack with agents, tool servers, safety guardrails, evaluation, prompt management, observability, and Kubernetes deployment..
AI Evaluation Suite — Standalone evaluation infrastructure with test harness, versioned prompts, and guardrails validation.
Go Microservice — Go HTTP service with observability and deployment.
Internal Tool — CLI tool or browser extension backed by MCP servers for internal team use..
MCP Toolkit — MCP server with evaluation harness, prompt library, and deployment.
Multi-Agent System — Multi-agent architecture with A2A protocol peers, MCP tool servers, and an orchestrating agent.
AI Proof of Concept — Minimal AI agent with evaluation harness.
RAG-Powered Agent — AI agent that uses retrieval-augmented generation as a tool.
Safe AI Agent — AI agent with safety guardrails, evaluation harness, and prompt management.