Skip to content
nanohypenanohype

CI Eval Pipeline

GitHub Actions workflow for eval-gated deployments. Runs LLM evaluation suites on every pull request, compares scores against a stored baseline, posts a markdown report as a PR comment, and fails the check if any suite regresses beyond a configurable threshold. Ships with a CLI for local runs, baseline management, and score comparison.

Nameci-eval
Version0.1.0
Categoryai-systems
LicenseApache-2.0
Personaengineering
Tagstypescript, ci, eval, github-actions, llm, ai
Sourcetemplates/ci-eval

Render it

Three front doors, one catalog. Pick whichever suits the caller.

# CLI
npx @nanohype/sdk render ci-eval --out ./my-app

# SDK
import { LocalSource, renderTemplate } from "@nanohype/sdk";
const result = await renderTemplate(source, "ci-eval", variables);

# MCP — from an agent
get_template({ name: "ci-eval" })

Prerequisites

ToolVersionWhy
node>=22Node.js runtime for TypeScript execution
gh (optional)—GitHub CLI for posting PR comments

Variables

Required

ProjectName string
Kebab-case project name, used in package.json and directory Must be lowercase kebab-case starting with a letter.

Optional

Description string — defaults to CI pipeline for eval-gated deployments
Short project description for package.json and README
EvalPath string — defaults to evals
Directory containing YAML eval suite files
RegressionThreshold string — defaults to 0.05
Maximum allowed score regression before failing the check (0.0–1.0)
LlmProvider string — defaults to anthropic
Default LLM provider name (built-in: anthropic, openai — or any custom registered provider)
IncludeTests bool — defaults to true
Include unit tests for baseline, reporter, and runner

What it produces

27 files. Placeholder names such as __APP_NAME__ are what the renderer substitutes into.

.env.example
.eval-baseline.json
.github/workflows/eval.yml
.gitignore
biome.json
package.json
src/__tests__/assertions.test.ts
src/__tests__/baseline.test.ts
src/__tests__/reporter.test.ts
src/__tests__/runner-cases.test.ts
src/__tests__/runner.test.ts
src/ci-eval/assertions.ts
src/ci-eval/baseline.ts
src/ci-eval/bootstrap.ts
src/ci-eval/config.ts
src/ci-eval/index.ts
src/ci-eval/logger.ts
src/ci-eval/providers/anthropic.ts
src/ci-eval/providers/index.ts
src/ci-eval/providers/openai.ts
src/ci-eval/providers/registry.ts
src/ci-eval/providers/types.ts
src/ci-eval/reporter.ts
src/ci-eval/runner.ts
src/ci-eval/types.ts
tsconfig.json
vitest.config.ts

Composes with