Skip to content
nanohypenanohype

CI Eval Pipeline

GitHub Actions workflow for eval-gated deployments. Runs LLM evaluation suites on every pull request, compares scores against a stored baseline, posts a markdown report as a PR comment, and fails the check if any suite regresses beyond a configurable threshold. The job calls Bedrock under an IAM role it assumes through GitHub OIDC federation. Ships with a CLI for local runs, baseline management, and score comparison.

Nameci-eval
Version0.1.0
Categoryai-systems
LicenseApache-2.0
Personaengineering
Tagstypescript, ci, eval, github-actions, llm, ai, bedrock, model-gateway
Sourcetemplates/ci-eval

Render it

Three front doors, one catalog. Pick whichever suits the caller.

# CLI
npx @nanohype/sdk render ci-eval --out ./my-app

# SDK
import { LocalSource, renderTemplate } from "@nanohype/sdk";
const result = await renderTemplate(source, "ci-eval", variables);

# MCP — from an agent
get_template({ name: "ci-eval" })

Prerequisites

ToolVersionWhy
node>=26Node.js runtime for TypeScript execution
gh (optional)—GitHub CLI for posting PR comments

Variables

Required

ProjectName string
Kebab-case project name, used in package.json and directory Must be lowercase kebab-case starting with a letter.

Optional

Description string — defaults to CI pipeline for eval-gated deployments
Short project description for package.json and README
EvalPath string — defaults to evals
Directory containing YAML eval suite files
RegressionThreshold string — defaults to 0.05
Maximum allowed score regression before failing the check (0.0–1.0)
LlmProvider string — defaults to bedrock
Default LLM provider name: bedrock for a run on a CI runner or a workstation, under an IAM role the run assumes (GitHub OIDC federation on a runner, an AWS profile on a workstation); gateway for a run inside the cluster, through the Platform ModelGateway with no credential; or any custom registered provider
PlatformName string — defaults to ${ProjectName}
Name of the Platform CR whose ModelGateway routes a run inside the cluster grades. The gateway provider defaults MODEL_GATEWAY_ENDPOINT to the Service the eks-agent-platform operator creates for it, <PlatformName>-gateway in namespace tenants-<PlatformName>
IncludeTests bool — defaults to true
Include unit tests for baseline, reporter, and runner

What it produces

34 files. Placeholder names such as __APP_NAME__ are what the renderer substitutes into.

.env.example
.eval-baseline.json
.github/workflows/eval.yml
.gitignore
biome.json
evals/log-triage.yaml
evals/pr-report.yaml
package.json
src/__tests__/assertions.test.ts
src/__tests__/baseline.test.ts
src/__tests__/bedrock.test.ts
src/__tests__/config.test.ts
src/__tests__/eval-cases.test.ts
src/__tests__/gateway.test.ts
src/__tests__/reporter.test.ts
src/__tests__/runner-cases.test.ts
src/__tests__/runner.test.ts
src/ci-eval/assertions.ts
src/ci-eval/baseline.ts
src/ci-eval/bootstrap.ts
src/ci-eval/cases.ts
src/ci-eval/config.ts
src/ci-eval/index.ts
src/ci-eval/logger.ts
src/ci-eval/providers/bedrock.ts
src/ci-eval/providers/gateway.ts
src/ci-eval/providers/index.ts
src/ci-eval/providers/registry.ts
src/ci-eval/providers/types.ts
src/ci-eval/reporter.ts
src/ci-eval/runner.ts
src/ci-eval/types.ts
tsconfig.json
vitest.config.ts

Composes with