EvalSuite

apiVersion

governance.nanohype.dev/v1alpha1

kind

EvalSuite

scope

Namespaced

reconciled by

eks-agent-platform

short names

eval

EvalSuite is a scheduled evaluation run against an AgentFleet's agents.

EvalSuiteSpec defines a periodic evaluation run against an AgentFleet.

What kubectl get shows

Column	Type	Meaning
Fleet	string	—
Schedule	string	—
LastScore	string	—
Phase	string	—

Spec

• agentFleetRefobjectrequired

  AgentFleetRef targets the fleet whose agents are under test.

  • namestringrequired

• platformRefobjectrequired

  LocalRef references a CR by name in the same namespace.

  • namestringrequired

• cases[]object

  Cases is the list of test cases (input prompt + expected criteria). In production these are typically loaded from an S3 manifest; this inline list is for small / dev suites.

  • inputstringrequired

  • namestringrequired

  • expectContains[]string

    ExpectContains: the output must contain every one of these substrings (golden / positive assertion). Empty = no positive-content assertion.

  • expectNotContains[]string

    ExpectNotContains: the output must contain none of these substrings (adversarial / data-leak assertion — e.g. a secret, PII, or a phrase that would indicate the agent complied with an injection). Empty = no forbidden-content assertion.

  • expectRefusalboolean

    ExpectRefusal: when true, the case passes only if the agent declined — either the model gateway reported a guardrail intervention, or the output matched a refusal. Use for adversarial prompts that should be blocked rather than answered.

  • maxCostUsdstring

    MaxCostUsd: if set, the case fails when the observed per-call cost exceeds this ceiling. A model with no pricing entry (unpriced) fails this assertion closed rather than passing on a misleading $0.

  • maxLatencyMsinteger (int32)

    MaxLatencyMs: if set (>0), the case fails when the observed round-trip latency exceeds this ceiling.

• casesFromManifeststring

  CasesFromManifest names the JSON file the suite's cases load from, such as `smoke.json`. The run reads `<platform>/manifests/<casesFromManifest>` in the eval-reports S3 bucket, under the referenced Platform's own prefix. The value is a bare file name: a letter or digit, then letters, digits, `.`, `_` and `-` with no two dots together, ending in `.json`. A path, a URI or a `..` segment is refused, so the key the run builds from this name stays under the Platform's `manifests/` prefix, the only one its eval role can read.

• passThresholdstringdefault "0.85"

  PassThreshold (0..1) is the required mean score for the run to be marked passing. Argo Rollouts AnalysisTemplate consumes this signal. Modeled as a string so reviewers see decimals in `kubectl get -o yaml` without int<->float coercion surprises; pattern enforces 0.0 .. 1.0.

• schedulestring

  Schedule (cron) — when to run the suite. Empty = manual only.