Multimodal Pipeline

Scaffolds a multimodal processing pipeline in TypeScript. Accepts image, audio, and video inputs, detects modality automatically, routes each input through a modality-specific processor, sends the result to a vision-capable LLM through the Platform's ModelGateway, which holds the model credential so the pipeline holds none, and returns structured output with Zod-validated schemas. Off the substrate, a Bedrock provider and Amazon Transcribe run under the AWS credential chain; no component holds a provider key. No heavy frameworks — every component is built from first principles: a model call goes to the ModelGateway over fetch, and Bedrock, Transcribe and S3 are reached through the AWS SDK. Processors and LLM providers use a self-registering registry pattern for easy extension.

Name	`multimodal-pipeline`
Version	`0.1.0`
Category	ai-systems
License	Apache-2.0
Persona	engineering
Tags	typescript, multimodal, vision, audio, video, llm, ai, structured-output, model-gateway, bedrock
Source	`templates/multimodal-pipeline`

Render it

Three front doors, one catalog. Pick whichever suits the caller.

# CLI
npx @nanohype/sdk render multimodal-pipeline --out ./my-app

# SDK
import { LocalSource, renderTemplate } from "@nanohype/sdk";
const result = await renderTemplate(source, "multimodal-pipeline", variables);

# MCP — from an agent
get_template({ name: "multimodal-pipeline" })

Prerequisites

Tool	Version	Why
`node`	`>=26`	Node.js runtime for TypeScript execution

Variables

Required

ProjectName string: Kebab-case project name, used in package.json and directory names Must be lowercase kebab-case starting with a letter.

Optional

Description string — defaults to A multimodal processing pipeline: Short project description for package.json and README
LlmProvider string — defaults to gateway: LLM provider for multimodal analysis: gateway (the Platform ModelGateway, which holds the Bedrock role on the nanohype substrate), bedrock (Bedrock called directly under the AWS credential chain, off the substrate), mock, or a custom name
PlatformName string — defaults to ${ProjectName}: Name of the Platform CR this project deploys under. The gateway provider defaults MODEL_GATEWAY_ENDPOINT to the Service the eks-agent-platform operator creates for it, <PlatformName>-gateway in namespace tenants-<PlatformName>
IncludeAudio bool — defaults to false: Include audio processing support (Amazon Transcribe transcription + LLM analysis). Transcription runs under the AWS credential chain, staging audio under TRANSCRIBE_S3_URI, and the ModelGateway serves no transcription route, so this is off the substrate, with LlmProvider bedrock
IncludeVideo bool — defaults to false: Include video processing support (frame extraction + LLM analysis)
IncludeTests bool — defaults to true: Include the vitest config and the pipeline and image processor suites, with the gateway and bedrock request suites and, with IncludeAudio, the transcribe suite

What it produces

47 files. Placeholder names such as __APP_NAME__ are what the renderer substitutes into.

.env.example
.gitignore
biome.json
package.json
README.md
src/__tests__/bedrock.test.ts
src/__tests__/cli.test.ts
src/__tests__/config.test.ts
src/__tests__/eval-cases.test.ts
src/__tests__/gateway.test.ts
src/__tests__/image.test.ts
src/__tests__/pipeline.test.ts
src/__tests__/process-file.test.ts
src/__tests__/transcribe.test.ts
src/bootstrap.ts
src/config.ts
src/eval/cases.ts
src/eval/cases/degrades-when-extraction-yields-no-speech.json
src/eval/cases/reads-the-sign-text-in-an-image.json
src/eval/cases/refuses-to-surface-the-prompt-or-credentials.json
src/eval/cases/summarises-a-transcript-routed-to-audio.json
src/eval/cases/treats-instructions-in-extracted-text-as-content.json
src/eval/runner.ts
src/index.ts
src/logger.ts
src/output/formatter.ts
src/output/types.ts
src/pipeline.ts
src/processors/audio.ts
src/processors/image.ts
src/processors/index.ts
src/processors/registry.ts
src/processors/types.ts
src/processors/video.ts
src/providers/bedrock.ts
src/providers/gateway.ts
src/providers/index.ts
src/providers/mock.ts
src/providers/registry.ts
src/providers/transcribe.ts
src/providers/types.ts
src/resilience/__tests__/circuit-breaker.test.ts
src/resilience/__tests__/request-timeout.test.ts
src/resilience/circuit-breaker.ts
src/resilience/request-timeout.ts
tsconfig.json
vitest.config.ts

Composes with

Pairs with agentic-loop
Pairs with rag-pipeline
Pairs with eval-harness
Nests inside monorepo