Skip to content
nanohypenanohype

Multimodal Pipeline

Scaffolds a multimodal processing pipeline in TypeScript. Accepts image, audio, and video inputs, detects modality automatically, routes each input through a modality-specific processor, sends the result to a vision or audio-capable LLM (Anthropic Claude, OpenAI GPT-4o), and returns structured output with Zod-validated schemas. No heavy frameworks — all components are built from first principles using provider SDKs directly. Processors and LLM providers use a self-registering registry pattern for easy extension.

Namemultimodal-pipeline
Version0.1.0
Categoryai-systems
LicenseApache-2.0
Personaengineering
Tagstypescript, multimodal, vision, audio, video, llm, ai, structured-output
Sourcetemplates/multimodal-pipeline

Render it

Three front doors, one catalog. Pick whichever suits the caller.

# CLI
npx @nanohype/sdk render multimodal-pipeline --out ./my-app

# SDK
import { LocalSource, renderTemplate } from "@nanohype/sdk";
const result = await renderTemplate(source, "multimodal-pipeline", variables);

# MCP — from an agent
get_template({ name: "multimodal-pipeline" })

Prerequisites

ToolVersionWhy
node>=22Node.js runtime for TypeScript execution

Variables

Required

ProjectName string
Kebab-case project name, used in package.json and directory names Must be lowercase kebab-case starting with a letter.

Optional

Description string — defaults to A multimodal processing pipeline
Short project description for package.json and README
LlmProvider string — defaults to anthropic
LLM provider for multimodal analysis (e.g. anthropic, openai)
IncludeAudio bool — defaults to true
Include audio processing support (whisper transcription + LLM analysis)
IncludeVideo bool — defaults to false
Include video processing support (frame extraction + LLM analysis)
IncludeTests bool — defaults to true
Include vitest test suite with pipeline and processor tests

What it produces

32 files. Placeholder names such as __APP_NAME__ are what the renderer substitutes into.

.env.example
.gitignore
biome.json
package.json
README.md
src/__tests__/image.test.ts
src/__tests__/pipeline.test.ts
src/__tests__/process-file.test.ts
src/bootstrap.ts
src/config.ts
src/index.ts
src/logger.ts
src/output/formatter.ts
src/output/types.ts
src/pipeline.ts
src/processors/audio.ts
src/processors/image.ts
src/processors/index.ts
src/processors/registry.ts
src/processors/types.ts
src/processors/video.ts
src/providers/anthropic.ts
src/providers/index.ts
src/providers/mock.ts
src/providers/openai.ts
src/providers/registry.ts
src/providers/types.ts
src/providers/whisper.ts
src/resilience/__tests__/circuit-breaker.test.ts
src/resilience/circuit-breaker.ts
tsconfig.json
vitest.config.ts

Composes with