Data Pipeline

Scaffolds an ETL pipeline for AI workloads in TypeScript. Implements document ingestion from files and web pages, configurable chunking strategies (recursive, fixed-size, semantic), embedding generation, and output adapters for indexing. No LangChain — all components are built from first principles using provider SDKs directly. Defaults to AWS Bedrock Titan v2 embeddings (on IRSA — no keys), with OpenAI as an alternate. Supports multiple ingest sources (PDF, Markdown, plain text, JSON, CSV, web — SSRF-guarded) and output formats (JSONL, console). Output is compatible with module-vector-store's VectorDocument shape.

Name	`data-pipeline`
Version	`0.1.0`
Category	ai-systems
License	Apache-2.0
Persona	engineering
Tags	typescript, etl, embeddings, pipeline, data, chunking, ai
Source	`templates/data-pipeline`

Render it

Three front doors, one catalog. Pick whichever suits the caller.

# CLI
npx @nanohype/sdk render data-pipeline --out ./my-app

# SDK
import { LocalSource, renderTemplate } from "@nanohype/sdk";
const result = await renderTemplate(source, "data-pipeline", variables);

# MCP — from an agent
get_template({ name: "data-pipeline" })

Prerequisites

Tool	Version	Why
`node`	`>=24`	Node.js runtime for TypeScript execution

Variables

Required

ProjectName string: Kebab-case project name, used in package.json and directory names Must be lowercase kebab-case starting with a letter.

Optional

Description string — defaults to Data pipeline for AI workloads: Short project description for package.json and README
EmbeddingProvider string — defaults to bedrock: Provider for text embeddings — bedrock (Titan v2, default) or openai
ChunkStrategy string — defaults to recursive: Text chunking strategy for splitting documents (e.g. recursive, fixed, semantic)
IncludeTests bool — defaults to true: Include vitest test suite with orchestrator, chunking, and registry tests

What it produces

45 files. Placeholder names such as __APP_NAME__ are what the renderer substitutes into.

.dockerignore
.env.example
.github/workflows/ci.yml
.gitignore
biome.json
Dockerfile
package.json
README.md
src/pipeline/__tests__/fixed-size.test.ts
src/pipeline/__tests__/orchestrator.test.ts
src/pipeline/__tests__/recursive.test.ts
src/pipeline/__tests__/registry.test.ts
src/pipeline/bootstrap.ts
src/pipeline/embed/bedrock.ts
src/pipeline/embed/index.ts
src/pipeline/embed/mock.ts
src/pipeline/embed/openai.ts
src/pipeline/embed/registry.ts
src/pipeline/embed/types.ts
src/pipeline/index.ts
src/pipeline/ingest/file.ts
src/pipeline/ingest/index.ts
src/pipeline/ingest/registry.ts
src/pipeline/ingest/types.ts
src/pipeline/ingest/url-guard.ts
src/pipeline/ingest/web.ts
src/pipeline/logger.ts
src/pipeline/metrics.ts
src/pipeline/orchestrator.ts
src/pipeline/output/console.ts
src/pipeline/output/index.ts
src/pipeline/output/json-file.ts
src/pipeline/output/registry.ts
src/pipeline/output/types.ts
src/pipeline/resilience/__tests__/circuit-breaker.test.ts
src/pipeline/resilience/circuit-breaker.ts
src/pipeline/transform/fixed-size.ts
src/pipeline/transform/index.ts
src/pipeline/transform/recursive.ts
src/pipeline/transform/registry.ts
src/pipeline/transform/semantic.ts
src/pipeline/transform/types.ts
src/pipeline/types.ts
tsconfig.json
vitest.config.ts

Composes with

Pairs with module-vector-store
Pairs with rag-pipeline
Pairs with eval-harness
Nests inside monorepo

Composites that use it

AI Platform — Full AI platform with HTTP service, LLM gateway, vector store, data pipeline, auth, billing, and monitoring.