Data Pipeline

Scaffolds an ETL pipeline for AI workloads in TypeScript. Implements document ingestion from files and web pages, configurable chunking strategies (recursive, fixed-size, semantic), embedding generation, and output adapters for indexing. No LangChain — every component is built from first principles: an embedding call goes to the ModelGateway over fetch, or to Bedrock through the AWS SDK. Defaults to embeddings through the Platform's ModelGateway (Titan v2), so the pipeline holds no model credential, with Bedrock called directly under an assumed IAM role as the off-substrate alternate. Supports multiple ingest sources (PDF, Markdown, plain text, JSON, CSV, web — SSRF-guarded) and output formats (JSONL, console). Output is compatible with module-vector-store's VectorDocument shape.

Name	`data-pipeline`
Version	`0.1.0`
Category	ai-systems
License	Apache-2.0
Persona	engineering
Tags	typescript, etl, embeddings, pipeline, data, chunking, ai, model-gateway
Source	`templates/data-pipeline`

Render it

Three front doors, one catalog. Pick whichever suits the caller.

# CLI
npx @nanohype/sdk render data-pipeline --out ./my-app

# SDK
import { LocalSource, renderTemplate } from "@nanohype/sdk";
const result = await renderTemplate(source, "data-pipeline", variables);

# MCP — from an agent
get_template({ name: "data-pipeline" })

Prerequisites

Tool	Version	Why
`node`	`>=26`	Node.js runtime for TypeScript execution

Variables

Required

ProjectName string: Kebab-case project name, used in package.json and directory names Must be lowercase kebab-case starting with a letter.

Optional

Description string — defaults to Data pipeline for AI workloads: Short project description for package.json and README
EmbeddingProvider string — defaults to gateway: Provider for text embeddings: gateway (the Platform ModelGateway, which holds the Bedrock role on the nanohype substrate), or bedrock (Titan v2, called directly under the AWS credential chain, off the substrate)
PlatformName string — defaults to ${ProjectName}: Name of the Platform CR this project deploys under. The gateway provider defaults MODEL_GATEWAY_ENDPOINT to the Service the eks-agent-platform operator creates for it, <PlatformName>-gateway in namespace tenants-<PlatformName>
ChunkStrategy string — defaults to recursive: Text chunking strategy for splitting documents (e.g. recursive, fixed, semantic)
IncludeTests bool — defaults to true: Include vitest test suite with orchestrator, chunking, registry and gateway request tests

What it produces

57 files. Placeholder names such as __APP_NAME__ are what the renderer substitutes into.

.dockerignore
.env.example
.github/workflows/ci.yml
.gitignore
biome.json
Dockerfile
package.json
README.md
src/pipeline/__tests__/embed-request-timeout.test.ts
src/pipeline/__tests__/eval-assertions.test.ts
src/pipeline/__tests__/eval-cases.test.ts
src/pipeline/__tests__/fixed-size.test.ts
src/pipeline/__tests__/gateway.test.ts
src/pipeline/__tests__/orchestrator.test.ts
src/pipeline/__tests__/recursive.test.ts
src/pipeline/__tests__/registry.test.ts
src/pipeline/__tests__/semantic.test.ts
src/pipeline/bootstrap.ts
src/pipeline/embed/bedrock.ts
src/pipeline/embed/gateway.ts
src/pipeline/embed/index.ts
src/pipeline/embed/mock.ts
src/pipeline/embed/registry.ts
src/pipeline/embed/request-timeout.ts
src/pipeline/embed/types.ts
src/pipeline/eval/assertions.ts
src/pipeline/eval/cases.ts
src/pipeline/eval/cases/bounds-a-document-with-no-sentence-boundary.json
src/pipeline/eval/cases/embeds-instructions-in-fetched-content-as-data.json
src/pipeline/eval/cases/splits-on-the-subject-boundary.json
src/pipeline/eval/runner.ts
src/pipeline/index.ts
src/pipeline/ingest/file.ts
src/pipeline/ingest/index.ts
src/pipeline/ingest/registry.ts
src/pipeline/ingest/types.ts
src/pipeline/ingest/url-guard.ts
src/pipeline/ingest/web.ts
src/pipeline/logger.ts
src/pipeline/metrics.ts
src/pipeline/orchestrator.ts
src/pipeline/output/console.ts
src/pipeline/output/index.ts
src/pipeline/output/json-file.ts
src/pipeline/output/registry.ts
src/pipeline/output/types.ts
src/pipeline/resilience/__tests__/circuit-breaker.test.ts
src/pipeline/resilience/circuit-breaker.ts
src/pipeline/transform/fixed-size.ts
src/pipeline/transform/index.ts
src/pipeline/transform/recursive.ts
src/pipeline/transform/registry.ts
src/pipeline/transform/semantic.ts
src/pipeline/transform/types.ts
src/pipeline/types.ts
tsconfig.json
vitest.config.ts

Composes with

Pairs with module-vector-store
Pairs with rag-pipeline
Pairs with eval-harness
Nests inside monorepo

Composites that use it

AI Platform — Full AI platform with HTTP service, LLM gateway, vector store, data pipeline, auth, billing, monitoring, and a Platform tenant chart whose ModelGateway serves the gateway's and the pipeline's model routes.