v3.0.5 is live·17 connectors · YAML transforms · 93 tests·pip install openingest
v3.0.5 · Open Source · PyPIMIT License
DataIngestion.
ZeroBoilerplate.

Register a dataset in YAML. OpenIngest handles discovery → validation → quality → transforms → PostgreSQL → Airflow automatically. No Python. No SQL.

174k
rows / run
17
connectors
4.21s
runtime
99.4%
quality
openingest · bash
$ openingest run
 
Discovering 8 datasets...
✓ customers → stg_customers replace 100.00%
✓ orders → stg_orders incremental 98.50%
✓ products → stg_products replace 100.00%
✓ sessions → stg_sessions replace 100.00%
✓ employees → stg_employees replace 100.00%
✓ events → stg_events incremental 99.20%
✓ order_items → stg_order_items replace 100.00%
✓ reviews → stg_reviews incremental 97.80%
 
Schema ── all 8 valid
Quality ── avg 99.4%
 
Rows : 174,777
Time : 4.21 sec
Status : SUCCESS ✓
python 3.12 · postgresql 15 · airflow 2.9connected
Scroll
01
Chapter 01 / The Problem

The old way is expensive.This is the fix.

Every data team builds the same ingestion layer from scratch — ad-hoc scripts, manual DDL, zero standards, zero observability. OpenIngest makes it a one-time YAML configuration problem.

✕
Without OpenIngest
Script per dataset · no standards
  • ✕New dataset → write a brand new Python script
  • ✕Manually author SQL DDL for each staging table
  • ✕Schema changes break pipelines silently
  • ✕Quality issues only appear in downstream reports
  • ✕Airflow DAG must be edited for every new source
  • ✕No run history, quality scores, or lineage
✓
With OpenIngest
Config-driven · zero repetition
  • ✓New dataset → add one YAML block, done
  • ✓Tables auto-created from inferred column types
  • ✓Schema validation blocks bad data before load
  • ✓Quality engine scores every dataset every run
  • ✓New YAML entry auto-generates a DAG task group
  • ✓Full metadata, quality, and lineage every run
Dataset DiscoverySchema ValidationData Quality EngineIncremental LoadingAuto DDLMetadata LoggingAirflow DAG17 ConnectorsYAML TransformsSlack + Email AlertsBuilt-in SchedulerDocker Ready
Dataset DiscoverySchema ValidationData Quality EngineIncremental LoadingAuto DDLMetadata LoggingAirflow DAG17 ConnectorsYAML TransformsSlack + Email AlertsBuilt-in SchedulerDocker Ready
02
Chapter 02 / How It Works

Nine stages.
Every run.

Click any stage to see what it does and which file owns it.

openingest_dynamic_pipeline
click any stage to inspect
configs/datasets.yamlzero Python
customers:
file: customers.csv
staging_table: stg_customers
load_strategy: replace
primary_key:
- customer_id
# That's it. OpenIngest handles the rest.
Chapter 03 / Connectors

Any source.
One config block.

17 connectors · v3.0
File Formats
Cloud Storage
REST / HTTP
Databases
File Transfer
SaaS

CSV

v1.0

Default format. Reads comma-separated files from data/raw/ or any local path. Encoding and separator are configurable.

built-in
configs/datasets.yaml
my_dataset:
source:
type: csv
file: customers.csv
encoding: utf-8
staging_table: stg_csv
load_strategy: replace
Plugin Architecture
# register any connector
ConnectorRegistry.register("my_db", MyConnector)
Full Docs

All 17 connectors with full config examples, authentication options, and tips.

docs/CONNECTORS.md →
All connectors
Chapter 04 / Transformation Engine

Six YAML steps.
No Python required.

Transformations run after quality checks, before the database write. Declare them in the transforms: block inside datasets.yaml.

renamestep 1/6

Rename columns from source names to clean pipeline names.

Effect on data
before
PM2.5, NO2(GT), Date Time
→
after
pm25, no2, ts
datasets.yaml
1
2
3
4
5
6
7
8
9
air_data:
staging_table: stg_air_clean
# source, load_strategy...
transforms:
- type: rename
columns:
"PM2.5":pm25
"NO2(GT)":no2
"Date Time":ts
source›
validate›
quality›
transform›
db
Chapter 05 / CLI

Everything
from the terminal.

Real output. Real runs. Every number is accurate.

Also available
openingest report
openingest dashboard
openingest scheduler start
openingest doctor
openingest infer
openingest profile
openingest graph
openingest init
openingest airflow build
openingest docker init
bash — openingest
❯openingest run
══════════════════════════════════════
OPENINGEST · OI-20260703-3BB09C
══════════════════════════════════════
 
✓ customers → stg_customers replace 100.00%
✓ orders → stg_orders incremental 98.50%
✓ products → stg_products replace 100.00%
✓ sessions → stg_sessions replace 100.00%
✓ employees → stg_employees replace 100.00%
✓ events → stg_events incremental 99.20%
✓ order_items → stg_order_items replace 100.00%
✓ reviews → stg_reviews incremental 97.80%
 
Rows Loaded : 174,777
Duration : 4.21 sec
Status : SUCCESS ✓
══════════════════════════════════════
By the numbers · OI-20260703-3BB09C

Real run. Real numbers.

—
rows loaded
per pipeline run
—
connectors
CSV to Google Sheets
—
pipeline runtime
8 datasets end-to-end
—
quality score
avg across all datasets
—
tests passing
ruff + mypy clean
—
transform types
rename→cast→filter→…

13 production-grade
components.

Every component is tested, wired together, and exercised on every real pipeline run.

Dynamic Discovery

Scans YAML and builds Dataset objects automatically. Zero code changes when adding sources.

core/discovery.py
Schema Validation

Required columns verified, extra fields caught. Pipeline stops before any data moves.

core/validation.py
Data Quality Engine

Non-null, unique, range, regex, custom df.eval() rules. 0–100% score every run.

core/quality.py
Incremental Loading

Watermark filter + SHA-256 hash CDC + upsert. State persisted between runs.

core/incremental.py
Auto Table Creation

Infers PostgreSQL types from source data. Tables created on first run. No SQL.

utils/db.py
Metadata Tracking

Run ID, status, duration, rows, quality logged to pipeline_runs every execution.

utils/metadata_logger.py
Airflow Integration

One task group per dataset. discover → validate → quality → ingest. Auto from YAML.

core/airflow/
Built-in Scheduler

scheduler start --cron @daily. No Airflow required for simple schedules.

core/scheduler.py
Slack + Email Alerts

Webhook and SMTP notifications on success/failure with configurable retry.

core/notifications.py
Data Lineage

ASCII + Mermaid + JSON lineage graph. openingest graph command.

core/lineage.py
Docker Ready

PostgreSQL + Airflow in one command. docker compose up -d.

docker-compose.yml
CI/CD Built-in

Ruff lint + Mypy type-check + Pytest coverage on every push. 93 tests.

.github/workflows/
Chapter 07 / Get Started

Up and running
in 5 steps.

From zero to 174,777 rows loaded in under 10 minutes.

01
Install
pip install openingest

Registers the openingest CLI. Windows: add Python Scripts to PATH.

02
Scaffold a project
openingest init my-pipeline
cd my-pipeline

Creates configs/, data/raw/, .env, and docker-compose.yml.

03
Configure database
# .env
DATABASE_URL=postgresql://user:password@localhost:5432/openingest
04
Start PostgreSQL
docker compose up -d

PostgreSQL on 5432. Airflow at localhost:8080 — admin / admin.

05
Infer config & run
openingest infer data/raw/customers.csv
openingest run

Discovers files, validates schemas, checks quality, loads PostgreSQL.

expected output
$ openingest run
 
Run ID : OI-20260703-3BB09C
 
✓customersstg_customers100.00%
✓ordersstg_orders 98.50%
✓productsstg_products100.00%
✓eventsstg_events 99.20%
✓order_itemsstg_order_items100.00%
 
Rows : 174,777
Duration : 4.21 sec
Status : SUCCESS ✓
Open Source · MIT · No vendor lock-in

Ship your first pipeline
in minutes.

No cloud account. No API key. No credit card.

$pip install openingest