One rig, governed end to end.
DataRig packages proven engines into a single-host runtime on your machine, and wraps every AI-generated action in a fail-closed governance layer.
The local runtime
DataRig Lite runs entirely through Docker Compose on Windows or macOS. No Kubernetes prerequisite. Every service binds to loopback only.
DataRig console
Single local web app (port 8088): chat, Data Explorer, activity, and tool links. The only surface you interact with.
Spark + Iceberg
Governed ingestion writes versioned Apache Iceberg v2 tables — atomic, schema-aware, never silently overwritten.
Apache Polaris
Open REST catalog with credential vending; Trino and Spark never hold raw storage credentials.
Kafka + Debezium
Change-data-capture backbone for PostgreSQL and SQL Server, with checkpointed, restart-safe streaming.
Airflow + dbt Core
Orchestrated transformations with tests, run-scoped docs, and in-place DAG refresh.
Trino + Superset
Federated SQL over Iceberg feeding governed dashboards provisioned from the business layer.
console 8088 · trino 8080 · airflow 8081 · dbt docs 8082 kafka connect 8083 · superset 8084 · grafana 3000 · polaris 8181
The governance layer
This is what makes agentic data engineering trustworthy: the agent can propose, but only exact, validated, approved plans can execute.
Adapter registry
Execution is denied unless a registered adapter exactly matches source, artifact type, destination, operation, and approved hash.
Hash-bound approvals
Approval binds to an exact content hash. Stale proposals after newer prompts are non-runnable by design.
Fail-closed validation
Validators reject secret-bearing fields, implicit table overwrites, arbitrary commands, mutating SQL, cyclic DAGs, and broken references.
Review-only honesty
Unsupported adapters stay visibly review-only. A tool name in a proposal is never treated as executability.
Local AI posture
Ollama by default; every provider response revalidated app-side. Codex bridge runs on the host so tokens never enter DataRig.
Industry-standard engines
- Apache Iceberg for atomic, versioned tables (canonical format)
- Apache Polaris for the REST catalog and credential vending
- Apache Spark for ingestion and distributed compute
- Kafka + Debezium for change data capture
- Trino for federated SQL queries
- Apache Airflow for orchestration
- dbt Core for SQL transformation and testing
- Apache Superset for dashboards and exploration
No proprietary forks. Your tables are portable open formats readable by any compatible engine — leaving means copying your data directory, not migrating a platform.
Cloud & Enterprise track (coming soon)
Once your workflows outgrow a laptop, paid editions run the same architecture as managed deployments: dedicated GCP projects with private networking, Cloud SQL-backed catalogs, Terraform-provisioned foundations, per-service secrets, and digest-pinned release automation. Same pipeline definitions, same governance — operated for you (Cloud) or inside your own cloud project (Enterprise).