Modern Data Stack Pipeline Architecture

End-to-end containerized ELT, code-first transformations, auto-suspending warehouse.

Built for teams with 3+ sources and multiple stakeholders. For a single source, a lighter setup (dlt + dbt + a managed warehouse) is enough.

Suggested Stacks:
Upgrade Outcomes & ROI
Click any outcome to highlight components: • Interactive
1. Save Time
Data updates itself, and fixes or new reports ship on the same day.
• Before: Scripts and exports run by hand; failures discovered only when a client asks. Stored procedures make changes slow and risky.
• With this pipeline: Pipelines run themselves every night and alert the team on Slack only when something breaks. Developers test changes on a copy of production in seconds, so fixes and new reports ship quicker with standardised flow.
Targets: Dagster + Snowflake Clone → Highlight Flow
2. Save Money
Save recurring cost from servers or seats.
• Before: Always-on database or orchestration servers billing 24/7; recurring monthly BI viewer license fees per user.
• With this pipeline: Compute runs only while jobs run, and reports have no per-viewer licence. Costs grow with the data you process, not with the number of people reading reports.
Targets: Snowflake Auto-Suspend + Evidence Static Host → Highlight Flow
3. Maintainable
Any data engineer can pick it up, and nothing depends on one person.
• Before: Logic hidden in stored procedures or one person's laptop. Breakages found by clients; rollback is guesswork.
• With this pipeline: Every rule, model and report lives in Git using standard open-source tools. A new engineer can read it, change it safely and roll it back, without relying on whoever built it.
Targets: dbt Core + GitHub Actions CI/CD → Highlight Flow
4. Scalability
New clients on the same data sources onboard through configuration, without custom code or per-client database clones.
• Before: Adding a client requires manual MySQL database setup in AWS, custom stored procedure parameters, and duplicating Power BI files.
• With this pipeline: Each client reuses the same dbt models and dashboard templates. Onboarding is a config entry, scripted Snowflake role setup, and an automated Evidence build behind Cloudflare Access, with row-level security enforced by Snowflake. A new type of data source still needs its own dlt source and dbt models, added once and reused for every later client.
Targets: Tenant Ingress + dlt + Multi-tenant Bronze → Highlight Flow
GitHub Actions Development and Production Workflow
Development Environment → Team Approval → Production Environment
Development (dev branch) Automatic
Feature branch → Pull Request to 'dev' → Gate 1: Preflight → Gate 2: dbt test → Gate 3: Evidence build → Deploy to Dev DB
Output: Internal Staging Dashboard Dev DB
Pull Request to 'main'
Team Review and Approval
Production (main branch) Gated
Merge to 'main' → Gate 1: Preflight → Gate 2: dbt test → Gate 3: Evidence build → Deploy to Prod DB
Output: Live Production Dashboard Prod DB
1 Tenant Ingress Auth & Registry
Tenant Ingress
Parameters:
• tenant_id
• org_id
SaaS Data Sources External
External APIs
REST / Webhooks
|
ECS Fargate (Docker)
2 dagster (Data Orchestration)
Schedules pipeline runs, gathers dlt and dbt execution logs, handles retries, and pushes failure notifications to Slack
dltHub
3 dlt
(Ingestion Engine)
snowflake
4 Snowflake
(Data Warehouse)
DEV_DB: Clone | PROD_DB: Live
Bronze Silver Gold
dbt
5 dbt Core
(Transformations)
Direct SQL Queries (Gold Production Marts):
6 evidence.dev
6 AI Model
Credentials
AWS Secrets Manager
API Tokens & DB Passwords
Logs
Amazon CloudWatch
Container Telemetry & Logs
#dev-ci
Channel silent
No active CI alerts (Channel silent)
Dev & PR Webhooks Private channel for Pre-merge checks
Webhook Dispatches
#data-ops
Channel silent
No active production alerts (Channel silent)
Production Webhooks Team channel's for P0 Incidents & Deploy Summaries
?
Operational FAQ: Engineering Failure Modes & CI/CD Mechanics
8 practical data engineering failure scenarios and handling solutions
Click to expand