Data Pipeline

A Data Pipeline is an automated sequence of steps that moves data from source systems to a destination, transforming it along the way so it arrives in a usable shape. A pipeline defines where data comes from, what happens to it, where it lands, when it runs, and what should occur when a step fails. Two patterns dominate. ETL transforms data before loading it, which suits systems with limited storage or strict schemas; ELT loads raw data first and transforms it inside the warehouse, which is now more common because cloud warehouse compute is elastic. In practice a team uses Fivetran or Airbyte to extract, dbt to define transformations as version-controlled SQL, and Apache Airflow, Dagster, or Prefect to schedule and orchestrate the runs. Streaming pipelines built on Kafka handle cases where nightly batches are too slow. The pitfall is silent failure: a job that completes but writes partial or stale data is worse than one that errors loudly, so pipelines need freshness checks, row-count assertions, and alerting on anomalies, not just on exceptions. Pipelines also accumulate dependencies, and an unowned pipeline nobody understands becomes the hardest thing in a data platform to change.