Glossary · Integration

Data Pipeline

An automated series of data processing steps that moves and transforms data from sources to destinations.

A data pipeline is a set of automated processes that move data from one or more sources through a series of transformation steps to a destination system. Pipelines handle the full data lifecycle: ingestion (collecting data from sources), transformation (cleaning, enriching, aggregating), validation (checking data quality and completeness), loading (writing to the destination), and monitoring (tracking pipeline health and data freshness). Pipelines can be batch-oriented (processing data in scheduled intervals), streaming (processing data in real-time), or hybrid (micro-batch). Modern data pipeline tools include Apache Airflow, Dagster, Prefect, dbt, Fivetran, and custom solutions built on message queues and stream processors.

Fig — every answer ships with the tables, rows and SQL behind it.