Data Ingestion & ETL

Data ingestion is the work of getting data from where it's created - databases, SaaS applications, files, event streams - to where it's used, usually a warehouse or lakehouse. It sounds simple and is famously not: sources change shape, volumes grow, some data must arrive in seconds and some can wait until tonight, and every broken pipeline is a dashboard quietly showing the wrong number. The tools in this category exist to make that movement reliable, and they split along a few important lines.

ETL vs ELT

The classic pattern is ETL - extract, transform, then load - where data is cleaned and reshaped before it lands. Tools like Informatica PowerCenter, Talend, SSIS, and Kettle grew up here. The modern pattern is ELT - load raw data first, transform it in the warehouse - which cheap cloud compute made practical. Managed extractors like Fivetran and Stitch, and warehouse-pushdown tools like Matillion, embody it, usually paired with dbt for the transform step.

Batch vs streaming vs CDC

Some data moves in batches on a schedule; some moves continuously as events. For real-time replication from databases, change data capture reads the transaction log so every insert, update, and delete streams out the moment it commits - Debezium is the open-source standard, and dataflow tools like Apache NiFi and StreamSets handle continuous, drift-tolerant movement across many systems.

Reverse ETL and activation

Ingestion increasingly runs both ways. Reverse ETL pushes modeled data back out of the warehouse into operational tools like CRMs - Census specializes here, Polytomic moves data in any direction, and Omnata runs the sync natively inside Snowflake.

Choosing a tool

Loading SaaS data into a warehouse? Managed ELT (Fivetran, Stitch) is usually the fastest path.
Real-time database replication? Log-based CDC with Debezium.
On-prem, complex transformations? Classic ETL (Informatica, SSIS, Talend, Kettle).
Pushing data back to business tools? Reverse ETL (Census, Polytomic, Omnata).

Most real stacks combine several: a managed extractor for inbound SaaS data, CDC for databases, dbt for transformation in the warehouse, and reverse ETL to activate the results. Match the tool to the direction, latency, and where you want transformation to happen.

In this section

Apache NiFi
Apache NiFi is an open-source dataflow tool for automating data movement between systems, with a visual canvas, backpressure, and full data provenance.
Census
Census is a reverse ETL platform that syncs modeled data out of the warehouse into operational tools like Salesforce and HubSpot - data activation built on the warehouse as source of truth.
Confluent Platform
Confluent Platform is the self-managed enterprise distribution of Apache Kafka, adding Schema Registry, Connect, ksqlDB, RBAC, and tiered storage to the open-source broker.
Debezium
Debezium is an open-source change data capture platform that streams every database insert, update, and delete from the transaction log into Kafka and beyond.
Fivetran
Fivetran is a managed ELT service with fully automated connectors and schema-drift handling that move source data into a cloud warehouse with minimal maintenance.
Informatica PowerCenter
Informatica PowerCenter is the enterprise, metadata-driven on-prem ETL platform for high-volume, mission-critical data integration, now succeeded by the Informatica cloud (IDMC).
Kettle (Pentaho Data Integration)
Kettle, now Pentaho Data Integration, is a mature open-source ETL tool with a visual Spoon designer for building transformations and jobs across on-prem and hybrid data.
Matillion
Matillion is a cloud-native ELT platform for building data pipelines visually and pushing transformations down to run as native SQL inside your data warehouse.
Microsoft SSIS
SQL Server Integration Services (SSIS) is Microsoft's ETL platform bundled with SQL Server, with visual control flow and data flow design for on-prem and Azure data integration.
Omnata
Omnata is a Snowflake Native App for data integration and reverse ETL that runs inside your Snowflake account, so data syncing to and from SaaS apps never leaves the Snowflake boundary.
Polytomic
Polytomic is a multi-directional data movement platform doing ELT, reverse ETL, and database/SaaS sync from one tool - flexible integration in whatever direction the use case needs.
Stitch
Stitch is a lightweight, developer-friendly cloud ELT service built on the open-source Singer standard, replicating source data into a warehouse with transparent row-based pricing.
StreamSets
StreamSets is a data integration platform built around drift-tolerant smart pipelines that absorb source schema changes, with batch and Spark-based engines under a central control hub.
Talend
Talend is a data integration suite spanning ETL, data quality, and governance - historically known for the open-source, Java-generating Talend Open Studio, now part of Qlik.