Change Data Capture in 2026: When Streaming Your Database to the Warehouse Is Worth It, and When Batch Still Wins
Last updated: August 2026
In the demo it looked effortless. The team aimed a change data capture connector at their production Postgres, and an insert on the orders table surfaced in the warehouse a heartbeat later. No overnight batch, no waiting until morning to see yesterday’s totals. The dashboard moved in real time, and everyone in the room agreed this was how it should have worked from the start.
The trouble showed up three weeks later, on a Saturday, and not in the warehouse. Over the weekend, with nobody watching, the connector drifted behind. Because it was reading from a Postgres replication slot, the database faithfully retained every write-ahead log segment the connector had not yet consumed, and that retained log stopped the disk from being reclaimed. By Sunday the production database that runs the actual business was sitting on a full disk and rejecting writes. A team that went hunting for fresher analytics had come within an inch of knocking over the source system instead.
This guide is for teams sizing up CDC, or already running it and feeling the operational drag. It walks through what CDC really is beneath the marketing, the concrete ways it fails in production, why the most dangerous failures land on the source database rather than the warehouse, how the 2026 tooling shakes out, and the honest call on when streaming your database earns its cost and when a plain batch load still comes out ahead.
What CDC actually is
Every transactional database writes a log of committed changes before it applies them to the tables themselves. Postgres calls it the write-ahead log, MySQL the binary log, Oracle the redo log. Every insert, update, and delete lands there first, carrying enough detail to reconstruct exactly what changed. Change data capture is the family of techniques that turns that stream into something a warehouse can ingest, so you move only what changed instead of reloading the whole table each time.
There are three ways to pull it off, and the differences are not cosmetic. Log-based CDC reads the transaction log directly and emits an event for every change; it is the production default wherever the source allows it, because it catches hard deletes, preserves ordering, and asks nothing of the source schema. Timestamp-based CDC skips the log and instead polls for rows whose updated_at is newer than the last run, which is simpler but cannot see deletes at all. Trigger-based CDC installs database triggers that record every change into a staging table, which is complete but taxes every single write on the source. Log-based reads the transaction log and captures all changes including deletes with near-zero impact on the running workload, which is why most production pipelines land on it, usually by way of Debezium, the canonical open-source implementation with connectors for the major databases.
That much is what the demo shows you. What it leaves out is where the real engineering lives.
Why the demo lies: the snapshot handoff
A connector switched on today can only stream changes from today onward. It has no idea the billion rows that already existed before it started even exist. So every real deployment has to load the current state first and then hand off to the live stream, and that handoff is where most implementations introduce silent data loss.
The naive orderings fail without ever raising their hand. Snapshot the table first and then start the connector, and any change made in the gap between the two is gone for good. Start the connector first and then snapshot, and you get the snapshot row plus replayed events for that same row, which gives you duplicates. The correct move is a coordinated handoff: the connector pins the snapshot to a specific log position, buffers the live events from that position while the snapshot reads, then replays the overlap through an idempotent merge so nothing gets counted twice. Debezium does this properly out of the box. Hand-rolled connectors and rushed setups often do not, and the symptom is ugly precisely because it hides: the row counts line up, so the pipeline looks healthy, while individual values quietly disagree with the source because rows changed mid-snapshot and were captured at their old state. It is the kind of wrong a green dashboard actively conceals, the same shape behind so many silent pipeline breaks.
The failure that hits the source, not the warehouse
Most data pipelines can only damage the warehouse. CDC is the unusual one, because its worst failures reach backward and threaten the operational database it reads from, the system your product actually depends on. The Saturday story above is the textbook case, and it turns up often enough to be a recurring thread in practitioner forums.
The mechanism is worth grasping before you switch CDC on. On Postgres, log-based CDC leans on a logical replication slot, which exists so the connector can restart without missing anything. To hold that guarantee, Postgres refuses to clean up any write-ahead log the slot has not yet acknowledged. If the connector stalls, crashes, or simply lags behind a heavy write burst, the unconsumed log accumulates, and an inactive slot can run the source database’s disk to zero. Slot disk usage belongs on your dashboards from day one, and a runbook for clearing a stuck slot is not a nice-to-have. The broader lesson is that CDC is not fire-and-forget, and the team that treats it as though it were tends to learn the operational contract at the least convenient possible moment.
Schema changes are where it quietly rots
The other slow leak is schema evolution, and it explains why a CDC pipeline that was correct in March is subtly wrong by September. The damage depends on the kind of change. Adding a column is usually harmless. Dropping one is trickier and breaks any consumer still expecting it. Renaming a column is the worst case, because the log records it as a drop followed by an add, which is ambiguous about whether the data should carry over, and most consumers read it as loss. Changing a column’s type is the silent-corruption case, where a downstream table either rejects the value or coerces it into something wrong.
None of this announces itself as an error unless something is watching for it. The discipline that holds up is to treat the stream’s schema as shared infrastructure and enforce compatibility before a change ships, which in practice means registering every stream with a schema registry and enforcing backward or full compatibility rules so a breaking change is rejected at deploy time rather than discovered in a dashboard. Pipelines that fail silently on a schema change are a production risk, not an edge case, and they are the single most common reason a CDC setup that hummed along for months starts producing quietly wrong numbers.
CDC is not built for bulk
One more limit catches teams off guard, because it cuts against the intuition that a real-time pipeline should cope with anything. CDC is designed for a steady flow of individual transactions, not for a single statement that rewrites millions of rows in one shot. When a nightly job or a migration fires a bulk update, the connector has to explode every affected row into its own change event, and that flood can back the pipeline up, blow through connector limits, and in some configurations be silently skipped and logged rather than delivered. The pattern that works is to route large batch operations around CDC entirely, loading them into the warehouse directly and reserving the change stream for the ordinary transaction traffic it was built to carry. A pipeline that assumes every write is a small one is a pipeline waiting for its first big backfill to break it.
The 2026 tooling landscape
If you conclude CDC is right for you, the market breaks into three camps, and the choice is mostly about how much operational weight you are willing to carry. The open-source route is Debezium on Kafka Connect: powerful, free, and yours to operate, which means running Kafka, the connectors, and the recovery playbooks all fall to you. Recent Debezium releases have filed down the rough edges with incremental snapshots that avoid table locks, better exactly-once support, and native support for Kafka’s KRaft mode, and Flink CDC connectors now let teams stream straight from a database into Flink with no Kafka in the middle.
The managed camp trades money for that operational weight. Fivetran, Airbyte, and Estuary run the sync on your behalf, with Estuary built specifically around unifying real-time CDC and batch backfills in one platform so teams stop maintaining two codebases for the same data. The cost model is the part to read closely, because managed CDC is usually billed on volume. Fivetran’s monthly active rows pricing is cheap at low volume and can escalate quickly on large, busy datasets, which is exactly the shape of a high-throughput CDC source, so the invoice can blindside a team that sized it on a pilot.
The third camp is warehouse-native, and it is the youngest. Snowflake shipped Openflow, built on the Apache NiFi project it acquired through Datavolo in 2024 and released in 2025, billed in Snowflake credits and bound to the Snowflake platform. Databricks counters with LakeFlow Connect for native ingestion into the lakehouse. These cut the number of vendors in your stack and raise your coupling to a single platform, which is the trade on the table. The right pick turns on whether you want zero maintenance and will pay for it, whether you want open source you can self-host, and how much you value staying portable.
Fresh is not the same as correct
There is a quieter reason to be careful with the whole premise, and it sits beneath the tooling debate. The entire pitch for CDC is freshness: the number is seconds old instead of a day old. But a fresher number is not a more correct one, and streaming a metric in real time mostly means a wrong or noisy figure now arrives faster and gets acted on sooner. That matters more as the consumer on the other end stops being a human who might pause at an odd value and becomes an automated rule or model that fires on whatever the stream reports. It is the same failure shape that makes an AI-generated answer so dangerous when it looks reasonable and is quietly wrong: nobody sees an error, so the mistake only surfaces downstream when someone tries to reconcile it against reality.
The trap is treating every real-time movement as signal. Most short-term motion in a live metric is noise, and a dashboard refreshing every few seconds invites people, and systems, to react to swings that would not survive a significance test. The discipline that helps is to separate the speed of delivery from the judgment about whether a change means anything: work out whether a movement is statistically real before anyone acts on it. QuantumLayers builds its insights engine around exactly that separation, computing significance and effect sizes deterministically so a change that reaches a decision is a verified movement rather than a live wobble that reads as urgent only because it is fresh. CDC can get your data to you in seconds. It cannot tell you whether the thing that just moved is worth moving on.
When CDC earns its keep, and when batch still wins
Peel away the appeal of real-time and the decision reduces to one question: is the freshness worth the operational contract you are signing?
Use CDC when seconds genuinely change a decision. Fraud checks, inventory that can oversell, operational systems reacting to live state, and syncs where deletes have to propagate correctly are the cases where log-based CDC pays for its overhead. If a human or a system acts on the data within minutes and an hour of staleness would cause real harm, the pipeline earns its keep. If instead the live data needs to flow the other way, from the warehouse back into the tools your team operates in, that is reverse ETL’s job rather than CDC’s.
Default to log-based when you do use it, and coordinate with the source team. The deletes-captured property alone justifies log-based over the query-based alternative for most sources, but it means DBA cooperation and slot monitoring on Postgres are part of the deal, not something you bolt on afterward.
Reach for batch or incremental loads when the freshness is decorative. A great deal of analytics is read the next morning no matter how fast it arrived, and for that, an hourly or nightly incremental load, driven by whatever orchestrator already schedules your pipelines, is cheaper to run, easier to reason about, and far less likely to page anyone at 3 a.m.
Do not put CDC on a table you could just reload. For a small reference table, a full nightly reload costs less to operate and reason about than any CDC mechanism. Streaming a thousand-row lookup table is effort spent buying freshness nobody asked for.
Start on one non-critical table. The teams that succeed validate the pipeline on something low-stakes first, learn the snapshot, schema, and monitoring disciplines on data that will not hurt if it wobbles, then expand. Flipping CDC on across the whole warehouse in a single move is how the Saturday disk incident happens.
The bottom line
Change data capture is a genuinely powerful way to keep a warehouse in step with the systems that feed it, and for workloads that need live data it is the right tool. The trap is adopting it for the demo feeling rather than the requirement. Beneath the tidy architecture diagram sit a snapshot handoff that loses data without a sound, a replication slot that can take down the database it reads, schema changes that rot the pipeline in the dark, and a cost model that punishes exactly the high-volume sources CDC is meant to serve.
So settle the requirement before the tooling. Put CDC where seconds actually change what someone does, run it log-based with the source team watching the slot, and send everything else through a batch load that lets the whole team sleep. Remember that arriving faster is not the same as being right, and that a real-time number still has to clear the bar of meaning something. Do that, and CDC becomes the quiet backbone it is meant to be. Skip it, and the freshest data in the company will be the thing that pages you on a Saturday.
