Change Data Capture in 2026: When Streaming Your Database to the Warehouse Is Worth It, and When Batch Still Wins
Last updated: August 2026
The pipeline worked beautifully in the demo. The team pointed a change data capture connector at their production Postgres database, and within seconds an insert on the orders table showed up in the warehouse. No nightly batch, no waiting until morning for yesterday’s numbers. The dashboard was live. Everyone agreed this was the way it should have worked all along.
The problem arrived three weeks later, on a Saturday, and it did not arrive in the warehouse. The connector fell behind over the weekend while nobody was watching, and because it was reading from a Postgres replication slot, the database dutifully held onto every write-ahead log segment the connector had not yet consumed. The log kept the disk from being reclaimed. By Sunday the production database that runs the actual business was pointed at a full disk and refusing writes. The data team had gone looking for fresher analytics and very nearly took down the source system instead.
This guide is for teams weighing CDC, or already running it and feeling the operational weight. It covers what CDC really is underneath the marketing, the specific ways it breaks in production, why the riskiest failures land on the source database rather than the warehouse, how the 2026 tooling landscape lines up, and the honest decision of when streaming your database is worth the cost and when a plain batch load still wins.
What CDC actually is
Every transactional database keeps a log of committed writes before applying them to the tables. Postgres calls it the write-ahead log, MySQL the binary log, Oracle the redo log. Each insert, update, and delete lands there first, with enough detail to reconstruct the change. Change data capture is the family of techniques that turns that stream of changes into something a warehouse can consume, so you load only what changed rather than reloading everything.
There are three ways to do it, and the difference is not cosmetic. Log-based CDC reads the transaction log directly and emits an event for every change, and it is the production default wherever the source allows it, because it captures hard deletes, preserves order, and requires no changes to the source schema. Timestamp-based CDC skips the log and instead queries for rows whose updated_at is newer than the last run, which is simpler but blind to deletes. Trigger-based CDC installs database triggers that write every change to a staging table, which is complete but taxes every write on the source. Log-based reads the transaction log directly and captures all changes including deletes with near-zero impact on the running workload, which is why most production pipelines use it, usually through Debezium, the canonical open-source implementation that ships connectors for the major databases.
That is the part the demo shows. The parts it hides are where the engineering lives.
Why the demo lies: the snapshot handoff
A connector that starts today can only stream changes from today forward. It knows nothing about the billion rows that existed before it turned on. So every real deployment has to load the existing state first, then hand off to the live stream, and that handoff is where most implementations introduce silent data loss.
The naive versions fail in ways that do not announce themselves. Snapshot the table first and then start the connector, and every change made during the gap between the two is simply gone. Start the connector first and then snapshot, and you get the snapshot row plus replayed events for the same row, which produces duplicates. The correct mechanic is a coordinated handoff, where the connector pins a snapshot at a specific log position, buffers the live events from that position while the snapshot reads, and then replays the overlap through an idempotent merge so nothing is double counted. Debezium does this correctly out of the box. Custom connectors and rushed setups frequently do not, and the tell is nasty: the row counts match, so the pipeline looks healthy, while individual values quietly disagree with the source because rows changed during the snapshot window and landed at their old state. It is the kind of wrong that a green dashboard actively hides, the same failure shape behind so many silent pipeline breaks.
The failure that hits the source, not the warehouse
Most data pipelines can only hurt the warehouse. CDC is unusual because its worst failures reach back and threaten the operational database it reads from, which is the system your product actually runs on. The Saturday story above is the canonical example, and it is common enough to be a recurring topic in practitioner forums.
The mechanism is worth understanding before you turn CDC on. On Postgres, log-based CDC uses a logical replication slot, and the slot exists so the connector can resume after a restart without missing anything. To make that guarantee, Postgres refuses to clean up any write-ahead log the slot has not yet acknowledged. If the connector stalls, crashes, or just falls behind a heavy write burst, the unconsumed log piles up, and an inactive slot can run the source database’s disk to zero. Slot disk usage is a metric you monitor from day one, and a runbook for replacing a stuck slot is not optional. The general lesson is that CDC is not fire-and-forget, and the team that treats it that way discovers the operational contract at the worst possible time.
Schema changes are where it quietly rots
The other slow leak is schema evolution, and it is the reason CDC pipelines that were correct in March are subtly wrong by September. The failure differs by the kind of change. Adding a column is usually safe. Dropping one is subtler and breaks consumers that still expect it. Renaming a column is the worst case, because the log records it as a drop followed by an add, which is ambiguous about whether the data should carry forward, and most consumers treat it as loss. Changing a column’s type is the silent-corruption case, where a downstream table either rejects the value or coerces it into something wrong.
None of this shows up as an error unless something is watching for it. The discipline that holds is to treat the stream’s schema as shared infrastructure and enforce compatibility before a change ships, which in practice means registering every stream with a schema registry and enforcing backward or full compatibility rules so a breaking change is rejected at deploy rather than discovered in a dashboard. Pipelines that break silently on a schema change are a production risk, not an edge case, and they are the single most common reason a CDC setup that worked for months starts producing quietly wrong numbers.
CDC is not built for bulk
One more limit surprises teams because it runs against the intuition that a real-time pipeline should handle anything. CDC is designed for a steady flow of individual transactions, not for a single statement that rewrites millions of rows at once. When a nightly job or a migration issues a bulk update, the connector has to turn every affected row into its own change event, and that flood can back the pipeline up, blow past connector limits, and in some configurations get silently skipped and logged rather than delivered. The pattern that works is to route large batch operations around CDC entirely, loading them into the warehouse directly and keeping the change stream for the ordinary transaction traffic it was built for. A pipeline that assumes every write is a small one is a pipeline waiting for the first big backfill to break it.
The 2026 tooling landscape
If you decide CDC is right, the market splits into three groups, and the choice is mostly about how much operational weight you want to carry. The open-source path is Debezium on Kafka Connect, which is powerful and free and hands you the full operational burden of running Kafka, the connectors, and the recovery playbooks yourself. Recent Debezium releases have sharpened the rough edges with incremental snapshots that avoid table locks, better exactly-once support, and native support for Kafka’s KRaft mode, and Flink CDC connectors now let teams stream straight from a database into Flink without Kafka in the middle.
The managed group trades money for that operational weight. Fivetran, Airbyte, and Estuary run the sync for you, with Estuary built specifically around unifying real-time CDC and batch backfills in one platform so teams stop maintaining two codebases for the same data. The cost model is the thing to read closely here, because managed CDC is usually priced on volume. Fivetran’s monthly active rows model is cheap at low volume and can escalate quickly on large, busy datasets, which is exactly the profile of a high-throughput CDC source, so the bill can surprise a team that sized it on a pilot.
The third group is warehouse-native, and it is the newest. Snowflake shipped Openflow, built on the Apache NiFi project it acquired through Datavolo in 2024 and released in 2025, billed in Snowflake credits and tied to the Snowflake platform. Databricks offers LakeFlow Connect for native ingestion into the lakehouse. These lower the number of vendors in your stack and raise the coupling to one platform, which is the trade you are making. The right pick depends on whether you want zero maintenance and will pay for it, whether you want open source you can self-host, and how much you value staying portable.
Fresh is not the same as correct
There is a quieter reason to be careful with the whole premise, and it sits underneath the tooling debate. The entire pitch for CDC is freshness: the number is only seconds old instead of a day old. But a fresher number is not a more correct one, and streaming a metric in real time mostly means that a wrong or noisy figure now arrives faster and gets acted on sooner. This matters more as the consumer on the other end stops being a human who might pause at an odd value and starts being an automated rule or model that fires on whatever the stream says.
The failure is treating every real-time movement as signal. Most short-term fluctuation in a live metric is noise, and a dashboard that updates every few seconds invites people, and systems, to react to swings that would not survive a significance test. The discipline that helps is to separate the speed of delivery from the judgment about whether a change means anything: compute whether a movement is statistically real before anyone acts on it. QuantumLayers builds its insights engine around that separation, computing significance and effect sizes deterministically so a change that reaches a decision is a verified movement rather than a live wobble that looks urgent because it is fresh. CDC can make your data arrive in seconds. It cannot tell you whether the thing that just moved is worth moving on.
When CDC earns its keep, and when batch still wins
Strip away the appeal of real-time and the decision comes down to whether the freshness is worth the operational contract you are signing.
Use CDC when seconds genuinely change a decision. Fraud checks, inventory that oversells, operational systems reacting to live state, and syncs where deletes must propagate correctly are the cases where log-based CDC pays for its overhead. If a human or a system acts on the data within minutes and being an hour stale would cause real harm, the pipeline earns its keep.
Default to log-based when you do use it, and coordinate with the source team. The deletes-captured property alone justifies log-based over the query-based alternative for most sources, but it means DBA cooperation and slot monitoring on Postgres are part of the deal, not an afterthought.
Reach for batch or incremental loads when the freshness is decorative. A great deal of analytics is consumed the next morning regardless of how fast it arrived, and for that, an hourly or nightly incremental load is cheaper to run, easier to reason about, and far less likely to page anyone at 3 a.m.
Do not put CDC on a table you could just reload. For a small reference table, a full nightly reload costs less to operate and reason about than any CDC mechanism. Streaming a thousand-row lookup table is effort spent buying freshness nobody asked for.
Start on one non-critical table. The teams that succeed validate the pipeline on something low-stakes first, learn the snapshot and schema and monitoring disciplines on data that will not hurt if it wobbles, then expand. Turning CDC on across the warehouse in one move is how the Saturday disk incident happens.
The bottom line
Change data capture is a genuinely powerful way to keep a warehouse in sync with the systems that feed it, and for the workloads that need live data it is the right tool. The trap is adopting it for the demo feeling rather than the requirement. Underneath the clean architecture diagram sit a snapshot handoff that loses data without a sound, a replication slot that can take down the database it reads, schema changes that rot the pipeline quietly, and a cost model that punishes exactly the high-volume sources CDC is meant for.
So decide on the requirement before the tooling. Put CDC where seconds actually change what someone does, run it log-based with the source team watching the slot, and send everything else through a batch load that will let the whole team sleep. Remember that arriving faster is not the same as being right, and that a real-time number still has to clear the bar of meaning something. Do that, and CDC becomes the quiet backbone it is supposed to be. Skip it, and the freshest data in the company will be the thing that pages you on a Saturday.
Lurika is an independent publication covering data analytics. We are not owned by any analytics vendor.
