Debezium Without Kafka: Streaming CDC from MongoDB to GCP PubSub in Production

2026-08-08 CDCDebezium-serverMongoDB

Debezium Without Kafka: Streaming CDC from MongoDB to GCP PubSub in Production

Almost everything written about Debezium assumes Kafka Connect. You deploy a Connect cluster, you POST a connector JSON to /connectors, you get topics. That's the documented path and it's the one every tutorial takes. We don't run Kafka. We run Debezium Server — the standalone Quarkus distribution — as a StatefulSet on GKE, reading MongoDB change streams and publishing to Google PubSub. No Connect cluster, no schema registry, no ZooKeeper-shaped hole in the architecture. This post is not a walkthrough of every property in application.properties. Debezium's reference docs already do that, and better. This is about the handful of decisions that actually determine whether the thing survives contact with production — and the one that silently broke during a MongoDB major version upgrade.

The shape of it

MongoDB Replica Set
  └── Change Streams (oplog-backed)
        │
        ▼
Debezium Server (StatefulSet, 1 replica)
  ├── Source: MongoDbConnector
  │     ├── capture.mode   = change_streams
  │     ├── snapshot.mode  = initial
  │     └── Offset → /debezium/data/offsets.dat  (on a PVC)
  │
  ├── Transform: RegexRouter → single topic
  │
  └── Sink: Google PubSub (ordered by document _id)

One pod. One file on a persistent volume. That file is the most important object in the entire system, and it's about 200 bytes.

The offset file is the whole durability model

Debezium Server has no distributed state store. There is no __consumer_offsets topic, no Connect internal topics, no external coordination. There is one file: properties

debezium.source.offset.storage.file.filename=/debezium/data/offsets.dat

That file holds the MongoDB resume token — the last change stream position Debezium successfully published. On startup the logic is brutally simple:

  • File exists → resume from that token. No snapshot.
  • File missing → fall back to snapshot.mode. In our case initial, which means a full scan of every included collection.

So the PVC isn't a nice-to-have for logs or scratch space. It is the only thing standing between a pod reschedule and re-emitting several million documents into your sink. If you take one thing from this post: the volume is the system. Size it, back it up, and make sure nothing in your deployment pipeline recreates it. The second thing that falls out of the offset file is your delivery guarantee: properties

debezium.source.offset.flush.interval.ms=10000

The resume token is written every 10 seconds. If the pod dies at second 9, the events from those 9 seconds were already published to PubSub, but the token on disk still points before them. On restart, they get republished. That is at-least-once delivery, and it is not a bug you can configure away — you can only shrink the window, at the cost of more disk writes. Which means the real requirement lands on your consumers:

Every consumer must be idempotent. Upsert by _id. Never blind insert. Never increment a counter on receipt without a dedupe key.

If your downstream can't tolerate a duplicate, CDC is not your problem — your consumer design is.

Deltas, not documents: change_streams vs change_streams_update_lookup

This is the setting I'd argue about the longest, because the "convenient" option is a quiet load multiplier on your primary. MongoDB's change stream emits only the fields that changed on an update: json

{
  "op": "u",
  "after": {
    "status": "delivered",
    "updatedAt": "2026-05-03T10:00:00Z"
  },
  "source": { "collection": "deliveries" }
}

That's usually not what a downstream consumer wants. It wants the whole document. Debezium offers exactly that: properties

debezium.source.capture.mode=change_streams_update_lookup

With update_lookup, after every single update event Debezium issues a findOne against MongoDB by _id and embeds the full current document in after. Read that again with your write volume in mind. Our highest-traffic collection is delivery records, and a delivery gets its status updated repeatedly through its lifecycle. update_lookup means one additional read against the replica set for every one of those updates, forever, at peak, during incidents. You've taken a feature that was supposed to remove load from your database and used it to add a read amplification factor of 2. There's a subtler problem: the lookup happens after the event, so it fetches the document's state now, not its state at the moment of the change. Under rapid successive writes, the after you receive may not correspond to the op you received. It's a "full document" that isn't quite the document from that event. So we stayed on: properties

debezium.source.capture.mode=change_streams

Consumers receive the delta, and either apply it to state they already hold, or re-fetch from MongoDB themselves when they genuinely need the full document. The load moves to the consumers that actually need it, instead of being paid by everyone on every event. The honest trade-off: consumers are more work to write. That's the right place for the complexity to live. (For completeness, there's a third mode — oplog — which reads the raw replication log directly. It predates change streams and is deprecated. Don't.)

One StatefulSet per collection

The obvious way to add a fourth collection to CDC is to append it to collection.include.list and restart the pod. We don't do that. Every collection we capture gets its own StatefulSet, its own ConfigMap, its own PVC, and its own offset file. This looks like more YAML for no reason until the first time you need to replay one collection. With a shared connector, wiping the offset to backfill districts also re-snapshots deliveries — several million documents dumped into your sink because you wanted to fix a lookup table. With isolated workloads, a full replay of one collection is a contained operation that the other pipelines never notice. The same isolation applies to failure. A poison event, a schema surprise, an OOM on one collection's pod — none of it touches the others. And each pipeline gets its own resource footprint, which matters when one collection is three orders of magnitude busier than the rest. The cost is real: more objects to manage, more PVCs, more pods. For a handful of collections it's clearly worth it. At fifty collections you'd want to rethink it. We're not at fifty.

Pinning the topic name instead of inheriting it

By default Debezium derives the topic name from the source: <prefix>.<db>.<collection>. That couples your sink naming to your database naming — rename a collection or move a database and your topic name follows it. We pin it explicitly instead: properties

debezium.transforms=RerouteData
debezium.transforms.RerouteData.type=org.apache.kafka.connect.transforms.RegexRouter
debezium.transforms.RerouteData.regex=(.*)
debezium.transforms.RerouteData.replacement=<topic-name>

The regex (.*) matches whatever topic name Debezium generated and replaces it wholesale with a name we choose. Because each collection runs in its own StatefulSet with its own ConfigMap, each one sets its own replacement value — so every collection lands on its own topic, and every consumer gets its own subscription. That's the level of isolation we decided to go with, and it's the same reasoning as one StatefulSet per collection, carried through to the sink: one collection's retention policy, IAM binding, subscriber set, and backlog are independent of every other collection's. The alternative — pointing several connectors at one shared topic and having consumers filter on payload.source.collection — is fewer objects to provision, but it makes every consumer pay deserialization cost on messages it will discard, and it couples unrelated pipelines through a single backlog. We'd rather manage more topics than debug that. Worth noting alongside it: properties

debezium.sink.pubsub.ordering.enabled=true

PubSub ordering keys are set to the document _id, so all changes to a single document arrive in order. Not global ordering — per-document ordering, which is the guarantee that actually matters for CDC. Be aware that ordering keys constrain PubSub's parallelism; a single very hot document becomes a serialization point.

Enable pre-images on the collection first — and mind your MongoDB version

Before you deploy anything, there is a step on the database side that is easy to skip: you have to turn pre-images on, per collection. A change stream on its own tells you what a document looks like after a write. It does not tell you what it looked like before. For Debezium to populate the before field on updates and deletes, MongoDB has to be explicitly told to retain the previous version of each document for that collection. This is opt-in, and it is scoped to a single collection — not the database, not the cluster. Every new collection you bring into CDC needs it applied again. The catch is that the command is not the same across MongoDB versions. On MongoDB 5.x: js

db.runCommand({
  collMod: "<collection>",
  recordPreImages: true
})

From MongoDB 6.0 onward, recordPreImages was replaced by a new option that covers both pre- and post-images: js

db.runCommand({
  collMod: "<collection>",
  changeStreamPreAndPostImages: { enabled: true }
})

Same intent, different API. Which means the runbook you wrote against MongoDB 5 is quietly wrong the moment you upgrade the replica set — and if you're rolling 5.0 → 6.0 → 7.0 like we were, "enable pre-images on the collection" is a line in the upgrade checklist, not a one-time setup step you did once and forgot. Here's why it's dangerous: nothing fails loudly. Debezium keeps running. Events keep flowing. Pods stay green, lag stays flat, dashboards stay boring. The before field just becomes null on every update and delete, and any consumer that computes a diff — audit trails, "what changed" notifications, reconciliation jobs — silently starts producing wrong output. You find out days later from a business report, not from an alert. Two things came out of this:

  1. Pre-image configuration is part of the upgrade checklist, not an application-level detail. Re-apply it per collection after a major version bump and verify it.
  2. Add an assertion, not just a dashboard. A periodic check that before is non-null on a collection where you expect pre-images turns a silent data-correctness bug into a page.

Test the collection before you deploy anything

Before pointing Debezium at a new collection, prove that change streams are actually working on it. This is a throwaway Node.js script — it opens a change stream against one collection and prints every event it receives. Nothing to deploy, no Debezium involved. You're isolating the database layer so that if something is broken, you find out here rather than three components downstream. js

const { MongoClient } = require("mongodb");

async function monitorChanges() {
  const client = new MongoClient("mongodb://<host>:27017/?replicaSet=rs0");
  await client.connect();

  const collection = client.db("<db>").collection("<collection>");
  console.log("Listening for changes...");

  collection.watch().on("change", (change) => {
    console.log(JSON.stringify(change, null, 2));
  });
}

monitorChanges().catch(console.error);

Update a document in another terminal. If nothing prints, your problem is in MongoDB, not in Debezium — and it's much cheaper to find that out now.

Operational reality: forcing a full replay

The procedure for "resend everything from the beginning" is: delete the offset file. But the PVC is ReadWriteOnce, so you can't inspect or modify it while the pod is running. That's a general Kubernetes annoyance worth knowing beyond Debezium. 1. Scale to zero and wait for full termination. bash

kubectl scale statefulset debezium -n cdc --replicas=0
kubectl get pods -n cdc -w

Don't skip the wait. The volume won't detach until the pod is actually gone. 2. Mount the now-free PVC with a throwaway pod. yaml

apiVersion: v1
kind: Pod
metadata:
  name: pvc-debug
  namespace: cdc
spec:
  restartPolicy: Never
  containers:
    - name: shell
      image: alpine:3.18
      command: ["sh", "-c", "sleep 1d"]
      volumeMounts:
        - mountPath: /debezium/data
          name: data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: debezium-data-volume-debezium-0

3. Wipe the offset, including dotfiles. bash

kubectl exec -it pvc-debug -n cdc -- /bin/sh
find /debezium/data -mindepth 1 -delete
ls -la /debezium/data/
exit

find -mindepth 1 -delete empties the directory without removing the mount point, and catches hidden files that rm -rf /debezium/data/* misses. 4. Clean up and scale back up. bash

kubectl delete pod pvc-debug -n cdc
kubectl scale statefulset debezium -n cdc --replicas=1
kubectl logs -f -n cdc debezium-0

Debezium finds no offset, honors snapshot.mode=initial, and re-emits every document as an r (read) event. Tell your consumers before you do this. A full replay of a large collection is indistinguishable, from the consumer's side, from a very sudden and very large burst of traffic. Idempotency is what makes it a non-event instead of an incident.

What I'd tell someone starting this

  • Debezium Server is a legitimate choice if you don't already run Kafka. Don't stand up a Connect cluster for one connector. The operational surface of a single Quarkus pod is dramatically smaller.
  • Everything hinges on one file. Protect the volume the way you'd protect the database.
  • At-least-once is the contract. Idempotent consumers are not optional, they're the price of admission.
  • Don't buy convenience with primary reads. update_lookup is a load decision disguised as a formatting option.
  • Major version upgrades change CDC prerequisites, quietly. Re-verify pre-images, and assert on the data rather than trusting a green dashboard.

The failure mode that cost us the most wasn't a crash, an OOM, or a config typo. It was a command that stopped being the right command, on a system that never stopped reporting healthy.

← back to blog