bridging streams and databases into feldera

the dangerous part of a streaming connector is not reading the next change.

it is deciding when the source is allowed to forget it.

in my postgres connector, that decision advances a confirmed lsn. in my kafka connector, it commits an offset for one topic partition.

both actions look like progress.

both can also lose data when they happen too early.

the rule i ended up using is simple:

the rule is shared. the source semantics are not.

postgres gives me transactions from the wal. kafka gives me independently ordered partition logs. treating them as the same system would hide the parts that matter most.

postgres starts with transaction boundaries

the postgres connector reads logical replication messages produced throughpgoutput.

those messages arrive individually, but the unit i care about is the committed transaction.

a simplified stream looks like this:

BEGIN
RELATION
INSERT
UPDATE
DELETE
COMMIT

when BEGIN arrives, i clear a transaction buffer.

as row changes arrive, i decode them and append them to that buffer. i do not send each row immediately.

when COMMIT arrives, i finally have the complete transaction that postgres made visible.

1. postgres begins the transaction.
2. rust collects and decodes the row changes.
3. postgres commits the transaction.
4. rust sends the complete transaction to feldera.
5. feldera processes the transaction.
6. rust confirms the ending lsn.

buffering between BEGIN and COMMIT is not just batching.

it preserves the transaction boundary.

if postgres commits three related changes, delivering only two before a crash would create a state that never existed in postgres.

the connector has to wait until the transaction is complete before treating it as deliverable.

the confirmed lsn is a claim

an lsn identifies a position in the postgres write ahead log.

the replication slot uses confirmed progress to track what the consumer has safely processed. once i advance that position, i am telling postgres that earlier wal does not need to be replayed for this connector.

that makes the acknowledgement order important.

this is wrong:

1. decode the transaction.
2. confirm end_lsn.
3. send the transaction to feldera.

if the process crashes after confirming the lsn but before delivery, postgres resumes after a transaction that feldera never received.

the decoder worked.

the replication connection worked.

the data was still lost.

the safer ordering is:

1. decode the complete transaction.
2. send the transaction to feldera.
3. wait for processing completion.
4. confirm end_lsn.

decoding proves that rust understood the bytes.

sending proves that rust attempted an http request.

neither proves that downstream processing completed.

then kafka changes the shape of progress

the kafka connector uses the same destination rule, but its source progress is not one wal position.

a kafka record is identified by:

topic
partition
offset

offset 40 in partition 0 and offset 40 in partition 1 are unrelated positions.

each partition has its own ordered log and its own committed progress.

partition 0: offsets 18, 19, 20
partition 1: offsets 52, 53, 54

this means the connector cannot maintain one global number calledlast_processed_offset.

it needs per-partition state.

(topic, partition): next safe position

records inside one partition must normally finish in order.

processing different partitions concurrently is useful. allowing offset 21 to complete before offset 20 in the same partition can be incorrect.

an insert followed by a delete is not equivalent to a delete followed by an insert.

json and avro meet in the middle

my kafka input has two payload paths.

one consumes json. the other consumes avro.

kafka itself does not care which format is inside the record. it stores bytes.

input: kafka record
payload type: &[u8]
decoder: selected by connector configuration

the json path turns the bytes into a typed change using json decoding.

source format: json
input: json bytes
decoder: json decoder
result: OrderChange

the avro path needs both the bytes and the correct writer schema.

source format: avro
input: avro bytes and writer schema
decoder: avro decoder
result: OrderChange

after decoding, the paths converge.

json result: OrderChange
avro result: OrderChange
destination representation: feldera request

this is where i wanted the connector architecture to stop caring about the source format.

json and avro fail differently. json exposes syntax and field type errors. avro can fail because the payload framing is wrong, the writer schema is missing, or the encoded record does not match the expected schema.

but once decoding succeeds, acknowledgement should follow the same policy.

the format should not decide when an offset becomes safe.

the kafka commit boundary

for one record, the flow is:

1. receive the record.
2. decode json or avro.
3. convert it into a destination event.
4. send it to feldera.
5. wait for processing completion.
6. commit the partition offset.

committing after recv() is too early.

committing after decoding is too early.

committing after the request leaves the process is too early.

the committed offset should represent completed external work.

a synchronous kafka commit only means the consumer waited for kafka to acknowledge the offset update.

it does not make the kafka commit and the feldera input one atomic operation.

that gap is where replay appears.

the failure both connectors share

the clean failure is easy.

the connector reads data, crashes before delivery, and leaves progress uncommitted.

postgres replays from the slot.

kafka redelivers from the committed partition position.

the difficult case starts when feldera succeeds but the connector cannot prove it.

the timeout does not tell me which part failed.

the request may never have reached feldera.

it may have reached ingress but failed during processing.

it may have completed, with only the response lost.

from the connector, those outcomes can look identical.

acknowledging the source would risk permanent loss.

not acknowledging causes replay when the first attempt actually succeeded.

the honest choice is replay.

that means duplicates are possible.

the postgres duplicate window

for postgres, the window looks like this:

1. the transaction is delivered.
2. feldera reports completion.
3. the connector crashes before confirming end_lsn.
4. postgres replays the transaction after restart.

the entire transaction can be delivered again.

transaction buffering prevents partial delivery before COMMIT. it does not make replay idempotent.

the transaction boundary and the acknowledgement boundary solve different problems.

the kafka duplicate window

for kafka, the same failure belongs to one partition position:

1. partition 2, offset 81 is delivered.
2. feldera reports completion.
3. the consumer crashes before committing the offset.
4. the group still resumes from offset 81.
5. offset 81 is delivered again.

multiple partitions make this harder.

at one moment, the connector may hold:

partition 0, offset 50: completed
partition 1, offset 82: waiting on http
partition 2, offset 14: failed avro decoding

those positions cannot be committed as one unit.

a rebalance can also revoke partition 1 while its request is still in flight. another consumer can receive the partition and replay from the last committed offset.

if the first request later completes, both consumers may have delivered work originating from the same record.

safe kafka acknowledgement therefore requires more than callingcommit_message.

it needs per-partition in-flight tracking, ordering, and a clear revocation policy.

completion is evidence, not coordination

feldera gives the connector a completion token for submitted input.

that is useful because a successful post and completed processing are not the same thing.

request accepted != input processed through the pipeline

i wait for the completion result before advancing the lsn or committing the kafka offset.

this gives me a stronger destination boundary.

it does not create a distributed transaction.

postgres does not know about the completion token.

kafka does not know about it either.

feldera does not atomically write the source checkpoint when it processes the input.

the connector is still joining two separate facts:

fact one: the destination completed this request
fact two: the source may now advance this position

a crash can always occur between them.

where the source models stop matching

postgres acknowledgement is transaction-oriented.

source unit: one committed postgres transaction
progress marker: one ending lsn

kafka acknowledgement is partition-oriented.

source unit: one ordered partition
progress markers: independently committed offsets

postgres requires me to preserve BEGIN and COMMIT.

kafka requires me to preserve per-partition ordering and survive group rebalances.

postgres can replay a complete transaction from the wal.

kafka can replay individual records from each partition.

the destination uncertainty is shared, but the recovery state is shaped by the source.

that distinction matters when moving from a working prototype to a connector that can survive failures without guessing.

what is still missing

the current rule prefers replay over loss.

that is the correct direction, but it is not exactly once delivery.

the postgres connector still needs a durable identity for each committed transaction, a coordinated snapshot to wal handoff, bounded handling for large transactions, and crash tests around every lsn acknowledgement point.

the kafka connector still needs explicit automatic commit settings, per-partition state, rebalance handling, bounded concurrency, a policy for invalid json and avro records, and crash tests around every offset commit.

both connectors need the destination to recognize that a replayed source position has already been applied.

the open question is this: what destination-visible identity can represent either a postgres transaction ending lsn or a kafka topic, partition, and offset, so that replay stays safe without applying the same change twice?

references