Post cover image

October 14, 2026

Dead-Letter Queues in Kafka: How to Recover Poison Messages Without Losing Events

Build reliable retry pipelines with Java, Spring Boot, and Kafka. Learn how to handle poison messages, preserve business correctness, and…

By The Turtle Code

18 min read

Build reliable retry pipelines with Java, Spring Boot, and Kafka. Learn how to handle poison messages, preserve business correctness, and recover failed events safely.

Introduction: The Message That Stopped an Entire Partition

Imagine an order-processing service built with Spring Boot and Apache Kafka.

Customers place orders, and the application publishes an OrderPlaced event. A Kafka consumer receives the event, reserves inventory, and updates the order's state.

For thousands of orders, everything works.

Then one event arrives with an unexpected field format. The consumer throws an exception every time it processes that record.

The consumer retries the operation.

It fails again.

None

Other records behind it may now be delayed, consumer lag increases, and order processing falls behind.

The service is still running, but the business workflow is no longer progressing as expected.

The problem is not merely a malformed event. One unprocessable record has become a reliability problem for the entire event-processing pipeline.

This is where retry policies, dead-letter topics, idempotent consumers, and controlled replay become essential.

A dead-letter topic gives failed records a separate path for investigation and recovery. But sending a record to another topic is only one part of the design.

A production-grade implementation must answer several questions:

  • Which failures deserve another attempt?
  • When should retries stop?
  • What if publishing to the dead-letter topic fails?
  • How do we avoid duplicate business operations?
  • What happens to event ordering?
  • How can operators replay failed records safely?
  • How do we know that failed work is actually being resolved?

Let's build a practical approach using Kafka, Java, Spring Boot, and Spring for Apache Kafka.

1. What Is a Poison Message?

A poison message is a record that repeatedly fails processing under the current application logic or environment.

The failure may be permanent, temporary, or dependent on external state.

Common examples include:

  • Invalid JSON or incompatible event schemas.
  • Missing required business fields.
  • Unsupported enum values.
  • Database constraint violations.
  • Temporary database or HTTP failures.
  • An event that triggers a programming bug.
  • A payload that exceeds processing limits.

Not every failed message is a poison message.

A database timeout may disappear after a retry. Malformed JSON is unlikely to become valid without a correction.

The recovery policy must distinguish between these cases.

Why infinite retries are dangerous

Unlimited retries can create several problems:

  • One record repeatedly consumes CPU and network resources.
  • Later records in the same partition may be delayed.
  • Consumer lag grows.
  • Logs fill with duplicate stack traces.
  • A downstream outage becomes more severe.
  • Restarting the service repeatedly fails to address the underlying cause.

Retries are necessary, but they must be bounded and connected to a recovery strategy.

2. What a Dead-Letter Topic Actually Does

Kafka stores records in topics. A dead-letter topic, often abbreviated DLT, stores records that could not be processed successfully under the configured recovery policy.

A typical flow looks like this:

orders
   |
   v
Order Consumer
   |
   +---- Success -------> Commit progress
   |
   +---- Retryable -----> Retry policy
   |
   +---- Retries exhausted
                 |
                 v
             orders.DLT
                 |
                 v
          Investigate / Replay

A DLT does not repair a message.

It preserves the failed record and gives the team a place to investigate it without repeatedly blocking the original processing path.

What should a dead-letter record contain?

Preserve enough information to identify the original record and understand the failure:

  • Original topic, partition, and offset.
  • Original key and payload.
  • Original timestamp.
  • Failure category and timestamp.
  • Exception information, where appropriate.
  • Number of attempts, when available.
  • Event ID and schema version, when available.

Retain the original payload whenever policy permits. An exception message alone is rarely enough to investigate serialization or business-data failures.

DLQ versus retry topic

These mechanisms solve different problems.

Mechanism         | Purpose
------------------+--------------------------------------------------------------
Immediate retry   | Retry a short-lived failure quickly
Backoff retry     | Wait before another attempt
Retry topic       | Defer processing and retry later
Dead-letter topic | Isolate records after recovery attempts are exhausted
Replay workflow   | Reprocess selected failed records under controlled conditions

A retry topic may be part of the recovery process, but it is not necessarily the final destination for a permanently failed record.

3. Choose the Right Retry Strategy

The correct strategy depends on failure duration, throughput, ordering requirements, and business impact.

Immediate retries

The consumer retries the same record without a significant delay.

This may work for a brief database connection interruption, but it is a poor fit for sustained outages. Repeated requests can overload an already struggling dependency.

Retries with backoff

Backoff increases the delay between attempts.

For example:

Attempt 1: Immediately
Attempt 2: After 1 second
Attempt 3: After 2 seconds
Attempt 4: After 4 seconds
Then: Recovery policy

Add jitter when many consumers might retry simultaneously. Otherwise, clients may retry in synchronized waves and create a thundering herd.

Always define a maximum attempt count and maximum delay.

Retry topics

Retry topics defer records to separate Kafka topics for processing after a delay.

orders
   |
   v
Consumer
   |
   +---- Success ------------------> Done
   |
   +---- Failure --> orders.retry.1s
                            |
                            v
                      Retry Consumer
                            |
                            +---- Success --> Done
                            |
                            +---- Failure --> orders.DLT

This approach can free the original consumer to process other records while the failed record waits.

However, retry topics can change business processing order. A later event may be processed before an earlier event that was delayed.

That matters for account balances, inventory, and order state transitions.

When should consumption pause?

Sometimes the problem is not one bad record. A critical database or downstream service may be unavailable for the entire consumer group.

Moving every valid record to retry topics during a major outage may generate unnecessary traffic.

Consider a circuit breaker, controlled pause, or backpressure when a shared dependency is unhealthy.

The strategy should distinguish between an individual poison message and a system-wide failure.

4. Classify Failures Before Configuring Spring Kafka

A practical policy might look like this:

Failure                       | Typical response
------------------------------+---------------------------------------------------
Temporary database timeout    | Retry with bounded backoff
HTTP 429                      | Retry according to rate-limit guidance
Temporary HTTP 503            | Retry with bounded backoff
Invalid JSON                  | Dead-letter
Unsupported event version     | Dead-letter or route to a compatible handler
Business validation failure   | Reject or dead-letter according to business policy
Invalid shared credentials    | Alert and consider pausing processing
Duplicate event               | Apply idempotent handling
Unknown programming exception | Bounded retry, alert, and investigate

These are starting points, not universal rules.

A 401 Unauthorized error may indicate a shared credential problem rather than a permanently invalid event. Automatically dead-lettering every affected record can hide a larger incident.

Likewise, a deployment bug may cause thousands of otherwise valid events to fail. Recovery must account for the scale and nature of the failure.

Avoid broad rules such as "retry every exception forever" or "dead-letter every exception immediately."

5. Configure Dead-Letter Handling in Spring Kafka

Spring for Apache Kafka provides components for retrying failed records and publishing exhausted failures to a dead-letter topic.

A common record-listener configuration uses:

  • DefaultErrorHandler for retry and recovery behavior.
  • DeadLetterPublishingRecoverer to publish failed records.
  • KafkaTemplate for producing records.
  • FixedBackOff or another backoff policy.

The following example illustrates the approach. Verify exact APIs and configuration against the Spring Kafka version managed by your Spring Boot application.

Add the dependency

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

Use Spring Boot's dependency management unless you have a deliberate reason to override the version.

Create the dead-letter topic

A conventional naming scheme is:

orders
orders.DLT

For a source topic named orders, Spring Kafka's DeadLetterPublishingRecoverer commonly uses orders.DLT as its default destination convention.

The default behavior can also retain the original partition number when the destination partition exists.

Create the DLT deliberately, with suitable partitions, replication, permissions, and retention. Do not assume every Kafka environment automatically creates it.

Configure the error handler

@Configuration
public class KafkaErrorHandlingConfiguration {

    @Bean
    DefaultErrorHandler kafkaErrorHandler(
            KafkaTemplate<Object, Object> kafkaTemplate) {

        DeadLetterPublishingRecoverer recoverer =
                new DeadLetterPublishingRecoverer(kafkaTemplate);

        DefaultErrorHandler handler =
                new DefaultErrorHandler(
                        recoverer,
                        new FixedBackOff(1_000L, 2L)
                );

        handler.addNotRetryableExceptions(
                IllegalArgumentException.class
        );

        return handler;
    }
}

This configures two retries after the initial failed attempt, with a one-second fixed backoff.

The record can therefore be attempted three times in total: the initial attempt plus two retries.

After retries are exhausted, the recoverer publishes the record to the DLT.

The example treats IllegalArgumentException as non-retryable. Use this classification only if it reliably represents permanent input failures in your application.

Configure the listener container

For an explicit listener container factory, register the handler:

@Bean
ConcurrentKafkaListenerContainerFactory<Object, Object>
kafkaListenerContainerFactory(
        ConsumerFactory<Object, Object> consumerFactory,
        DefaultErrorHandler kafkaErrorHandler) {

    var factory =
            new ConcurrentKafkaListenerContainerFactory<Object, Object>();

    factory.setConsumerFactory(consumerFactory);
    factory.setCommonErrorHandler(kafkaErrorHandler);

    return factory;
}

Integrate this configuration with your existing factory if one already exists.

Ensure the key and value types match your actual serializers and producer configuration. A recovery path that cannot serialize the failed record is not a reliable recovery path.

Create a thin listener

@Component
public class OrderEventListener {

    private final OrderService orderService;

    public OrderEventListener(OrderService orderService) {
        this.orderService = orderService;
    }

    @KafkaListener(
            topics = "orders",
            groupId = "order-processor"
    )
    public void consume(OrderEvent event) {
        orderService.process(event);
    }
}

The listener stays focused on business processing.

Retry and recovery are managed by the container rather than by manually catching every exception inside the listener.

6. Handle Deserialization Failures

Not every failure reaches the listener as a normal Java exception.

If Kafka cannot deserialize a record, the listener may never receive an OrderEvent instance.

Spring Kafka provides ErrorHandlingDeserializer to expose deserialization failures to the container's error-handling machinery.

For a JSON value and string key, configuration may look like this:

spring:
  kafka:
    consumer:
      key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring.deserializer.key.delegate.class: org.apache.kafka.common.serialization.StringDeserializer
        spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer
        spring.json.value.default.type: com.example.events.OrderEvent
        spring.json.trusted.packages: com.example.events

Adapt the delegate deserializers and JSON settings to your event contract and Spring Kafka version.

Do not trust arbitrary type headers or deserialize untrusted data into unrestricted Java classes.

Preserve malformed payloads

For malformed records, the original bytes may be essential for investigation.

Your recovery workflow must handle records that cannot be deserialized into the normal domain class.

Test this with genuinely malformed payloads. A configuration that handles business exceptions may behave differently for deserialization failures.

7. Design the DLT as a Production Interface

A dead-letter topic is not merely an implementation detail. It becomes an operational interface between the consumer, the recovery process, and the engineers responsible for failed work.

Partitioning

If your recoverer retains the source partition number, the DLT must have the corresponding partition for every source partition that may be routed to it.

For example, a 12-partition source topic generally needs at least 12 destination partitions for a one-to-one partition mapping.

Verify the actual routing behavior and topic configuration.

The DLT does not preserve the complete processing order of the original topic. A failed record may arrive after later records have already succeeded.

Keys and headers

Preserve the original key where appropriate. It may represent an order ID, account ID, or another partitioning key.

Spring Kafka's dead-letter support adds origin and exception metadata. The exact headers depend on the version and configuration.

Test the actual metadata produced by your deployed version instead of relying on undocumented assumptions.

Retention and ownership

Define:

  • Retention and cleanup.
  • Access permissions.
  • Investigation ownership.
  • Replay approval.
  • Payload redaction.
  • Audit requirements.

Too-short retention may delete records before investigation. Unlimited retention may keep sensitive data longer than necessary.

A DLT with no operational owner simply becomes a backlog of unfinished work.

8. The Critical Failure: DLT Publication Fails

Suppose the consumer fails to process a record and attempts to publish it to orders.DLT.

But the broker is unavailable, authentication fails, or the destination topic cannot accept the record.

What happens to the source record?

If the consumer commits progress despite failed recovery publication, the record may be lost from the application's processing path.

Calling a recoverer does not automatically mean recovery succeeded.

The recovery process must verify publication according to the application's delivery requirements.

Configure and test how your Spring Kafka version handles asynchronous send failures. Ensure publication failures are surfaced to the container and are not silently treated as successful recovery.

When recovery fails, the original record should remain eligible for redelivery according to the configured error-handler and acknowledgment semantics.

Understand the acknowledgment boundary

The intended sequence is conceptually:

Processing fails
       |
       v
Publish record to DLT
       |
       v
Confirm recovery publication
       |
       v
Allow source progress to advance

If the DLT publication succeeds but the source offset is not committed, the record may be delivered again and published again.

This creates duplicates rather than loss.

Therefore, design for both failure modes:

  • Losing a record because progress advances too early.
  • Publishing a duplicate because progress does not advance after recovery.

Include the original topic, partition, and offset in the recovery metadata. Together, these identify the original record within its source topic's history.

Producer idempotence helps prevent certain duplicate writes caused by producer retries. It does not make the entire consumer-to-DLT workflow atomic.

9. Kafka Transactions and Exactly-Once Processing

Kafka transactions can atomically publish Kafka records and commit consumed offsets when the application is configured appropriately.

They are useful for Kafka-to-Kafka processing and certain transactional recovery workflows.

However, a Kafka transaction cannot automatically roll back a PostgreSQL update or an external HTTP request.

Consider:

1. Database update succeeds.
2. Application crashes.
3. Kafka offset is not committed.
4. Kafka redelivers the record.

The database update may happen twice unless the operation is idempotent.

Use a database inbox or deduplication table

Store a processed-event identifier in the same database transaction as the business update.

CREATE TABLE processed_events (
    consumer_name VARCHAR(100) NOT NULL,
    event_id UUID NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (consumer_name, event_id)
);

Then process the event transactionally:

@Transactional
public void process(OrderEvent event) {
    boolean firstProcessing =
            processedEventRepository.insertIfAbsent(
                    "order-processor",
                    event.eventId()
            );

    if (!firstProcessing) {
        return;
    }

    orderRepository.applyOrderEvent(event);
}

insertIfAbsent must use an atomic database operation, such as an insert with conflict handling. A separate read followed by an insert is vulnerable to concurrent races.

The deduplication insert and business update must share the same database transaction. If the business update fails, the deduplication record must roll back too.

This protects against duplicate processing of the same logical event, assuming the event ID is stable and unique.

External side effects need their own protection

For payments, emails, and external API calls, database deduplication alone may not prevent duplicate effects.

Use the downstream provider's idempotency mechanism where available. Otherwise, design a durable intent, reconciliation, and retry process appropriate to the operation.

If a database change must reliably trigger a Kafka event, the transactional outbox pattern can help. Store the business change and outbox record in the same database transaction, then publish the outbox record separately.

Be precise about guarantees. "Exactly once" should describe a specific boundary, not a blanket promise about the entire distributed application.

10. Retry Topics Versus Blocking Retries

Spring Kafka supports non-blocking retry-topic workflows that defer failed records without keeping the original consumer occupied.

A simplified annotation-based configuration looks like this:

@RetryableTopic(
        attempts = "4",
        backoff = @Backoff(
                delay = 1_000,
                multiplier = 2.0
        ),
        dltTopicSuffix = "-dlt"
)
@KafkaListener(
        topics = "orders",
        groupId = "order-processor"
)
public void consume(OrderEvent event) {
    orderService.process(event);
}

@DltHandler
public void handleDeadLetter(OrderEvent event) {
    deadLetterAuditService.record(event);
}

This illustrates the basic model. Exact attributes, topic creation, serialization behavior, and handler signatures depend on the Spring Kafka version and application configuration.

Do not casually layer this retry-topic mechanism on top of an independent blocking-retry policy. Overlapping retry mechanisms can make attempt counts and final recovery behavior difficult to reason about.

When retry topics make sense

Use them when failures need a meaningful delay, the original consumer should continue processing, and the business can tolerate the ordering implications.

When blocking retries make sense

Use blocking retries when failures are likely to be brief, retry counts are small, and partition ordering matters more than uninterrupted progress through later records.

Understand ordering changes

Consider a partition containing:

OrderCreated
PaymentAuthorized
OrderShipped

If PaymentAuthorized moves to a retry topic, OrderShipped may be processed first.

The original topic's order has not changed, but the order of business effects has.

For sequence-sensitive workflows, consider aggregate-version checks, state-transition validation, per-key sequencing, or pausing later events for the affected aggregate.

Kafka partition ordering alone does not guarantee business ordering after records are split across independent retry paths.

11. Build a DLT Inspection Workflow

A dead-letter topic is useful only if failed records can be investigated and acted upon.

A dedicated inspection workflow might look like this:

orders.DLT
     |
     v
Inspection Consumer
     |
     +----> Failure metadata store
     |
     +----> Alerting / incident system
     |
     +----> Recovery dashboard

The inspection consumer might store the original topic, partition, offset, failure category, and timestamp.

Avoid copying sensitive payloads into additional systems unless there is a justified operational need.

Avoid dead-letter loops

If the DLT consumer fails and uses the same recovery strategy, it may send records to another DLT indefinitely.

Define what happens when inspection fails. Depending on the importance of the audit record, the consumer may pause, retry within a limit, or use a separately monitored operational path.

Record a stable failure category

Useful categories include:

INVALID_SCHEMA
BUSINESS_VALIDATION
DEPENDENCY_TIMEOUT
DATABASE_CONSTRAINT
UNKNOWN_PROCESSING_ERROR

Avoid using raw exception messages as metric labels or database keys. They may contain sensitive values and create high-cardinality metrics.

12. Replay Failed Messages Safely

Replaying a record is not simply copying it back into the original topic.

The event may have failed because of invalid data, a software defect, a temporary outage, or a business rule that has since changed.

A safe replay workflow

  1. Identify the failed record and its original metadata.
  2. Investigate the failure and root cause.
  3. Correct the application, configuration, or data issue.
  4. Decide whether the original payload can be replayed unchanged.
  5. Select a bounded set of records.
  6. Check whether replay could violate business invariants.
  7. Publish through a controlled replay mechanism.
  8. Record who initiated the replay and why.
  9. Monitor the resulting processing outcomes.
  10. Confirm that the records reached an acceptable terminal state.

For high-impact business events, require approval before replaying large batches.

Why blind replay is dangerous

Suppose a payment event failed because a downstream service timed out.

The payment provider may still have completed the payment even though your application never received the response.

Blindly replaying the event could initiate a second charge.

Check the current business state and use a stable idempotency key before repeating irreversible operations.

Preserve original identity

Replay metadata might include:

replayId
replayedAt
replayedBy
replayReason
originalTopic
originalPartition
originalOffset

Keep the original event ID unchanged. Use a new replay-operation ID for each replay attempt.

Publishing an old record back to a live topic does not restore its original offset. The replayed record normally receives a new offset, so later events may already have changed the aggregate state.

For strict ordering, consider an isolated replay consumer, coordinated processing for the affected aggregate, or rebuilding state before resuming live traffic.

13. Make the Business Operation Idempotent

At-least-once delivery means a record may be processed more than once.

This can happen after a crash, a failed offset commit, a DLT publication followed by redelivery, or an operator replay.

An event should carry a stable ID:

public record OrderEvent(
        UUID eventId,
        UUID orderId,
        String eventType,
        long aggregateVersion
) {
}

The event ID identifies the logical event. The aggregate version can help enforce sequence constraints.

Do not generate a new event ID every time the consumer receives the same message.

Deduplication is not enough for every operation

Two distinct events can represent the same business intent.

For example, a producer bug may create two different event IDs for the same payment request. Event-ID deduplication will not catch that.

For payments, use a business idempotency key and the provider's supported mechanism. For state transitions, validate the current aggregate version and state.

The right idempotency boundary depends on what must happen only once.

14. Keep Business Logic Separate From Recovery

Avoid mixing business processing, retries, DLT publication, and incident management inside one listener method.

Prefer clear responsibilities:

Kafka Listener
     |
     v
Business Service
     |
     +----> Validate event
     |
     +----> Apply business transaction
     |
     +----> Return success or raise failure

Container Error Handler
     |
     +----> Retry
     |
     +----> Recover to DLT

DLT Workflow
     |
     +----> Audit
     |
     +----> Alert
     |
     +----> Support controlled replay

The business service owns domain rules and database consistency. The container error handler owns delivery recovery. The operational workflow owns investigation and replay.

Never swallow processing exceptions

This is dangerous:

@KafkaListener(topics = "orders")
public void consume(OrderEvent event) {
    try {
        orderService.process(event);
    } catch (Exception ex) {
        log.error("Processing failed", ex);
    }
}

The exception is swallowed. The listener returns normally, so the container may consider the record successfully processed under the configured acknowledgment semantics.

The retry handler may never see the failure.

If processing fails, propagate the exception or invoke an explicit recovery policy that preserves the intended delivery guarantees.

15. Test the Complete Failure Lifecycle

A production DLT design needs integration tests, not just a unit test that verifies the recoverer was constructed.

Transient failure

Make the consumer fail twice and succeed on the third attempt.

Verify that the expected number of attempts occurs, the business effect happens once, and the record does not reach the DLT.

Permanent failure

Publish a record that fails deterministic validation.

Verify that it is not retried unnecessarily, the original payload and key are preserved, and origin metadata is available.

DLT publication failure

Simulate a failure while publishing to the DLT.

Verify that the source record is not silently skipped and the failure is observable. Test what happens when the DLT becomes available again.

Malformed payload

Publish bytes that cannot be deserialized into the expected event type.

Verify that the recovery path retains the original bytes and that the inspection tooling can handle the record safely.

Duplicate delivery

Force a failure after the database transaction commits but before source progress is committed.

When Kafka redelivers the record, verify that deduplication prevents duplicate business effects.

Ordering and replay

Publish multiple events for the same key. Force one event to fail while later events are available.

Verify the actual ordering behavior of the selected retry strategy and ensure the business state remains valid.

Testcontainers can help exercise real Kafka behavior and a test database rather than relying exclusively on mocks.

16. Monitor Recovery, Not Just Consumer Lag

A consumer can successfully route failures to the DLT while business work remains unfinished.

Its source lag may return to normal even though thousands of orders still need investigation.

Monitor the recovery pipeline separately.

Metric                        | Why it matters
------------------------------+-----------------------------------
Consumer lag                  | Measures normal processing backlog
DLT records produced          | Reveals exhausted failures
DLT backlog                   | Shows accumulated recovery work
Oldest DLT record age         | Identifies unresolved failures
Retry attempts by category    | Highlights recurring failures
DLT publication failures      | Detects recovery-path problems
Replay success rate           | Measures recovery effectiveness
Duplicate processing rate     | Helps identify delivery issues
Processing latency by outcome | Reveals the cost of recovery

Use bounded labels such as consumer name, outcome, and failure category. Avoid event IDs, order IDs, and raw exception messages as metric labels.

Useful alerts include sudden DLT volume increases, old unresolved records, recovery publication failures, and repeated replay failures.

A routine invalid input may not need an on-call page. A sudden wave of valid orders failing after a deployment may indicate a production incident.

17. Secure Dead-Letter Topics

Failed records can contain the same sensitive information as normal events.

Treat DLTs as protected data stores.

  • Restrict read, publish, replay, and delete permissions.
  • Separate investigation permissions from replay permissions where practical.
  • Avoid copying sensitive payloads into unrestricted logs.
  • Limit diagnostic headers and exception details.
  • Apply appropriate encryption and retention.
  • Audit replay operations.
  • Require approval for sensitive or large-scale replays.

Deleting a DLT record does not necessarily remove copies from backups, logs, monitoring systems, or exported incident records.

18. A Production Architecture

A complete order-processing recovery path might look like this:

Kafka: orders
                           |
                           v
                 Spring Boot Consumer
                           |
                           v
                  Business Processing
                           |
                +----------+----------+
                |                     |
                v                     v
             Success                Failure
                |                     |
                v                     v
         Commit Progress        Classify Failure
                                      |
                          +-----------+-----------+
                          |                       |
                          v                       v
                    Transient Error         Permanent Error
                          |                       |
                          v                       v
                    Bounded Retry             Dead-Letter
                          |                       |
                  +-------+-------+               v
                  |               |         Inspection / Audit
                  v               v               |
               Success         Exhausted          v
                  |               |          Controlled Replay
                  v               v
             Commit Progress  Publish to DLT
                                  |
                                  v
                            Confirm Recovery
                                  |
                                  v
                            Commit Progress

This is a conceptual flow. Actual commit boundaries depend on acknowledgment mode, container configuration, transactions, and recovery implementation.

When the database is unavailable, the service may retry within a defined budget or pause consumption if the outage affects the whole system.

When the event is malformed, it can be routed to the DLT with its original payload and useful metadata.

When DLT publication fails, the consumer must not silently treat the record as recovered.

When replay is requested, the workflow must preserve event identity and enforce normal validation and idempotency controls.

19. Common Mistakes to Avoid

Retrying every exception indefinitely. This can block partitions and amplify outages.

Dead-lettering every exception immediately. Temporary failures become unnecessary operational work.

Swallowing exceptions. The container may treat failed processing as successful.

Ignoring deserialization errors. Some failures occur before the listener receives a domain object.

Assuming DLT publication cannot fail. Recovery has its own dependencies and failure modes.

Committing progress before recovery is durable. The source record may be skipped even though the DLT write failed.

Assuming Kafka transactions cover PostgreSQL. Cross-system side effects need explicit consistency design.

Using retry topics without considering ordering. Later events may be processed before earlier failed events.

Replaying every record blindly. Some operations can produce duplicate payments or invalid state transitions.

Monitoring only the original consumer group. Source lag can look healthy while the recovery backlog grows.

20. Production Readiness Checklist

Before deploying a dead-letter workflow, confirm that:

  • [ ] Transient, permanent, and system-wide failures are classified.
  • [ ] Retry attempts and backoff are bounded.
  • [ ] Retry-topic ordering implications are understood.
  • [ ] DLT partitions and retention are configured deliberately.
  • [ ] Original keys, payloads, and record identity are preserved.
  • [ ] Recovery publication failures are surfaced.
  • [ ] The source record is not silently skipped when recovery fails.
  • [ ] Business processing is idempotent where required.
  • [ ] External side effects have appropriate idempotency controls.
  • [ ] Replay operations are authorized, audited, and bounded.
  • [ ] DLT backlog and oldest-record age are monitored.
  • [ ] Deserialization failures are tested.
  • [ ] Duplicate delivery and publication failure are tested.
  • [ ] A team owns investigation and recovery.

Frequently Asked Questions

What is the difference between a Kafka DLQ and a DLT?

A dead-letter queue is a general term for holding failed messages separately. Kafka commonly implements this with a dead-letter topic, which stores failed records in another Kafka topic.

How many times should a message be retried?

There is no universal number. Choose a retry budget based on dependency recovery time, processing cost, and business latency requirements.

Should every failed Kafka message go to the DLT?

No. Transient failures generally deserve bounded retries. System-wide failures may require pausing consumption or raising an incident. Permanent input failures may be dead-lettered or handled as explicit business rejections.

Does publishing to the DLT guarantee that a message cannot be lost?

Not automatically. The application must verify recovery publication and coordinate it with source offset progress. Test publication failure and crashes between publication and offset commitment.

Does the DLT preserve ordering?

It can retain origin metadata, but it does not preserve the complete processing order of the original topic. Retry topics and replay can allow later events to be processed first.

Do Kafka transactions eliminate duplicate database updates?

No. Kafka transactions can atomically combine Kafka output records and consumed offsets under suitable configuration. Database updates and external API calls require their own consistency and idempotency strategy.

Should DLT records be replayed automatically?

Only when the failure category and business semantics make replay safe. Invalid data may require correction, and payment events may require checking the provider's state before another attempt.

Are retry topics always better than blocking retries?

No. Retry topics can free the original consumer, but they may change ordering. Blocking retries may be preferable for short failures when partition order is important.

Conclusion: A Dead-Letter Topic Is a Recovery Contract

Kafka makes it possible to process events at scale, but distributed systems still encounter malformed records, unavailable dependencies, incompatible schemas, and application defects.

A dead-letter topic gives those failures a controlled destination. The difficult engineering work lies in what happens around it.

Retries must be bounded. Failures must be classified. DLT publication must be verified. Offset progress must follow the configured recovery guarantees. Business operations must tolerate duplicate delivery. Replay must respect ordering and current business state.

The recovery pipeline also needs monitoring, access controls, retention policies, and an accountable owner.

Most importantly, moving a message to another topic is not the same as resolving the underlying business problem.

Reliable event processing is not the absence of failures. It is the ability to detect failures, contain their impact, preserve what is needed for recovery, and restore correct processing safely.

That is what turns a dead-letter topic from a convenient Kafka feature into a production-grade reliability mechanism.