Handling poison messages with retries and dead-letter topics in Boomi Event Streams

Your order.created events flow nicely from the e-shop to the warehouse, invoicing and the CRM. Then one morning an order arrives for customer C-1042, who exists in the e-shop but not yet in the ERP. The invoicing consumer fails, the event is redelivered, it fails again, and again. This article explains what such a poison message does to an event-driven integration, the retry and dead-letter pattern that contains it, and how to build it with Boomi Event Streams.

What a poison message does

In an event-driven setup, a consumer acknowledges an event after processing it. If processing fails and the event is not acknowledged, the broker delivers it again. That is exactly what you want for a temporary problem, such as the ERP being restarted. It is exactly what you don’t want for a permanent one:

The same event is retried endlessly, filling logs and wasting runtime capacity.
Depending on the broker and settings, it can hold up the events behind it, so one bad order delays every good one.
If someone “solves” it by acknowledging failures, the order silently disappears and is discovered when the customer asks for the invoice.

The pattern: classify, retry a little, then dead-letter

Classify the error

Not every failure deserves a retry. A useful split:

Kind	Examples	What to do
Transient	ERP timeout, HTTP 503, connection refused, lock on a record	Retry with a delay
Permanent	Customer not found, invalid VAT number, malformed JSON, unknown SKU	Dead-letter immediately

Retrying a permanent error five times only delays the inevitable and adds noise.

Retry a limited number of times

For transient errors, retry a small, fixed number of times with a growing delay (for example after 10 seconds, 1 minute, 5 minutes). If the error persists beyond that, treat it as permanent for now.

Move it aside: the dead-letter topic

A dead-letter topic (DLT) is an ordinary topic where failed events go, together with information about why they failed. Moving the event there lets the consumer acknowledge it and continue with the next one. Nothing is lost; it is parked.

Fix and replay

Someone looks at the dead-letter topic, fixes the cause (creates customer C-1042 in the ERP, corrects the mapping), and replays the event to the original consumer. Because consumers are idempotent, replaying an event that was in fact partly processed is safe.

orders.created ──► invoicing consumer ──ok──► ERP
                         │
                   fails N times / permanent error
                         ▼
             orders.created.invoicing.dlt ──► alert ──► fix ──► replay ──► invoicing consumer

Note that the DLT is per consumer. If invoicing fails but the warehouse succeeded, only invoicing needs to replay.

Building it in Boomi

Boomi Event Streams is built on Apache Pulsar, which has native support for redelivery and dead-letter topics on subscriptions. Depending on your release, the Boomi Event Streams connector’s listen operation may expose settings for redelivery and a dead-letter topic directly; check the current connector documentation for what is available and how it is labelled. The approach below builds the pattern explicitly in the consumer process, which works either way and gives you full control over error classification and the information stored with each failed event.

Step 1: Create the dead-letter topic

In Event Streams, in the same environment as orders.created, create a topic for the invoicing consumer’s failures, for example orders.created.invoicing.dlt, and a subscription on it named replay. Do the same for each consumer that needs one. A naming convention (<topic>.<consumer>.dlt) makes it obvious where a failed event came from.

Step 2: Wrap the work in a Try/Catch

In the invoicing consumer process (from Publish once, consume many: pub/sub with Boomi Event Streams), place a Try/Catch shape right after the Start shape, so that everything which talks to the ERP sits on the Try path.

Set the Try/Catch to catch errors per document, so one failing event doesn’t fail the others in the same execution.
Use the Try/Catch retry count for transient errors: the shape can re-run the Try path a few times before sending the document to the Catch path. Keep the number small.

For control over delays between attempts, some teams instead implement retries with a counter in a dynamic document property and a re-publish to a separate retry topic. Start simple and add that only if you need it.

Step 3: Classify on the Catch path

On the Catch path, the error message is available through the document’s try/catch message property. Use a Decision or Route shape to separate the known permanent errors (customer not found, validation errors from the ERP) from everything else. Both end up in the dead-letter topic, but you may want to alert differently: a permanent data error goes to the sales back office, a persistent technical error to IT.

Before your own validation, add checks on the Try path for the permanent errors you can predict, such as “customer exists in the ERP” or “every SKU is known”. Failing early with a clear message is far better than an obscure ERP error.

Step 4: Publish to the dead-letter topic

Still on the Catch path:

Use a Map or Message shape to wrap the original event with failure details:

{
  "failedAt": "2026-10-23T09:14:09Z",
  "consumer": "invoicing",
  "sourceTopic": "orders.created",
  "errorType": "permanent",
  "errorMessage": "Customer C-1042 not found in ERP",
  "attempts": 1,
  "event": { "eventId": "7f1c2e9a-4b1d-4c55-9a51-2d1f0b7c9e10", "type": "order.created", "orderId": "58213", "customerId": "C-1042" }
}

Send it with a Boomi Event Streams connector shape using a Produce operation on orders.created.invoicing.dlt.
End the Catch path normally, so the original event is acknowledged and the consumer moves on. Make sure the DLT publish itself succeeded first; if it fails, let the process error so the original event is redelivered rather than lost.

Step 5: Alert and inspect

Add a notification on the Catch path (for example an email or a message to a Teams/Slack channel via its connector) with the order number, customer and error message, not just “process failed”.
Watch the backlog of the replay subscription on the DLT in Event Streams. A non-empty DLT means work for someone.
Use Process Reporting to look at the failed execution’s documents when the message alone isn’t enough.

Step 6: Replay after fixing

Build a small replay process:

Start with the Event Streams connector consuming from orders.created.invoicing.dlt on the replay subscription. Many teams keep this listener stopped and start it only when they want to replay, or run the replay process manually, so events aren’t replayed before the cause is fixed.
Extract the original event from the wrapper with a Map shape.
Hand it back to the invoicing logic. The cleanest option is to publish it to a dedicated retry topic that only the invoicing consumer listens to, such as orders.created.invoicing.retry. Re-publishing to orders.created itself would deliver the event to the warehouse and CRM again; idempotency protects you, but it is noise.

Because the invoicing consumer checks whether the web order already exists before creating it, replaying an event that was partly processed doesn’t create a duplicate invoice.

Before you go live

Test with a real poison message. Send an order for a customer that doesn’t exist in the ERP and follow it to the DLT, the alert and back through the replay.
Set retention on DLTs deliberately. Events in a DLT must survive long enough for someone to fix them, including over a long weekend.
Give the DLT an owner. A dead-letter topic nobody looks at is just a slower way to lose orders.
Don’t dead-letter on a full outage. If the ERP is down, every event will fail. Consider stopping the listener instead, so events wait in the main subscription rather than flooding the DLT.
Log the event ID everywhere, so you can trace one order through producer, consumer, DLT and replay.

For the general error handling and retry patterns used in non-event integrations, see Error handling and retries in Boomi.

Summary

Poison messages are inevitable; lost or stuck orders are not. Classify errors, retry transient ones a few times, park permanent ones in a per-consumer dead-letter topic with a clear explanation, and replay once the cause is fixed. In Boomi, Event Streams holds the topics and Try/Catch plus the Event Streams connector implement the flow. For the concepts behind it, see Event-driven integration: what it is and when you need it.