Keeping business automations reliable

Start with one important outcome, such as an order reaching the fulfillment system. List the states an operator needs to understand: received, ready, in progress, completed, rejected and awaiting review. Decide which system is authoritative for each state. A successful HTTP response may only mean that a request was received; it does not prove the business action completed. Keep the incoming identifier and the destination identifier so the team can trace a case without searching every system manually.

For important events, require a durable record before telling the sender that delivery succeeded. Record enough information to process or reconcile the event, with appropriate access and retention controls. Then separate processing from receipt so a slow downstream service does not keep the incoming connection waiting indefinitely. Verify the actual behavior of each provider. Stripe, for example, documents repeated webhook delivery and events arriving out of order, so a design that assumes one perfectly ordered delivery would be unsafe for that integration.

Use a stable identifier to recognize an event that has already been processed. Also define the business action being protected: two different notifications about the same order should not create two fulfillment requests. An idempotency key is an identifier that lets a compatible receiving API recognize a repeated request as the same attempt. Confirm its scope and retention in that API. Keep this separate from an operator deliberately authorizing a genuinely new action.

A timeout means the caller did not receive a result. The other system may already have committed the change. Before retrying a write without duplicate protection, look up the original business record or reconcile the destination state. Retry temporary failures with a delay and a limit. Invalid data, revoked credentials and missing mappings need correction rather than endless retries. Show the operator whether work is definitely rejected, still pending or uncertain; those states require different decisions.

Create an exception view that explains the affected business record, the failing step and the last known result. Restrict replay to authorized people and preserve the original identifiers. Do not ask an operator to duplicate a customer record as a way to restart the flow. Record who made a correction and why. Monitor old pending work as well as explicit errors: a workflow that stops receiving events may appear quiet while orders accumulate elsewhere.

Before release, demonstrate duplicate delivery, delayed delivery, an unavailable destination, invalid input, a process restart and a timeout after a successful write. For each, record the expected business state and the operator’s next action. Compare source and destination totals or identifiers to find missing work. Confirm alerts reach a responsible person and that the recovery instructions work for someone other than the original builder. Choose review frequency and tolerable delay according to the process impact.