Webhook failures are rarely caused by one mysterious problem. This guide gives you a repeatable way to inspect requests, separate transport failures from application errors, verify signatures safely, understand retries and duplicates, and replay a delivery without losing the evidence needed to fix it.
Overview
A webhook is an HTTP request sent by one system when an event occurs. The receiving endpoint must accept the request, authenticate it, parse its payload, perform the required work, and return an appropriate response within the sender’s delivery limits. A failure at any one of those stages can appear as a generic “delivery failed” message.
Start by treating every delivery as an observable transaction. Record the event identifier, delivery attempt, request timestamp, target URL, HTTP method, response status, response body, and elapsed time. Compare that information with the receiver’s application logs and infrastructure logs. The goal is to answer four questions:
- Did the request reach the expected network boundary?
- Did the endpoint return a response, and what kind?
- Did authentication and payload validation succeed?
- Was the event processed once, more than once, or not at all?
Keep provider-specific behavior separate from application behavior. Signature headers, timestamp formats, retry schedules, timeout limits, event schemas, and replay controls differ between services. Confirm those details in the sender’s current documentation rather than copying assumptions from another integration.
For related HTTP investigation, use the HTTP status code troubleshooting matrix. If the payload is JSON, a jq workflow or a carefully selected JSON formatter can make comparisons easier. Do not paste live secrets, tokens, or personal data into an untrusted online tool; the guide to choosing online developer utilities safely provides a useful review checklist.
Checklist by scenario
The webhook is rejected or returns an error
- Confirm the URL. Check the scheme, hostname, path, port, and environment. A production sender pointed at a staging endpoint can look like an application failure.
- Check DNS and TLS. Verify that the hostname resolves correctly and that the certificate is valid for the requested host. If a proxy, gateway, or load balancer sits in front of the application, inspect its logs too.
- Inspect the status code. A client-side error usually indicates a request, authentication, routing, or validation problem. A server-side error points toward the receiver or an upstream dependency. A redirect may be a problem even when a browser appears to follow it successfully.
- Compare the request contract. Check the method, required headers, content type, body encoding, and expected event fields. Do not assume that a request accepted by a local test endpoint matches the production payload.
- Read the response body and server logs. A generic sender dashboard message is less useful than the receiver’s validation error, exception, or gateway record.
When status codes are involved, record the exact response rather than summarizing it as “the API failed.” Status, headers, response body, and timing often distinguish an authentication problem from a routing or timeout problem.
Signature verification fails
- Preserve the raw body. Many schemes sign the exact bytes received, not a parsed and re-serialized JSON object. Verify the signature before transforming the body.
- Use the correct secret. Check the environment, endpoint, account, and key version. During rotation, determine whether the sender expects one active secret or supports a controlled overlap.
- Match the provider’s signed message format. The input may combine a timestamp and raw body, or use another defined construction. Do not substitute a familiar HMAC pattern without confirming the specification.
- Check encoding and comparison behavior. Hex and Base64 representations are not interchangeable. Use a constant-time comparison where appropriate, and reject malformed values without exposing diagnostic secrets.
- Validate the timestamp window. If the scheme includes a timestamp, check clock synchronization, units, parsing, and the allowed tolerance. A valid signature with an old timestamp may still need to be rejected.
Log safe metadata such as the event ID, signature header name, timestamp age, and verification result. Avoid logging the secret, complete authorization headers, or unredacted sensitive payloads.
Deliveries are duplicated
Assume that a webhook may be delivered more than once. Retries can follow a timeout, a connection reset, a non-success response, or an intermediary failure even when the receiver eventually completed the work. Build idempotency into the handler:
- Extract a stable event or delivery identifier.
- Store a processing record with a uniqueness constraint.
- Return the same safe outcome for a recognized event instead of performing the side effect again.
- Separate “received” from “completed” if processing is asynchronous.
- Keep enough status to retry a failed job without creating a second charge, ticket, deployment, or notification.
Do not rely only on an in-memory flag. Multiple application instances, restarts, and delayed retries require a durable coordination mechanism suited to the workflow.
Events arrive late or not at all
- Check the sender’s delivery history and retry state.
- Compare the event creation time, delivery time, and receiver processing time.
- Look for queue backlogs, worker failures, rate limits, maintenance windows, and dependency errors.
- Confirm that filtering rules, event subscriptions, and environment settings include the event type being tested.
- Determine whether the sender supports replay or manual redelivery, and use it only after making the receiver safe to run more than once.
Late delivery can be a normal consequence of retries or asynchronous processing, while a missing delivery may indicate filtering, a disabled endpoint, an exhausted retry policy, or an event that was never emitted. Treat those as separate hypotheses.
The payload cannot be parsed
Check the declared Content-Type, character encoding, compression, and body size before investigating business logic. Capture the raw request in a controlled test environment and compare it with the parser’s expected input. Common mismatches include form-encoded data treated as JSON, an empty body caused by middleware, invalid escaping, unexpected nulls, and schema changes that the receiver treats as fatal.
Validate required fields explicitly and decide how unknown fields should be handled. A forward-compatible consumer often ignores fields it does not need while still rejecting missing fields that are essential to safe processing. Use an OpenAPI or schema validation workflow where it fits the integration; the OpenAPI validation guide covers a structured way to compare request and response contracts.
What to double-check before replaying a request
A replay is useful only when it preserves the evidence and does not create an unintended side effect. Follow this sequence:
- Capture the original details. Save a redacted copy of the method, URL, relevant headers, raw body, event ID, response, and timestamps.
- Choose a safe destination. Prefer a local, staging, or diagnostic endpoint. If production is unavoidable, confirm idempotency and understand what the event will trigger.
- Recreate the body exactly. Avoid formatting or parsing JSON between capture and verification if the signature depends on raw bytes.
- Generate a fresh signature when required. Some schemes bind the signature to a timestamp, so an old header may correctly fail during replay. Use the documented test or signing process instead of weakening verification.
- Preserve the original event ID intentionally. Keeping it helps reproduce deduplication behavior; changing it may test a new-event path. Record which behavior you selected.
- Observe every layer. Watch the edge proxy, application logs, queue, worker, database, and downstream service. A successful HTTP response does not prove that asynchronous work completed.
- Compare outcomes. Note what changed between the failed and successful attempt: URL, headers, body, signature, timing, deployment, configuration, or dependency state.
A minimal diagnostic request might look like this, but replace every placeholder and follow the sender’s authentication rules:
curl -i -X POST 'https://example.test/webhooks/events' \
-H 'Content-Type: application/json' \
-H 'X-Event-Id: example-event-id' \
--data-binary @payload.jsonUse --data-binary when preserving the exact file bytes matters. Never place a real secret in shell history, shared tickets, or copied command output.
Common mistakes
- Returning success before processing safely. A fast response can prevent retries, but acknowledging work that was not durably recorded can lose events.
- Returning an error after completing the side effect. The sender may retry, producing a duplicate unless the operation is idempotent.
- Parsing before verifying. Re-serialization can alter whitespace, number representation, or character escaping and invalidate a signature.
- Logging full requests by default. Payloads may contain credentials, personal information, or payment-related data. Redact deliberately and limit retention.
- Testing only with a browser. Browsers follow redirects, hide some headers, and do not reproduce the sender’s timeout or retry behavior.
- Assuming a 2xx response means completion. If a queue accepts the event, monitor the worker and final state separately.
- Changing several variables at once. Rotate one secret, change one route, or update one parser at a time where possible so the cause remains identifiable.
For a broader developer workflow, compact browser-based utilities can help inspect JSON, Base64, or related formats, but use them only with sanitized data. The browser developer tools guide compares practical inspection workflows.
When to revisit
Review the webhook checklist before seasonal planning cycles, major releases, infrastructure migrations, and changes to queues, gateways, secrets, or event schemas. Revisit it whenever a provider changes its signature scheme, retry policy, timeout limit, event format, or replay controls. Also review it after an incident, even if the immediate fix was simple; an incident often reveals missing observability or an unsafe retry assumption.
Keep a short integration record with the endpoint owner, event types, authentication method, expected response behavior, idempotency key, redaction rules, replay procedure, and last verification date. Run a controlled test after endpoint changes and confirm both the success path and a failure path such as an invalid signature or temporary downstream outage.
When the next delivery fails, do not start by rewriting the handler. First collect the request and response evidence, classify the failure as transport, authentication, parsing, processing, or retry-related, then change the smallest relevant variable. That discipline turns webhook debugging into a repeatable engineering workflow rather than a sequence of guesses.