Your Webhook Endpoint Is a Tiny Distributed System

Sep 9, 2026

Rob

Webhook endpoints always seem simple when you build the first version.

Add a route, create a controller action, parse some JSON, update a record and return 200. Pretty standard Rails stuff.

Then the real requirements start showing up. You need to verify that Stripe or GitHub actually sent the request. The provider wants a response quickly, so the useful work moves into a background job. The same event arrives twice. A worker dies after doing half of the work. Two related events get processed at the same time. Another event shows up out of order.

At some point, that little controller action has picked up a surprising amount of infrastructure around it.

Nobody starts by saying, “I need a distributed system for this webhook.” You normally get there one completely reasonable requirement at a time.

It is still a small system, of course. We are not building Kafka and twelve services here. But once a webhook is production-ready, you have an external trust boundary, durable ingress, asynchronous workers, duplicate delivery, retries, concurrency, ordering problems and a handful of failure states that all need to agree with each other.

That is the part of webhooks I find interesting. A tiny HTTP endpoint becomes a pretty good microcosm of a much larger distributed system.

First, can you trust the request?

A webhook is public ingress into your application, so before doing anything useful with the payload, you need to answer the obvious security question: did the provider actually send this?

Most providers solve this with a shared secret and a signature over the request body. GitHub, for example, sends an HMAC-SHA256 signature in X-Hub-Signature-256. Stripe signs its payload and includes a timestamp in Stripe-Signature.

The important detail is that those signatures are based on the raw request body, not a Ruby hash after Rails has parsed it.

So the order here matters:

Read the exact request body.
Verify the signature.
Parse the payload only after you trust it.

Rails gives us request.raw_post for the first part.

A minimal GitHub controller can look something like this:

class Webhooks::GithubController < ApplicationController
  skip_forgery_protection

  def create
    raw_body = request.raw_post

    return head :unauthorized unless valid_signature?(raw_body)

    payload = JSON.parse(raw_body)

    # Persist and dispatch the verified event here.

    head :accepted
  rescue JSON::ParserError
    head :bad_request
  end

  private
    def valid_signature?(raw_body)
      secret = Rails.application.credentials.dig(:github, :webhook_secret)

      expected = "sha256=" + OpenSSL::HMAC.hexdigest(
        OpenSSL::Digest.new("sha256"),
        secret,
        raw_body
      )

      provided = request.headers["X-Hub-Signature-256"].to_s

      ActiveSupport::SecurityUtils.secure_compare(expected, provided)
    end
end

There are only a few lines here, but each one matters. The signature is compared with Rails’ secure_compare, the body is not parsed until after verification, and the endpoint skips normal browser CSRF protection because this request is not coming from one of our application’s browser sessions. The provider signature is what authenticates this request.

For Stripe snapshot events, I would use the SDK instead of rebuilding its signature format myself:

raw_body = request.raw_post
signature = request.headers["Stripe-Signature"]
secret = Rails.application.credentials.dig(:stripe, :webhook_secret)

event = Stripe::Webhook.construct_event(raw_body, signature, secret)

Stripe’s verifier also validates the signed timestamp against its tolerance, which helps with replay protection. Stripe’s docs are also very specific about passing the unmodified request body to the verifier.

Stripe’s newer thin events use a different SDK entry point, so this example is specifically for the snapshot Event objects used by the rest of this article.

So even before we process the event, the webhook endpoint already has its own trust boundary and its own security rules.

Receiving and processing are two different jobs

Ok, now that we trust the request, the next problem is how long we keep the provider waiting.

The webhook provider is making a normal HTTP request to our application. It does not want to wait while we update a bunch of records, call another API, generate something, send email or do whatever the event actually triggers.

GitHub.com expects a 2xx response within 10 seconds. Stripe’s guidance is to return a successful 2xx before doing complex processing that could time out.

That leads to a fairly boring ingress path, which is exactly what I want:

verify
  ↓
validate enough to identify the event
  ↓
persist a durable receipt
  ↓
enqueue work
  ↓
return 2xx

Personally, I do not want the request itself provisioning an account, recalculating billing, synchronizing a repository or doing some other multi-step business process. The request should establish that I trust the event and that my application has accepted responsibility for it.

New Rails 8 applications use Solid Queue as the default Active Job backend in production. A dispatch job can stay very normal Rails code:

class WebhookDispatchJob < ApplicationJob
  queue_as :webhooks

  def perform(delivery_id)
    delivery = WebhookDelivery.find(delivery_id)

    # Dispatch to provider/event-specific handling.
  end
end

That gets the slow work out of the HTTP request, but just throwing perform_later into the controller is not the whole reliability story.

If you return a 2xx and then discover that the event never made it into a durable place, the provider thinks you have the event and you may not actually have it anymore.

This is where the acknowledgement boundary becomes important.

What does `2xx` actually mean?

I like storing a durable webhook receipt in the application database before telling the provider the event was accepted.

The first version does not need to be complicated:

class CreateWebhookDeliveries < ActiveRecord::Migration[8.0]
  def change
    create_table :webhook_deliveries do |t|
      t.string :provider, null: false
      t.string :external_id, null: false
      t.string :event_type, null: false
      t.json :payload, null: false
      t.datetime :processed_at
      t.datetime :failed_at
      t.text :last_error
      t.timestamps
    end

    add_index :webhook_deliveries,
      [:provider, :external_id],
      unique: true
  end
end

With that in place, GitHub ingress can start looking more like this:

def create
  raw_body = request.raw_post
  return head :unauthorized unless valid_signature?(raw_body)

  delivery_id = request.headers["X-GitHub-Delivery"].presence
  event_type = request.headers["X-GitHub-Event"].presence

  return head :bad_request unless delivery_id && event_type

  payload = JSON.parse(raw_body)

  delivery = WebhookDelivery.create_or_find_by!(
    provider: "github",
    external_id: delivery_id
  ) do |record|
    record.event_type = event_type
    record.payload = payload
  end

  WebhookDispatchJob.perform_later(delivery.id)

  head :accepted
rescue JSON::ParserError
  head :bad_request
end

One thing to call out here: the unique database index is doing real work. An exists? check followed by create! still has a race between those two queries. Two concurrent requests can both pass the check. The unique (provider, external_id) constraint is the final guard that prevents two receipt rows for the same provider event.

The receipt also gives us somewhere to recover from the slightly awkward gap between inserting application data and enqueuing a job.

Solid Queue has an important detail here. In a default Rails 8 production setup, Solid Queue is configured on a separate database from the application’s primary database. Creating a WebhookDelivery in one database and inserting a Solid Queue job in another is not one atomic transaction.

In the controller above, create_or_find_by! has already finished its transaction before perform_later runs, so the receipt is committed before enqueueing starts. If receipt creation and enqueueing happen inside a larger application transaction, Rails gives Active Job enqueue_after_transaction_commit to defer the enqueue until that transaction commits:

class WebhookDispatchJob < ApplicationJob
  queue_as :webhooks
  self.enqueue_after_transaction_commit = true
end

That prevents work from being enqueued for a transaction that later rolls back. It still does not create one atomic transaction across the application and queue databases.

If the receipt commits and the queue insertion fails, I still want that event to be recoverable. A recurring reconciliation job that looks for old, unprocessed receipts is a simple solution and gives you another chance to enqueue the work.

Now 2xx has a much more useful meaning. It does not mean all of the business logic finished. It means the application has durably accepted the event and can recover the remaining work if something goes wrong.

Duplicate delivery is normal

This is probably the first webhook problem that surprises people when they have only worked with the happy path.

Imagine this:

Stripe sends an event.
Your application processes it successfully.
The response gets lost or arrives too late.
Stripe does not know the work succeeded.
Stripe sends the event again.

That is not some wild production edge case. Two systems communicated over a network and disagreed about whether the operation finished.

Stripe explicitly documents that the same event can be delivered more than once. GitHub gives every delivery an X-GitHub-Delivery GUID and keeps that same GUID when a delivery is redelivered.

This is why the (provider, external_id) unique index is useful. It gives us delivery idempotency. Receiving the same provider event over and over does not create a new logical event in our application every time.

However, delivery idempotency is not the same as business idempotency.

Suppose two different webhook events can both lead to the same business operation. Or suppose a job calls an external API successfully and dies before it records processed_at. The unique webhook receipt does not stop that external operation from happening again on the retry.

For those cases, I like the idempotency boundary to describe the actual operation:

class CreateProvisioningOperations < ActiveRecord::Migration[8.0]
  def change
    create_table :provisioning_operations do |t|
      t.references :account, null: false, foreign_key: true
      t.string :operation_key, null: false
      t.datetime :completed_at
      t.timestamps
    end

    add_index :provisioning_operations,
      [:account_id, :operation_key],
      unique: true
  end
end

That operation_key could be based on a provider object ID and the action you are performing. If the next external service supports its own idempotency keys, I would use those too when making that request.

This is also why I try not to think in terms of “exactly once” webhook processing. We do not control exactly how many times a provider attempts delivery, how many times a job is retried or whether a worker dies at a convenient point.

What we can control is whether doing the work again is safe.

Idempotency does not remove concurrency

There is another small wrinkle in the controller example above.

Two copies of the same event can both resolve to the same WebhookDelivery, and both requests can enqueue WebhookDispatchJob.

If the job starts with this:

return if delivery.processed_at?

…that helps with a later retry, but it does not protect against two workers starting at the same time. Both workers can read processed_at == nil before either one updates it.

For work that stays entirely inside our database, a row lock can make that state transition explicit:

class WebhookDispatchJob < ApplicationJob
  queue_as :webhooks

  def perform(delivery_id)
    delivery = WebhookDelivery.find(delivery_id)

    delivery.with_lock do
      return if delivery.processed_at?

      Webhooks::Github::Dispatch.call(delivery)
      delivery.update!(processed_at: Time.current)
    end
  end
end

That example assumes Dispatch is doing database work that belongs inside the transaction.

I would not keep a row lock open while making a slow request to another service. Once the handler has external side effects, that business operation should normally have its own durable idempotency protection instead of assuming a database lock in Rails can control what another system does.

So now our tiny webhook endpoint has a concurrency model too. Fun!

There are two retry systems

Once processing moves to Solid Queue, there are two completely separate retry loops to think about: the provider retrying webhook delivery and our application retrying the async processing.

For Stripe, the flow looks roughly like this:

Stripe
  ↓ webhook delivery
Rails ingress
  ↓ durable receipt
Solid Queue
  ↓ processing attempt
business state

If Rails does not return a successful response, Stripe can retry the webhook delivery for up to three days in live mode using exponential backoff.

Once Rails has accepted the event, the job can fail independently. Active Job does not automatically retry every failed job for you; retry behavior is something you configure. A transient network failure could use something like:

class WebhookDispatchJob < ApplicationJob
  queue_as :webhooks

  retry_on Net::OpenTimeout,
    Net::ReadTimeout,
    wait: :polynomially_longer,
    attempts: 5

  def perform(delivery_id)
    # ...
  end
end

I would not put every exception behind one giant retry rule. A timeout to another API may be temporary. A malformed payload is not. A missing mapping or a bug in the handler probably needs a fix rather than five delayed attempts at the exact same broken code.

Provider behavior is not universal here either. GitHub does not automatically redeliver a failed webhook delivery. You can redeliver manually or through the API, but that is very different from Stripe retrying failed deliveries on its own.

That is another reason I like owning the durable receipt and having a reconciliation path inside the Rails application. Recovery should not depend entirely on whatever retry policy the provider happens to use.

And then events arrive out of order

Even after making every individual event safe to retry, you can still get the wrong result if your code assumes events arrive in the order they happened.

Stripe does not guarantee event ordering. GitHub also documents that webhook deliveries can arrive in a different order than the underlying events occurred.

A billing flow could end up looking like this:

subscription.updated
subscription.deleted
subscription.updated   # delayed older event

If each handler blindly writes its payload to the local record, that delayed event can move the local state backward.

The fix depends on the provider. Sometimes there is a useful sequence or version. Be careful with timestamps: Stripe’s snapshot Event created value only has second-level precision, and Stripe says not to use it to determine event order.

For important state, another option is to treat the webhook as a notification that something changed and then fetch the provider’s current canonical object before making a local decision. That read can fail when the object has been deleted or is no longer available, and it is not an ordering guarantee by itself. Concurrent handlers still need a provider-specific version or sequence when one exists, local serialization, or a reconciliation rule.

The main point is to make the ordering assumption explicit. “This is the webhook I am processing now” does not automatically mean “this is the newest state.”

At this point we have duplicate messages, retries, concurrent workers and events showing up out of order. The distributed-system comparison is doing a lot less metaphorical work than it did at the beginning.

Every provider has its own little rules

There is definitely common webhook infrastructure worth sharing, but I would be careful about trying to make Stripe, GitHub, Shopify, Twilio and every other provider look identical too early.

Verification happens before trust. Durable receipts can share a lifecycle. Jobs can share dispatch infrastructure. Logging and admin tooling can be normalized.

The provider details still matter.

Stripe and GitHub alone are enough to show the differences:

Signatures: Stripe uses a timestamped Stripe-Signature format and provides an SDK verifier. GitHub uses X-Hub-Signature-256 with HMAC-SHA256.
Retry behavior: Stripe automatically retries failed live deliveries for up to three days. GitHub does not automatically redeliver failures.
Ordering: both document that deliveries can arrive out of order.
Delivery identity: Stripe events have event IDs. GitHub provides a delivery GUID in X-GitHub-Delivery that stays the same across redelivery.
Response expectations: GitHub expects a 2xx within 10 seconds. Stripe recommends returning a 2xx quickly before complex processing.

I normally prefer a common ingress shape with provider-specific verification and dispatch over a generic abstraction that hides details the application actually needs to know.

In Rails, this can stay pretty conventional:

Webhooks::GithubController
        ↓
GithubSignatureVerifier
        ↓
WebhookDelivery
        ↓
WebhookDispatchJob
        ↓
Webhooks::Github::Dispatch
        ↓
provider/event-specific handler

There does not need to be a framework behind every box. The useful part is having an obvious place for each responsibility so that Stripe-specific logic does not slowly leak into GitHub processing and vice versa.

A `200` is not observability

Once the useful work is asynchronous, the HTTP request stops telling us very much about what eventually happened.

A successful webhook response can mean the signature passed, a receipt was stored and the work was enqueued while absolutely none of the actual business processing has happened yet.

Eventually you need to answer questions about the event lifecycle instead of the HTTP request.

Did we receive delivery abc123? Was it a duplicate? What event type was it? Did processing ever finish? How many times did it fail? What was the last error? Can I replay it after fixing the bug?

This is another place where the WebhookDelivery model earns its keep. It gives logs, admin tooling, reconciliation and manual replay a stable record to work from.

I usually want enough state to distinguish something along these lines:

received → processing → processed
                  ↘ failed

You do not need to build a workflow engine for this. A few timestamps, an error field and a clear job boundary may be plenty. But once processing can happen seconds or minutes after the request, “the endpoint returned 200” is not a useful debugging answer anymore.

Ok, so we built a tiny distributed system

Let’s look back at everything our original controller picked up along the way.

We now have an untrusted network boundary, cryptographic verification, a durable message receipt, database uniqueness, an asynchronous queue, concurrent workers, duplicate delivery, an internal retry policy, a provider retry policy, out-of-order events, business idempotency, failure state, reconciliation, replay and observability.

None of those are especially exotic problems by themselves. The interesting part is how many of them get packed into a feature that started as POST /webhooks/stripe.

This does not mean the first webhook endpoint needs fifteen classes and a homegrown framework. I would still start pretty small. I just want the important boundaries to be obvious:

POST /webhooks/:provider
        ↓
verify the raw payload
        ↓
persist one durable receipt
        ↓
enqueue Solid Queue work
        ↓
provider-specific handler
        ↓
idempotent business operation
        ↓
record the outcome

From there, add complexity when a real failure mode calls for it.

The controller can stay boring. That is probably the best outcome.

The complicated part was never parsing the JSON. It was accepting work from another system when you do not control how many times the message arrives, when it arrives, what order it arrives in or which process eventually finishes it.

That is a lot hiding behind one Rails controller action.