Free sample · Chapter 2
The Durable Handoff
Once perform_later returns, it feels like your part is over. The request is done, the job is "in the queue," and a worker will pick it up. Yet a great many reliability problems in background work begin right there, because a lot happens to the work between the moment you hand it off and the moment a worker looks at it. Time passes. The deploy that was running when you enqueued has been replaced. The record you were looking at may have changed, or been deleted, or the transaction around your call may have rolled back. In rare cases the job never reaches the queue at all.
Consider two jobs at the point where a request hands off responsibility. Registration sends a welcome email. If that enqueue is lost, it’s annoying, but nobody is waiting for a promised result. A report is different: the user clicked a button, we said we accepted the work, and then we owe them an answer. Both need correct arguments and transaction timing. Only the report needs a durable record that recovery can find if publication fails.
What crosses the boundary
When you call perform_later, Active Job doesn’t hold on to your arguments. It serializes them into a payload, the adapter writes that payload somewhere durable (for Solid Queue, a row in solid_queue_jobs), and the objects you passed are gone as far as the job is concerned. A worker process, possibly on a different machine, possibly running a different release of your code, later reads the payload, deserializes it, and calls perform.
You can see exactly what will cross by serializing a job without enqueueing it. The report job takes an account ID and a report request ID, and the reference application has a test that looks at the payload:
test "the queued arguments are plain ids, with no request-local state serialized" do
job = GenerateReportJob.new(@account.id, @request.id)
assert_equal [ @account.id, @request.id ], job.serialize["arguments"]
end
Both are integers. Active Job also serializes strings, booleans, nil, symbols, hashes, times and dates, and anything that includes GlobalID::Identification (which is every Active Record model). Pass an object it doesn’t know how to serialize and Active Job raises ActiveJob::SerializationError, which is at least a loud failure. Outside a transaction it comes out of perform_later itself. Inside a transaction with deferral on (which the transaction section below turns on), it comes out at commit, after your rows are already saved.
The more dangerous cases are the ones that serialize without complaint. A hash holding a copy of a record’s attributes serializes fine. So does a symbol-keyed options hash, a time computed from "now," or a string that happens to be a secret. The queue accepted it, and nothing in the framework asks whether what it accepted still means the right thing when a worker reads it an hour later, or after a deploy, or after an outage.
So the first design question for any job is what the job will see later. Whatever the answer is, it has to be small and stable enough to survive the wait.
Identity, execution-time state, and event-time snapshots
Three different kinds of data tend to get mixed together in job arguments, and they want different treatment.
Identity answers "which thing is this about?" An account ID and a report request ID are identity. They’re small, they don’t change, and they stay meaningful for as long as the row exists. This is what belongs in the arguments.
Execution-time state is whatever is true when the job actually runs: whether the account is still active, whether the user still has access, what the current balance is, what the latest saved settings are. You can’t know it when you enqueue, because by definition it hasn’t happened yet. The job should read it fresh. A job that carries the state in its arguments is carrying a guess about the future.
Event-time snapshots are the opposite case. Some facts are only correct as they were at the moment the user asked. A report request for September should produce September’s report whether the worker starts a second later or the queue backs up past midnight. If the job computes the period from "this month" when it starts, a late start gives the wrong answer with no error anywhere.
The reference application records that kind of input on the request row itself, in a parameters column, at the moment of the request:
class ReportRequestsController < ApplicationController
before_action :set_current_account
def create
request = ReportRequest.submit!(
account: Current.account,
requested_by: Current.account.users.find(params.require(:requested_by_id)),
kind: params.require(:kind),
parameters: params.fetch(:parameters, {}).permit(:period_start, :period_end).to_h
)
render json: { id: request.id, status: request.status }, status: :accepted
end
private
# Request-local context. The job does not receive it; it gets account_id.
def set_current_account
Current.account = Account.find(params[:account_id])
end
end
The period_start and period_end the user chose are written to report_requests.parameters (a jsonb column) when the row is created. The job gets IDs, loads the row, and reads the frozen period from it. That puts the snapshot in the database where it can be inspected, kept for as long as you like, and survives a retry or a recovery run without being re-derived. It also keeps the queue payload tiny, which matters because queue payloads are copied into logs and dashboards.
The response matters too. The controller renders 202 Accepted and the row’s status, which is pending. It doesn’t say "done." That distinction between accepted and completed is the thread running through the rest of this book, and the status column is what lets the interface, a recovery job, and an operator all agree on which one is true.
Stable IDs and explicit tenant identity
The job itself looks like this at the top:
class GenerateReportJob < ApplicationJob
queue_as :default
...
def perform(account_id, report_request_id)
# Scoped lookup: an id from another account finds nothing.
request = ReportRequest.find_by(id: report_request_id, account_id: account_id)
# Policy: a request that no longer exists has no one waiting for it, so do nothing.
return unless request
# Another execution already claimed or finished it.
return unless request.claim!
...
end
end
(The retry configuration and the rest of perform are elided here. Chapter 3 covers failure and Chapter 4 covers the claim on the third line.)
Two arguments rather than one. The report request ID would be enough to find the row, and in a single-tenant application it would be enough, full stop. In a multi-tenant application the second argument, account_id, is doing real work. The lookup is find_by(id:, account_id:), so a job that somehow carries a request ID belonging to a different account finds nothing and does nothing. The reference test for this is short:
test "a request id from another account finds nothing and is left untouched" do
other_account, _other_user = create_account("Other")
GenerateReportJob.perform_now(other_account.id, @request.id)
assert_equal 0, ReportBuilder.calls
assert_equal "pending", @request.reload.status
end
The mismatched pair is more likely to come from a bug than from a normal request: a console script using the wrong loop variable, a data migration that moved records, or an accidental controller change. If the two IDs don’t belong together, the scoped lookup returns nothing. That’s better than building one customer’s report with another customer’s data, and it gives you something useful to log.
Using find_by with both keys is a small choice, and an Account.find(account_id).report_requests.find(id) chain would do the same job. The difference is the failure mode. The chain raises ActiveRecord::RecordNotFound for either a missing account or a mismatched request, and find_by followed by return unless request lets us choose what a missing record means. We’ll come back to that choice below, because it is a policy decision and not something the framework makes for you.
Request-local state does not travel
There’s one more thing the job above doesn’t do: it never touches Current. The controller sets Current.account for the duration of the request, and that is fine for the request. But Current is cleared when the request ends, and the job runs in a different execution context entirely. Active Job doesn’t carry it over.
The reference app makes this a test instead of a claim:
test "Current.account set during a request is gone by the time the job runs" do
account, user = create_account
seen = :unset
ReportBuilder.script(->(_r) { seen = Current.account })
post account_report_requests_path(account), params: { requested_by_id: user.id, kind: "usage" }
assert_nil Current.account, "request-local state does not outlive the request"
perform_enqueued_jobs
assert_nil seen, "the job rebuilt its context from account_id, not from Current"
assert_equal "completed", ReportRequest.last.status
end
This matters most in a place that doesn’t look like a job at all. If code in a request calls GenerateReportJob.perform_now, it runs inside the request and happily sees the controller’s Current.account. It works in development, where you run everything inline, and fails in production, where the same code runs on a worker with nothing set. Writing every job as if Current, the session, and the request don’t exist keeps the inline and the queued paths identical.
The same goes for the actor. requested_by_id is stored on the row as data: who asked. It’s a record of the event, and not a permission context. A job that wants to know whether the requester may still see this report should ask the question when it runs, against current data, rather than assuming the user who clicked the button a day ago still has the access they had then.
GlobalID: valid, with a different failure shape
Passing the record itself is perfectly idiomatic Rails. The welcome email does it:
UserMailer.welcome(user).deliver_later
Behind that line, Active Job serializes user as a GlobalID, a string like gid://reference/User/42, and the worker calls GlobalID::Locator.locate to load a fresh copy from the database when the job runs. So the job doesn’t get a stale snapshot of the user’s attributes. It gets a lookup by ID, the same as the explicit version, just spelled differently. The reference app checks that too:
test "a GlobalID argument is serialized as a GlobalID, not as the record" do
argument = UserNoticeJob.new(@user).serialize["arguments"].first
assert_equal "gid://reference/User/#{@user.id}", argument["_aj_globalid"]
end
For a welcome email, that is plenty. There is one record, the lookup is by ID, and nothing about tenancy or workflow state is at stake. We wouldn’t rewrite it.
Where GlobalID becomes a poor fit is at workflow boundaries, for two reasons that have little to do with safety.
First, the failure shape. When the record is gone, the lookup fails during deserialization, before perform runs. The error is ActiveJob::DeserializationError, with the original ActiveRecord::RecordNotFound as its cause. A rescue_from ActiveRecord::RecordNotFound or a handler written around the body of perform never sees it. The reference app has a small job that makes the point:
# GlobalID style: the job receives the record itself, serialized as a GlobalID
# and loaded when the job runs. If the user was deleted in between, loading raises
# ActiveJob::DeserializationError before perform runs. Discarding is right here
# because a notice for a user who no longer exists has no value.
class UserNoticeJob < ApplicationJob
discard_on ActiveJob::DeserializationError
def perform(user)
UserMailer.notice(user).deliver_now
end
end
And the tests that pin down both sides of it:
test "a GlobalID record deleted before the job runs raises DeserializationError, which the job discards" do
UserNoticeJob.perform_later(@user)
@user.destroy!
assert_nothing_raised { perform_enqueued_jobs }
assert_empty ActionMailer::Base.deliveries
end
test "without discard_on the same deletion fails the job with DeserializationError" do
StrictNoticeJob.perform_later(@user)
@user.destroy!
assert_raises(ActiveJob::DeserializationError) { perform_enqueued_jobs }
end
Discarding is the right decision for this job. Nobody needs a notice addressed to a deleted user. But it is a decision, and with GlobalID it is made by a class-level declaration far from the code that knows what the record means. With explicit IDs the decision sits inside perform, next to the code that can tell whether a missing row is harmless.
Second, a GlobalID argument carries no tenant. gid://reference/User/42 identifies a user and nothing else. If the tenant matters (and it matters for nearly every record in a multi-tenant app), the job either trusts the global lookup or has to check the record’s account after loading it. Explicit IDs make the tenant a visible, required argument.
None of this makes GlobalID wrong. For a mailer that takes a user, a cache-warming job, or any low-stakes job where a missing record means "skip it," it’s the least code and it works. Reach for explicit identifiers when the job drives a workflow someone is waiting on, when the tenant boundary matters to the lookup, or when "the record is gone" needs a different answer from "the lookup blew up."
Missing records and changed eligibility
A job that runs later can find that the world has moved. There are two different ways for that to happen, and the right response differs.
The first is that the record is gone. For GenerateReportJob, the answer written in the code is a no-op: a request that has been deleted has nobody waiting for it, so the job returns. The reference test pins that down:
test "a missing request is a no-op, not a failure" do
@request.destroy!
assert_nothing_raised { GenerateReportJob.perform_now(@account.id, @request.id) }
assert_equal 0, ReportBuilder.calls
end
Compare the provider operation job, the one the reference application uses for refunds, which we’ll meet properly in Chapter 4. Its lookup reads:
operation = ProviderOperation.where(account_id: account_id).find(provider_operation_id)
Same scoped lookup, opposite policy. The comment in that file explains: the operation row is written before the job is enqueued, so a missing row can’t be "nothing to do." Either something deleted money-related state or a bug enqueued the wrong ID, and the job should fail visibly. find raises RecordNotFound, the job fails, and someone looks.
The decision to make for every job is which of these a missing record is. It’s a business question about whether anyone is harmed if the work silently doesn’t happen, and neither answer is the safe default. Swallowing a missing record for a refund hides a real problem. Failing loudly on a deleted report request fills your error tracker with noise about nothing.
The second kind of change is that the record exists but the work is no longer appropriate: the account was suspended, the subscription lapsed, the user lost access, the report was already completed by an earlier execution. Enqueue-time checks can’t rule these out, because the check happened before the wait. When eligibility matters, check it again at the start of perform, against current data, and decide whether an ineligible record is a no-op (nothing to do any more) or a failure (something that should have been prevented got through). The claim in GenerateReportJob, the conditional update that moves a request from pending to processing, is one form of that check, and Chapter 4 takes it apart.
The transaction boundary
Now to when the job is handed off. Registration creates an account and its first user and sends a welcome email, all inside one transaction:
class Registration
def self.create!(account_name:, email:, name:)
ActiveRecord::Base.transaction do
account = Account.create!(name: account_name)
user = account.users.create!(email: email, name: name)
UserMailer.welcome(user).deliver_later
user
end
end
end
The deliver_later call is inside the transaction, which is the natural place for it: that’s where the user exists, in the code’s own mind. But the transaction hasn’t committed yet. Two things can go wrong.
If the enqueue happens immediately, a worker could pick up the job before the commit and look for a user that other database connections can’t see yet. The job fails with a missing record (or, with a GlobalID, with DeserializationError), and whether it retries successfully depends on timing.
And if the transaction rolls back after the enqueue, the job exists for a user who was never created.
How likely each of these is depends on where the queue lives, and the reference application uses Rails' default for Solid Queue: a separate queue database next to the primary one.
# Rails' default Solid Queue topology: a separate `queue` database next to the
# primary one. A job row therefore never shares a transaction with a domain row.
default: &default
adapter: postgresql
...
development:
primary:
<<: *default
database: background_jobs_reference_development
queue:
<<: *default
database: background_jobs_reference_queue_development
migrations_paths: db/queue_migrate
Writing a job row means writing to a different database, through a different connection, in its own transaction. It doesn’t join yours. That has a consequence we can demonstrate. The reference suite has a test class that runs real commits, real rollbacks, and real Solid Queue, with an undeferred job (the Rails 8.1 default) and a deferred one:
class UndeferredJob < ActiveJob::Base # enqueue_after_transaction_commit is false: the Active Job 8.1 default
def perform(*) = nil
end
test "without deferral a job enqueued in a transaction that rolls back still exists in the queue" do
ActiveRecord::Base.transaction do
UndeferredJob.perform_later(1)
raise ActiveRecord::Rollback
end
assert_equal 1, SolidQueue::Job.where(class_name: UndeferredJob.name).count, "the hazard: a job for data that never committed"
end
test "without deferral a worker could pick the job up before the data it needs commits" do
ActiveRecord::Base.transaction do
UndeferredJob.perform_later(1)
assert_equal 1, SolidQueue::Job.count, "visible in the queue database while the primary transaction is still open"
assert_equal 1, SolidQueue::ReadyExecution.count
end
end
The first test is the orphan: the application’s transaction rolled back, and the job is still sitting in the queue. The second is the race: while the primary transaction is still open and uncommitted, the job and its ready execution are already visible in the queue database, which is exactly what a worker polls. Nothing prevents a worker from claiming it right now.
Deferring the enqueue
Active Job has a setting for this. When enqueue_after_transaction_commit is true on a job class, calling perform_later inside a transaction doesn’t enqueue immediately. Active Job registers the enqueue to run after the outermost transaction commits, and drops it if the transaction rolls back.
In Rails 8.1 this is opt-in. The attribute defaults to false. The old global config.active_job.enqueue_after_transaction_commit setting was removed in 8.1, and load_defaults 8.1 doesn’t turn it on. (Earlier Rails versions had a different shape here, with :default, :always, and :never values and an adapter that could answer for itself. If you’re reading a version older than 8.1, read that version’s source rather than applying this.) The setting that works is the class attribute, so the reference app turns it on in ApplicationJob:
class ApplicationJob < ActiveJob::Base
# Active Job 8.1 defaults this to false: perform_later inside a transaction
# enqueues immediately. Setting it to true defers the enqueue until the
# outermost transaction commits and drops it on rollback. See README.
self.enqueue_after_transaction_commit = true
end
One line, and every job that inherits from ApplicationJob is covered. The two tests that showed the hazard now have deferred counterparts:
class DeferredJob < ActiveJob::Base
self.enqueue_after_transaction_commit = true
def perform(*) = nil
end
test "with deferral a job enqueued in a transaction that rolls back is never published" do
ActiveRecord::Base.transaction do
DeferredJob.perform_later(1)
raise ActiveRecord::Rollback
end
assert_equal 0, SolidQueue::Job.count
end
There’s a trap in the way this is set up. deliver_later doesn’t enqueue one of your jobs. It enqueues ActionMailer::MailDeliveryJob, and that class inherits from ActiveJob::Base directly, not from ApplicationJob. A setting on ApplicationJob doesn’t reach it, which means the welcome email in the registration code above would still be enqueued immediately, hazard and all, with the app looking correctly configured. The same applies to anything else you’ve put on ApplicationJob, such as retry_on or queue_as.
The reference app sets the attribute on the mail delivery job itself:
# Mail delivery jobs are plain Active Job classes, not ApplicationJob subclasses,
# so they need the same deferral: no email job is published for a transaction
# that rolls back.
ActiveSupport.on_load(:action_mailer) do
ActionMailer::MailDeliveryJob.enqueue_after_transaction_commit = true
end
And a test confirms the mailer behaves as intended, which is the check worth running in your own application, because the setting is easy to believe in without ever observing it:
test "a mailer's deliver_later inside a transaction waits for commit" do
_account, user = create_account
ActiveRecord::Base.transaction do
UserMailer.welcome(user).deliver_later
assert_equal 0, SolidQueue::Job.count
end
job = SolidQueue::Job.sole
assert_equal "ActionMailer::MailDeliveryJob", job.class_name
assert_equal "critical", job.queue_name
end
Inside the block nothing is in the queue. After the block exits (commit), one MailDeliveryJob is there. A companion test checks that a failed registration, one that violates the unique email index, rolls back the account and queues no mail at all.
Where after-commit is enough
For the welcome email this is the whole treatment. The email is sent after the account and user are committed, never for a rolled-back registration, and never before the worker can see the user. If the process dies in the narrow gap between commit and enqueue, the user doesn’t get a welcome email. We accept that. Nobody is waiting for it, the user is already in the product, and the machinery to guarantee it would cost more than the occasional loss. The policy is worth writing next to the code, since it is the kind of decision that vanishes from memory: a comment on Registration saying the email is best effort and may be lost between commit and enqueue is enough. The mail delivery initializer above is what makes the "after commit" half of that comment true.
What after-commit does not give you
Deferral prevents a job from running early or for data that never existed. That’s all it prevents. It does not promise the job will exist after a successful commit, and it’s worth being exact about why, because the behavior is easy to misread.
A success signal that isn’t one
With deferral active inside a transaction, perform_later returns the job instance with successfully_enqueued? set to true. Nothing has been written to the queue yet. The reference suite shows the whole sequence:
test "perform_later inside an open transaction returns the job instance but publishes nothing until commit" do
account, _user = create_account
returned = nil
ActiveRecord::Base.transaction do
returned = GenerateReportJob.perform_later(account.id, 1)
assert_instance_of GenerateReportJob, returned
assert returned.successfully_enqueued?, "reports success optimistically while deferred"
assert_equal 0, SolidQueue::Job.count, "nothing is written to the queue yet"
end
assert_equal 1, SolidQueue::Job.count, "the enqueue runs right after commit"
end
The return value tells you the call was accepted, not that the job is in the queue. Code that checks successfully_enqueued? inside a transaction, perhaps to decide what to tell the user, is checking something that can’t yet be known.
You may also have read that a failed enqueue makes perform_later return false. With Solid Queue that is not how failure arrives. false comes back when an enqueue callback halts the job or when an adapter raises ActiveJob::EnqueueError. Solid Queue raises SolidQueue::Job::EnqueueError, which is a plain StandardError and not an ActiveJob::EnqueueError, so it propagates as an exception. Checking the return value for false won’t catch it.
The failure after commit
Combine the two facts. The business row has committed, and then the deferred enqueue runs and raises, perhaps because the queue database is unreachable at that instant. The exception comes out of the after-commit callback, and the caller sees it, but the row is already durable. The reference suite simulates exactly that:
test "an enqueue failure after commit raises from submit! and leaves a pending request that recovery publishes again" do
account, user = create_account
failing = ->(*) { raise SolidQueue::Job::EnqueueError, "queue database unavailable" }
request = nil
assert_raises(SolidQueue::Job::EnqueueError) do
SolidQueue::Job.stub(:enqueue, failing) do
request = ReportRequest.submit!(account: account, requested_by: user, kind: "usage", parameters: {})
end
end
request = ReportRequest.sole
assert_equal "pending", request.status, "the accepted request is durable even though the caller saw an error"
assert_equal 0, SolidQueue::Job.count
travel 20.minutes
RecoverReportRequestsJob.perform_now
assert_equal 1, SolidQueue::Job.where(class_name: "GenerateReportJob").count
drain_solid_queue
assert_equal "completed", request.reload.status
end
Here the controller gets an exception even though the request was accepted and saved. If the controller turned that exception into a 500, the user might retry and create a second request, so what the response says about a committed-but-unpublished request deserves a thought. A process killed between the commit and the enqueue produces the same stranded row and no exception anywhere, since nothing is alive to raise it.
One more behavior from the suite is worth knowing because it’s quiet: an ActiveJob::EnqueueError raised by an adapter during the deferred enqueue is swallowed. The reference test shows the request staying pending and the queue staying empty with no error surfaced. Whichever way the enqueue fails after commit, you should assume nobody is going to tell you about it.
Written as a timeline, the gap looks like this:
1. The request commits a ReportRequest with status "pending".
2. The process is killed, or the queue database is unreachable, before the enqueue completes.
3. The user holds a pending request. No job exists. Nothing will ever run it.
After-commit deferral shrank the window in which this can happen to the interval between commit and enqueue. It didn’t close it.
Durable intent, in proportion
For a welcome email we stop at step one of the timeline above. For a report the user was told we accepted, we can’t. Having said "we have your request," silence afterward is a broken promise, and nothing in the system would notice.
The fix doesn’t have to be elaborate. We already wrote a row for the request, in the same transaction as the acceptance. That row, with its pending status, is a durable statement that someone wants this work done. The job in the queue is an optimization on top of it, a way to get the work started promptly, and the row is the source of truth. If a job goes missing, the row is still there, and a periodic sweep can find rows that have been pending too long and publish them again.
Durable intent here doesn’t need a separate outbox table, an event store, or a publisher process. The business row already carries everything an outbox row would, and it’s already committed atomically with the acceptance.
The model carries the policy
The submit method writes the row and requests the job in one transaction, and the class carries the constants and scopes the sweep uses:
# The pending row is the durable record of intent. Enqueueing GenerateReportJob is
# an optimization on top of it: RecoverReportRequestsJob can always find work that
# was accepted but never published, or claimed and then abandoned.
class ReportRequest < ApplicationRecord
STATUSES = %w[pending processing completed failed].freeze
# Longer than the longest retry wait (up to 296.4 seconds with 15% jitter
# before the fifth attempt), so the sweep avoids duplicating scheduled retries.
PENDING_RECOVERY_AFTER = 15.minutes
STALE_PROCESSING_AFTER = 30.minutes
RECOVERY_BATCH_SIZE = 100
# The sweep is a retry layer with its own budget: after this many republishes or
# claim resets the request is failed instead of recovered again.
MAX_RECOVERY_ATTEMPTS = 3
class RecoveryExhausted < StandardError; end
belongs_to :account
belongs_to :requested_by, class_name: "User"
validates :kind, presence: true
validates :status, inclusion: { in: STATUSES }
scope :stale_pending, ->(now = Time.current) { where(status: "pending").where(updated_at: ...(now - PENDING_RECOVERY_AFTER)) }
scope :stale_processing, ->(now = Time.current) { where(status: "processing").where(started_at: ...(now - STALE_PROCESSING_AFTER)) }
# Persist the intent and ask for the job in one transaction. The enqueue is
# deferred until the transaction commits (see ApplicationJob).
def self.submit!(account:, requested_by:, kind:, parameters:)
transaction do
request = account.report_requests.create!(requested_by: requested_by, kind: kind, parameters: parameters)
GenerateReportJob.perform_later(account.id, request.id)
request
end
end
...
end
The claim and completion methods that the elision hides are the subject of Chapter 4. For now, submit! is the piece that matters: the row is created, perform_later is called inside the same transaction, and with deferral on, the enqueue waits for the commit.
PENDING_RECOVERY_AFTER is fifteen minutes, well above the longest scheduled retry_on wait, so the sweep doesn’t publish a second job while the original retry is still waiting (we’ll calculate that wait in Chapter 3). STALE_PROCESSING_AFTER is thirty minutes, intended to be longer than a normal report run. Neither number is a Rails default. If your reports run for forty minutes, thirty is a bad choice; if users need an answer in five, fifteen may be too long.
The sweep
The recovery job is short:
# Recurring sweep (see config/recurring.yml). Finds accepted work whose job was
# never published, or whose claim was abandoned, and publishes it again.
# GenerateReportJob's claim makes a duplicate enqueue harmless. The sweep is itself a
# retry layer, so each request gets MAX_RECOVERY_ATTEMPTS recoveries and is then
# failed and reported, not recovered forever.
class RecoverReportRequestsJob < ApplicationJob
queue_as :default
def perform
now = Time.current
# An abandoned claim goes back to pending and is published again right away.
ReportRequest.stale_processing(now).limit(ReportRequest::RECOVERY_BATCH_SIZE).each do |request|
next request.give_up_recovery!(now) if request.recovery_exhausted?
GenerateReportJob.perform_later(request.account_id, request.id) if request.reset_stale_claim!(now)
end
ReportRequest.stale_pending(now).limit(ReportRequest::RECOVERY_BATCH_SIZE).each do |request|
next request.give_up_recovery!(now) if request.recovery_exhausted?
# Spending the attempt also touches updated_at, so the next sweep does not
# re-enqueue it before the recovery period passes again. Conditional: a job that
# already completed the request keeps its updated_at and spends nothing.
GenerateReportJob.perform_later(request.account_id, request.id) if request.spend_republish!(now)
end
end
end
It does two things. Rows that have been processing longer than the timeout are presumably abandoned by a worker that died; the sweep returns them to pending with reset_stale_claim! and publishes them again. And rows that have been pending longer than the recovery threshold are published again too, covering the stranded case from the timeline. A batch limit keeps a large backlog from enqueueing thousands of jobs in one run.
The spend_republish! call is easy to skim past. Besides counting the attempt, it touches updated_at. Without that, a request that’s been pending for twenty minutes would be republished on every sweep until a worker picked it up, which would pile duplicates into a slow queue. Touching updated_at pushes the request back behind the threshold, so it gets republished at most once per recovery period. The update verifies both that the request is still pending and that updated_at is still older than the recovery threshold. A request released recently for retry cannot be republished on the strength of an old snapshot.
A sweep is a retry layer, and a retry layer needs its own budget. Without one, a request whose job crashes every time (a poison request) would be reset after thirty minutes, published again, crash again, and repeat for as long as the sweep runs. So recovery_attempts counts each republish or claim reset, in a conditional update that refuses to go past MAX_RECOVERY_ATTEMPTS. When a request has used all three, the next sweep marks it failed with a message that says recovery gave up, and reports it with Rails.error.report(…, handled: true). The failure is then visible in business status and in error reporting, which is the same rule Chapter 3 set for a job that exhausts its retries. The count is spent only on requests the sweep touched, so a request that completes normally never sees it, and reopen! resets it for an operator who has fixed the cause.
Every recovery mutation rechecks staleness in its own SQL UPDATE, not just in the sweep’s initial SELECT. A worker can release a stale claim and another worker can take it again before recovery runs. Checking only status could reset that new, live claim. reset_stale_claim! therefore checks started_at against the cutoff, spend_republish! checks updated_at, and exhaustion checks the stale predicate plus the spent budget before marking failed. A fresh retry keeps its claim and no error is reported. The two indexes the sweep needs are (status, updated_at) and (status, started_at).
It runs on a schedule. The entry in config/recurring.yml is two lines:
production: &recurring
recover_report_requests:
class: RecoverReportRequestsJob
schedule: every 5 minutes
...
(Chapter 7 covers recurring configuration and what Solid Queue does and doesn’t do about missed runs. Here it only needs to be true that the sweep runs periodically.)
The thirty-minute cutoff does not prove the original worker died. It might just be slow. If recovery resets its claim and republishes the request, both executions can build the report. We accept that cost here because report generation is repeat-safe and a completed or failed row cannot be overwritten by a late completion. You wouldn’t make the same trade for an arbitrary external effect; that needs an ownership generation or fencing check at completion. The sweep only solves lost publication if the job on the other end can safely run again.
The test that proves the sweep recovers the stranded row is the one shown earlier: the enqueue fails after commit, the request stays pending, twenty minutes pass, RecoverReportRequestsJob.perform_now publishes a GenerateReportJob, the queue drains, and the request is completed. Two companion tests prove the sweep leaves things alone that it should: a fresh pending request is not republished, and a processing claim younger than its timeout is not reset.
Why the report gets this and the welcome email doesn’t
It’s tempting to look at the sweep and decide every job deserves one. It doesn’t, and the reason is worth stating in terms of what each loss costs.
A lost welcome email costs one email that nobody was waiting for. There is no row to find, no promise to keep, and nobody would notice. Adding a sweep would require inventing a table to record "welcome emails we meant to send," and then a job to read it, to guard against a loss we’ve already decided to accept. It would be complexity with no one to benefit from it.
A lost report request costs a user who pressed a button, saw "we’re preparing your report," and waits indefinitely. We told them something and then failed to follow through. The row already exists because the feature needs it anyway; the sweep is a short job that reads it. For this job the cost and the benefit are reversed.
What decides is whether the application made a promise it has to keep. Importance isn’t the test: a high-volume, low-stakes job with no promise attached doesn’t need recovery, and a low-volume one that was promised does. And when the promise involves money or an irreversible effect in another system, a sweep isn’t enough; Chapter 4 adds a persisted operation with a stable identity for those.
Be clear, too, about what the sweep doesn’t do. It publishes jobs. It doesn’t make the work happen exactly once, and it can’t detect a loss that left no row behind. Its guarantee is bounded by the row: if an accepted request exists in a non-terminal state, someone will eventually publish a job for it.
A note on shared databases
Rails normally generates Solid Queue tables in a separate database. You can put them in the application database instead and insert the job row on the same connection and in the same transaction as the business row. That removes the commit/enqueue gap for that particular handoff, at the cost of queue traffic on your primary database, shared connection capacity, and a different migration setup. If your application uses that arrangement, document it and decide whether a publication sweep is still needed. The reference sticks to the generated separate-database setup, where assuming atomic publication would be wrong.
Arguments that outlive a deploy
One last property of the handoff is easy to forget until it bites. A job sits in the queue between enqueue and execution, and that interval can span a deploy. The worker that runs it may be a newer release than the one that enqueued it. Whatever shape of arguments the old code serialized, the new code has to be able to receive.
That gives three rules of thumb.
Add arguments as optional ones with defaults. If GenerateReportJob later needs a delivery option, make it a keyword with a default:
def perform(account_id, report_request_id, deliver_by_email: false)
...
end
Jobs queued by the old release have two arguments and run fine with the default. Jobs enqueued by the new release pass the extra keyword. During a rolling deploy an old worker can pick up a job from the new release, and old code that receives a keyword it doesn’t declare raises ArgumentError (the reference tests it). Deploy the receiving side first, or keep the old code tolerant of the new keyword.
Don’t remove or reorder positional arguments while jobs in the old shape could still be queued. Dropping account_id or swapping the order silently misreads old payloads, and since both are integers, the misreading looks like a record not found and not like a type error. If you need a different shape, add it alongside, accept both for a while, and delete the old one after the queue has drained.
Don’t rename or move a job class without leaving the old name resolvable. The payload records the class name as a string, and a worker that can’t resolve it can’t run the job. Keeping the old constant defined, as an alias or a thin subclass, until the old rows are gone makes a rename safe. The reference tests the alias case and the failure when the constant is gone (a NameError).
A queued payload may outlive the release that created it. Keeping it to an account ID and a request ID gives an older job very little to misunderstand. The period, format, and delivery options stay on the request row, where ordinary migrations can change them.