Queues and Background Jobs in PHP: A Practical Guide

Some work does not belong in the request-response cycle. Sending an email, generating a PDF report, resizing an uploaded image, calling a slow third-party API — forcing a user to wait on any of these inside a normal HTTP request makes the application feel slow and fragile, since a single slow dependency now blocks every request that touches it. Queues move that work out of the request cycle entirely, into background workers that process it independently.

Dispatching a Job

A queued job is, at its simplest, a class describing one unit of deferrable work plus the data it needs to do that work. Dispatching it pushes a serialized representation onto a queue (database table, Redis list, or a managed service) for a separate worker process to pick up and execute later.

class SendWelcomeEmail implements ShouldQueue
{
    public function __construct(private int $userId) {}

    public function handle(): void
    {
        $user = User::findOrFail($this->userId);
        Mail::to($user)->send(new WelcomeMail($user));
    }
}

SendWelcomeEmail::dispatch($user->id);

Note that the job stores the user's id, not the full User object. Serializing an entire Eloquent model into the queue payload risks the data going stale by the time the job actually runs, and bloats the payload unnecessarily; re-fetching fresh data inside handle() is the safer default.

Jobs Must Be Idempotent

A queue worker can fail mid-job (a server restart, a deploy, an unhandled exception) and most queue systems will retry a failed job automatically. If a job is not idempotent — safe to run more than once with the same input — that retry can cause real damage: a payment charged twice, a welcome email sent three times, a counter incremented twice for one event. Designing every job to check whether its work has already been done, or to use operations that are naturally safe to repeat, is not optional polish; it is the baseline requirement for using queues safely at all.

public function handle(): void
{
    if (Payment::where('order_id', $this->orderId)->where('status', 'completed')->exists()) {
        return;
    }
    // proceed with charge
}

Failed Jobs Need a Real Plan, Not Just Logging

A job that fails and gets silently logged to a file nobody monitors is functionally the same as a job that silently never ran at all, from the business's perspective. Failed jobs need visibility (alerting, a dashboard, a dead-letter queue you actually check) and a defined retry strategy — exponential backoff for jobs depending on a flaky external API, immediate alerting for jobs where failure has direct business consequences like a missed payment.

Queue Priority and Isolation

Not all background work has equal urgency. A password-reset email is urgent; a weekly analytics report is not. Running every job through a single shared queue means a burst of low-priority report-generation jobs can delay an urgent password-reset email behind it in line. Separate queues by priority, and run separate worker processes for high-priority queues, so urgent work is never stuck waiting behind a backlog of work that could comfortably wait.

Job Timeouts and Memory Limits

A queued job with no timeout can hang indefinitely on a stuck external API call, tying up a worker process that could otherwise be processing other jobs. Setting an explicit timeout per job, shorter than the queue worker's overall timeout, ensures a single stuck job fails fast and frees the worker rather than silently blocking the queue behind it.

class GenerateReport implements ShouldQueue
{
    public int $timeout = 120;
    public int $tries = 3;
}

Monitoring Queue Depth, Not Just Failures

A queue that is steadily growing — more jobs arriving than workers can process — is a slow-motion outage that pure failure-rate monitoring will miss entirely, since no individual job is failing. Tracking queue depth over time, and alerting when it trends upward persistently rather than just spiking briefly, catches a worker capacity problem before users start noticing delayed emails or reports.

Job Chaining and Batching

Some workflows need several jobs to run in a specific sequence, or need to know when an entire group of related jobs has finished. Job chaining runs jobs strictly in order, stopping the chain if one fails; job batching runs jobs in parallel while still letting you register a callback for when the whole batch completes, useful for a "process these 500 records and notify when done" workflow.

Bus::batch([
    new ProcessRecord($id1),
    new ProcessRecord($id2),
])->then(function () {
    Notification::send($admin, new BatchCompleteNotification());
})->dispatch();

Avoid Overloading a Single Job With Too Much Work

A job that processes ten thousand records in a single handle() call looks simple but fails badly — if it dies at record 8000, a naive retry reprocesses the first 8000 unnecessarily, and a sufficiently large batch can exceed memory or timeout limits outright. Splitting large batch work into many smaller jobs, or using chunked processing with checkpointing, makes partial failure recoverable instead of catastrophic.

Case Study: The Double-Charged Customers

A payment-processing job retried automatically after a worker crashed mid-execution, immediately after the charge had succeeded but before the job had recorded that success to the database. The retry charged the same customer a second time. Over a weekend with elevated worker crashes due to an unrelated memory leak, several dozen customers were charged twice before anyone noticed, surfaced eventually through support tickets rather than monitoring. The fix was making the job idempotent by checking for an existing successful payment record before charging, keyed on the order id, exactly the pattern described earlier in this guide — a fix that took an afternoon, versus the multi-day cleanup and customer-trust repair the incident itself required.

A Glossary for This Topic

Queue: an ordered list of pending units of work waiting for a worker to process them. Worker: a long-running process that pulls jobs off a queue and executes them. Idempotent: safe to execute more than once with the same input without unintended duplicate effects. Dead-letter queue: a separate queue holding jobs that failed all their retry attempts, for manual inspection. Job batching: dispatching a group of related jobs together with a shared completion callback.

Frequently Asked Questions

How many retries should a job get? It depends on the failure mode — transient network errors usually warrant a few retries with backoff, while errors from invalid input should not be retried at all since retrying will not change the outcome.

Can I run queues without a separate worker process? Technically a queue can run synchronously within the request, but that defeats the entire purpose of moving slow work out of the request cycle.

Should every queued job send a notification on failure? Only ones where failure has direct business consequences; routine, low-stakes jobs are usually fine surfaced through a dashboard rather than individual alerts.

Step-by-Step: Making an Existing Job Idempotent

First, identify the side effect the job performs that would cause real harm if duplicated (a charge, a sent email, an external API call). Second, add a durable record of that side effect having completed, keyed on a stable identifier (an order id), separate from the job's own internal state. Third, check that record at the start of handle() and return early if the side effect already completed. Fourth, ensure the check-and-record happens atomically enough to handle near-simultaneous retries, using a database unique constraint as a backstop if needed. Fifth, write a test that calls handle() twice and asserts the side effect happened only once.

A Comparison Table: Job Failure Handling Strategies

Security Considerations Checklist

Never deserialize job payloads from an untrusted source without validation, since a queue is sometimes accessible to more systems than the application itself realizes (a shared Redis instance, a misconfigured queue endpoint). Ensure jobs handling sensitive operations (payments, permission changes) re-verify authorization at execution time rather than trusting that authorization checked correctly when the job was originally dispatched, since significant time can pass between dispatch and execution during which permissions may have changed. Avoid logging full job payloads if they contain sensitive data.

Accessibility Considerations

Background jobs have no direct accessibility dimension, but user-facing consequences of job failures do — a failed job that silently never sends a password-reset email, with no user-facing indication anything went wrong, leaves a user stuck with no accessible path forward; surfacing failure state back to the user interface, not just to internal logs, matters here.

How This Plays Out at Different Scales

A small application with light background work can often get away with a single queue and minimal monitoring. A growing application needs the priority-isolation and depth-monitoring practices described earlier, since a single shared queue and failure-only alerting both break down as job volume and variety grow. A large-scale system typically needs dedicated infrastructure per job category, sophisticated retry and dead-letter tooling, and serious capacity planning for worker scaling under load.

What to Do When You Inherit Jobs With No Idempotency Guarantees

Inheriting a queue system full of jobs that assume they will only ever run exactly once, with no safeguards against the retries every queue system performs by default, is a serious but addressable risk. Prioritize auditing jobs by the cost of duplication first — a job that charges money or sends an irreversible external notification needs idempotency fixed immediately, while a job that merely recalculates a cached value that will simply be recalculated correctly again is lower urgency — rather than attempting to retrofit every job in the system simultaneously.

Final Checklist Before Trusting a Queue System in Production

Every job with a side effect that would cause real harm if duplicated has an explicit idempotency check. Job timeouts are set explicitly, shorter than the worker's overall timeout. Queue depth is monitored over time, not just failure rate. Failed jobs have a defined, monitored destination (dead-letter queue, alerting), not just silent logging. High-priority queues are isolated from low-priority bulk work.

Closing Thought, Revisited

Queues let an application feel fast by moving slow work out of the user's way, but that same deferral means failures happen invisibly, away from any user watching a loading spinner. The discipline that makes queues trustworthy — idempotency, monitoring, real failure handling — is what turns "fire and forget" into "fire and verify," which is the only version of background processing worth running in a system handling anything that actually matters.

Local Queue Testing Without a Real Worker

Laravel's sync queue driver runs jobs immediately, inline, with no separate worker process needed — useful for fast local development and for tests that need a job's effects to happen synchronously and deterministically. Switching to a real queue driver (database, Redis) only for staging and production, while developing locally against sync, is a common and reasonable setup, as long as you also test against a real queue driver at least occasionally to catch any behavior that differs under genuine asynchronous execution.

Job Versioning During Deploys

A rolling deploy can briefly have old and new application code running simultaneously, which matters if a job's class structure changes between versions — a job dispatched by old code but picked up by a new worker expecting a different constructor signature can fail unpredictably. Keeping job class signatures backward-compatible across a deploy, or draining the queue before deploying a breaking job change, avoids this class of deploy-timing bug.

Choosing a Queue Driver

The database queue driver is simple to operate since it needs no extra infrastructure beyond the database you already have, but it adds load to that same database under heavy queue volume. Redis-backed queues handle much higher throughput with lower latency but add an operational dependency. For most applications starting out, the database driver is the pragmatic default; migrating to Redis becomes worth the added complexity once queue volume is high enough that database load from queue polling becomes a measurable problem.

Graceful Worker Shutdown During Deploys

Killing queue workers abruptly during a deploy can interrupt a job mid-execution, leaving partial side effects behind. Allowing workers to finish their current job before shutting down, and only then restarting with new code, avoids this specific deploy-timing failure mode, at the cost of a slightly slower deploy while in-flight jobs finish.

Scheduled Jobs Versus Queued Jobs

A scheduled job runs on a fixed time-based trigger (daily, hourly), independent of any specific user action, while a queued job is dispatched in response to something happening (a user signing up, an order placed). Conflating the two — running heavy scheduled work synchronously inside the scheduler process rather than dispatching it to the queue — can block the scheduler from running other scheduled tasks on time; dispatching scheduled work to the queue rather than running it inline keeps the scheduler itself fast and reliable.

Visibility Into What a Worker Is Currently Processing

When a deploy needs to happen and workers need to be cycled, knowing exactly which job each running worker is currently mid-execution on matters for deciding whether to wait for it to finish or force a restart. Tooling that surfaces current job state per worker, not just aggregate queue depth, makes this kind of operational decision during a deploy far less of a guessing game than relying on log-tailing alone.

Testing Job Failure Paths Explicitly

It is easy to test that a job succeeds under normal conditions and forget to test what happens when its external dependency fails entirely — does the job retry correctly, does it eventually land in the dead-letter queue, does any partial side effect get cleaned up. Writing an explicit test that forces the dependency to fail and asserting on the resulting job state catches failure-handling bugs before they surface for the first time in a real production incident.