Md Nayeem Hossain | Software Engineer

Background Jobs with BullMQ: Retries, Backoff and Dead Letters#

Any request that sends an email, generates a report, calls a slow third-party API or processes a large file should not make the user wait. Moving that work to a background job is the obvious fix. Making those jobs reliable is less obvious.

I use BullMQ on Redis for most Node.js projects. These are the patterns that keep queues healthy in production.

The basic shape#

A producer adds jobs; one or more workers process them:

1import { Queue, Worker } from "bullmq";2 3const connection = { host: process.env.REDIS_HOST, port: 6379 };4 5export const emailQueue = new Queue("email", { connection });6 7await emailQueue.add(8  "welcome",9  { userId },10  { attempts: 5, backoff: { type: "exponential", delay: 2_000 }, removeOnComplete: 1_000 },11);12 13new Worker(14  "email",15  async (job) => {16    const user = await users.findById(job.data.userId);17    await mailer.sendWelcome(user.email, user.name);18  },19  { connection, concurrency: 10 },20);

Pass IDs, not data#

Notice the job payload is { userId }, not the whole user. Jobs can sit in a queue for minutes or, during an outage, hours. Data copied into a payload goes stale. Loading fresh data inside the worker means a retry uses the current state, and payloads stay small.

Every job must be idempotent#

BullMQ, like every queue, delivers at least once. A worker can finish the work and crash before acknowledging it, and the job runs again. Design every handler so running it twice is harmless:

Check whether the work is already done ("was this invoice already sent?").
Use idempotency keys when calling external APIs.
Use upserts instead of inserts.

If you take one thing from this post, take this one.

Retries with exponential backoff#

Most failures are temporary: a provider rate-limits you, a network call times out. Retrying immediately usually fails again and adds load to the struggling dependency. Exponential backoff — 2s, 4s, 8s, 16s — gives it time to recover.

Not every error should be retried, though. A 400 from an API because the email address is invalid will fail forever. Mark those as unrecoverable so they stop immediately:

1import { UnrecoverableError } from "bullmq";2 3if (error.status === 400) {4  throw new UnrecoverableError(`Invalid recipient: ${error.message}`);5}6throw error; // retry everything else

Dead letters: failures you can act on#

After the final attempt, a job ends up in the failed set. That is your dead-letter queue, and it needs an owner. In practice:

Alert when failed jobs exceed a threshold, broken down by job type.
Keep failed jobs long enough to inspect the error and payload.
Provide a replay path — a small admin action or script to retry failed jobs once the cause is fixed.

A failed-jobs set that nobody looks at is just a slower way of losing data.

Timeouts and stalled jobs#

A worker that hangs forever holds a job forever. Protect against it:

Put timeouts on every external call inside a job.
Rely on BullMQ's stalled-job detection: if a worker stops renewing its lock (because it crashed or the event loop is blocked), the job is returned to the queue.
Avoid blocking the event loop with heavy synchronous work. Use sandboxed processors or worker threads for CPU-heavy tasks.

Concurrency and rate limits#

Set concurrency per queue based on what the downstream system can handle, not what your server can handle. If an email provider allows 50 requests per second, use BullMQ's built-in limiter:

1new Worker("email", handler, {2  connection,3  concurrency: 20,4  limiter: { max: 50, duration: 1_000 },5});

Separate queues by priority and by dependency. A flood of report-generation jobs should never delay password-reset emails.

Scheduling and repeatable jobs#

For cron-style work — nightly cleanups, hourly syncs — use repeatable jobs instead of setInterval in your API process. They survive restarts and run once across all your worker instances, rather than once per instance.

Graceful shutdown#

During deploys, workers receive a termination signal. Close them properly so in-flight jobs finish instead of being interrupted:

1process.on("SIGTERM", async () => {2  await worker.close(); // waits for active jobs to complete3  process.exit(0);4});

Observe your queues#

The metrics I watch for every queue: waiting count, active count, failed count, job duration and the age of the oldest waiting job. A growing backlog is often the first sign of a downstream problem.

Conclusion#

Reliable background processing comes down to a few rules: pass IDs, make handlers idempotent, retry transient errors with backoff and stop on permanent ones, give failed jobs an owner, and shut down gracefully. Follow them and your queues become the most dependable part of the system rather than the most mysterious.