5 Ways Your Celery Tasks Are Failing Silently (And How to Catch Them)

Silent failures are the worst kind. Your users are waiting for their data export, your nightly backup didn't run, and you won't know until someone complains. Here's how to catch Celery failures before they become user-facing problems.

The problem with background tasks

Background tasks are designed to be "fire and forget." You queue them up, they run asynchronously, and you move on. But in production, this invisibility becomes a liability.

I learned this the hard way running a data migration. The migration was processing millions of records and ran for days. Everything looked fine—no errors in the logs, no crashes—but it had been stuck for 6 hours with no progress. I only discovered this because I manually SSH'd into the server to check.

Celery has the same problem. Most teams set up error tracking (Sentry, etc.) and call it done. But that only catches exceptions. It doesn't catch tasks that never start because no worker picked them up, or tasks that started but never finished because a worker was killed or restarted mid-execution. These silent failures are invisible unless you're specifically looking for them.

Here are the most common ways Celery tasks fail silently, and how to catch them.

1. The stuck task (no progress updates)

What happens

Your task starts processing, makes some progress, then... nothing. It's still "running" according to Celery, but it stopped updating 20 minutes ago.

Common causes: - Worker killed mid-task (OOM, deployment, server restart) without acks_late enabled - Worker restarted and task wasn't re-queued - External API hanging (no timeout set) - Database query locked waiting for another transaction

Real example

@app.task
def process_export(user_id):
    user = User.objects.get(id=user_id)
    data = fetch_data_from_api(user)  # ← Hangs here if API is down

    for item in data:
        process_item(item)

    send_email(user, "Your export is ready!")

If fetch_data_from_api() hangs (no timeout), the task sits there forever. Celery thinks it's running. The user never gets their email.

How to catch it

Solution 1: Heartbeat Detection (Stale Timeout)

Set a maximum time between task updates. If the task goes silent for too long, alert.

With TaskBadger's Celery integration, your tasks automatically report progress. You can use update_value for tasks with meaningful progress to track, or ping for tasks where you just want to confirm the task is still alive:

@app.task(bind=True, base=taskbadger.celery.Task)
def process_export(self, items):
    task = self.taskbadger_task

    for i, item in enumerate(items):
        do_something(item)

        # Report progress; value_step throttles updates to at most every 100 items
        task.update_value(value=i, value_step=100)

    # Mark the task as complete
    task.success(value=len(items))

For tasks without meaningful progress metrics, use ping to send a heartbeat:

@app.task(bind=True, base=taskbadger.celery.Task)
def sync_data(self):
    task = self.taskbadger_task

    for record in get_records():
        process_record(record)
        task.ping(rate_limit=5)  # Heartbeat, at most once every 5 seconds

If TaskBadger doesn't receive an update within the stale timeout, it marks the task as stale and triggers your configured alerts.

Solution 2: Maximum Runtime

Set a hard limit on how long a task should take.

@app.task(time_limit=3600)  # 1 hour hard limit
def process_export(user_id):
    # If task exceeds 1 hour, Celery kills it
    ...

When Celery kills a task for exceeding the time limit, it raises a SoftTimeLimitExceeded or TimeLimitExceeded exception. If you have Sentry or another error tracking system configured, you'll get an error notification. TaskBadger will also mark the task as failed.

Without stale detection though, a task that's stuck but hasn't hit the time limit looks identical to a running task. You won't know until users complain.

2. The silent exception (caught and swallowed)

What happens

Your code catches an exception but doesn't re-raise it. Celery thinks the task succeeded. It didn't.

Real example

@app.task
def send_password_reset(user_id):
    try:
        user = User.objects.get(id=user_id)
        send_email(user, reset_link)
    except Exception as e:
        logger.error(f"Failed to send password reset: {e}")
        # ← Exception caught, task "succeeds", user never gets email

From Celery's perspective, this task completed successfully. The user never gets their password reset email. They try again. It fails again. They give up.

How to catch it

Re-raise after logging

@app.task
def send_password_reset(user_id):
    try:
        user = User.objects.get(id=user_id)
        send_email(user, reset_link)
    except Exception as e:
        logger.error(f"Failed to send password reset: {e}")
        raise  # ← Let Celery know it failed

If you're using TaskBadger's Celery integration, exceptions are automatically recorded and the task is marked as error—similar to how Sentry captures exceptions. You don't need to add explicit failure tracking code.

A silent exception is worse than a loud one. At least when something blows up, you know about it.

3. The forgotten scheduled task (didn't run at all)

What happens

Your nightly backup, daily aggregation, or hourly sync didn't run. Celery Beat crashed, the cron expression was wrong, or the worker wasn't listening to that queue.

You won't know until: - You try to restore from a backup that doesn't exist - Your dashboard shows stale data from 3 days ago - A customer complains their integration hasn't synced in a week

Real example

# In your Celery Beat schedule
app.conf.beat_schedule = {
    'nightly-backup': {
        'task': 'tasks.backup_database',
        'schedule': crontab(hour=2, minute=0),
    },
}

What can go wrong: - Celery Beat isn't running (service died, forgot to start it) - Task queued to wrong queue, no worker listening - Schedule expression is wrong (runs Feb 31st only) - Server clock is wrong (timezone mismatch)

How to catch it

Check-in pattern: Tasks "check in" when they run. Alert if they don't check in on schedule.

import taskbadger as tb

@app.task
def backup_database():
    task = tb.Task.create(
        "nightly-backup",
        max_runtime=3600  # Seconds. Alert if the backup runs longer than 1 hour
    )

    try:
        run_backup()
        task.success()
    except Exception as e:
        task.error(str(e))
        raise

Then set up a monitor in TaskBadger dashboard: - Alert if: Task doesn't run within 25 hours of last run - Alert to: Email

Alternative: Use a service like Cronitor or healthchecks.io specifically for scheduled job monitoring.

This one will ruin your week. You discover your backups haven't been running the moment you actually need to restore from one.

4. The queue backlog (tasks pile up unnoticed)

What happens

Tasks are being queued faster than workers can process them. Queue wait time grows from seconds to minutes to hours. Users think your site is broken.

Common causes: - Traffic spike (Black Friday, viral post) - Slow downstream API (third-party service degraded) - Worker died, not enough workers for load - Database query got slower (missing index, table lock)

What it looks like

Hour 1: Queue length = 100, wait time = 5 seconds
Hour 2: Queue length = 500, wait time = 30 seconds
Hour 3: Queue length = 2,000, wait time = 10 minutes  ← Problem!
Hour 4: Queue length = 8,000, wait time = 45 minutes  ← Crisis!

Users requested a report export 45 minutes ago. It hasn't even started processing yet.

How to catch it

Monitor queue wait time (not just queue length).

Why wait time over queue length? Because 1,000 tasks in the queue is fine if workers process them in 10 seconds. It's a disaster if workers process them in 10 minutes.

# With TaskBadger's Celery integration, this is automatic
import taskbadger
from taskbadger.systems import CelerySystemIntegration

taskbadger.init(
    token="<your-token>",
    systems=[CelerySystemIntegration()],
)

Then set up alerts: - Alert if: Average queue wait time > 2 minutes - Response: Spin up more workers (auto-scaling or manual)

Manual check (Redis):

# Check queue length
redis-cli llen celery

# But this doesn't tell you wait time!
# You need to track when tasks were queued vs started

By the time you notice the backlog, hundreds of users have already given up waiting.

5. Slow task degradation

One more thing worth tracking: task duration trends. Tasks that used to take 5 seconds might gradually creep up to 30 seconds as your data grows—and you won't notice until it's a problem. TaskBadger tracks durations for individual tasks and shows aggregated durations by task type, so you can spot these trends before they become incidents.

The one metric that catches most of these

Notice a pattern? Most of these silent failures show up in queue wait time—the gap between when a task is queued and when a worker picks it up.

Stuck tasks tie up workers, so the queue backs up. Silent exceptions trigger retries, adding more tasks. Slow tasks mean each worker handles fewer per minute. All roads lead to the same place: longer queue wait times.

Queue wait time is the closest proxy you have for what users actually experience. Monitor it, and you'll catch most silent failures before they become visible.

How to implement this (practical steps)

Minimum viable monitoring (15 minutes)

1. Enable Celery events (required for all monitoring):

CELERY_WORKER_SEND_TASK_EVENTS = True
CELERY_TASK_SEND_SENT_EVENT = True

2. Run Flower locally (for development debugging):

pip install flower
celery -A your_app flower

3. Add TaskBadger (for production monitoring):

pip install taskbadger

import taskbadger
from taskbadger.systems import CelerySystemIntegration

taskbadger.init(
    token="<your-token>",
    systems=[CelerySystemIntegration()],
)

4. Set up alerts in the dashboard:

• Alert on task failures
• Alert on queue wait time > 5 minutes
• Alert if scheduled tasks don't run

Total cost: $0-99/month depending on volume | Setup time: 15 minutes

For larger teams

Add Sentry for error tracking:

import sentry_sdk
from sentry_sdk.integrations.celery import CeleryIntegration

sentry_sdk.init(
    dsn="your-dsn",
    integrations=[CeleryIntegration()],
)

Or Prometheus/Grafana if you already have infrastructure:

pip install celery-exporter
celery-exporter --broker-url=redis://localhost:6379/0

Wrapping up

Silent failures are the norm in Celery, not the exception. Design your monitoring around that assumption.

The single most useful metric is queue wait time—it surfaces stuck tasks, slow degradation, and backlog problems before users notice. Beyond that, set stale timeouts so you know when tasks go quiet, and verify that your scheduled tasks actually ran (don't just assume Celery Beat is doing its job).

Layer your tools: Flower for local debugging, something like TaskBadger for production monitoring, Sentry for error tracking. You don't need zero failures—you just need to hear about them before your users do.

Resources

• TaskBadger Celery Integration Docs
• Celery Monitoring Best Practices
• Flower Documentation

Have you experienced a silent Celery failure disaster? We've all been there.

This post was written by Simon, founder of TaskBadger. TaskBadger helps you monitor background jobs and catch silent failures before they become user-facing problems. Try it free at taskbadger.net.