Building Scalable APIs with Django & Celery

When an API gets slow, the first instinct is a bigger server. More often the problem is elsewhere: the request is doing work that doesn't belong there. Emails, Telegram messages, image processing, calls to third-party APIs — none of it is worth making the user wait. Here is how I move that work to the background with Django and Celery, with real examples from WeliCoDev and NutriZone.

1. Rule: the request is only for the response

Keep only what the response needs inside the HTTP request: validation, a database write, the reply. Everything else goes to a queue.

• Django accepts the request and enqueues a task;
• Redis is the broker;
• Celery worker runs tasks in the background;
• Celery beat runs scheduled jobs.

2. The classic bug: enqueuing before the commit

Call task.delay(obj.pk) right after saving and the worker may not find the row — the transaction isn't committed yet. Use transaction.on_commit:

from django.db import transaction

def on_article_published(instance):
    transaction.on_commit(
        lambda: post_to_channel.delay("blog.Article", instance.pk)
    )

3. Idempotency

Celery may retry a task and workers restart. Every task must survive running twice. For channel posts I use two guards: a timestamp on the object and a table of sent posts.

@shared_task
def post_to_channel(model_label, pk):
    obj = Model.objects.filter(pk=pk).first()
    if obj is None or obj.posted_to_channel_at:
        return
    if ChannelPost.objects.filter(object_id=pk, status="sent").exists():
        return
    post = channel.post_object(obj)
    if post.status == "sent":
        Model.objects.filter(pk=pk).update(posted_to_channel_at=timezone.now())

Pass only the pk; read fresh state from the database when the task runs.

4. Third-party limits and retries

Telegram allows a bot roughly 25 messages per second. For broadcasts: throttle, retry on transient errors, and on permanent ones (403 — the user blocked the bot) stop and record it.

5. Schedules: Celery beat

CELERY_BEAT_SCHEDULE = {
    "send-due-telegram-posts": {
        "task": "apps.telegram_bot.tasks.send_due",
        "schedule": 60.0,
    },
}

Set CELERY_TIMEZONE to your project's zone (ours is Asia/Tashkent), or “11:00” will mean UTC.

6. Production: separate containers

Web, worker and beat share one image but run as separate services. Scale workers independently; run exactly one beat, or jobs fire twice.

7. Checklist

• Pass IDs, not objects.
• After saving — transaction.on_commit.
• Make every task idempotent.
• Handle 4xx and 5xx differently.
• Exactly one beat.
• In tests, mock external APIs and assert the double-call case.

Conclusion

Keep requests light, hand heavy work to the queue, and write every task to be safely repeatable — the API becomes fast and resilient on its own.