Deferred tasks maximum delay

[jribbens]

Just a note to make sure django-tasks doesn't make the same mistake that celery did...

celery introduced eta / countdown in 2009. These allow you to do the equivalent of DEP-0014's "deferred tasks". That's a great feature.

In 2023, thirteen years later, they changed the documentation for this feature to say "Oh, sorry, didn't we say, you can't actually defer tasks for more than a minute or two at most? It may appear to work but will cause all sorts of subtle problems later. We probably should have mentioned that. But we didn't. Sucks to be you".

So, django-tasks should either:

• document that the maximum supported delay is <x> minutes (or whatever),
• or, far preferably, explicitly document that you can defer tasks for as long as you want. 5 minutes, 5 hours, 5 months, all are good and will be dealt with efficiently.

[RealOrangeOne]

I'd not considered the need to document it, but currently yes it's assumed that run_after can be any point in the future, no matter how far.

There might be an argument for capping it to something sensible, so people are encouraged to use something like cron (which is probably more applicable for that use-case), but I don't think that's necessary.

[jribbens]

I think it's important to document, because otherwise people who write back-end plugins later may assume that they can get away with things like taking any task from the queue and if it turns out it's deferred then sleep() until it's ready, which is what I think happened with celery.

A cap is not unreasonable, but if there is one I think it should be at least a month (i.e. 32 x 86400 seconds).

cron (which I think I've already mentioned is something I think django-tasks should be supporting natively, like celerybeat) is great for doing things like "once every day, process payments for customer renewals happening that day" but useless for things like "send an email to newly registered users 1 hour, 48 hours, and 72 hours after they register".

[RealOrangeOne]

Documenting is probably the way to go then. PRs welcome 🙂

useless for things like "send an email to newly registered users 1 hour, 48 hours, and 72 hours after they register"

Not quite true. A cron task which runs every hour, finds the users which registered an hour ago, and emails them, is perfectly sensible. In fact, i'd tend towards implementing a feature that way rather than using run_after. It's much easier then to handle cases like the user having deleted their account, or unsubscribing, or something like that.

[jribbens]

Not quite true. A cron task which runs every hour, finds the users which registered an hour ago, and emails them, is perfectly sensible.

That's why I included "1 hour" in my example. If the requirement is "1 hour after registration" and in fact they'll get it "between 0 and 60 (or 60 and 120) minutes after registration" then that may well not be acceptable. Plus, suppose you can't tell from the database that the action should be triggered, you'd have to re-implement a run_after queue in your own application, plus a polling event to see if things are waiting, just to avoid using a feature that already exists in in django-tasks!