Storing the exception in the result store

[ksamuel]

Some queue libs do something quite handy, when getting the result of task, if the task raised an exception, then getting the result raises the same exception.

This allows to deal with errors in a natural way with multiple try/except (so good idea support, natural pattern matching, log support), and to debug errors that happen in the other process.

I don't know if this makes sense for django.tasks, but we do need:

• To have information on the error that happened.
• A way to handle it in a rich manner.
• A mechanism to introspect the problem and debug.

Because debugging task queue errors in production is very hard without it, especially if you don't have good telemetry.

[RealOrangeOne]

getting the result raises the same exception.

This feels like the wrong approach, as you'd need to potentially catch multiple exceptions wherever you get the result.

I do agree though there needs to be a way of capturing the exception. However it's not simple due to our constraint on JSON-only serialization. We could fairly easily pass through the type of exception which was raised, but we couldn't reliably send other details without resorting to pickle.

The current compromise is to not store it at all, but log it so it can be captured by error monitoring tooling.

I'd love some ideas for how this could be solved!

[ksamuel]

Logging only is not an option because setting up logging is a non trivial problem in Python. The stdlib logging module has terrible default and an unintuitive api, and half of the django devs I met don't know how to settings.LOGGING work.

I do believe it's great to have it, I always set it up myself with rich on top, but it can't be the only source.

One additional info source is to serialize the stack trace to text and store it, with the error datetime, to at least be able to read it from the task result later. It's not perfect, but it will allow displaying it in a shell, print it, create an admin for it, etc. We could go as far as dumping the repr() of locals() (barring a size limit) in another attribute.

Having a dead letter queue with a live shell is also great for debugging, nothing beats having access to the object in memory, and no serialization is involved, but it's much harder to pull off, especially since it means dealing with a dead letter queue per process.

Both is ideal but I understand even if django mascot is a poney, we can't always have one.

[RealOrangeOne]

Storing the text representation of the stacktrace is probably better than nothing, but it's really only useful for debugging - your code can't really action it.

The error type and description are probably enough for most cases, which would be fairly easy to serialize.

tblib, which Django somewhat depends on already for parallel tests, seems to implement serialization (or more importantly, deserialization) for exceptions without using pickle. That might be the ideal direction to go. However, it's another dependency for Django, which might be a difficult sell.

[ksamuel]

From a design perspective, maybe it's better to:

• Dedicate error info at the task result level for debugging, so it's kept simple.
• Let the user deal with error handling and decision-making (retry, re-enqueue) in a callback called from the task itself (could be set from a decorator).
• Give a free json field to tasks where one can save arbitrary metadata from inside the task or the aforementioned callback for anything more complicated that must be done way after the fact.

Because trying to cater to everything may require indeed big dependencies.

[RealOrangeOne]

I've written a fairly basic exception serializer which can be used to store task errors: #68