You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It appears that the Redis consumer will not reconnect after an ack failure. It is believe that the ack failure occurred because the stream was dropped from the Redis instance after exceeding its maximum cache size. In flight messages could not be ack'd because the stream key no longer existed. The producer to this stream re-created a new key and the consumer never recovered.
The text was updated successfully, but these errors were encountered:
the stream was lost because it exceeded its MAXLEN setting
the CC application reported errors acking messages
the CC application did not report any errors reading from the non-existent stream
the stream was recreated at some point, however the CC application still did not receive any new messages until restarted
things to investigate:
what's the behavior of the redis client when we read from a non-existent stream? does it produce an error or just return "0 messages"? is there a difference between the stream not existing to begin with vs. the stream disappearing mid-way?
why did the CC application no receive new messages after the stream was recreated? does it have to do with the consumer group? does it need to be recreated as well / rejoined by the application?
options to fix this issue:
detect when streams don't exist and throw an error from the input source which will terminate the CC application (brute force fix)
gracefully handle rejoining consumer groups for re-created streams (in case that turns out to be a problem)
streams are already created as part of the initialization logic. when the application detects that a stream is gone it could just re-create it
In this case I think if the app had crashed when that first ack failed everything would have self-healed. Propagating that exception might be a decent enough short term fix.
It appears that the Redis consumer will not reconnect after an ack failure. It is believe that the ack failure occurred because the stream was dropped from the Redis instance after exceeding its maximum cache size. In flight messages could not be ack'd because the stream key no longer existed. The producer to this stream re-created a new key and the consumer never recovered.
The text was updated successfully, but these errors were encountered: