[bug]: TrackPayments occasionally returns two failed payments

[feelancer21]

Background

I am currently building a tool to store all my failed payments in a sql database. I noticed that occasionally payments with the status FAILED are returned twice with different failure_reason. Which in the end led to a unique constraint violation in my database.

With bos probe the behavior is reproducible. But it also occurs with payments outside of bos, e.g. very rarely with the regolancer.

 {'paymentHash': '1bfa4cc64968a5b7e2409761b9d98cccefb291b2c95b1e7a04c4cf621d243943', 'value': '10000', 'creationDate': '1740109172', 'paymentPreimage': '0000000000000000000000000000000000000000000000000000000000000000', 'valueSat': '10000', 'valueMsat': '10000000', 'status': 'FAILED', 'creationTimeNs': '1740109172616344874', 'htlcs': [{'status': 'FAILED', 'route': {'totalTimeLock': 884822, 'totalAmt': '10000', 'hops': [{'chanId': '###################', 'chanCapacity': '6000000', 'amtToForward': '10000', 'expiry': 884822, 'amtToForwardMsat': '10000000', 'pubKey': '################', 'tlvPayload': True, 'mppRecord': {'totalAmtMsat': '10000000', 'paymentAddr': 'g9+TbZjgqXCY0bVlEaE1hnWNX23I69xmIorddIW6jbg='}}], 'totalAmtMsat': '10000000', 'firstHopAmountMsat': '10000000'}, 'attemptTimeNs': '1740109172629398991', 'resolveTimeNs': '1740109192783215207', 'failure': {'code': 'INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS', 'failureSourceIndex': 1, 'height': 884669}, 'attemptId': '44732122'}], 'paymentIndex': '44664171', 'failureReason': 'FAILURE_REASON_INCORRECT_PAYMENT_DETAILS'}
 
 
 {'paymentHash': '1bfa4cc64968a5b7e2409761b9d98cccefb291b2c95b1e7a04c4cf621d243943', 'value': '10000', 'creationDate': '1740109172', 'paymentPreimage': '0000000000000000000000000000000000000000000000000000000000000000', 'valueSat': '10000', 'valueMsat': '10000000', 'status': 'FAILED', 'creationTimeNs': '1740109172616344874', 'htlcs': [{'status': 'FAILED', 'route': {'totalTimeLock': 884822, 'totalAmt': '10000', 'hops': [{'chanId': '###################', 'chanCapacity': '6000000', 'amtToForward': '10000', 'expiry': 884822, 'amtToForwardMsat': '10000000', 'pubKey': '################', 'tlvPayload': True, 'mppRecord': {'totalAmtMsat': '10000000', 'paymentAddr': 'g9+TbZjgqXCY0bVlEaE1hnWNX23I69xmIorddIW6jbg='}}], 'totalAmtMsat': '10000000', 'firstHopAmountMsat': '10000000'}, 'attemptTimeNs': '1740109172629398991', 'resolveTimeNs': '1740109192783215207', 'failure': {'code': 'INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS', 'failureSourceIndex': 1, 'height': 884669}, 'attemptId': '44732122'}], 'paymentIndex': '44664171', 'failureReason': 'FAILURE_REASON_ERROR'}

Your environment

lnd 0.18.5

Expected behaviour

A failed payment should only be sent once so that a clear reason for the error can be identified.