You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This was described on slack and I've seen it as well:
When TRD fails, the next run attempts to run the failed payments again. But this results in a lot of calls to tzkt api. Recently, it seems that this endpoint has been rate-limited, causing failure like the example below (taken on ghostnet TRD).
A workaround is to delete the failed payment folder (assuming it's a solid failure and not a partial payment). But it would be good to look into why it's querying every
│ 2023-06-22 21:39:11,754 - producer - INFO - Summary 4 paid, 0 done, 0 injected, 9 fail, 1 avoided │
│ Exception in thread producer: │
│ Traceback (most recent call last): │
│ File "/app/.local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn │
│ conn = connection.create_connection( │
│ File "/app/.local/lib/python3.10/site-packages/urllib3/util/connection.py", line 72, in create_connection │
│ for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): │
│ File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo │
│ for res in _socket.getaddrinfo(host, port, family, type, proto, flags): │
│ socket.gaierror: [Errno -3] Try again │
│ │
│ During handling of the above exception, another exception occurred: │
│ │
│ Traceback (most recent call last): │
│ File "/app/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen │
│ httplib_response = self._make_request( │
│ File "/app/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request │
│ self._validate_conn(conn) │
│ File "/app/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn │
│ conn.connect() │
│ File "/app/.local/lib/python3.10/site-packages/urllib3/connection.py", line 363, in connect │
│ self.sock = conn = self._new_conn() │
│ File "/app/.local/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn │
│ raise NewConnectionError( │
│ urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7ff7a2064460>: Failed to establish a new connection: [Errno -3] Try again │
│ │
│ During handling of the above exception, another exception occurred:
but the only other occurence of update_current_balances() function in the producer logic is in CalculatePhase4 which is for founders/owners.
We are not actually querying individual delegators balances with tzkt API during initial payment: we are bulk querying the indexer with a balance list of delegators self.reward_provider_model.delegator_balance_dict.
Only during retries, we do it, which takes a very long time and often fails.
A solution would be to modify the retry logic to query all balances again using the same API as initial payment. I'm not going to do this now, instead I'll just record my observations here.
This was described on slack and I've seen it as well:
When TRD fails, the next run attempts to run the failed payments again. But this results in a lot of calls to tzkt api. Recently, it seems that this endpoint has been rate-limited, causing failure like the example below (taken on ghostnet TRD).
A workaround is to delete the failed payment folder (assuming it's a solid failure and not a partial payment). But it would be good to look into why it's querying every
Link to baker slack discussion: https://tezos-baking.slack.com/archives/CQ35AM8KE/p1685978794781209
The text was updated successfully, but these errors were encountered: