Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bug]: Channels Stuck in "Waiting to Close" State Despite Confirmed Transactions #9532

Open
jjacks95 opened this issue Feb 20, 2025 · 11 comments
Open

[bug]: Channels Stuck in "Waiting to Close" State Despite Confirmed Transactions #9532

jjacks95 opened this issue Feb 20, 2025 · 11 comments
Labels
bug Unintended code behaviour needs triage

Comments

@jjacks95
Copy link

Description:

Our LND node is encountering an issue where channels remain in the "Waiting to Close" state even though the associated close transactions have been successfully confirmed and broadcasted on the blockchain.

Observed Behavior:

  1. Rebroadcasting Confirmed Transactions:
    The node continuously attempts to rebroadcast transactions that have already been successfully broadcasted and confirmed. This behavior is redundant and may indicate an internal state mismatch.

  2. Pending Sweeps Incorrectly Displayed:
    The "Pending Sweeps" section lists transactions as pending even when they are fully confirmed on the mempool. This discrepancy suggests that LND is not correctly updating its internal state or transaction status.

  3. Channels Stuck in "Waiting to Close" State:
    Channels that have been closed with confirmed transactions remain listed as "Waiting to Close". This state should be updated automatically once the closure is confirmed, but the node does not reflect this. Not sure what is causing as these dont even transition into the pending force closure state.

  4. Database Query Overload at Startup:
    Each startup triggers an excessive number of PostgreSQL queries, leading to a bottleneck. Many queries get stuck, and if not manually terminated, they eventually cause the node to crash.

Expected Behavior:

  • Once close transactions are confirmed, channels should transition from "Pending Force Close" or "Waiting to Close" to a fully closed state assuming all parts have also matured to expected block height.
  • "Pending Sweeps" should accurately reflect the confirmed status of transactions.
  • Startup should not generate an excessive volume of PostgreSQL queries which are never resolved.

Additional Information:

  • LND version: v0.18.5
  • PostgreSQL version: 15.4
  • Bitcoin Core Version: v28.1.0
  • Operating System: Ubuntu x86_64 GNU/Linux

This issue significantly impacts node stability and operational reliability, particularly during restarts or when managing channel closures. Any insights or guidance on addressing this behavior would be greatly appreciated. I am unsure what I am looking for in the logs to explain the behaviour so if there is more I can provide just let me know.

Also I know that there is usually a 2 week window of delay for force closures, I am mostly concerned with the fact that these channels have not moved into a forced closure state displaying the maturity_height and blocks_til_maturity. As well hoping to understand why confirmed transactions would still appear in wallet pendingsweeps.

PSQL Example

PSQL Example

Pending Channels Output

{
    "total_limbo_balance":  "66083957",
    "pending_open_channels":  [],
    "pending_closing_channels":  [],
    "pending_force_closing_channels":  [
        {
            "channel":  {
                "remote_node_pub":  "03b65c19de2da9d35895b37e6fa263cefbf8d184e6b61c0713747ebf12409d219f",
                "channel_point":  "18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2",
                "capacity":  "50000000",
                "local_balance":  "49475418",
                "remote_balance":  "522905",
                "local_chan_reserve_sat":  "0",
                "remote_chan_reserve_sat":  "0",
                "initiator":  "INITIATOR_REMOTE",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "",
                "private":  false,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "closing_txid":  "9e0b26d6352c6fc5770e40b1701cfa7c01bbd288c4ee4fe4e5396162af13a5ae",
            "limbo_balance":  "49475748",
            "maturity_height":  886442,
            "blocks_til_maturity":  1934,
            "recovered_balance":  "0",
            "pending_htlcs":  [],
            "anchor":  "LIMBO"
        },
        {
            "channel":  {
                "remote_node_pub":  "032ae3168ba52314da581d6b6693c562b437a9cf805933d4d69e7801547e07302e",
                "channel_point":  "1ee8abab4991c14d4c9c3e60f3f2cb1ac09853cb6e6446b63c031b401f222626:1",
                "capacity":  "20000000",
                "local_balance":  "9902354",
                "remote_balance":  "9922390",
                "local_chan_reserve_sat":  "0",
                "remote_chan_reserve_sat":  "0",
                "initiator":  "INITIATOR_REMOTE",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "",
                "private":  false,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "closing_txid":  "5f4aac37834c34aea8a41f823112e929ff2b63bd20e003199c2e867766b29885",
            "limbo_balance":  "10076489",
            "maturity_height":  885175,
            "blocks_til_maturity":  667,
            "recovered_balance":  "330",
            "pending_htlcs":  [
                {
                    "incoming":  false,
                    "amount":  "25001",
                    "outpoint":  "c0341757138a4e6632e73758e8ae0e9484ec6a4ca36ac5bcbcec3809953876f2:0",
                    "maturity_height":  883157,
                    "blocks_til_maturity":  -1351,
                    "stage":  1
                },
                {
                    "incoming":  false,
                    "amount":  "37509",
                    "outpoint":  "8f41386086663d64a2a47084f668d0c30d7dc1884c9870a3fd608b879ccff3b6:0",
                    "maturity_height":  883431,
                    "blocks_til_maturity":  -1077,
                    "stage":  1
                },
                {
                    "incoming":  false,
                    "amount":  "51489",
                    "outpoint":  "1e7fe71c6c575d03d12a118e3d90135b3beff24f711e29af695343b2730f2574:0",
                    "maturity_height":  883220,
                    "blocks_til_maturity":  -1288,
                    "stage":  1
                },
                {
                    "incoming":  false,
                    "amount":  "60136",
                    "outpoint":  "725b1d8d8d87b2fb15c8dcc558b9bde4d2a2bd1e5459e0daafd8a2d64285328f:0",
                    "maturity_height":  883393,
                    "blocks_til_maturity":  -1115,
                    "stage":  1
                }
            ],
            "anchor":  "RECOVERED"
        },
        {
            "channel":  {
                "remote_node_pub":  "02bd87b47c03654eeda9d8ba96ae2f1a73543ce703721326801d2945e5ef173475",
                "channel_point":  "3d6c98bcc39e106987c65678d36f5845163f85373c6f8fbfd5f09872d795ee52:1",
                "capacity":  "10000000",
                "local_balance":  "2670208",
                "remote_balance":  "6932571",
                "local_chan_reserve_sat":  "0",
                "remote_chan_reserve_sat":  "0",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "",
                "private":  false,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "closing_txid":  "f50b8a3c0475b6eb02d46b6c21984aedfcd48219d7de1fefbf775ca1889c25a7",
            "limbo_balance":  "0",
            "maturity_height":  0,
            "blocks_til_maturity":  0,
            "recovered_balance":  "330",
            "pending_htlcs":  [],
            "anchor":  "RECOVERED"
        },
        {
            "channel":  {
                "remote_node_pub":  "02cc7f5d4d167ec871341f8f2a4c175546b71b417166ac7a9ac5b5793678f15d1b",
                "channel_point":  "a33b675ecfa23cfe02ed1522e63c89959ad4df509c61aea8f886a8bb6b5f3d8c:25",
                "capacity":  "2000000",
                "local_balance":  "27971",
                "remote_balance":  "1902337",
                "local_chan_reserve_sat":  "0",
                "remote_chan_reserve_sat":  "0",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "STATIC_REMOTE_KEY",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "",
                "private":  false,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "closing_txid":  "753c98bc131e148c487d0eb7a3324c586ed244d0d35341adab55fa454020efc1",
            "limbo_balance":  "68949",
            "maturity_height":  0,
            "blocks_til_maturity":  0,
            "recovered_balance":  "0",
            "pending_htlcs":  [
                {
                    "incoming":  true,
                    "amount":  "68949",
                    "outpoint":  "753c98bc131e148c487d0eb7a3324c586ed244d0d35341adab55fa454020efc1:1",
                    "maturity_height":  884704,
                    "blocks_til_maturity":  196,
                    "stage":  1
                }
            ],
            "anchor":  "LIMBO"
        },
        {
            "channel":  {
                "remote_node_pub":  "02c4ad243f18d8ed3a40e98725bc6a795b941d7e3d64be9f7dab38144f1eb09cb8",
                "channel_point":  "8c0fe38f8d7344585a70540cc903bb537e3e783cb135d71617a479dafa06cdb3:0",
                "capacity":  "10000000",
                "local_balance":  "5718523",
                "remote_balance":  "3899085",
                "local_chan_reserve_sat":  "0",
                "remote_chan_reserve_sat":  "0",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "",
                "private":  false,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "closing_txid":  "b3ff92833fd0d9a9d63738318ce3d1e232af6806bcc88d908fd7cef73d7cb772",
            "limbo_balance":  "215866",
            "maturity_height":  0,
            "blocks_til_maturity":  0,
            "recovered_balance":  "330",
            "pending_htlcs":  [
                {
                    "incoming":  false,
                    "amount":  "215866",
                    "outpoint":  "65647085079d74a2818dd9ab1bd2056fed8d00eb04f5dfd3320018bf73f1c781:0",
                    "maturity_height":  883638,
                    "blocks_til_maturity":  -870,
                    "stage":  1
                }
            ],
            "anchor":  "RECOVERED"
        }
    ],
    "waiting_close_channels":  [
        {
            "channel":  {
                "remote_node_pub":  "023e09c43b215bd3dbf483bcb409da3322ea5ea3b046f74698b89ee9ea785dd30a",
                "channel_point":  "00b688b8a3744d2acf05e850a44431f71b145aaf74a71d099b9c3629073c63b4:3",
                "capacity":  "10000000",
                "local_balance":  "100890",
                "remote_balance":  "9898113",
                "local_chan_reserve_sat":  "100000",
                "remote_chan_reserve_sat":  "100000",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "1",
                "chan_status_flags":  "ChanStatusCoopBroadcasted|ChanStatusRemoteCloseInitiator",
                "private":  false,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "limbo_balance":  "100890",
            "commitments":  {
                "local_txid":  "10244f03e25c662a00631e908faadd1c9c9af63e3f79b6f6ef1f3138c83634ef",
                "remote_txid":  "03e1c335faea1d0e71c82b3733839b32e1bc42985eddf046a6a4c3b01027b886",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "336",
                "remote_commit_fee_sat":  "336",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  "27d28158f84cd7d3376c497587fc245bbe0ea91f14264e5d51536d7f2b8bfd36",
            "closing_tx_hex":  ""
        },
        {
            "channel":  {
                "remote_node_pub":  "028ba0d4be716834f38a8f1a9203855f48feba85ca260e5024703d365b8571705b",
                "channel_point":  "ebdba1f9d2a671c43a5a7a2d15b0871bcb9d8c8dbd90f739f990c604220b5e5b:14",
                "capacity":  "2000000",
                "local_balance":  "22982",
                "remote_balance":  "1976045",
                "local_chan_reserve_sat":  "20000",
                "remote_chan_reserve_sat":  "20000",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "156",
                "chan_status_flags":  "ChanStatusCoopBroadcasted|ChanStatusRemoteCloseInitiator",
                "private":  false,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "limbo_balance":  "22982",
            "commitments":  {
                "local_txid":  "fd732c31fc407ab4b1b4c8bebc122a5624cefb5c76f203c5ab2070f052bd36cb",
                "remote_txid":  "b4d73f0a70eae3397b218f5115069795d6219ec896c88372cdb4674ff2bb1bf8",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "312",
                "remote_commit_fee_sat":  "312",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  "d50c4590cefec6a5b43fe0ceec3ae4415e01db7bc490655f819c3e66e4af60ae",
            "closing_tx_hex":  ""
        },
        {
            "channel":  {
                "remote_node_pub":  "02b9d77259938b94f948bb166b688101464cb60752b95221311ef3dbb01c72f967",
                "channel_point":  "84bd71b8cad47b2dd812f609d6360afa3118f4918858888eea7caa9327c7f45a:0",
                "capacity":  "9000000",
                "local_balance":  "91987",
                "remote_balance":  "8906789",
                "local_chan_reserve_sat":  "90000",
                "remote_chan_reserve_sat":  "90000",
                "initiator":  "INITIATOR_REMOTE",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "142",
                "chan_status_flags":  "ChanStatusBorked|ChanStatusCommitBroadcasted|ChanStatusLocalCloseInitiator",
                "private":  false,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "limbo_balance":  "91987",
            "commitments":  {
                "local_txid":  "55eea93792a9403d88bc42d1a4e05399b2c9d6623f8168da9fb3e268b6ac89bc",
                "remote_txid":  "e73e13dce8b8eca937820edb89138ae261582f6ffea3bf584767182a444a789b",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "563",
                "remote_commit_fee_sat":  "563",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  "55eea93792a9403d88bc42d1a4e05399b2c9d6623f8168da9fb3e268b6ac89bc",
            "closing_tx_hex":  ""
        },
        {
            "channel":  {
                "remote_node_pub":  "02dd4a9cf224e290851f6af83072e2496479bdf4b65e1733e92e24d42a0ba89a75",
                "channel_point":  "785bbe84183391ed381132bca5650a2f631f0cada709a66de8f0277edea05f35:1",
                "capacity":  "2000000",
                "local_balance":  "21397",
                "remote_balance":  "1977379",
                "local_chan_reserve_sat":  "20000",
                "remote_chan_reserve_sat":  "20000",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "1",
                "chan_status_flags":  "ChanStatusCoopBroadcasted|ChanStatusRemoteCloseInitiator",
                "private":  false,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "limbo_balance":  "21397",
            "commitments":  {
                "local_txid":  "e04fa97c94887f2686f12c22fd352f81afdc73534a6763457ba9a5e68c38ad68",
                "remote_txid":  "3d5397d522406eba875339c07f20bbecc1cac055a4be2997b5998c7dfbd2b240",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "563",
                "remote_commit_fee_sat":  "563",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  "8a5d6d26b3c93679e7b822e47fb5f4026c8ab8711d73ff0145a45f81fafd31ee",
            "closing_tx_hex":  ""
        },
        {
            "channel":  {
                "remote_node_pub":  "030a0504e5726782e1c344837b7bfcdc750ed34748fde02b0641c7a73c5f9bbf01",
                "channel_point":  "73203d79cb5d3f333b6279e29bb6f6c55526c76cc940178b540c776f9af5747b:1",
                "capacity":  "2000000",
                "local_balance":  "30903",
                "remote_balance":  "1968123",
                "local_chan_reserve_sat":  "20000",
                "remote_chan_reserve_sat":  "20000",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "346",
                "chan_status_flags":  "ChanStatusCoopBroadcasted|ChanStatusRemoteCloseInitiator",
                "private":  false,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "limbo_balance":  "30903",
            "commitments":  {
                "local_txid":  "869303288986f4cfd44fa434c14fe6cd938bac953047d7cb33bb6d1c8406a0f9",
                "remote_txid":  "3dd4596d4babd7d3bb38556b3190f4f45fa1b4e5dbd533a8cad12f323bc9a80c",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "313",
                "remote_commit_fee_sat":  "313",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  "",
            "closing_tx_hex":  ""
        },
        {
            "channel":  {
                "remote_node_pub":  "0336350b10294b9d8759944709db961cb9ef8e4c7d3a80d057684ae5b0d841c101",
                "channel_point":  "058107f6e8633b5d31f00629879c01591d5b63239b0cd83a64f7c379acc2b3b7:5",
                "capacity":  "2000000",
                "local_balance":  "1973214",
                "remote_balance":  "25813",
                "local_chan_reserve_sat":  "20000",
                "remote_chan_reserve_sat":  "20000",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "166",
                "chan_status_flags":  "ChanStatusBorked|ChanStatusCommitBroadcasted|ChanStatusLocalCloseInitiator",
                "private":  false,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "limbo_balance":  "1973214",
            "commitments":  {
                "local_txid":  "368d1dc3b0464d2d09c1e2759611b178598a21577ad9ed92801a5a661444e1a1",
                "remote_txid":  "833e1a86de000805ef594e959a8510cdd41293017b6451e0e7b73837ac53c518",
                "remote_pending_txid":  "66ab16fb98cbfe697692a738265a01725882e8d98e7b244a5c136fda4570b662",
                "local_commit_fee_sat":  "312",
                "remote_commit_fee_sat":  "360",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  "368d1dc3b0464d2d09c1e2759611b178598a21577ad9ed92801a5a661444e1a1",
            "closing_tx_hex":  ""
        },
        {
            "channel":  {
                "remote_node_pub":  "037f66e84e38fc2787d578599dfe1fcb7b71f9de4fb1e453c5ab85c05f5ce8c2e3",
                "channel_point":  "2f78990960d9c76adb03edeb4fce04963e7839c9148e2ab3f2a7644a75f46e19:15",
                "capacity":  "5000000",
                "local_balance":  "56167",
                "remote_balance":  "4942860",
                "local_chan_reserve_sat":  "50000",
                "remote_chan_reserve_sat":  "50000",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "58",
                "chan_status_flags":  "ChanStatusCoopBroadcasted|ChanStatusRemoteCloseInitiator",
                "private":  false,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "limbo_balance":  "56167",
            "commitments":  {
                "local_txid":  "4d81e53acc663a7e062702008acd46d7201cf9a9661913e8c4944cb9abdc1d67",
                "remote_txid":  "25c62b90652ca1038cf4428f20e23cbb784a7037dac3800c2cef16040e8c4785",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "312",
                "remote_commit_fee_sat":  "312",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  "e9f96bdde43c52d088c594165b3a7c6e6c6a477b3ed2160c0efb4ec91125bb6c",
            "closing_tx_hex":  ""
        },
        {
            "channel":  {
                "remote_node_pub":  "03d5e14480d49d3bfe65cb87d03a863712c606aa8e6720567743ee0a1cdef5fef2",
                "channel_point":  "8e735dba3c3bd0243ba71c519b0fe82a84e06a9f3c46d9332a168df4ea8741d2:1",
                "capacity":  "5000000",
                "local_balance":  "3897200",
                "remote_balance":  "1101576",
                "local_chan_reserve_sat":  "50000",
                "remote_chan_reserve_sat":  "50000",
                "initiator":  "INITIATOR_REMOTE",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "20",
                "chan_status_flags":  "ChanStatusBorked|ChanStatusCommitBroadcasted|ChanStatusLocalCloseInitiator",
                "private":  false,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "limbo_balance":  "3897200",
            "commitments":  {
                "local_txid":  "2365cebffb9515f0d7aa0b0a1e5b3dd515a576856e7061599b94e475994d4813",
                "remote_txid":  "464695637aa22be2be9f77780e5362fb044bbd64d06eb15b4f30f384f201464b",
                "remote_pending_txid":  "56d949834c7080f9c37c5ce5fa962840399906eb8853f6b2316af4c406c2e6cc",
                "local_commit_fee_sat":  "563",
                "remote_commit_fee_sat":  "649",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  "2365cebffb9515f0d7aa0b0a1e5b3dd515a576856e7061599b94e475994d4813",
            "closing_tx_hex":  ""
        },
        {
            "channel":  {
                "remote_node_pub":  "03e438d94f95f635d8e19b0d6e4aaa75d15e7b0fd8f28f674ca47043aa8ab677ac",
                "channel_point":  "e528c19815e85ee04a35a3d04c976a5c8868ee71e638e213e27826b87e594b84:30",
                "capacity":  "2000000",
                "local_balance":  "52165",
                "remote_balance":  "1946825",
                "local_chan_reserve_sat":  "20000",
                "remote_chan_reserve_sat":  "20000",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "2",
                "chan_status_flags":  "ChanStatusCoopBroadcasted|ChanStatusRemoteCloseInitiator",
                "private":  false,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "limbo_balance":  "52165",
            "commitments":  {
                "local_txid":  "136ce7ee67c36c097d3d0aeee8d581b992e0ae64107f1ebcafd818ac118a5893",
                "remote_txid":  "545f5ec9d2d58e138fcf97dfc316a989b747712ac540569112a41b704c494abc",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "349",
                "remote_commit_fee_sat":  "349",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  "07a020f2a36162a569f57b2bbcf2374da9812cd6175d620c91531cbd8c7fb061",
            "closing_tx_hex":  ""
        }
    ]
}

Pending Sweeps Output

{
    "pending_sweeps": [
        {
            "outpoint": "9e0b26d6352c6fc5770e40b1701cfa7c01bbd288c4ee4fe4e5396162af13a5ae:1",
            "witness_type": "COMMITMENT_ANCHOR",
            "amount_sat": 330,
            "sat_per_vbyte": 0,
            "broadcast_attempts": 0,
            "requested_sat_per_vbyte": 0,
            "immediate": false,
            "budget": 330,
            "deadline_height": 885516,
            "next_broadcast_height": 0,
            "requested_conf_target": 0,
            "force": false
        },
        {
            "outpoint": "55eea93792a9403d88bc42d1a4e05399b2c9d6623f8168da9fb3e268b6ac89bc:0",
            "witness_type": "COMMITMENT_ANCHOR",
            "amount_sat": 330,
            "sat_per_vbyte": 0,
            "broadcast_attempts": 0,
            "requested_sat_per_vbyte": 0,
            "immediate": false,
            "budget": 330,
            "deadline_height": 885516,
            "next_broadcast_height": 0,
            "requested_conf_target": 0,
            "force": false
        },
        {
            "outpoint": "f50b8a3c0475b6eb02d46b6c21984aedfcd48219d7de1fefbf775ca1889c25a7:0",
            "witness_type": "COMMITMENT_ANCHOR",
            "amount_sat": 330,
            "sat_per_vbyte": 0,
            "broadcast_attempts": 2,
            "requested_sat_per_vbyte": 0,
            "immediate": false,
            "budget": 330,
            "deadline_height": 885516,
            "next_broadcast_height": 0,
            "requested_conf_target": 0,
            "force": false
        },
        {
            "outpoint": "56d949834c7080f9c37c5ce5fa962840399906eb8853f6b2316af4c406c2e6cc:1",
            "witness_type": "COMMITMENT_ANCHOR",
            "amount_sat": 330,
            "sat_per_vbyte": 0,
            "broadcast_attempts": 0,
            "requested_sat_per_vbyte": 0,
            "immediate": false,
            "budget": 330,
            "deadline_height": 885516,
            "next_broadcast_height": 0,
            "requested_conf_target": 0,
            "force": false
        },
        {
            "outpoint": "b3ff92833fd0d9a9d63738318ce3d1e232af6806bcc88d908fd7cef73d7cb772:1",
            "witness_type": "COMMITMENT_ANCHOR",
            "amount_sat": 330,
            "sat_per_vbyte": 0,
            "broadcast_attempts": 2,
            "requested_sat_per_vbyte": 0,
            "immediate": false,
            "budget": 330,
            "deadline_height": 885516,
            "next_broadcast_height": 0,
            "requested_conf_target": 0,
            "force": false
        },
        {
            "outpoint": "e73e13dce8b8eca937820edb89138ae261582f6ffea3bf584767182a444a789b:0",
            "witness_type": "COMMITMENT_ANCHOR",
            "amount_sat": 330,
            "sat_per_vbyte": 0,
            "broadcast_attempts": 0,
            "requested_sat_per_vbyte": 0,
            "immediate": false,
            "budget": 330,
            "deadline_height": 885516,
            "next_broadcast_height": 0,
            "requested_conf_target": 0,
            "force": false
        },
        {
            "outpoint": "2365cebffb9515f0d7aa0b0a1e5b3dd515a576856e7061599b94e475994d4813:1",
            "witness_type": "COMMITMENT_ANCHOR",
            "amount_sat": 330,
            "sat_per_vbyte": 0,
            "broadcast_attempts": 0,
            "requested_sat_per_vbyte": 0,
            "immediate": false,
            "budget": 330,
            "deadline_height": 885516,
            "next_broadcast_height": 0,
            "requested_conf_target": 0,
            "force": false
        },
        {
            "outpoint": "5f4aac37834c34aea8a41f823112e929ff2b63bd20e003199c2e867766b29885:1",
            "witness_type": "COMMITMENT_ANCHOR",
            "amount_sat": 330,
            "sat_per_vbyte": 0,
            "broadcast_attempts": 0,
            "requested_sat_per_vbyte": 0,
            "immediate": false,
            "budget": 330,
            "deadline_height": 885516,
            "next_broadcast_height": 0,
            "requested_conf_target": 0,
            "force": false
        },
        {
            "outpoint": "464695637aa22be2be9f77780e5362fb044bbd64d06eb15b4f30f384f201464b:1",
            "witness_type": "COMMITMENT_ANCHOR",
            "amount_sat": 330,
            "sat_per_vbyte": 0,
            "broadcast_attempts": 0,
            "requested_sat_per_vbyte": 0,
            "immediate": false,
            "budget": 330,
            "deadline_height": 885516,
            "next_broadcast_height": 0,
            "requested_conf_target": 0,
            "force": false
        },
        {
            "outpoint": "833e1a86de000805ef594e959a8510cdd41293017b6451e0e7b73837ac53c518:1",
            "witness_type": "COMMITMENT_ANCHOR",
            "amount_sat": 330,
            "sat_per_vbyte": 0,
            "broadcast_attempts": 0,
            "requested_sat_per_vbyte": 0,
            "immediate": false,
            "budget": 330,
            "deadline_height": 885516,
            "next_broadcast_height": 0,
            "requested_conf_target": 0,
            "force": false
        },
        {
            "outpoint": "f50b8a3c0475b6eb02d46b6c21984aedfcd48219d7de1fefbf775ca1889c25a7:3",
            "witness_type": "HTLC_OFFERED_REMOTE_TIMEOUT",
            "amount_sat": 58313,
            "sat_per_vbyte": 0,
            "broadcast_attempts": 0,
            "requested_sat_per_vbyte": 0,
            "immediate": true,
            "budget": 116626,
            "deadline_height": 885516,
            "next_broadcast_height": 0,
            "requested_conf_target": 0,
            "force": false
        },
        {
            "outpoint": "f50b8a3c0475b6eb02d46b6c21984aedfcd48219d7de1fefbf775ca1889c25a7:2",
            "witness_type": "HTLC_OFFERED_REMOTE_TIMEOUT",
            "amount_sat": 47613,
            "sat_per_vbyte": 0,
            "broadcast_attempts": 0,
            "requested_sat_per_vbyte": 0,
            "immediate": true,
            "budget": 95226,
            "deadline_height": 885516,
            "next_broadcast_height": 0,
            "requested_conf_target": 0,
            "force": false
        },
        {
            "outpoint": "c160c37638bd249dded33362c99bb67f571ff45829fcddfa826480050318be79:1",
            "witness_type": "COMMITMENT_ANCHOR",
            "amount_sat": 330,
            "sat_per_vbyte": 0,
            "broadcast_attempts": 0,
            "requested_sat_per_vbyte": 0,
            "immediate": false,
            "budget": 330,
            "deadline_height": 885516,
            "next_broadcast_height": 0,
            "requested_conf_target": 0,
            "force": false
        },
        {
            "outpoint": "368d1dc3b0464d2d09c1e2759611b178598a21577ad9ed92801a5a661444e1a1:1",
            "witness_type": "COMMITMENT_ANCHOR",
            "amount_sat": 330,
            "sat_per_vbyte": 0,
            "broadcast_attempts": 0,
            "requested_sat_per_vbyte": 0,
            "immediate": false,
            "budget": 330,
            "deadline_height": 885516,
            "next_broadcast_height": 0,
            "requested_conf_target": 0,
            "force": false
        },
        {
            "outpoint": "66ab16fb98cbfe697692a738265a01725882e8d98e7b244a5c136fda4570b662:1",
            "witness_type": "COMMITMENT_ANCHOR",
            "amount_sat": 330,
            "sat_per_vbyte": 0,
            "broadcast_attempts": 0,
            "requested_sat_per_vbyte": 0,
            "immediate": false,
            "budget": 45752,
            "deadline_height": 884519,
            "next_broadcast_height": 0,
            "requested_conf_target": 0,
            "force": false
        }
    ]
}

Log

  • Example of logged output related to a channel stuck in waiting to close
2025-02-19 22:24:21.934 [TRC] CNCT: Attempting to stop ChainWatcher(18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2)
2025-02-19 22:24:21.950 [TRC] CNCT: Attempting to stop ChannelArbitrator(18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2)
2025-02-19 22:24:21.950 [DBG] CNCT: Stopping ChannelArbitrator(18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2)
2025-02-19 22:38:24.331 [INF] CNCT: Re-publishing force close tx(9e0b26d6352c6fc5770e40b1701cfa7c01bbd288c4ee4fe4e5396162af13a5ae) for channel 18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2
2025-02-19 22:38:24.348 [WRN] CNCT: Channel 18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2 is in state ChanStatusCoopBroadcasted, but no coop closing tx to re-publish...
2025-02-19 22:38:26.347 [INF] CNCT: Close observer for ChannelPoint(18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2) active
2025-02-19 22:38:26.935 [INF] CNCT: ChannelArbitrator(18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2): starting state=StateCommitmentBroadcasted, trigger=chainTrigger, triggerHeight=884508
2025-02-19 22:38:28.740 [INF] CNCT: ChannelArbitrator(18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2): no HTLCs at stake, sweeping anchor with default deadline
2025-02-19 22:38:28.741 [INF] CNCT: ChannelArbitrator(18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2): offering anchor from remote commitment c160c37638bd249dded33362c99bb67f571ff45829fcddfa826480050318be79:1 to sweeper with deadline=None, budget=0.00000330 BTC
2025-02-19 22:38:28.822 [INF] CNCT: ChannelArbitrator(18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2): no HTLCs at stake, sweeping anchor with default deadline
2025-02-19 22:38:28.822 [INF] CNCT: ChannelArbitrator(18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2): offering anchor from local commitment 9e0b26d6352c6fc5770e40b1701cfa7c01bbd288c4ee4fe4e5396162af13a5ae:1 to sweeper with deadline=None, budget=0.00000330 BTC
2025-02-19 22:38:29.081 [INF] CNCT: ChannelArbitrator(18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2): trigger chainTrigger moving from state StateCommitmentBroadcasted to StateCommitmentBroadcasted
2025-02-19 22:38:33.322 [INF] NTFN: Historical spend dispatch finished for request outpoint=18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2, script=0 6896c6edb9fdda9368e1f715cd1e7c702f644b2be7538aec34af7a50718250f5 (start=884426 end=884508) with details: 9e0b26d6352c6fc5770e40b1701cfa7c01bbd288c4ee4fe4e5396162af13a5ae[0] spending 18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2 at height=884426
2025-02-19 22:38:33.331 [INF] CNCT: Local unilateral close of ChannelPoint(18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2) detected
2025-02-19 22:38:33.332 [INF] CNCT: ChannelArbitrator(18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2): local on-chain channel close
2025-02-19 22:38:59.083 [WRN] CHBU: Replacing disk backup for ChannelPoint(18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2) w/ newer version
2025-02-19 22:39:30.901 [INF] CNCT: ChannelArbitrator(18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2): trigger localCloseTrigger moving from state StateCommitmentBroadcasted to StateContractClosed
2025-02-19 22:39:31.148 [INF] CNCT: ChannelArbitrator(18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2): still awaiting contract resolution
2025-02-19 22:39:55.083 [WRN] CHBU: Replacing disk backup for ChannelPoint(18769fb7c4d29ca3e7df2f397444457aab458a7e405be3c7bfcad06b67402419:2) w/ newer version
  • Example of attempted rebroadcast of already broadcasted transaction
2025-02-19 22:38:24.444 [INF] CNCT: Re-publishing coop close tx(2bccf7bb2b554f6f5d42fe1bbb8c32a153905bd1b1921d0b37c8bd990638d021) for channel b380e5a88d21180f6ea643db5c1496aaa2e7fd71975af327d8733be96febbd5e:2
2025-02-19 22:38:24.462 [ERR] BTCN: Broadcast attempt failed: Transaction already in block chain
2025-02-19 22:38:24.463 [WRN] BTWL: Transaction 2bccf7bb2b554f6f5d42fe1bbb8c32a153905bd1b1921d0b37c8bd990638d021 not accepted by mempool: txn-already-known
2025-02-19 22:38:24.471 [INF] LNWL: 2bccf7bb2b554f6f5d42fe1bbb8c32a153905bd1b1921d0b37c8bd990638d021: tx already confirmed
2025-02-19 22:38:24.478 [INF] NTFN: New confirmation subscription: conf_id=10, txid=2bccf7bb2b554f6f5d42fe1bbb8c32a153905bd1b1921d0b37c8bd990638d021, num_confs=6 height_hint=884508
@jjacks95 jjacks95 added bug Unintended code behaviour needs triage labels Feb 20, 2025
@guggero
Copy link
Collaborator

guggero commented Feb 20, 2025

Does your wallet show as being synced? Is your chain backend a pruned node?
Sounds like the chain re-scans don't finish properly for some reason. Do you see any errors related to that?
Might be worth trying starting with --reset-wallet-transactions, but that will take a while.

@jjacks95
Copy link
Author

Wallet seems to be synced as I was able to open other channels while this was happening. Backend is full bitcoind node.

What type of error would I be looking for so I can search the logs? I do suspect that a reset may fix the issue just trying to exhaust other possible options to avoid the lengthy downtime. Also would love to discover why this happened, and if it was preventable.

@guggero
Copy link
Collaborator

guggero commented Feb 21, 2025

Can you just grep by [ERR] please? You can send me the result directly if you prefer that. I'm on Keybase or Slack with the same username.

@yyforyongyu
Copy link
Member

You may want to change the config values for the budget, since some of the HTLCs have already reached their deadlines, by default half of their value will be used as fees when they are swept,

lnd/sample-lnd.conf

Lines 1834 to 1872 in 27440e8

[sweeper.budget]
; The amount in satoshis to allocate as the budget to pay fees when sweeping
; the to_local output. If set, the budget calculated using the ratio (if set)
; will be capped at this value.
; sweeper.budget.tolocal=
; The ratio of the value in to_local output to allocate as the budget to pay
; fees when sweeping it.
; sweeper.budget.tolocalratio=0.5
; The amount in satoshis to allocate as the budget to pay fees when CPFPing a
; force close tx using the anchor output. If set, the budget calculated using
; the ratio (if set) will be capped at this value.
; sweeper.budget.anchorcpfp=
; The ratio of a special value to allocate as the budget to pay fees when
; CPFPing a force close tx using the anchor output. The special value is the
; sum of all time-sensitive HTLCs on this commitment subtracted by their
; budgets.
; sweeper.budget.anchorcpfpratio=0.5
; The amount in satoshis to allocate as the budget to pay fees when sweeping a
; time-sensitive (first-level) HTLC. If set, the budget calculated using the
; ratio (if set) will be capped at this value.
; sweeper.budget.deadlinehtlc=
; The ratio of the value in a time-sensitive (first-level) HTLC to allocate as
; the budget to pay fees when sweeping it.
; sweeper.budget.deadlinehtlcratio=0.5
; The amount in satoshis to allocate as the budget to pay fees when sweeping a
; non-time-sensitive (second-level) HTLC. If set, the budget calculated using
; the ratio (if set) will be capped at this value.
; sweeper.budget.nodeadlinehtlc=
; The ratio of the value in a non-time-sensitive (second-level) HTLC to
; allocate as the budget to pay fees when sweeping it.
; sweeper.budget.nodeadlinehtlcratio=0.5

@ziggie1984
Copy link
Collaborator

ziggie1984 commented Feb 21, 2025 •
edited
Loading

Can you try disabling the height_hint cache and check if the closing transactions resolve ?

in the lnd.conf
height-hint-cache-query-disable=true

@jjacks95
Copy link
Author

Can you try disabling the height_hint cache and check if the closing transactions resolve ?

in the lnd.conf height-hint-cache-query-disable=true

Thanks I will try this and report back.

@jjacks95
Copy link
Author

jjacks95 commented Feb 21, 2025 •
edited
Loading

Can you just grep by [ERR] please? You can send me the result directly if you prefer that. I'm on Keybase or Slack with the same username.

@guggero Ok will do with the next restart, and send you some output.

@jjacks95
Copy link
Author

You may want to change the config values for the budget, since some of the HTLCs have already reached their deadlines, by default half of their value will be used as fees when they are swept,

lnd/sample-lnd.conf

Lines 1834 to 1872 in 27440e8

[sweeper.budget]

; The amount in satoshis to allocate as the budget to pay fees when sweeping
; the to_local output. If set, the budget calculated using the ratio (if set)
; will be capped at this value.
; sweeper.budget.tolocal=

; The ratio of the value in to_local output to allocate as the budget to pay
; fees when sweeping it.
; sweeper.budget.tolocalratio=0.5

; The amount in satoshis to allocate as the budget to pay fees when CPFPing a
; force close tx using the anchor output. If set, the budget calculated using
; the ratio (if set) will be capped at this value.
; sweeper.budget.anchorcpfp=

; The ratio of a special value to allocate as the budget to pay fees when
; CPFPing a force close tx using the anchor output. The special value is the
; sum of all time-sensitive HTLCs on this commitment subtracted by their
; budgets.
; sweeper.budget.anchorcpfpratio=0.5

; The amount in satoshis to allocate as the budget to pay fees when sweeping a
; time-sensitive (first-level) HTLC. If set, the budget calculated using the
; ratio (if set) will be capped at this value.
; sweeper.budget.deadlinehtlc=

; The ratio of the value in a time-sensitive (first-level) HTLC to allocate as
; the budget to pay fees when sweeping it.
; sweeper.budget.deadlinehtlcratio=0.5

; The amount in satoshis to allocate as the budget to pay fees when sweeping a
; non-time-sensitive (second-level) HTLC. If set, the budget calculated using
; the ratio (if set) will be capped at this value.
; sweeper.budget.nodeadlinehtlc=

; The ratio of the value in a non-time-sensitive (second-level) HTLC to
; allocate as the budget to pay fees when sweeping it.
; sweeper.budget.nodeadlinehtlcratio=0.5

@yyforyongyu are you suggesting I increase the ratio? I am not sure that the issue is related to the economical sweeping seems a lot of the pending sweeps have been broadcasted and confirmed but not updated.

@Roasbeef
Copy link
Member

@jjacks95 is the bitciond node running with a txindex activated?

@jjacks95
Copy link
Author

@Roasbeef Yes it is

@guggero
Copy link
Collaborator

guggero commented Feb 24, 2025

Thanks for the logs. So I see a lot of these errors:

GRPH: process network updates got: unable to add node 030f35e2e50cd3e097cb243e137191d28b9628ed85b5e24b0a56c8940a8c51b1c3 to the graph: db tx retries exceeded
unable to add node 022c402216eee911831d0d21e8ff5d9f2ac664b7c24cb674cf9e90b00f99a3d7c3 to the graph: db tx retries exceeded

In addition with the fact that there are a lot of pending queries for the walletdb_kv bucket, I think there might be another deadlock in the wallet code.
@Roasbeef I think we might need to revert the removal of the write lock, at least for the wallet DB, until we've re-written that code to not lock up so easily...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Unintended code behaviour needs triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Roasbeef @guggero @yyforyongyu @jjacks95 @ziggie1984