Recoverer/claim broadcasting race causes permanent duplicate payment in local instance

[danielgranhao]

The issue occurs as follows:

1. Atrack_new_blocks loop iteration starts, and does a full scan of the liquid onchain wallet
2. A receive swap claim tx is broadcasted (and the claim tx id is set in the db)
3. The same track_new_blocks iteration continues and tries to recover data from the swap. 2 txs are found in the lockup script history, both pending (due to 0-conf), but neither are known local wallet txs (wallet is out of date), so we see the log: Found unconfirmed lockup and refund txs while recovering data for Receive Swap Jusa6EMuaCRd. End result is that the swap gets both its lockup and claim tx id cleared.
4. Swapper sends duplicate update transaction.mempool (see Network issues can cause same swap to be subscribed to multiple times #704). Given the claim tx id was erased, we proceed with broadcasting a new claim tx
5. Only one of the claim txs confirms, but the other still gets added to the payment_tx_data table. There is no implemented mechanism to clean this table, causing a permanent pending payment to appear in the payments list.

We should:

• Prevent the race issue between the claiming and the recoverer tasks
• As a fallback, add some mechanism for cleaning the payment_tx_data table as part of the recovery - Clean payment txs data on sync #731

[roeierez]

Great analysis @danielgranhao !