Lightning: Transactions sometimes not broadcasted

[laanwj]

A friend of mine running a lightning node (on testnet) on a Rpi3 has had this problem twice: channels seem to stay stuck in a certain status forever.

We investigated this closer, and it turns out that the transactions were never successfully broadcasted at all!. Neither the local mempool nor blockcypher knows about them.

(no, there is no walletbroadcast=0 in bitcoin.conf)

We looked further and it seems that many of the transactions in the output table are not known to the network.
It cannot withdraw, this gives an error about missing inputs (#767), as one'd expect if the parent transaction is unknown.

This does not seem to be issue #709 as there is no transaction to watch - it's not on the network so will never confirm. It also doesn't seem to be #749, as the client is up to date with the block chain at this point (though it has been out of date in the past).

It might be due to slowness of the system, sometimes the thing becomes as good as unresponsive and I could fully see bitcoin-cli commands timing out. Or maybe some other communication issue between lightningd and bitcoind, or even a bitcoind (fee?) issue.

@kvaciral

[windsok]

I also experienced this once when opening a mainnet channel. I was able to take the transaction and manually submit it to blockchain.info's pushtx interface.

[laanwj]

Are the raw transactions stored somewhere? I couldn't find them, otherwise
I'd try that!

On Jan 27, 2018 20:09, "windsok" [e-mail] wrote:

I also experienced this once when opening a mainnet channel. I was able to
take the transaction and manually submit it to blockchain.info's pushtx
interface.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ElementsProject/lightning/issues/803#issuecomment-361007570,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHutpyqBQdQ4VA8eJ66LDX9J3nIPP7Pks5tO3SCgaJpZM4RvNOH
.

[cdecker]

I have extracted the txs from the logs by grepping for sendrawtransaction and submitted them manually. Seems we get a bit out of sync and that compounds over time. I'll add a tx log to the DB and see if we can rebroadcast them, if they don't confirm.

[windsok]

Are the raw transactions stored somewhere?

The raw funding tx is included in the JSON response of fundchannel, or in the logs as cdecker mentions.

[laanwj]

The raw funding tx is included in the JSON response of fundchannel, or in the logs as cdecker mentions.

The JSON responses were not stored in this case, as this was just manual testing w/ the command line.

And in the log, on only at log level debug, I suppose?

I'll add a tx log to the DB and see if we can rebroadcast them, if they don't confirm.

Yes, a dedicated tx log seems like a useful thing for recovering cases like this.

[jacobyandmeyers]

Advise adding the tx log as utf/ascii vs. blob. Either that or decode on the command line with lightning-cli. Any chance that the raw transaction is stored in the DB in one of those blobs?

Also, this occurred on a high end system with plenty of horsepower (I7). One nuance may be that I have opened connections with every node I found with listnodes ~125. Opened/funded channels with the nodes that seemed "big."

Now that I think about it. I funded several channels quickly. Suspect a race condition.

[fixone]

One other scenario that I've encountered is that the bitcoind refuses to broadcast the transaction (in my case because the fees were considered too small), but because of the fact that lightningd and bitcoind were somehow out of sync, the error message was not correctly interpreted. Now the lightningd waits for channel to lock (CHANNELD_AWAITING_LOCKIN), but bitcoind won't accept the transaction. This creates problems down the line, because subsequent fundchannels will reference inputs that have not been seen. Is there a way to close the channel that's in CHANNELD_AWAITING_LOCKIN state?

[sfmcnally]

are the transactions stored somewhere so that we can rebroadcast them manually?

[sfmcnally]

when I use dev-sign-last-tx and broadcast the transaction manually, I get the following error:
Missing parents for 1d33e85f8ddebe4ed55de4468ce4a16ce10f914df2d991e180d1a349cef98619 while inserting: [adec6492d7ebb0740902913cc56544985af5d995736ca346c52fddabde396315]

[sfmcnally]

here is the raw transaction:
02000000000101156339deabdd2fc546a36c7395d9f55a984465c53c91020974b0ebd79264ecad00000000003530368001867f0100000000002200205c44ab47439eba495f486239888430b81e315a7bb192c3819a7297db794a4d6f04004830450221009e5b7c6095922d754462b5cd6204ee3a629103f599f5c53077393274117cbf1a02206372735122fde7963fec268770e4dbba6062fcae8406159f87c88266ca2210e901473044022061d8412876235ca893cfd7c898065a59cb90014b08a1e18d2c128a724fb0fc9a022062ae2b712b99c27db854b60e3ce047211470d44081112fd92085bac99fe3c9c801475221036116c806190959b3fd69bce1248eb0c97d605f88a09095830d455c60a881eda3210392ce215659b2310d3d8403463836ca10504424a2776d9707fb720906988c1b7a52ae429bfc20

[jacobyandmeyers]

@sfmcnally I have been going through the best odds spot (the lightningd.sqlite3 DB). No luck finding raw transactions so far but that is where I would look. I don't think you will find it :(. Options include grabbing the private key and/or backing transactions out of your output DB (in that sqlite3 file). The DB is not too hard to parse as each of the blobs are hex encoded. I have not had much time to play with reversing transactions today so sorry I don't have better advice.

[sfmcnally]

@cdecker @jacobyandmeyers

This trasnaction did not broadcast:

./lightning-cli fundchannel 031c527494f295220d26a1814c18e7db3f4f03203ce1a53a0b74b52ed570a3972a 100000
{ "tx" : "02000000000102105a6c4b82e28d0063cf5cfdb1da49d888458262cb25008a68451ec7277009750000000000ffffffff156339deabdd2fc546a36c7395d9f55a984465c53c91020974b0ebd79264ecad0100000000ffffffff02a0860100000000002200206adcda7193ed197fceb601335cab3739c5f5206b9c0a986a86daa7b62bba72c463a50100000000001600145f9b6723be1a4c9395976ae96bf87b9d8c1c99fd0247304402204dfb23eb5f53757e03c1db43dfae297a9bab1dc17d2c21895bb2dbaf27efbea002202e76a7b5203e348a0901ffcb2bb477fe63970e17071684459c93cc6b0e1e7fc4012102128910f0a2f8c28eabf3baddeba64f5d0cb75db597337cd83eedf0db8badd013024830450221008ffc4a53f1515bb6890e3d335c607cfbae0e5375ef0a255706f72d9397e35875022012352e99f524c1ce6003df0d51719bc85f0859575d9d0771bbe95a19f3db5c63012102b7d1d954e9be0ee44456b5734c37400bc0e8b2405d6b959c241597e9aa42167100000000", "txid" : "c4092b5757099739e2dfeee9da736d634ea4989cc21ac8cb5a8e4b464dec3ae2" }

when I try to broadcast in manually on blockchain.info, I get this error:

Missing parents for c4092b5757099739e2dfeee9da736d634ea4989cc21ac8cb5a8e4b464dec3ae2 while inserting: [adec6492d7ebb0740902913cc56544985af5d995736ca346c52fddabde396315]

The funds were removed from the lightning wallet. Where did they go?

[cdecker]

We have recently merged PR #842 that should allow lightningd to correctly broadcast transactions again. In addition we created PR #864 so that users can sync with the UTXO set as seen by bitcoind which will thus recover funds that were marked as spent by an unsuccessful funding attempt.

@fixone @laanwj could you pull the latest master, compile and use dev-rescan-outputs to recover funds and attempt to open a new channel (with another peer)?

[ghost]

@cdecker Thanks a lot for implementing "dev-rescan-outputs"!!! Had the same problem as van der Laan's friend (MAINNET): 2 channels in CHANNELD_AWAITING_LOCKIN that had not sent their lock-in transactions and 1 channel in CHANNELD_AWAITING_LOCKIN the lock-in transaction I got confirmed by broadcasting the raw tx via blockchain.info but nevertheless remained stuck in CHANNELD_AWAITING_LOCKIN. I then tried to withdraw funds but that wouldn't work because of missing inputs.

As you requested, this is what I did:
Stopped lightning, git pull; make and restarted lightning. I waited till the lightning block height was synced with bitcoind.
cli/lightning-cli dev-rescan-outputs
{ "outputs" :
[
{ "txid" : "54fd538649816e716f60437e0d24fb00d3181e3cb888d6c6c9b31cc1eba61abe", "output" : 7, "oldstate" : 2, "newstate" : 2 },
{ "txid" : "aeea4f536faa1c2ae812fa839672e52fbc5d89d06cf72ef2ffd036ac83aac00f", "output" : 1, "oldstate" : 0, "newstate" : 2 },
{ "txid" : "cadfc924300bd5f796aec2abea4ca08d4d43a504abe63c5de763e8f7d92725a3", "output" : 1, "oldstate" : 2, "newstate" : 2 },
{ "txid" : "a4208c1b30430319797dd4d1e40fa0db1f7b83187c4c79ff86d55b0b974d380b", "output" : 1, "oldstate" : 2, "newstate" : 0 },
{ "txid" : "e197172a26cf419cebb939113aa831474f9e6a3bd229d45f10d5ab1c1bdeee8b", "output" : 1, "oldstate" : 2, "newstate" : 2 },
{ "txid" : "86b371eb0a156c4963d8eacf9999dd24daa9b207f5787b7b3726689e0ce4fc7b", "output" : 1, "oldstate" : 2, "newstate" : 2 } ] }

Now my withdraw worked! "txid" : "5cf6d093ba7027ec771b7de6f428324ab03a784513322b7dea063207922928ff"

After connecting, funded a channel (fee too low but we got lucky, it confirmed quickly):
cli/lightning-cli fundchannel 0371190acfb2e92bd1faa6ce4d12ff248798515a92ac903ac14d31b5172d9b2917 2000
The funding transaction was broadcasted this time!
"txid" : "b363caed4eb458d017a4f678b135b9ad186f240dc7a46cb209ada8433386a970"

After a confirmation, the channel moved from CHANNELD_AWAITING_LOCKIN to CHANNELD_NORMAL and after 6 confirmations "public" in "cli/lightning-cli listchannels" moved from false to true,

So thanks again. This is a big improvement, especially for mainnet :-)

[laanwj]

@cdecker Thanks a lot, I got word that it worked perfectly and he was able to successfully withdraw the funds.