Lisk-sdk: Mainnet stuck because of bad transaction

[simonmorgenthaler]

This seems to be the bad transaction: https://explorer.lisk.io/tx/14467342019343228805

Error in pgsql.log:
2017-01-08 15:24:11 UTC ERROR: value too long for type character varying(64)
2017-01-08 15:24:11 UTC STATEMENT: insert into "mem_accounts2u_delegates" ("accountId", "dependentId") values ('202243225L', '8a6d629685b18e17e5f534065bad4984a8aa6b499c5783c3e65f61779e6da06czz');

The string '8a6d629685b18e17e5f534065bad4984a8aa6b499c5783c3e65f61779e6da06czz' is 66 chars long

My nodes are all in the status "Syncing"

[Gr33nDrag0n69]

cc001 infos is right (from what i posted in lisk chat) will try to add other things here

[Gr33nDrag0n69]

The transaction is linked to the same delegate that got the chain stuck last summer
axlsaddek
https://explorer.lisknode.io/address/202243225L

[Gr33nDrag0n69]

The chain isnt forked and everyone is stuck at same height with 100% consensus

[Gr33nDrag0n69]

rebuild wont help and clearing memtable too because the tx isnt expired and get rebroadcasted

[Gr33nDrag0n69]

a quick fix could be to extend the size of problematic field in the db so it get written in the chain but i fear we could have a biger problem after

[simonmorgenthaler]

Question is: why and how is this string too long?

[gregorst3]

The problem starts from the lenght of the wallet... It's all related

[Gr33nDrag0n69]

also, i dont know if related but i got a lot of this one (using debug level)

[dbg] 2017-01-08 16:42:07 | Broadcast relays exhausted - {"type":2,"amount":0,"senderPublicKey":"ac395d9a9144768284f29b1aa3f6a0237b03be4b37c4cf526630097540d43ffc","timestamp":19777491,"asset":{"delegate":{"username":"virtum","publicKey":"ac395d9a9144768284f29b1aa3f6a0237b03be4b37c4cf526630097540d43ffc"}},"signature":"9358d80a413a74f127562171c79ad42ec5b2e691014dc4bb9d2e2cfd4074b6f80bf6dd680fc9f29644595a9360c762eac0338f423290a494fb091155ac275800","id":"12145996864416172549","fee":2500000000,"senderId":"12492293221048511653L","relays":2}

[Gr33nDrag0n69]

but i think this is normal behavior not related

[Gr33nDrag0n69]

the account behind transaction is weird too, so it might be a weird fuck related to that

[Gr33nDrag0n69]

i dont know cc001

[viper-tkd]

Did anybody notice the word bad in the too long string: The string 8a6d629685b18e17e5f534065bad4984a8aa6b499c5783c3e65f61779e6da06czz

I don't know if it's a coincidence or if it was made like that to break the network.

[Odsejen]

just looked at the transactions from & to the the account "axlsaddek", one connected acount is the Lisk address "24213L" is this really a common address?

[Gr33nDrag0n69]

viper does the lenght is 3 chars too long ? thats odd

[viper-tkd]

@Gr33nDrag0n69 Dunno. I didn't check the length of the field in the database. But I suspect it is.

[simonmorgenthaler]

@viper-tkd @Gr33nDrag0n69 the string is 66 chars long. 2 chars too long

[vrlcrypt]

The bad transaction was made by @AxlSaddek.

Months ago he reported this issue (about Address length <16):
https://github.com/LiskHQ/lisk-explorer/issues/9

[1towia]

There is some checks missing here I think :
https://github.com/LiskHQ/lisk/blob/master/logic/vote.js#L157

The code doesn't verify if the publicKeys in the transaction are valids.

[simonmorgenthaler]

it just expired:
[inf] 2017-01-08 17:09:16 | Expired transaction: 14467342019343228805 received at: Sun, 08 Jan 2017 14:08:56 GMT

[simonmorgenthaler]

after reload the nodes seems ok for a short time. I guess the transaction is still broadcasted
and make the nodes stuck again
The transaction is expired only on 2 of my 3 nodes

[simonmorgenthaler]

after reload:
OK, then these messages:
[ERR] 2017-01-08 17:27:33 | Failed to get height from peer: 213.136.81.40:8000
[ERR] 2017-01-08 17:27:33 | Failed to apply unconfirmed transaction: 8224227198097948535 - Failed to add vote, account has already voted for this delegate
[ERR] 2017-01-08 17:27:34 | Failed to apply unconfirmed transaction: 1219768254938248202 - Failed to add vote, account has already voted for this delegate
[ERR] 2017-01-08 17:27:34 | Failed to apply unconfirmed transaction: 12145996864416172549 - Account does not have enough LSK: 12492293221048511653L balance: 1
[ERR] 2017-01-08 17:27:34 | Failed to apply unconfirmed transaction: 989175740950637023 - Account does not have enough LSK: 17836753152065928804L balance: 493.2
[inf] 2017-01-08 17:27:44 | Starting sync

then stuck again

[Odsejen]

Here the tx "14467342019343228805" is expired (appr. at 17:09 ) on all 4 nodes,

[Gr33nDrag0n69]

just retry a rebuild after the tx expired, no go

[viper-tkd]

same here on all nodes

[Odsejen]

ok no go, just tried a rebuild with one of the previous nodes which showed the tx "x" is expired, now i have similar messages as @simonmorgenthaler mentioned (and as before stuck at xxx836), but no expiration messages anymore, but instead these "failed to apply" messages and broadhash consensus is/was in all cases 100%.

• Failed to apply unconfirmed transaction: 17649204730636712354 - Failed to remove vote, account has not voted for this delegate
• Failed to apply unconfirmed transaction: 8224227198097948535 - Failed to add vote, account has already voted for this delegate
• Failed to apply unconfirmed transaction: 1219768254938248202 - Failed to add vote, account has already voted for this delegate

[vrlcrypt]

Look:
https://github.com/LiskHQ/lisk/issues/165

[1towia]

@vrlc92 He didn't used the testnet, like last time. Not good @AxlSaddek.