Neo: Send "notfound" message when inventory is not found.

[lock9]

(Originally from @decentralisedkev on #366)

Summary
When nodeA asks nodeB for a header, if nodeB does not have the header, he will not tell nodeA. nodeA will wait for nodeB until some set of time, which slows down nodeA.

Note: I know we have 'notfound' message but I can't affirm this is being used for that. If this is already implemented this way, we should close this issue.

Do you have any solution you want to propose?
Once nodeB cannot find the header, he should send back a notfound message with the hashes of the inv that he could not find, along with a separate message of the headers that he could find.

Where in the software does this update applies to?

• P2P (TCP)

@shargon could you confirm if this already exists?

[shargon]

I think that is not implemented yet

[ixje]

I think it should only send notfound if it cannot find _any_ of the requested hashes. This way a client that does flight tracking doesn't have to wait for exceeding their set time threshold on the request. A client can determine what is missing from the response it receives; e.g.

scenario 1

• -> request headers 1 - 100
• <- not found
• result; node doesn't have any of the requested headers

scenario 2

• -> request headers 1 - 100
• <- send headers 1 - 10
• result; client can diff their requested vs received headers and learn that node does not have the other 90 headers

The original proposal would increase traffic quite a bit. For scenario 2 it would

• send the full headers for 1 - 10
• send notfound + 90 * 32 bytes (uint256) for all headers not found.

The client already has all these hashes so it is only generating traffic because the client is too lazy to track the request itselves. In the worst case scenario it would request the maximum 2000 headers, the node only has one and you'll get 1999 * 32 = 63968 bytes (excluding the Message bytes) of unnecessary traffic.

[shargon]

Maybe we can relay the message if we don't have this information

[ixje]

Maybe we can relay the message if we don't have this information

How would the information get back to the requesting party? I think we should minimize the responsibilities of the remote node in the syncing process and keep them as basic as possible. It either gives the requested data or not (or optionally informs it can't because it's missing). After that it is up to the requesting party to find an alternative way to get the data it wants (e.g. by trying a different node).

I still secretly hope we can get a new command as suggested in #522 to skip all this header requesting and sending overload. From the P2P handshake we can tell the current height of the remote node and we can sync by just sending

• getblocks(start_height=0)
• getblocks(start_height=500)
• getblocks(start_height=1000)
• etc

to get the full blocks, without needing any hash or figuring out if the remote node actually has the hash. Once local_node.height == remote_node.height you can use the very lightweight ping/pong messages to keep in sync. The current inv broadcasting that happens in neo-cli can be replaced by ping/pong broadcasting.

With the above you'll not even have to ask the client if they have certain hashes or blocks, you know before even asking based on their height!