Gridcoin-research: Use P2P for stats file distribution

[a123b]

So I heard PrimeGrid limited access to the stats files because Gridcoin wallets participating in stats gathering would all download the stats files at around the same time, effectively DDoS-ing the project servers.

It seems to me that letting the clients share the files with each other in a peer-to-peer fashion similar to BitTorrent would be an ideal solution for this problem.
This would not only greatly relieve the projects' stats servers and Gridcoin's "trusted stats mirrors", but also could also remove the need of trusted mirrors entirely in the long run.

Of course, the clients would have to check integrity of the files with the project's server. This would be possible once feature request BOINC/boinc#2259 gets implemented in BOINC. Until then, the Gridcoin stats mirrors (which are already used/trusted anyway, as I understand) could be used to calculate and provide the sha256 checksums. Once said feature request is implemented and greatly adopted, using the feature could be added as a requirement for whitelisted projects, so the trusted mirrors could be avoided entirely.

[jamescowens]

I agree and this is essentially what I proposed yesterday on Cryptocurrency and Discord.

[jamescowens]

We will not use an external file distribution network though...

[denravonska]

Bittorrent and IPFS are both viable options. We have to either integrate a third party protocol or roll our own.

[jamescowens]

I am not enamored with depending on any third party protocol to distribute information that is critical to determine payouts on the chain.

[a123b]

Doesn't creating a new protocol from the ground up sound very difficult to maintain and optimize? I feel like it would be a very hard and time-consuming task to build something that is as efficient and reliable as commonly used and well-tested libraries developed specifically for this purpose...

[denravonska]

@jamescowens They are just protocols so as long as they meet our requirements they should be fine. One requirement is strong hashes on each block which might rule out Bittorrent. Regardless, I would prefer an established protocol rather than building our own.

Edit: I agree if the effort to develop and maintain our own protocol is less than or equal to implementing support for a third party one. Reducing dependencies is always a good thing if it's free:ish.

[jamescowens]

I am ok with using the protocols :) What I am referring to is third party infrastructure...

[tomasbrod]

I'd like to remind you, that Gridcoin network (well bitcoin) is a peer to peer protocol and that this protocol supports sharing of strongly hashed objects. Unconfirmed transactions are shared this way.
However you cant just make a random node download and share the stat file. Even if it is hashed, how other nodes know what is the correct hash? With a contribution to BOINC that @denravonska started, we can have the correct hash available from the server. But that just changes the situation from few thousand nodes downloading a hundred mega file, into about million of nodes downloading 65-byte file from the poor project server. Due to overhead, the later is not much better.
For rest, I would like to point you to analysis I made a year and a half ago.

[jamescowens]

And you know what I am suggesting in the meantime for the scrapers... to have multiple scraper instances download the stats and then independently hash the files and put the hashes in a manifest. The nodes will then select a random scraper to pull down the files, hash them and then connect to an alternate scraper and pull the manifest from the alternate and cross compare the hashes. It is the best we have until we can hash at the source and distribute internally.

[grctest]

Wouldn't @tomasbrod's DWP proposal be a far simpler solution than a P2P file sharing mechanism which doesn't solve the stats proxy centralization issue?

[jamescowens]

Unfortunately not. DWP is an excellent way of solving the NN scalability problem by controlling (and essentially randomly selecting) the nodes to do stats pulldown. This still requires the stats sites to be open, because the nodes are randomly selected by DWP, and a locked down BOINC site requires credentials. These two things together would require that every node possess the credentials to download the sites (because you don't know a priori which node is going to be selected), which is not what the BOINC sites intended when they issued credentials for stats access. Hence there has to be an intermediate layer that possesses the credentials to download the stats. This brings us full circle back to the scrapers.

We can't get too religious about this CM. The BOINC sites themselves are a form of centralization. If we really were going to be idealistic about this, the nodes should be using GRC blockchain natively to record the statistics directly in the blockchain, but that is never going to happen. There is no way for nodes to independently, _directly_ calculate or verify the work units processed and credits earned by a given nodes' research, because the nodes do not possess the rules or machinery necessary to directly validate processed work units. Hence there will always be a degree of third party dependency for research rewards. There is little additional value in total decentralization of stats download by the nodes, when the statistics calculations themselves are centralized. Hence the correct pragmatic solution for this is several scrapers with hash calculations so the nodes can at least verify that the stats on the scraper selected have not been compromised.

I will point out this third party dependency is also at the root of not being able to construct a Proof-of-Research for staking purposes that does not have vulnerabilities. In general the proof-of-stake algorithm must be based on objects that are in 100% possession of the node and are independently verifiable on a rules basis for both the generating (staking) node and accepting node to serve as a basis for staking. UTXO value meets this definition. Magnitude does not, and never will.

[jamescowens]

BTW, it is also much easier to set up several scrapers and do hash cross-verification than it is to do either file sharing or DWP, and we have to do the scrapers anyway, so this is the logical next step, not file sharing.

[dopeshitnetworks-irc-dopeshit-net]

I'd like to remind you, that Gridcoin network (well bitcoin) is a peer to peer protocol and that this protocol supports sharing of strongly hashed objects. Unconfirmed transactions are shared this way.
However you cant just make a random node download and share the stat file. Even if it is hashed, how other nodes know what is the correct hash? With a contribution to BOINC that @denravonska started, we can have the correct hash available from the server. But that just changes the situation from few thousand nodes downloading a hundred mega file, into about million of nodes downloading 65-byte file from the poor project server. Due to overhead, the later is not much better.
For rest, I would like to point you to analysis I made a year and a half ago.

@tomasbrod's are you talking about a year and a half ago being July 2017 and the w00t doc on how to hack and cause Gridcoin's value to just tank and never recover for the analysis?

@a123b dafaq is " Gridcoin's "trusted stats mirrors" " that is as invalid and forced as " Gridcoin's " trusted community members. " that are forced on us and not trusted nor should they be introduced as trusted or trust-able. They are as trust-able as Donald Trump Jr.

I do not trust anything in this donkey show at this point in time. To much drama, censorship and back room deals still going on with supportively trusted community members own personal agenda's due to this being a financial institution and financial gains. Do you really think Gridcoin has anything to do with science? No its to do with $$$ and none of us would be here if it did not , lie to yourself if you wish.
Gridcoin has many centralizations look at the once again outdated seednodes in /src/net.cpp around lines 113-117. The ironic thing is there is nothing trust-able about this project due to paranoia and ignorance and our " trusted community members " stunting the growth of Gridcoin while the value of it tanked hardcore.
Why do you think Rob washed his hands of Gridcoin and cashed out all of his GRC and it was/is A LOT in the millions of GRC and went on to his Bible pay project. I am sure it being a mashing on bitcoin , peercoin , blackcoin and I always forget #4 and I am sure other snippets from coins that don't get mentioned.
Really even being 1gbps files should not matter but we still have to understand there are projects like Amicable Numbers who's running the project on a laptop within his home on a cable connection , collatz runs at home too and 80% of these projects run at someones home where they typically have a massive downstream in comparison to their upstream IE: I get 280mbps down and 20mbps up and most do not pay the extra $50 for unlimited over their ISP's cap of 1tb a month traffic. At a university ( back in the day on t1's ) with their OC3's up to 10gbps/10gbps asynchronous data rates and unlimited traffic. That is why University backing is key... So maybe we work with the lesser bandwidth projects and come up with a solution , primegrid makes me lol because they use rackspace.net and should not bitch unless they are metered. This is why donations to projects make a difference , they can afford unlimited up/down traffic a month. I never heard about stat mirrors but I do know we constantly have superblocks that dis-include projects that are whitelisted and this needs to be addressed and fixed before Gridcoin tanks the rest of the way due to the BS issues internally and ethically along with users not getting payment for work done because you guys refuse help from people anyone whom isnt in the inner circle and yes I will say it " grc8 clique " for example people like me and rhino have tried to contribute ( i started on day 3 ) and we get shut down because we are not part of " the clique " although he was at one time but you guys kicked him out and started censoring him too.. So American for real , its like Trump runs the show and facts are fake news...

Also "

[denravonska]

@grcjamezz Gridcoin didn't tank from the w00t document. Quite the opposite, in January we reached an ATH.

Stats downloading is an issue for projects. I can't remember how large the Primegrid stats are but when 1000+ (?) nodes download the files at least once simultaneously every day it's going to take its toll. We can't require the projects to oversize their server and bandwidth capacity just because we haven't come up with a better solution.

Every coin needs a seed node. I know you have the round robin DNS idea but that's just moving the centralized list from the visible code to the DNS holder, unless your idea has another solution for it. I do agree that we shouldn't be using dead nodes. We need to fix that.

Regarding Rob's funds, it's hard to say where they went. Maybe he moved them, maybe he sold them.

[tomasbrod]

@grcjamezz The "trusted stats mirrors" are not to be trusted from gridcoin perspective. It's that projects 2 projects already do not give public access to their stats, they give the access only to entities they trust. In order to continue whitelisting those projects, we need a mechanism to pull the stats.

[jamescowens]

Just wanted to do a quick update on this. Tomas and I are well on our way to getting the plumbing done to distribute the scraper files (hashed and signed by the way) using the existing wallet messaging system. The solution is actually very elegant. There are two streams of work on this active. Please see https://github.com/jamescowens/Gridcoin-Research/commits/integrated_scraper and https://github.com/tomasbrod/Gridcoin-Wallet/commits/chk. Also please refer to the wiki https://github.com/gridcoin-community/Gridcoin-Research/wiki/Scraper-Brainstorming

The new scraper works and is successfully pulling the stats and filtering/parsing them and storing the files correctly. We have been able to construct and send manifest across the network with the files.

[jamescowens]

This issue will be closed when the Scraper/NN code is accepted into the development branch. It is operational on #testnet already.

[jamescowens]

Closing this issue now that the new scraper/NN is merged into the development branch! :)