Gridcoin-research: Neural Network Can't Download Stats from World Community Grid

[3ullShark]

WCG was not in the NN this morning.

Since WCG moved to the IBM cloud it looks like they have changed the URL's to their stats from user.gz to user.xml.gz and team.gz to team.xml.gz.

It looks like the extensions are hardcoded into the VB.NET

The logs read:

May 23 2017 00:11, Syncing Team world community grid http://www.worldcommunitygrid.org/boinc/stats/team.gz
May 23 2017 00:11, Error while downloading master team gz file: The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Team world community grid http://www.worldcommunitygrid.org/boinc/stats/team.gz
May 23 2017 00:11, Error while downloading master team gz file: The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Team world community grid http://www.worldcommunitygrid.org/boinc/stats/team.gz
May 23 2017 00:11, Error while downloading master team gz file: The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Team world community grid http://www.worldcommunitygrid.org/boinc/stats/team.gz
May 23 2017 00:11, Error while downloading master team gz file: The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Team world community grid http://www.worldcommunitygrid.org/boinc/stats/team.gz
May 23 2017 00:11, Error while downloading master team gz file: The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Project world community grid http://www.worldcommunitygrid.org/boinc/stats/user.gz
May 23 2017 00:11, Error while downloading master project rac gz file : The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Project world community grid http://www.worldcommunitygrid.org/boinc/stats/user.gz
May 23 2017 00:11, Error while downloading master project rac gz file : The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Project world community grid http://www.worldcommunitygrid.org/boinc/stats/user.gz
May 23 2017 00:11, Error while downloading master project rac gz file : The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Project world community grid http://www.worldcommunitygrid.org/boinc/stats/user.gz
May 23 2017 00:11, Error while downloading master project rac gz file : The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Project world community grid http://www.worldcommunitygrid.org/boinc/stats/user.gz

[tomasbrod]

Immediately after the migration this urls returned 403. Then WCG fixed it and it was working for some time. Now they return 404.

[3ullShark]

I've tweeted asking for a 301 redirect but I doubt they will do it. Looks like we might need to change it in the VB.NET

[tomasbrod]

Please, check the situation with einstein@home too.

[tomasbrod]

The URIs should be configurable from project beacons. In case oter projects decide to change location.

[denravonska]

What's the status now? Does this solve the issue?

[iFoggz]

@denravonska @3ullShark had mentioned it had worked for him. I don't have windows to test it myself.

[3ullShark]

What are peoples thoughts on changing the way the URL's are handled by setting up an official NGINX proxy server. That way we can map sub-domains to the real project URL.

For example:

The client would connect to the proxy server at:
https://wcg.gridcoin.us/user.xml.gz

The proxy server would pull the result directly from the project server at:
https://download.worldcommunitygrid.org/boinc/stats/user.xml.gz

The benefit of this is we could change the project URL's without having to update the client. We could also cache the results on the proxy server to reduce the load on the project servers. It might need a beefy machine to run though as more users start to use Gridcoin.

[iFoggz]

proxy server kind of centralizes the access point and there is no guarentee gridcoin.us would have uptime of 100%. I agree something else should be done.
I thought of a few ideas for in the contract. I know adding an additional field is troublesome. Also I don't think you can update a contract only add/delete

We could maybe consider adding support for future to allow extra information in project URL as Porting to c++ is already considered.

Example.
ProjectUrl;statsfile
ex https://www.worldcommunitygrid.com/boinc/stats/;user.xml.gz
or
ProjectUrl;Type
ex https://www.worldcommunitgrid.com/boinc/stats/;1
ex https://www.worldcommunitgrid.com/boinc/stats/;2
where type 1 is user.gz
where type 2 is user.xml.gz

[tomasbrod]

Also I don't think you can update a contract only add/delete

You can update a contract by sending new with the same MT (type) and MK (key aka name).

We could maybe consider adding support for future to allow extra information in project URL as Porting to c++ is already considered.

Yes, definitely. The project beacons should more flexible, preferably XML formatted. You can add this info and other your ideas to the NN porting wiki @Foggyx420.

[3ullShark]

Yeah, there is the centralization issue, however we always have access to the code and could change the client at any time.

About the up-time. At least we are in control. In fact, if the project website went down during the day we would still have a copy of the stats if cached and therefore the up-time could be improved.

However if we are to change it in code. Then I think allowing the full URL's and two of them would be the best option providing you could update them.

Also, why do we need the user.xml.gz? Isn't all the required info in the team.xml.gz file? Or am I missing something?

[tomasbrod]

Your proposed level of centralization, @3ullShark, is equivalent to computing magnitudes on the gridcoin.us server and just serving them to wallets. Which is of course unacceptable.

[3ullShark]

I wouldn't go that far but yeah it's not ideal. If that's the case then how can we trust the client. It is downloaded from gridcoin.us

[moisespr123]

So, the thing is, can the one in charge of updating the Neural Network can just point to the correct WCG files here? https://download.worldcommunitygrid.org/boinc/stats/

[3ullShark]

Not the way the files are downloaded currently. The filenames are hard coded in code. It could be changed so that the full URL's are used in the download code by following @tomasbrod suggestion of improving the project contracts to use XML or JSON.

[moisespr123]

In that case, an update could be made to check both variants of the file.
If one fails, then checks for the other files to download it. But this
update needs to be done urgently, as it is like saying that WCG is not
whitelisted...

On May 26, 2017 12:01 PM, "3ullShark" [e-mail] wrote:

Not the way the files are downloaded currently. The filenames are hard
coded in code. It could be changed so that the full URL's are used in the
download code by following @tomasbrod https://github.com/tomasbrod
suggestion of improving the project contracts to use XML or JSON.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/gridcoin/Gridcoin-Research/issues/323#issuecomment-304321038,
or mute the thread
https://github.com/notifications/unsubscribe-auth/APkf6nX4PuPGb9_ixDVY3pwYkps5zrtyks5r9vdfgaJpZM4Nks_C
.

[3ullShark]

I've already created a pull request for a quick fix for WCG but I think it will be a while before it gets released.

Also, how many people are likely to update their clients? Wouldn't there need to be over 50% for consensus?

[moisespr123]

They'll update when they notice that their WCG isn't updating or showing :)

[moisespr123]

After checking the code, I conclude, the VB.NET is outdated. I believe it was for the original Gridcoin wallet?
It seems the feature we're looking for is in the rpcblockchain.cpp file, particularly the GetTeamURLs function.

According to what's said in the previous comments, it appears that someone needs to issue a new beacon to update the URLs for the project.

[3ullShark]

@moisesmcardona that's weird because after making this change https://github.com/gridcoin/Gridcoin-Research/commit/f3c7de557a912a74c5f0cb591d47febf51691d0f and re-compiling the boincstake project the NN started to download the files on my client.

[moisespr123]

Yes, but I believe that component was initially from Gridcoin Classic. The current Gridcoin Wallet is written entirely in C++ and to add or update a project in the Neural Network, it works sending a Beacon.

I was told in the Telegram group that either Rob or RTMoney_ can add or change the project whitelist (kinda centralized, right?).

[3ullShark]

I'm almost certain the C++ is calling .NET code. This happens when you open diagnostics or the neural network. It also happens when the C++ initiates a sync dpor2.

[moisespr123]

Could make sense now that you mention, since Linux doesn't have the Neural Network.

PS: Now that Microsoft has the .NET Core standard, I think it would be possible to port the current .NET code to Linux, but it'll only work in Command Line since it doesn't have GUI capacities.

[3ullShark]

Unfortunately I think it would be more work than just porting it to C++. It's written in VB.NET and WinForms. There's also dependencies that might not be available in .NET Core and some dependencies we do not have source code for.

I'm in the process of analysing the code and putting together a wiki page on how it works so that we can start porting to C++

[moisespr123]

Great.

In the short term, I think this compiles into a .dll file. Could it just be handed to users while we await an official wallet update?

[denravonska]

@3ullShark Great! Can you try to isolate parts which can be ported on their own? Databases, data processing, communication etc.

[3ullShark]

@moisesmcardona yes it compiles into a DLL called BoincStake. I've tested it and it works on my client. I could upload the DLL or you could follow the instructions I put together in this wiki page

@denravonska yeah I can try and do that.

[moisespr123]

Perfect. Thanks for the instructions. I'll follow it while we await for this release

@denravonska, any estimates when we can have an official release?

[3ullShark]

It might be better if we pull the NN commit directly into master and then just release it for Windows now.

I've tried contacting WCG asking for a 301 redirect but they are not responding.

[moisespr123]

sounds great

[tomasbrod]

Good work @3ullShark! I hope you have seen the other wiki page on Neural Net and I just want to point out that there are two now. Keep reading the NN code and explain to us how it works. Then we can start writing some portable code. I don't think it is too complicated. The NN just has to collect project stats and assign magnitudes.
I support to cherry-pick the commit into master as a quick fix.

[3ullShark]

Thanks @tomasbrod. I've seen your wiki. I will add the suggested improvements from my wiki to yours and just continue on with adding the .NET implementation details to my doc.

@denravonska what are your thoughts on cherry-picking the commit and building a quick Windows release?

[moisespr123]

Just wanted to say Thanks for the Quick Fix! The last superblock has WCG back in :D

[3ullShark]

@moisesmcardona that's weird. There hasn't been a fix released.

[moisespr123]

Looking at my stats at gridcoinstats.eu, it now show WorldCommunityGrid in the Neural Network. It wasn't there before, and my mag went up too.

[moisespr123]

https://gridcoinstats.eu/project/world%20community%20grid

[3ullShark]

I think gridcoinstats.eu pulls the stats directly from the project websites.

[3ullShark]

Have you checked the NN in your client? Before I patched mine it would not show WCG and I was getting errors in debug2.log saying it could not download the stats.

[moisespr123]

I'm checking, but I can say that my mag went up (310 now. Was at 275)

[moisespr123]

It will take some time for the NN to load. (Not Responding) message...

[moisespr123]

You're right. It isn't in the list. However, it says I have 215 mag but the wallet says 310... Gridcoinstats says 335... Who's right?

[3ullShark]

I would say the wallet is right. You will find out when you stake a block.

[denravonska]

@3ullShark We can pull the fix to master if we still need it.

[3ullShark]

@denravonska yeah I think we still need it. @moisesmcardona checked his NN and WCG was not in the list.

[tomasbrod]

The whole time since GRC stat files are renamed magnitude for wcg should been zero.
If it was non-zero then there is a serious problem with NN consensus.

[3ullShark]

@tomasbrod I think it was always 0 for me until I patched my client. I assume I will never be in consensus now since I have WCG data and no one else does.

You could be right though, since not many people have complained.

[3ullShark]

It looks like WCG have fixed this by storing user.gz and user.xml.gz on their server now.

[moisespr123]

Yup. It looks like they restored the old filenames. Still, this fix should go ahead just in case this happens in the future.

[grctest]

Right, so what's the deal then? Is there the two files? Is there the one file? WCG really dropping the ball here!

[tomasbrod]

It is us not being flexible enough. The file names should be specifiable in the project beacon.

[moisespr123]

They have both files for now. They are both the same. One with the .gz extension and the other as .xml.gz, but they are identical.

[3ullShark]

@tomasbrod it looks like the reason for the file names not being in the beacon is because there are two files to download.

1. The team.xml.gz
2. The user.xml.gz

The team.xml.gz is downloaded to find the Gridcoin team ID.

The team ID is then used to calculate magnitudes for users with that team ID.

I assume the reason for not specifying both URL's in the beacon was to save space. A possible alternative would have been to store the team ID in the beacon so that the team.xml.gz file does not need to be downloaded at all. Then the full path to the user.xml.gz file could be used.

e.g

<WHITELIST>Amicable_Numbers<COL>http://sech.me/boinc/Amicable/stats.user.xml.gz<COL>[TEAMID]</WHITELIST>`

// INSTEAD OF
<WHITELIST>Amicable_Numbers<COL>http://sech.me/boinc/Amicable/@</WHITELIST>

With the team requirement being removed the problem goes away providing the project beacons are updated with the full paths to user.xml.

[moisespr123]

Sounds like a great idea. Definitely only downloading the users.xml.gz file would help save precious bandwidth for both us and the BOINC project server. I believe to do this, there would need to be a mandatory release right? Making the old NN obsolete since it would have an additional field that the old NN would not understand.

[moisespr123]

So, is this fixed in the .9.0 release?

[denravonska]

@moisesmcardona Yup, for now :)