Gridcoin-research: Improved project fallback mechanisms

[denravonska]

Right now the NN participants use the following path when downloading statistic files:

• Download stats for day N from harvester site
• If file is not available, download current stats from project site
• If file is not available, exclude project from superblock

This can lead to missing projects in the block even if they were just out for a brief moment. In that case it might be better to use the stats from the day before:

• Download stats for day N from harvester site
• If file is not available, download current stats from project site
• If file is not available, download stats for day N-1 from harvester site
• If file is not available, exclude project from superblock

Here we can add a few days lookback if we want to, but we should probably keep it very short.

[3ullShark]

How about 7 days? Should be enough time for project to do maintenance

[denravonska]

Yeah, maybe. The idea is to have it long enough to balance small downtimes but short enough to not make project DDOS worth it.

[tomasbrod]

I don't like this. When implemented and a project goes offline, users will be rewarded for work, we don't know they are doing.

[denravonska]

@tomasbrod I think that's better than not rewarding at all and giving the mag to other projects. At least for a short time.

[startailcoon]

I would think allowing the usage of 1 previous days results are good enough. Longer than that and the stats are beginning to skew. Besides, if a project is down for longer it's having issues affecting users ability to do work.

[dopeshitnetworks-irc-dopeshit-net]

i really wish i knew more about how it all funtions and the programming side of things and have an idea to work the greylist and a SB checks and balance system together.
With it a SB missing projects would freeze rac/mag if a project is triggered on a check from the greylist system of the whitelisted projects. It says , was project in last SB = no ok is it in this sb ? if = no save , if = yes pay for both SB if sb 3 still no save so saves till that projects in the SB. but the greylist would put the project in a container not actually on the proposed greylist idea because its NN related and not project.
This is based off my first project jail system idea , now called greylist. My original idea was if a project didnt have work for 3 weeks your rac/mag froze for the project on the NN , i thought it seemed fair as its not a users fault if they have put all their resources in 1 project. When the project returned your stats continued.. This has evolved over the past 5-6 months of chatting about it and my bringing it up. So the other day Dutch had a post and it made me think , why not work that into the greylist as a checks and balance system where it checks the WL and if the project is in the SB and WL A+ , if not its statistics are frozen until the next SB. Now what if its not in 2-3 SB in a row? Not a problem , its been saved as the info has been halted and no payout received has been flagged maybe in a way we used it like the haves tor or even stole the harvester and worked it into the greylist to create project+cpid.tar.gz with data files in the container ( it would be a database ) and literally a checks and balances ( FDIC insurance but for Gridcoin ) that could be like SB ### ID for security and the block hash for 2nd layer of security for creation of the file and has the info from the projects that should be in the SB , then if its 1-2-3-4-5+ superblocks it would NEVER matter about loss of PoR. Maybe even some way to work it into the coin output for the day and actually being able to scrape the total and put it floating in the blockchain to decentralize it till the next correct full WL SB is created and solution and resolution magically has ended up in the users wallet for those missing days vs not getting PoR...
Per ddos , sorry but I come from that world too and mitigation these days , yes after like the ntpd amp and i have ssl based ddos tools and github is full of toys for skids , hell even all those cisco's have not been patched let alone heart bleed vuln boxes still exist on the web. Back before I retired , as a sysadmin for cwie/ccbill we resold prolexic and that was for some customers $80k a month for ddos mitigation. Now adays you can buy an openvz vps 2 core/512mb $4mo and pay $2mo for up to 700,000pps or 600gbps DDOS. I am sorry but not storm or krashed or if we want to talk about GNAA and everyone from from the underground irc networks and their ddos networks and all the hard chats of the ddos world I want to say than the project admin should be investing into their project we are investing in. They should use cloudflare or amazon or someones reverse proxy or setup their own for their project. Yes I know there are projects ran these peoples laptops , linux and boinc-server on cable modems @ home or DSL not everybody is seti@home on UC Berkelys's oc148s but our projects that we back under the whitelist as we have invested our community into them , they should put what nowadays costs only $10-20mo in order to mitigate ddos or have services hosted correctly and I am seeing projects move to say like Amazon Light Sail. I think if our investors and the small group of crunchers whom are going to only vote yes on projects with SSL we need to start asking about hardware and connection in the application and ddos mitigation? Hell ask their providers ASN , lets draw up a boinc project peerng map from tier providers and draw up a gridcoin project security list of projects that are ssl and projects that are non ssl and what is hosted on what and where... DDOSing boinc pojects is pointless , but since @tomasbrod at the last mumble session you and I were joking about when the eclipse happened freezing the blockchain and releasing it when the eclipse was over , joking but " i could ddos the gridcoin network and cause that to happen " we were just joking , I had no idea you would hack the 1m coin as we were talking about it I predicted the eclipse being 1m turn over.. Soooo , that makes me think about and consider in this whole conversation 1 thing that hasn't been happening and in my opinion needs to start as we grow. I am sorry but I am in the mood to try to explain positive ideas.
Full nodes , many are like boinc projects and ran @home on cable/dsl and it seems less in the wiki list are in data centers with ddos mitigation. We have talked about nodes being ddosed causing forks and have we been lucky? Do need requirements in order to be in the wiki list and move that lost and make a real page on the " real " gridcoin website if there is one. Require a vps/dedi/colo or ddos mitigated IP of some sort like we require ssl? DDOS mitigation actually gets used , so its actually important because if some actually calls himself a blackhat could just say F gridcoin and ddos our nodelist and well ,... What then , yes I am guilty too but 4 are in data centers with ddos mitigation and only 1 listed node is at home. Or a min down/up stream result from a testfile @ www.gridcoin.us/speedtest or other nodes as you can install the standard speedtest.net embeded test in your webserver. If we have no nodes then it doesn't matter if we have projects and a bunch of un-networked computer systems that are supposed to form a spider web from the seed nodes out. Or can we get rid of the lines around 113 in /src/net.cpp and have node.gridcoin.us ONLY hand out the other nodes to connect to and the official list is anon and done like an apache/mysql load balancing cluster thus making our node network fully anonymous . No wiki list needed , just a new flag in the code that would be like the current server=1 and listen=1 adding seed=1? These are literally just the nodes we have listed currently in the wiki I am refering to , full nodes not coin holding wallets unless that is how you roll.. Ok I got in type off IRC a bunch of ideas stuck in my head , sorry new meds yes this is blog/rant but its something I hope can get some wheels turning more on this.. I don't think 1-3 SB blocks worth of backup is harmful , if we are going to worry about ddos lets actually be real and worry about what WE control not the boinc admins , btw they should be able to put up a " we are currently under ddos " msg on the main boinc forum and thanks to #gridcoin and bots and startails news updats from multiple sources we would know. Please has purely been attempted to be on track and clear , if I had not taken so much care in trying to explain and be clear it would be much shorter..

[iFoggz]

@denravonska doesn't your project fall back work fix for this?

[denravonska]

@jamescowens The new scraper handles this, right?

[jamescowens]

Yes. The two constants
static int64_t SCRAPER_FILE_RETENTION_TIME = 48 * 3600;
static int64_t SCRAPER_CMANIFEST_RETENTION_TIME = 48 * 3600;

control the retention of statistics files on the scraper and manifests in memory. This is currently set to 48 hours. This is a balance between eliminating dropouts and not allowing too old statistics to be used for magnitudes.

We may want to adjust the 48 hours as we gain experience with the new scraper/NN, but I think 48 hours is a good start.

I think we can close this.

[jamescowens]

We will close this when the new scraper/NN is merged into development.

[jamescowens]

Closing.