Gridcoin-research: Port Neural Network to something OS agnostic

[denravonska]

(WiP)

To extend the amount of nodes capable of driving the neural network it should be ported from Visual Basic to something portable.

See ongoing NN documentation in the wiki.

Requirements

• NN should run on headless wallets
• Enabled by default, disabled via config/arguments
• Replaces VB implementation on Windows

Identify functionality

Identify which tasks the NN is responsible for and where they are located in the code.

Incorporating improvements

The wiki mentions several improvements which can be done to secure the network and reduce the traffic. We should consider implementing them while rewriting, even if it might render the networks incompatible.

[grctest]

Related:
https://github.com/Erkan-Yilmaz/Gridcoin-tasks/issues/101
https://github.com/Erkan-Yilmaz/Gridcoin-tasks/issues/94

[3ullShark]

I know this is not going to be very popular but we could potentially port the NN/DPOR Network very quick if we used the new .NET Core which is cross-platform.

The reason being we could re-use some of the existing code and a lot of the functionality takes longer to implement in C++. Things like downloading, decompressing and working with databases all seem like a lot of work to implement when using C++.

Bare in mind, since it's .NET Core and not old fashioned .NET it will not have a GUI and would just be executed as a process from the C++ code.

Here is an example:

main.cpp

#include <iostream>
#include <stdexcept>
#include <stdio.h>
#include <string>

std::string exec(const char* cmd) {
    char buffer[128];
    std::string result = "";
    FILE* pipe = popen(cmd, "r");
    if (!pipe) throw std::runtime_error("popen() failed!");
    try {
        while (!feof(pipe)) {
            if (fgets(buffer, 128, pipe) != NULL)
                result += buffer;
        }
    } catch (...) {
        pclose(pipe);
        throw;
    }
    pclose(pipe);
    return result;
}

int main()
{   
    std::string result = exec("dotnet managed.dll --datadirectory=\"C:\\Users\\Gridcoin\\AppData\\Roaming\\GridcoinResearch\" --cmd=syncdpor2 --cmddata=\"<XML></XML>\"");
    std::cout << result;
    return 0;
}

Program.cs

```c#
using System;
using System.IO;

namespace DPOR
{
class Program
{
static void Main(string[] args)
{
string dataDirectory = "";

        foreach(var arg in args)
        {
            var split = arg.Split(new char[] {'='}, StringSplitOptions.RemoveEmptyEntries);

            switch(split[0])
            {
                case "--datadirectory":
                    dataDirectory = split[1];
                    break;
            }
        }

        if (File.Exists(Path.Combine(dataDirectory, "gridcoinresearch.conf")))
        {
            Console.WriteLine(File.ReadAllText(Path.Combine(dataDirectory, "gridcoinresearch.conf")));
        }
    }
}

}

**output**

cpumining=true
email=xxx
addnode=node.gridcoin.us
UpdatingLeaderboard=false
boincdatadir=C:\ProgramData\BOINC\
```

If executing it as a process is not ideal it is possible to self host the .NET Core CLR inside the C++ application but it's a bit more involved setting it up.

According to the Stack Overflow 2017 Survey C# is in the top 4 most used languages therefore it might be easier to get more people to help out.

Just a thought :-)

[3ullShark]

I've started working on a .NET Core port as a proof of concept and if I can get it working it might make it easier to port the C# code to C++ later down the line. Here is the link to the repo and branch if anyone has suggestions or wants to help out.

https://github.com/3ullShark/Gridcoin-Research/tree/dotnetcore/contrib/Gridcoin-DPOR

[fooforever]

@3ullShark How far have you gotten with this? I think it's still a big issue for the wallet that we're relying on only a subset of nodes to do the vital NN work. Can you suggest anywhere I can help?

[jamescowens]

I thought it would be appropriate to make a note here that the core team is jointly working on a native C++ replacement for the NN. I have taken the lead, based on a C++ BOINC stats scraper prototype that @Foggyx420 wrote. This rewrite maintains protocol compatibility with the existing NN, but uses a radically different data-gathering and distribution design from the existing network. Please see https://docs.google.com/document/d/15UBHdRqDAXB_uzJH_3bF9ujSCexjmU6UbFUlPBrwYKQ/edit?usp=sharing for an initial write up, along with issue #1324.

Also see https://github.com/gridcoin-community/ScraperProxy for @Foggyx420's protoype of the scraper itself.

There is also a good discussion here...
https://github.com/gridcoin-community/Gridcoin-Research/wiki/Scraper-Brainstorming

This new approach solves many problems, including scalability limitations, cross-platform capability, removal of team requirement, reduced dropped projects, etc.

[jamescowens]

The project for this is being managed on my fork at https://github.com/jamescowens/Gridcoin-Research/projects/1.
The development branch is https://github.com/jamescowens/Gridcoin-Research/tree/integrated_scraper

[jamescowens]

I am closing this issue as the new scraper/NN has been merged into development. We still have open Gridcoin-community tasks associated with this issue.