Beaker: Issues sharing dats over the network

[specious]

Operation System: macOS Sierra 10.12.6 (16G29)
Beaker Version: git commit a44d4272cc3fa817c2b3b31d6ac69a74c19fbb10

I created a dat archive via the "New site" wizard that includes a dat.json and index.html and I can view it just fine.

dat://dbfd9a74d014399bdcd04ea6a9fce0c675661bd8800ae024f6a76a118a910349/dat.json

{
  "url": "dat://dbfd9a74d014399bdcd04ea6a9fce0c675661bd8800ae024f6a76a118a910349/",
  "title": "Ildar Sagdejev",
  "description": "Take a look at my software development artifacts.",
  "author": "Ildar Sagdejev <[email protected]> (https://keybase.io/tknomad)"
}

However, several different people have tried to load it across the internet, and none of them can reach it.

datbase.org can't find it either:

• https://datbase.org/view?query=dbfd9a74d014399bdcd04ea6a9fce0c675661bd8800ae024f6a76a118a910349

Meanwhile, I can quite successfully view other known dat archives published by other people.

I'm attaching the debug-20171230.log generated by

• starting Beaker
• navigating to beaker://library/
• inspecting beaker://library/dbfd9a74d014399bdcd04ea6a9fce0c675661bd8800ae024f6a76a118a910349
• loading the dat: dat://dbfd9a74d014399bdcd04ea6a9fce0c675661bd8800ae024f6a76a118a910349/
• loading https://datbase.org/view?query=dbfd9a74d014399bdcd04ea6a9fce0c675661bd8800ae024f6a76a118a910349 which reported that it could not find the dat
• exiting Beaker

The swam debugger reports no peers connected to this archive:

The swarm debugger log:

closing connection (550424ms) id=55 type=tcp host=35.193.163.202:43465
got handshake (144ms) id=56 type=tcp host=35.193.163.202:43465
closing connection (13228ms) id=56 type=tcp host=35.193.163.202:43465
got handshake (808ms) id=57 type=tcp host=35.193.163.202:43465

Searching the debug log for my dat key returns one line:

$ ag dbfd9a74d014399bdcd04ea6a9fce0c675661bd8800ae024f6a76a118a910349 debug-20171230.log
47:Sat, 30 Dec 2017 18:53:23 GMT dat Swarming archive, discovery key: 9b83f6487324413f737b89c11ddd2c01c7cc1a0c17e20de31e1a582b9cd6420b key=dbfd9a74d014399bdcd04ea6a9fce0c675661bd8800ae024f6a76a118a910349

Any advice on what I can do to troubleshoot this situation?

[pfrazee]

Dat has some known issues with peer discovery and connection right now. Maf is working on it, and I know Jim has been helping with some network emulators to help fuzz test. I'm going to close because it's a known issue.

Rehosting via hashbase tends to be a fairly consistent solution for now 😒

[specious]

Hold on, why not reopen if this is a known issue?

Someone might wonder why their dat isn't reachable and find this thread and possibly contribute to figuring it out.

[specious]

Let's say I'm interested in troubleshooting swarm connection and discovery situations.

In app/package.json I see that Beaker depends on mafintosh/discovery-swarm which uses maxogden/discovery-channel to "search for a key across multiple discovery networks and find peers who answer".

If I want to manually simulate the peer discovery that Beaker performs in the background, how can I do it?

[pfrazee]

You'd take the discovery module, import it into a test CLI app, then do one of the following:

1. Run it on various real world configurations (ie two laptops on a LAN, two laptops over WAN, etc) and look for failures.
2. Create a simulated network using containers and look for failures.

Option 2 is what @jimpick is working on now: https://github.com/jimpick/dat-network-simulator

You can also run tests on multiple Beaker instances. The challenge is getting proper debugging output -- the swarm debugger is obviously not helpful enough, so one option is to expand the amount of data it emits in its log. Alternatively, you can search through the internal code of the Dat modules and instrument it more. Most modules have already be instrumented using https://www.npmjs.com/package/debug, but you might need more than what's already there.

The task is to uncover where the failure is occurring, which means recognizing what the correct behavior is, and what's happening instead. It does appear, based on your log, that the failure is in discovery between peer devices. I assume that because the peers that are discovered are using TCP, and peers only ever connect to each other using UTP. The most reliable mechanisms for peer-discovery are in the https://www.npmjs.com/package/dns-discovery module, which uses a combination of multicast UDP (for LANs) and a centralized tracker that the Dat team runs using a modified form of DNS. That's what I'd debug, to see why connections are failing. Previously, I had started to look myself, but stopped when @mafintosh suggested he was going to be replacing the module entirely with a new DHT-based solution. That said, it's looking like the timeline for that is going to be too long, so debugging dns-discovery would still be worth it.

[D1plo1d]

I've tested datPeers on 3 networks in Toronto on Ubuntu 18.10 and while LAN datPeers connections have worked, datPeers haven't been able to connecting across networks for me anywhere I've tried it. Would it be safe to assume my datPeers issue(s) are caused by the same connectivity issue(s) as publishing a dat archive?

Also is there an upstream issue for this in dat, discovery-swarm, dns-discovery or another lib?

[pfrazee]

Would it be safe to assume my datPeers issue(s) are caused by the same connectivity issue(s) as publishing a dat archive?

Yes

Also is there an upstream issue for this in dat, discovery-swarm, dns-discovery or another lib?

The current plan is to address this with a new discovery stack, hyperswarm. I'm also going to be adding analytics to help us gather information on why failures are occurring.

[Fastidious]

The current plan is to address this with a new discovery stack, hyperswarm.

Where to follow the status on this?

[pfrazee]

You can follow the repos at https://github.com/hyperswarm

[RangerMauve]

This is the high level API that's going to be used: https://github.com/hyperswarm/network

[danrobi11]

Please take example on i2p browser: https://geti2p.net/en/browser
Implement a network status health report to help the users to know if they are firewalled or not.
Beakerbrowser user version 0.8.8
Thanks.

[D1plo1d]

@pfrazee does this mean the connectivity issues have been resolved? What version of Beaker should the fixes land in?
Thanks!

[pfrazee]

@D1plo1d Yep! Beaker 1.0 has the new hyperswarm stack and we've been getting really good results with connectivity. I haven't had a failed load yet across a number of user tests! So fingers crossed that we have a winner.

[D1plo1d]

That's awesome! Can't wait to try it out!

[pfrazee]

<3 !

[aishv123]

Why is beaker not able to load the fritter site