Rippled: How to allow peer connections in stand-alone-like mode?

[codedot]

I am looking for configuration and/or command line arguments for rippled to run in a stand-alone-like mode that would accept at least one peer connection. This is needed to test alternative implementations, for example this work in progress (see #2413 and #2449 for details).

I have already tried the following options:

1. Public network peers respond with either 503 or 403, never 101 as expected. Anyway, this approach is not very productive as it is impossible to access debug output of the other side of the sockets.
2. With default configuration, rippled is unstable due to #2448 and always responds with 503.
3. Specifying --standalone option makes rippled drop all incoming connections.

Please advise.

[mDuo13]

The very definition of "stand-alone mode" is that rippled does not make connections to other instances, so this won't be "fixed" as written.

However, you _can_ reconfigure your ports to accept fixed IPs and limit the number of connections specifically. I believe how this works is:

• Remove r.ripple.com 51235 from the [ips] block
• Add the IPs you want to connect to in the [ips_fixed] block
• In the "peer protocol" port, set the limit to your desired number of connections
• Start with --start command

[codedot]

Add the IPs you want to connect to in the [ips_fixed] block

I do not want rippled to connect to any peers, only to accept incoming connections. If both [ips] and [ips_fixed] blocks are empty, rippled still connects to r.ripple.com 51235.

[codedot]

@mDuo13 I followed your advice. The rippled still responds with 503.

[mDuo13]

I see, you're referring to this code.

I don't think this will change. If the server can't connect to its peers, it's not able to engage in consensus. And if the server is in stand-alone mode, it shouldn't connect to its peers (and it doesn't use consensus, just ledger_accept).

It sounds to me like your problem is less about this behavior and more that your machine has insufficient specs to run rippled.

[codedot]

It sounds to me like your problem is less about this behavior and more that your machine has insufficient specs to run rippled.

As mentioned above, all public peers I tried to connect to respond with either 503 or 403. Do they all have insufficient specs? It sounds to me like rippled is never tested before released since it does not even have a mode in which testing is possible.

[mDuo13]

Oh, I misunderstood you. I thought you meant _your_ rippled was responding with 503.

Not sure why other servers would return 503/403, but if I had to guess it's because they don't want more peers at the time.

And you're absolutely, completely wrong about rippled not being tested before being released. Aside from the (incredibly thorough) unit tests, every nightly is tested on the Test Net for a minimum of a few days before being released.

[codedot]

And you're absolutely, completely wrong about rippled not being tested before being released. Aside from the (incredibly thorough) unit tests, every nightly is tested on the Test Net for a minimum of a few days before being released.

And how do you test that rippled can respond with something other than 503 and 403?

[nbougalis]

If you don't want rippled to connect to _any_ peers but wish to allow incoming connections ensure that you have port of type peer configured (that's where other servers will connect) and simply set in your config file:

[peer_private]
1

[codedot]

This does not resolve the issue, because setting

[peer_private]
1

in the configuration file only prevents broadcasting the server's address. The node still connects to either the configured peers (if any), or r.ripple.com 51235 (if none are configured).

[nbougalis]

That is not correct. Peers with peer_private set do not make outgoing connections except to peers specified in [ips_fixed]. See https://github.com/ripple/rippled/blob/3ce4dda5cbdc66df705d1ad31acd3651d974e2ea/src/ripple/app/main/Application.cpp#L1384

If they do, that is a bug. Please verify and, if necessary, open a new issue.

[codedot]

I just checked again. The server responds with 503. This does not resolve my issue.

Taking into account that this question:

And how do you test that rippled can respond with something other than 503 and 403?

remains unanswered for half a year, I assume no such test is ever done.

I request reopening issues #2448, #2449, and #2450. They should remain open until they are confirmed to have been resolved.

[nbougalis]

Can you provide me with more details on _how_ you are trying to connect? I just connected to several servers using RPC, WebSocket and the peer protocol and didn't have a 503 or 403.

It's meaningless to reopen this issue (#2450) because the question it poses is "how to allow peer connections in standalone mode" and standalone mode is, by definition, designed to make allow peer connections.

If you want to run a server that does not make outgoing connections, then the correct mechanism to do that is what I described above: set peer_private in the configuration file; with that configuration the server will allow incoming connections to its peer port.

Alternatively, if you want to create a network of peers that don't connect to the main network, you can do that too. On all the servers that you want to have join your network:

1. shut down the server
2. remove the peerfinder.sqlite file
3. edit the config file to remove all entries under the [ips] block. Add the IP and port of a server that you want to connect to which has an open peer port.

[MarkusTeufelberger]

curl -v -H "Upgrade: RTXP/1.2" -H "Connection: Upgrade" https://r.ripple.com:51235/ gives a 503
curl -v https://r.ripple.com:51235/ gives a 403

Maybe that's what @codedot means? I would guess that returning peer IPs and a 503 in the first case is actually a bug, but it might be there for compatibility reasons...

[nbougalis]

The 503 is expected when you connected to a server that can't service your request (which could be because it's busy, but could be because it doesn't understand what you're asking to do). If that happens, it tries to be helpful and returns a 503 along with a set of IPs to try to connect to.

The reason you're getting a 503 however, isn't because the server is busy. It's because you're not passing in all the required information to it. You're missing a Connect-As header, which causes the server to call makeRedirectResponse (https://github.com/ripple/rippled/blob/develop/src/ripple/overlay/impl/OverlayImpl.cpp#L219).

While the server can't establish a connection because you've sent a request that's missing fields that are required, it still tries to be helpful. It provides a list of peers, on the theory that _just maybe_, they are running a newer version that can understand the request you sent and _maybe_ help you connect. This is not a bug, but it's probably more reasonable to send you an error telling you you're not talking the right language before simply dropping the connection.

That is exactly what happens if you retry but add a -k (to prevent SSL cert check issues) and -H "Connect-As: Peer" to your first curl command by the way:

Request:

GET / HTTP/1.1
Host: r.ripple.com:51235
User-Agent: curl/7.59.0
Accept: */*
Upgrade: RTXP/1.2
Connection: Upgrade
Connect-As: Peer

Response

HTTP/1.1 400 Bad Request
Server: rippled-1.1.1
Remote-Address: 35.167.133.144
Connection: close
Content-Length: 29

Unable to parse HELLO message

Rest assured, rippled returns values other than 503 and 403. If it did not, no connections would be established. But I guess this does highlight the need for updating #2449 sooner rather than later.

[codedot]

Adding the Connect-As: Peer header does indeed help to get the expected upgrade response with code 101 out of r.ripple.com:51235 using this work in progress:

$ node upgrade.js r.ripple.com:51235
{ peer: 'r.ripple.com:51235',
  status: 101,
  headers: 
   { connection: 'Upgrade',
     upgrade: 'RTXP/1.2',
     'connect-as': 'Peer',
     server: 'rippled-1.1.1',
     crawl: 'public',
     'public-key': 'n9KJb7NMxGySRcjCqh69xEPMUhwJx22qntYYXsnUqYgjsJhNoW7g',
     'session-signature': 'MEUCIQDf3H+eb6iOPWtGc4DivGltzcWCrLspRAqemUmItZ683QIgTrGANgM/IEQxPCuXZq5p0CoXDkUXKPFwXvzImTZBeGE=',
     'network-time': '594560045',
     'closed-ledger': 'cVvCY0U+WCpbiE5HTXQYrs6jHaH06I+yq4LxiXyHPz8=',
     'previous-ledger': 'BmdN9ojMCSPSnVBNaxaLDUToEGNVV6OaNPewaDm+EsA=' },
  check: true }

However, the local rippled server running with this configuration still responds with 503:

$ node upgrade.js 127.0.0.1:51235
{ peer: '127.0.0.1:51235',
  status: 503,
  headers: 
   { server: 'rippled-1.1.1',
     'remote-address': '127.0.0.1',
     'content-type': 'application/json',
     connection: 'close',
     'transfer-encoding': 'chunked' },
  body: '{"peer-ips":[]}' }

What configuration is needed for the local rippled server to get the expected response 101? Until such configuration is found, this issue should be open.

[MarkusTeufelberger]

I don't think that you can connect on localhost:

https://github.com/ripple/rippled/blob/7b0367730cf11b05395f24b537247d7341ea106a/src/ripple/overlay/impl/TMHello.cpp#L412-L432

[codedot]

I don't think that is the case, because neither of the debug messages show up in the log after the server responds with 503 even with

[rpc_startup]
{ "command": "log_level", "severity": "info" }

[nbougalis]

_Since we're having a conversation, I'm going to reopen this issue. But again, note that "allowing peer connections in stand-alone mode" is a non-sequitur. The discussion is about the protocol mechanics of establishing a peer connection._

---

I'm betting the problem is that you have peer_private set. To be sure, please execute the RPC command print peerfinder against the server that's returning the error. Please paste the config and counts objects here.

More specifically, is that this is being triggered from https://github.com/ripple/rippled/blob/develop/src/ripple/peerfinder/impl/Logic.h#L396-L402

That calls into https://github.com/ripple/rippled/blob/develop/src/ripple/peerfinder/impl/Counts.h#L77-L78

Which uses the value m_in_max calculcated from https://github.com/ripple/rippled/blob/develop/src/ripple/peerfinder/impl/Counts.h#L126-L150. If peer_private is set, this will make m_in_max == 0.

With that said, I believe that the way we calculate inbound/outbound slots can be improved and should, probably, be something that the admin can explicitly configure; we can do it automatically as a fallback. I'll have to open an issue for that.

But even if there were nothing actionable on the rippled side, it may make sense to enhance the code to improve the error information returned, to allow us to quickly decide what the issue is.

[codedot]

I didn't open this issue for a chat. The description of the issue has always been the same:

I am looking for configuration and/or command line arguments for rippled to run in a stand-alone-like mode that would accept at least one peer connection.

Above, I have already attached the configuration with which the local rippled server is running and responds with 503. Please note that the configuration includes

[peer_private]
1

as recommended by @nbougalis previously:

set peer_private in the configuration file; with that configuration the server will allow incoming connections to its peer port.

Once I obtain a configuration for rippled so it doesn't connect to any peers but allows incoming peer connections and responds to peers with 101 as expected, I will close this issue.

[nbougalis]

I didn't open this issue for a chat.

I didn't suggest you were. I simply said that I was reopening this issue, which you requested, because we seem to be discussing something other than allowing connections in stand-alone mode.

I am looking for configuration and/or command line arguments for rippled to run in a stand-alone-like mode that would accept at least one peer connection.

peer_private will do that just fine, unless you want to allow _incoming_ connections. Earlier in this thread I incorrectly stated that a server with [peer_private] would allow incoming connections, but a quick review of the code suggests otherwise. Goes to show, you learn something new every day. :nerd_face:

I asked you to paste the output of the print peerfinder command against the server that's returning the error. Can you do this so we ca go down the path of verifying that this is the issue, and deciding how this should be fixed, if at all? Please paste the result of the config and counts objects returned from that command here.

Alternatively, you can simply disable peer_private (remove the entry from your config file) and just add the following:

[ips]
192.0.2.1 51235

This will prevent your server from using r.ripple.com, because you specified at least one entry. However, the specified IP is marked as reserved by RFC 5737 and nobody should be using it, which means that your server will be unable to connect to it.

However, if the 503 you receive was caused by peer_private resulting in the server to not allocate any inbound slots, this will address it and allow your code to establish a connection.

If this has solved your issue, please go ahead and close this issue at this time. For me the takeaways from this conversation are simple:

1. We need to prioritize #2449 to accurately describe the connection protocol and the various caveats that exist.
2. We need to improve the error messages reported by the server when a connection is rejected to allow for easier debugging.
3. We need knobs that allow administrators to exercise more control over the kind of peer connections their server establishes (e.g. incoming vs. outgoing) and not overload the peer_private directive.

[codedot]

unless you want to allow incoming connections.

This issue is specifically about incoming peer connections (one can only accept(2) a connection request, not make a request). I don't understand where ambiguity could be coming from in this case, assuming some basic knowledge of how sockets work.

I asked you to paste the output of the print peerfinder command against the server that's returning the error.

In the attached configuration file, RPC was disabled, so I could not do that.

Anyway, that rather dirty hack

[ips]
192.0.2.1 51235

finally resolves the issue:

$ node upgrade.js 127.0.0.1:51235
{ peer: '127.0.0.1:51235',
  status: 101,
  headers: 
   { connection: 'Upgrade',
     upgrade: 'RTXP/1.2',
     'connect-as': 'Peer',
     server: 'rippled-1.1.1',
     crawl: 'public',
     'public-key': 'n9KAULfJX8hirkv3aVHZhuk2qBvwe7ePUL7eZQ1xwAwuV4jdSTKo',
     'session-signature': 'MEUCIQCDYyZSOvH4xq9HtCIwnUbZI59cWpLB6ksAJ4JuWlJuuwIgZyFsl75ko9xxTfelf9n+0Vtpc2MV896j7Pp1XC9NWXE=',
     'network-time': '594596428',
     'closed-ledger': 'caR3cbSIRQqckJ/YMwFTsdI9hH1VIYyp/Kp26MADeP4=',
     'previous-ledger': 'q4aKbP7sd5wv+EXArwCmQiWZhq9AwBl2p/hCtpGJNsc=' },
  check: true }

so I am closing it now.

[nbougalis]

Glad to hear this worked, thanks. This conversation has highlighted the need for more flexible configuration of connectivity options for rippled. I plan on creating a new issue with some details, and will tag you on it if that's OK?

I don't understand where ambiguity could be coming from in this case, assuming some basic knowledge of how sockets work.

There was no ambiguity; your request was very clear. There simply was some confusion on my part.

[codedot]

OK.