Rippled: ValidatorSite:WRN Problem retrieving from https://vl.ripple.com 104:Connection reset by peer

[RyanHubs]

See how to configure rippled full node to make the "validator_list_expires" field "2018-Aug-22 00:00:00.000 000" as never
We have observed that sometimes the block height will not be updated when the time exceeds validator_list_expires.
I wonder if this field is relevant.

stack=[RippledError(noNetwork, { error: 'noNetwork',||error_code: 17,||error_message: 'InsufficientNetworkMode'

[RyanHubs]

@nbougalis
@miguelportilla
Please help to see the problem. Thank you~~

[wilsonianb]

Published validator lists must have an expiration. Publishers are expected to create an updated list (with a later expiration) before the current list expires.
If you really don't want a list that can expire, you can instead manually configure your list of trusted validators (using [validators]). This is not recommended if running on the production network, as the UNLs of other servers using published lists are expected to change over time, and insufficient UNL overlap can result in your server forking from rest of the network.
https://github.com/ripple/rippled/blob/develop/cfg/validators-example.txt#L13

[RyanHubs]

@wilsonianb
At 2018-Aug-22 00:00:00.000000000, I observed that the server's block had not changed("complete_ledgers" : "40191164-40967932"), at which point after restarting the server, the block height changed, and the validator_list_expires time became new("validator_list_expires" : "2018-Aug-29 00:00:00.000000000").

please help, I want to know if there is a way that block height update is not affected by validator_list_expire?

[wilsonianb]

rippled will not advance its validated ledger / block height if its configured validator list is expired.
It sounds like your rippled was unable to fetch the updated list until you restarted.
Would you mind checking for any ValidatorSite or ValidatorList entries in your log file (/var/log/rippled/debug.log)?

[RyanHubs]

@wilsonianb
my rippled.cfg use validators_file like this:

[validators_file]
validators.txt

and validators.txt with the content default:

[validator_list_sites]
https://vl.ripple.com

[validator_list_keys]
ED2677ABFFD1B33AC6FBC3062B71F1E8397C1505E1C42C64D11AD1B28FF73F4734

[scottschurr]

I don't want to interrupt if there's already a useful discussion going on. However I have a couple of observations to make about the server_infos posted.

The pre-restart server_info shows a number of transitions between Full, Tracking, and Syncing. Sometimes this indicates hardware or configuration issues. Potential issues include insufficient memory or perhaps background processes that sporadically prevent rippled from keeping up with the network.

The post-restart server_info shows a total uptime of 187 seconds. This is not enough time to allow the server to sync to the network. My server regularly requires in excess of 5 minutes to sync to the network. If it's recreating the database it can take considerably more than this 5 minutes. Your personal experiences with sync time will be influenced by your hardware and your by network connection. But, at the very least, I suggest that you allow rippled to attempt to sync for at least 10 minutes before throwing in the towel.

I hope this helps.

[RyanHubs]

@wilsonianb
The following information can be filtered from the debug.log
What does this mean "Connection reset by peer"?

2018-Aug-16 15:41:01.827242111 ValidatorSite:WRN Problem retrieving from https://vl.ripple.com 104:Connection reset by peer
2018-Aug-22 00:13:19.706261787 ValidatorList:WRN New quorum of 18446744073709551615 exceeds the number of trusted validators (0)

[RyanHubs]

@scottschurr
If there is a problem with the hardware or configuration, why the block synchronization is normal before validator_list_expires time?
This makes me very puzzled.

[MarkusTeufelberger]

You only have 10 peers in all examples, are you maybe NATed?

[RyanHubs]

@MarkusTeufelberger
What's the meaning of "NATed" ? hahaha...

[MarkusTeufelberger]

https://en.wikipedia.org/wiki/Network_address_translation

[RyanHubs]

@MarkusTeufelber
Yes, the formal environment network is isolated, exposing port(51235) only.
Is this problem related to "ValidatorSite:WRN Problem retrieving from https://vl.ripple.com 104:Connection reset by peer"?

[RyanHubs]

Hi @wilsonianb @miguelportilla @nbougalis

Please help to see how to handle better.
Can you help me solve this problem? Thank you.
ValidatorSite:WRN Problem retrieving from https://vl.ripple.com 104:Connection reset by peer

[wilsonianb]

@LuoRyan what operating system are you on?

[RyanHubs]

@wilsonianb
Our operating system is CentOS7.

$sudo cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[wilsonianb]

Can you try changing https://vl.ripple.com to http://vl.ripple.com in your validators.txt?
The validator list itself is signed, so no TLS won't be an issue.

[RyanHubs]

@wilsonianb
According to your solution, it should be fine. I'll watch it for a while.
Thank you~~

2018-Sep-03 02:38:16.951707241 ValidatorSite:DBG Loading configured validator list sites
2018-Sep-03 02:38:16.951750748 ValidatorSite:DBG Loaded 1 sites
2018-Sep-03 02:38:17.374695296 ValidatorSite:DBG Applied new validator list from http://vl.ripple.com

Will this error affect data acquisition?
2018-Sep-04 22:28:20.061270651 ValidatorSite:WRN Problem retrieving from http://vl.ripple.com 110:Connection timed out

[RyanHubs]

@wilsonianb
Using this url(http://vl.ripple.com), it looks like there are still problems

2018-Sep-10 18:18:20.269277835 ValidatorSite:WRN Problem retrieving from http://vl.ripple.com 110:Connection timed out
2018-Sep-10 18:53:30.539995077 ValidatorSite:WRN Problem retrieving from http://vl.ripple.com 104:Connection reset by peer

[MarkusTeufelberger]

Maybe rippled doesn't follow redirects? Make sure to change it to https.

Edit: OK, I checked, there's no redirect (only in the browser, probably HSTS) and the file is hosted on http too.

[RyanHubs]

@MarkusTeufelberger @wilsonianb
Can this problem be solved? ☹

[MarkusTeufelberger]

I suspect that the problem is somewhere on your end (firewalls, proxies...?) since it is not a problem in general.

[RyanHubs]

@MarkusTeufelberger

After executing the command, I can get the correct response result.☹

$curl -X GET \

https://vl.ripple.com/ \
-H 'Cache-Control: no-cache' \
-H 'Postman-Token: 9186bb08-93bb-4041-ad61-1d038ed35bd7'
{
"public_key": "ED2677ABFFD1B33AC6FBC3062B71F1E8397C1505E1C42C64D11AD1B28FF73F4734",
"manifest": "JAAAAAFxIe0md6v/0bM6xvvDBitx8eg5fBUF4cQsZNEa0bKP9z9HNHMh7V0AnEi5D4odY9X2sx+cY8B3OHNjJvMhARRPtTHmWnAhdkDFcg53dAQS1WDMQDLIs2wwwHpScrUnjp1iZwwTXVXXsaRxLztycioto3JgImGdukXubbrjeqCNU02f7Y/+6w0BcBJA3M0EOU+39hmB8vwfgernXZIDQ1+o0dnuXjX73oDLgsacwXzLBVOdBpSAsJwYD+nW8YaSacOHEsWaPlof05EsAg==",
"blob" : "eyJzZXF1ZW5jZSI6MzIsImV4cGljQyNzk1Q0...

[RyanHubs]

@MarkusTeufelberger
Is it possible that there is a problem with the configuration file(/opt/ripple/etc/rippled.cfg) for our live machine?
Can you help us look at our configuration(rippled.cfg)?

[server]
port_rpc_admin_local
port_peer
port_ws_admin_local

[port_rpc_admin_local]
port = 5005
ip = 0.0.0.0
admin = 127.0.0.1
protocol = http

[port_peer]
port = 51235
ip = 0.0.0.0
protocol = peer

[port_ws_admin_local]
port = 6006
ip = 0.0.0.0
admin = 127.0.0.1
protocol = ws

[ips]
r.ripple.com 51235

[ips_fixed]

s2.ripple.com 51234

[ledger_history]
100000

[node_size]
tiny

[node_db]
type=RocksDB
path=/data/home/data/rocksdb/
open_files=2000
filter_bits=12
cache_mb=256
file_size_mb=8
file_size_mult=2
online_delete=500000
advisory_delete=0

earliest_seq=39165153

[shard_db]

type=NuDB

path=/data/home/data/shard/

max_size_gb=800

[database_path]
/data/home/data/wallet/

[debug_logfile]
/data/home/logs/debug.log

[sntp_servers]
time.windows.com
time.apple.com
time.nist.gov
pool.ntp.org

[validators_file]
validators.txt

[rpc_startup]
{ "command": "log_level", "severity": "warning" }

[ssl_verify]
1

[wilsonianb]

@LuoRyan Can you confirm that this is an intermittent issue?

Run:

/opt/ripple/bin/rippled log_level ValidatorSite debug

Wait a while, and then check that you have Validator list with current sequence from ... being logged every five minutes (except for any times Problem retrieving from ... happens).

If this is the case, you should be ok as long as you get at least one successful retrieval during the window of time after a new list has been published and before the previous list expires (usually at least several hours).

[RyanHubs]

@wilsonianb
Yes, we can see the log of "Validator list with current sequence from https://vl.ripple.com".

2018-Sep-16 23:56:41.189041853 LoadMonitor:WRN Job: processLedgerData run: 1476ms wait: 0ms
2018-Sep-17 00:07:04.732697416 ValidatorSite:WRN Problem retrieving from https://vl.ripple.com 104:Connection reset by peer
2018-Sep-17 01:19:12.135825016 LoadMonitor:WRN Job: processLedgerData run: 2207ms wait: 0ms
2018-Sep-17 01:28:00.336308695 ValidatorSite:WRN Problem retrieving from https://vl.ripple.com 104:Connection reset by peer
2018-Sep-17 01:38:01.552484383 ValidatorSite:DBG Validator list with current sequence from https://vl.ripple.com
2018-Sep-17 01:59:16.183673935 Flow:WRN DirectStepI: can't receive IOUs from issuer without auth. src: rBtiyfuSCQrcAKevnvfNUJVmaGx856ZVkk
2018-Sep-17 02:03:02.014298944 ValidatorSite:DBG Validator list with current sequence from https://vl.ripple.com
2018-Sep-17 02:03:15.580179273 Flow:WRN DirectStepI: can't receive IOUs from issuer without auth. src: rBtiyfuSCQrcAKevnvfNUJVmaGx856ZVkk
2018-Sep-17 02:16:01.862713764 ValidatorSite:WRN Problem retrieving from https://vl.ripple.com 104:Connection reset by peer
2018-Sep-17 02:16:03.309101685 ValidatorSite:DBG Validator list with current sequence from https://vl.ripple.com

[nbougalis]

We've made several changes to this code with rippled 1.1.1 which I believe should address this issue. I'm closing it, but if you continue having problems, please feel free to reopen.

[hippo-dalaoshe]

We've made several changes to this code with rippled 1.1.1 which I believe should address this issue. I'm closing it, but if you continue having problems, please feel free to reopen.

excuse me, I meet the same problem New quorum of 18446744073709551615 exceeds the number of trusted validators

the detail info in debug.log is as follow:

2019-Jan-14 23:37:36.676261277 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 16698ms
2019-Jan-14 23:37:36.676282620 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 19580ms
2019-Jan-14 23:37:36.676443416 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 13696ms
2019-Jan-14 23:37:36.676523252 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 7694ms
2019-Jan-14 23:37:36.676569343 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 10696ms
2019-Jan-14 23:37:36.676671027 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 1693ms
2019-Jan-14 23:37:36.676684432 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 4692ms
2019-Jan-14 23:45:31.170064459 Peer:WRN [325056] onReadMessage: Connection reset by peer
2019-Jan-14 23:47:25.541382117 LoadMonitor:WRN Job: processLedgerData run: 10764ms wait: 0ms
2019-Jan-14 23:47:25.541496212 LoadMonitor:WRN Job: InboundLedger run: 0ms wait: 8367ms
2019-Jan-14 23:47:25.541525938 LoadMonitor:WRN Job: processLedgerData run: 10360ms wait: 0ms
2019-Jan-14 23:47:25.573421814 LoadMonitor:WRN Job: InboundLedger run: 32ms wait: 7866ms
2019-Jan-14 23:47:25.573432698 LoadMonitor:WRN Job: InboundLedger run: 32ms wait: 4829ms
2019-Jan-14 23:51:13.438445475 LoadMonitor:WRN Job: processLedgerData run: 1389ms wait: 0ms
2019-Jan-14 23:51:13.438548683 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 1229ms
2019-Jan-14 23:51:13.438597187 LoadMonitor:WRN Job: processLedgerData run: 1448ms wait: 0ms
2019-Jan-14 23:51:27.901913617 Peer:WRN [325098] onReadMessage: Connection reset by peer
2019-Jan-15 00:00:00.235820248 ValidatorList:WRN New quorum of 18446744073709551615 exceeds the number of trusted validators (0)
2019-Jan-15 00:00:22.243129908 NetworkOPs:WRN We are not running on the consensus ledger
2019-Jan-15 00:00:22.243207871 ValidatorList:WRN New quorum of 18446744073709551615 exceeds the number of trusted validators (0)
2019-Jan-15 00:00:22.243263837 LedgerConsensus:WRN Need consensus ledger 4059DF5BA121A44C3F19AD366E9B77A375385D8AA0892CE255E8AACDCFAAB123
2019-Jan-15 00:00:23.247307864 LedgerConsensus:WRN View of consensus changed during open status=open,  mode=wrongLedger
2019-Jan-15 00:00:23.247352930 LedgerConsensus:WRN 4059DF5BA121A44C3F19AD366E9B77A375385D8AA0892CE255E8AACDCFAAB123 to DDFBFD8C6E372C52A1787557E5236EE0444A25255D16CF900B5F4EC4B1D890FC
2019-Jan-15 00:00:23.247404692 LedgerConsensus:WRN {"accepted":true,"account_hash":"B13D1AD764F1DD3253127A3DA02C6141B9093A2A7986B8E9940CE9143C995767","close_flags":0,"close_time":600825620,"close_time_human":"2019-Jan-15 00:00:20.000000000","close_time_resolution":10,"closed":true,"hash":"DDFBFD8C6E372C52A1787557E5236EE0444A25255D16CF900B5F4EC4B1D890FC","ledger_hash":"DDFBFD8C6E372C52A1787557E5236EE0444A25255D16CF900B5F4EC4B1D890FC","ledger_index":"16090546","parent_close_time":600825601,"parent_hash":"E7FA10FF2E422A41A5B4437422029B9265C6B60491A3C6043F182CCB32D50160","seqNum":"16090546","totalCoins":"99997075910029940","total_coins":"99997075910029940","transaction_hash":"0000000000000000000000000000000000000000000000000000000000000000"}

you can see that, before 2019-Jan-14 23:51:13 everything is ok, and my node can receive and persist the closed ledger normally.
but at 2019-Jan-14 23:51:27 the Peer:WRN [325098] onReadMessage: Connection reset by peer message happen, and then at 2019-Jan-15 00:00:00 it tell me New quorum of 18446744073709551615 exceeds the number of trusted validators, and at 2019-Jan-15 00:00:23, my node is at View of consensus changed during open status=open, mode=wrongLedger, after that my node can only receive closed ledger but cannot sync it.

my node is run at ubuntu18.04 in testnet with rippled 1.1.1
I have restart the node, and lose all the ledger I have sync, can you tell me how to avoid this problem?

[livingrock7]

same problem
tried http as well
can you please check the logs and advise??

2020-Mar-23 09:23:30.515434163 OrderBookDB:DBG Advancing from 0 to 3 2020-Mar-23 09:23:30.515511800 OrderBookDB:DBG OrderBookDB::update> 2020-Mar-23 09:23:30.515599844 OrderBookDB:DBG OrderBookDB::update< 0 books found 2020-Mar-23 09:23:30.532429909 ValidatorList:DBG Loading configured trusted validator list publisher keys 2020-Mar-23 09:23:30.532472156 ValidatorList:DBG Loaded 1 keys 2020-Mar-23 09:23:30.532480219 ValidatorList:DBG Loading configured validator keys 2020-Mar-23 09:23:30.532485205 ValidatorList:DBG Loaded 0 entries 2020-Mar-23 09:23:30.532490659 ValidatorSite:DBG Loading configured validator list sites 2020-Mar-23 09:23:30.532578268 ValidatorSite:DBG Loaded 1 sites 2020-Mar-23 09:23:30.572387108 NetworkOPs:NFO Consensus time for #3 with LCL 3458C8DD476FC28BA254854D3665542635E7E3A13426BA7CBBA8A487071C234F 2020-Mar-23 09:23:30.572426562 ValidatorList:DBG 0 of 0 listed validators eligible for inclusion in the trusted set 2020-Mar-23 09:23:30.572435970 ValidatorList:DBG Using quorum of 18446744073709551615 for new set of 0 trusted validators (0 added, 0 removed) 2020-Mar-23 09:23:30.572440632 ValidatorList:WRN New quorum of 18446744073709551615 exceeds the number of trusted validators (0) 2020-Mar-23 09:23:30.572449528 LedgerConsensus:NFO Entering consensus process, watching, synced=no 2020-Mar-23 09:23:30.572464365 LedgerConsensus:NFO Consensus mode change before=observing, after=observing 2020-Mar-23 09:23:30.572454806 ValidatorSite:DBG Starting request for http://vl.ripple.com 2020-Mar-23 09:23:30.572490013 NetworkOPs:DBG Initiating consensus engine 2020-Mar-23 09:23:30.630246286 Server:NFO Opened 'port_rpc_admin_local' (ip=127.0.0.1:5005, admin IPs:127.0.0.1, http) 2020-Mar-23 09:23:30.630332786 Server:NFO Opened 'port_peer' (ip=0.0.0.0:51235, peer) 2020-Mar-23 09:23:30.630439681 Server:NFO Opened 'port_ws_admin_local' (ip=127.0.0.1:6006, admin IPs:127.0.0.1, ws)