Rippled: Rippled Treshold limit

[Edoin]

Consumer entry 127.0.0.1 dropped with balance 93099 at or above drop threshold 1500

This error got to me for some time now, causes me to stop the nginx for about 5 minutes and start it again. Recently this is showing about every 5 - 10 mins after I restart my private rippled.

[mtrippled]

There's a configuration option for your websocket and http rpc ports in rippled.cfg that sets listed client IP's as administrative, and bypasses load throttling:

Example:
[port_rpc]
port = 51234
ip = 0.0.0.0
admin = 127.0.0.1
protocol = https, http

[Edoin]

Hello Mark,

Thanks for answering my question, but I have this as my port_rpc config

[port_rpc]
port = 7000
ip = 127.0.0.1
admin = 127.0.0.1
protocol = http

and also I have been looking for the variable to bypass the load. Is send_queue_limit the variable to be filled to bypass load throttling?

[mtrippled]

Actually, given your config with the admin=127.0.0.1, this shouldn't be happening at all. Can you please look for the following in the debug_logfile:
grep 'Resource:DBG New .*endpoint "127.0.0.1"'

Also please include the log entries showing disconnection like what you originally posted:
"Consumer entry 127.0.0.1 dropped with balance"

[Edoin]

I was not able to find any logs concerning 'Resource:DBG New .*endpoint "127.0.0.1"' on our .

This is the log entry like the one I posted.

2017-Sep-12 07:26:40 Resource:WRN Consumer entry 127.0.0.1 dropped with balance 1504 at or above drop threshold 1500
2017-Sep-12 07:26:40 Resource:WRN Consumer entry 127.0.0.1 dropped with balance 1514 at or above drop threshold 1500
2017-Sep-12 07:26:40 Resource:WRN Consumer entry 127.0.0.1 dropped with balance 1523 at or above drop threshold 1500


2017-Sep-12 07:28:30 Resource:WRN Consumer entry 127.0.0.1 dropped with balance 194269 at or above drop threshold 1500
2017-Sep-12 07:28:30 Resource:WRN Consumer entry 127.0.0.1 dropped with balance 194278 at or above drop threshold 1500
2017-Sep-12 07:28:30 Resource:WRN Consumer entry 127.0.0.1 dropped with balance 194287 at or above drop threshold 1500

[mtrippled]

OK, your logging verbosity is not high enough. The default logs to debug and higher. In rippled.cfg, comment out the lines that look something like:
[rpc_startup]
{ "command" : "log_level", "severity" : "warning" }

Then restart rippled. Then you'll be able to see those entries.

[Edoin]

But in restarting our private rippled would cause the lost of current ledgers right? is there a safe way to restart the rippled without losing the current ledgers?

I have read about the --load but is it really working?

[inmyth]

Is there a solution to this issue ?

[kyofight]

I encounter the same problem, is there a fix for this?

[nbougalis]

--load has nothing to do with this issue, @edoin. When you restart rippled, it will reacquire the latest current ledger and then backfill, as needed.

@inmyth, @kyofight: what exactly are you doing to encounter this? It should only occur on servers that are receiving way too many queries, each imposing a large load.

[kyofight]

@nbougalis thanks for your reply.
i am running a private ripple network of 6 ripple instances. The network had run for few days. And all of the instances were down due to electricity problem. I tried to start all servers and i found out that all of the ledgers are gone, even if i use the --load to start the instances, the ledgers still cannot be recovered. Later i do a PoC, i found out as long as there is a full history instance standing (not dead), i am able to start other instances to backfill the ledgers. But for the case, all instances are dead, i can do nothing to get ledgers back.

[inmyth]

@nbougalis
@Edoin (OP)'s problem is not the same as @kyofight problem.
In case of OP, he encounters spikes in incoming requests that turn his rippled instances unresponsive (hangs / no longer returns any response). He has to clear all websocket connections, wait for a few minutes before the instance can start serving requests again. In his case, no ledgers are lost.
@kyofight shut down his instances and had to do full restart, both the server and rippled, which usually causes a "hole" in the ledgers. And it looks in his case, he started from ledger 1 again, maybe due to mis-config. In my experience ledger structure or any data stored in RocksDb or NuDb may not be fully recovered or rebuilt. In fact depending on config, past ledgers will eventually disappear. However txn data stored in Sqlite are safe so you can try retrieving your data from there.

[gituser]

I've just stumbled across this problem as well.

To reproduce:

You need to send many queries from admin ip to the rippled instance in few seconds (e.g. 100 or more) and rippled will just return NULL instead of the actual rpc response. Actually doesn't matter which query you do via RPC.

Here is a snip from the logs:

2018-May-18 13:08:46.587088482 Resource:WRN Consumer entry x.x.x.x dropped with balance 15000 at or above drop threshold 15000
2018-May-18 13:08:47.080254060 Resource:WRN Consumer entry x.x.x.x dropped with balance 15094 at or above drop threshold 15000
2018-May-18 13:08:47.080874815 Resource:WRN Consumer entry x.x.x.x dropped with balance 15188 at or above drop threshold 15000
2018-May-18 13:08:47.081445930 Resource:WRN Consumer entry x.x.x.x dropped with balance 15281 at or above drop threshold 15000
2018-May-18 13:08:47.081999736 Resource:WRN Consumer entry x.x.x.x dropped with balance 15375 at or above drop threshold 15000
2018-May-18 13:08:47.082566998 Resource:WRN Consumer entry x.x.x.x dropped with balance 15469 at or above drop threshold 15000
2018-May-18 13:08:47.083078002 Resource:WRN Consumer entry x.x.x.x dropped with balance 15563 at or above drop threshold 15000
2018-May-18 13:08:47.083577652 Resource:WRN Consumer entry x.x.x.x dropped with balance 15656 at or above drop threshold 15000
2018-May-18 13:08:47.084054002 Resource:WRN Consumer entry x.x.x.x dropped with balance 15750 at or above drop threshold 15000
2018-May-18 13:08:47.084638382 Resource:WRN Consumer entry x.x.x.x dropped with balance 15844 at or above drop threshold 15000
2018-May-18 13:08:47.085271442 Resource:WRN Consumer entry x.x.x.x dropped with balance 15938 at or above drop threshold 15000
2018-May-18 13:08:47.085712221 Resource:WRN Consumer entry x.x.x.x dropped with balance 16031 at or above drop threshold 15000
2018-May-18 13:08:47.086230294 Resource:WRN Consumer entry x.x.x.x dropped with balance 16125 at or above drop threshold 15000
2018-May-18 13:08:47.086815052 Resource:WRN Consumer entry x.x.x.x dropped with balance 16219 at or above drop threshold 15000
2018-May-18 13:08:47.087320031 Resource:WRN Consumer entry x.x.x.x dropped with balance 16313 at or above drop threshold 15000
2018-May-18 13:08:47.087914573 Resource:WRN Consumer entry x.x.x.x dropped with balance 16406 at or above drop threshold 15000
2018-May-18 13:08:47.088389661 Resource:WRN Consumer entry x.x.x.x dropped with balance 16500 at or above drop threshold 15000
2018-May-18 13:08:47.088869624 Resource:WRN Consumer entry x.x.x.x dropped with balance 16594 at or above drop threshold 15000
2018-May-18 13:08:47.089345307 Resource:WRN Consumer entry x.x.x.x dropped with balance 16688 at or above drop threshold 15000
2018-May-18 13:08:47.089782426 Resource:WRN Consumer entry x.x.x.x dropped with balance 16781 at or above drop threshold 15000

Of course x.x.x.x is listed as admin in the configuration but it doesn't help.

I've been testing this with rippled v1.0.0 on testnet, but it's also true for mainnet.

[movitto]

Greetings! I'm running into a similar issue when attempting to sync my client against a ripple instance running in a production datacenter. I recently changed a few things in my setup, including updating rippled to 1.0, updating my client logic (in development), and tweaking the rippled.cfg settings (particularily setting send_queue_limit to its max value of 65535) but have had no luck, every so often, I see these "Consumer Entry x.x.x.x dropped" messages in the log and my connections are dropped.

Before I go into the details of my workflow, please note my (single) rippled instance is deployed on hardware sufficiently satisfying the "development" requirements, specifically:

• the instance is deployed on a Linode 16GB VM with
• 16 GB of memory
• 6 Cores
• a 320 GB SSD
• 40 Gbps network in and
• 6000 Mbps network out

My rippled config is mostly the stock setup with websockets enabled, ssl ceritifcates enabled, the node size set to medium and the send_queue_limit increased as mentioned above.

My client workflow consists of:

• synchronizing accounts sequentially via the datav2 api
• querying my rippled instance for account balances, trust lines, offers, and info; pulling in all data, even on subsequent pages if 'marker' is set.
• also independently following the ledger, and updating these entities in the local db when transactions modify them

I can specify the number of workers to allocate to client operations, and am trying to maximize the throughput which I receive data. If I set the client to use 3 workers for the account 'details' synchronization, everything operates smoothly with no disconnections. But if I bump this to 5 I start seeing the aforementioned dropped connections and 'Consumer entry x.x.x.x dropped' log messages.

---

So my question is, is it possible to achieve a higher throughput / synchronization rate using the hardware and deployment detailed above? With 3 details workers, I synchronize accounts at a rate of ~3-5 per second, which projected over the ~1.2M accounts will take me ~3days to complete. When I increase the number of workers, I see initial and subsequent rates of ~10-20 accounts per second (for 5 and 10 workers respectively) which would reduce the sync time to < 1/2 a day, if not for the periodic disconnections caused by the load balancer (about once a minute all clients become disconnected).

Appreciate any insights that could be offered and for the continued work on the awesome project!

[movitto]

One particularity to note is that I don't see any change in the 'balance' or 'drop threshold' values in the log as I changed the send_queue_limit. Regardless if it is 1000, 10000, or the max 65535 I always see the threshold limit being 15000 . I'm not sure if this indicates an issue or if this is as designed, with the limit being fixed but the load balance scaling different depending on the queue_limit setting.