Rippled: Recent fee rises and TxQ issues
During today's large increase in transaction fees, possibly due to a certain tweet, rippled is spewing lots of the following logs. Market makers connected to this instance are receiving "Can not queue at this time." responses. Is this the intended behaviour? The market makers are trying to use low transaction fees to get their offers onto the books and to make money by keeping the fees low. The end effect of the transaction fee increment process seems to be that the market makers are getting priced out of the market, which is ironic :-)
2017-Aug-28 22:47:18 TxQ:WRN Queue is full, and transaction B6B9A046340CFBE78DEE4B1D133DDFF2D2B071806B346F34DB4771FCE6E5997E fee is lower than end item's account average fee
2017-Aug-28 22:47:21 TxQ:WRN Queue is full, and transaction C92AC1CBCF48EDB1FDBB790CCFBD6EA74F5BDE6ACE2D27352FB74B5EF472AD98 fee is lower than end item's account average fee
2017-Aug-28 22:47:21 TxQ:WRN Queue is full, and transaction 2C5AF5FB72BA96A8CBBA6980D98ED42C6105627856F96DC5269726F46DAB1586 fee is lower than end item's account average fee
2017-Aug-28 22:47:21 TxQ:WRN Queue is full, and transaction 1701E6EAC50CA0D9994AED3BECF49E14FA6DCDC000F175A202820A7376B80A3D would kick a transaction from the same account (rfepxFxQKMSwn1otQRk35TgXbfLX6p8JBX) out of the queue.
2017-Aug-28 22:47:22 TxQ:WRN Queue is full, and transaction 9DCDD2AF543932BD43954C20BD3365396A83AEF2F94339027C3DE572C6B8173E fee is lower than end item's account average fee
2017-Aug-28 22:47:22 TxQ:WRN Queue is full, and transaction F1F70AF66AEE4C497BB97522C6034BCEC3B3E8D92277D97D10F8BCB621142FC5 fee is lower than end item's account average fee
2017-Aug-28 22:47:23 TxQ:WRN Queue is full, and transaction 9696B9F9E3195423214B310B7F4885F0B90F032C4E57DCA4090BEC4F818682B7 fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Removing last item of account rfepxFxQKMSwn1otQRk35TgXbfLX6p8JBXfrom queue with average fee of27297 in favor of 10982E55CC5219B0B4B90B5B2D205965196171F8C8A720800EF14D6B9F789C21 with fee of 446796
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction A1DEDCCF06002577E40DA3C2302674C68957B0F479264B66056D4D1457CC2EFE fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction 7AB84471D805ADE13BE32F5E83506D14A35AF08FAD790AD8F4AAF244904A00C1 fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction E7EC553FB99FB7948005A09AA41AE94418D80109926E557B0A089571D39FEDBF fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction 688C1E363C3B9E8391BEA044A9D57E8B45CAD6CC90DB2992C3B205588D62E03E fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction 4361FE30EFBF320166B9289B8B4263D86D0381BD826D78A096DDE1E512F99E78 fee is lower than end item's account average fee
2017-Aug-28 22:47:26 TxQ:WRN Queue is full, and transaction C801A86FEA2707A656CA741DE727BD5D87F054F0F2830A5BE832245BE9A75B70 fee is lower than end item's account average fee
2017-Aug-28 22:47:26 TxQ:WRN Queue is full, and transaction 5FD6ED2C1B61AEC719E0CE0485E712AE4771B448F8112E09E9F4B648200A79E7 fee is lower than end item's account average fee
2017-Aug-28 22:47:30 TxQ:WRN Queue is full, and transaction AAE9D9E47BE9DD6B7B2648D277F3F9266C80224CE250042F846B0A1BFF5843EB fee is lower than end item's account average fee
2017-Aug-28 22:47:31 TxQ:WRN Queue is full, and transaction 22C24DBD28C4FAA2F36E584330A719C919D8FB0DCA57407613663D9933DE4540 fee is lower than end item's account average fee
2017-Aug-28 22:47:31 TxQ:WRN Queue is full, and transaction AD735AF3B1298BDF5B355FE922BA318977D5FEA2C851A68E8DF9E610C9C377A1 fee is lower than end item's account average fee
2017-Aug-28 22:47:31 TxQ:WRN Queue is full, and transaction 40B59A6B2F425ACFAD39CA4F1458031993B39DBDEEBBAF137138A57643B727F7 fee is lower than end item's account average fee
2017-Aug-28 22:47:32 TxQ:WRN Queue is full, and transaction A749A097EBD0EC4B2F7F47F0D1393102A332DF4369D6336E2225DE8815F568B7 fee is lower than end item's account average fee
2017-Aug-28 22:47:32 TxQ:WRN Removing last item of account rfepxFxQKMSwn1otQRk35TgXbfLX6p8JBXfrom queue with average fee of27469 in favor of 9C888D0ACA02C4D15AAE2253433EEF82B257C6B556AAC6BB99B9AA4E1DBF3F14 with fee of 38400
q2017-Aug-28 22:47:33 TxQ:WRN Queue is full, and transaction 6C0A59D1EB3E48494A30AEDFA6C85D3B6ABDABCA117B96DE67B344412551A353 fee is lower than end item's account average fee
2017-Aug-28 22:47:34 TxQ:WRN Queue is full, and transaction 50F231BEAC67BAB309FCBA3A19E2198A7183DC73339F34BD0D274AEF506E22F8 fee is lower than end item's account average fee
2017-Aug-28 22:47:34 TxQ:WRN Queue is full, and transaction 5DB0C39C7AA8F82D385D5A6B2709566989A18F2C83D86321D221BE8497F03F48 fee is lower than end item's account average fee
2017-Aug-28 22:47:35 TxQ:WRN Queue is full, and transaction 38B08A4E6CE70ABBE29053344069600C91A44BA9F8756AD4BB550874B603ABF6 fee is lower than end item's account average fee
2017-Aug-28 22:47:35 TxQ:WRN Queue is full, and transaction A32A8EF77A14D6E1039769DB11F86569D6857BE3662EC517F4168653C7BDD550 fee is lower than end item's account average fee
2017-Aug-28 22:47:36 TxQ:WRN Queue is full, and transaction C234E7971A60621DE677272426217020F656ACB14B305C2914E6DDD101C9A470 fee is lower than end item's account average fee
2017-Aug-28 22:47:36 TxQ:WRN Queue is full, and transaction 6BE17EAA6D8C93BBB1DCB4D39800B005BDCD44BF4C662AD0BBE9716852FD8B54 fee is lower than end item's account average fee
2017-Aug-28 22:47:37 TxQ:WRN Queue is full, and transaction 936F4E461768015C8F49C3297DAF4DC53EC628B9D6EBEE8BAE62A598CD076731 fee is lower than end item's account average fee
2017-Aug-28 22:47:37 TxQ:WRN Queue is full, and transaction B002C8025DA8A38029F45AB900B8017CABF655EE692323FFAB7063D15E684FC9 fee is lower than end item's account average fee
2017-Aug-28 22:47:37 TxQ:WRN Queue is full, and transaction 89C6E7F4C12A80EF58E063D72E863B044A04D2B995F0DB8FD5543E296F494A5D fee is lower than end item's account average fee
2017-Aug-28 22:47:38 TxQ:WRN Queue is full, and transaction 77658D10FE4E7285922EC58880D1C6BDA1D7C1FB16C8D303F6DBF7641FCE50B9 fee is lower than end item's account average fee
donovanhide
All 68 comments
@donovanhide Thanks for the feedback and your question. As far as we can tell, yes, everything is processing exactly as designed. To prevent runaway resource usage and/or abuse, the queue was implemented with a size limit of 20 ledgers' worth of transactions (https://github.com/ripple/rippled/blob/develop/src/ripple/app/misc/FeeEscalation.md#other-constants).
If the network was staying healthy, and there were no problems with consensus times (and I haven't seen any evidence of problems yet), then it's safe to presume that the expected ledger size was at least 50, which means there were at least 1,000 transactions in the queue, which is honestly pretty impressive!
ximinez
on 29 Aug 2017
It looks like that's the case. For example, here's a fee distribution from one of those ledgers. The first column is the number of transactions, the second is the fee paid in drops:
4 10
5 12
13 30
4 68
12 70
1 100
2 490
1 800
1 5191
9 10500
You'll notice that several 10 drop transactions still managed to get into the ledger and the total number of transactions in the ledger was 52.
JoelKatz
on 29 Aug 2017
Thanks for the responses. I guess the real question is how do I choose a fee which can make a profit based on the information fed out from rippled. For example here are two recent ledgers and the fee amounts advertised during the time before each ledger closes:
Ledger | TxnCount | Max Fee | Fees
32299043 | 56 | 9700 | 5550,5739,5932,6128,6328,6530,6736,6945,7157,7372,7590,7812,8037,8265,8496,8730,8968,9209,9453,9700,10
32299042 | 36 | 8265 | 5363,5739,5932,6128,6328,6736,7372,7590,7812,8037,8265,5180,5363
32299043 seems to end on a fee of 10 drops, while 32299042 ends on 5363. I don't really know how to interpret this data. A market maker needs to be able to adjust his/her prices based on external data and/or trades occurring on the rippled ledger in a timely manner. Without being able to alter his/her offers in a timely manner, money will be lost or opportunities missed. If the fees are too high then profits will also decrease. Finding the balance to this problem is difficult, especially when the fees are sustained at such high levels as they were yesterday.
donovanhide
on 29 Aug 2017
I don't recognize the format of that data, but if those are all the fees that were paid by transactions on that ledger, it means that any transactions that paid less than those amounts were either put into the queue or kept in the queue.
Remember, that when a new ledger is opened, transactions are first pulled off the queue in order from highest fee to lowest. As those txs are added, the open ledger fee can start rising. And if it rises above the fee paid by the remaining highest fee tx in the queue, that tx will be left in the queue. Then both the open ledger and the queue are opened up for submissions. Transactions can be put in front of other transactions in the queue if they're willing to pay more. It's a trade-off between fee paid and time waited, and I have no idea how to make that decision for you.
Oh, there is one additional set of data points I can offer. If you use rippled's ledger command with ledger_index: current and queue: true (IIRC), you can see all of the transactions, including their fees, in both the open ledgers _and_ the queue. I don't know if that'll help you make a decision for your own fees, but that's what we can give you.
ximinez
on 29 Aug 2017
The Fees values list is a result of this calculation based on the incoming ServerStream messages recorded from the notification of a ledger close until the next ledger close:
Based on this information:
https://ripple.com/build/transaction-cost/#server-state
The basic problem statement for a typical market-maker might be:
Every t seconds I want to submit n offers to replace existing offers from a set O which need to have their prices updated due to changed external data. These offers need to be updated within t seconds, otherwise the next batch will clash. I need to know what fee will accomplish this goal with a certainty greater than x percent. I can then use this value to adjust the offer prices accordingly.
I'd say that this is a usability problem. Currently, an understanding of a complex queue at the rippled end is required and the client needs to manage a further queue to dynamically feed the rippled queue. This is further complicated by the need to deal with cancellation of existing offers, which can get messy, as there is no way to mark an offer with a client-generated id, other than hacking the expires value.
If there was an estimateFee API call which could take parameters t,n and x and return a single fee value estimate, that would be very much more usable :-) Even better would be if this was a subscribe-able stream.
donovanhide
on 29 Aug 2017
...
{
"LastLedgerSequence" : 32370764,
"account" : "rG1dLiK6p4e6w3mvAVCEGAFv6TcbMN5wHY",
"auth_change" : false,
"fee" : "12",
"fee_level" : "307",
"max_spend_drops" : "12",
"preflight_result" : "tesSUCCESS",
"retries_remaining" : 10,
"tx" : "F4011E477CA7EEE9ED3CBBD4D8A1768321578F4B6E6B76ED1DED6134A40DEA1A"
},
...
Given the above section of the ledger current queue output, if that was my transaction, and I wanted to push it faster into a ledger, are you saying to re-sign it with a fee of 307 raised from the level of 12, and that would raise its chances closer to 100% of successful submission?
donovanhide
on 29 Aug 2017
ximinez
on 29 Aug 2017
Honestly, that link does not help or answer the question :-)
fee_level is 12/10*256
How do I use the information from the queue_data to help select a more appropriate fee?
donovanhide
on 29 Aug 2017
I think the crux of this problem you're describing is that you're asking rippled to predict the future actions of the other users / transactions / servers on the ripple network, and that's really just impossible without a well trained neural network or a time machine. All we can do is describe the past and present. If you want a (nearly) 100% chance of successful submission, the only real way is to pay the current open ledger fee (from the fee command, or using the load_factor from server_info). If you want to get into a given position in the queue, you can grab the ledger info as described above and pay a fee that will give you a fee level slightly higher than the transaction in that position, but there is absolutely no way I know of to predict or guarantee whether you'll stay in that position. The queue is essentially an auction, and you can't predict what others will bid.
ximinez
on 30 Aug 2017
I agree that a predictive statistical model of varying number of transactions combined with varying levels of fees yielding a chance of success output would be very useful, but it would certainly cost a large amount of XRP to produce the training data on the live network :-)
I'm more than happy to use the present values in rippled, which seems to be the open ledger fee. What I need to know is if the values in the server subscribe stream are appropriate. Strangely, I see the response is no longer documented here:
https://ripple.com/build/rippled-apis/#subscribe
except the brief excerpt:
Stream: server - Information about the server status, such as load_base (the current load level of the server), random (a randomly-generated value), and others, subject to change.
The formula used is here:
If I were to select the highest seen value after the last ledger close, and just before submitting my transactions, would that figure be a fair representation of the open ledger fee? Unfortunately there are four separate API responses where fee information is presented: server_state, server_info, server stream and fee and the docs are all a little vague about the fields in each.
donovanhide
on 30 Aug 2017
Well, today it seems like:
rpcyXFq6ArWozyYLUE52FnNDjWWcnGYTGe
and
rDfdaugfiyePPcNKHL2soyL3wDmya32Ya3
are having fun paying huge fees and inflating the transaction fee for everyone else, resulting in hardly any transactions going through and servers reporting they are full. I'm sure they will probably run out of XRP and it's non-malicious, but it is very much observable that the transaction volume topples when the fee rises to current levels, which does suggest most bots do not have a suitable fee selection strategy, which goes back to the original point of this issue.
donovanhide
on 11 Sep 2017
I wonder if it's the same issue that rippled when I do send some ripples gives me transaction hash today but when I look it at https://ripple.com/build/ripple-info-tool/ it says tx not found.
What do you guys think? I do not see any errors in rippled log though.
gituser
on 11 Sep 2017
Ledger 32683368 is a good example of dysfunction. 4 out of 5 transactions all paying over 2 cents worth of XRP to be in an empty ledger, preceded by not particularly full ledgers.
donovanhide
on 11 Sep 2017
All day today I'm getting 'transaction not found' on ripple-info-tool whilst locally transactions are accepted.
I've resent now few of them, but I'm worried that I might send two times.
gituser
on 11 Sep 2017
@gituser You need to check the _validated_ transactions up until your specified LastLedgerSequence. The response you get when you submit is effectively useless.
donovanhide
on 11 Sep 2017
@donovanhide how do I do this?
gituser
on 11 Sep 2017
@gituser Attempt to interpret this:
https://ripple.com/build/reliable-transaction-submission/
into your own code :-)
donovanhide
on 11 Sep 2017
@donovanhide the thing is the method I've been using for sending was working perfectly fine until today. Is there some network issues at the moment?
I can see also via account_info that my account is "validated" : false
gituser
on 11 Sep 2017
@gituser I can't really help you debug your code, but this issue is that today, and on previous days, when some accounts raise the overall transaction fee, most other bots seem to fail to get their transactions into a ledger.
donovanhide
on 11 Sep 2017
@donovanhide ok, I'm looking into that documentation you sent before to make proper submission of the transaction in Ripple network.
Thank you.
gituser
on 11 Sep 2017
@donovanhide so I read about reliable transaction posting and it seems just too much hassle to implement all of that and I'm not sure that even after I do create the system which will check for LastLedgerSequence and re-submit transactions it will work better and won't give me more headache. In current system the transaction is only sent once. And if something goes wrong you've got to re-send it manually, but this also protects from accidental double sending.
So here's workaround I've used - I've just increased and made fee static to 0.5 XRP instead of using fee_mult_max parameter after this all transactions seems to be mined much more reliable.
gituser
on 12 Sep 2017
Yeah the fees are getting "ridiculously" high for market makers...they are not low enough to not be considered in the risk model. And it seems there are not too many transactions per second. Are the validators overloaded by some "malicious" attack that don't claim any XRP and the load_factor goes high? How is the load_factor calculated?
tuloski
on 13 Sep 2017
./rippled -q json ledger '{"ledger_index":"current","queue":true}' | grep account | wc -l
366
Certainly lots of transactions in the queue, not many getting into ledgers...
donovanhide
on 13 Sep 2017
We've escalated and are investigating. We have definitely seen evidence of undesirable emergent behavior.
JoelKatz
on 13 Sep 2017
I'm seeing rpJrC46bPSUiWwTuuRGSiqHdSaCNWTFBZv consistently at the top of the queue with high fees. Oddly seems to lead to:
https://www.eobot.com/user/436753
Which suggests that maybe the ledger is being used for mining payouts and the payout bot has some bad fee calculation code?
Edit: seems that address is also linked to https://www.jubi.com/ ...
donovanhide
on 13 Sep 2017
What is the "latency" I see in rippled code to calculate if the server is overloaded? Where that time is coming from? I see there are 2ms removed for jitter so I suppose it is related to the communication with the other nodes, but it's weird that it is the only measure of load of the network. And then there are a lot of magic numbers (+8 *4) :)...hard to decode that code.
tuloski
on 13 Sep 2017
The load factor I see is very high (minimum 3000 up to 400000 or more), so the fee is correctly calculated based on the loadFactor. Is there one of the validators having high load and then it propagates to the network?
s-west.ripple.com where I connect has a loadFactorServer = 1, so it might be another one.
tuloski
on 13 Sep 2017
Queue even longer this morning:
./rippled -q json ledger '{"ledger_index":"current","queue":true}' | grep account | wc -l
1019
Useful script:
./rippled -q json ledger '{"ledger_index":"current","queue":true}' | grep account | sort | uniq -c|sort -nr
366 "account" : "rECfeqVQCWfUdFjV2E3teK3xNHtegLAX4a",
102 "account" : "rBTp7LRcgYAXLDMUeKGCudamjienArx8eh",
72 "account" : "rK3Grwc6uQS9G2Mfff1S5UUdnttCRVwtEN",
62 "account" : "r9XRCuhi5uDZY9gPvpiCy6kWDjeH3wM4LU",
49 "account" : "rpJrC46bPSUiWwTuuRGSiqHdSaCNWTFBZv",
47 "account" : "r4aqu2zbAdX2QQ9bdeuGwifUMhtEeqx3pY",
38 "account" : "rJapan2paeQA81SvYk1iEpgmvSh9Z7xaJs",
37 "account" : "rabZ5dJdhqjJPnMzriaHp6VdLGMftPwXGh",
35 "account" : "r9Gps6fB9YLuZ87rWx7M9TgLAGK2zsz5s6",
19 "account" : "r9hNR8G9nARPL7LSAQoLv5EZdVjHBiXr7Q",
12 "account" : "r9KG7Du7aFmABzMvDnwuvPaEoMu4Eurwok",
11 "account" : "rsheL3o95iWwE8fnzVcNNeAS8iFwN6UwTw",
10 "account" : "rwfGzgd4bUStS9gA5xUhCmg1J86TMtmGMo",
10 "account" : "rnHXzwFLSen6hpTsKtVCNUpVKQsTZr8TeD",
10 "account" : "rMQTGsAtFa1Aow2G9YRJabBwj1y1kTHLxN",
10 "account" : "rEr3hxu5aim5tDWwH7H8BK47K91tR8c7FM",
10 "account" : "rD4G6gtD2KwHqsRf7pcyA8r1neUzXT61ix",
10 "account" : "r9Vv4yJeVFYqLBuZ85muKmxHvPzmZUDFMM",
9 "account" : "rLAS7brcrD4H1tb2U7qFKEZAR5L2LuMxoy",
9 "account" : "razorxL2wrHEheeJ7c3BZ14gAkciU7tMTd",
8 "account" : "rhS2H7ETM3wBkFETvYycoUm9FEDYi44Pg4",
5 "account" : "rP45RuF983ttNc6zT7WtthnTSDC1iaGhg1",
5 "account" : "rH3uSRUJYoJhK4kL9x1mzUhDimKE2n3oT6",
5 "account" : "rEBhG5cxEcmM9rTwZgsMyS7Hxs3xdLuJ6",
4 "account" : "rnwmbU2Luj9ZWcisU6VGopMBsdMPhbq88B",
4 "account" : "rnMCfd99pwRE8u4LxE43Z1pDzock2kXbLQ",
3 "account" : "rUYMTVAPdEvTA46TZnwBwx6CGwoqg7HsG",
3 "account" : "rspZhZwvM8y6iSJERn3LpfXoEHkAk5ond1",
3 "account" : "rLJFygDvrqbGgwNew4WbiSTomMS95Tqqiq",
3 "account" : "rLHtvB1kZimYJsDCqCX4iQxRtSAqqFtrA7",
3 "account" : "rKd19spe7AruEeMFxwTG9jmXdBUPeHU9i7",
3 "account" : "rHENTPe3nMDLGJd59e5a3tYH9BXz1nFhvA",
3 "account" : "rcyS4CeCZVYvTiKcxj6Sx32ibKwcDHLds",
3 "account" : "rbokx39budXTQJRz2oEQiNfe9FwHKbWRB",
3 "account" : "rBeToNo4AwHaNbRX2n4BNCYKtpTyFLQwkj",
3 "account" : "r5oibBjEvqt4TtLpbKx2qf7Bru5a8Y4DT",
3 "account" : "r4WXyQXoWwk4zEKnWQwBNf563XyPXjhL7U",
2 "account" : "rUSqPxBWpHGGYawAg2e2nnC7EEZAY5eSRw",
2 "account" : "rUeFPRGNjtcbtezyQKKiDcS1eQyYLQ1gcr",
2 "account" : "rsc1JP8UNUtpwxH2XN5xBiqTsP6Hu3Dmfi",
2 "account" : "rPMzWvu7WZmWtTQb4bH9HdxGKHeWNy4vyN",
2 "account" : "rpa9GNAHVoQQq2Z533ZsD8TX4Tsu5d9f4v",
2 "account" : "rLELbUxAfArviMX1yWj4ccmxnaHP159Q8",
2 "account" : "rKiXhWVFvihz9zr1VqPxCifyGoCbwUuTS1",
2 "account" : "rhub8VRN55s94qWKDv6jmDy1pUykJzF3wq",
2 "account" : "rEp4SYVfwQgGbWCJV4tBAgGNpG2KeiaY1W",
2 "account" : "raL7tMvfQUL9sHQMRdwWpEJFRo5bxtzeiE",
2 "account" : "r3CmdWT7sgBw9FossNwp5g8xdo7iBJwFMM",
2 "account" : "r1eb6z9HtARaHSSz2hygEDi8yubA1bsJu",
1 "account" : "rUFYLgGfX8CECpDcqbbC6CWbDg5BzD1EJk",
1 "account" : "rpP2JgiMyTF5jR5hLG3xHCPi1knBb1v9cM",
1 "account" : "rnVSYhajS62ZzSQu9crNNeibJUmFGWKnRc",
1 "account" : "rND7fYoHXX8zFPreKNYWgQnvGj9dLMhSe8",
1 "account" : "rKaVa3JczUZQehKEbFKjYqdc29UJjG3gQJ",
1 "account" : "rJ2V5E9eXhgzRFwcWaKpeuVJytfmkTXztZ",
1 "account" : "rH1GonJfbvgAiMeWzGFB59Wo75Yd5pPz7",
1 "account" : "rDAtpYpng9FQugwGbZvcrcYZRZJ6jpzMx7",
1 "account" : "rDAN8tzydyNfnNf2bfUQY6iR96UbpvNsze",
1 "account" : "rchGBxcD1A1C2tdxF6papQYZ8kjRKMYcL",
1 "account" : "raq3FG6ftAVXw7F9ePX2zGKfGzADc8SEdc",
1 "account" : "raG6ntBY6yDUwLiaVU29pbirMTv375o9MZ",
Looking at the TXQ trace logs, I'm seeing lines similar to:
2017-Sep-14 09:32:08 TxQ:TRC Transaction 5C0B9B414422157B71DB8C0D95295F3E9A07BD2C89F0C8A06ED796001D781CA2 from account rP45RuF983ttNc6zT7WtthnTSDC1iaGhg1 has fee level of 2560 needs at least 1991940 to get in the open ledger, which has 147 entries.
The fee_level of 1991940 seems to come from here:
Not sure that the maths behind this calculation is properly justified.
donovanhide
on 14 Sep 2017
@donovanhide isn't the fee calculated as base_fee * load_factor? BTW it's pretty hard to follow rippled code on fees. I gave it a shot yesterday but without comments in many parts of the code is very hard to follow.
tuloski
on 14 Sep 2017
@tuloski Yeah, the code is complicated and possibly only makes sense to the people who wrote it :-)
I think the TXQ fee and/or fee_level is treated separately from the actual fee for the open ledger. What I have noticed is that when I restart rippled, the TXQ doesn't immediately fill up to the high levels it had before the restart, which means there is a lot of local state floating around, which probably means this is a nightmare to debug :-)
Ed is obviously working on it:
https://github.com/ximinez/rippled/commit/ff660dc2b9d132b7b1646c560ffa458c387db48e
donovanhide
on 14 Sep 2017
In my idea the fee should be only the one to protect the network, i.e. the open ledger one. What's the meaning of having a local fee?
tuloski
on 14 Sep 2017
I guess the idea is that the TXQ protects the network from floods of low fee transactions, especially when the fee is high. Of course you could just write a custom client that connects to the other peers and sends the transactions no matter what the fee is. The TXQ is probably also meant to help the user get his/her transactions into a later ledger, if the fee is high now. But that is not working at the moment...
Another option is to have a global queue, shared by all other peers, but then you aren't charging a fee for the distribution of those transactions which fail, but do consume network bandwidth.
donovanhide
on 14 Sep 2017
Now even with 0.5XRP fixed fee transactions ain't going into the network.
I'm getting transaction not found via rpc-tool here https://ripple.com/build/ripple-info-tool/
Even after resubmitting yet again completely new transaction it's still not validated.
As for transactions in the queue there seems to be not much:
./rippled -q json ledger '{"ledger_index":"current","queue":true}' | grep account | wc -l
41
Any good validators out there which are reliable? I'm using
[ips]
r.ripple.com 51235
in the config..
What is happening? For months the system worked just fine.
gituser
on 14 Sep 2017
I think there is a difference per rippled server. I am using a private rippled. An I am now dividing the api.getFee() with 100 to get those lower transaction fees. Most transactions are now queued, but none of them fails. Most fees that I am using now are smaller then 0.001 XRP. So, apparently, different rippled's show different behavior.
jnr101
on 14 Sep 2017
I do use my own rippled instance, but it doesn't help.
Do you resend these transactions manually?
Do you use LastLedgerSequence on the submit ?
Also I do not see my transactions in the queue in the current ledger via:
./rippled -q json ledger '{"ledger_index":"current","queue":true}' |grep 'txhash'
gituser
on 14 Sep 2017
- No, I do not resend the transactions in any way
- I do not set the LastLedgerSequence parameter (thus using the default of +3 current validated ledger)
- I check the transactions via other tooling, they certainly made it in the ledger(s)
jnr101
on 14 Sep 2017
@jnr101 if the fees are different on different rippled, then it's even more worrisome :(
Are you sure? Can you connect to your rippled and to a public from RL and compare the api.getFee()?
tuloski
on 15 Sep 2017
Anyone had the situtation when unconfirmed (validated=false) account balance Sequence number is more than validated balance Sequence number?
It seems that's the problem for my main account to send out new transactions.
I've already added LastLedgerSequence into all my new outgoing transactions which I'm getting from server_info and adding to it 4, but it doesn't help transactions are stuck and not being validated.
Any suggestions how to fix the difference of Sequence numbers for validated/unvalidated states of the account?
gituser
on 15 Sep 2017
@gituser Yep, I've seen that.. pass "validated" in the "ledger_index" field for the account_info API call:
donovanhide
on 15 Sep 2017
@donovanhide I did that.
The question is: why my particular account isn't able to send transactions into the network without a problem, why they do hang if they are sent from this account?
If I try from another account (where Sequence number is the same for both "ledger": "validated" and without that param) the transaction is validated almost instantly..
Or maybe my account was banned put on some limit could be that's the case?
gituser
on 15 Sep 2017
It's because your transactions are stuck in the queue... There is some complicated, increased fee trick to overwrite transactions in the queue, but I've never tried it.
donovanhide
on 15 Sep 2017
@donovanhide any link to that trick/technique ?
also I've checked the queue via account_info call it gives me for ledger_index: "current":
"queue_data" : {
"txn_count" : 0
}
Is there any way to list these stuck transactions somehow ?
gituser
on 15 Sep 2017
Step 4 here:
https://github.com/ripple/rippled/blob/develop/src/ripple/app/misc/FeeEscalation.md#transaction-queue
increase the fee by 25%. There's obviously no point doing this if the transaction has all the same values.
If you share your account id I can tell you if your transactions are sitting in my queue, plenty are :-)
donovanhide
on 15 Sep 2017
Easier still, here's my current full queue:
./rippled -q json ledger '{"ledger_index":"current","queue":true}' | grep account | sort | uniq -c|sort -n
1 "account" : "r3mcYA8MUfZ9RSKhbNYzfMUP7Jsq9vNBha",
1 "account" : "r9Gps6fB9YLuZ87rWx7M9TgLAGK2zsz5s6",
1 "account" : "rB1za2ZVgDnNB7u8LbVN61k5nCByBUtXCA",
1 "account" : "rfKTnx7UE2Ab596Gan6DFrxeKtijQxr2nZ",
1 "account" : "rG7eucYLyK39JjptdTksiKKdGEPGSFPeW3",
1 "account" : "rGFVGdAo2iwJPZNkRqQ7EC5UEcdiFsBYoN",
1 "account" : "rHmgCr7g5i425UmfYQH7EnBQhocG3oT756",
1 "account" : "rKPNA16qJ9Vpymu4xeRq98c4Nsk8jwEZar",
1 "account" : "rMhPVE4SnkcbbpgZwxqZAiv5sRTYh3Lfrf",
2 "account" : "rJBpryFMKP7dPRjaVJSogkVHrf8ydoQMcx",
2 "account" : "rLHtvB1kZimYJsDCqCX4iQxRtSAqqFtrA7",
2 "account" : "rUZwBRmxtK9PwoJqAsgJg5P5was3Bd7wjA",
2 "account" : "rwvCPrwkL8dr1pGfrk74v8FXus7APEd8Lk",
3 "account" : "rEp4SYVfwQgGbWCJV4tBAgGNpG2KeiaY1W",
3 "account" : "rPandaauXvGmajQAng65b3sByFeMWxX8en",
7 "account" : "rnMCfd99pwRE8u4LxE43Z1pDzock2kXbLQ",
10 "account" : "rPvVZ7jE5QzkiUhHFxf6s4EanMRqzGC4bs",
34 "account" : "rK3VruB2uFg3XALfX9UffHEQmjXEyGWgfc",
38 "account" : "rJM7t2aG7aXid9j8LcnC6X3rf9GhX3ZKgN",
42 "account" : "rGHDkp1JG9ccieSVyvXywU7MxcXBAYSKDn",
44 "account" : "r9KG7Du7aFmABzMvDnwuvPaEoMu4Eurwok",
46 "account" : "rL3iwiaRjQoPXZD1U7hNGSsSj3sj3CoYNg",
93 "account" : "rusty54REyh8LnsupyXD61WEbnoAjcKwb",
105 "account" : "rhTqdCG4HAM9uFeiQAtfpH9uKjK4VBxGeT",
115 "account" : "rGskymq2eMkCopmzP9od3gfRJrC5V193x1",
131 "account" : "rabbit1eyDGsiWzRH4wL4vnk7giSxWyMeb",
200 "account" : "razorxL2wrHEheeJ7c3BZ14gAkciU7tMTd",
312 "account" : "rJapan2paeQA81SvYk1iEpgmvSh9Z7xaJs",
There is no my account in your listings, furthermore I've checked on my rippled using the same query as you did and there is no indication of my account as well.
It's super weird.
I'm sorry but I don't want to affiliate myself with this particular account in public.
I can't even determine TXHASH-es which are stuck :/
gituser
on 15 Sep 2017
Are you getting tefPAST_SEQ,terPRE_SEQ,telINSUF_FEE_P or terQUEUED responses when you submit? It's kind of irrelevant, because tx's with those submit responses can succeed anyway, but might give you a clue as to what is going wrong...
donovanhide
on 15 Sep 2017
No I don't. I'm getting tesSUCCESS for the new transactions.
The old ones of course I can try to find in the history, but I'm not running full rippled node, I've set pruning to 2000 there so old data is being deleted and also by looking via https://ripple.com/build/ripple-info-tool/ on all those transactions I'm getting Transaction not found.
EDIT: I've just checked the transaction which I sent earlier (about 1.5 hours ago) and it's not found anywhere neither on the tool neither in my local rippled instance.
gituser
on 15 Sep 2017
You can imagine the Ops department in a bank phoning the dev late at night saying the exact same thing :-) There are definite usability questions about the transaction queue... Batch submission with atomic guarantees would go a long way to simplify the user experience.
My best advice is just wait for the fix; my market makers cannot operate at a profit with the way things are currently working.
donovanhide
on 15 Sep 2017
I can see also via account_info that my account is "validated" : false
That just means that you are asking the server for the most recent information, which is unvalidated. Most rippled queries default to querying the open ledger. Typically this gets you the most current information, taking into account even unverified transactions. That's usually what you want. (For example, if you submit a transaction to send XRP to someone else, you don't want to see that you still have the XRP just because the transaction isn't fully validated yet.)
Unfortunately, at the moment, open ledgers are diverging a bit and querying the open ledger can show you things that are less likely to correlate with future network states. Querying validated information is always safe unless you're querying a misconfigured server or the network's consensus logic as a whole is broken.
My best advice is just wait for the fix; my market makers cannot operate at a profit with the way things are currently working.
Anything relying on being able to clear large numbers of transactions at low cost should probably wait for a fix.
JoelKatz
on 15 Sep 2017
@JoelKatz,
I understand the difference between querying unvalidated vs validated open ledger.
The question is how to reliably submit new transaction?
1) I've added LastLedgerSequence which I get from server_state according to documentation and I'm adding to the resulting value 4.
2) If I omit Sequence in the transaction it defaults to the unvalidated open ledger (account_info without ledger:"validated") Sequence
3) If I specify Sequence in the transaction from the validated ledger (account_info with ledger: "validated") it gives me error tefPAST_SEQ
So what should be the logic regarding 2) & 3) if I submit let's say simultaneously (at the same time) multiple transactions will be there a problem?
Also, another question:
according to the documentation https://ripple.com/build/rippleapi/#transaction-fees there is maxFee parameter but If I specify maxFee the signing fails.
And another question:
How to replace some transaction with Sequence number? I do get on submit tefPAST_SEQ on my local rippled and transaction isn't going to the network.
@donovanhide
You can imagine the Ops department in a bank phoning the dev late at night saying the exact same thing :-) There are definite usability questions about the transaction queue... Batch submission with atomic guarantees would go a long way to simplify the user experience.
That's very true.
My best advice is just wait for the fix; my market makers cannot operate at a profit with the way things are currently working.
Seems it is the only way, but I'd like to understand more about how ripple works in terms of sending/confirming transactions and why some are confirmed faster than others.
gituser
on 15 Sep 2017
@tuloski,
I did some checks and wrote a very simple program only doing the api.getFee() for each ledger.
C1 = Client@home_local_VM
C2 = Client@remote_VM1
S1 = Rippled@wss://s1.ripple.com
S2 = Rippled@remote_VM2
I tried all combinations. It turns out that each rippled gives different fees back. (not sure if that is worriesome, I can imagine it will also look at local load). When connecting to the same rippled it will give the same fees back. It was at first a bit confusing, but even when connecting to s1.ripple.com it sometimes gives different fees. This is because there are several rippleds behind that address. The factor of difference can be significant. Below are 2 connections to s1.ripple.com at the same time:
[15/09/2017 21:15:03] Connected..
[15/09/2017 21:15:06] Fee (0.10004165625)
[15/09/2017 21:15:09] Fee (0.10004165625)
[15/09/2017 21:15:13] Fee (0.098415)
[15/09/2017 21:15:17] Fee (0.098415)
[15/09/2017 21:15:21] Fee (0.10086000000000002)
[15/09/2017 21:15:24] Fee (0.097606640625)
[15/09/2017 21:15:27] Fee (0.097606640625)
[15/09/2017 21:15:30] Fee (0.097606640625)
[15/09/2017 21:15:33] Fee (0.098415)
[15/09/2017 21:15:38] Fee (0.099226640625)
[15/09/2017 21:14:54] Connected..
[15/09/2017 21:14:57] Fee (0.152006625)
[15/09/2017 21:15:01] Fee (0.15)
[15/09/2017 21:15:05] Fee (0.151001625)
[15/09/2017 21:15:09] Fee (0.152006625)
[15/09/2017 21:15:12] Fee (0.15)
[15/09/2017 21:15:17] Fee (0.15)
[15/09/2017 21:15:20] Fee (0.152006625)
[15/09/2017 21:15:23] Fee (0.14900165625)
[15/09/2017 21:15:26] Fee (0.14900165625)
[15/09/2017 21:15:29] Fee (0.14900165625)
[15/09/2017 21:15:33] Fee (0.15)
[15/09/2017 21:15:38] Fee (0.151001625)
The client (C1 or C2) do not make the difference in fees.
jnr101
on 15 Sep 2017
could it be that the systems are busy due to increasing use of some of the new functionality, which then would use more system resources then perhaps anticipated? in that case it would be just organic growth and the fee-escalation should be loosened somewhat. Or maybe we're dealing with non-optimal implementation of the new functionality..(But I am just wildly guessing here of course)
jnr101
on 15 Sep 2017
@gituser I'm using the MaxFee without problem...just the transactions don't go through because my max fee is often not enough and it fails with telINSUF_FEE_P
@jnr101 I don't get why 2 different rippled should give different fees. At the end the fee that matters is the open ledger, because you could modify your rippled code to "ask" you 0 fees, but then it would fail in the consensus. In your case your rippled was giving a lower fee and then the transaction was validated? So now I can say that whatever I thought to know about the fees is not true :)
tuloski
on 15 Sep 2017
@tuloski
I'm using the MaxFee without problem...just the transactions don't go through because my max fee is often not enough and it fails with telINSUF_FEE_P
I'm getting this when I try to sign transaction with MaxFee field inside tx_json structure:
Field "tx_json.MaxFee" is unknown.
gituser
on 15 Sep 2017
@tuloski All I can say is my transactions validate with apparently a lower fee setting than what @gituser is using. I don't know why. But I can say that my instance of rippled is configured 'tiny' and running on relatively fast system. And perhaps of influence, I do not allow inbound connections on my rippled
You can try the test with the fees with this code:
'use strict'
const {RippleAPI} = require('ripple-lib')
var moment = require('moment')
const feeCushion = 1.2
const api = new RippleAPI({
feeCushion: feeCushion,
server: 'wss://s1.ripple.com' // Public rippled server hosted by Ripple, Inc.
});
function timestamp() {
return moment().format('DD/MM/YYYY HH:mm:ss')
}
api.on('error', (errorCode, errorMessage) => {
console.log(errorCode + ': ' + errorMessage);
});
api.on('ledger', ledger => {
return api.getFee().then(fee => {
console.log("[" + timestamp() + "] Fee (" + fee + ")")
}).catch(console.error);
});
api.on('error', (errorCode, errorMessage, data) => {
console.log(errorCode + ': ' + errorMessage);
});
api.connect().then(() => {
console.log("[" + timestamp() + "] Connected..");
}).catch(console.error);
install/run with
npm install moment ripple-lib
node getFee.js
start two instances and when it connects to different server you will find different fees
For comparison here are some fee results from my own rippled:
[15/09/2017 22:08:05] Connected..
[15/09/2017 22:08:07] Fee (0.06402665625000001)
[15/09/2017 22:08:11] Fee (0.06272662500000001)
[15/09/2017 22:08:15] Fee (0.063375)
[15/09/2017 22:08:19] Fee (0.063375)
[15/09/2017 22:08:23] Fee (0.06402665625000001)
[15/09/2017 22:08:26] Fee (0.062081625)
[15/09/2017 22:08:29] Fee (0.063375)
[15/09/2017 22:08:33] Fee (0.5360574375)
[15/09/2017 22:08:36] Fee (0.06402665625000001)
[15/09/2017 22:08:39] Fee (0.062081625)
jnr101
on 15 Sep 2017
Regarding: Sequence number issue it seems to be a regression in the v0.70.1, I've switched back to the v0.70.0 and a miracle - now the Sequence for both validated/unvalidated account_info is the same and I also can send now transaction with previous Sequence!
EDIT: hm, no, after some further testing there is a difference again between Sequence numbers for validated / unvalidated account_info even on v0.70.0.
gituser
on 15 Sep 2017
@donovanhide do you have maximum_txn_per_account specified in your config to something greater than 366? I'm not seeing any accounts with more than the default 10 queued transactions.
https://github.com/ripple/rippled/blob/develop/doc/rippled-example.cfg#L535-L538
wilsonianb
on 16 Sep 2017
@wilsonianb
Yep, had it whacked up since Ed advised me to do so about a year ago :-) I submit batches of market maker transactions greater than 10 about once a minute:
$ grep maximum_txn_per_account rippled.cfg
maximum_txn_per_account=1000
I'm now seeing less transactions per ledger than a Chinese Bitcoin Exchange circa 1st October 2017 :-)
donovanhide
on 19 Sep 2017
BTW I'm seeing many transactions with low fees (0.000011) getting into the ledger. I don't get how is that possible...they might have their local rippled with low load and low fee, but then why don't they fail in the consensus? How can I retrieve the open ledger fee?
I see mainly these accounts with very low fee: rUp8CvtneUUL6qYaxfZwUHg2cwYuo5jugG, rEp4SYVfwQgGbWCJV4tBAgGNpG2KeiaY1W, rD8LigXE7165r3VWhSQ4FwzJy7PNrTMwUq
tuloski
on 19 Sep 2017
Hi Ed (@ximinez)!
thought I'd give the 0.70.2 hotfix an early quick spin. The queue submission responses are a lot more enlightening than the old blanket telINSUF_FEE_P, which is great :-)
So, my market making bots now can submit transactions to the queue, and now strangely, exactly 10 from each batch always seems to succeed and the rest get queued locally, but do not succeed before the next batch, roughly a minute later. Is this the maximum_txn_per_account setting being applied at the validator end? Maybe I'm being premature, and 0.70.2 need to be deployed elsewhere :-)
donovanhide
on 19 Sep 2017
Is this the maximum_txn_per_account setting being applied at the validator end?
Most likely.
ximinez
on 19 Sep 2017
We don't expect the fixes in 0.70.2 to have much effect until they're more widely deployed. We are still testing and finalizing the build.
JoelKatz
on 19 Sep 2017
Queues have been drained and local load_factor seems to be back to normal. Guess the fix has been deployed to the validators. Any chance of a tagged release?
donovanhide
on 21 Sep 2017
Soon鈩笍
ximinez
on 21 Sep 2017
0.70.2 has been released to master and tagged. Enjoy!
I'm going to go ahead and close this issue. If you have any further problems, feel free to reopen or open a new issue.
ximinez
on 22 Sep 2017
Nice work. Trading has resumed :-)
If you do get a spare moment after this slog, would be good to get some thoughts on building a local model for predicting a transaction fee for a range of possible transaction numbers in the next ledger:
https://www.xrpchat.com/topic/9809-questions-on-fees/?do=findComment&comment=98593
donovanhide
on 22 Sep 2017
Related issues
MarkusTeufelberger
路
5Comments
tuloski
路
9Comments
mDuo13
路
5Comments
gituser
路
5Comments
tuloski
路
8Comments