Go: Handle path payments with a fixed source amount

[tomquisel]

As pointed out in https://github.com/stellar/stellar-protocol/issues/243, path payments don't handle the case where you have a fixed amount of the source asset. For example "Get me the most EUR possible for my 10 USD". This case is becoming increasingly important for Stellar applications.

Jed created CAP-24 to handle this case, but there's debate over whether it will be added or added soon. An alternative is to try to simulate Jed's PathPaymentStrictSend with a transaction containing multiple offers, plus cleanup logic.

Either way, we'll need a way to find paths for the "fixed source amount" case. This issue is a feature request, to add support for that case to the path finding algorithm.

It's unclear what the best interface is. It could be as simple as adding a source_amount parameter to the existing /paths endpoint, but that risks confusing users and making them think we already have CAP-24 in place when we don't. Maybe it should be a separate endpoint or a special flag, making it clear that path payments don't support that kind of operation yet.

[tamirms]

@tomquisel @morleyzhi

in this new path finding endpoint would you also provide a destination asset? Or, would you only provide the source asset and source amount without specifying any destination asset?

If you provide source_asset, source_amount, and destination_asset, then we will try to find the best path payments which spends source_amount of source_asset to end up with destination_asset. This parameter set seems to match the example use case you stated: "Get me the most EUR possible for my 10 USD."

Alternatively, we could make the new path finding endpoint completely mirror the existing path finding endpoint. In that case you would provide the following parameters:

• destination account which will be receiving the path payment
• source asset which will be spent by the sender
• source asset amount which will be spent by the spender

Using this scheme, the new endpoint would return all payment paths which:

• start by spending source_asset_amount of source_asset
• end by delivering an asset for which destination_account has a trustline

[morleyzhi]

Yes, our intention is that the endpoint takes in a destination asset.

[tomquisel]

It sounds like we'd end up with the following ways of calling path finding:

• Fix destination_asset, destination_amount, source_account, return paths from any source_asset (the current API)
• set source_asset, source_amount, and destination_asset, return best paths from source_asset to destination_asset (the new addition)

What's interesting is that if you fix source_asset and destination_asset, then you no longer need to provide either account. It's just telling you what's possible with the orderbooks. I find that pretty appealing.

It doesn't seem good that the interface is asymmetric, though. When destination_amount is fixed, you need to provide a source_account and you'll get back many possible source_assets. When source_amount is fixed, you have to fully specify both the source and destination assets.

The most generic and symmetric approach would be to allow all 4 combinations:

Fixed destination amount (the current kind of path payment):

• destination_asset, destination_amount, source_account
• destination_asset, destination_amount, source_asset

Fixed source amount (the new kind of path payment):

• source_asset, source_amount, destination_asset
• source_asset, source_amount, destination_account

I think that's probably what we should do to to allow devs flexibility. But is it too confusing? We could split /paths into /receivepaths and /sendpaths perhaps, but I don't love it because those names don't properly describe what's happening. /strictsendpaths and /strictreceivepaths properly describe the operations, but who gives API endpoints names like that?

The simpler option is to just extend /paths to accept any of those 4 combinations.

@morleyzhi @tamirms what are your thoughts? @accordeiro what do you think would be most intuitive?

[tomquisel]

Thinking about it more, the mirror to what we have right now doesn't make a lot of sense. Who would want to fix what they're sending and then pick between different assets the destination could receive?

This brings me back to Morley's original suggestion, to just add this interface for the new kind of path payment:

• source_asset, source_amount, destination_asset

[marcinx]

@tomquisel I think your last suggestion makes a lot of sense, is straight forward and has least risk for any type of regression for existing implementations on /paths. Probably just a question of appropriate naming for the new endpoint/interface.

Over here we're happy if the existing /paths endpoint keeps its current destination_asset, destination_amount, source_account structure and returns paths for all assets which are trusted by source_account. The last part is actually quite important for us because it lets us fetch live exchange rates for all assets supported on our platform with a single request.

[marcinx]

Thinking more about it, it might actually make sense to use destination_account in the new interface and return paths for all supported assets on destination_account (instead of only using destination_asset). Consider this:

1. User holds x BTC on "web platform".
2. User wants to withdraw x BTC to some "other account".
3. User types in public key of "other account".
4. "web platform" invokes new /inversePaths (?) endpoint and receives paths for all assets supported in "other account".
5. "web platform" presents user an option of all assets in "other account" along with target amount received if he withdraws x BTC.
6. User selects which asset he wants to receive in his other account and initiates path payment.

This way it would be similar (but inverted) to what we have in the existing /paths endpoint.

This would actually be a huge win for us over here because we could calculate with a single request how much of each asset the user would receive when he withdraws from his balance. Much easier and faster than invoking /account and then iterating x times over the existing trustlines and needing to invoke the new /inversePaths (?) endpoint for each trustline on that account.

[tamirms]

It should definitely be possible to handle all variants of the path finding request within the same endpoint. The main drawback of using one endpoint for all cases is that it becomes more difficult to document the semantics of the path finding endpoint.

[marcinx]

Out of curiosity... will these calculations be done in memory within the scope of the other ongoing /paths refactoring?

[tamirms]

@marcinx yes, that is the plan

[accordeiro]

I agree with @tamirms that making the semantics clear for the API users is a big challenge, especially when using the same endpoint for all cases. I also like @tomquisel's last suggestion, so I thought about a few alternative ideas to accomplish that:

a. Add a nested URL path to detail the operation:

• /paths/strict-send/ – would contain the new functionality, and take source_asset, source_amount and destination_asset params
• /paths/strict-receive/ – would basically be an alias of the current /paths functionality

b. Add a type (which could be either "strict_send" or "strict_receive") parameter to /paths, so we can easily explain/document what are the other required parameters based on the chosen type (and assume type="strict_receive" by default for backwards compatibility).

I like a better, so we can also have a different response format for each – what do you think?

[tomquisel]

@accordeiro I love the URI scheme in a.

[marcinx]

I'm trying to test the new pathfinding endpoint. Waiting for the ingestion to finish. Is it supposed to take a very long time? I'm a few hours in already.

I'm curious to test the performance now but was a bit disappointed to see that strict-send is constrained into a single destination asset (instead of destination account). Is there a chance to make strict-send synchronous to strict-receive in the future? Currently it behaves like this:

/paths/strict-send returns payment paths where both the source and destination assets are fixed.

However, there are use cases where you want to know the resulting amounts for all assets/trustlines in the destination account. Why do we restrict ourselves to one destination asset only instead of the entire destination account (and all its trustlines) like in strict-receive?

In strict-receive I get all amounts for all assets in the source account, while in strict-send I need to iterate over all trustlines and then invoke the endpoint individually for each one of them and that's very inefficient and slow.

I pointed out a use case in an earlier comment above on how this would be useful in beautiful UX/UI design and @johansten added a few thoughts on Keybase. It would be nice to see this come to fruition in a later release :)

Other than that I'm excited to test and will be glad to report back my performance tests on the new /paths endpoint.

@tomquisel @tamirms

[bartekn]

I'm trying to test the new pathfinding endpoint. Waiting for the ingestion to finish. Is it supposed to take a very long time? I'm a few hours in already.

Sorry to hear this. When Horizon is ingesting state it requires a storage for temporary objects (this is because of the way history buckets are designed). To make story short: the first version with experimental ingestion (0.19.0) stored these objects in memory however it required around 4GB of RAM for public network buckets. We decided to move the store to a temporary table in Postgres. It worked well during pre-release tests however we found out that under higher load on a DB it's actually slow. In a meantime, public network buckets were merged/rewritten to a new version (CAP-5) what changed memory requirements to ~1.5GB. To solve the issue you experienced, we're going to allow users to change the store they want to use, with memory storage as a default. We will also explore alternative stores (like a local filesystem temporary file).

This is experimental feature so such things unfortunately can happen. If you can't wait any longer for the state sync, I recommend rolling back to 0.19.0.

[tamirms]

@marcinx

I'm curious to test the performance now but was a bit disappointed to see that strict-send is constrained into a single destination asset (instead of destination account). Is there a chance to make strict-send synchronous to strict-receive in the future?

We will modify the endpoint to address your feedback. The /paths/strict-send endpoint will be modified to take the following parameters:

• source_asset
• source_amount
• destination_asset
• destination_account

However, destination_asset and destination_account cannot both be present it has to be one or the other. If destination_asset is provided then all payment paths will end at destination_asset. Otherwise, if destination_account is provided, then all payment paths will end at an asset held by that account

[marcinx]

Brilliant!

[bartekn]

Added to Horizon 0.21.0 milestone as one feature requested by @marcinx is currently in review: #1664.

[bartekn]

Deployed to staging environment: https://horizon-blue.stellar.org