Go-ethereum: eth_listTransactions

[tymat]

This feature has been discussed on gitter in the past and we would like to see this get implemented for accounts and contracts.

_Proposal_

eth_listTransactions

Returns a list of transaction hashes for a given external or contract account

Parameters

String - the account or contract address

params: [
  "0x385acafdb80b71ae001f1dbd0d65e62ec2fff055"
]

Returns

DATA - A list of transaction hashes

[gavofyork]

This is useless - transactions are merely one way of two ways to send a message to an object ("contract") in Ethereum; CALL would be the other way.

Use logs. It's what they were designed for.

[gavofyork]

would close.

[Gustav-Simonsson]

@gavofyork It's far from useless. For example, it would be very useful for keeping track of non-contract txs, which cannot use logs.

This feature has been requested from several people. We shouldn't force users to only use the platform in a certain way, and If users desire a API call which doesn't conflict with anything or have any significant drawbacks, we should consider adding it.

I'm in favour of adding this.

[obscuren]

So am I. But let the community decide on this. Get enough +1 and it will eventually be considered/added.

[ethernomad]

Maybe messages to non-contract addresses should automatically get a blank log entry? It would store the originating transaction id.

[frozeman]

:+1: I fully agree with gustav here. Even though most of the cool stuff can happen in contracts, being able to simply send money from one account to the other will be heavily used. And then you need a easy way to display its transactions (e.g. the wallet needs that for example :) )

_I would add a second parameter, which is the nonce, from which on you want to get transactions. This way you don't need to request the whole list over and over again._

[tymat]

eth_listTransactions is very much needed for Ether to Ether transactions. For wallet devs we need a way to present this information to users so they know how their Ether was spent and when it was spent and which TX hashes they occurred in. Adding a separate persistence layer to store this data for easy access is cumbersome and would have to be constantly updated on the client side.

This is very useful and much needed.

[slothbag]

+1 I found myself looking for this feature.. would be great to get it included.

[VoR0220]

:+1:

[VoR0220]

this is much more efficient IMO than the logs...and there's a lot of use cases for this feature for execution in other contracts.

[VoR0220]

I would add a third optional parameter for number of transactions from the top or bottom (perhaps something python-esque?)

[frozeman]

Though i +1 this feature @VoR0220 you can't get logs only from transactions, as there can also be logs created when one contract interacts with another.. So only relying on this feature won't work.

I don't like the third parameter actually, as a nonce (from which nonce to get tx on), is more than enough to control the tx flow.

[VoR0220]

how does nonce control how many transactions you would like to see?

[d11e9]

:+1:

[frozeman]

If you say i want to get all transactions from nonce 50 and there a re a total of 60 tx, you will get 10

[bas-vk]

@frozeman, what if you want to retrieve tx [100...150] when there are 10.000+ transactions?

[VoR0220]

@frozeman but that's for a nonce in the chain correct? So I would only get the transactions for that block, no (I might be totally off here and feel free to correct me)? However, if I wanted to go and see the past 50 or so transactions for a particular address, how does that nonce benefit me if I want to get specific with the transactions displayed?

[frozeman]

nonce is an ear increasing number of transactions from an account.
With web3.eth.getTransactionCount('0x12345..') you basically get the next used nonce (as nonces start from 0, like array.length)

And then you can subtract whatever number you want to go back and ask for those tx.

[frozeman]

@bas-vk you couldn't :)

[obscuren]

While easy to implement its difficult to do it right. Indexing all transactions would require quite significant storage and most certainly doesn't scale.

One way to do this is as how we MIP mapped LOGs and use a similar query mechanism, but this is an expensive and CPU/IO intensive operation because there are far more transactions than there are LOGs.

Doing this for LOGs is actually fine since you more or less paid for it where as transactions incurs no additional cost in terms of upfront payment. letting the user decide what to map isn't an option either since it just won't work for light clients. Light clients depend on archive and full nodes for data retrieval and can't make use of this "user setting" option because they don't receive any other data than the data that is explicit requested.

I understand this is a much requested feature but understand that it comes at a great, expensive cost and, as I said earlier, difficult to do it right.

[frozeman]

I don't expect dapps requesting this endpoint often, rather only once when a dapp syncs for the first time. It will then keep this information in a local database anyway. And only requests the last x tx.

Remote nodes would have to rate limit their requests anyway somehow.

[m888m]

I second the need for transaction listings.
Regarding implementation, to compromise between system load and user friendliness, I suggest to use nonce as single argument, and implement a filter on the gui where users can limit time, counterparties, amounts etc.

[obscuren]

@frozeman that's quite an assumption you are making there and neither solves the issue at hand I described above.

@m888m the nonce is hardly the issue

[obscuren]

Remote nodes would have to rate limit their requests anyway somehow.

It's not about rate limiting either. Please reread

[rfikki]

list of transaction hashes for a given external or contract account. +1

[sammy007]

+1, with nonce param.

[m888m]

@obscuren nonce was not my main point, the need for user friendly transaction listings was. To push end user adoption, we definitely need a user-friendly way to show past transactions that happened during a certain time period and with certain counter parties. Furthermore, the transaction fees occurred for a set of transactions must be easily established.

[obscuren]

@m888m mine neither. I understand what everyone has reiterated over and over again but as I've mentioned above in great detail; there are issues with the requested feature. It's not as simple as it seems.

[tymat]

Indexing of this can be an optional setting when starting up the node. It doesn't have to be a requirement so clients that do not have CPU power can just turn the feature off.

Right now most of the web wallets and mist based wallets are forced to use some external persistence mechanism to have this data available to the user so any efforts to save CPU/etc... becomes a zero sum effort and you add an additional TCP/socket round-trip just to get the data from the blocks and then another connection to the persistence layer. Is the cost of doing indexing on the node greater than that? Probably not.

[caktux]

Got my vote. Maybe index the node's accounts by default and all accounts only optionally?

[ghost]

+1 add getTransactions(address) to API.

[karalabe]

Just a thought. As already mentioned, doing a full index of transactions would be costly both CPU and storage wise. However, if we consider that most people will only ever care about a handful of accounts, we could do a heuristics whereby a node would only index transactions for:

• Etherbase
• All accounts reported by eth.accounts
• Any account previously requested

This way the storage price would be linear to the operations you want to perform (i.e. you only expand CPU/HDD resources on the ops you personally need).

Note, this is not a +1 for this issue. I'm not yet fully sold on this. It's only a discussion idea that we could consider if/when implementing this feature.

[wildbunny]

If you guys are considering adding this, please can we have a 'since block' parameter? That's how exchanges sync their coin's blockchains and it would save them time to have this parameter.

[tymat]

+1 on the idea of doing a heuristic approach.

[chrisfranko]

+1 would like to see this

[altsheets]

_Still_ no solution for this? (In September I got an answer "might do that as a weekend project").

You are funny people.

If I cannot simply check for transactions out of and into my account, it is pretty useless as a currency, no?

Until you start listening to your userbase ... please publish a fully working code that achieves the same result, that we can use for the time being.

Thanks.

[altsheets]

i.e. publish working code for getting all (recent) transactions of an account.
I suppose it is done with this: https://github.com/ethereum/wiki/wiki/JavaScript-API#web3ethfilter ?

Please give working example code not for "pending" but for transactions of an account .

Thanks a lot!

[altsheets]

Fiddling around with those suboptimal manual instructions.

I am stuck here:

var options = {address: "0xf2cc0eeaaaed313542cb262b0b8c3972425143f0"};

var myfilter= web3.eth.filter(options);
console.log(myfilter);

myfilter.get(function (error, log) {
    console.log("get error:", error); 
    console.log("get log:", log); 
});

results in

f {options: Object, implementation: Object, filterId: null, callbacks: Array[0], pollFilters: Array[0]}

get error: Error: could not decode, invalid type on field : not a number or string(…)
get log: undefined

Any hint highly appreciated, thanks.

[altsheets]

P.S.: Also tried this one

var options = {fromBlock: 0, toBlock: 'latest', address: "0xf2cc0eeaaaed313542cb262b0b8c3972425143f0"};

var myfilter = web3.eth.filter(options);

same results.

P.S.: Using v1.2.2.0 ( because chocolatey https://github.com/ethereum/go-ethereum/issues/2086#issuecomment-166142038 ).

[frozeman]

@altsheets filters are not for getting past transactions, this issue doesn't belong here ;)

This is meant as a discussion about adding some. You issue would be better in the http://github.com/ethereum/web3.js/issues repo.

Please file there again, and ideally give the line number in web3.js where the error is thrown.

BTW just tested and works for me:

var options = {address: "0xf2cc0eeaaaed313542cb262b0b8c3972425143f0"};

var myfilter= web3.eth.filter(options);
console.log(myfilter);

myfilter.get(function (error, log) {
    console.log("get error:", error); 
    console.log("get log:", log); 
});

VM176:5 Filter {requestManager: RequestManager, options: Object, implementation: Object, filterId: null, callbacks: Array[0]…}
Filter {requestManager: RequestManager, options: Object, implementation: Object, filterId: null, callbacks: Array[0]…}
VM176:8 get error: null
VM176:9 get log: []

[altsheets]

Thank you very much for your answer.

filters are not for getting past transactions

Ah.
That explains it why all my such code attempts were futile :-)

Perhaps the wiki could be improved to state that fact "not past".

So then the existing API is telling us really only one thing:
getTransactionCount ... not more.

Sigh.

I guess the situation forces a lot of ETH developers to be iterating over the whole chain, building own databases of those results, etc. (same seems to apply to my _uncles rewards question_ https://github.com/ethereum/go-ethereum/issues/2088#issuecomment-168728900 ?).

---

Idea:

Is there perhaps a ready made (accessing geth, preferably in Python) tool which does all that (iterating the whole chain, looking at each block, storing into a private database), which I could use to simply adapt to my needs?

(I still think getTransactions(address) should be part of the base system. But if not, then) we need to find our workarounds to correct their decisions - Just because the ETH base code architects have decided to not support such questions ... doesn't mean that they exist, and want to be examined. So I guess others have already partly solved this task, right?

Where is such a blockchain-iterator-dump-results-into-database project? Thanks.

---

You issue would be better in the http://github.com/ethereum/web3.js/issues repo.

Thank you very much for that hint. Will be looking there. (Now that I know that I won't get past transaction with that code anyways, it doesn't really matter to fix it, though). It is not always quite clear where to ask which question, and where to make which suggestion. Sorry for polluting your space here :)

So ... back on topic:

[altsheets]

+1 add getTransactions(address) to API.

[frozeman]

To be correct, filters, don't give your transactions at all, expect for filter('pending'), which gives you pending transactions.

If you use ethereum for smart contracts, than tx doesn't matter so much, as they only tell the input person of an action on the blockchain, not what happens then.

But i definitely see a use case for getting past transactions of an account too. Though as @obscuren described, this is not without overhead in space in the node, as thinks need then to be index properly and we are currently looking more for where to trim ;)

[altsheets]

Thanks.

Just an idea: Perhaps ... space saving could be to store only the block _height_ of all the blocks that contain transactions to/from an account. The block height needs much less space than the transaction id.

If I know the blocks in which to look, it is fast to collect all the transaction ids for a certain account.

And then it is easy and straight forward to allow 'since block' and 'to block' parameters.

[tgerring]

What is being described is database indexes which is a non-trivial topic and why we're not likely to get a solution to aggregates.

Although storing an index to a particular block of data will be faster than a full scan of the database, this is not a complete solution without also considering demands for covering indexes or how that might be implemented in the client. This can be a large and complex topic and would detract from other important work going on.

[altsheets]

would detract from other important work going on

I get that. Focus on the most important and innovative topics, for sure.

On the other hand, this is the best funded team out there, so there should be time to also solve side issues.

And for some users it is _not_ even a side issue that we want to see _tx of an addr_.

But ... I see no hope for us here. So:

---

What about the other idea (https://github.com/ethereum/go-ethereum/issues/1897#issuecomment-169354056) then - create a new repo which is intended to solve all "for this question you have to iterate over all blocks" questions?

In that framework repo, all necessary nuts and bolts would already be there (e.g. connect to provider, connect to local database, blocks iterator, store results per block, update-when-new-block-is-mined, etc) - and (provided I know how to code) ... I can fork it and simply plug-in my questions, like:

• "in which blocks were all transactions from my address", or
• "in which blocks were all transactions from all addresses", or
• "what is the total supply, i.e. how many uncles were there ever mined" (see https://github.com/ethereum/go-ethereum/issues/2088 ), or
• "what was the difficulty evolution since block 1",
• etc.

Then we who are interested in questions that the ethereum team finds uninteresting, would be helped. We would simply generate the answers ourselves.

What do you think?

[frozeman]

Write it up :) It should be fairly easy in any language. You have web3.eth.getBlock(1234, true), which includes even all transactions and there you go..

Ethereum is a smart contract based blockchain, we might in the future even drop normal accounts completely and make even them smart contracts, then you could fire logs for you account and have them listed, for example. Read here https://blog.ethereum.org/2015/12/24/understanding-serenity-part-i-abstraction/

[Joter271]

+1 for this command too.

[bricky]

+1 add getTransactions(address) to API.

[obscuren]

I lack the understanding for this feature request.

Why not use the system as is? Is there a particular "feature" you're missing?

[wildbunny]

@obscuren Try getting a list of transactions on a particular account without having to spend a single Ether, then you will see the need.

[obscuren]

@wildbunny and that's exactly where you go wrong! You assume you'll get something for free, well, I hate to break it to you; you ain't ;-)

Ethereum provides you with the means and tools to build whatever you like and you're "complaining" (do not take this word too literal and offensive) feature X doesn't exist, well, how about you go and build it (not for free of course, no) :-D

I can maybe help, though:

/*
calldata 0 .. 31: address
calldata 31 .. calldatasize: tx input
*/

// create a log3 with even "UserTx" and 2 additional fields to filter on (from & to).
log3("UserTx", tx.from, calldata[:31]);
delegate(calldata[:31], tx.value, input);

The assembler equivalent (if that's your thing) would be:

;;;;; prepare calling options
PUSH1 0         ; caller starts at the zero'd index
CALLDATALOAD        ; push address to stack

;;;;; create a log event {
DUP1            ; dup address for log stack[-1] == TO
CALLER          ; push the caller to the stack
PUSH1 0         ; the log's data offset (0) (i.e nothing)
PUSH1 0         ; the log's data size (0)
PUSH6 0x557365725478    ; push "UserTx" to stack
LOG3                    ; create a log
;;;;; }

;; set memory 0 .. CALLDATASIZE to memory for call input
CALLDATASIZE        ; set the end of read for CALLDATALOAD
PUSH1 32        ; set the calldata offset
PUSH1 0         ; start of memory write for call data
CALLDATACOPY        ; copy call data 32 .. CALLDATASIZE

;;;;; prepare delegate call {
;; output (0), txs don't have output
PUSH1 0                 ; out offset
PUSH1 0                 ; out size

;; push the memory size to read for the delegate call input
PUSH1 32                ; subtract 32 for the address field
CALLDATASIZE            ; load input size
SUB                     ; CALLDATASIZE - 32

;; push the memory offset
PUSH1 0

;; push the recipient
PUSH1 0         ; caller starts at the zero'd index
CALLDATALOAD        ; push address to stack

;; push gas amount to the stack
PUSH 20000              ; push 2k to subtract the gas
GAS                     ; push current gas
SUB                     ; leave actual gas input on stack

DELEGATECALL            ; perform the call
;;;;; }

Warning: I did not test the above assembler but it should give you an idea on how to solve the issue.

Now all you got to do is sit tight, wait for homestead, deploy the above code, smile and pay :-D

[wildbunny]

@obscuren What you are saying is equivalent to asking every single user to pay to receive a transaction. Would that be acceptable?

[obscuren]

Receive!? I doubt you actual read the above code.

This is about sending and the sender pays, not the receiver. What makes you think the receiver pays the price? :-|

[wildbunny]

Maybe this is my lack of understanding on how ethereum works - I thought scripts needed to pay a gas price when they run? You're saying the sender pays the script's gas price?

[obscuren]

@wildbunny aye :)

[wildbunny]

@obscuren Then the only argument would be that services will compete to offer their users the lowest prices, which means there is still a genuine need for this functionality, because the alternative is to parse the blockchain manually.

[karalabe]

@wildbunny Just to show you the complexity:

• A fast synced client does not have historical data about past transactions that were received by an account, only the logs, so if a client fast syncs, it cannot ever figure out what happened in the past with plain accounts.
• A state pruned client deletes its own historical data, so again, it cannot figure out what happened in the past with non-contract accounts.
• Light clients don't have any data whatsoever, so they need to pull everything from the network. However they can only query consensus and generally available data (past intermediate history is not generally available), so to get past transactions going in to an account, they would need to download the entire history to trace it.

Ethereum was designed to be a platform for other applications to build upon. Unless you're writing a wallet, applications don't care about historical data, only the current state of the system. Even wallets only care for analysis/retrospection purposes. Similarly to block explorers, various analysis tools are not in the scope of the main Ethereum project.

The purpose of the project is enabling people to create decentralized application, and Ethereum revolves around these applications/contracts. Plain user accounts are simply a means of interaction. If you are abusing the platform's design (by creating software that revolves around non-contracts, or around contracts not under your control), then be prepared to pay the development/maintenance price of gathering the data you need.

[wildbunny]

@karalabe If you wanted contracts to be at the core of the system, you shouldn't have enabled the direct transfer of eth. To call the design of a system which uses that capability 'abuse' is frankly absurd, especially when such a design is actually cheaper for users!

[frozeman]

@wildbunny you might like this then: https://blog.ethereum.org/2015/12/24/understanding-serenity-part-i-abstraction/
"Currently, there are two types of accounts in Ethereum: externally owned accounts, controlled by a private key, and contracts, controlled by code. For externally owned accounts, we specify a particular digital signature algorithm (secp256k1 ECDSA) and a particular sequence number (aka. nonce) scheme, where every transaction must include a sequence number one higher than the previous, in order to prevent replay attacks. The primary change that we will make in order to increase abstraction is this: rather than having these two distinct types of accounts, we will now have only one – contracts. There is also a special “entry point” account, 0x0000000000000000000000000000000000000000, that anyone can send from by sending a transaction. Hence, instead of the signature+nonce verification logic of accounts being in the protocol, it is now up to the user to put this into a contract that will be securing their own account."

[karalabe]

It's also cheaper for you to give a 1 dollar bill to your friend directly than to do a wire transfer but you're not accusing your bank of not knowing about the transaction ;)

[obscuren]

@wildbunny not only do we want it, it is. We have provided you with examples how to solve your issue and how you can build this feature yourself (as intended) but you seem not be satisfied with the solutions, answers and rational. We are sorry for this and perhaps you could try one of the following:

1. Go back to Bitcoin
2. Fork and add the functionality and start a new ethereum client
3. Find another blockchain that suits your needs.

[wildbunny]

@obscuren Clearly it currently isn't, otherwise we wouldn't even be having this conversation. If the link which @frozeman posted comes to fruition, this entire discussion is irrelevant anyway since direct transfer will be disabled.

It is a shame that you cannot see the business model of providing customers the lowest cost service as being relevant, but again, largely irrelevant in the face of your future plans for ethereum.

[biafra23]

+1 for eth_listTransactions. Mist would also benefit from having that.

NOTE Accounts can't display incoming transactions, but hold and send ether. To see incoming transactions create a wallet contract to store ether.

Even with a wallet contract Mist would not be able to list adress to address transactions.

[karalabe]

@wildbunny Our problem here is that you feel entitled for us to solve your business model. Nobody is stopping you to implement the functionality yourself and to use it as you see fit. The information is there. We've tried to tell you a lot of times why it is prohibitively expensive resource and complexity wise for us to include it in the client. You make certain assumptions that do not hold for the entire ecosystem, and we need to cater for everyone, not for a single particular use case.

[wildbunny]

@karalabe I do not feel at all entitled. I am merely stating facts. Indeed, this is not just 'my' business model, this is the business model of every single exchange which lists ETH currently.

[karalabe]

Clearly poloniex and kraken cannot trade ETH currently because they lack this feature. Oh wait, they're doing just fine since we launched and didn't even mention that they would need it.

[obscuren]

Perhaps you should read up on ethereum again, it clearly is. This conversation has nothing to do with the core of ethereum it self but rather a feature which isn't implemented.

Customers? Ethereum doesn't have customers (anymore), it has users of all kind. We provide these users with a platform as I've tried to point out only so many times. The platform provides you with tools to build anything which we didn't.

I'm sorry but I still fail to understand the specific need for this (apart from getting it for free)

[obscuren]

@wildbunny exchanges can use contracts to solve this problem. They aren't informed well enough or don't need it. I've already gave several pointers how you can solve this issue (and even move the costs to the consumers).

[wildbunny]

@obscuren If contracts were at the core of ethereum, I wouldn't be requesting a way to get a list of transactions sent to an account, because it would be impossible, since there would be no accounts, only contracts.

Furthermore, the people who build applications on ethereum wish to attract customers (otherwise what is the point?), therefore competing applications will seek to reduce fees. I cannot explain that any more clearly.

[Joter271]

Right now my process is inserting all ethereum transactions in database. Then I will be able to get all data I need, because this client is kinda useless in comparison to Bitcoin :/

But I get the picture why such commands are not supported. Probably indexing related?

[wildbunny]

@karalabe Oh wait, yes, they're parsing the blockchain, just like I am, which is a massive waste of resources.

[biafra23]

I for one would not need to get the transactions for any account. For me it would be enough to get the incoming and outgoing transactions for any account the geth client manages.

[wildbunny]

@biafra23 Agreed, this would be sufficient.

[karalabe]

And what would happen when you add an account to geth?

[bricky]

@biafra23 ditto. I'd just need the equivalent of bitcoin's listransactions.

[obscuren]

@wildbunny sigh you can! How many times do I have to tell you that you can? Are you going to keep ignoring all of the information that is given to you that doesn't help in strengthen your argument?

"The people" seem to be doing pretty fine without this feature except a very specific niche (and even then you can solve the damn issue).

Stop ignoring the information given freely (hey, lucky you!) to you and start thinking about how you can actual solve your issue.

[wildbunny]

@obscuren System A, user pays minimum transaction fee to send a deposit. System B, user pays minimum transaction fee + gas fee to send a deposit. Which is better for the user?

[obscuren]

@wildbunny @joter85 @bricky use a contract and a LOG. No need to parse the block chain. Stop ignoring the suggestions above that fix your specific issue.

[biafra23]

@obscuren can I prevent people sending ether to accounts directly? If so, how?

[obscuren]

@wildbunny that depends on the use case. Again, we provide a generic system that benefits all, not a selective group of people.

[obscuren]

@biafra23 assuming you are the recipient (or your system) you control where it gets send to.

If out going you control where to being send from anyway

[obscuren]

@biafra23 I'm happy to help you solve your issue if there's one but I guess it's better to use gitter

[bricky]

@obscuren using contracts in place of accounts only fixes the issue if it's a viable option. It's often not. (e.g. apparently shapeshift can't accept funds sent from wallets: https://www.reddit.com/r/ethereum/comments/45x39g/lost_all_my_ethereum_trying_to_use_shapeshiftio/)

I'm with you 100% btw: it's obvious that I'm not using the system as it was intended to be - it doesn't make my problem go away though :(

btw: I'm currently implementing the second database approach, as it has other benefits which I'll use. So consider the issue resolved from my point of view (but the feature request remains)

[wildbunny]

@obscuren Your condescending tone is unconstructive; I have already 'solved' the problem, but my solution was inelegant and wasteful therefore I put in the original support request with the hope that it would actually benefit all users of the system who are not involved with contracts.

[obscuren]

@bricky I'll take a stab at it and see if I can somehow help.

@wildbunny you're misreading. Read up and you'll see I'm actually trying to help you. I spent 30 minutes trying to write something up for you (though you didn't even mentioned it once). I'll continue to help you out if you require it, but not if you blatantly keep ignoring it :-/

[VoR0220]

well this is getting nasty rather quickly. @wildbunny how do you suggest the problem be solved then? Because I can tell you this command is not buildable under the current circumstances. You could have deduced that from just reading this entire thread. Of course we would all love this feature. And if you think everyone here is so wrong, prove us wrong.

[wildbunny]

@VoR0220 Perhaps it would be possible to implement such a feature for full nodes only, via a restricted API call?

[VoR0220]

potentially. But I'm not here to make that happen.

[wildbunny]

@VoR0220 Then I misunderstand your question

[frozeman]

@wildbunny @bricky there is one big flaw with the eth_listTransactions, you wouldn't see "transactions" (message calls) from one _contract_ to an account address, even if you have all transactions listed. So you would miss many tx out.

Exactly for this reason we introduces "logs" in contracts.

[bricky]

@frozeman understood.
@karalabe @obscuren (and indeed everyone else!) thanks.

[altsheets]

Interesting debate, thanks. As a side effect, we are learning a lot about ethereum. And you about your users' expectations.

I am back because I am reading that the parse-the-whole-chain approach has been implemented already:

I'm currently implementing the second database approach (@bricky)

and

I have already 'solved' the problem, but my solution was inelegant and wasteful (@wildbunny)

Would one of you publish the code? Of course, I could be writing the 9th version of the same thing, but I'd prefer to build on top of what is already functioning. See my blockchain-iterator-dump-results-into-database project idea in https://github.com/ethereum/go-ethereum/issues/1897#issuecomment-169354056 and https://github.com/ethereum/go-ethereum/issues/1897#issuecomment-170055698 up there.

Of course, only for those of us who are _abusing_ ETH as a currency ... ;-)

Thanks.

[frozeman]

@altsheets remember, you can't track incoming funds from contracts this way....

[altsheets]

@frozeman Yes, I read that. Good to know. Has to be added then. Or the user has to be notified that he is seeing incomplete data.

[wildbunny]

@frozeman Is there yet a way to prove that funds from a contract actually entered any given account? None of the blockchain explorers support this the last time I checked, making it impossible to support in my application.

[obscuren]

Yes, by using logs.

[wildbunny]

@obscuren Aren't logs only good for contemporary transactions? We need to be able to query historical transactions as well for robustness sake.

[bricky]

@altsheets I'm using @obscuren's approach from your other issue here and dumping the data into an external db. Certainly not an elegant solution.

[obscuren]

LOGs can be queried/searched (historical) and watched.

[wildbunny]

@obscuren My point is, if I suffer a power outage, or some other hardware loss, which includes the logs, can I query the chain to get the data back? That is what's required for full robustness.

[obscuren]

Yes. LOGs can be fetched at any given time.

var filter = contract.SomeEvent({param: "one"}, function(error, event) {
    // notifications would happen in here
})

filter.get(function(error, logs) {
     // historical data
});

[wildbunny]

@obscuren Even if my HD blows up? ... and I get a replacement

[karalabe]

@wildbunny Please stop trolling.

[wildbunny]

@karalabe This is a genuine question

[obscuren]

@wildbunny yes, even if your HD blows up. They are stored in the receipts of the blockchain. They are part of the consensus rules.

[obscuren]

That's why it's costing you Gas.

[wildbunny]

@obscuren Perfect, thank you

[frozeman]

@wildbunny there is a fromBlock: 0 option, which gives you passed logs you may have missed ;) Please read the RPC docs or web3.js docs

[Gustav-Simonsson]

@altsheets As I also enjoy abusing Ethereum as a currency, I started on a quick & dirty impl here: https://github.com/Gustav-Simonsson/go-ethereum/tree/list_transactions

It uses sqlite to index txs on both sender and recipient. It's WIP and pretty hacky - have to rewrite it quite a bit before it'll be ready for others to use.

Preliminary benchmarks:

• sqlite DB size after indexing all mainnet txs: 600MB
• JSON-RPC call listTransactions on address with 60 txs: <1ms. On address with 25k txs: <2s.
• Indexing from scratch: 12 hours on a ivy bridge i7 machine.

All of these metrics can probably be improved a lot, as the current hack is just a first stab and not optimised.

TODOs:

• Index during normal sync instead of traversing blockchain from HEAD to genesis.
• Handle forks & reorgs (by integrating into the code that already handles this for the leveldb instance used for blocks)
• Include txs from tx pool in API return
• Return more than the tx hash, perhaps configurable in the API call
• cmd line flags to make the call & required indexing optional, and perhaps also configure from which block to begin indexing (to save time if one knows how far back the txs one is interested in go)

[altsheets]

wow @Gustav-Simonsson you are a star!

I gather it is mainly in eth/addr_txs.go ?

Indexing from scratch: 12 hours

Oops. But 12 * 3600 / 1018621 = 0.04241 is not too bad, actually (there is the 42 again! :-) )

not optimised

Which are the bottlenecks?

Could multithreading help? Query 500 blocks at a time? This supersimple question is not time-ordered.

todo: Handle forks & reorgs

yes, and that is actually one reason why I had refrained from just doing it myself.

Fantastic work!

Rawr

[fairglu]

This kind of information is also required for 3rd party auditing, currently ethereum is hard and error-prone to audit, even for basic transfers and coin generation. There is no point in having sophisticated gas and difficulty mechanics if the monetary part is not sound and auditable (not just for isolated accounts, but for all blockchain accounts and transactions). This is also needs to be auditable efficiently and in real-time, so that fool play can be detected and acted upon quickly.

While the core ethereum team is out there to build something, and spurious transactions are more likely to be bugs than scams, this cannot be said or assumed for clones and private chains. Since the code is very complex, it is very hard to justify code and consensus can be trusted, even with multiple implementations (actually multiple implementations for a private chain will only make the 3rd party auditing more expensive).

In summary, this is not a care-bear world, without standardized auditing (of which tx listing is a key part), things lean more on the trust side than on the trust-less one.

[dmxhZGp1c2hh]

+1

Does anyone thinks that not knowing when, from who and how much you received is a cool feature?
Messages like Accounts can't display incoming transactions. does not make any sense. They can track amount, you can use them to send and receive ether, but somehow you can't track this.
Think about this from average user point of view. He needs to go to 3rd party site to check who send him ether.

As I understood the only option is to create contract. So I need to pay to receive ether? Really?

[altsheets]

:-)

[obscuren]

I can only reiterate about what has already been said which makes this feature questionable and difficult to implement correct.

listTransactions will not work for light-clients. light-clients and full clients need to support the same set of functionality. Even if full nodes would index _all_ transactions and _all_ calls it still wouldn't be feasible since there's no way to actually verify that the transactions the node is giving you, are valid and correct. The only part which can be validated are the transaction, but since we're also dealing with contracts (which don't send transactions but invoke an EVM opcodes) we can't actually verify that value transfers that occurred during the execution of the contract are valid (a full node might be malicious).

As I understood the only option is to create contract. So I need to pay to receive ether? Really?

Pay to receive ether? Of course not. The sender is paying, not the receiver. The only part that would _cost_ you ether is the deployment of the contract, which costs you something like 0.001 cents, if not less. This can't be an argument not to use contracts.

[obscuren]

Please note that I'm not arguing that this isn't a good/nice/cool/epic feature. I'm arguing that, while seemingly easy to implement at first sight, is actually quite difficult to do correct, if at all.

This isn't bitcoin where the only thing you have are value transfers. Our core is much more complicated that a feature like this becomes difficult to implement or impossible to implement correctly at all.

[Joter271]

@obscuren well that explains it then. last paragraph. thanks

[dmxhZGp1c2hh]

@obscuren, understood.

[altsheets]

just to clarify, this approach https://github.com/ethereum/go-ethereum/issues/1897#issuecomment-185276610 is futile then?

---

EDIT: and with _futile_ I meant, that what Gustav-Simonsson seems to have implemented _already_ ... would lead to incorrect results? (Because if not, then just make it optional, or ship it as a standalone code/extension - and everyone's happy?). Working, correct code is the best argument possible. But some of the more recent comments since then would only be understandable, if his approach is ... simply wrong. I am confused. And that is why I asked for clarification - not for restating arguments from before his post, or inventing new ones. If there is already a solution in code, to answer the OP's question - then why argue whether such a solution should exist at all, or not? Makes no sense to me. So: Is his approach only slow/mem-hungry - or is it leading to incorrect results?

[karalabe]

@altsheets Concurrent fast sync takes 11 mins on my machine, uses 2.2GB. The proposed solution needed 12h (!!), used 0.6GB extra data. Guess the feasibility ;)

On a serious note, a pruned node does _not_ have the data you want. Isn't stored anywhere, so to get it, you need to crunch through and replay the entire Ethereum history, so fast sync is out of the question. Pruning is out of the question. It's not part of the consensus rules, so light clients are out of the question.

The only place where this can be implemented generically is for archive nodes, but those will be very rare. Average users won't have the data needed to index such transactions.

This is analogous to the real world: if I look at 5 dollars in my wallet, I don't expect the universe to tell me all the people that ever had those banknotes. No do I want a list with all the cash I've spent and received throughout my lifetime. I know how much I have now and I don't care who gave me how much when. Note, this is for cash. Now when I open a bank account and get a credit card, I expect to see all of my transaction history, but I don't get that for free, I pay my yearly maintenance fees to access that data. It's the same with Ethereum: the Ether is a plain account is cash: you know how much you have, you see the currently pending changes, but no history; whereas a contract can act as a bank account, just you get to choose all the rules of it. The more featured you wany, the more you need to pay the bank to get it, or in this case, you need to support the network with tx fees to maintain extra data on your behalf.

Just as a closing remark, the same way as it would be awesome to have automatic past tx history for plain accounts, it would be awesome to have tx data for my personal cash expenditure... But it's not something most pepole need, and it's not sometging that bothers you in real life. Simply assume that a plain account is cold hard cash, whereas a contract is a bank account working accordong to your rules.

[karalabe]

Typed it on my phone, sorry for the many typos.

[dmxhZGp1c2hh]

@altsheets, it really make sense. The problem is that if you have bitcoin background it's kinda hard to implement same features the same way. For example my users have virtual balances and to transfer some amount to balance they use their own bitcoin address. It's easy to track, because each address belongs to some user. In ethereum I don't see same ways.

So creating account for each user is not an option. OK. I can create contract that will track transactions. But this way all my users will send ether to same contract address and how to distinguish who is who? Now I need to ask users to provide some additional data. Do I ask them to provide address, so I can track them later or to provide extra data in each transaction? Well if they use some 3rd party service mby they don't know exactly from which address ether will be send or don't have permissions to add extra data in transactions.

In bitcoin (or similar altcoin) you just create new address for every event you want to track and check later for address in transactions. Super easy and flexible.

[dmxhZGp1c2hh]

@altsheets, about implementation. listtransactions in bitcoin only returns related transaction for your wallet. So you don't need to save data about all address <-> transaction relations.

What about this logic. We have array of our account addresses (let's call it addresses). When node gets new transactions, it checks if sender or receiver is in our addresses array. If yes, we put txid of transaction in some cache table. Now we can use already implemented API to get info about transactions.

[obscuren]

@altsheets as I said, I'm not arguing the usefulness of the feature I'm arguing the solutions that have either been suggested or created. If you read my response above you'll find the answer as to why this doesn't work, i.e. The results can not be guaranteed, for anything but full nodes. Since full nodes will not be dominating our network in the future makes this feature request extremely difficult if not at all possible to implement correctly (i.e. The results can never be verified).

[dmxhZGp1c2hh]

The results can not be guaranteed, for anything but full nodes.

Is that a problem? Full node will have some additional API for tracking, makes sense.

[obscuren]

That would mean some dapps won't work with chains that were created with the fast sync option.

Mist will be our primary application to serve dapps and mist always uses the fast sync and therefore it will be impossible to support such a feature.

This feature won't work with light clients either and I expect light client to eventually dominate the network mostly.

I'm not going to make a promise but we could make such a feature available under, say, admin. This means you are free to use the feature but isn't part of standard web3 and will never be either. but please note that no dapp, including the wallet dapp or mist, will ever be able to make use of this. This would purely be considered a _administrative utility_.

[obscuren]

For example an exchange could use this feature if it wants to be a full/archive node indefinitely.

[fairglu]

I do not see why nodes that do not perform full validation are considered in this discussion.

Clients that have to trust other nodes will be intrinsically unsafe for services and dapps, and thus unusable, especially if/when full archive nodes are rare and sybil attacks have become easy. Any use case with real world consequences will need accounting and auditing capability, which means full history for a reasonable time frame (at least weeks, years in many cases). This means running full validating nodes (with data that can be trusted through blockchain trustless mechanisms and consensus) or... rely on "trusted" payment processors and web services to run those nodes for you.

Just as for other blockchains, even a light client is vastly more resource-intensive than relying on a web service ala blockchain.info or coinbase. And if a light client cannot provide full, validated information, it would be both less practical and less safe. There would be zero incentive in running your own light client if it cannot match the information provided by explorers like ether camp's full transparency.

So the problem may not be an easy one to solve, but it is a necessary cost to be paid for a light client to be useful in a wild, hostile world.

[obscuren]

@fairglu all of the current functionality that is supported by web3 (blocks, receipts, logs, etc) can be verified by a light client. We must consider light clients in this discussion because:

1. light clients shouldn't be any less than archive nodes in terms of web3 functionality;
2. light client and nodes synced using --fast should not be considered sub-sets of web3.

Transactions that come from the call eth_listTransactions can not be verified without having the full chain at your disposal and even then it becomes incredible impractical to validate the transactions as you'd need to parse the entire chain and figure which transactions are valid and which aren't.

The problem is however solvable by using the Ethereum system as intended as I've only pointed out so many times. If you're looking for a simple value transfer network with said functionality your best option is the Bitcoin network.

In any case DApps that rely on transaction tracking are very likely doing it wrong and I'm sure there are more than enough people willing to help solve whatever issue it you're having, including myself. You're always welcome to join our gitter to discuss your issue you're having :-)

[obscuren]

Locking this issue but leaving it open.

We're considering a suitable implementation for this feature.

No ETA.