Hledger: Allow more than 2 postings in a csv-imported transaction

[MatthiasKauer]

Hi,
This is about the implementation for the change we had discussed in https://github.com/simonmichael/hledger/issues/624

I have written something to get

matthias@xubfiend:~/dvl/hledger$ stack exec hledger -- -f ~/hledger_test.csv print
2017/01/02 Ate something
    assets:bank:checking          USD100
    food                         USD-100
    FoodTax                      USD0.87
    GeneralTax                   USD1.23

out of

date,curr,acc1,amt1,acc2,amt2,desc,acct3,amt3,acct4,amt4
2017-01-02,USD,BofA,100,food,-100,"Ate something",FoodTax,0.87,GeneralTax,1.23

and

account1 assets:bank:checking

date %1
amount %4
account2 %5
account3 %8
description %7
amount3 %9
amount4 %11
account4 %10

date-format %Y-%m-%d

currency %2

skip 1

I am rather inexperienced with Haskell and the solution is quite rough at the moment.
Hopefully, you have some feedback to get me into the right direction.
There were also some interface questions that came up and should be discussed.

Best regards,
Matthias

[simonmichael]

This was previously discussed in #267 (and recently #630). Let's continue here. More expressive CSV conversion is something we definitely want. Personally, I need it to better represent transactions containing fees, and transactions involving currency conversions. As I said on #630 I feel the need to review and clarify the user-facing design here, as a precursor to clarifying the docs, the implementation, and adding the new features. There are other paths we could go down. Let me share some thoughts here and see what you think.

What's the goal of this issue ? allow more than two postings in transactions converted from CSV. There are other improvements in expressiveness of CSV conversion that may come, but let's focus on more than two postings.

What are some success criteria ? Preserve existing capabilities. Robustness, generality, intuitiveness for the user. As cheap as possible to develop and to maintain. Solve real problems, avoid overengineering.

Plan 1: numbered amount fields. We already have amount, account1, and account2 fields. Generalise these, allowing account1/2/3/4 and amount1/2/3/4. #630 is prototyping this.

How can this work in detail ? I am guessing it could be something like this: We allow up to four amountN declarations. Each one generates a posting. Each one should have a corresponding accountN declaration.

Maybe an additional posting is added automatically when needed to balance the transaction, with a default account name. Maybe an additional accountN is allowed (1 greater than the number of amountNs) to customize this posting's account.

Maybe if no amount fields are defined, we generate postingless transactions instead of an error.

amount still works as before, for backwards compatibility. Maybe amount is considered equivalent to amount1.

Unless we replace them with some other mechanism (eg, conditional rules which can check specific fields), amount-in/amount-out still need to be supported as alternatives to amount to handle the case of separate debit/credit fields in the CSV. By consistency we would probably need to allow amountN-in/amountN-out as an alternative for each amountN also.

Error checking/rules validation gets more complicated. Some things to check: each amountN has a corresponding accountN and vice versa (except for the "plus 1" case mentioned above). The Ns are consecutive and start at 1. The usual "amount OR (amount-in AND amount-out)" existence check, for each N.

Error checks are further complicated by conditional blocks, which may contain account/amount declarations. Ideally we'd want to report any malformed/incomplete rules immediately at startup, and not at runtime when processing individual CSV records. This sounds hard to do in full correctness. How do we currently handle this ?

Plan 2: transaction templates in rules file. There is an old wish to allow full journal entry templates in the rules file, allowing more freedom in generating custom transactions. If implemented, these would naturally provide more control over the postings. Transaction templates might avoid many of plan 1's complications.

The current system of declaring just the fields needed can be quite concise and presumably would remain, with transaction templates as an optional power feature used mainly in conditional blocks. A top-level default transaction template could also be used.

It's possible transaction templates could turn out to be a more useful UI than the current system, allowing a simplification of the rules file format.

[simonmichael]

Posting order. hledger 1.4 made this more consistent: the account1/amount posting comes first, and the docs say "It's conventional and recommended to use account1 for the account whose CSV we are reading". I have been enjoying this scheme - it's usually FROM ASSET, which I don't need to touch, then TO EXPENSE, which I sometimes copy/paste to split up into more detailed expenses.

In plan 1, account1/amount1 would be first, with the others following in numeric order, and any auto-balancing posting coming last. Contrary to the above, I think this would tend to put the base account's posting last:

# Date,Description,Payment,Transaction Fee
# 2017/11/6,foo,$100,$2
2017/11/6 foo
  expenses:misc        $100
  expenses:bank fees     $2
  assets:checking     $-102

If you want it to be first, you'd need to declare all amounts/postings explicitly, and this might not be so easy, eg if there are multiple amount fields but their total is not available. (For two amount fields, you could do it by using more postings, eg like:)

2017/11/6 foo
  assets:checking     $-100
  assets:checking       $-2
  expenses:misc        $100
  expenses:bank fees     $2

[simonmichael]

Field declarations (the current rules system) can be very concise and easy looking in simple cases. In not-so-simple cases they can get more complicated, interact with each other, and feel like programming. Providing limited freedom, they tend to generate valid journal entries.

Transaction templates (if implemented) would reuse knowledge of the journal syntax, with slight additions (interpolation, conditionals). They might be more obvious to users and require less mental modelling than field declarations. They will also make it easy to generate invalid journal entries. These would cause a parse error during the subsequent journal parsing stage, which might or might not be easy to relate to a position in the CSV or rules files. Warning about malformed transaction templates when parsing the rules file would be better, but might not be possible in all cases.

[simonmichael]

Just for completeness:

Plan 3: haskell code. Rather than keep making a more expressive rules DSL, we could somehow allow arbitrarily powerful CSV conversions using haskell code, as suggested by joeyh. How to do this reliably is not clear, and using it is not going to be pleasant except for programmers.

Plan 4: an easier programming language. Instead of haskell, configure rules with some kinder, gentler, more portable programming language (eg pandoc embeds lua). Again, appealing only to programmers, possibly easier to implement.

Plan 5: external converters. Instead of or as an alternative to the built-in CSV converter, accept some easy standard input format (eg pandoc accepts json or native pandoc serialization format) and let users write their own conversion scripts any way they like. Again, programmers only. Ties in with a larger goal of making it easy to quickly add support for new data formats in general (not just CSV).

[simonmichael]

Plan 6: transaction templates in journal. There is a desire for a journal directive to define standard transactions, which could be used for easier data entry, stronger error checking, etc. Perhaps with some extra annotations advising where to interpolate data, the CSV reader could use these too.

Plan 7: transaction templates from past transactions. Rather than explicitly defined standard transactions, we could look for a past journal transaction that's similar, as add and iadd do, and use that as at least a partial template. In theory, less need for explicit configuration - just import once, fix up transactions manually (maybe add some tag annotations), and next import will give better conversions. But, less reliable and harder to detect and debug problems.

[MatthiasKauer]

Wow, you have really put a lot of thought into this.
Allow me to comment on your various plans.

1. Numbered fields: This is what I attempted to implement.
   I found it tricky to formulate what amounts we should allow to be missing since we can infer them.
   
   
   
   • Validating the rules file also becomes more complicated but I had not gotten that far.
   
   
   • You also raise another issue: the amounts in the file may not be balanced.
     
     That could be accepted and left to manual cleanup maybe.
   
   
   • amount1 should be an alias for amount in my opinion.
   
   
   • amount1 & amount2 can fully replace amount-in & amount-out. Maintaining both
     
     functionalities feels tricky.
   
   
2. Transaction templates: What do you mean by that? Something like
   %1 %2 %3 %4 %5 %6
3. Haskell code: Is Haskell embeddable? Or would I then need to install a compiler before importing csv files?
4. Another programming language: A good option in my opinion. That would also let me do inline calculations and solve the cases where bank reports "buy for 100$ of which $2 are commission" but I need "98$ trade plus $2 commission" in my ledger.
   hs-duktape and hsLua look like the most promising options to me.
   Care should be taken that we don't make the build too complicated or lose platforms like Windows over this.
5. Json: would also work for me
6. Templates: Ok, I understand now from the link you posted. This feels complicated. Can this even be done without machine learning to get the user's intent?

Finally, I'd like to say that I doubt there are many users who appreciate the command line and text files but aren't familiar with at least Python or Javascript.

If I had to vote right now, I'd see how well duktape integrates into the build toolchain.
That might actually be easier than getting amount3, amount4 to behave correctly.
Then we'd still have to think about an interface: Would hledger pass individual rows to a Javascript function? Or would we pass the whole CSV file and expect JSON representing a list of transactions in return?

[adept]

Out of curiosity: do you consider "csv import" to be something that is done
once or something that is done repeatedly (for the same file)?

I am asking because I run a system where i retain all the original csv
files + import rules + makefile to generate .journal files from these, and
have a master journal that !includes all the generated files and contains
all manually-entered transactions. From time to time I go back and tweak
the rules, which causes all .journal files to be regenerated. I am using
hledger since 2010ish (if not earlier), and I found this approach to be
more robust than "convert once + manually fix all unknowns/conversion
mistakes".

But now it occurs to me that maybe i am an outlier, and typical use case is
"convert once + manually fix"

On Wed, Nov 8, 2017 at 3:37 AM, MatthiasKauer [e-mail]
wrote:

Wow, you have really put a lot of thought into this.
Allow me to comment on your various plans.

1. Numbered fields: This is what I attempted to implement.
   I found it tricky to formulate what amounts we should allow to be
   missing since we can infer them.
   
   
   
   • Validating the rules file also becomes more complicated but I had
     
     not gotten that far.
   
   
   • You also raise another issue: the amounts in the file may not be
     
     balanced.
     
     That could be accepted and left to manual cleanup maybe.
   
   
   • amount1 should be an alias for amount in my opinion.
   
   
   • amount1 & amount2 can fully replace amount-in & amount-out.
     
     Maintaining both
     
     functionalities feels tricky.
   
   

2. Transaction templates: What do you mean by that? Something like

   %1 %2
   %3 %4
   %5 %6

3. Haskell code: Is Haskell embeddable? Or would I then need to
   install a compiler before importing csv files?

4. Another programming language: A good option in my opinion. That
   would also let me do inline calculations and solve the cases where bank
   reports "buy for 100$ of which $2 are commission" but I need "98$ trade
   plus $2 commission" in my ledger.
   hs-duktape and hsLua look like the most promising options to me.
   Care should be taken that we don't make the build too complicated or
   lose platforms like Windows over this.
5. Json: would also work for me
6. Templates: Ok, I understand now from the link you posted. This
   feels complicated. Can this even be done without machine learning to get
   the user's intent?

Finally, I'd like to say that I doubt there are many users who appreciate
the command line and text files but aren't familiar with at least Python or
Javascript.

If I had to vote right now, I'd see how well duktape integrates into the
build toolchain.
That might actually be easier than getting amount3, amount4 to behave
correctly.
Then we'd still have to think about an interface: Would hledger pass
individual rows to a Javascript function? Or would we pass the whole CSV
file and expect JSON representing a list of transactions in return?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/simonmichael/hledger/issues/627#issuecomment-342701031,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHNKqao1PyOtnbnzu4d7vQmipTCrBzJks5s0SHggaJpZM4P6PKe
.

--
D. Astapov

[simonmichael]

@adept: interesting question. I like to have one yearly journal file that sequentialises all transactions, so I need to convert and append just the recent CSV transactions. I don't re-do old conversions. (I do occasionally rewrite old transactions in general). We should discuss more elsewhere (mail list ?)

[ghost]

["ghost" == simonmichael]

@MatthiasKauer, thanks for the feedback.

• Numbered fields: This is what I attempted to implement.
I found it tricky to formulate what amounts we should allow to be missing since we can infer them.

I forgot about this. A particular CSV record might be missing any of the amounts. This makes clear that only some errors can be checked at rule parsing time, others must be handled while processing the CSV.

I think we don't need to do any inferring in the CSV converter - just interpolate the available amounts according to the rules. The later journal-finalising stage will do the usual inferring and complain if it can't. (And if that happens, it means the user needs either more complex or simpler rules to handle that CSV, eg conditionals).

  • You also raise another issue: the amounts in the file may not be balanced.

We needn't and shouldn't expect the CSV to include all balanced amounts. Currently it always adds a balancing posting, which is easy with one amount. With multiple amounts, this sounds harder. Maybe we add an amountless final posting so the journal finaliser can balance it for us ? Maybe we call the journal finaliser's inferring/balancing operation a bit sooner, or a simplified version of it which avoids the complexities of balance assertions ? [Yes, contradicting the above here..]

  • amount1 should be an alias for amount in my opinion.

amount implies two postings. It might be useful for amount1 to be different, eg to allow generation of single-entry transactions (using unbalanced postings, sometimes useful eg for timelogs).

  • amount1 & amount2 can fully replace amount-in & amount-out. Maintaining both functionalities feels tricky.

Could you explain how ? I haven't understood how you could replace amount-in/amount-out with amount1/2.

• Transaction templates: What do you mean by that? Something like
%1 %2
%3 %4
%5 %6

You're right to make this more concrete. Yes I suppose that could be one. I was thinking of something like:

fields date,description,payment,fee,comment
entry
%date %description ; %comment
assets:checking
expenses:misc $%payment
expenses:fees $%fee

But this looks less practical than I thought. To set the "expenses:misc" to something more specific, eg in a conditional, you'd have to repeat the whole template. Or, we allow unbound variables in templates which must be redefined, eg:

fields date,description,payment,fee,comment
entry
%date %description ; %comment
assets:checking
%account $%payment
expenses:fees $%fee
if SUPERMARKET
account expenses:food
if MANAGEMENT
account expenses:rent

This starts to feel a bit complicated. How might this look with field declarations ? Perhaps:

fields date,description,amount1,amount2,comment
account3 assets:checking
account2 expenses:fees
if SUPERMARKET
account1 expenses:food
if MANAGEMENT
account1 expenses:rent

Note how the assets:checking amount must be inferred here, and therefore it must be the last posting (account3).

• Haskell code: Is Haskell embeddable? Or would I then need to install a compiler before importing csv files?

Not very. There is stuff like hint (http://hackage.haskell.org/package/hint). I suspect this could only be a neat add-on for power users, not the sole mechanism for converting CSV.

• Another programming language: A good option in my opinion. That would also let me do inline calculations and solve the cases where bank reports "buy for 100$ of which $2 are commission" but I need "98$ trade plus $2 commission" in my ledger.
hs-duktape and hsLua look like the most promising options to me.
Care should be taken that we don't make the build too complicated or lose platforms like Windows over this.

hs-duktape (http://hackage.haskell.org/package/hs-duktape) and hslua (http://hackage.haskell.org/package/hslua) look fun. Yes, breaking cross-platform builds or hurting installability would rule them out. I'd be happy to hear how they work out.

• simonmichael

[MatthiasKauer]

Regarding amount1/amount2 over amount-in/out. From what I saw they are direct replacements.

Consider:

account1 asset:cash
account2 expense:food

If we calculate the missing amount, then

amount-in %1
amount-out %2

and

amount1 %1
amount2 %2

would yield exactly the same thing, no?

Also: I have taken a quick look at hs-duktape for Windows. With a basic appveyor config stack seems to have built it without complaining: https://ci.appveyor.com/project/MatthiasKauer/hs-duktape/build/1.0.2

[simonmichael]

An example will help me. Given

Date,Description,Debit,Credit
2017/11/1,foo,10,0
2017/11/2/bar,0,15

what could the rules look like to produce

2017/11/1 foo
  assets:checking   10
  income:misc

2017/11/2 bar
  assets:checking  -15
  expenses:misc

?

[simonmichael]

Also @MatthiasKauer, what do you feel is the most promising direction now ? Do you want to give the numbered accounts/amounts another shot, or.. ?

[simonmichael]

PS tshirtman points out that if you're going to be programming, there's no special reason to have it embedded in hledger - just write a script, right ?

[tshirtman]

Yes, that's the route i took, because even if the above difficulties are solved, there are always special cases, as in the file i export from bitstamp, where all the amounts/values are positive, and you only know how to use them by the latest field, Sub Type, which indicate if it's buy or sell order…

In the latter case, i have to negate the Amount and Value fields.

I guess you could incorporate condications in the template mentionned above, but it really starts to feel like you want something Turing complete.

So, with that in mind, and considering it's pretty much undebugged, here is an excerpt from my csv file, and the script i'm working on to make that work.

Type,Datetime,Account,Amount,Value,Rate,Fee,Sub Type
Deposit,"Nov. 07, 2017, 12:28 AM",Main Account,6.37178000 ETH,,,,
Market,"Nov. 07, 2017, 12:14 PM",Main Account,10.98587000 LTC,0.09 BTC,0.00792154 BTC,0.00022000 BTC,Sell
Market,"Nov. 07, 2017, 08:51 PM",Main Account,0.31804472 ETH,0.01 BTC,0.04130982 BTC,0.00002000 BTC,Sell
Market,"Nov. 07, 2017, 08:53 PM",Main Account,1.00000000 ETH,0.04 BTC,0.04130982 BTC,0.00006000 BTC,Sell
Market,"Nov. 07, 2017, 08:56 PM",Main Account,0.23202336 ETH,0.01 BTC,0.04130982 BTC,0.00002000 BTC,Sell

#!/usr/bin/env python3.6

from csv import reader
from sys import argv
from time import strptime, strftime
from textwrap import dedent

if not len(argv) > 1:
    print("usage: {} filename.csv".format(argv[0]))
    exit(0)


def main():
    filename = argv[1]

    with open(filename) as f:
        r = reader(f)

        for i, l in enumerate(r):
            if i == 0:
                continue
            Type, Datetime, Account, Amount, Value, Rate, Fee, SubType = l
            date = strptime(Datetime, '%b. %d, %Y, %H:%M %p')
            date = strftime("%Y/%m/%d", date)
            description = f"{Type} {SubType}"

            Account2 = 'market'
            if Amount:
                Amount = Amount.split(' ')
                Amount, Currency_Amount = Amount
                Amount = float(Amount)

            if Value:
                Value = Value.split(' ')
                Value, Currency_Value = Value
                Value = float(Value)

                Account2 = 'market'

                assert (
                    Type in ('Market', 'Limit') and
                    SubType in ('Sell', 'Buy')
                )

                if SubType == 'Sell':
                    # reverte the transaction
                    Amount, Value = -Amount, -Value

            else:
                assert Type in ('Deposit', 'Withdrawal')
                Value = Amount
                Currency_Value = Currency_Amount
                Account2 = 'ext'
                if Type == 'Withdrawal':
                    Account, Account2 = Account2, Account

            if Fee:
                Fee = Fee.split(' ')
                Fee, Currency_Fee = Fee
                Fee = float(Fee)

            else:
                Fee = 0
                Currency_Fee = None

            print(dedent(
                f'''
                {date} {description}
                    {Account} \t\t\t{Amount:f} {Currency_Amount}
                    {Account2} \t\t\t{-Amount:f} {Currency_Amount}
                    bitstamp  \t\t\t{Fee:f} {Currency_Value}''' + (
                f'''
                    {Account2}\t\t\t{Value:f} {Currency_Value}
                    {Account} \t\t\t{-Value-Fee:f} {Currency_Value}
                ''' if SubType in ('Sell', 'Buy') else '')
            ))


if __name__ == '__main__':
    main()

(this is for example only, i'm not sure i'm interpreting the data from bitstamp correctly yet, especially considering the fee field)

this gives me an output of the form of hldeger format

2017/11/07 Deposit 
    Main Account                        6.371780 ETH
    ext                         -6.371780 ETH
    bitstamp                    0.000000 ETH

2017/11/07 Market Sell
    Main Account                        -10.985870 LTC
    market                      10.985870 LTC
    bitstamp                    0.000220 BTC
    market                      -0.090000 BTC
    Main Account                        0.089780 BTC


2017/11/07 Market Sell
    Main Account                        -0.318045 ETH
    market                      0.318045 ETH
    bitstamp                    0.000020 BTC
    market                      -0.010000 BTC
    Main Account                        0.009980 BTC

which i can just process/check with hledger -f bitstamp.journal (and save with the nicer alignments from hledger).

[simonmichael]

I asked @tshirtman to share that here as food for thought. Seems like a nice real-world test case for any new scheme we consider implementing.

[MatthiasKauer]

Yeah, maybe I should also just go the Python route. What @tshirtman shared looks quite reasonable.
That'd save me a lot of hassle. I have to preprocess my csv files with Python already anyway. Hmm.

Anyway, let me address your example:

Date,Description,Debit,Credit
2017/11/1,foo,10,0
2017/11/2/bar,0,15

I don't see how to address this with account1, account2.
But wouldn't amount-in/amount-out choke similarly?
We would need the "0" to be "", no?

So, recapping, we could:

• consider csv import to be outside the main concern of hledger
• pursue the account3,4,5 solution further
• embed some kind of language

The numbered accounts approach has several open questions.
Externalizing solves a lot of problems.
Embedding doesn't seem to offer real benefits over externalizing.
Hmm. I'm a bit torn right now, to be honest, but externalizing does sound good.

[simonmichael]

Re amount-in/amount-out: you're right it does choke on the 0, that issue is open somewhere. So assume empty fields instead of 0. I think the point is made, amount-in/amount-out still have a purpose and aren't superseded by the proposed numbered amount fields, as far as we can see right now.

[simonmichael]

570

[simonmichael]

Re where to go from here: personally I am feeling that I'd like a little more power in the CSV rules, at least to be able to generate a third posting for transaction fees, because they are robust and low hassle (they work the same way on all platforms and don't require anything extra like the right python version or support libs or command-line know-how).

And, I think we should encourage custom conversion scripts like the above as the escape hatch for more advanced needs, doing what we can to make this easy to set up. Some ideas in this regard:

• provide some nice template scripts in various languages, in docs or repo
• provide a command that creates such starter scripts for you ("hledger new csv-reader-py ..." or something)
• automatically use suitably-named scripts. Eg when reading some.csv, if an executable some.csv-read[.EXT] file exists in the same directory, use it as a pre-filter (run it with some.csv as argument and try to parse its output). Possibly enable this for all file reading. This may or may not be worth the trouble but it's worth thinking about.

[MatthiasKauer]

Re amount-in/amount-out: you're right it does choke on the 0, that issue is open somewhere. So assume empty fields instead of 0. I think the point is made, amount-in/amount-out still have a purpose and aren't superseded by the proposed numbered amount fields, as far as we can see right now.

I disagree. As soon as you have empty fields. account1 and account2 are really equivalent.
That part was actually working in my implementation already.

On the real issue: Maybe it'd be sufficient to add such scripts to the documentation and link it from the rules file that is automatically created.

[simonmichael]

I disagree. As soon as you have empty fields. account1 and account2 are really equivalent.

Quite possibly I'm being dim. Show me (a mockup of) the rules for the example I gave ?

[MatthiasKauer]

I had implemented the following:

o BC3uh matthias@xubfiend:~/dvl/hledger (more-csv-fields)*$ cat t.21881.csv
10/2009/09,Flubber Co,50,
11/2009/09,Flubber Co,,50
o BC3vJ matthias@xubfiend:~/dvl/hledger (more-csv-fields)*$ cat t.21881.csv.rules 
account1 Assets:MyAccount
date %1
date-format %d/%Y/%m
description %2
amount1 %3
amount2 %4
currency $
o BC3vP matthias@xubfiend:~/dvl/hledger (more-csv-fields)*$ stack exec hledger --  --rules-file t.21881.csv.rules -f t.21881.csv print
2009/09/10 Flubber Co
    Assets:MyAccount             $50
    unassigned                  $-50

2009/09/11 Flubber Co
    unassigned                   $50
    Assets:MyAccount            $-50

[simonmichael]

In your example above, how does it know to put MyAccount on the second posting in the second transaction ?

[simonmichael]

PS %d/%Y/%m ?! I didn't think that was used anywhere.

[MatthiasKauer]

In your example above, how does it know to put MyAccount on the second posting in the second transaction ?

That's the problem with ordering I currently have. "MyAccount" is still account1.

PS %d/%Y/%m ?! I didn't think that was used anywhere.

I don't know about that. I got that example by adjusting and working with a failing test.
I believe it was https://github.com/simonmichael/hledger/blob/7698d3171ec104bb579a2a48ad1b4171a918b3ca/tests/csv/csv-read.test#L17

[simonmichael]

I got that example by adjusting and working with a failing test.

Ah I see, it's from an imaginary pathological example. Was just curious!

@MatthiasKauer, are you interested in working further on this and #630 ?
I'm still feeling like https://github.com/simonmichael/hledger/issues/627#issuecomment-343678473, and think that draft docs are the best way to get traction.

[MatthiasKauer]

Looking at it now, I agree with the second part of what you said in https://github.com/simonmichael/hledger/issues/627#issuecomment-343678473

More power in csv rules would be nice but the implementation is tricky (for me?).
Maybe adding a Python script like the one from @tshirtman above to a documentation page would be sufficient. Generating such a script and maybe including it in tests should be the second step in my opinion.

[apauley]

I've added the option of having a custom construct script to the utility I use for my CSV imports, which allows you to generate any arbitrary hledger transaction:
https://github.com/apauley/hledger-makeitso#the-construct-script

This is in line with the script that @tshirtman showed above.

[simonmichael]

Interesting!

[simonmichael]

This has now landed in the form of @adept's #1095. The end result looks not unlike @MatthiasKauer's prototype here, but we worked through a lot of gnarly testing, design and compatibility questions (so far). We've dropped the old principle of "valid rules will generate valid transactions", and the complicated validity checks discussed above, in favour of a free approach where you can generate whatever you want, for now. Closing this one.