Hledger: Support different separators for CSV import [$10]

[uu1101]

CSV files served by some banks use a semicolon ";" as a separator instead of a comma. It would be useful being able to specify the separator in the rules file.

---

A $10 bounty on this issue was claimed.

[simonmichael]

Our csv lib (or another) might support this now ? Personally I'd like the ability to read TSV (tab-separated values).

[larsen]

I had a look at the code of Text.CSV: it doesn't seem to me the delimiter is replaceable with characters other than comma. The author is clear about it: the module is an implementation of RFC4180, that does not mention alternatives for the field separator.

An option to implement this feature could be switching to something like cassava, but I'm not sure about hledger dependency policies.
@simonmichael what would you recommend?

(I'd be happy to help with this task)

[simonmichael]

In general, we like light dependencies that just work, including on windows. cassava-megaparsec at http://hackage.haskell.org/packages/#cat:CSV is new and might be interesting ? After the 0.28 release https://github.com/simonmichael/hledger/pull/366 will be landing, and we'll be using megaparsec.

[pmatos]

I am interested in this since my bank serves semi-colon separated values.

[simonmichael]

Status update: 0.28 hasn't happened yet but #366 did land, so there's no reason to wait if someone wants to experiment with new CSV libs.

[peti]

Has anyone tackled this project yet? It's quite common for CSV files to use separators other than ,, really, and it would be nice to have support for this (seemingly trivial) fact of life.

[simonmichael]

Anyone up for trying cassava-megaparsec ? It seems to handle any delimiter (but only one at a time, you might have to make several attempts I guess).

[ony]

Me ended up with python script . Default Python library csv is much more powerful than cassava-megaparsec. csv-conduit have more format settings than cassava-megaparsec but still lacks support for configuring format like "Use ""Excel"" export\nMake your CSV parser fail"; 1138; "Have a fun".

[simonmichael]

https://groups.google.com/forum/#!topic/hledger/_aXEM1ssutc

[ghost]

I just converted a file with semicolons to a csv file with cat bank.csv | sed 's/,/./g' | sed 's/;/,/g' > bank.csv in the meanwhile. Also make sure in german speaking languages it is quite common to use a , instead of . to split decimals.

[sorsasampo]

Osuuspankki, one of the largest banks in Finland, serves semicolon-separated fields as well. Probably because in Finland we use comma as a decimal separator - and it seems to be pretty popular in Europe:

https://en.wikipedia.org/wiki/Decimal_separator#Examples_of_use

So finnish and european users would probably be very happy about a configurable separator!

[simonmichael]

So would I. There's a lot of interest in this, who could take a crack at it ? Support available in #hledger if needed.

[simonmichael]

PS this issue has now been open for 4 years. :-)

[simonmichael]

Related helper by @sorsasampo: https://gitlab.com/sorsasampo/hledger-osuuspankki (reddit)

[lucamolteni]

I could work on this during Zurihac 2018.

[lucamolteni]

Can you please review this?

https://github.com/simonmichael/hledger/pull/829

Thank you

[simonmichael]

Done and merged. @lucamolteni I think you need to mark it resolved on bountysource.

[simonmichael]

Oh, perhaps I need to close this issue so you can do that. I was planning to discuss more here, but that makes sense. Done.

[simonmichael]

I was thinking the separator char is a property of the file and does not change between runs, so we should move it into the rules file. And actually, I see that's what this issue specified, but I don't want to delay this bounty any further. One of us will follow up before the end of month release. I also want to make this work for tab separators, and allow output in ssv and tsv formats.

[simonmichael]

Follow-on issues created, bounty complete. Thanks @lucamolteni !