Hledger: move --separator flag into the CSV rules file

[simonmichael]

Follow-on from #176. We'd like to replace the --separator flag with a separator CHAR directive in the rules file, since it belongs more to the file than the command invocation.

[schoettl]

What if CHAR is a white-space char, tab or space?

The config file would look like

which, I think, is still better than hledger --separator=$'\t'.

Should the config file parser parse "separator" token, one " ", and then one character?

Or better (and more consistent) parse "separator" token, one or more " ", and then Maybe one character? If it parses Nothing for CHAR, it could default to space as separator.

[simonmichael]

Good question. Till now, spaces surrounding csv rule values have always been ignored. I think we should require that a non-empty value must be visible to the eye in some way.

Eg, by writing an escape sequence like \t instead of a literal tab. Unfortunately there's no escape sequence for space.

Or, by enclosing the separator in either single or double quotes:

separator  ' '      # a single space
separator  "  \t"   # two spaces and a tab

In this case, we'd need to decide how thoroughly to support quotes. Would they be allowed/ignored/useful in other csv rules ? Would they support quote escaping, like "\"" ? Would there be any different behaviour of escape sequences (like \t or \n) in unquoted, single-quoted, double-quoted values ? Would an unbalanced quote be an error ?

Or, we could use some made-up marker:

separator
separator sst

[cfellinger]

On Sun, Feb 10, 2019 at 09:33:24AM -0800, Simon Michael wrote:

Good question. Till now, spaces surrounding csv rule values have
always been ignored. I think we should require that a non-empty
value must be visible to the eye in some way.

Eg, by writing an escape sequence like \t instead of a literal
tab. Unfortunately there's no escape sequence
for space.

One might use a visible space "␣" (utf8: U2423, OPEN BOX) for those
places where an actual explicit space is needed. We at home use it
nowadays to write our CSV harvesting rules and have a shortcut for
it in our ~/.XCompose files:

: "␣" U2423 # OPEN BOX

As an aside, I remember clearly the first time I used lowercase
letters in the commandline interface of some file transmission
program: the fear it wouldn't work every where and such, but the huge
improve in readability. The same is happening lately as I slowely
catch up and realise that we trully entered the Unicode age.

--
groetjes, carel

[schoettl]

I don't like the idea of a visible UTF-8 space symbol.

• It's hard to input for most users. I'd have to search for a unicode symbol internet page, then search for the symbol (what if there are multiple similar symbols?), then copy & paste it. Other users may never have done this workflow.
• It's not self-explanatory. Granted, it's a special case anyway. But it's also a very uncommon way for config files or CLI to represent the space character.
• You then cannot use this symbol as separator in CSV files.

I also hope that we can keep a simple config file format for now, without quoted-string parsing.

How about parsing:

  string "' '"
| string "\" \""
| string "\\t"
| char

after removing leading whitespace?

Example config file:

separator \t
separator ;
separator ' '
separator " "

BTW: ~/.XComose is interesting. I use Neo keyboard layout, so I already have a lot of symbols at my fingertips.

[simonmichael]

separator \t
separator ;
separator ' '
separator " "

@schoettl that sounds ok. Except.. people might try to use '' or "" quotes in other places. Would they be allowed everywhere ? Or would we say that they are a special syntax only for enclosing spaces ? Would it allow 2 or more spaces ? Would it be a little odd that we introduce backslash for tabs, and quotes for spaces ?

I think a backslash syntax avoids these questions. Eg:

separator ;
separator \t
separator \s

or:

separator ;
separator \tab
separator \space

or:

separator ;
separator <tab>
separator <space>

[simonmichael]

or:

separator ;
separator TAB
separator SPACE

[simonmichael]

Prior use of s, FWIW: in regular expressions it means "whitespace character", including tab and newline.

[simonmichael]

PPS: separator \space is no good for steady pace. separator SPACE is no good for ALL YOUR SPACESHIPS. Which leaves the <tab>/<space> option.

Or, overrule the prior use of \s and say that in a hledger CSV rules file it means a space character only. (But not in a hledger regular expression..)

[simonmichael]

I'll try that one more time.

separator SPACE is no good, eg: ALL YOUR SPACESHIPS.

separator \space - wouldn't handle A\spaceB\spaceC. I guess that's ok, never mind.

We could also: overrule the prior use of \s and say that in a hledger CSV rules file it means a space character only. (But not in a hledger regular expression..)

[simonmichael]

No; still confused. The SPACE example would work fine, disregard that too.

[kepi]

I didn't study how separator works now in depth, but if I understand correctly that separator could be regular expression, IMHO it can work as:

separator /\t/
separator     / /
separator   /[;,\t]+/

Simply using / character to surround regular expression, which is defacto standard. In this way you still could ignore spaces around, have clear surrounding of separator and exact way how to parse and use the separator.

I'm not sure if this is good idea, but it would work for me just fine :)

Edit: if separator can be regular expression but it may be only string, I would personaly vote for using regular expression always as it will simplify the things. If that is not preferable, my idea is probably worthless :)

[simonmichael]

Multi character delimiters and regexp delimiters might be useful, but currently cassava supports only a single character.

[adimit]

Here is how I see the problem space:

• Cassava supports only singe char delimiters. It is reasonable to assume that multi-char delimiters are rare enough to be a special case we don't need to cover
• Using quotes is somewhat misleading: quotes aren't supported anywhere else. It would be better not to use quotes
• Using the delimiter characters for space and tabs verbatim is problematic for readability reasons

The way I see it, the separator directive takes one character only. If you give it more than one character, it errors out. However, if you give it more than one character and form a special reserved word, such as SPACE or SPC or TAB, it will be translated to the corresponding obvious separator.

Therefore, my suggestion is to use

separator SPACE
separator TAB
separator ;
separator þ

Do not laugh about that last one. I once worked with a (very expensive proprietary) DB system that actually used the letter Thorn to delineate records internally! And it was leaking that representation left and right… :roll_eyes:

Something like separator foo would be illegal.

If the design is OK for you @simonmichael I could write a PR.

[adimit]

I believe this issue should be addressed with #1161 now, no?

[simonmichael]

Right you are! Thanks.