Openrefine: Improvement : Split multivalued-cells by regex/special characters

[ettorerizza]

Unless I am mistaken, the "split multi-valued cells" feature only accepts literal characters as separators. For example, we can't split values separated by a line break like these.

The solution I found is to replace line breaks with a comma (value.replace(/\n/, ','), but it's just a workaround. Would anyone have a better solution or idea how to improve this feature?

[thadguidry]

@ettorerizza Your not mistaken, Split multi-valued cells is String only currently.
But the enhancement could easily be done by someone. Like @ostephens :)

Take the code that is here: https://github.com/OpenRefine/OpenRefine/blob/master/main/src/com/google/refine/expr/functions/strings/Split.java#L69

and implement it similarly here: https://github.com/OpenRefine/OpenRefine/blob/master/main/src/com/google/refine/expr/functions/strings/SplitByCharType.java#L53

[ostephens]

@thadguidry happy to have a look at this

[ostephens]

The current behaviour is to trim leading and trailing whitespace from any values created by the split. This makes less sense to me once you add in the ability to split in a more precise way (either by regular expression or by string lengths).

Which seems the most desirable behaviour:

• Never trim
• Trim when split by text separator but not otherwise
• User configurable option to trim (or not) values created by split (and if this option, what should the default option be - trim or not trim)

[thadguidry]

@ostephens my preference would be "never trim" because I'm a purist and because I work with UTF-8 quite a bit and unprintable control characters that are significant to my work.

Eventually, what will help with all that is ...
I want to see in OpenRefine a toggle mode for DISPLAY UNPRINTABLE CHARS for our users (like the ACK and CR/LF below)...so that they show up. This is work to be done in the future in part of our UI Enhancement project(s).

[ostephens]

Thanks @thadguidry. Pinging @ettorerizza as the person who opened this issue, but I'm inclined to go with 'never trim' at the moment - this puts control with the user, and it is always possible to follow up a 'split multi-valued cells' with a 'trim' if you need to

[ettorerizza]

Thank you for this work @ostephens ! Regarding the automatic trim, I have no specific opinion, except that I personally prefer doing this kind of thing myself rather than letting the software perform "under the hood" operations that I may not wish.

Just a detail, perhaps unimportant: this change breaks the symmetry between "split mutlivalued cells" and "join multivalued cells", its sister feature. I'm not sure it could be a problem for the users, but I point it out anyway.

[ettorerizza]

Issue #1113 is a good example of the frustration that can be felt when a trim is done without being asked.

[ostephens]

Thanks @ettorerizza. I've gone ahead with the 'never trim' option at the moment.

In terms of the 'join' function - do you have any views on how it should work if it is to 'mirror' this updated split function?

(Changing behaviour of join to not add whitespace as reported in #1113 is probably v straightforward - I can definitely have a look at that, I'm wondering about any other changes you might have in mind to 'mirror' the update 'split')

[ettorerizza]

@ostephens I do not have a specific use case in mind. No doubt there is at least one person on this earth who dreams of being able to join multivalued cells by \n, but I am not sure that it is very frequent.

[ostephens]

@ettorerizza ok thanks - once this is finished off I'll have a look at #1113 at least

[wetneb]

Never trimming seems to be the correct approach here as the user can always trim afterwards or adapt the regex to eat whitespace on both sides.

[thadguidry]

@ostephens Awesome job on contributions Owen ! Your Rocking ! Keep 'em coming !

[ostephens]

Fixed by #1278 (now merged thanks @wetneb)