Psychopy: Potentially add UTF-8 BOM header to data files

[heaton2016]

Hi Jon:
I find this problem when I'm running a quite simple program, and I've found a way to fix it. Just change 'utf-8' to 'utf-8-sig' in experiment Handler when saving. Maybe this experience can do some help.

[problem] In this study, I created a condition.xlsx to make Instructions change in every repeat. And I have to input the instructions in Chinese. When I run the program, everything is quite fine.Of course, Chinese characters can be displayed quit well. However, the .csvfile in the output data folder was in a mess, especially the columns including Chinese instructions.

[fix] And I track to psychopy.data.experiment.saveAsWideText() function. In this function, the parameter encoding = 'utf-8' may not be suitable for windows system, so I change it to utf-8-sig. Finally, the output file is correct.

Therefore, I'm wondering if this experience could help us to make psychopy3 more compatible to windows PC? Especially in Non-English countries.
p.s.
Thank you for your excellent efforts for psychology researches :+1: . And if you are interested in seeking cooperation in China, don't hesitate to contact me. And I own a Master's Degree in Applied Psychology in East China Normal University, ShangHai. :)

[some information]
Psychopy version: v3.0.0.b13
OS name: Microsoft Windows 7 enterprise
OS version: 6.1.7601 Service Pack 1 Build 7601

[hoechenberger]

Hello @heaton2016, how did you open & view the CSV file? Excel? Could you try another software like LibreOffice instead?

The BOM added by utf-8-sig should really not be necessary. The file contents themselves will be exactly the same, except for an added header. I‘m afraid adding a header might interfere with other parts of people‘s workflows, e.g. when concatenating files; see e.g. https://en.m.wikipedia.org/wiki/Byte_order_mark#UTF-8

We _could_ try it and see if the test suites still pass, though...

Edit Adding the BOM might even cause trouble when importing the file for further processing, e.g. in R or JASP (remember, it is an "invisible" character, so it might be difficult for users to spot where a seemingly odd behavior originates from!). We need to carefully check how other software packages handle a BOM before adding it by default!

[heaton2016]

Hi @hoechenberger
Thank you for your suggestion! I've installed LibreOffice just now. The CSV file containing Chinese character is not tidy until I set LibreOffice as default. You are right, EXCEL may add BOM to the file.

I agree with you. UTF-8 is quite well, considering we need to use R for analysis. utf-8-sig should never replace 'utf-8'.

However, I'm afraid that most Windows users use Excel to open CSV file, which may frustrate these Psychopy learners when their non-English files are messy. Especially when their mother language is not English, they are more likely to use other languages in the condition file. Maybe we could find out other solutions to it, for example add a Tip for windows users :)

[hoechenberger]

I created a CSV file with a UTF-8 BOM:

import pandas as pd

df = pd.DataFrame(dict(foo=[1,2,3], bar=['ä', 'ø', 'ü']))
df.to_csv('/tmp/foo.csv', index=False, encoding='utf-8-sig')

and could successfully (i.e., simply using the defaults) open it via

• LibreOffice 6.1.0.3
• Numbers 5.3
• pandas 0.23.4
• R 3.5.1 (read.csv())
• JASP 0.9.2

[peircej]

Can you upload a csv with this encoding so I can take a look and test on a couple more things? (Excel and SPSS are the most common apps for users)

[hoechenberger]

Sure: example.csv

[hoechenberger]

@peircej You can also create your own using the code snippet I posted above

[peircej]

@peircej You can also create your own using the code snippet I posted above

Ah yes, sorry! :-) I can't see a way to even detect the difference between these files, let alone find a way they break existing behaviour so I agree that if this fixes an issue for someone (as it apparently does) then this should be new default for writing files.

[hoechenberger]

I can't see a way to even detect the difference between these files

If you're on macOS, what you can do is view the file(s) with less in the terminal; it displays the BOM for me.

[hoechenberger]

so I agree that if this fixes an issue for someone (as it apparently does) then this should be new default for writing files.

Do you think we could have another beta with this change?

[peircej]

I'm actually feeling pretty comfortable that this isn't going to cause any problems, or it does they'll be pretty rare corner cases. Rare enough that having one more beta release probably won't reveal the problem (we'll need thousands of testers before one finds a problem and we'll only get that when we release final 3.0.0)

[hoechenberger]

I leave this up to you to decide :) But I also think we've done some quite thorough testing, so should be good in MOST cases!!