Label-studio: Encoding

[RasmusEdvardsen]

Hello there,

I'm using your project to annotate data with scandinavian letters (æ, ø, å) from a text file, like so:
label-studio init poc-front --input-path=data.txt --input-format=text --label-config=config.xml
I then start the front, navigate to labeling, and I am greeted with 'Ã¥' instead of 'å'.

Obviously this is an encoding issue, and I was wondering if you would consider the standard input text format to be utf-8
I suggest the following change:

from:
with io.open(path) as f:
to:
with io.open(path, encoding="utf8") as f:

https://github.com/heartexlabs/label-studio/blob/8209b3a286a0443b772e73540aad24bcf27e6712/label_studio/tasks.py#L84

[PhilipMay]

I have the same issue.

[PhilipMay]

Is it fixed here but not released? #300

[RasmusEdvardsen]

No, this would have to be fixed in tasks.py, I haven't made a PR on it - should I?

[PhilipMay]

I would be your first test user... ;-)

[RasmusEdvardsen]

There, you should be able to read both .txt files and .json files from input-path now - @makseq could you or someone else have a look at this? Thanks

[makseq]

Please, check fixes in release/0.7.3 https://github.com/heartexlabs/label-studio/pull/320 . I've copy your code to this branch. Unfortunately, your PR branch is very different to master.

[PhilipMay]

I have an issue with 0.7.3. See here: #346

[Nighthyst]

Hello, I have the same issue with french characters in the GUI as you can see :

Is the fix already effective ? I am trying to labellize named entity in sentences, so this is my task.csv (test) file :

Bonjour à tous

As you can see the "à" character was changed just as other french accents.

I am working on a Windows 10 Professional with the latest version of label-studio installed with pip (Anaconda). So my version of label-studio if the 0.7.3 and my Python version is 3.7.6.

Also I have an unrealated question. Is there any template for NER task with pre-labellized data (sentences where entities' span were already predicted with SpaCy for instance) ?

[RasmusEdvardsen]

@Nighthyst did you try to initialize the project like above?:
label-studio init poc-front --input-path=data.txt --input-format=text --label-config=config.xml

As for your unrelated question:
You can add a backend to your label-studio setup (under the model pane in the ui), that can act as a bridge between your model or just data, and it will prefill the "predictions" key of the task when labelling:
Backend model

Alternatively, if you have the data, and you don't want to implement the "model" part, you can just prefill the tasks you import in to label-studio with predictions as above.
Tasks - completions & predictions

[Nighthyst]

Well thank you : if I init the project this way it works. Before I started my project with the quickstart commands :

# Requires >=Python3.5
pip install label-studio

# Initialize the project in labeling_project path
label-studio init labeling_project

# Start the server at http://localhost:8080
label-studio start labeling_project

and then I imported my data.

Thank you also for my unrelated question 👍

[RasmusEdvardsen]

Could someone take a look at the above PR? @makseq maybe?
W.r.t. the init and importing from the UI not working, I think it's a small issue with encoding - let me know what you think

[RasmusEdvardsen]

@Nighthyst can you confirm that the error occurs when you import tasks in json format from the /import UI ?

[makseq]

@RasmusEdvardsen yes, it seems you are right. I'll check you PR a little bit later.

[makseq]

I've merged it to release/0.7.4.

[Nighthyst]

@RasmusEdvardsen Both json and plain text formats create this encoding problem when importing tasks

[makseq]

@Nighthyst please, check release/0.7.4

[Nighthyst]

@makseq I installed Label Studio with your realease/0.7.4 branch with the command :

pip install --upgrade git+https://github.com/heartexlabs/label-studio.git@release/0.7.4

This time I can create an empty project like this :

label-studio start labeling_project

import my tasks and accents are correct :

[saguillo2000]

Is there anyway to change the encoding from UTF-8 to ISO-8859-1 or any other encode in the text plain?

[makseq]

@saguillo2000 it's easier to convert all your data to utf8. LS works in utf8.