Label-studio: New annotation type guide

[VictorAtPL]

Hi,

When can we expect the guide how to extend annotations types in your app?

I need to annotate the part of pictures, in general:
image; many polygons; each polygon should have label like text/logo.

I also need an annotation like image to text (for OCR fine-tuning).

I am able to try writing the code myself, although I need your guidance. Could you tell me what files should I modify to add more annotation types?

[deppp]

Hey @VictorAtPL

Yep, we're working on updating the documentation as well as releasing a "Getting Started" guide. It will be covering both embedding and extending the functionality with additional data types.

There is an initial implementation of the polygons, try out this config:

<View>
  <PolygonLabels name="poly" toName="img">
    <Label value="text"></Label>
    <Label value="logo"></Label>
  </PolygonLabels>
  <Image name="img" value="$image_url"></Image>
</View>

For an image to text, you can use a config like this:

<View>
  <Image name="img" value="$image_url"></Image>
  <Header value="Describe an image above:"></Header>
  <TextArea name="txt" toName="img"></TextArea>
</View>

To add more annotation types, I'd first check how other types are implemented here for example the one for Text.

Let me know right away if you get stuck somewhere. We're glad to help!

[VictorAtPL]

@deppp,
Thank you for your response. I will test it till Wednesday and get back here to presumably ask more questions.

If it's ok for you, please do not close this issue for now.

[VictorAtPL]

@deppp @shevchenkonik
Sorry guys, I didn't have time to look at it, but will in next week for sure.

[VictorAtPL]

I made some little tests and it looks like the polygons are now not usable. I will make a list of changes I need to introduce to make it usable in my use case and I will try to address them.

Could you tell me what was the purpose of _Normalization_ in _Label_? Except making Polygon labels I need also to write some comment what is inside that (it varies every label, because it's text). Should I use _Normalization_ or develop another type in. e. _TextPolygonaLabels_?

[VictorAtPL]

@deppp @shevchenkonik
My question above is still up to date (about the _Normalization_).

I am now struggling with another problem. I try to run "frontend" (written in React) live, so not the compiled version run by backend. I try to do it by running the backend, and then executing npm run start with NODE_ENV=production. The frontend starts, but it looks like it's not in the production, because I can not see "go to task list" link.

If you have some time, please answer me to these two questions.

[VictorAtPL]

There is also a special built-in environment variable called NODE_ENV. You can read it from process.env.NODE_ENV. When you run npm start, it is always equal to 'development', when you run npm test it is always equal to 'test', and when you run npm run build to make a production bundle, it is always equal to 'production'. You cannot override NODE_ENV manually. This prevents developers from accidentally deploying a slow development build to production. source

Well, so how can I debug the frontend? If I need to compile it every time and run it via backend to see the effect, it would take ages to progress with the code.

[VictorAtPL]

I do not understand the way, how files with task and completions for it are created.

When I place one task in tasks.json file and make a submit for it in frontend, it creates a file with completions, but without data, id and task_path keys. File such as this cannot be loaded later via frontend for in. e. update annotations.

What's wrong with it? The task file should be created even before making completion, or the first completion should be saved with the fields containing information about task?

It seems like there is some bug, or I use the app not how I should be.

[VictorAtPL]

I hope #48 fixes the problem described above.

Unfortunately I will have to look at the PolygonLabels, because it looks like they are not loaded in the task view mode. It has the highest priority on my TODO list.

[deppp]

Could you tell me what was the purpose of _Normalization_ in _Label_? Except making Polygon labels I need also to write some comment what is inside that (it varies every label, because it's text). Should I use _Normalization_ or develop another type in. e. _TextPolygonaLabels_?

Yep, normalization is a bit misleading name right now, as it's just a way to add a piece of extra information to the labeled part of data (entity). Alternatively, based on your use case, you can create a new tag that subclasses PolygonLabels and adds an input for that additional info you need.

When you're saying that it's not usable, can you describe what's going wrong when you're using it?

[VictorAtPL]

@deppp
https://youtu.be/on70S27QFhM

As you can see the main problem is that the annotations doesn't load after save.

Here are more code which could help in debugging the problem:
https://gist.github.com/VictorAtPL/2349f2f4e75a0d9237d0ea39c6190169

[deppp]

@VictorAtPL we've recently released an updated version for the Polygons, as well as the website, here some links you may find useful:

• Polygons documentation https://labelstud.io/tags/rectanglelabels.html
• For large enough number of labels, it makes sense to make it into a two-column layout, look at the example configuration here: https://labelstud.io/templates/image_polygons.html

Please let me know if the updated version works well for you.

[VictorAtPL]

@deppp
I've pulled your code and tested it deeply.

I think a couple of things doesn't work at all, a couple of them are not intuitive and a couple of things are lack.

Here is detailed list:

• [BUG] #100 The reset button does not work properly. The image is not visible after pushing this button.
• [BUG] #102 After adding polygon (annotation ended by clicking in the first point), deleting it with Bksp and selecting next polygon's label with keyboard shortcut, there is no possibility to add the polygon.

"Failed to resolve reference 'bUaZuGflcd' to type 'PolygonRegionModel'"

• [BUG] #103 After adding one polygon, submitting task, adding the second one and updating the task, the second polygon is not being saved (not loaded after refreshing page via F5).
• [BUG] #87 Does undo do anything in polygon labels?
• [BUG] #104 If you add polygon, zoom in the image and click on the polygon, it is moved somewhere where it is not visible.
• [IMPROVEMENT] When you drag the edge (a dot is shown) it should create new point of polygon between the two nearest points and place it in the place you are moving the mouse. Right now it starts to move whole of the polygon, which is not intuitive.
• [IMPROVEMENT] The way of ending annotation of polygon - by clicking the first point of annotation (by the way we can forget which one was it and just search thru all points to look for the one which point will get bigger) - is not intuitive. The annotation of polygon should finish in following scenarios:
  
  
  
  • pushing ESC/Enter button,
  
  
  • Clicking any of the polygon's points or clicking inside the area of polygon.
  
  
• [IMPROVEMENT] If one would like to annotate second polygon, which should be as near as possible with the first one, s/he has to zoom the image, because clicking the first polygon makes it selected instead of adding a point to the second polygon. I think it should be possible for annotation to overlap others. A researcher is the person responsible for managing the fact that 2+ polygons overlaps. S/he can do it at least two ways: The Pascal VOC dataset way (void label) or classify a pixel as label which polygon is earlier/later in JSON file.
• [NEW FEATURE] When annotating, pushing Delete should cancel whole process and delete current polygon.
• [NEW FEATURE] If you have selected polygon, clicking outside it should deselect it.
• [NEW FEATURE] There is no way to change the label of currently existing polygons. Pushing CMD/CTRL + label shortcut could change the polygon's label.

Overall, your app is the best open-source annotation tool in the market, but for my case I have to work a little bit on details of polygon labels.

The bugs are presented in following video:
https://www.youtube.com/watch?v=15ssMjgaXQY&feature=youtu.be

I will take care of the improvements & new features in the end of this month, or next month. But for the sake of your users use-experience, it would be nice if you fixed the bugs.

[deppp]

@VictorAtPL Thanks for such a thorough testing and bug reporting! That's really helpful!

We'll be looking into the bugs you've listed this week. Can I please ask you to add them as separate issues so that we have a more natural way to track what gets fixed and when. Thanks!

[VictorAtPL]

@deppp
Sure, but next week. I am quite busy nowadays. I hope you can wait.

[shevchenkonik]

@VictorAtPL
Thank you, your bug report is very helpful. Every item of your report will be a new issue. But I will write on this issue if the bug or feature will be ready.

_[BUG] The reset button does not work properly. The image is not visible after pushing this button._
Will be fixed in patch release 0.2.1-4: https://github.com/heartexlabs/label-studio/pull/98

[VictorAtPL]

@shevchenkonik
I've added three bugs listed above as new issues. The rest (2 bugs) have already corresponding issues and as you stated, one of them is already fixed.

I won't create issues for improvements & new features, because I suppose you've got a plenty of work even without it. I will try to do it on my own at the end of the month or next month.

I consider this issue as resolved, so I am closing this issue. Thank you.