Js-lingui: Keep PO headers unchanged for existing catalogs

[FredyC]

Describe the bug
When I load extracted PO catalogs into some sort of editor and do some changes, it does modify headers in its own way. Later when I rerun extract again, headers are again modified, mostly lower casing, but nothing is lost.

This is mostly an issue about file versioning as when headers will be changing between commits, it means a bunch of unrelated changes all the time.

To Reproduce
You can use for example known POEdit to see this happening or simply edit PO file manually to change the casing of some header.

Expected behavior
Headers should be kept completely unchanged for existing catalogs.

Additional context
This is upstream issue https://github.com/smhg/gettext-parser/issues/12 which does not seem to be getting fixed any time soon. So it's either about forking it or copying out relevant parts.

[tricoder42]

gettext-parser also doesn't handle obsolete packages in a way I want, so maybe we should write own parser. Before I tried to find alternatives for gettext-parser, but nothing seemed to be better.

What about write the parser using peg.js. We need just read/write po file, nothing else.

[FredyC]

I am not feeling for PegJS, it says simple, but my head is spinning just by looking at examples :)

I have also found this one, so it's probably worth the try before getting hammer :) https://github.com/rubenv/pofile

[tricoder42]

Looks good! Interested in trying it on your own? :)

[FredyC]

Can you please elaborate more on how _obsolete_ should be working differently? It's just a flag that should go after #, right? I wasn't investigating much what's wrong with the current implementation.

Strangely enough that pofile package tries to identify obsolete items with prefix #~ which doesn't seem to be anything official. It does parse flags per references just fine, but this is something extra.

[tricoder42]

I'm not sure if there's official specification for obsolete messages. I used to work with Django web framework and they simply comment out obsolete messages. I use obsolete flag as a workaround, because gettext-parser doesn't allow commenting messages.

[FredyC]

Alright, I see now. The #~ seems to be an actual comment in PO file (see example). Based on this the module decides that particular translation is obsolete. I will set obsolete flag when message is commented out, seems like best course of action.

There is also support for msgctxt which denotes context. Based on reference doc, when the context is present, you can have a translation with the same msgid but with different context. That sounds like very useful feature in cases you are using messages as keys and sometimes it might be unclear in which context that message has been used. I have verified it with Loco service and it indeed works, some other will probably support it too. I will try to implement it from the side of format and perhaps there could a special macro that would add context to the message.

Edit: I am not entirely sure how to approach the context since it basically allows for same keys in a catalog. Only, the way seems that the value in the catalog would be an array of possible translations and it has to pick one. Well, I guess that needs some major refactor, so I will leave that for 3.0.

[FredyC]

One more thing... Currently, the description is considered to be a one-liner, but Gettext supports multiple #. lines for comments (official term - extracted comment). The pofile will parse all #. correctly and give out array of string. I don't like the idea of joining it together and then separating it again. I am thinking of keeping description prop to be a first comment and also to have comments with the rest of those. What do you think?

[tricoder42]

Alright, I see now. The #~ seems to be an actual comment in PO file (see example). Based on this the module decides that particular translation is obsolete. I will set obsolete flag when message is commented out, seems like best course of action.

Yes, that's awesome 👍

here is also support for msgctxt which denotes context. Based on reference doc, when the context is present, you can have a translation with the same msgid but with different context. That sounds like very useful feature in cases you are using messages as keys and sometimes it might be unclear in which context that message has been used. I have verified it with Loco service and it indeed works, some other will probably support it too. I will try to implement it from the side of format and perhaps there could a special macro that would add context to the message.

I wouldn't implement context, at least not right now. So far I saw only one or two uses and the workaround isn't difficult. On the other hand, library expects ID to be unique, so we either need to do massive refactoring and figure out some hack.

One more thing... Currently, the description is considered to be a one-liner, but Gettext supports multiple #. lines for comments (official term - extracted comment). The pofile will parse all #. correctly and give out array of string. I don't like the idea of joining it together and then separating it again. I am thinking of keeping description prop to be a first comment and also to have comments with the rest of those. What do you think?

It's hard to say how external tools will handle #.... Lingui will always add/replace description of message, so if we assume that the first #. is description extracted from source then we're fine until external tool changes order of messages. This is "extracted comment", so the multiline description should be extracted from source by lingui, right? External tools shouldn't modify this description.

[FredyC]

This is "extracted comment", so the multiline description should be extracted from source by lingui, right? External tools shouldn't modify this description.

I understand it the same way. However, I can imagine that a developer might want to add additional comments to a translation manually before sending it to a translator. In that case, it would make sense to keep those in there. It would probably need to be documented to use #. for additional comments and add it after the first line of description. Ideally, if this could be handled on source code level, to have multiple comments.

I wouldn't implement context, at least not right now.

Yea, I've edited comment above probably after you read it :)

[FredyC]

There is another small issue. Apparently msgstr can be repeated for a single translation as PO supports its own plural syntax. I suppose we are not going to do that. Point is that pofile extracts msgstr as an array. I am thinking what to do with it. Grab the first one and ignore the rest?

[tricoder42]

I would ignore it and raise a warning. Lingui CLI will never generate array or msgstrs, so it can only happen when someone imports a PO file with unsupported message format.

[tricoder42]

I understand it the same way. However, I can imagine that a developer might want to add additional comments to a translation manually before sending it to a translator. In that case, it would make sense to keep those in there. It would probably need to be documented to use #. for additional comments and add it after the first line of description. Ideally, if this could be handled on source code level, to have multiple comments.

This is gonna be tricky. In such case the developer would edit locale-specific catalog, probably only in one locale, but right now the same description is used for all locales...

[FredyC]

This is gonna be tricky. In such case, the developer would edit locale-specific catalog, probably only in one locale, but right now the same description is used for all locales...

You are right, it shouldn't be supported. It should show a warning that for any extract comments out of source code the pure # comment should be used. And I will actually transform additional extracted comments to translator comments. It would be just another property comments so it doesn't break anything.

I would ignore it and raise a warning. Lingui CLI will never generate array or msgstrs, so it can only happen when someone imports a PO file with unsupported message format.

👍

[tricoder42]

FYI, at the moment the description property isn't used at all. It's just for the next major release, when description will be added to macros.

[FredyC]

Yea, that would be a reason why I was unable to find its use :) Have you actually thought about multiline descriptions? Because those should be probably split into multiple #. lines. If you plan something like that, I would prepare it right away to have it as an array.

[tricoder42]

The description can be any string. So, unless we prevent it, they can do something like this:

t({ 
  id: "msg.id",
  description: `
    Message description
    (This is the second line)
  `
})

Even if we prevent it, they still can do Message description\n(This is the second line)

I'm very sceptical than anybody will actually use it, so it's probaly just nice to have.

[FredyC]

Alright, I see now. The #~ seems to be an actual comment in PO file (see example). Based on this the module decides that particular translation is obsolete. I will set obsolete flag when message is commented out, seems like best course of action.

This needs some second thought. Obviously, the obsolete flag is not supported by any tools and they would simply ignore it causing translator to be translating something that's not needed anymore.

#, obsolete
msgid "obsolete"
msgstr "Obsolete message"

Opposite to this the notation with #~ which is supported in tools I've tried (POEdit, Loco) so it's much more reliable.

#~ msgid "obsolete"
#~ msgstr "Obsolete message"

I would drop the obsolete flag, keep parsing of it only for a backward compatibility, but new PO files will be written as commented out. Is that ok like that?

[tricoder42]

Absolutely! The obsolete flag was just a workaround, because I didn't know about #~ and simply commenting out msgs didn't work.

Great job 👏

[tricoder42]

Released in v2.6.0