Caddy: Translations & more helpful error messages

[mholt]

Error messages

In the upcoming versions of Caddy, I want to make error messages not only descriptive of what went wrong, but to suggest common remedies as well.

I intuitively know what may be wrong because I understand the internals. But this does not help everyone. With error messages that offer suggestions, users will not have to scratch their heads to know what to try in order to fix the problem.

This will not be a trivial change to make, as I think the best way to do it involves creating kind of a "database" of error messages mapped to solutions. This may be a new package with no dependencies on other packages in Caddy so that it can be imported everywhere. Basically, every error that Caddy could return should be classified, where each error class has an explanation and a list of possible solutions.

Translations

On a related note, I would also like this new package/database to contain translations of textual output into other languages. Even though we can't directly translate most Go standard library error and log messages, we can provide translations for everything else we show the user.

To prevent the binary size from growing any larger, we could use custom build tags to selectively compile translation files so that only the desired languages are included.

Implementation

I haven't yet started implementing this but I want help. Not only with translations, but with design ideas/proposals/suggestions. Remember that most strings will need a way to be templated or formatted with some variable values.

I'll probably post my own implementation proposal soon, but I want to hear your thoughts too!

Once we hit the sweet spot for a design, implementation work can begin on a branch.

[mholt]

This is also related to #337.

[navopw]

Cool :+1: I would help with the german and russian translation!

[Cludch]

Could help with the German translation aswell.

[jeyemwey]

Can also help with German translation!

[captncraig]

One thing I'd like considered is that sometimes it is best to not advertise what server you are running. It would be nice to have a "silent" mode that uses extremely basic responses (maybe activated by cmd flag), or else let user supply all of the relevant error documents, even for built in errors.

[mholt]

Cool, thanks for the offers to help so far.

Another thought, perhaps instead of build tags, is a command line flag where you provide a language file that contains the translations. For example caddy -lang deutsch.json so that languages can be swapped out if needed with the same binary.

And build tags / language files are only needed if the binary size increases too much.

[captncraig]

The nice things about it embedded in the binary is only one file to deploy. Build tags would help keep the binary small, but it may not be horrible to embed them all and switch with a flag. Hard to say before you test it.

[navopw]

@mholt I would prefer english names for the language files :smile:

[jeyemwey]

@captncraig Yep, I see your point. All languages should be in the binary, you might want to choose the log-language in the Caddyfile. :+1:
@NAAVOO I would prefer ISO 639-1 codes as they are easier to identity and standardized (de :de: VS _deutsch_; _german_; _allemand_; _alemán_). :smile:

[mholt]

Oh, yeah, meant to clarify, we'll use the ISO codes wherever a language is to be named.

[gomes]

I can help with pt-br translation.

[humboldtux]

I can help with french version.

Hashicorp has just released https://github.com/hashicorp/hil for for string interpolations.

[ulisesflynn]

Willing to help with Error Messages and or Translation (Spanish or Portuguese)

[jeyemwey]

@humboldtux Broken link.. :disappointed:

[humboldtux]

@jeyemwey fixed, thank you.

[jordic]

I could help with spanish translations

[alexandreteles]

I can help with the Brazilian Portuguese translation.

Em dom, 7 de fev de 2016 15:13, Jordi Collell [email protected]
escreveu:

I could help with spanish translations

—
Reply to this email directly or view it on GitHub
https://github.com/mholt/caddy/issues/574#issuecomment-181070366.

[mholt]

Wow, thanks for all the interest in helping!

I have stubbed out a very basic brainstorm in the translate-experiment branch. (Indeed, it's just an experiment for right now!) It doesn't yet solve the problem of providing help texts for error messages.

I know there are i18n and gettext packages that do this, but I'd rather not use one since they seem so heavyweight for what, in our case, is essentially a map lookup.

What do you think of the basic idea? I'm torn on how to key the texts. Right now the key is simply the English text, but should it be a special string like "flag_agree_helptext"? Or just a number?

And, for those texts which are error messages, we need a way to match up concise one-liners (for logs, etc) with a more verbose explanation without getting in the way or making it hard to maintain. Or should this package be separated from the error helptexts completely? Help text is only going to be helpful if the operator is present, which in our case, is only when the server starts. (No need for helptexts in the logs, I think.) Maybe in every if err != nil block we just print an additional line or paragraph to stderr.

[captncraig]

A few ideas I have:

• I'm not sure how you plan on using these strings. Strongly typed variables would be nice. Something like:

import . "github.com/mholt/caddy/caddy/tanslation"
...
fmt.Println(S.AgreeToLETOS)
fmt.Println(S.ErrorOpeningLog(logFile))

This can be accomplished with some code generation. S would be a struct that just has a bunch of string variables (or functions for printf type things) that get loaded at startup time once a language is selected.

• The ultimate translating experience would probably be a web-app for editing translations. A previous job used an automatic setup and it was pretty nice. If each language is a json file containing an array of {Id, String, Timestamp}. You can compare a translation to the source file and find any strings which don't exist, or which have been changed in the source more recently than the last translation. Show a view of original text and let them edit the translated text. A couple of simple views wouldn't take too long to throw together. A generate phase could convert the json files to go source for easy consumption.

^ that sounds complicated in retrospect, but I really don't think it is so bad tbh.

[mholt]

I'm not sure how you plan on using these strings. Strongly typed variables would be nice.

This is modeled after GNU gettext, which allows you to use strings as keys so that the context is preserved for someone reading the program. It also makes it clear how to use format strings properly because the %s are visible.

I thought about using variables but then we have to use build tags to compile in only the desired language or, I guess in your case, use a struct. But frankly I'm not a fan of adding steps to the build process by using code generation.

You can compare a translation to the source file and find any strings which don't exist, or which have been changed in the source more recently than the last translation.

I'm actually considering writing a simple test that will iterate the translations and look for discrepancies, like where strings are missing or empty. So go test could catch those.

[mholt]

@jeyemwey Thanks for stubbing out the German translation!

Before we add more, what do you all think of this structure/design? Is this something you'd be willing to contribute to or would you like to see it set up a different way (and why)?

[captncraig]

The only problem with that approach is you can never ever change the english text once it is established as the key.

[mholt]

@captncraig You can, but it does require a search+replace. The thing is, even if you use variable names and you change the text, you have to change all the translations anyway. So I'm not sure variables are saving us there.

[alexandreteles]

@captncraig said:
The ultimate translating experience would probably be a web-app for editing translations

There is any chance that you was using pottle? This software is a good plataform for oline translating :+1:

@mholt said:
This is modeled after GNU gettext

Can't we use gettext files (so we could use poedit or pootle) and some gettext binding for Go, like https://github.com/gosexy/gettext?

Do you, @mholt, plan to create a translation team or will be making revisions by yourself? I think that you could select some people to take care of submitted translations once we get defined how Caddy wil handle this problem.

[mholt]

@alexandreteles I hadn't seen pottle before, thanks for the link. So a little more information about my preferred approach to this: I want to keep it as simple and least disruptive as possible, in pure Go, and self-contained. I don't want to add steps to the build process or external data files (outside of the source code) if possible. I think I wouldn't mind using build tags, but we'll have to see how much space the different languages take up...

As for maintenance, I realize this is not a particularly easy task. I will definitely be depending on the community to stay involved to help keep translations as current as possible, but I also accept the fact that they will lag behind the English. But naturally, the more demand there is for another language, the more it will be updated by the community.

To make maintenance easier, I can write a quick test that does a simple diff between the keys in the maps for each language, so it's easy to see which languages are missing various strings. (Better yet would be a test that traverses the source code for translate.Text() calls and looks at which strings are in fact missing in the various language files.)

Does this sound good? Would appreciate all your feedback before we scale this up, because it won't be easy to reverse the decision later.

[mholt]

Who would be willing to head this up? Responsibility would be to delegate and coordinate the efforts of translating, integrate the translated strings throughout all of Caddy, etc. We have plenty of volunteers for translators, but to finish bringing this to fruition in the project, I need help.

There's no way I have the time to do this on top of everything else :smile:

[carlisia]

@mholt could this be useful: https://github.com/nicksnyder/go-i18n

[minhoryang]

I can help for ko-KR (Korean)
And you should check crowdin.com (which I used for Tunnelblick translation)

[mholt]

Thanks for the links! I'll check them out, but I also encourage anyone who would like to take over managing the translation project to consider them and find a good way to implement the solution. (Preferably avoid a third party lib if we can? But maybe we can learn from how they do it.)

Anyone interested just comment and feel free to take the reins. :wink:

[mholt]

Would anyone like to take this over after 0.9 stable is released? I don't have the bandwidth to take it on myself in the next year or so, since I'll be working on other aspects of Caddy. If nobody volunteers for now, I'll close the issue and mark it as deferred so that we'll come back to it later...

[Bitti09]

@carlisia that: https://github.com/maximilien/i18n4go seems to be a little easier or not?

[carlisia]

@Bitti09 I haven't used either, probably some trial run with these and/or even other packages could be a good idea.

[fedelibre]

I may help with italian, once this is implemented

[wendigo]

Count me in for polish translation :)

[mholt]

Kind of realizing this feature may require a _lot_ of consistent maintenance from a lot of people, because if a translation gets out-of-date, that could be messy. This is not a task to take on lightly, as it'll be a long-term thing. Maybe we should do this at around v1.0...

[Bitti09]

Maybe add a fallback to English when the translation is not up2date so that
the translator have some time .

On Wed, Aug 9, 2017, 06:23 Matt Holt [e-mail] wrote:

Kind of realizing this feature may require a lot of consistent
maintenance from a lot of people, because if a translation gets
out-of-date, that could be messy. This is not a task to take on lightly, as
it'll be a long-term thing. Maybe we should do this at around v1.0...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mholt/caddy/issues/574#issuecomment-321148905, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AALapXyFNqviV9cV_w2SXzuJA6Zxtfn4ks5sWTQ4gaJpZM4HTiCU
.

[mholt]

@Bitti09 But how to know when the translation is out-of-date?

[Bitti09]

1. Add a version string to translation file or
2. Compare with English language file:
   Strings not found in translation then there is a high change of an outdated
   translation

On Wed, Aug 9, 2017, 07:26 Matt Holt [e-mail] wrote:

@Bitti09 https://github.com/bitti09 But how to know when the
translation is out-of-date?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mholt/caddy/issues/574#issuecomment-321156098, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AALapdJqD4l5Dj6qyKzrdrM-V0sqBkHIks5sWUMQgaJpZM4HTiCU
.

[20zinnm]

Any movement on this issue? @mholt I would suggest that you stick with an existing i18n library, given that this is a major issue already and many others have come before in solving it. Any translation library/package you write will probably need to include string interpolation and pluralization, so that's already two major hurdles.

[mholt]

None yet. I basically am looking for a few people:

• Someone committed enough long-term to lead the project of building in i18n support, and coordinating the effort

• People who are committed enough long-term to help maintain translations for their language(s)

Sounds like we have a few people above who volunteered to help with languages, so we really need someone to step up and be in charge of coordinating it.

[20zinnm]

@mholt I did a lot of this when I worked on Minecraft plugins back in the day--you never know what people want their server to say, so you let them configure the messages, similar to how translations work.
My suggestion is to, at least initially, avoid the mentality of "must have full translations before rollout" because realistically that's not achievable without a paid team of translators. Instead, if we set up the framework for i18n and progressively translate/extrapolate messages from the code-base to the i18n framework, it makes it a lot easier to work on.
Additionally, with a progressive rollout we can start building tools like translation UIs and whatnot. Instead of coordinating a team with dedicated people, we can just accept translation PR's for different modules until all of Caddy is translated.
Finally, I think different dialects could function like addons--when you are downloading Caddy, you can choose the dialect you want it to use (with a default fall-back to English while we fill in other languages).

If any of this is repetitive or rambling I apologize in advance, I've been traveling and am deathly tired.

[20zinnm]

I should note that go-i18n splits the translation files into completed all and untranslated so we have a good idea of what we still need to translate. If JSON is a little unappealing, we could use HOCON or YAML, which are a little more human-friendly.

[mholt]

@20zinnm Thanks for the feedback, your experience is valuable. Definitely not rambling. Your suggestions are reasonable and I think we can go with at least most of them (the translation UI sounds like a bit of work). However, I still need someone in charge of the i18n effort. Dialects/languages as plugins is also definitely possible. Ideally, this'll work without requiring any external resources at runtime.

[zuohuadong]

I can help for zh-CN (Chinese)

[mholt]

I think the best place for translations to exist is in the documentation and not in the software itself; we might look into this for Caddy 2.