Bedrock: Developer Workflow

[hoosteeno]

Success Criteria

• [x] Developer can write rules on a view that decide what template to show for a particular request (e.g. new localized or old localized depending on whether a fluent file or some fluent IDs are present)
• [x] Developer can deactivate a particular page in a locale
• [x] Developer can wrap a section of HTML in a template in a conditional that will check for the presence of certain fluent IDs in the fluent file for a locale requesting the page
• [x] Developer can ascertain which locales have a particular fluent file activated (e.g. via django management command)
• [x] Developer can specify required strings in a fluent file
• [x] Developer can easily convert an en-US .lang file to an en .ftl file
• [x] Developer can easily convert a legacy template to a Fluent template
• [x] Developer can easily convert existing translations from .lang files to .ftl files

[pmac]

The things checked off above are being done in #7679 but are not yet in master. I'm using the check list to mark progress.

[pmac]

I've added some management commands for converting .lang files and templates to the new system more quickly. In #7679 I used them to convert the /mission page. I did the following:

$ ./manage.py lang_to_ftl mozorg/mission
$ ./manage.py template_to_ftl mozorg/mission bedrock/mozorg/templates/mozorg/mission.html

And then I updated mozorg/urls.py to point to the new template (I've kept the old one for now) and refer to the new mission.ftl file. It works pretty well. It won't get everything but it should give developers a big head-start on any page conversion. I've now started working on the script to convert the existing translations in .lang files to .ftl files as well.

[pmac]

And now... this

$ ./manage.py port_lang_translations mozorg/mission

Which takes the ftl file from the first command and converts all of the existing translated .lang files into .ftl files ready for commit and push to the www-l10n repo.

[alexgibson]

I tried out this process porting the about page and here was my experience. You can see my branch here for context.

Overall, things worked great - I was able to port the page very quickly locally 💯

Convert en-US .lang file to .ftl

./manage.py lang_to_ftl mozorg/about-2019

Worked without issues 👍

Template conversion

./manage.py template_to_ftl mozorg/about-2019 bedrock/mozorg/templates/mozorg/about.html

This process seemed to work fine, but after pointing the url to the new template and trying to view the page I got this:

Turns out I had to stop docker and run make build again, in order for it to pick up the newly generated fluent file. Not a show stopper, but it took a while. Is there a quicker command to run here?

Strings inside macros were extracted OK in step one, but weren't covered in the template conversion e.g.

{{ card(
  class='mzp-c-card-medium',
  title=_('Pioneers of The Open Web'),
  ga_title='Pioneers of The Open Web',
  desc=_('Our leadership has been at the forefront of building a healthier internet since Day 1. What began as an alternative to corporate domination has grown into a global force for good online.'),
  image_url='mozorg/about/leaders.jpg',
  include_highres_image=True,
  aspect_ratio='mzp-has-aspect-3-2',
  link_url=url('mozorg.about.leadership')
)}}

I had to manually update these toftl() calls in the new template. This was easy enough to do using find/replace - but there were 22 instances in this one template. We use macros a lot these days for Protocol. Perhaps we could try improve this in the template conversion?

Locale string conversion.

./manage.py port_lang_translations mozorg/about-2019.ftl

This ran ok and I was able to inspect the translated files - and they appeared ok. I wasn't able to preview these in the context of the web page running locally however. Do these files need to be committed to https://github.com/mozmeao/www-l10n first for that to happen?

[pmac]

I had to manually update these toftl() calls in the new template. This was easy enough to do using find/replace - but there were 22 instances in this one template. We use macros a lot these days for Protocol. Perhaps we could try improve this in the template conversion?

ah yes! my replacement thing looks for the opening {{ _(. I'll add this case.

This ran ok and I was able to inspect the translated files - and they appeared ok. I wasn't able to preview these in the context of the web page running locally however. Do these files need to be committed to https://github.com/mozmeao/www-l10n first for that to happen?

That should work. I may need to add some directories to the ones mapped from local in docker.

[alexgibson]

I set L10N_CACHE_TIMEOUT=10 in my local .env and the docker cahcing issues seem to go away 👍

[pmac]

I've tweaked a bunch of things about template and lang file conversion. Please give it another shot when you can. The about page is still a good test, but I also tried it with ./manage.py template_to_ftl mozorg/about/governance/policies/participation bedrock/mozorg/templates/mozorg/about/governance/policies/participation.html and had a good bit of success with trans blocks as well.

One issue is that there are a few places in that participation page that just use %s as a place holder with no variable name, and that isn't supported in the new system, so I just print warnings in that case. Let me know if you can think of a better solution.

[alexgibson]

@pmac the participation page above doesn't have a .lang file associated with it, since the page is en-US only, so when I run the template conversion command I just get a stream of NO MATCH warnings. Nothing was actually converted in the template, so it's a bit hard to compare.

What's the best way to go about creating a new .ftl file from scratch, is it just a manual process?

[alexgibson]

I tried this on the history page which is mostly {% trans %} blocks and things worked pretty well:

./manage.py lang_to_ftl mozorg/about/history-details

./manage.py template_to_ftl mozorg/about/history-details bedrock/mozorg/templates/mozorg/about/history.html

It got caught out by one string:

NO MATCH: By 2002, <a href="%(over90)s">well over 90% of Internet users</a> were browsing with Internet Explorer.

Extracting locale string also worked well (no caching issues locally):

./manage.py port_lang_translations mozorg/about/history-details.ftl

[alexgibson]

One issue is that there are a few places in that participation page that just use %s as a place holder with no variable name, and that isn't supported in the new system, so I just print warnings in that case. Let me know if you can think of a better solution.

I'll give this some thought, but I think a warning in the console should be enough to catch this in most cases.

[pmac]

the participation page above doesn't have a .lang file associated with it, since the page is en-US only

The locale/en-US/mozorg/about/governance/policies/participation.lang file exists in my locale directory, and I see it in the repo:

https://github.com/mozilla-l10n/bedrock-l10n/blob/master/en-US/mozorg/about/governance/policies/participation.lang

[alexgibson]

Oh huh, I looked but didn't find it. I'll check again thanks

[pmac]

but it's no big deal. participation was just one I chose because it had a lot of both _() and lang blocks. It also has lang blocks with a lot of arguments, so it was a good test for that. I'll also try the history page and investigate that one string. Thanks again for trying all of this!

[alexgibson]

So I ran ./manage.py lang_to_ftl mozorg/about/governance/policies/participation.lang and have a couple of thoughts:

I saw three console warnings reading:

WARNING: Place-holder with no variable name found in string. Convert "%s" to a Fluent variable in the new file.

This is red/bold and hard to miss, which is nice. But delving into the lang file to find those isn't so easy. Think we could add a comment in the .ftl file above the string reading # VARIABLE ERROR!, or something similar? This would make it easier to catch in code review as well.

Unrelated, but I also noticed it copied over existing TAG comments:

# TAG: participation_nov2018

Are these obsolete now, should we ignore them?

[pmac]

Are these obsolete now, should we ignore them?

They almost certainly are. Are tag comments always formatted that way? i.e. TAG: tag_name?

[pmac]

The issue with that string it didn't match from history is that the string on the page uses a % which in the extracted string is %%. I should be able to fix that.

[alexgibson]

They almost certainly are. Are tag comments always formatted that way? i.e. TAG: tag_name?

It looks like tag names are either using underscores or dashes, but they consistently start with ## TAG: e.g.

• ## TAG: quantum-firstrun-whatsnew
• ## TAG: platform_strings_04232019

https://github.com/mozilla-l10n/www.mozilla.org/search?q=%23+TAG&unscoped_q=%23+TAG

[pmac]

Okay. Just pushed:

• tag comment ignore
• fix for unmatched string due to %%
• Fixing %% in strings in ftl files
• Replace %s in strings with { $VARIABLE_MISSING } for easy search/replace

[pmac]

I moved the two "may move" items to more appropriate (IMO) cards. So this one is nearly there. I just want to document all of these things and then the PR should be ready to merge.