Translate-plugin: [BOUNTY] Translating page's url & implement translatable indexes

[BMCouto]

A bounty has been placed on this issue

Scope definition

• [x] CMS Page URLs should be translatable (see below for technical description)
• [x] Static Page URLs should be translatable (see #156)
• [x] Models should support index definitions and lookup mechanism

---

It would be nice to be able to translate the path when creating a new page.

Taken from #112:

Basically, if I have three languages, for instance "en", "fr", "de", and an about page,
I currently can access :

/en/about
/fr/about
/de/about
For SEO purposes, I would prefer to access (as the same route) :

/en/about
/fr/a-propos
/de/uber-uns

[yordivd]

Is this possible? Can't find anything

[annelbco]

+1 on this.

[flo-sch]

+1

[hansek]

:+1: Is there any plan to implement this by Rain Lab / @daftspunk ?

[artras]

+1

[oryxgd]

+1 as well

[fdeneux]

+1

[gergo85]

+1

[sajjad-ser]

+1

[wateryoma]

👍

[kikobalco]

+1

[amdad]

+1

[KGE]

+1

[nacr]

+1

[marcoelho97]

+1

[Deathz0r]

+1

[manuelbua]

+1

[aurukas]

+1

[jan-vince]

+1

[jan-vince]

I understand this is not a priority, so I was looking for some workaround.

This is, what I have so far:

1. Install Rainlab.Translate plugin
Set languages - I used: Czech [cs] primary, English [en]
Force URL schema in plugin's component (as plugin do not change automatically locale for base URLs like / - if I change from /en/contact to /kontakt, locale is still en, which is maybe a bug)

2. Add all matching URLs in CMS page URL field:
My URL for contact page is: /:slug|^bkontaktb|bcontactb$

_"Kontakt" is Czech word for "contact"._

3. Add URL checks in PHP part of CMS page:

function onStart(){
$slug = $this->param('slug');
$locale = App::getLocale();

$sessionLocale = (Session::get('rainlab'));
if( isset($sessionLocale['translate']['locale']) ){
    $locale = $sessionLocale['translate']['locale'];
}

// prevent mixing of pages with different locale (eg. /cs/contact)
if($slug == "kontakt" && $locale != "cs"){
    $this->controller->setStatusCode(404);
    return $this->controller->run('/404');
    // Or redirect to correct page:
    // Session::pull('rainlab', 'translate');
    // return Redirect::to('/kontakt', 301); 
}

if($slug == "contact" && $locale != "en"){
    $this->controller->setStatusCode(404);
    return $this->controller->run('/404');
}
}

Right now I don't know about any other way.
The best solution would be translatable URL field in October pages, but this can take some time.
This workaround alows me to use October for multilingual sites.

_I would be grateful for any better solutions!_

[manuelbua]

I'm not sure we are given any sort of attention in this thread: is there someone who can chime in and shed some light on this issue after 2 years being open?

My opinion is that, given the current state of the web and SEO, multilingual URLs are an obvious part of a CMS and should be handled correctly: the number one thing we are probably all interested now, however, is that it would be really great _to know the developers' ideas on this_. Anyone?

[artras]

@daftspunk ?

[Amekiir]

+1

[daftspunk]

@manuelbua For issue escalations the current advice is to submit a bounty for it:
https://www.bountysource.com/issues/5736226-translating-page-s-url

[manuelbua]

@daftspunk Thanks for the advice, I didn't intend to sound harsh, but i didn't really questioned about your timeline _at all_: i think it would be beneficial to everyone to at least know if this feature is planned or not, or if you have any other solutions, or what solutions October has to offer at this time.

I can see people coding their own solutions and workarounds for this, but any direction from you as the lead developer would be very much appreciated.

[daftspunk]

@manuelbua I haven't had time to dedicate enough thought to it at this stage. I would hate to give poor advice on something not properly considered.

The solution requires discovery and it appears complex. To put things in to perspective, there are about 500 issues we are working through. There are 0 bounties. Bounty driven issues are addressed before regular issues, it is the only way we can prioritize the massive workload. Thanks for your understanding.

[manuelbua]

Thanks for getting back to us and clarifying that.

[aurukas]

I encourage everyone to give at least 5$ as a bounty for this demanded
feature. I did just that.

On Fri, Jul 8, 2016 at 3:03 PM dudez [email protected] wrote:

Thanks for getting back to us and clarifying that.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rainlab/translate-plugin/issues/25#issuecomment-231342745,
or mute the thread
https://github.com/notifications/unsubscribe/ABhalb6_F8T_ZMZe0oDhZIZ__P542bS-ks5qTjyBgaJpZM4C2Vyk
.

[artras]

• 5$ from me !

[thomasilmer]

Placed a 40$ bounty, this functionality is needed for a lot of our projects.

[jan-vince]

And $5 from me.

_I would love to see Translation plugin's functionality in the core of October once ..._

[daftspunk]

Discovery complete: This is possible and planned for implementation.

It will function as follows:

1. The CMS Page "URL" input will display a locale picker, where the user can toggle between locales
2. Translated URLs will be stored to the page's viewBag component

```
url = "/contact"

[viewBag]
localeUrl[cs] = "/kontakt"
```

1. Only one page URL can exist at one time, for example, when locale cs is active, only the /kontakt URL is available, the URL /contact will return 404. This is a non-negotiable feature due to the technical reasons 1 page = 1 URL. For duplicate content reasons it is desirable anyway.
2. The onSwitchLocale AJAX handler will need to be modified to redirect to the new URL if it is different, as opposed to the Redirect::refresh currently in place. i.e en -> cs refresh will 404.

Prototype code

// Extend all CMS Pages (front-end execution context only)
!App::runningInBackend() && \Cms\Classes\Page::extend(function($page) {

    // As soon as the template is hydrated
    $page->bindEvent('model.afterFetch', function() use ($page) {

        // Check the page viewBag against the active locale
        // ..

        // If the active locale has a custom URL, rewrite the URL
        $page->url = '/kontakt';

    });

});

FYI - The above prototype code is made possible thanks to the Halycon engine we implemented in the stable version, released just 7 days ago. This means it was not easily implemented in 2014, but it is possible today! :tada:

[acasar]

How would that affect SEO? The main reason I see for translating URLs is to make search engines happy. If all non-default languages return 404 when Google is crawling the site, doesn't that defeat the purpose?

[amdad]

Only one page URL can exist at one time, for example, when locale cs is active, only the /kontakt URL is available, the URL /contact will return 404.

So this will generate 404s randomly? How SEO be like in this situation? It's like have both eyes but you can watch only through one at the time. :wink:

[amdad]

I see @acasar have same concern. Also from user perspective this solution is not very friendly.

[daftspunk]

The above so far is just a technical description, it probably needs to be explained from a UX perspective. Assuming you have 3 languages set up:

• en: English
• fr: French
• ru: Russian

There is a page with the following content:

url = "/contact"

[viewBag]
localeUrl[ru] = "/контакт"
==
<p>Page content</p>

Note: "Contact" in French is "Contact" so a translated URL is not given, or needed.

• /fr/contact - Page in French
• /en/contact - Page in English
• /ru/контакт - Page in Russian
• /ru/contact - 404

If the page has no URL override specified, then the default URL still operates. I hope this clears things up.

[acasar]

@daftspunk Sounds great :+1: Placed a $10 bounty as well, would love to see this feature implemented.

[amdad]

Will this work without lang prefixes too?

/contact - Page in French
/contact - Page in English
/контакт - Page in Russian

[daftspunk]

Will this work without lang prefixes?

Absolutely. The prefixes are given just to clarify what locale is selected, it can be selected using the URL or from the user's session.

[amdad]

What about database driven plugins?

[daftspunk]

@amdad Nothing changes from the functionality today, just page URLs can be translated. Do database driven plugins work today? Yes.

[amdad]

Hm, I mean slugs translation. I've tested a little bit and it worked in some way. But URL translation for plugins like Rainlab.Blog will be available? This proposal cover file pages only?

[daftspunk]

I see what you mean. The blog post slug will not be translated because it is not driven by the URL schema, that is out of scope for this issue. The blog plugin would need to be modified to support multiple slugs, probably using a look up table of some kind. Please log an issue for it over at the blog repo.

[amdad]

I think more about.. How _any_ plugin can have translated slugs for categories/aliases etc. Do October have some method for it? So Rl.Blog would be as example.

[daftspunk]

You raise an excellent point, at the moment the translated values are stored in a single row in JSON, not great for look up purposes. So we would need a dedicate table for slugs, that can store each slug in a separate row and is indexed for performance.

Then provide an API for the plugins to use, part of the TranslatableModel behavior, something like

class Post {
    public $translatableIndexes = ['slug'];
}

Post::whereLocaleIndex('slug', 'контакт')->first();

Let's include this as part of this issue and use the blog as the first implementation.

[jan-vince]

Hi @daftspunk,
what I would like to end up is that this file (index.htm):

url = "/kontakt"
[viewBag]
localeUrl[en] = "/contact"
==
<p>{{ 'page.content'|_ }}</p>

produce this:

http://www.domain.cz/kontakt

• this page has Czech locale
• the content in <p>is translated to Czech language

http://www.domain.cz/en/contact

• this page has English locale
• the content in <p>is translated to English language

The plugin should not produce duplicates like

http://www.domain.cz/cs/kontakt
http://www.domain.cz/en/kontakt
http://www.domain.cz/cs/contact

Problem now is, that locale in session overrides URL schema (when you return from /en page to / just by changing URL, locale remains EN). I have to check it and change every time like this:

function onInit(){
    if($this->page->url == "/"){
        Session::put('rainlab.translate.locale', 'cs');   
    }
}

---

Your approach with [viewBag] is right I think (probably only possible).

But the routing should be reviewed. Maybe possibility to disable locale prefixes?

For me, this would be ok (if locale prefixes are disabled):

url = "/kontakt"
[viewBag]
localeUrl[cs] = "/kontakt"
localeUrl[en] = "/en/contact"
==
<p>{{ 'page.content'|_ }}</p>

The result will be:

http://www.domain.cz/kontakt
http://www.domain.cz/en/contact

Nothing more...

[daftspunk]

Update: CMS pages now support translated URLs, please conduct testing of this feature. Edge updates should be enabled to get the latest changes.

Guys, thanks for funding this, for participating and using October, it is because of you that this plugin has come so far already.

[amdad]

Yea, I thought it may be probably broken update. We have missing files in updates before. When Rainlab ML richeditor widget was not included into update. And... here was the same. Now works properly.

Offtop:
Oh, 353 with SCSS !!! Coool!

[jan-vince]

@amdad - It works for me in build 353 and Translate plugin version 1.2.0

@daftspunk

Right now these all 4 versions works, so it has bad impact on SEO (but I wrote about it before - solution is to use rel=canonical) - I just don't like the feeling, October is creating something behind my back :)

These all works, but I believe only that two URLs defined in Page's URL field, should work:

/test-cs
/cs/test-cs _this should not work, i didn't defined it in page's URL_
/test-en
/en/test-en _this should not work either_

But I can definitely live with that :)

_Do you plan to support Rainlab.Pages plugin with translatable URLs too?_

BTW great job!

[daftspunk]

@jan-vince This is out of scope unfortunately, the locale should not be guessed from the URL, instead it should be specified from the URL prefix or by the session (specified by the user). /test-en does not work rightfully so because your default is set to cs and this is the selected language at the time. Please try to test the plugin within the scope mentioned above.

[acasar]

@daftspunk Thanks for implementing this! Will give it a test run tomorrow.

@amdad Be considerate and watch your language. Your reply is really shameful. Prefixes are standard practice in multilingual websites and almost everyone is using them. OctoberCMS should aim to cover 90% of use cases and not every edge case possible. And I believe it's doing that very well.

[jan-vince]

@amdad If the length was a problem, sorry to write it so long. I can confirm what you said.

@daftspunk:

I just wanted to say this:

When we have translated URLs, why don't use them?

We know where /contact leads and what locale it should have (EN). Same for /kontakt (CS).

Why to force anyone to use locale prefixes?

This is what I meant with "locale should by guessed from URL".

_If anyone wants to use locale prefixes, it can be done from Translate components 'Force URL schema._

Is it too difficult to support both approaches?

[daftspunk]

It is not difficult to support two different approaches, we do it already for forceUrl. The problem is the requirements are not solid enough, for example, if I have a url /contact for en and fr how does it know what locale to use? This is why the prefix must explicitly declare it as part of the URL schema.

If I can decode what you are saying correctly, I think you want the default locale to be prefix-less, so you would have

• /contact (default)
• /fr/contact (french)
• /cs/kontakt (czech)

This is possible if we say include an option to make the default locale "naked" where it has no prefix (forceUrl with naked default). However, the others must have one, otherwise the solution becomes too magical and convoluted.

There is a quote from Albert Einstein that expresses our philosophy the best:

"Everything should be made as simple as possible, but not simpler".

[daftspunk]

Also keep in mind, just because we don't want to develop it, doesn't mean you can't do it yourself. It is possible (and easy) to make a plugin that scans the pages and compiles a bunch of redirects based on the translated page URLs. That is essentially what I see is needed here. It is not something we want to include in this plugin and is out of scope from what was agreed to at the time the bounty was created.

[daftspunk]

Guys this is getting off topic. I'm going to clean up this thread to keep it focused. Thanks for your feedback so far.

[daftspunk]

Update: Static Pages now support translated URLs (5fa5379d6857d51a93340ce71ed2d5167f351b66). These improvements have created several bugs in the Static Pages plugin, so consider this partially implemented. There is still work to be done.

From initial tests:

• Static menus do not show translated URLs
• The locale picker does not redirect correctly
• Prefix not added to URLs generated by RainLab.Pages (https://github.com/rainlab/pages-plugin/commit/e7793b8638263e088e73754f14e9d0694287cc27)

[Kunilt]

When we can expect models definitions indexing and lookup mechanism functionality?

[daftspunk]

Still doing discovery work for this piece @Kunilt, the bulk of this feature is available now.

[daftspunk]

Update: Discovery on translatable indexes is complete (This feature has been implemented as part of the discovery process).

Indexed attributes

Translatable model attributes can also be declared as an index by passing the $transatable attribute value as an array. The first value is the attribute name, the other values represent options, in this case setting the option index to true.

    public $translatable = [
        'name'
        ['slug', 'index' => true]
    ];

Once an attribute is indexed, you may use the transWhere method to apply a basic query to the model.

Post::transWhere('slug', 'hello-world')->first();

The transWhere method accepts a third argument to explicitly pass a locale value, otherwise it will be detected from the enivronment.

Post::transWhere('slug', 'hello-world', 'en')->first();

[daftspunk]

This feature is now complete. Thanks to everyone who participated. The changes will be rolled out in stable after ~7 days of testing. To obtain the improvements for testing, enable the cms.edgeUpdates setting and perform a system update.

The following versions are needed.

• October core Build 362+
• RainLab.Pages v1.2.10+
• RainLab.Blog v1.2.10+
• RainLab.Translate v1.2.4+

[tombien]

Hello, great feature! I think it would be great to make Static menu items Titles (in popupdialog) also translatable to make this languange support more complex. Are you planning it?

[thomasilmer]

Hello @daftspunk , I have been testing the new feature and so far it's great!

I just have one small issue with implementing the translatable slugs. Let's say I have a page with this signature:

url = "/projects/:slug"

[viewBag]
localeUrl[nl] = "/projecten/:slug"
localeUrl[de] = "/projekte/:slug"

In which ":slug" is different for every language, e.g.:
en => house
nl => huis
de => haus

Currently we have a custom plugin which has a backend menu for adding projects (Basically adds a Project model with some properties), which has a translatable index attribute "slug". In the onRun() function of "/projects/:slug" we check the database for a Project model which has the current slug as attribute. The page is then populated with properties of the Project model.

This works fine, untill you use the localePicker, because only the "projects/projecten/projekte" part of the url is switched. The ":slug" part stays the same as before the locale switch. This can result in people being redirected to "projects/huis" for example. This URL (rightfully) gives a 404.

I tried looking at how RainLab.Pages plugin tries to solve this, but I haven't found the same functionality there.

What is a nice solution to translate the slug when the onSwitchLocale is called?

I also have a second issue which is somewhat similar. Let's say I don't use the language prefixes /en/, /nl/ or /de/, but instead use domains to determine the locale:

example.com => english
example.nl => dutch
example.de => german

Currently we set the session locale based on the domain and it works. Except again, when we switch language with the localePicker, the domain won't change with it. Currently we have a temporary fix by redirecting to the domain homepage (not using the localePicker), but it would be a lot nicer to redirect to the current page in a different language.

Is there an event for onSwitchLocale? Because then we could probably set the domain after onSwitchLocale ourselves, but currently I don't know how to fix this.

Sorry if this is in the wrong topic, but I felt like this is partly in the scope of the new translatableIndexes.

[Kunilt]

@daftspunk what about eager loading attribute translations for models?

For example I'm querying hundred of models with one translatable attribute and now for every model Translator makes additional DB query to get this attribute translated.

[daftspunk]

The bounty is accepted in the majority. Thanks to those who participated.

Glad you like the new features. The last 3 comments are out of scope for this. As a guide, let's try to keep 1:1 issue:issue ratio. I will try my best to address these quickly:

@tombien it would be nice, yes. For now we do have a workaround of using key names for menu labels, that can be seeded in the theme.yaml file. See the Bonjour theme for more info.

@thomasilmer this issue was noticed as well. It is difficult to solve because the localePicker component has no idea about the Post model, nor should it (separation of concerns). We would need to think of a recommended way to handle this. Log a new issue for it please and we can discuss options there.

@Kunilt Looking at the code, eager loading isn't supported at this stage. Log a new issue for it and include some sample code that demonstrates the issue (used for peer review and solution formulation).