Easy_localization: Flat json data in initial load, decreasing CPU footprint

[danilofuchs]

In the current implementation, every time the user calls the tr() method, the library recursively searches through the localization tree until it finds a valid string.

We could use flat to flatten the localization map into a dot-separated map.

Example:

flatten({
  'key1': {'keyA': 'valueI'},
  'key2': {'keyB': 'valueII'},
  'key3': {
    'a': {
      'b': {'c': 2}
    }
  }
});

// {
//   'key1.keyA': 'valueI',
//   'key2.keyB': 'valueII',
//   'key3.a.b.c': 2
// };

This flat map should be saved and used across the library.

Then, accessing the array is really simple and performatic:

final key =  "my.translation.key";
final message = messages[key] ?? key; // "My awesome translated text" ?? "my.translation.key"

[spiritinlife]

@danilofuchs Thats a good point and optimisation.
I was thinking that instead of creating a flat map on load which could cause some delay ( haven't tested it ) we could just cache the keys in a dotted way every time one is accessed and then we can resolve them from cache instead of looking them up again.

[spiritinlife]

Here is a possible solution @aissat @danilofuchs

Map<String, String> _cachedNestedSentences;
Object _getFromMap(String key, Map<String, Object> value) {
    if (!value.containsKey(key))
      print(
          '[easy_localization] Missing message: "$key" for locale: "${this._locale}", using key as fallback.');

    return value[key] ?? key;
 }
  bool isCached(String key) => this._cachedNestedSentences.containsKey(key);

  String _resolve(String key) {
    if (this.isCached(key)) return this._cachedNestedSentences[key];

    List<String> keys = key.split('.');
    String kHead = keys.first;

    var value = this._getFromMap(kHead, this._sentences);

    for (var i = 1; i < keys.length; i++)
      if (value is! String) value = this._getFromMap(keys[i], value);

    if (keys.length > 1) // add to cache if nested key
      this._cachedNestedSentences[key] = value;

    return value;
  }

[aissat]

Here is another solution using flat PKG @spiritinlife @danilofuchs

class CachedBundleAssetLoader extends AssetLoader {
  const CachedBundleAssetLoader();

  @override
  Future<String> load(String localePath) async {
    return flatten(rootBundle.loadString(localePath));
  }
}

[aissat]

Here is a possible solution @aissat @danilofuchs

Map<String, String> _cachedNestedSentences;
Object _getFromMap(String key, Map<String, Object> value) {
    if (!value.containsKey(key))
      print(
          '[easy_localization] Missing message: "$key" for locale: "${this._locale}", using key as fallback.');

    return value[key] ?? key;
 }
  bool isCached(String key) => this._cachedNestedSentences.containsKey(key);

  String _resolve(String key) {
    if (this.isCached(key)) return this._cachedNestedSentences[key];

    List<String> keys = key.split('.');
    String kHead = keys.first;

    var value = this._getFromMap(kHead, this._sentences);

    for (var i = 1; i < keys.length; i++)
      if (value is! String) value = this._getFromMap(keys[i], value);

    if (keys.length > 1) // add to cache if nested key
      this._cachedNestedSentences[key] = value;

    return value;
  }

I like it this, coz it is better to minimize the use of external libraries (PKG)

[spiritinlife]

@aissat Nice thinking using the AssetLoader the developer can actually do that nice !

In my approach we lazy cache only the nested keys which i think is a good approach mem and perf wise

[aissat]

@aissat Nice thinking using the AssetLoader the developer can actually do that nice !

In my approach we lazy cache only the nested keys which i think is a good approach mem and perf wise

yeah we should use this approach like a default optimisation

[spiritinlife]

@aissat @danilofuchs Changed my mind on this :)

1. CachedBundleAssetLoader shouldn't be named like that, maybe FlatJsonBundleAssetLoader
2. Also it shouldn't exist at all, since it heavily depends on how the library internally works.

I am thinking that maybe we should ditch this idea of flattening or we should add it as an option, something like computeNestedValues or flattenNestedValues in EasyLocalizationDelegate.

Thoughts?

[danilofuchs]

Maybe I didn't express correctly the benefit of flattening.

When analyzing the internal lib code that resolves a key (seen here), you can see it uses a recursive approach, separating each string between . in subpaths and passing through a sublist of these paths each iteration.

It becomes very verbose very quickly, not having a standard way of finding out if a key exists or not.

If we flat it first, when accessing the array, it is very simple:

message = messages[key]

If the message exists, it is a string. If not, null (then we can show a warning).

The optimization part may not be as important as simplifying the internal code and allowing for more complex rules. I could set up a sample project comparing speed improvements made by these changes.

I also found a bug in the current approach. What I found is that, in the current implementation (on master), if we do the following (which is very common when using the intl simple JSON format):

// en.json
{
    "my.flat.key": "This key is flat in the JSON definition"
}

When accessing it in this library, the recursive approach is not able to access it, as it will try to access a json field like this:

// Trying to access tr("my.flat.key")
{
    "my": {
        "flat": {
            "key": "Not really a field"
        }
    }
}

This specific case is simple to solve with a direct comparison, but if the map is in any unconventional way (mixing . separated keys with nested), only flattening could solve it (as I can see).

[danilofuchs]

Also, @aissat. I please ask you to not commit directly on master when discussing these new behaviours. We are still analyzing the best approach and a PR should be opened with a draft implementation so that we can all take a look, make tests and approve it before publishing.

[spiritinlife]

@aissat Exactly as @danilofuchs is saying, please not commit directly to master :)

@danilofuchs I see your point, my only concern is what happens on very big language files.