Packages: Scope for keys in mappings/dictionaries (e.g. JSON, YAML)

[FichteFoll]

Terminology:

• A _colored_ scope is a scope name that most color schemes define a color for. Examples include string, entity or constant and specifications of those.
• An _uncolored_ scope is a scope name that is usually not colored, mainly meta and puncutation scopes.

Introduction

I'd like to discuss the standardization of a specific scope name for key elements in dictionaries or mappings or objects or whatever you call them in your language.

Currently, the JSON definition doesn't have any specific highlighting for keys which it could certainly use and I inteded to add for a while (see also https://github.com/wbond/package_control_channel/pull/5496). There are different scopes for the key and value, but in a way that makes it impossible to specifically target the key with a selector.

In the YAML definition entity.name.tag is used, which is a "steal" from xml-like syntaxes where entity.name.tag should denote a tag name.

In most _programming_ languages, no special highlighting is done.

Earlier this year, the JavaScript syntax was updated with a _colored_ scope for keys in object literals, but many people seemed opposed of this change (hurr change is bad durr), thus causing the change to be reverted and an _uncolored_ meta scope being used instead.

Concluding questions

1. Should keys be highlighted with a "colored" scope and which one should it be? I personally like entity.name.key.
2. Should keys not be highlighted and get an "uncolored" meta scope instead?
3. Decide on a case-by-case basis? How complex would this be for color scheme creation?

[keith-hall]

Personally, the first thing I do when I install ST on a new machine is modify the color scheme to highlight JSON keys. To me, it just looks wrong to have it not colored. Especially as punctuation is not colored, its harder to distinguish at a glance, where a key ends and the punctuation begins. Keys are strings and should be highlighted - simple as IMO ;)

As for scope name, I have an additional question - do we want any keys to show up in "Goto Anything"? I believe that at a minimum, "root" level keys should appear here for easier document traversal, but that's just my 2 cents. entity.name.key works for me, I like that the beginning is consistent with xml "keys" - entity.name.tag.

[keith-hall]

The sublime-syntax definition PR uses entity.name[...] for contexts, which I find super useful - I can just Goto Definition on an include or push etc. to find it. Not sure if it would work for/benefit other YAML documents or not though if it were made generic for all YAML files, as I don't use YAML for anything else.

Obviously I realise that a different selector could be used for symbols, but again, I like the consistency - all symbols are therefore automatically/by default colored the same by the color schemes, no extra work required.

[FichteFoll]

Oh, I might as well voice my opinion too. I'm strongly in favor of 1 for all languages. It just makes things more consistent and it makes sense to highlight "string keys" differently compared to "normal strings".
There are always going to be some of these people.

The YAML-tmLanguage also uses some entity.name scope for pattern keys and adds them to the symbol list. For YAML in general, I would use https://github.com/ddiachkov/sublime-yaml-nav instead, which computes the full path of a key when pressing ctrl+r (overriding key binding).

PS: I need to check out the sublime-syntax def PR and see if I can work with it. Since I made the YAML def, I might tune it to be more precise and won't have to start from scratch, which I originally intended.

[wbond]

Combining a meta scope with a proper scope for the key makes the most "general" sense to me. For instance, meta.key string.quoted.double. That way keys can be targeted by color schemes, however I can see in some situations wanting keys to be indexed, in which case entity.name makes sense.

I don't know if this is going to be something we can fully standardize.

[keith-hall]

One thing I did for my sublime-snippet PR (https://github.com/sublimehq/Packages/pull/576) was to use the meta.toc-list scope, to allow easier navigation using Goto Symbol. Probably overkill for snippets, but also allows color schemes to give the "keys" different colors if they desire.

[FichteFoll]

Because I work a lot with syntax definition for JSON-based file formats currently, I took a look into this again.

I have collected a few examples for "keys" in different languages.

JSON

{"key": "value", "key2": null, "3": 3, "key4": 3.14}

JSON only allows strings as keys, which must be quoted.

JavaScript

var o = {key: "value", "key2": null, 3: 3, "key4": 3.14}

JS seems to allow quoted strings as keys and treats everything else as an unquoted string.

When looking up values, it seems that they are converted to strings, because o[3] and o["3"] do the same thing.
You can also use o[undefined] if o has a key named "undefined".

Python

d = {"key": "value", "key2": None, 3: 3, "key4": 3.14}

Python evaluates keys the same way it does for values.
Some objects are not allowed as keys because they are unhashable (e.g. lists and dicts).

Lua

local t = {key = "value", key2 = nil, [3] = 3, ["key4"] = 3.14}

Lua only allows unquoted strings as its table keys, unless you wrap the key in [] brackets, in which case it is evaluated.

(Unlike in Python, you can also use tables as keys but they only compare by reference.)

---

With these examples, we can observe:

1. Not all languages only use strings for keys.
2. Some languages have string keys quoted, some do not.

This leads myself to the following conclusions, respectively:

1. An explicit scope with entity isn't good since it would potentially mask other scopes such as constant.numeric or entire structures (e.g. tuples as keys in Python dicts).
   A meta.key scope for the entire key would be most appropriate.
2. All languages with unquoted string keys should have their keys scoped as such (i.e. string.unquoted.key), in addition to the meta scope.

This allows color schemes to target the entire key in general (meta.key) and string keys specifically (meta.key string).

Remaining issue

When strings appear in a structure/expression that is used as a key (tuples or function calls in dicts for Python), they would get matched by meta.key string as well.
It might be possible to exclude these matching meta.key string - meta.key meta instead, but this remains to be tested. This also implies that there should not be meta scope names between meta.key and string if they both cover the same token(s).

[wbond]

If I recall correctly, JavaScript has keys scoped as unquoted strings and there was a huge backlash over the change. See #141.

While I understand the argument for string.unquoted, I don't think that is going to be a tenable change. I think the only option would be to introduce something like identifier.key or variable.key for object literal keys in JS.

Also note that JavaScript supports expression as keys:

{
[myvar==true ? 'one' : 'two']: 'value'
}

[FichteFoll]

If I recall correctly, JavaScript has keys scoped as unquoted strings and there was a huge backlash over the change. See #141.

Yes, and they do have a point there, because to them the change did exactly the opposite of what we are trying to achieve here. We want to standardize scopes for different colorization of keys, while the change they complained about was that their keys look the same as the values. Special-casing JavaScript object keys with a different scope name seems contradictory here.

JS and Lua share a common trait (besides unquoted keys) in that they both use the same data structure for both "mappings" and "objects" and use attribute-access syntax synonymously with item-access syntax (t.key1 == t["key1"]). You also touched on this earlier (https://github.com/sublimehq/Packages/issues/141#issuecomment-182826789). I will need to think about this and probably have to look at some real-world JavaScript to judge on this.

This post was edited.

[FichteFoll]

Patching the seemingly most popular color scheme, Monokai, to support these scope changes should be an easy step forward and pave the way for other scheme authors.

Ensuring the other most popular color schemes (on Package Control) are updated before the next beta build should be possible as well. We might want to consider a mailing list, a platform (the forum) or just a github issue with announcements about the default packages and scope naming standards for color scheme developers.

What I'm mostly concerned is: If we are to averse from change, nothing may change at all. I'm sure some people won't like the changes to Python decorators either because their color will change, for example. There will also be people who prefer to have their JSON keys be colored the same as their values (God knows why).
We should be looking for ways that allow us to do these changes when we consider them appropriate.