Packages: Interpolated strings

[FichteFoll]

I've compiled a few scope names for languages that have interpolation in strings (incomplete due to lack of time right now):

Bash: $(fk yea) -> string.interpolated; "${apples}" -> variable.other.bracket
C#: not implemented
Groovy: source.groovy.embedded.source
JavaScript: string.template into source.js.embedded.template
Perl:
Php:
Python: see https://github.com/sublimehq/Packages/issues/715#issuecomment-289097254 (meta.string.interpolated for the entire string and meta.interpolation source.python.embedded for the interpolated parts)
Ruby: source.ruby.embedded.source
Scala:

(for examples, see https://en.wikipedia.org/wiki/String_interpolation)

All these add on top of the string scope and do not clear it. Punctuation scopes are inconsistent.
This raises the following items:

1) I remember that you, @wbond, considered using clear_scopes for interpolated strings to clear the string scope so that the interpolated code would not be using the string scoping. I like this a lot, and will most likely use it for Python's f-strings. We should apply that to the other syntaxes too, however.

2) I intend to also add punctuation.definition.string.interpolation.begin and .end respectively, so that color schemes could target punctuation.definition.string and would match all the string edge punctuation.

3) It seems like source.<language>.embedded.source is the most common scope for interpolated sections. Should this be the standard?

[wbond]

So, my reaction is:

1. Yes, I do believe clear_scopes is the correct thing to use here. The downside is now you can't select the whole string using string.* scopes. Thus I propose strings have a meta.string.* that will include all string parts, interpolated begin/end tokens and all embedded source.
2. Since this is the beginning of a non-atomic element, I think the scope punctuation.section.interpolation.* should be used for begin and end tokens. These tokens should, IMO, not get string.* since they aren't technically string contents, but markers.
3. Yes, I think that works for the scope. The punctuation.section.interpolation.* scopes should not be part of this, but should be just before and just after it.

[wbond]

I suppose the meta.string only technically needs to be added to strings that can contain interpolated sections. Should it be included on all strings, for consistency? For example, single quoted strings in PHP.

[FichteFoll]

1. Ah yes, I wanted to include a mention of a meta scope like meta.string but forgot in a hurry. Wonder if View.extract_scope breaks when re-tokinizing the initial meta.string string context with a meta.string source.*.embedded.
   Actually, if a scope: key defines two scopes, would clear_scopes clear both with one number or would it require one for each space-separated scope?
2. True.
3. Why the .embedded.source though? I mean, source is already in there.

[wbond]

Number 1: I am not sure about extract_scope(). clear_scopes operates on space-separated scope names. If a context pushes two scopes, any children would need to use clear_scopes: 2. If clear_scopes: 1 was used, only the farthest-right scope would be removed.

Number 3: I think the .embedded part is probably useful. I agree that the source probably isn't needed.

[FichteFoll]

By the way, I'm slowly making progress with python. However, it is not possible to do it as precise as I wanted because of with_prototype recursion (expressions are allowed within template strings but strings are expressions as well), so I'll have to go with the less precise method.

I do have a suggestion on how to resolve these recursion loops that I will propose soon-ish on the core issues repo, once I'm back from vacation.

[djspiewak]

On my phone right now, but just pointing out that the Scala mode handles string scoping in exactly the way you propose: with punctuation scoping and clear_scopes. Can't recall if it has a meta scope in the string as a whole, but it certainly should.

[michaelblyons]

Since this discussion was opened, C# interpolation was merged in. It looks like

• punctuation.section.interpolation.begin.cs
  
  
  
  • meta.string.interpolated.cs source.cs (with clear_scopes: 2)
  
  
• punctuation.section.interpolation.end.cs

[FichteFoll]

Python:

f"result: {value:{width}.{precision}}\n"
#         ^ punctuation.section.interpolation.begin.python - source source
#          ^^^^^ source source.python.embedded
#               ^^ - source source
#                ^ punctuation.section.interpolation.begin.python
#                 ^^^^^ source source.python.embedded
#                      ^ punctuation.section.interpolation.end.python
#                       ^ - source source
#                        ^ punctuation.section.interpolation.begin.python
#                         ^^^^^^^^^ source source.python.embedded
#                                  ^^ punctuation.section.interpolation.end.python - source source
#                                    ^^ constant.character.escape
#          ^^^^^^^^^^^^^^^^^^^^^^^^^^ meta.interpolation.python
#               ^^^^^^^^^^^^^^^^^^^^ meta.format-spec.python
#          ^^^^^^ - meta.interpolation.python meta.interpolation.python
#                ^^^^^^^ meta.interpolation.python meta.interpolation.python
#                       ^ - meta.interpolation.python meta.interpolation.python
#                        ^^^^^^^^^^^ meta.interpolation.python meta.interpolation.python
#                                   ^^^ - meta.interpolation.python meta.interpolation.python

Edit: Forgot to mention that the entire thing is meta.string.interpolated.python, whereas normal strings are meta.string.python.

[FichteFoll]

Newest discussion on this topic: #1432 (highly relevant).

[wbond]

Alright, these elements are going to be part of the scope naming doc with the next release of Sublime Text. Thanks for all of the discussion!