Packages: $top_level_main special context name is not documented

[aziz]

In TeX.sublime-syntax line 41 there's this include:

- include: $top_level_main

Is this $top_level_main simply a bug or does it mean something?

A quick search showed that it's been used in other places as well.
namely: C, C++, Objective-C, D, LaTeX and TeX

[FichteFoll]

It's equivalent to the old $base and includes the most top level context, i.e. the one that is displayed in the status bar.

[aziz]

Yes! Thanks :+1:
maybe It should be mentioned somewhere in the docs

[randy3k]

$top_level_main causes problems when we are trying to include those syntaxes in markdown.

See for example,
https://github.com/jonschlinkert/sublime-markdown-extended/issues/49
https://github.com/sekogan/MarkdownLight/issues/2

Is there anyway to fix this problem?

[FichteFoll]

https://github.com/textmate/textmate/pull/1276 is the proper issue to reference.

[wbond]

Just a heads up that in general $top_level_main is kinda broken. I recently overhauled the C, C++, Objective-C and Objective-C++ syntaxes and removed $top_level_main from all.

The core issue is that $top_level_main tends to be used to attempt some sort of syntax inheritance. However, the scopes references by a .sublime-syntax file in include: rules are hard-coded to the syntax file is was defined in. Thus if you use $top_level_main in C++ to copy a structure from C, if there are any contexts that include others, they will always be hard-coded to the C version and not see the C++ version. And this needs to be that way, otherwise things could be broken by "downstream" syntaxes.

To accomplish some sort of inheritance of syntaxes, there would need to be a way to include contexts in a syntax in such a way that all include: rules would first pull from the "current" syntax, and fall back to the included syntax when searching for context names. However, this could have its own pitfalls to where it would be easy to accidentally break some nested context if a name was reused.

Anyway, that concept of functionality currently does not exist. Because of that, copy-paste/duplication seems like the most "correct" answer.

[MattDMo]

I just want to make sure I understand this. Say I have two languages, X and Y, where Y is sort of a superset of X, and so inherits a bunch from X. In X.sublime-syntax, a certain context has

- include: foo

where foo is defined elsewhere in X.sublime-syntax. All good.

However, in Y.sublime-syntax, there is _also_ a context named foo. When Y includes X, you want the foo context applied to the view being parsed as the foo context from Y, _not_ X, is that what you're saying? And if Y.sublime-syntax doesn't include a foo context, then the one from X will be applied? What if _both_ have a foo context, and in one situation I want X:foo to apply (pretend it's embedded code or something...) and in another situation I want Y:foo to apply? Will there be a mechanism to choose?

This is going to require a huge amount of coordination between language defs, as we could have a situation sometime in the future when the language itself changes dramatically (think Python 2 -> Python 3, or Java throughout the years) and the .sublime-syntax would need to be heavily modified, but such changes may not apply to the related syntaxes. I'm not saying it's not doable, or even that it's not a good idea, I'm just trying to think of all the ramifications up front.

[wbond]

@MattDMo Yes, that is what I am saying. For instance, an "expression" from C is nested inside of parens, and all sorts of other constructs. "statements" are nested inside of curly braces, and preprocessor directives and all sort of other things. When working in C++, I want the C++ definition of "expression", which is really the C definition of "expression", plus some things added. However, I can't reuse all of the "nested" contexts from C in C++ because they all use the C version of "expression". I'd really rather they use the C++ version.

If you look at the current C, C++, Objective-C and Objective-C++ definitions:

• C stands alone
• C++ reuses parts of C, but a large part is copy pasted to handle different definitions for "expression", "statements" and "global"
• Objective-C reuses parts of C, but a large part is copy pasted to handle different definitions for "expression", "statements" and "global"
• Objective-C++ reuses parts of C, C++ and Objective-C and is pretty much all copy-paste the various previous definitions

$top_level_main is a disaster because is breaks embedding (placing C in fenced code blocks of Markdown, or JavaScript is a script tag of HTML), and it provides only a sledge hammer to deal with language inheritance. Previously C, C++ and Objective-C each used $top_level_main, with Objective-C++ just including both C++ and Objective-C. It was a disaster with all sort of different lookbehinds and gross hacks, C++ syntax was present in the C syntax to prevent C contexts from consuming C++ constructs, etc.

Basically, $top_level_main is useless. I think the only "real" usage left is TeX and LaTex in the default packages. I'm planning on removing those when I get to refactoring/improving them. The D syntax uses it right now, but it doesn't need to. The instances of that can just be replaced with main. I believe the only reason it is present is because D was forked off of C++ at one point.

The whole C-family of languages would benefit from an alternative to include, such as import or something like that that overlays the local versions of contexts. And yes, it would require coordination, however right now maintaining the C-family is a huge pain. That said, I didn't want to implement any new features without having a good sense of what real problems need to be solved. Now that I have a better idea, my plan is to discuss ideas with Jon to see if there is a reasonable solution.

[aziz]

It might be informative to read this thread on Textmate forum as well. People expressing their problems with developing a syntax grammar for D

[wbond]

@aziz I believe that most of the issues raised in that thread are not a problem for .sublime-syntax. There have only been a few deficiencies I've run into while working on JavaScript, Rust and the C-family. In most cases creating more precise contexts is the answer, but currently partial composition of different syntaxes together isn't really possible. Luckily it isn't really needed very often.