Packages: [RFC] Preprocessor Keywords

[deathaxe]

Motivation

Many IDEs highlight preprocessor keywords with a dedicated color to separate them from normal runtime code, which improves readability in situations, when single lines or hunks are to be excluded from compiling.

Example

C/C++

#include <stdio.h>
void main() {

    if (_var) {
#ifdef DEBUG
    printf("My debug message %s", _var)
#endif
        switch (_other) {
            case LBL: ;
            default: ;
        }
    }
    return 0   
}

Actual Situation

The C/C++ define the following scopes to support highlighting preprocessor keywords:

• meta.preprocessor keyword.control.import
• ~meta.preprocessor keyword.declaration~
• ~meta.preprocessor keyword.other~

The C# syntax uses:

• keyword.control.preprocessor
• keyword.other.preprocessor

The JavaScript syntax uses:

• keyword.control.import-export

Issues

No 1

While the C/C++ approach seems a nice solution in order to reuse existing keyword scopes in a new preprocessor context it causes normal keywords to be highlighted as preprocessor keywords in #define statements.

#define IF if
// ^ meta.preprocessor keyword.control
//         ^^ meta.preprocessor keyword.control

_Desired Situation:_ The keyword if should be highlighted with its normal keyword color, while #define is highlighted with the preprocessor keyword color.

No 2

It seems keyword.control.import is used in some syntaxes to scope keywords, which belong to commands which are executed once upon import-time of a script file, which arises the question:

Can script commands which are executed during module loading be compared to preprocessor statements which are executed by the compiler in syntaxes like C?

Proposal

Prefered Solution

Introduce a dedicated keyword.preprocessor scope:

• keyword.preprocessor.control
• keyword.preprocessor.control.conditional
• keyword.preprocessor.control.flow
• keyword.preprocessor.declaration
• keyword.preprocessor.operator
• keyword.preprocessor.other

Pro

• Color schemes need to define keyword.preprocessor only to ensure all preprocessor keywords are highlgihted with a dedicated color.
• All subscopes of control can be used as in normal keywords.

Contra

• Introduces a new 2nd level scope.

Alternative Solution

Just follow the C# approach and scope preprocessor keywords as:

• keyword.control.preprocessor
• keyword.control.preprocessor.conditional
• keyword.control.preprocessor.flow
• keyword.declaration.preprocessor
• keyword.operator.preprocessor
• keyword.other.preprocessor

Pro

• Reuses existing top-level scopes.

Contra

• Color schemes need to define selectors for all listed scopes to highlight preprocessor keywords only. The more the preprocessor subscope is moved to the right, the more scopes a color scheme needs to address.
• The preprocessor subscope breaks scopes like keyword.control.conditional.

Notes

This issue applies to other syntaxes as well.

The original idea came up, while working on Erlang's Typing Language. Both spec and when are scoped via keyword.control in the current internal WIP branch and by changing the color scheme I found both to be scoped as preprocessor, while only spec is meant to be so.

-spec mod:foo({X, integer()}) -> X when X :: atom()

[deathaxe]

The C/C++ syntax definition of Visual Studio Code uses keyword.control.directive for all kinds of preprocessord keywords combined with meta.preprocessor.

scope | keywords
----------------------------------------|----------------------------------------
keyword.control.directive.conditional | #if, #ifdef, #else, #endif, ...
keyword.control.directive.diagnostic | #warning, #error
keyword.control.directive.import | #import
keyword.control.directive.include | #include
keyword.control.directive.<keyword> | #<keyword>

This way all preprocessor directive keywords may be addressed via keyword.control.directive.

This is actually the scope the current _Erlang_ implementation uses.

[wbond]

I'm fine with lumping them all under keyword.control.directive. From looking online, that seems like a more appropriate class than preprocessor. Either way we are going to break backwards compatibility, but it should be fairly minor, and hopefully picked up by keyword.control in most situations.

I think import and preprocessor should probably be distinct? Lots of syntaxes have imports, but they feel pretty semantically different than the preprocessor.

[deathaxe]

I think import and preprocessor should probably be distinct? Lots of syntaxes have imports, but they feel pretty semantically different than the preprocessor.

_Already saw what you mean in C# for instance._

The answer mainly depends on what we understand as import. I personally got confused by the C scopes keyword.control.import.ifdef and friends as it is not clear, whether import means "executed during compile/import time" or whether it means just importing something.

The scope keyword.control.directive feels like a general approach to classify everything which does not belong to normal runtime code and is executed at compile-time (compiler) or import-time (interpreter).

We still have keyword.control.import which should be used for import-like statements only (i.e.: uses, import, from .. import ..). So it's clear to see they are importing something.

All preprocessor like statements (#ifdef, #define, ...) then are scoped as directive. If a syntax like C handles imports via preprocessor, I still would add it to directive to be able to highlight all the same way.

[FHTheron]

I definitely agree that "directive" seems to be the correct term (incl. for #include).

This ties in more with the modules/namespace discussion, but I would almost call the following lines "imports" and they are definitely not preprocessor directives.
using namespace std;
using std::cout;
That might not be a good idea, but it is just another bit of ambiguity regarding the real meaning of import in different languages.

[FichteFoll]

I definitely agree that "directive" seems to be the correct term (incl. for #include).

I would almost call the following lines "imports" and they are definitely not preprocessor directives.

These two statements seem contractionary.

Anyway, I support the sentiment that preprocessor and similar directives should get keyword.control.directive but imports be different in that they impose different semantics in languages where the referenced file isn't inserted into the code literally and don't need to be handled at compile time necessarily. (In the flow keyword issue it was suggested to scope tham as keyword.import. #1228)

[deathaxe]

These two statements seem contractionary.

The main issue is, that no clear distinction between normal runtime code and such which is executed at import time or those which is just a hint for a compiler exists at the moment.

Everything is somehow mixed up with the existing keyword.control or keyword.other scopes

Same issue may exist with meta scopes and is related with the "annotation" discussion, too.

I see two possible main strategies to handle these kind of stuff.

1. Create unique meta.scope rules for each part and reuse existing keywords. This would result in all statements to be highlighted the same way by default. Different colors could be applied by using a combination of meta and keyword scopes. The down side is that constructs like #define IF if wouldn't cause tha last if to be highlighted the same way as #define if both share the same basic keyword.control scope and are spanned by the same meta.preprocessor or something like that. _(Please excuse the trivial and maybe none-real-world-example)_

2. Create unique keyword. ... sub scope structures which address the different types of keywords. A clear "tree" would help to highlight different types of keywords with one selector and avoid conflicts, but may introduce longer/more detailed sub scopes.

Approach 2 is what I had in mind, when opening this issue as it seems to be a more robust way to avoid conflicts.

(In the flow keyword issue it was suggested to scope tham as keyword.import. #1228)

The keyword.control.import is an existing scope being used in several syntaxes (C/C++) to highlight includes or even normal preprocessor directives. The keyword.control.directive is used by Erlang and is the choice in C/C++ like syntaxes of Atom and VS Code to scope preprocessor like statements. The keyword using is scoped as keyword.other.using.cs by VS Code.

The TextMate scope naming guidelines only use keyword.control, keyword.operator and keyword.other as 2nd-level scope. Only those 3 may be addressed in older color schemes.

_From a perspective of backward compatibility I think we should keep using these existing scopes._

So C/C++ would be modified to use keyword.control.directive. ... for all the #... keywords which are "executed" at compile time.

Everything like using in C# or import, from, as in python would be scoped as keyword.control.import as it is executed at import time.

We recently decided to add keyword.declaration as the 4th second level scope.

_From a perspective of short and logical keywords I find keyword.control not optimal for the discussed kind of keywords as control looks like a scope name for normal runtime code._

That's why I find the proposal of keyword.import worth thinking about. But this would mean to also introduce keyword.directive to be consequent.

This would enable color schemes to distinguish between preprocessor/import/runtime just on the base of few 2nd-level scopes

_Compile Time_

• keyword.directive

_Import Time_

• keyword.import

_Runtime_

• keyword.control
• keyword.declaration
• keyword.operator
• keyword.other

The downside might be, that some older color schemes might not address the new scopes and may break highlighting. I checked a few color schemes - all seem to address keyword. But there are far too many to make a safe statement.

[FichteFoll]

In an effort to clean up the keyword.control scope and instead introduce more 2nd level scopes, similar to #1228, I should probably voice that I strongly prefer keyword.directive over yet another keyword.control subscope. (I did put my :+1: in here earlier, but it probably wasn't too clear what I referred to exactly.)

keyword really should be a scope selector matched by all color schemes. Anything within that will be a keyword and should be highlighted to the user, so I believe it is safe to bank on that and introduce new scopes to clean up the current mess.

[deathaxe]

With regards to the last part of https://github.com/sublimehq/Packages/issues/1228#issuecomment-549066281 and my last comment here, I start wondering about whether to merge the two 2nd-level scopes import and directive would be a general solution for scoping "pre-runtime" code without distinction whether it is import-time or compile-time. _(I think both is basically the same.)_

What I have in mind is:

  directive
    conditional   // #if , #else, ...
    declaration   // #define, #pragma, ...
    import        // #include, import, from, using, with, ...
    other

The reason for keyword.directive.declaration rather than meta.preprocessor keyword.declaration is just to be able to highlight #define in another way than any other declaration keyword in the expression part.

Example:

#define MY_CLASS class
   ^ meta.preprocessor keyword.directive.declaration
                   ^ meta.preprocessor keyword.declaration

The class is a keyword.declaration while #define should be something else. Both are covered by meta.preprocessor.