Solidity: Add new keywords to syntax highlighters

[chriseth]

We will add new keywords with version 0.5.0 and of course we would like to have nice syntax highlighting for them.

In which places do we need to add them?

• [x] docs/utils/SolidityLexer
• [ ] Remix's ACE highlighter

[chriseth]

I am especially lost in how Solidity syntax highlighting ends up in github eventually. Pinging @veox, @oktapodia and @davidhq because I think they might be able to help.

[axic]

docs/utils/SolidityLexer and Remix's ACE highlighter are two things we "maintain"

[veox]

Linguist Github uses https://github.com/davidhq/SublimeEthereum for the grammar definitions, as a git submodule pulled in by linguist.

@davidhq has not responded to a notification, so I'm not sure he's aware of the fact; but there has been some activity in the repo a month ago, so a PR might just do.

---

Thanks for pinging, BTW. I'll try to update the plug-in Pygments lexer in a more timely fashion than usually.

(The Rouge lexer has not been merged, so nothing-to-do there.)

[davidhq]

I'm finally able to join this conversation, sorry for some delay. Yes, I saw the notification but then didn't investigate in detail how everything works now and in fact how new syntax would get into github highlighting in he future. I will try to see that now but yes, any PR would get merged and if new things are taken from this repo, to make it quicker you could hint at them and I'd do my best to include. So before more typing, I should understand the recent updates more in a day or two.

[davidhq]

Or also if it helps, to add some people as collaborators directly to https://github.com/davidhq/SublimeEthereum or someone can fork this repo to https://github.com/ethereum and I make a new reference from packagecontrol.io (where Sublime plugin is installed from).

I missed this important part:

...for the grammar definitions, as a git submodule pulled in by linguist.

ok that then means my repo is a reference and you can all propose how to do it best as per previous paragraph.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

__This issue now has a funding of 0.15 ETH (106.28 USD @ $708.52/ETH) attached to it.__

• If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $8,601.63 more funded OSS Work available on the Gitcoin Issue Explorer

[csggene3]

I have little applied experience but would like to make an attempt at a practical contribution. I won't "start" on the gitcoin issue page unless given the go ahead here, but I'll begin studying the project and syntax. I thought making small additions like tags might be more plausible than attempting to "fix" small issues, but I'm happy to be told otherwise. thx much. cheers.

[davidhq]

Actually for now the project should be up to date, can you check? I updated about a week ago, there should not be any missing keywords. Linguist probably hasn’t auto-updated yet, I’ll check in a few hours when I get home.

I plan to bring the project to new sublime syntax engine before trying (others welcome!) any upgrades like for example special coloring of function modifiers. This was impossible currently.

[davidhq]

I checked, last release 6.1.0 of Linguist was 27 days ago, seems they release every 2-3 months or so. You can see currently referenced commits for all pulled external grammars here: https://github.com/github/linguist/tree/master/vendor/grammars

Since release 6.1.0 this commit was done on SublimeEthereum package: https://github.com/davidhq/SublimeEthereum/commit/114080d4dbf5b51acd9ffd5e12513df333219b3a (some missing keywords and lots of polishing improvements).

So for current version 0.4.23/24 of Solidity it should be up to date, for upcoming keywords or other updates in 0.5+ I will follow the changes .. if things are already known but not in docs, any pull requests are welcome... best is for smaller things until syntax highlighter is refactored to new format... In any case, if this is done in a month or so, it should be a part of next Linguist release.

[chriseth]

The first step here would be to create a list of syntax highlighting tools for solidity that are out there, then we should add the new 0.5.0 keywords like constructor and emit (perhaps others).

[davidhq]

So in SublimeEthereum these two are already added, for Atom the syntax highlighting is/was auto-generated from SublimeEthereum as well, I have written to the author: https://github.com/caktux/language-ethereum/issues/4 and just now again: https://github.com/caktux/language-ethereum/issues/18

Not sure if this is maintained, maybe you can try contacting him as well so in the future it's more streamlined?

Regarding other places, I don't know, Remix uses their own highlighter which doesn't work via Regex-es...

About Solidity docs - nothing is used there (no colors) - should I investigate how could this be done, possibly also based on SublimeEthereum package? UPDATE: can it be done with Linguist already?

[chriseth]

The solidty docs do use highlighting. The module is inside docs/utils. If you could replace this with something else, it would be great!

[veox]

The solidty docs do use highlighting. The module is inside docs/utils. If you could replace this with something else, it would be great!

I see the docs use Sphinx; it could use the pygments-lexer-solidity package. Will submit PR.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

__Work has been started__.

These users each claimed they can complete the work by 8 months, 3 weeks ago.
Please review their action plans below:

1) raphaelhuefner has started work.

Questions

• https://github.com/ethereum/solidity : Why can't I find any .. code-block:: solidity in /docs/*.rst ?
• https://github.com/ethereum/remix-ide : I could not recognize any test coverage for the existing syntax highlighting. Am I missing something obvious here?

Action Plan

• Collect new keywords from here: https://github.com/ethereum/solidity/pull/3627/files#diff-526895adda8a40510dd8574b69c948d7
• For /docs/utils/SolidityLexer.py:
  
  
  
  • Match all new keywords with appropriate pygments' Keyword Token subtypes (from this list: http://pygments.org/docs/tokens/#keyword-tokens ) ... for this I will need to double-check with either @chriseth or @axic if I got the classification right
  
  
  • Insert them into the list of root tokens, according to subtype: https://github.com/ethereum/solidity/blob/develop/docs/utils/SolidityLexer.py#L50-L81
  
  
  • Create a test .rst file with examples for all new keywords in .. code-block:: solidity blocks
  
  
  • Run sphinx-build -b html and test that the new keywords are recognized and correctly classified.
  
  
• For Remix's ACE highlighter:
  
  
  
  • Insert new keywords into https://github.com/ethereum/remix-ide/blob/master/src/app/editor/mode-solidity.js#L60 and https://github.com/ethereum/remix-ide/blob/master/src/app/editor/mode-solidity.js#L76 and possibly a few other places I haven't figured out yet.
  
  
  • Test syntax highlighting manually on my local remix-ide instance.
  
  

Learn more on the Gitcoin Issue Details page.

[raphaelhuefner]

I'll hold off on the /docs/utils/SolidityLexer.py part until https://github.com/ethereum/solidity/pull/4180 is decided either way. I'll do the Remix highlighter first.

[gitcoinbot]

@raphaelhuefner Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] warning (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

@raphaelhuefner due to inactivity, we have escalated this issue to Gitcoin's moderation team. Let us know if you believe this has been done in error!

• [x] warning (3 days)
• [x] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[vs77bb]

Hi @raphaelhuefner are you waiting for guidance here from @chriseth or is this in progress?

[raphaelhuefner]

Hi @vs77bb this is still in progress, sorry for not getting back to you earlier!

[gitcoinbot]

@raphaelhuefner Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] warning (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[raphaelhuefner]

@gitcoinbot Still working on it: Started discussing a change of plans for the Remix IDE: https://github.com/ethereum/remix-ide/pull/1333#issuecomment-395038412

[gitcoinbot]

@raphaelhuefner. :wave: thanks for the atMention, but you need to install @gitcoinbot on this repo for me to be able to respond. More details in the documentation.

:v:
@gitcoinbot

[vs77bb]

@raphaelhuefner Thanks for the update!

[gitcoinbot]

@raphaelhuefner Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] warning (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[raphaelhuefner]

Yes, I'm still working on this. I agreed with @yann300 to make the ACE syntax highlighting into a NPM module of it's own:

I'm currently implementing a way to test the highlighting via editor.getSession().getTokens() (see https://ace.c9.io/#nav=api&api=edit_session )

[gitcoinbot]

@raphaelhuefner Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] warning (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[vs77bb]

Thanks for the update @raphaelhuefner; snoozed Gitcoin Bot for 10 days here.

[raphaelhuefner]

Status update for the ACE part: https://github.com/ethereum/remix-ide/issues/1359#issuecomment-400242538

My hope is that the tokenization test data could prove useful for other syntax highlighters as well... time will tell, I guess.

[raphaelhuefner]

Still working on the ACE part: https://github.com/ethereum/remix-ide/issues/1359#issuecomment-401518222

[raphaelhuefner]

Currently battling nested parentheses inside function definitions: https://github.com/ethereum/remix-ide/issues/1359#issuecomment-402183455

[raphaelhuefner]

Got the nested parentheses resolved: https://github.com/ethereum/remix-ide/issues/1359#issuecomment-404217516 ... next up is re-integration with Remix IDE

[raphaelhuefner]

Got a version 0.1.0 of the NPM package out: https://www.npmjs.com/package/ace-mode-solidity/v/0.1.0

Here is the PR against Remix IDE to start using that package: https://github.com/ethereum/remix-ide/pull/1415

[axic]

@vs77bb I am a bit confused, it seems there are two bounties now: this and https://github.com/ethereum/remix-ide/issues/1359.

The Remix changes are covered by the Remix bounty I guess.

The Solidity changes (local to this repo) were solved by #4180.

[vs77bb]

Hi @axic I was also slightly confused, sorry for the delay 😄

I just paid out @raphaelhuefner for his great work diagnosing and solving the Remix bounty. I'm not sure if @veox is interested in a Gitcoin bounty, but I am happy to pay him out for his work on #4180 as well. Thoughts on a bit of ETH for your work @veox? 🙂

[veox]

@vs77bb I don't think I've spent more than a couple hours on the whole thing (the PR took 20 minutes at first judging from timestamps, using prior work (available regardless of this issue); the rest was figuring out NatSpec peculiarities).

I think @raphaelhuefner has spent much more time with the Remix IDE part, as much as "original work" pertaining to this issue is concerned.

Perhaps this would finally "help" me register on gitcoin, though... (EDIT: Ha! It uses OAuth or something, and I've already logged in previously.)

---

Considering the amount of issue-specific work done, and the size of the bounties at current ETH valuation, I wouldn't mind it all going to @raphaelhuefner, if only to expedite the resolution of these bounties (less fiddling with switching assignees).

[chriseth]

Are there any other syntax highlighters that need to be handled before this can be closed?

[chriseth]

Closing because all tasks pertaining to this repository are done.

[gitcoinbot]

⚡️ A tip worth 0.15000 ETH (27.7 USD @ $184.66/ETH) has been granted to @raphaelhuefner for this issue from @vs77bb. ⚡️

Nice work @raphaelhuefner! Your tip has automatically been deposited in the ETH address we have on file.

• $31853.28 in Funded OSS Work Available at: https://gitcoin.co/explorer
• Incentivize contributions to your repo: Send a Tip or Fund a PR
• No Email? Get help on the Gitcoin Slack

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

__The funding of 0.15 ETH (27.7 USD @ $184.66/ETH) attached to this issue has been approved & issued.__

• Learn more on the Gitcoin Issue Details page
  
  
  
  • Questions? Checkout Gitcoin Help or the Gitcoin Slack
  
  
  • $31,853.28 more funded OSS Work available on the Gitcoin Issue Explorer