Php_codesniffer: JS Tokenizer generates errors when processing some decimals

Created on 10 Mar 2016 · 9Comments · Source: squizlabs/PHP_CodeSniffer

In trying to figure out why testing with my custom standard is failing to complete, I ran a quick test against the Pear standard in PHPCS just to make sure things are running right. I ended up getting the following errors at about 74%

PHP Notice:  Undefined index: bracket_closer in /CodeSniffer/Tokenizers/JS.php on line 1062

PHP Stack trace:
PHP   1. {main}() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:0
PHP   2. PHP_CodeSniffer_CLI->runphpcs() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:25
PHP   3. PHP_CodeSniffer_CLI->process() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:104
PHP   4. PHP_CodeSniffer->processFiles() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:886
PHP   5. PHP_CodeSniffer->processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:619
PHP   6. PHP_CodeSniffer->_processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1714
PHP   7. PHP_CodeSniffer_File->start() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1836
PHP   8. PHP_CodeSniffer_File->_parse() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/File.php:471
PHP   9. PHP_CodeSniffer_File::tokenizeString() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/File.php:689
PHP  10. PHP_CodeSniffer_Tokenizers_JS->processAdditional() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/File.php:1447

This error

PHP Notice:  Undefined index: content in /CodeSniffer/Fixer.php on line 139

PHP Stack trace:
PHP   1. {main}() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:0
PHP   2. PHP_CodeSniffer_CLI->runphpcs() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:25
PHP   3. PHP_CodeSniffer_CLI->process() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:104
PHP   4. PHP_CodeSniffer->processFiles() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:886
PHP   5. PHP_CodeSniffer->processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:619
PHP   6. PHP_CodeSniffer->_processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1714
PHP   7. PHP_CodeSniffer_File->start() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1836
PHP   8. PHP_CodeSniffer_Fixer->startFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/File.php:472

This error

PHP Notice:  Undefined index: bracket_closer in CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 951

PHP Stack trace:
PHP   1. {main}() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:0
PHP   2. PHP_CodeSniffer_CLI->runphpcs() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:25
PHP   3. PHP_CodeSniffer_CLI->process() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:104
PHP   4. PHP_CodeSniffer->processFiles() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:886
PHP   5. PHP_CodeSniffer->processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:619
PHP   6. PHP_CodeSniffer->_processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1714
PHP   7. PHP_CodeSniffer_File->start() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1836
PHP   8. Generic_Sniffs_WhiteSpace_ScopeIndentSniff->process() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/File.php:567

this error

PHP Notice:  Undefined index: line in CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 952

PHP Stack trace:
PHP   1. {main}() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:0
PHP   2. PHP_CodeSniffer_CLI->runphpcs() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:25
PHP   3. PHP_CodeSniffer_CLI->process() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:104
PHP   4. PHP_CodeSniffer->processFiles() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:886
PHP   5. PHP_CodeSniffer->processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:619
PHP   6. PHP_CodeSniffer->_processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1714
PHP   7. PHP_CodeSniffer_File->start() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1836
PHP   8. Generic_Sniffs_WhiteSpace_ScopeIndentSniff->process() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/File.php:567

this error

PHP Notice:  Undefined offset: 2845 in CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 217

PHP Stack trace:
PHP   1. {main}() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:0
PHP   2. PHP_CodeSniffer_CLI->runphpcs() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:25
PHP   3. PHP_CodeSniffer_CLI->process() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:104
PHP   4. PHP_CodeSniffer->processFiles() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:886
PHP   5. PHP_CodeSniffer->processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:619
PHP   6. PHP_CodeSniffer->_processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1714
PHP   7. PHP_CodeSniffer_File->start() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1836
PHP   8. Generic_Sniffs_WhiteSpace_ScopeIndentSniff->process() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/File.php:567

and lots more errors
see Travis-ci RAW log for details and all the errors
Also Failing with similar errors using the PSR2 standard
And with the PSR1 standard
And with the PHPCS standard

a file that you analyze: Testing against this Repo on Travis-CI
standard definition you're using: PEAR (for this test only)
used PHP_CodeSniffer version: Current stable 2.5.1
used PHP version: 5.6.5

Bug

Source

photodude

All 9 comments

This is the file that is causing the errors: media/editors/codemirror/addon/search/search.js

gsherwood on 11 Mar 2016

Smallest bit of code to replicate:

function() {
  if (condition)
    foo = .4
}

gsherwood on 11 Mar 2016

I made a tweak to the file you mentioned to bypass this bug for the moment. (just added a 0 in front of the decimal as a temporary work around)

After bypassing the bug, Looks like there is another file in this repo throwing an error in the JS Tokenizer
/media/editors/codemirror/mode/soy/soy.js

photodude on 11 Mar 2016

After bypassing the bug, Looks like there is another file in this repo throwing an error in the JS Tokenizer
/media/editors/codemirror/mode/soy/soy.js

Yep, I'm on that now. It's a regex tokenizing issue.

gsherwood on 11 Mar 2016

Both those issues are now fixed. Are you able to give it a test with your custom standard?

gsherwood on 11 Mar 2016

🎉1

It took some work, and a lot of trial and error, but I finally figured out getting the PHPCS master branch and the custom standard in a state that I can run them on Travis-CI.

Looks good, for the fix on the second file

I'll have to run another test with my temporary workaround for the decimal bug removed. But it's probably not an issue anymore.

photodude on 11 Mar 2016

Thanks for testing. I'll close this issue, but please post again if it's not fixed for you.

gsherwood on 11 Mar 2016

🎉1

Will do, Thanks for addressing this issue. (now I just need to fix the issue with the FileCommentSniff in our custom standard, maybe someday before phpcs 3 we'll get the our custom standard out of a beta state)

photodude on 11 Mar 2016

Hello,

I'm running into a simila issue:

Script ./bin/phpcs --report=checkstyle --ignore=*/om/*,*/map/*,*/Ressources/*,*/Tests/* --report-file=build/logs/checkstyle.xml src/ handling the phpcs event returned with an error

  Error Output: PHP Notice:  Undefined index: bracket_closer in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Tokenizers/JS.php on line 1071                          
  PHP Notice:  Undefined index: content in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Fixer.php on line 139                                                        
  PHP Notice:  Undefined index: bracket_closer in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 988  
  PHP Notice:  Undefined index: line in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 989            
  PHP Notice:  Undefined offset: 8 in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 218              
  PHP Notice:  Undefined offset: 8 in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 238              
  PHP Notice:  Undefined offset: 8 in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 345              
  PHP Notice:  Undefined offset: 8 in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 563              
  PHP Notice:  Undefined offset: 8 in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 813              
  PHP Notice:  Undefined offset: 8 in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 814              
  PHP Notice:  Undefined offset: 8 in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 822              
  PHP Notice:  Undefined offset: 8 in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 823              
  PHP Notice:  Undefined offset: 8 in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 832              
  PHP Notice:  Undefined offset: 8 in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 838              
  PHP Notice:  Undefined offset: 8 in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 839              
  PHP Notice:  Undefined offset: 8 in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 873              
  PHP Notice:  Undefined offset: 8 in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 907              
  PHP Notice:  Undefined offset: 8 in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 986              
  PHP Notice:  Undefined offset: 8 in /home/ph/www/squarebreak/vendor/squizlabs/php_codesniffer/CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/DisallowTabIndentSniff.php on line 80

I will try to find out which JS file is causing the issue.