Semgrep: Parse Error in files from popular Frameworks/Libraries

[erwanlr]

semgrep v0.28.0 fails to parse some files from popular PHP Frameworks/Libraries.

To reproduce, download the latest version / master code via the link given and run semgrep again it

• https://github.com/sebastianbergmann/phpunit

semgrep warn: parse error
  --> phpunit-master/src/Framework/Constraint/Operator/LogicalNot.php:12
12 | use function array_map;
   |     ^^^^^^^^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse LogicalNot.php as php

semgrep warn: parse error
  --> phpunit-master/src/Framework/TestCase.php:12
12 | use const LC_ALL;
   |     ^^^^^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse TestCase.php as php

semgrep warn: parse error
  --> phpunit-master/src/TextUI/XmlConfiguration/Group/Groups.php:39
39 |     public function include(): GroupCollection
   |                     ^^^^^^^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse Groups.php as php

[...]
ran 3 rules on 323 files: 0 findings
164 files could not be analyzed;

• https://github.com/PHPOffice/PHPWord

semgrep warn: parse error
  --> PHPWord-develop/src/PhpWord/Writer/Word2007/Element/Shape.php:21
21 | use PhpOffice\PhpWord\Element\Shape as ShapeElement;
   |                               ^^^^^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse Shape.php as php

semgrep warn: parse error
  --> PHPWord-develop/src/PhpWord/Element/Shape.php:20
20 | use PhpOffice\PhpWord\Style\Shape as ShapeStyle;
   |                             ^^^^^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse Shape.php as php

semgrep warn: parse error
  --> PHPWord-develop/src/PhpWord/Writer/Word2007/Style/Shape.php:25
25 | class Shape extends AbstractStyle
   |       ^^^^^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse Shape.php as php

semgrep warn: parse error
  --> PHPWord-develop/src/PhpWord/Style/Shape.php:26
26 | class Shape extends AbstractStyle
   |       ^^^^^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse Shape.php as php

[...]
ran 3 rules on 284 files: 0 findings
4 files could not be analyzed;

• https://github.com/symfony/symfony

semgrep warn: parse error
  --> symfony-5.x/src/Symfony/Bridge/Doctrine/Validator/DoctrineLoader.php:52
52 |         } catch (MappingException | OrmMappingException $exception) {
   |                                   ^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse DoctrineLoader.php as php

semgrep warn: parse error
  --> symfony-5.x/src/Symfony/Component/HttpClient/HttpClientTrait.php:305
305 |             return $generatorToCallable((static function ($body) { yield from $body; })($body));
    |                                                                               ^^^^^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse HttpClientTrait.php as php

semgrep warn: parse error
  --> symfony-5.x/src/Symfony/Contracts/Service/ServiceLocatorTrait.php:117
117 |         return new class($message) extends \InvalidArgumentException implements NotFoundExceptionInterface {
    |                    ^^^^^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse ServiceLocatorTrait.php as php

semgrep warn: parse error
  --> symfony-5.x/src/Symfony/Component/Cache/Adapter/PhpArrayAdapter.php:92
92 |             get_from_pool:
   |                          ^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse PhpArrayAdapter.php as php

semgrep warn: parse error
  --> symfony-5.x/src/Symfony/Component/VarDumper/Dumper/CliDumper.php:455
455 |             goto href;
    |                  ^^^^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse CliDumper.php as php

semgrep warn: parse error
  --> symfony-5.x/src/Symfony/Bridge/Doctrine/PropertyInfo/DoctrineExtractor.php:149
149 |                         case Types::ARRAY:
    |                                          ^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse DoctrineExtractor.php as php

semgrep warn: parse error
  --> symfony-5.x/src/Symfony/Component/DependencyInjection/Loader/Configurator/Traits/PublicTrait.php:19
19 |     final public function public(): self
   |                           ^^^^^^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse PublicTrait.php as php

semgrep warn: parse error
  --> symfony-5.x/src/Symfony/Component/HttpClient/NativeHttpClient.php:54
54 |             [, $this->defaultOptions] = self::prepareRequest(null, null, $defaultOptions, $this->defaultOptions);
   |              ^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse NativeHttpClient.php as php

semgrep warn: parse error
  --> symfony-5.x/src/Symfony/Component/OptionsResolver/OptionConfigurator.php:69
69 |     public function default($value): self
   |                     ^^^^^^^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse OptionConfigurator.php as php

semgrep warn: parse error
  --> symfony-5.x/src/Symfony/Component/DependencyInjection/Loader/Configurator/DefaultsConfigurator.php:65
65 |     final public function instanceof(string $fqcn): InstanceofConfigurator
   |                           ^^^^^^^^^^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse DefaultsConfigurator.php as php

semgrep warn: parse error
  --> symfony-5.x/src/Symfony/Bridge/Doctrine/Types/AbstractUidType.php:47
47 |             return $this->getUidClass()::fromString($value);
   |                                        ^^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse AbstractUidType.php as php

semgrep warn: parse error
  --> symfony-5.x/src/Symfony/Component/HttpClient/Response/AmpResponse.php:228
228 |             if (null === $response = yield from self::getPushedResponse($request, $multi, $info, $headers, $options, $logger)) {
    |                                                 ^^^^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse AmpResponse.php as php

[...]
ran 3 rules on 3058 files: 0 findings
211 files could not be analyzed;

[aryx]

Thx for the report!

[aryx]

Is there one parsing bug you really would like us to fix? Just to prioritize things.
Is this high-priority for you to be able to use semgrep?

[erwanlr]

Is there one parsing bug you really would like us to fix? Just to prioritize things.

Not really, they are all on the same level for me I would say. It happened when trying to parse some code I can't post here, but managed to find open source code with the same parsing issues.

Is this high-priority for you to be able to use semgrep?

Would not say it's a high one, but it's a bit annoying to not be able to fully parse some code to scan. Can still use semgrep on basic code.

[rezen]

I'm curious, would it be possible to have partial parsing? For example if the parser runs into something that it doesn't recognize, it could label it as such and continue on? I imagine this would be challenging around not being able to parse language constructs that create separate blocks/scopes (such as new class(...)). I suspect this probably would not work, but I figure why not throw ideas out there!

[aryx]

PHPWord now parses correctly at 100%
(using the develop version of semgrep)

[aryx]

phpunit parsing should improve now. All the parsing errors in PHPunit mentioned above have been fixed
(but there are some still remaining).

[aryx]

Use of anonymous classes has been fixed, see https://github.com/returntocorp/semgrep/issues/2003

[aryx]

Type annotations on closures are not correctly supported. See https://github.com/returntocorp/semgrep/issues/2001

[aryx]

Use grouped declarations are now supported: https://github.com/returntocorp/semgrep/issues/2004

[aryx]

Most errors reported above are now fixed. I'll close for now; the popular framework libraries
are still not parsing fully at 100% but we can create a separate issue with new errors.

[erwanlr]

Thank you very much @aryx !