Php-parser: Dose it parse any project with a lot of file?

[Rivendall]

Hello.
I think, this is a good library an useful.
I have a problem.
When i use single file, out put is correct, but use one project include some file with commands for example requier and include, php parser create AST for any file separately and then create cfg for any file separately.
Can php parser , parse project and recognize include or require?

Thank you.

[ichiriac]

Hi @Rivendall,

Depends on what you try to achieve, but maybe this project would be simpler to use as it scans recursively files :

https://github.com/glayzzle/php-reflection

By the way, I've benchmarked it on magento2 repository, 18500 files parsed and put in memory with php-reflection project in about 35sec (but mostly depends on you disk performance mine was an SSD), and memory footprint about 400Mb.

Hope it helps :smile:

[Rivendall]

Thank you for your answer.
I need to parser for php, that get php code and give AST for its. Because, I want to know, how to work?
for example any parser, get one project file by file, and create AST for any file separately. Dose your parser do its in the same way? and I want to know, how to parse include, require and ... expression.
Thank you.

[ichiriac]

for example any parser, get one project file by file, and create AST for any file separately. Dose your parser do its in the same way?

Yes, it does.

and I want to know, how to parse include, require and ... expression.

Ok, you need some help for bootstraping something. If you haven't do it yet, start a new project on github and let me know its URL, then I will be able to help you more in details.

In order to start, you can take a look at the AST format here : http://glayzzle.com/php-parser/#demo

For example, on this kind of code :

<?php include('something.php');

The resulting AST would be :

{
  kind: 'program',
  children: [
    {
      kind: 'include', // it's the same node kin for require & include
      once: false,
      require: false, // if true means it's a require statement
      target: {
       kind: 'parenthesis',
       inner: {
         kind: 'string',
         value: 'something.php'
       }
      }
    }
  ]
}

I've made a sample code in order to show you how to parse a list of PHP files and do what ever you what with them : https://gist.github.com/ichiriac/9b622420f3af9da370b2cb0706b19d34

Hope it helps

[Rivendall]

Thank you for your help.
OK, I'll start a new project coming soon and use of your help. I'll need your help.
I have another question. Do you know about Static code analyzer? for example RIPS or WAP and etc.
Someone of Static code analyzers use php parser for parse code after analysis. I don't understand how detect vulnerability in one file that its input is other file and they use include for their join.
Do you understand my question?

[ichiriac]

OK, I'll start a new project coming soon and use of your help. I'll need your help.

Glad to help you if I can. You can chat here : https://gitter.im/glayzzle/Lobby, it will be more easy to discuss.

I have another question. Do you know about Static code analyzer? for example RIPS or WAP and etc.

No, sorry, I've never used them

Someone of Static code analyzers use php parser for parse code after analysis. I don't understand how detect vulnerability in one file that its input is other file and they use include for their join. Do you understand my question?

Yes I understand the question. You should not focus on includes because that's not a safe way to detect dependencies (autoload mechanisms, or dynamic includes, or function encapsulated includes, ini setted headers files ...etc).

Instead, like in php-reflection, you should scan every file, and then do the static analysis.

Let me give an approximative example (a way to go, but could be others solutions) :

<?php
// this is the index.php file :
$ok = isAdminAuth($_GET['pwd']);
if (!$ok) die('Go away !');

And another file (included somehow) :

<?php
function isAdminAuth($pwd) {
  return mysql_query("SELECT * FROM users WHERE user = 'admin' AND password = '".$pwd."'");
}

If you want to detect a possible (obvious) sql injection, you may do 2 passes :

• First pass scan every file and detect potential dangerous functions
• Second pass, again scan every file, and when a call is done, check if the variable can be harmfull

At the first pass, you detect function declarations, scan their body to detect dangerous functions like mysql_query. If the function call contains expressions that involves a variable, then check if it's passed from arguments. If so, you create in memory a list of functions and for each the list of harmfull arguments positions.

At the second pass, you scan variables globals variables like $_POST, $_GET, $_REQUEST, $_COOKIE, $_SERVER and the most tricky par their usage.

You can say a variable becomes safe when a sanitizing function is used like mysql_string_escape for example.

Note, doing it with passes allows to avoid putting in memory AST but only structures you need to track, so you project will be able to scan almost illimited codebase size files. You will need to parse twice each file so it may be slower (but take in account that it takes 40sec for 18 000 files, so duration may not be a problem))

You can get some examples here :

• https://www.npmjs.com/package/evalscan : it's using php-parser in order to detect eval/preg calls, the project is not on github so you may install it with npm in order to get the source code
• https://github.com/DaGhostman/ts-php-inspections : this one makes static analysis in order to detect code optimisations (a kind of phpMessDetector)

This kind of tool is not so easy to do because there is plenty of edge cases, so in order to avoid that your scan code to become too complex, the structure of your API will have to be well written from start, so you may start by writing your test cases first ...

I close the issue as it's more a conversation, we can continue on the chat if you want to.

[Rivendall]

OK. Sure, I need your help. Thank you.