Php-language-server: Caching

[Strate]

Hi! Thank you for this project firstly! It is really good idea & implementation!

I want to suggest you to improve usage experience on large projects. I have 20332 php files, and scanning project on each vscode startup takes too long.

Seems that there is no persistent cache of scan result. It would be good to save scan results between working sessions.

And, seems that scan is single-threaded. I think that you could use n of forks, each one is for 1 machine core.

What do you think about it? How do you think, is it hard to implement? Would you accept any help?

[kaloyan-raev]

Hi and welcome!

I have 20332 php files, and scanning project on each vscode startup takes too long.

How long exactly does it take for you?

If you have Xdebug active, you should disable it. This will give you 5-10x speed up.

Seems that there is no persistent cache of scan result. It would be good to save scan results between working sessions.

We had a persistent cache for a short time. It was an early attempt and it caused some performance issues, so we removed it for now. It is still in the plan to have such cache.

And, seems that scan is single-threaded. I think that you could use n of forks, each one is for 1 machine core.

This makes sense. If you have the skills to implement it, it would be best to create a pull request, so we can review and discuss that particular implementation there.

[felixfbecker]

I had caching implemented https://github.com/felixfbecker/php-language-server/pull/82 but I took it out again because as long as not all request methods are implemented, the cache format will have to change frequently and that would be a breaking change everytime or require us or users to delete the cache on every update. I'm also moving away from accessing the file system directly, so I would rather like to see a caching solution implemented in the LSP.

I thought about using child processes for parsing before, and that would speed it up a lot. It will always be single-threaded though because PHP is single-threaded. So the challenge here is the inter-process communication, you need to serialize the data, send it to the parent and unserialize it. Or maybe use some IPC solution from composer. Spawning one process per file would be too much, so you would want to spawn (as you said) for every CPU core, and that means the IPC needs to support streaming the ASTs of multiple documents. PR would be very welcome here.

[vakata]

I added basic vendor dir caching for personal usage, but I thought I'd share: https://github.com/vakata/php-language-server/tree/vendor-cache
The idea is that the vendor dir is usually quite huge and takes some time to scan, this solution scans the vendor dir only if the hash in composer.lock has changed or if the internal string version is changed - making it simple to invalidate if the php-language-server version is bumped.

[felixfbecker]

@vakata I will implement https://github.com/sourcegraph/language-server-protocol/pull/14 soon with a file system fallback, that will cache each dependency index by the version

[vakata]

Great! :)

[staabm]

you might also consider to activate opcache in the cli (especially on 7.1.*).
http://php.net/manual/en/opcache.configuration.php#ini.opcache.enable-cli

this works pretty good on linux but on windows is not yet stable, so segfaults in some situations.

[felixfbecker]

@staabm

OPcache improves PHP performance by storing precompiled script bytecode in shared memory, thereby removing the need for PHP to load and parse scripts on each request.

There are no "requests" in CLI mode, this would have no benefit

[staabm]

opcache also has a temp-file opcache which in combination with a enabled cli-opcache can speedup things a lot:
http://php.net/manual/en/opcache.configuration.php#ini.opcache.file-cache

[felixfbecker]

Afaik those performance improvements only affect startup time, which is pretty irrelevant for a language server. Also, if a user wants to set this, he is free to do so in php.ini.

[staabm]

as of php 7.1 opcache also has "optimizer" improvements which might influence the runtime after startup.

but as you said, anybody can configure at will.

[TheColorRed]

Multi-threading can be simulated, by creating a php webserver, which could be done via the built in server php comes bundled with. You could then send curl requests to the server with curl_multi_init and curl_multi_exec. This technically isn't multi-threading, but it is probably the best way to simulate it.

[felixfbecker]

@TheColorRed That wouldn't be better than exec + IPC. But there is already a caching PR open #260

[jens1o]

Can we close that?