Pest: Compatibility: Psalm

[AlexMartinFR]

Problem:
It seems Pest doesn't work well with Psalm.

I didn't know this tool from the Vimeo engineering team, but its domain name was created in February 2019, and it's already got 3K stars of Github.
I'm throwing this issue because it could be interesting to see what would be needed to make Pest & Psalm work well together.

Report from our Discord channel*:

Even though the tests are running, my editor is highlighting errors everywhere in the test code. Psalm thinks the test function does not exist.
Ah man, just saw Pest is using PhpStan instead of Psalm. Psalm is way better.
It is more strict about the types, catching way more type issues, which is sort of the whole point of those things
Psalm also makes it much easier to add advanced information about your code in the docblocks quickly and easily for things you would have to write an entire plugin for in phpstan
I won't use php without Psalm. Using php with Psalm is like already being on php version 10

• _Cf Discord, #help, June 2nd, 2020._

[owenvoke]

Psalm has been around for quite a while, but I'm surprised there would be issues with it. Psalm should know that the test() function exists (and does in my test project), and also should not throw errors. However, depending on the level this may change. So the only thing I've had so far is Psalm not being able to find the assertions in higher order proxies, which is because test() and it() are defined as returning TestCall|TestCase|mixed in the doc comments.

PHPStan and Psalm are both great, and have their pros and cons. 👍

I might have a look into this later.

[still-dreaming-1]

I will look into this further since I am familiar with Psalm. I know it is possible to make a separate plugin package that is a Psalm plugin, to help Psalm understand Pest code. It is also possible my editor was just being weird at the time, so I will do more investigation.

If it does turn out that Psalm really is not understanding Pest code, an alternative to making a plugin is for you guys to use Psalm yourself either instead of, or in addition to PHPStan. This would have a dual effect of helping check the types and correctness of your code, and of making sure Psalm will understand Pest tests in other projects.

[still-dreaming-1]

Psalm does know about the functions. This was just my experience while trying to set up pest for the first time. I probably just forgot that the Psalm language server / the Psalm VS Code plugin gets confused when you change too many things on it, like installing a new package. I probably just needed to close and reopen VS Code. The thing is, I thought for sure I conceived of this, and ran Psalm manually from the terminal with the cache disabled, and still saw these error, but I guess I did something wrong, since it is all working now.

You can either close the issue, or leave it open for about a week, and I will report any Psalm issues here in the mean time. It is highly likely I will run into some of the main ones within the next week, if there are any.

[owenvoke]

I think it's worth keeping the issue open for now, it would be good to see if there's any issues that we could easily fix. 👍 Thanks @still-dreaming-1

[still-dreaming-1]

When I use a ->with( similar to the first examples in Datasets, Psalm doesn't know what type the it( or test( returns, so it doesn't recognize ->with as a defined method.

I think even if basic issues like these got resolved with improved annotations, it still might be worth creating a plugin for this for Psalm. They maintain one for phpunit, which allows some pretty cool things that I don't think you could get without a plugin. For example in phpunit, if you use a data provide, Psalm knows if the types returned by the provider match what is needed where it is used. Here is their plugin for phpunit: https://github.com/psalm/psalm-plugin-phpunit/commits/master

[still-dreaming-1]

It turns out I'm going to have to distance myself from responding to these for about the next 3 weeks. I probably won't post anything new or respond until after that. It is likely I will eventually get back to trying out pest with Psalm after that, and should become more engaged at that time.

[owenvoke]

Might take a look at the plugin stuff shortly as I'm quite interested in how it works. 👍

May also give us some ideas for our plugins in Pest.

[still-dreaming-1]

Upon further reflection, I think you might be able to use Psalm templates to add types for this kind of thing. So you might not need a plugin. I could be wrong, but I think plugins are more used for projects that don't want to use Psalm, so someone else maintains a plugin in order to generate the right stubs for the types. But if you add Psalm to pest, it should point out where your types are not totally complete or contradict something, then fixing those errors might be enough for other projects to be able to use pest and get full typing support from Psalm. The advantage of that approach is the pest developers would immediately know when they change some code that triggers a Psalm issue, as it would be part of the QA for pest, possibly running on a CI server. I basically treat Psalm as a compiler for php, as in I think of the issues it reports as compiler errors.

If you try this out, you might be overwhelmed by the number of issues it finds, as it will probably surface some technical debt in the code that would have to be addressed to fix all the issues. Psalm also allows you to suppress all the current issues into a baseline, and then just fix new ones that come up from new development. Anyway, I really am going to try not to respond to these too often for 2 or 3 weeks, but I wanted to at least say this last thing in case I was leading you down a bad path with the plugin idea. I think starting out with using Psalm in a project rather than creating a plugin right away is also a good approach to start getting familiar with Psalm.

I used to use both PHPStan and Psalm in a code base I maintain, so it is possible to use both. Sometimes they cover each other's blind spots. Overall after I got comfortable with both, I found Psalm provided better protection and was easier to maintain, and continuing to use both was no longer worth it in that project, so now it just uses Psalm.

[Synchro]

I've run into this too; psalm doesn't seem to know that pest isn't phpunit:

it(
    'rejects an empty header',
    function () {
        $message = new Header('');
    }
)->throws(InvalidArgumentException::class);

ERROR: PossiblyUndefinedMethod - tests/Unit/HeaderTest.php:11:4 - Method PHPUnit\Framework\TestCase::throws does not exist (see https://psalm.dev/108)
)->throws(HeaderException::class);

I guess this might be a result of accessing the files directly rather than in the context of pest, so for now I'm not letting psalm check my tests. FWIW, phpstan doesn't have any problem with them.

[nunomaduro]

No plans to have this.

[simPod]

Anyone found a solution to this? Obviously the typehint @return TestCall|TestCase|mixed is wrong, not sure why this is closed.

return new HigherOrderTapProxy
...
return new TestCall

I don't see any TestCase there