Psalm: [Feature proposal] Running Psalm with an error baseline

[ErikBooijCB]

Hi! I have an idea for a feature that would help us a lot in implementing Psalm in our large, legacy codebase. The problem we currently face, is that running Psalm on the settings we would like, yields roughly ~4,000 errors (don't ask). Too much to fix at once and therefore a huge road block for making Psalm a blocking step in our CI.

The idea I have is to be able to initialize a baseline with existing errors and ignore those in processing the results. It would look something like this:

$ ./vendor/bin/psalm --set-baseline <filename>

It would generate and XML file (I'd prefer JSON, but to keep it in line with the configuration file) with the current errors. Running Psalm with the baseline would be:

$ ./vendor/bin/psalm --with-baseline <filename>

This would allow us to make Psalm blocking right away, while allowing us to work on these existing issues as time permits.

To be clear, I'm not asking you to build this, I would be putting in that effort, I just want to gauge if you'd be open to the feature, when the PR comes and if so, if you have any suggestions/tips I should keep in mind.

[muglug]

First, this is a brilliant, amazing idea.

I foresee a file like

baseline.xml:

<files>
    <file name="src/A/B.php">
        <PossiblyUndefinedVariable />
    </file>
    <file name="src/A/C.php">
        <UndefinedThisPropertyFetch />
    </file>
</files>

All these issues show up as info when you run Psalm - you still see them, they don't go away, and if you use an LSP plugin you'll be able to see them in your editor.

Additionally when you run with --with-baseline Psalm will update the baseline file automatically when you remove issues in the files it covers. That allows for seamless code improvement.

[muglug]

yields roughly ~4,000 errors (don't ask)

This is a shame-free zone! Also, when running on Vimeo's codebase:

1 errors found
------------------------------
19679 other issues found.

[ErikBooijCB]

Awesome, thanks! Will be taking a look at it shortly and I'll either post issues I run into here, or if all is smooth, open a PR 👍

[ErikBooijCB]

Merged in https://github.com/vimeo/psalm/pull/1049