When the GitHub app adds a new check on GitHub, it overwrites the previous entry if the stack name is the same.
For example, imagine in PR #1 we have 2 apps, app1 and app2. In each of these apps, we also have Org/development as the stack name.
What will happen is that when app1 runs preview, the check will show the status for org/development, and then when app2 runs, it will update the same status entry from before, but with the status for app2.
This is because the entries are done with the stack names and not the stack + project. As in, it's done with Stack instead of Project/Stack.

Thanks for the bug report. This does seem like part of the fallout of #2039.
This has been fixed in the Pulumi Service codebase. It will be deployed sometime tomorrow. After it goes out, you should start seeing the qualified stack names like website/production and backend/production in the GitHub checks UI.
Woohoo! Thanks for the quick fix!
Most helpful comment
This has been fixed in the Pulumi Service codebase. It will be deployed sometime tomorrow. After it goes out, you should start seeing the qualified stack names like
website/productionandbackend/productionin the GitHub checks UI.