Cataclysm-dda: Can we start getting at least a mandatory 10 minutes of testing before major PR merges?

[Regularitee]

In recent days, both the freezing and the farming patches have broken game systems. Both of these had outstanding issues that could have been found out within a few minutes of testing of normal, non-exceptional gameplay. But they weren't, and instead were merged with the master and broke the game's systems for every player that updated.

I understand these are experimental builds, but can we get even a half-hearted attempt at quality control (at least for MAJOR game system overhauls, like freezing and farming?)

[Lorith]

I can't add anything to this except that I fully agree. I have no problem with odd situations having bugs, or bugs that require heavy testing. But if you would literally catch it in under two minutes of testing normal gameplay related to the feature, there is a serious problem.

[cainiaowu]

It would need some volunteers to checkout the branch and build that branch and do the playtesting, which will consume the precious dev time. There is a reason why only a few people left merging.

Rather complaining, anyone can take this heavy load off the dev by doing this themselves and add feedback in review comments.

[Duhuh123]

I git clone the repository and compile the code to play the game.
If there is a branch for play-tested code (beta), and another branch (alpha) that contains all of the untested patches, I might try new things from time to time.

[cainiaowu]

Just looL at that PR queue, 48 is quite small but it means that the jerkins need to keep 48x(numbers of platforms) different downloadable versions in the server. And the queue can easily go up to 100 back in the "slow" days.
So yeah, just build it yourself maybe a better solution.

[Regularitee]

It would need some volunteers to checkout the branch and build that branch and do the playtesting, which will consume the precious dev time.

It wouldn't be if we just had the authors of the PRs themselves do a final ten minute test on default settings before authorizing the final merger. Particularly if it's only the major overhaul PR mergers (which are, what, twice a month?).

[hameleona]

5 noob cents, to keep in mind, since we get forggoten often:

1. You want more testers, you need noobs. Noobs like me WON'T go and learn how to use github properly. Maybe we make an account and whine. If there is more involved than 5 clicks - they will not test anything.
2. Whatever the simplest way of getting the game is - noobs will play that and bitch, moan and whine. So, no matter how many "experimental" labels you put - what the launcher gives is what's being played.
3. I'm sorry but farming was not tested at all. It's not even meant to be played at default season length. I know you owe nothing to no one, but the freezing hot fresh meat and weeds growing by the minute are things that should have been caught the moment anybody loaded up the game. They got the moment the update hit.
4. If it breaks an old system, noobs will whine. And honestly can you blame them? It's hard to get older builds and there can be 4+ updates in a day, making the revert function in the launcher useless.
5. I actually like the new changes, but some common sense at approving stuff may be welcome. Even I know temperature is broken by default. Why merge something entirely dependent on temperature? Farming got from - plant, fert, forget to FarmSim 2050. It's bound to have major problems.

[Robik81]

Every author should test his work before he makes a PR. And I hope it is the case, actually.

If there are authors who do not, they should imagine potential embarrassment, could help with motivation.

Problem with self-testing is, that authors tend to not do 'dumb' stuff on unconscious level and thus might not find everything even if they try very hard, external feedback is quite important.

Mergers should also do testing before merging, but if you look at the enormous amount of PRs and number of mergers, it is just not possible. To be honest, I think that mergers should do only minimal level of testing. Why? Because I find the occasional 'testing on live subjects' preferable to much longer PR queues.

My suggestion to players who do not want to bother with such stuff is to look out for major PRs and backup their game AND save before upgrading.

[ZhilkinSerg]

If you are thinking that PRs are getting merged without testing - you are wrong. If you you are thinking that testing prevents bugs from appearing in game - you are dead wrong.

[sfsworms]

Maybe a stupid question, but is there a special flag or something for bigger PR that might need more extensive testing so we poor non-devs have an easier time helping with the testing?

[ZhilkinSerg]

Maybe a stupid question, but is there a special flag or something for bigger PR that might need more extensive testing so we poor non-devs have an easier time helping with the testing?

We've added PUBLIC TEST label since, but we really need to configure automatic PR builds to make public PR test process easier.

https://github.com/CleverRaven/Cataclysm-DDA/pulls?q=is%3Aopen+is%3Apr+label%3A%22PUBLIC+TEST%22