Valkyrie: Score moderation process and average duration

[antontimmermans]

NOTE: this is not a bug report for the Valkyrie app, but a review process proposal for the scoring system.

The scoring of the scenario's is a big advantage of Valkyrie. It gives users the opportunity to give feedback on what they like or dislike about the scenario, and scenario writers (or scenario translators) can use the feedback to improve the scenario or translation.
The average play time calculation is also very helpful.

However, sometimes scenario are scored extreme low, and from the variable log or play times you can see that this was not a full play, but just someone misusing the system to target a scenario's score.
Also sometimes there are extreme time recordings like 23 hrs or 10 minutes game play for a scenario that normally runs on average 80 minutes. These outlier times will mess-up the average play time calculation.

There for I have the following proposal as a process:

• Scenario writers that detect these weird situations, can leave a comment in the google spreadsheet that collects the scores in the cell that they consider a wrong value, and a proposal on what the correction should be.
• On a regular basis these comments are reviewed by (a to be selected group of reviewers.
• Reasons for an correction request could be:
  
  
  
  • An extreme Low or extreme High time recording to be corrected to the current average playtime
  
  
  • disable a review if it is clear that is not a normal play review (for example: play time is 10 minutes, #rounds played is 2, a complaint about investigator dead not being handled while other parameter indicate it was (and a test proves it works properly))
  
  
• Not a reason for correction:
  
  
  
  • Negative Feedback or score when all other parameters suggest that it is a normal play session. (#rounds played roughly the same as all other play sessions of the scenario, similar time played)
  
  

Each modification is documented with a comment from the reviewers.

To prevent that this is to much administration for one person, this load of this review task should done by a trusted group of reviewers. I believe current people capable of modifying the spreadsheet are @scrubbless and @BenRQ. They could decide if this group should be extended, and who that could be.

This idea was shortly discussed on discord with @scrubbless

[BenRQ]

Thank you @antontimmermans for this clear proposal.

I've been doing this filter/fix on the statistics file from time to time but not enough to make those stats fully reliable.

I have a proposal to improve this system, and avoid manual intervention.
Instead of using "real" average on score and average time played, we could use a mean score without outliers.
It's quite easy to do, and we could exclude the 5 or 10% extremes values.

The score and average time are all calculated with scripts.
So I can easily change it and it would apply for all values instantaneously.

Tell me what you think about it.

[scrubbless]

I like the proposal, however I want to be clear about my view.

No score or review should be modified, if there is evidence big foul play they can be deleted wholesale (rename the scenario name ideally). I don't want score tampering, that information should be read only and non tangible.

The timing system is off and I wonder if it's to do with the save system? If that can be overhauled it would save a lot of manual intervention.

[BenRQ]

For the strange timings, I'm pretty sure that some people have such a long game that they pause their gaming session, let the application run in background and go back to the game after a day (or a week).

[antontimmermans]

For the game duration, it may be feasible to detect outliers automatically as per your proposal. But you would need sufficient plays on a scenario to know what is the real average and what the outliers are.
Pausing the game is likely the reason for some of the long playtimes but they do influence the average playtime, leaving a wrong calculation of actual average playtime (pausing is not playing). I don't think it is easy to implement a pause detection in the game. For example, detect running in the background, or time between events (clicked or otherwise triggered) exceeds 15 minutes.

The automatic filtering for scores could mask valid feedback. If someone does not like my scenario after playing it fully and gives it a low score with arguments, then I will not like the low score but it would still be valid feedback.
It will be suspicious if from obvious game stats you can see that the player did not really play the scenario. This is something the scenario writer will be able to see. Through a comment (with proper arguments) in the sheet, a review can be requested.
I also agree that no score should be modified. Disabling is a better option.

[BenRQ]

The automatic filtering for scores could mask valid feedback. If someone does not like my scenario after playing it fully and gives it a low score with arguments, then I will not like the low score but it would still be valid feedback.

Maybe we could only exclude outliers for duration then, but still keep it all of the score.
It would be a lot easier than trying to detect pause.

[BenRQ]

I have a first implementation available, here are the results:

https://docs.google.com/spreadsheets/d/1Vopk9dqAC-Jev-tARiRgktsyxEONBo7v0EFZUCoKwJc/edit?usp=sharing

Few meaningful examples with a big difference:

scenario name | Old avg duration | New avg duration | Difference
-- | -- | -- | --
expeditiontoterror.valkyrie | 339 | 225.7692308 | 113.2307692
inthedark.valkyrie | 308.3181818 | 205.45 | 102.8681818
main_street_market_mayham.valkyrie | 303.3076923 | 201.75 | 101.5576923
strainonreality.valkyrie | 359.8685446 | 265.4236453 | 94.44489928
wiltshire.valkyrie | 207.6935484 | 121.7647059 | 85.9288425
stressandstrain.valkyrie | 255.8939929 | 175.7843866 | 80.10960632
birdsshop.valkyrie | 192.8333333 | 139.862069 | 52.97126437
saviors.valkyrie | 176.0864198 | 131.9605263 | 44.12589344

It looks like it's doing what it is supposed to.
I can put it online if you think results are goods.

[BenRQ]

New algorithm has been uploaded.
As agreed, we don't want to mess with the notes so from my point of view ticket is closed.

Feel free to tell me otherwise
We can still revert if someone sees any issue, or thinks the new script is not doing a good job.