Smokedetector: Should we reconsider the core threshold (again)?

[angussidney]

See the previous discussion here: https://github.com/Charcoal-SE/SmokeDetector/issues/1594

A number of people have shown that they don't like the current thresholds for the 'core' role - or at least were confused / surprised by the changes and think that we should revisit the decision.

Personally, I think that we never really came to a conclusive decision last time we discussed this - only 3 or 4 people were actively discussing the issue with varying opinions, and multiple people weren't even aware that the discussion was taking place.

Do we want to change this? Should it continue to be based on feedbacks (if so, is the current threshold okay or not), or something else?

[honnza]

One thing to note that the number of feedbacks in general will drop once we switch to a consistent 5-autoflag mode. Do we have enough data to predict this?

[K-Davis1]

I don't think feedbacks will drop. Feedbacks are different than flags. If people are here just for the flags and this is what turns them off then so be it.

---

But I am up for either keeping "as-is" or upping the threshold. I don't think we should lower it.

[Undo1]

Metasmoke requires at least 2 feedbacks on each post, I don't think feedback shrinkage will be a huge issue. If it is, we can adjust then.

[iBug]

60 feedbacks in 30d is too low and I've always objected it. Some mentioned 200~250 and I think that's a reasonable number.

But still, I think it's better to have multiple criteria for this. Feedback count should be one. I also like my previous proposal: identity in CHQ.

[ArtOfCode-]

I'll repeat what I've said previously here: I'd like the _primary_ component or the _majority_ of the core system to be an objective, automatable requirement. Something metasmoke can do by itself. That takes a chunk of work away from admins (no, it's not a huge job to give out core, but it means we don't run into complaints about subjectivity).

Having a smaller secondary subjective component is fine (we have one at the moment - admins can pin the core role for people who don't meet 60/30).

[angussidney]

I stand by my original opinion that core should be awarded on a subjective basis, especially when some of the features of the core role require you to be an active member of the organisation to understand their proper use. If by the end of this discussion we find that we're having trouble balancing the overall threshold due to the large variation of the features, maybe we should consider discussing splitting some of the features into a new role?

---

Feedback isn't a very good indicator of whether or not you're a 'core' member of our team - it doesn't necessarily correlate to chat participation, speaking up in discussions, code/blacklist contributions etc. I know none of those are required for the core role, but feedback shouldn't be the only requirement, especially when increasing your feedback stats is extremely easy - simply use FIRE to give feedback to as many old posts as you like.

Basing the core role on a statistical measure only would only hurt long-time core contributors who take a short break, or contribute in ways other than feedback. Even just using a _combination_ of an objective measure (e.g. feedbacks) and a subjective measure (admin discresion for the outliers) would still result in gamification, which is a very valid problem - especially when there's outsider concerns about the gamification of autoflagging.

Using an admin-decides approach without any fixed criteria (which we currently use for all of the other roles anyway) would solve all of the above problems. I know that sometimes some things fall through the cracks, and that occasionally people don't get the role when they should do - but that's why people should just ask. More often than not, the answer will be yes.

I know that as users of SE we hate doing things subjectively, but it isn't necessarily a bad thing here in my opinion.

[Undo1]

I'll apply the same filter to this as any other concern: We'll address gamification when there's evidence of gamification both happening and having a real negative impact. Dealing in what-ifs isn't a great way to make these decisions.

I'm fine either way, just want to make sure we do things for the right reasons.

[WELZ-gh]

I support the idea of making it a higher threshold, however, I would like for there to be a way for admins to Pin (as Art mentioned), but it doesn't need to be indefinitely, it can be for 1 ,3 6 and 12 months. (for users who've given many contributions, but may not be able to do so at this moment - they're still a Core user, just not able to feedback for whatever reasons.)

---

Maybe we can have Smokey post a message into chat like:

@[username] you are now a Core user, read more here.

With this it will be very easy for regulars to notice if anybody truly is gaming. When you suddenly see a random user is core, and you've never (or only once) even seen them in chat, it would raise some alarms.

[j-f1]

Note that you can’t ^{[citation needed]} give feedback without approval from an admin, so it’s unlikely that random users would get core access.

[Undo1]

Citation: I just looked at metasmoke; new users do not have feedback permissions (whether that's enforced well is another question)

[tripleee]

I'm with Angus on this one -- if we don't automate all of our roles, making this particular one (more) objective doesn't actually help build any confidence that we are not subjective and/or know what we are doing.

Having a human make the final decision is actually an important feature for building and maintaining that confidence. If there is always a member of the privileged caste making the call, we ensure that we are at least making a conscious decision at every point of the process.

Gamification is just one part of this; we also need to defend ourselves against infiltration etc -- in short, the list of "what ifs" can be made infinitely long and complex, and even though every individual decision will be mundane and undramatic, having someone performing that chore puts in a useful emergency brake into the system.

[magisch]

I don't think we need an automated process for this. Almost everything is done by common sense here (privileges in general, code admin, admin, github access, people, etc) so having core in there won't be a big change of pace and I suspect not a lot of work for the admins to figure.

[ferrybig]

I think basing our core user group on feedbacks only is wrong because it's not primary goal of CHQ.

Our primary goal is maintaining a bot that detects spam, people in the core user group should (ideally) know:

• How the code works (they can prove this by contributing code (Not only the blacklist (Nested parentheses look weird ) ) )
• How to setup a smoke detector instance from scratch, incase there are problems with a MS that caused all smokey's to crash (even a smokey without MS integration is more useful than none)
• Have experience with basic problems of Smokey that are annoying on the long run (like putting 1 instance in standby if multiple are running)

While feedback is useful, it is more part of our secondary goal, while it helps in the long run to make our primary goal more accurate, it doesn't add anything in the meantime.

One other drawback of basing the core role on feedback, is that it either forces people to use userscripts, or encourages spamming in the chat to add the feedback.

TL;DR: We cannot live without people knowing how the code works, but we can live if we have no-one that adds feedback (and it is easier to find people for this), and because of that basing the core role on feedback seems wrong to me.

[ArtOfCode-]

I'll concede on automation, then :)

That said, I don't think it'd be fair to base it on code or other technical knowledge either. We have dedicated people who are certainly part of our core who have all degrees of technical ability, from the know-all developers to the technically uninitiated. I think some degree of awareness of how the system works is good, but by the time you can be considered core you've almost certainly already got it.

[SulphurDioxide]

My first reaction was surprise at having lost the core role. When I first started at CHQ, it was quite flattering to find that one day I'd been mysteriously given the core role because one of the admins felt it was about time and that I was part of the core team. I like that human element because it shows that people want you to be there.

How about instead of actually providing the core role based on activity, we could have an admin page that _suggests_ users for core role eligibility based on activity instead. That way, it is more difficult for users to slip through the cracks.

[makyen]

IMO, what needs to happen is that the existing "core" role needs to be reexamined as to it's purpose, and an additional role added.

"Core" is serving multiple purposes

There are both the functional privileges which "core" gives and an indicator that you've contributed substantially to the project. IMO, the problem is that including increased autoflagging in the "core" role treats that responsibility as a reward, which isn't how we should be treating it.

"Core" as recognition for substantial contribution

The assignment to the "core" role to indicate that someone has substantially contributed to the project is beneficial, for multiple reasons. Users should not be removed, unless they break from the project for some reason which indicates their prior contributions should not be recognized. Assignment/removal should not be automated. However, in addition to admins manually noting that a user has been contributing, there could be thresholds of activity which bring users to the attention of admins for potential inclusion.

People are rightly upset to no longer be considered "core" when they have contributed substantially to the project. Once in this group, they should not be removed, except by intentional action by the admins.

Current additional privileges for "core"

Some additional privileges being associated with/as a reward for the "core" role is a good thing. The ones which are granted currently appear reasonable, with the exception of increased autoflagging.

Autoflagging shouldn't be a reward, it's a responsibility

There appears to be some feeling (outside and, to an extent, inside the project) that increased autoflagging is a reward for being a "core" member. In my opinion, if we look at increased autoflagging as, or treat it as, a reward, then we are confirming that autoflagging is, in part, "gaming" the flag system. This is something that we really don't want to be the case.

There should be objectively clear reasons for granting increased autoflagging which don't treat it as a reward for older, historical, contributions to the project. Given that someone should be in "core" for exactly that reason (or for current contributions), increased autoflagging should not be associated with the "core" role.

Increased autoflagging is for different reasons than "core"

As I understand it, the reason increased autoflags are given to people is that they are users who are both experienced with the system and are currently available to respond to pings in Chat, should there be any problems with the autoflags placed for them (e.g. it turns out that the post is fp and the flags should be retracted). A person's availability for this is indicated by their presence in Chat on a regular basis, at least as someone who lurks and posts a message often enough to be pingable.

It is desirable that who is getting increased autoflagging be something that dynamically adjust to include only those who are currently active and available to respond, should the need arise. Thus, this is something which should be automatically allocated dynamically, without the need for admins to manually add or remove people. (This is the opposite of how the "core" role should be assigned, which is an additional reason to split increased autoflagging out of "core".)

Currently, the system's best indicator of a person's availability is that they have recently provided some amount of feedback. Thus, it's reasonable for inclusion in the group of users who receive additional autoflagging be assigned based on the quantity of recent feedback provided by the user, as an indicator that they are currently available.

Proposal

1. Create a new role: "main autoflagger" (someone should come up with a good name)
   
   
   
   • The users who have this role are included in increased autoflagging. If the user doesn't have this role, then they are not included in increased autoflagging.
   
   
   • Assignment to/removal from this role is automatic, with the ability to manually disqualify yourself (using metasmoke to turn off your own autoflaging is sufficient; e.g. when you go on vacation).
   
   
   • If the user has X feedbacks in the last N days, then they have this role. If they don't have that many feedbacks in that time, then they don't have this role.
     
     
     
     
     
     • X and N should be chosen to indicate the user is currently active. (i.e. N should be small; e.g. 7 or 14 days).
     
     
     
   
   
   • If desired, there could be other prerequisites for this role, which could be used to indicate that the user has overall experience with the system. Some possible criteria to indicate overall experience could be: > Q all-time feedbacks; has had some other role for T time; etc.
   
   
2. Remove increased autoflagging from the "core" role.
3. Change "core" membership to be manually added/removed by admins.
   
   
   
   • Perhaps have various automatic triggers which suggest to admins that a user should be considered for inclusion in "core" (e.g. > 1,000 all-time feedbacks; > M PRs submitted; etc.)
   
   

[angussidney]

I'm going to steal a suggestion from my previous comment on splitting the roles:

1. Give all accounts the ability to sign up for autoflagging (à la the current 'flagger' role) - we've only ever revoked flagging permissions once, and even then you could argue that it was their right to participate in autoflagging as a normal SE user, especially since it doesn't have any harmful effect on our project
2. Change the 'flagger' role so that it grants additional autoflags (similar to Mayken's 'main flagger'). Transfer current automatic core settings over to this rile.
3. Keep the 'core' role the same, minus the additional autoflags and automatic role granting.

[j-f1]

minus the additional autoflags

and the automatic addition/removal of the role

[angussidney]

We appear to have come to a conclusion; would someone be able to change MS to suit the new agreement?

[ArtOfCode-]

MS changes made.

[makyen]

@ArtOfCode- How much of this was actually implemented? I'm not pushing for implementation, I'm just not wanting to tell people the wrong things are in effect.

What appears to be the case:

• Core is back to being completely manual wrt. additions and removals.
• Core remains as the role used to determine if the user gets additional autoflags. I'm assuming this because all users still have the "Flagger" role, and there's no new role which people have been assigned to indicate getting additional autoflags.

[Undo1]

That's all correct @makyen.