Salt: Reconfigure Stale Bot to exclude more labels

[sagetherage]

Description of Issue

Add to the slatebot exemptLabels:

• Bug
• Feature
• Test Failure

Setup

N/A

Steps to Reproduce Issue

github > saltstack org > settings > Installed GitHub Apps > stalebot > configure
requires Admin access to the org

Versions Report

N/A

[Oloremo]

What about Confirmed?

[sagetherage]

Confirmed label is included in this merged PR: https://github.com/saltstack/salt/pull/55626/files

[max-arnold]

What stalebot should do with untriaged tickets that have no labels?

[sagetherage]

Very good point @max-arnold! Our process is to triage sooner than Stale bot, but in the event of this risk the process does need to catch our human errors, and what better way than with automation! Any ideas?

My first idea: GitHub does support templates that will give a label automatically and we could update the issue template to include a label for all new issues submitted then exclude that label in the stale bot config. Recently, we have been using needs-triage I am also ok with new or something similar.

There are likely many ideas to consider. This ticket is to gather some, so I hope you and others will give us some input. :)

[sagetherage]

We are using the old issue template in GitHub and should update: https://help.github.com/en/github/building-a-strong-community/about-issue-and-pull-request-templates I can open a ticket to update to the latest and add to include a label that is then excluded here.

[sagetherage]

https://github.com/saltstack/salt/issues/55853

[Ch3LL]

i just had a thought, can stalebot do the reverse and only update issues with specific labels @waynew ?

If so we could just ensure that stalebot is only updating issues with Pending Discussions or Info Needed as we want to ensure those issues are getting updates to them.

[cachedout]

If so we could just ensure that stalebot is only updating issues with Pending Discussions or Info Needed as we want to ensure those issues are getting updates to them.

This is what the community has been asking for.

[Ch3LL]

looked a bit more into stalebot, they do have this:

# Only issues or pull requests with all of these labels are check if stale. Defaults to `[]` (disabled)
onlyLabels: []

but it seems from reading that it means an issue would need to include ALL of the labels, both info needed and pending discussion so for now just going to exempt labels.

[sagetherage]

Going to go ahead and add these labels to help with current status of stale bot closing those that we know we want them to ignore and will keep this issue open as we want to continue to look into only include as @Ch3LL states above and/or more options.

[sagetherage]

Discussion today on this topic to look at more robust options. GitHub now offers Actions to use for workflows that does give us more flexibility, but isn't a fork of stale bot we would maintain. If we look at the use of Actions for a different .yml file then it would require an SEP. Leaving this open to discuss and will communicate in other platforms.

[sagetherage]

https://github.com/saltstack/salt/issues/55913#issue-551874567 suggestion to configure longer than 30 days - discussed with Core team 1/24/2020 at this time we will keep this as a consideration for future configurations, but since the change this week added more labels that will be excluded from stalebot, we see this as sufficient ATM. We can consider this especially if there is more feedback.

[cachedout]

Hi @sagetherage

I briefly subscribed to the repo again to see if the noise level has gone down and while it has improved, there still seems to be a high number of issues which stalebot is tagging and probably should not. Here are some examples which have landed in my inbox from the previous twelve hours:

https://github.com/saltstack/salt/issues/46616#issuecomment-583802528
https://github.com/saltstack/salt/issues/49931#issuecomment-583764565
https://github.com/saltstack/salt/issues/49575#issuecomment-583759044
https://github.com/saltstack/salt/issues/48493#issuecomment-583759039

It seems that the most egregious offenders are those where Pending Discussion or the Info Needed labels have not been removed after a user has reported that the issue is still valid.

Perhaps it would be worth taking the time to examine those few hundred issues and ensure they are correctly labeled?

[timwsuqld]

It would be great to not have stalebot trigger at 30 days of inactivity, until you've sorted out the labels issues. Lots of old issues have those labels and so every 30 days at the moment I'm being forced to unstale issues.

[sagetherage]

@cachedout I have attempted to take care of those listed above and go through stale tickets this week and agree we don't have the number of days correct, yet. We seem to have swung the pendulum from one end to the other. As I understand, 30 days to trigger the bot is where we want to get be, eventually, ideally, but currently if there is more manual work to correct the bot then the bot isn't configured for our workflow. I suggest increasing the trigger date to 60 or 90 days, at least. @timwsuqld do you have an opinion on what may be a number of days to trigger?

[cachedout]

This isn't the hill I want to die on, but the only question I'd ask is -- what is the problem to which stalebot is a solution?

There are plenty of open, active, and recent issues to work on so it doesn't seem like it's the case that anybody is spending time working on an issue which is no longer applicable.

Having open issues, even when they are old, is _valuable_ to people who are searching for information about a Salt issue and would like to see that it's the activity (or lack thereof). Closing the issue without fixing it doesn't aid in that kind of searching. I can't see how this would aid in any type of project planning.

What's happening now falls, pretty generally, into one of the following buckets, all of which apply at pretty much any time span.

1. The issue is still valid and the submitter has stopped responding. Closing the issue automatically is very much the wrong behavior in this case. It may still be picked up by another users, or users seeing similar issues may be helped by seeing that it is still open.

2. The issue is not valid and the submitter has stopped responding. Here, while stalebot might have accidentally done the "right" thing, it's still not valuable to somebody searching for the issue. Again, my question is, "What actual harm is done by having this issue open?"

3. The issue is still valid and the contributor is still responsive. This annoys the contributors and, to be honest, offends a lot of people.

4. The issue is not valid and the contributor is still responsive. I don't think this happens too much. :)

Perhaps a middle ground might be this -- keep stalebot around, but set it to only label issues as stale and never to close them.

This way, you can filter as you please and there is some indication to users who are searching through the repo for information that the conversation is not active.

Users who have spent time contributing to Salt by filing a bug report aren't annoyed, and the bot is, at the very least, only minimally invasive. For bonus points, I suppose, you could make an "unstale" bot that removes the label after a conversation is restarted.

Again, my preference is that the bot be removed entirely but since it sounds like there is no appetite for that option, perhaps this compromise might at least satisfy all parties to some degree.

[sagetherage]

@cachedout thank you for the thought and analysis. We will take this to the next round of discussion on this topic. As you say Mike, there has been a long and elaborate discussion on this topic - we'll include you in as we pick up discussion on this soon.

[timwsuqld]

@sagetherage 90 days would be more reasonable for me at this stage
@cachedout I totally agree. Having things just marked as stale would be more helpful. This way we can filter stale issues and manually triage them. I understand the use of stalebot when people dump "support" requests into an issue, and then don't respond after 30 days requesting more information. I think this is where Salt are trying to get to, with the correct labels being applied to issues, once it's accepted as a valid issue, stalebot will leave it alone. That way, issues that aren't verified and are waiting more information, don't end up clogging the system being open years later but not having enough information to reproduce the issue.

[timwsuqld]

@sagetherage When is this discussion going to happen? I'm losing track of the "stale" tickets I need to comment on every 30 days, it's a bit ridiculous. You guys said you aim to triage issues before stalebot gets to them, but there is a backlog of tickets that haven't been given labels that stalebot is still rampaging against. At the least, until that backlog has been actioned, please adjust stalebot to have a longer time before marking as stale.

[timwsuqld]

@sagetherage Can we please triage Pending Discussion and Info Needed because it's becoming increasingly frustrating to keep relevant issues open. Either Triage them, or change stalebot to 90 days until you can.

The current situation isn't helping the community, and stalebot is not reopening bugs once closed, it's a real pain at the moment, and a waste of my time.

[sagetherage]

@timwsuqld I will put in a PR today to at least update the days. Apologies for not responding sooner.

[sagetherage]

Stale bot is no longer closing issues in the configuration and later today I will remove it completely from the Salt Repo

[timwsuqld]

@sagetherage Thank you for this. I hope whatever we do to replace it works better for both the Devs and the community. I know it's a hard line to walk, thank you for listening to the community.