Community: Create 2019 Contributor Survey

[parispittman]

Umbrella issue below
other refs: #4166 , #4392

Per our the contributor experience charter, we need to execute on a contributor survey to get a pulse on the upstream community. We've made progress over the years and would like to add a designer on this year so we can have a beautiful report at the end.

Goals:

• gather demographics for contributor base (try to map to past data questions for trends)
• create a priority list for contributor experience projects
• determine what needs improvement and benchmark whats working
• continue an important feedback loop

Tool:
SurveyMonkey (CNCF)

Tasks:

• review questions from last year and determine what stays and goes
  list of questions here: https://github.com/kubernetes/community/blob/master/sig-contributor-experience/contribex-survey-2018.md
• add additional questions for this year
• create a survey page for this year modeled from last year (same as above list of questions)

Notes from retro of last survey:

• Be transparent about # goal to hit. If we want 200, say that.
• Don't make all questions required.
• Incentives are good. We did codes for some kind of swag last year.
• Advertise on contributor heavy channels and list them on the survey page for transparency.
• According to the distribution of responses, when doing demographics, 'less than 6 months' and 'just started' are too close in definition to the reader. Need to either clarify those or use something else.
• Include links where definitions may be fuzzy or mean something different to everyone.
• Make sure all ranges match and have consistency throughout the survey (ex: 1=least useful to 5=most useful)
• Figure out another goal/phrase/wording for this question as it didn't give the results intended:
  Which of the following tooling do you find useful? (question 8)

Timeline:

• finish survey and testing by end of August
• send out with timebox of EOM September
• goal for contributor summit to have results and talk about them during the welcome and use some of the pretty graphs as posters (and potentially do deep dives in sessions)

[parispittman]

/help-wanted

need to break the tasks down into individual issues and time box them. this is not a good issue for first time contributors unfortunately but will happily take at least a partner from our current base.

[jberkus]

Per discussion today I'd like to add some questions about the community meeting. I feel like we need to poll what people would like to see out of a community meeting. Here's my suggestion:

Q: How many times did you attend, or watch a video of, the weekly Community Meeting last year?
- Zero
- 1-3
- 4-9
- 10-20
- 20+

Q: Which of the below would make you likely to attend more of the Community Meetings? Check all that apply.
- Having a full agenda with descriptions posted several days ahead
- Eliminating the demo at the beginning
- Additional developer tips/news content
- Using the meeting to discuss project-wide development topics & roadblocks
- Fewer meetings in my personal schedule
- Different timeslot for the meeting
- Other:

I think we could eliminate the "what do you like most about the community meeting" question we had last year.

[cblecker]

cc: @vllry @BenTheElder @munnerz @justaugustus

I want to get a couple questions added around prow testing comments and notifications. Still mulling over exact wording, but want to make sure we get them in.

[parispittman]

ok - i modified and left comments for all of the questions we asked last year. Please take a look.

1- How long have you been contributing to Kubernetes?

• less than one year
• one to two years
• two to three years
• three+ years

2- What level of the Contributor Ladder do you consider yourself to be on? (pick the highest if you are multiple OWNERs files)
-there's a contributor ladder?
-not yet a member but working on it
-member
-reviewer
-approver
-subproject owner

3- Are you interested in advancing to the next level of the Contributor Ladder?
-no
-yes
-if I had help/mentoring/support
-if i had more free time

4- What region of the world are you in?
(REMOVING: What timezone are you most often in? (Check your UTC offset here))
-list the same regions we did last year

5- How many other open source projects not in the Kubernetes ecosystem do you contribute to? (example nodejs, debian)

• this is my first open source project!
• 1 other
• 2 or more

6- Please rate the below parts of the contribution process by how challenging they are, from 1 (not a problem) to 5 (a frequent blocker):
Code/Documentation review
Communication
GitHub tools and processes (not our customized tooling)
Finding the right SIG for your contributions
Our CI, labels, and crafted customized automation
Debugging test failures
Finding appropriate issues to work on
Setting up development environment
Having PRs rejected
Writing documentation and notes

(REMOVING:
Which of the following tooling do you find useful?
What tool above is the least useful and why?
Wish something was automated that isn’t? List it here.
How do you perceive the current notification volume and utility?)

7- How often do you interact with Kubernetes GitHub via the following methods (1 - not at all, 5 - frequently):
GitHub website
Desktop application
Mobile
E-mail notifications

8- Do you agree with the following statements (1 - strongly disagree, 5 - strongly agree):
"I understand enough about how Kubernetes CI works to be able to diagnose my own PR failures"
"When something is broken in my PR, I can read the comments from CI and understand why"
"The number of test failures unrelated to my PR severely impacts my ability or desire to contribute"
"There are too many notifications to be helpful when I open a PR"

9- Does your employer support your contributions to Kubernetes?
yes i can contribute on company time
no i need to use my own time

10- How often do you contribute upstream (code, docs, issue triage, etc.)?
//NEED TO GET SELECTIONS FROM LAST YEAR

11- What areas of Kubernetes do you contribute to? Please check all that apply.
Core code inside of kubernetes/kubernetes
Code inside of another repo in the kubernetes/* GitHub organization
Documentation
Testing & Infrastructure
Advocacy and events
Community & Project management; SIG Chair etc.
Plugins & Drivers (CSI, CNI, cloud providers)
Related projects (Kubeadm, Helm, container runtimes, etc.)
Don’t contribute yet, hoping to start soon
I can contribute during the day but it's not officially supported //may need rewording

12 - Are there specific ways the project could make contributing easier for you?
open ended question

(REMOVING: What conferences have you previously attended or are planning to attend?)

(REMOVING: Do you have any suggestions on how to make the Contributor Summits more valuable to you (N/A if not applicable)?)

(POSSIBLY REMOVING: How many Kubernetes Contributor Summits have you attended?)

13 - Which of the below would make you likely to attend more of the Community Meetings? Check all that apply.

• Nothing, I attend and think they are great
• Having a full agenda with descriptions posted several days ahead
• Eliminating the demo at the beginning
• Additional developer tips/news content
• Using the meeting to discuss project-wide development topics & roadblocks
• Fewer meetings in my personal schedule
• Different timeslot for the meeting
• Other:

14 - Some of the major projects contributor experience is working on are listed below, check one that is most important to you that we carry through to completion:
Mentoring programs for all contributor membership levels [link to community membership doc]
GitHub Management
Delivering valuable contributor summits at relevant events
Governance operations (youtube, zoom, etc.)
Keeping our community safe on our various communication platforms through moderation guidelines and new approaches
Cleaning up OWNERs files
Contributor Documentation (guide, noncode guide, developer guide, contributor site)

15- What is missing from that list entirely that's important to you/your SIG? Why?

16 - Of our various communications channels, please rate which ones you use and/or check most frequently on a 1-5 scale, where 1 is “never”, 3 is “several times a month” and 5 is “every day”.
Google Groups/Mailing Lists
Slack
discuss.kubernetes.io
Zoom video conferencing/meetings
Discussions on Github Issues and PRs
Unofficial channels (IRC, WeChat, etc.)
YouTube recordings (community meetings, SIG/WG meetings, etc.)
Google Docs/Forms/Sheets, etc (meeting agendas, etc)

17 - Which of these channels is most likely to reach you first for news about decisions, changes, additions, and/or announcements to the contributor process or community matters?
kubernetes-dev mailing list
Dedicated discuss.k8s.io forum for contributors
Contributor Experience mailing list
Slack
Twitter
A dedicated contributor site
Kubernetes blog
k/community repo in GH (Issues and/or PRs)

(POSSIBLY REMOVING: Do you think Slack adds value to the project for users and/or contributors?
yes for users
yes for contributors
yes for both
no for both
no for users
no for contributors)

18- Have you ever used the Help Wanted and/or Good First Issue labels on issues you file to find contributors?

(REMOVING: Are you interested in mentoring a Kubernetes upstream Intern for Outreachy or Google Summer of Code? We are also looking for organizations to sponsor if your employer is interested.)

19 - Have you watched or participated in an episode of our YouTube mentoring series Meet Our Contributors? If you have specific suggestions, leave them at the end of the survey.
no - didn't know this was a thing
yes - it was helpful
yes - it wasn't helpful
no - i don't need mentoring, guidance, or other resources

(REMOVING: How useful did you find Meet Our Contributors? (1 - not useful at all; 5 - extremely useful)If you have suggestions on improvements, leave those in the feedback box at the end of the survey.)

20 - Are you available to mentor other Kubernetes contributors through one or more of our many mentorship programs?

I'm inexperienced and don't know enough to mentor
Yes, please contact me (you'll include your email in the last question)
I am already mentoring (you'll include your email in the last question because you want swag!)
I would like to but I don't have the time

20 - What remains a blocker to becoming a mentor?
time
company/employer doesn't support OSS mentoring unless it's a coworker
//add more here

21 -
Do you have any comments, questions, or clarifications for your answers on this survey? Leave the general feedback here. If you leave your email, we will circle back with you:

ADD: How often do you attend the weekly community meeting?
[insert question bank]

[nikhita]

A question about how useful people find comments from test failures on PRs would be useful: https://twitter.com/tophee/status/1169147984724074496

xref https://github.com/kubernetes/community/issues/3621
/cc @cblecker @BenTheElder @munnerz

[jberkus]

Would be lovely to have this in something that allowed line-item comments, but until GiST supports that I don't have a suggestion. Some tweaks, mainly to the available answers:

3- Are you interested in advancing to the next level of the Contributor Ladder?
-no
-yes
-maybe if I had help/mentoring/support

remove word "maybe"
add "if I had more free time"

5- How many other open source projects not in the Kubernetes ecosystem do you contribute to? (example nodejs, debian)

* this is my first oss project!

s/oss/open source

* 1 more

change to "1 other"

* 2+

change to "2 or more"

6- Please rate the below parts of the contribution process by how challenging they are, from 1 (not a problem) to 5 (a frequent blocker):
Code/Documentation review
Communication
GitHub tools and processes (not our customized tooling)
Finding the right SIG for your contributions
Our CI, labels, and crafted customized automation
Debugging test failures
Finding appropriate issues to work on
Setting up development environment
Having PRs rejected

Add: "Writing documentation and notes"
(this is to cover our non-english speakers who often struggle with this)

7- Which of the following tooling do you find useful?
automatic /retest of flakes (fejta-bot)
automatic labeling of stale issues (fejta-bot)

s/fejta-bot/k8s-triage-robot/

issue commands like /assign, /kind bug (k8s-ci-robot)
PR commands like /approve, /lint (k8s-ci-robot)
automatic merging of approved PRs (k8s-merge-robot and k8s-ci-bot)
//NEED A DECISION ON THIS FROM CHRISTOPH/TEST INFRA

8- What tool above is the least useful and why?
//NEED A DECISION ON THIS FROM CHRISTOPH/TEST INFRA
based on data analysts comments from last time, this question had the least conclusive responses/no trends/etc. - need to rethink this one

Josh: I'd suggest just cutting the question. We can decide "not useful" based on not checked in the prior question.

9- Wish something was automated that isn’t? List it here.
//NEED A DECISION ON THIS FROM CHRISTOPH/TEST INFRA

10- How do you perceive the current notification volume and utility?
//NEED A DECISION ON THIS FROM CHRISTOPH/TEST INFRA
we didn't get much ROI from this question last year

Josh: yes, the answer to this one seems likely to be predictable. IF we want a GH notification question (not saying we do) here's one that would give us more determinative answers:

1. How are you dealing with GitHub notification volume? Choose the one that's most similar to your situation.

• I don't get that many notifications
• I get tons of notifications, but have my own system for dealing with them
• I'm ignoring most notifications because there's too many

11- Does your employer support your contributions to Kubernetes?
yes i can contribute on company time
no i need to use my own time

add: "I can contribute during the day but it's not officially supported" (or better wording to that effect)

12- How often do you contribute upstream (code, docs, issue triage, etc.)?
//NEED TO GET SELECTIONS FROM LAST YEAR

13- What areas of Kubernetes do you contribute to? Please check all that apply.
Core code inside of kubernetes/kubernetes
Code inside of another repo in the kubernetes/* GitHub organization Documentation
Testing & Infrastructure
Advocacy and events
Community & Project management; SIG Chair etc.
Plugins & Drivers (CSI, CNI, cloud providers)
Related projects (Kubeadm, Helm, container runtimes, etc.)
Don’t contribute yet, hoping to start soon

Wait, wasn't there a "documentation" option for the above?

14 - Are there specific ways the project could make contributing easier for you?
open ended question

(REMOVING: What conferences have you previously attended or are planning to attend?)

(REMOVING: Do you have any suggestions on how to make the Contributor Summits more valuable to you (N/A if not applicable)?)

(POSSIBLY REMOVING: How many Kubernetes Contributor Summits have you attended?)

I'd like to keep one Summit question on the survey, probably this one. What I specifically want to see is whether the CSes are reaching a diverse group of contributors, or whether it's 80-90% the same minority of contributors. Since some of the CSes don't have good registration/attendence records, we need to handle this via survey. The limitation on this is that, of course, a lot of contributors don't fill out the survey.

I might want to change it to: "How many Contributor Summits have you attended in the last 2 years?", because we don't really care about 2016 at this point. Answers to the modified question would be:

• None
• None yet, but I'm attending my first one this year
• 1
• 2
• 3 or more

15 - Which of the below would make you likely to attend more of the Community Meetings? Check all that apply.

* Nothing, I attend and think they are great

* Having a full agenda with descriptions posted several days ahead

* Eliminating the demo at the beginning

* Additional developer tips/news content

* Using the meeting to discuss project-wide development topics & roadblocks

* Fewer meetings in my personal schedule

* Different timeslot for the meeting

* Other:

We're dropping "How often do you attend the weekly Community Meeting, or watch it on the live stream or Youtube?" ? Without that question, the answer to (15) becomes less meaningful. The "things we need to change" means something different from a frequent attendee than it does from an infrequent/never attendee.

16 - Some of the major projects we are working on are listed below, check one that is most important to you that we carry through to completion:

s/we/Contributor Experience/

Mentoring programs for all contributor membership levels [link to community membership doc]
GitHub Management
Delivering valuable contributor summits at relevant events
Governance operations (youtube, zoom, etc.)
Keeping our community safe on our various communication platforms through moderation guidelines and new approaches
Cleaning up OWNERs files
Contributor Documentation (guide, noncode guide, developer guide, contributor site)

Is this list of projects up to date?

17 - What is missing from that list entirely that's important to you/your SIG? Why?

18 - Of our various communications channels, please rate which ones you use and/or check most frequently on a 1-5 scale, where 1 is “never”, 3 is “several times a month” and 5 is “every day”.
Google Groups/Mailing Lists
Slack
discuss.kubernetes.io
Zoom video conferencing/meetings
Discussions on Github Issues and PRs
Unofficial channels (IRC, Hangouts, Twitter, etc.)

add "WeChat" to the list of unofficial channels

add option: "Related project Slacks (Helm, CoreOS, Golang, Prometheus, etc.)

YouTube recordings (community meetings, SIG/WG meetings, etc.)
Google Docs/Forms/Sheets, etc (meeting agendas, etc)

20 - Do you think Slack adds value to the project for users and/or contributors?
yes for users
yes for contributors
yes for both
no for both
no for users
no for contributors

Suggest dropping this question, unless you have a specific goal for it? I think we passed the point of potentially dumping Slack.

23 - What remains a blocker to becoming a mentor?
time
company/employer doesn't support OSS mentoring unless it's a coworker
//add more here

It's a bit odd to have this question without a prior question about "do you want to mentor for Kubernetes". Without a question about whether people are already mentoring, a question abotu why they can't doesn't make much sense. So we should add back in a "do you want to mentor" question, or remove (23) Suggested mentoring question if we go with the former:

Are you available to mentor other Kubernetes contributors through one or more of our many mentorship programs?

• I'm inexperienced and could use mentoring
• Yes, please contact me
• I am already mentoring
• Yes, but I don't have the time
• I'm not sure how to get started but contact me anyway

[cblecker]

Specific questions on testing:

• Remove question 7-10
  

  - Add following questions:

How often do you interact with Kubernetes GitHub via the following methods (1 - not at all, 5 - frequently):

• GitHub website
• Desktop application
• Mobile

- E-mail notifications

Do you agree with the following statements (1 - strongly disagree, 5 - strongly agree):

• "I understand enough about how Kubernetes CI works to be able to diagnose my own PR failures"
• "When something is broken in my PR, I can read the comments from CI and understand why"
• "The number of test failures unrelated to my PR severely impacts my ability or desire to contribute"
• "There are too many notifications to be helpful when I open a PR"

Thoughts?: @BenTheElder @vllry @munnerz @justaugustus @Katharine

[parispittman]

addressed everyones comments to date in my previous comment above with the updated question bank. ill email that to the mailing list and get some other opinions. id like to target next week to send this out.

[parispittman]

https://www.surveymonkey.com/r/VYRJZ5G

survey launched yesterday
END DATE: JAN 15 2019 @ 5PM PT

opening a new issue for post survey data tasks

/close

[k8s-ci-robot]

@parispittman: Closing this issue.

In response to this:

https://www.surveymonkey.com/r/VYRJZ5G

survey launched yesterday
END DATE: JAN 15 2019 @ 5PM PT

opening a new issue for post survey data tasks

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.