Tag-security: Retrospective for cloud native security whitepaper (CNSWP)

[PushkarJ]

Description: Perform retrospective for Cloud native security whitepaper

Impact: Cloud native security whitepaper version 1.0, was released Nov 18, 2020. This is a huge milestone but we are just getting started and the contributors fully expect to publish future versions of this paper.

Some of the content in future versions, will change based on the natural evolution of security in cloud native. However, some changes could also be influenced by feedback from the readers of the paper received anecdotally or through more formal means e.g. CNCF survey, twitter poll

Scope: Limited to readers and contributors of cloud native security whitepaper. Large part of SIG-Security members would fall in either of these two categories. The formal group to gather feedback could be CNCF marketing or communications team
Scope:
Conduct a survey and analyze results to determine the effectiveness of the Cloud Native Security Whitepaper.

TO DO

• [x] STAG Leadership Sponsor: @TheFoxAtWork
• [x] Project leader(s): @PushkarJ
• [x] Identify sources for feedback and the questions we want to get answers for
• [x] Get feedback from identified sources
• [x] Visualize / Aggregate the data that is measurable
• [x] Carve out insights, from the anecdotal (subjective feedback) and aggregated (objective feedback)
• [ ] Discuss which insights could be used on improving next version of whitepaper
• [ ] Identify a contributor who will take ownership to ensure the agreed upon insights are accounted for in next version of whitepaper

Tagging OG leads of cloud native security whitepaper @TheFoxAtWork and @lumjjb for further feedback :)

[magnologan]

I think I can help gathering those data via surveys and putting it all together in a DB like ELK for search and interpretation of the results. Do we have a template survey already or should we create one from scratch?

[jlk]

Do we need ELK? Sounds like a google sheet would be Good Enough. I think the LF community survey has 1200 participants...just don't want to over-engineer :)

[magnologan]

No need, just a suggestion. I just don't know the amount of data we'll get I've just find it easier to get proper info out of it from importing the excel/csv sheet into ELK or a Splunk to visualize and filter the data better. Just personal preference that I've done before with CVE data and Docker Image vulns. Whatever is easier I'm fine with it.

[jlk]

:) all for suggestions!

[faisalrazzak]

I think our primary focus initially should be: What information we are trying to extract from the feedback/retrospective ? Once determined, we can determine what is the best way to collect and manage the data around it.

I want to extract information on following topics:

1. Check to see how the whitepaper was seen by the larger cloud native community and did it help/not help in convincing C-level folks in organizations.
2. What topics/domains/aspects were missing in the 1st version of whitepaper ?

[achetal01]

One of the goals should be to identify what level of depth the consumers ant in this paper. Maybe we need to have two flavors one for the C-Level and One for actual platform owners and app devs.

[TheFoxAtWork]

related to #495 ( some items from the retro may deserve to be their own breakout topics.)

[TheFoxAtWork]

Wanted to drop this here while still in my mind: either the next iteration or one of the separate deep dive papers needs to include explanation and importance of verifiable reproducible builds

[chasemp]

To that end, recently found myself quoting https://reproducible-builds.org/ internally. Good reference resource.

[joshuagl]

Wanted to drop this here while still in my mind: either the next iteration or one of the separate deep dive papers needs to include explanation and importance of verifiable reproducible builds

I would be very happy to contribute to that! I have been wondering how we can better advocate for verifiably reproducible builds (and ephemeral build systems, and build system hygiene more generally) through the SIG.
A colleague and I tried to drum up some noise about this a year or so ago but didn't make much headway before getting side-tracked).

How can I help make this deep dive/section of the next white paper happen?

[TheFoxAtWork]

@PushkarJ I'd like to move forward with the retrospective so we can create a subset of topic areas for deep dive papers. Would get a few folks on this issue (interested in contributing to the survey) together for a meeting to figure out the questions we will ask? We should leverage the CNCF for running the survey for us.
We will need:

• survey questions
• date to open survey
• help from CNCF to execute the survey
• date to close the survey
• plan for going through the responses

[PushkarJ]

@TheFoxAtWork going through some crazy times last few weeks but I am back now. I will bring this up in next SIG meeting and will setup a time to discuss this with interested participants.

[PushkarJ]

For everyone who have graciously offered to contribute, hop on to #sig-security-whitepaper slack channel under CNCF slack workspace for further discussions on meeting logistics

[PushkarJ]

I think our primary focus initially should be: What information we are trying to extract from the feedback/retrospective ? Once determined, we can determine what is the best way to collect and manage the data around it.

I want to extract information on following topics:

1. Check to see how the whitepaper was seen by the larger cloud native community and did it help/not help in convincing C-level folks in organizations.
2. What topics/domains/aspects were missing in the 1st version of whitepaper ?

Agree on first point, for which I think the surveys will help. For second point, we have a chance to cross reference work being done by supply chain security wg: https://github.com/cncf/sig-security/issues/510

[apmarshall]

Brainstorming some questions for survey:

Identifying Who the Actual Audience of the Paper Was:

1. What job title best fits your role? (CSO/CISO, CTO, C-Suite, Engineering Manager, SRE, Developer, Security Architect, etc)
2. How many years of experience do you have with cloud computing, programming, or engineering? (give tiers, something like 1-2, 3-5, 5-10, 10-15, 15+)
3. How often do you work on issues related to security in your job? (scale with %s)
4. Describe your level of comfort/experience with security in a cloud environment: (1-5, total newbie - expert)

Questions About Whitepaper Content

1. (pick one) What is the most important area of security for cloud environments? (dependency verification/supply chain, repo scanning, service registry, secure communication channels, monitoring/logging, etc. Sure there are lots of things we could put on this list)
2. (pick one) What area of security for cloud environments do you believe is the most challenging for organizations to master? (same list)
3. After each of the above, 1-5 scale: How well does the whitepaper address this area?
4. How are you likely to use or refer to this whitepaper in the future? (as a primer for executives, reading material for training new staff, a reference/manual, etc)

[PushkarJ]

Update

We have finalized top 10 questions for the survey. The list can be found here: https://docs.google.com/document/d/15I1Yg-Qy6LCCipYZMB-vFaobMtn7TMQQERzRZXQy_6k/edit

Next step: To identify audience and send the survey out!

[PushkarJ]

Survey is out: https://www.surveymonkey.com/r/CNCF-SIG-Security-EndUser-2021 :)

[PushkarJ]

Hi Team 👋🏼 As discussed in today's call, @amye shared that we have had close to 70 responses to our cloud native security survey after it went out late March.

As a next step, we will go with lazy consensus on closing the survey i.e. if anyone has any concerns or thoughts about closing the survey, please comment here by June 4th. After this period is over, on Monday June 7th we will close the survey and move towards data collection, feedback review and start drafting changes we want to make in the next version of the whitepaper 🙂

[PushkarJ]

Since the survey is now closed, we compiled the initial results in this draft PR: https://github.com/cncf/surveys/pull/34

Suggestions, comments, edits are welcome!!