Tag-security: [Suggestion] Define what an Observer role is during assessment

[MVrachev]

Description: I noticed there is the so-called Observer role in the assessment matrix.
I read all the documentation there which is about the different roles in a security assessment but found none information about this role.

Impact: I believe that if there's better documentation of the requirements and responsibilities of an observer role it will be easier for newcomers to join in the security assessments. They will be more confident to join even if they don't have so much experience and eventually this could lead to a new group of security reviewers.

Scope: How much effort will this take?
I don't think it will take so much time. This item is worth discussing in a future meeting and when everything is clear the solution is pretty straight forward.

[rficcaglia]

I think I injected that term but happy to change it to something else. Essentially it addresses the role of participants who closely monitor the assessors to a) learn the process but equally important b) provide feedback on how the process went. The explicit volunteering (ie not just passively monitoring the calls and git commits) means you are going to attend all working sessions as if you were an assessor, and actively review all work product as the assessment team produces it to provide feedback, before such materials are disseminated broadly.

The idea here is that after actively participating as an "observer" you are more or less qualified to be an assessor.

Maybe intern is a better term? :)

[MVrachev]

Maybe intern is a better term? :)

Yes, I think there should be a more appropriate term to describe a role which is actively involved.

My biggest question here is what are the pre-requirements to be an Observer/Intern.

[JustinCappos]

My biggest question here is what are the pre-requirements to be an Observer/Intern.

In my view, it is just a willingness to participate and learn. No skills are required.

(If you are in this role and lost, look up terms and try to understand them. It will come naturally after a surprisingly short while.)

[rficcaglia]

I agree that “interning” should be open to everyone lest this become an
echo chamber of homogenous perspectives.

That said, I think there ought to be some candid discussions with
volunteers about what domain knowledge is useful for a given assessment.

Someone completely new to kubernetes for example would probably find it
difficult to participate effectively other than just a scribe.

Since we don’t seem to have a flood of volunteers, I think we will likely
need to be flexible in any case.

On Mon, Aug 26, 2019 at 6:57 AM Justin Cappos [e-mail]
wrote:

My biggest question here is what are the pre-requirements to be an
Observer/Intern.

In my view, it is just a willingness to participate and learn. No skills
are required.

(If you are in this role and lost, look up terms and try to understand
them. It will come naturally after a surprisingly short while.)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/cncf/sig-security/issues/256?email_source=notifications&email_token=AAGENIWO543FPMEKA25FATDQGPOOHA5CNFSM4IO7YLR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5EOLRI#issuecomment-524871109,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGENIR5344AXUNCHFHT6JDQGPOOHANCNFSM4IO7YLRQ
.

[rficcaglia]

An example from recent events. If someone was assessing a crypto based
system, and could not fully understand an analysis such as this:

https://golovnev.org/papers/election.pdf

Then probably they would find it hard to contribute productively to an
assessment of a crypto based project.

I’m not asserting that every assessor needs to be able to produce this
level of analysis de novo, only that they need to be capable consumers of
such materials to be able to apply lessons learned to the projects they
assess.

On Mon, Aug 26, 2019 at 7:12 AM Robert Ficcaglia [e-mail]
wrote:

I agree that “interning” should be open to everyone lest this become an
echo chamber of homogenous perspectives.

That said, I think there ought to be some candid discussions with
volunteers about what domain knowledge is useful for a given assessment.

Someone completely new to kubernetes for example would probably find it
difficult to participate effectively other than just a scribe.

Since we don’t seem to have a flood of volunteers, I think we will likely
need to be flexible in any case.

On Mon, Aug 26, 2019 at 6:57 AM Justin Cappos [e-mail]
wrote:

My biggest question here is what are the pre-requirements to be an
Observer/Intern.

In my view, it is just a willingness to participate and learn. No skills
are required.

(If you are in this role and lost, look up terms and try to understand
them. It will come naturally after a surprisingly short while.)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/cncf/sig-security/issues/256?email_source=notifications&email_token=AAGENIWO543FPMEKA25FATDQGPOOHA5CNFSM4IO7YLR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5EOLRI#issuecomment-524871109,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGENIR5344AXUNCHFHT6JDQGPOOHANCNFSM4IO7YLRQ
.

[MVrachev]

I agree that “interning” should be open to everyone lest this become an echo chamber of homogenous perspectives. That said, I think there ought to be some candid discussions with volunteers about what domain knowledge is useful for a given assessment. Someone completely new to kubernetes for example would probably find it difficult to participate effectively other than just a scribe. Since we don’t seem to have a flood of volunteers, I think we will likely need to be flexible in any case.

I agree with @JustinCappos that the Observer/Intern program should be open to everyone.
On another hand, I also agree with @rficcaglia that there should be some kind of competence for the reviewed project so that the Observer/Intern could be useful for the security assessment.

I suggest defining a process in which

1. The decision to make a security assessment on a particular project is taken.
2. Make an announcement on our team meeting and/or somewhere on the README about the upcoming security assessment which would start a week or two later.
3. Invite contributors as Observers/Interns
4. Appoint a date and time a few days after step 3 on which there will be a discussion with the candidate Observers/Interns so that the security reviewers can decide if the candidates have enough base knowledge of the project.
5. If the security reviewers decide that there are capable enough Observers then they can assign them a small task list with to-do items before the security assessment has started.

I believe it's important to give time between steps 3 and 4 so that the candidate for Observers/Interns can try and read about the project.

[lumjjb]

I think ideally the net effect of someone as an intern/observer should provide a net non-negative effect on the process. I think this could be in various capacities. Being invited to the discussions and just listening in itself is super helpful. For such "passive observers", we can have any amount of them per security review.

However, observers which are active I think should be limited to a project - to ensure that there aren't "too many cooks". i.e. in a 30 min call, if each person has a comment, then it would really drag out the review process. Maybe "active observers" can make suggestions to the Assessment doc, which would be approved by a security reviewer? Also, there should be a certain ratio to the number of reviewers to active observers to ensure reviewers are not overburdened.

[rficcaglia]

@lumjjb I appreciate the too many cooks, however I also see (currently) a dearth of volunteers to dig in and do the dirty work...therefore having "enough" to spread the load (especially where the expertise required may be so specific) is ideal. the fine line between too many cooks and enough cooks can be determined by the lead I think. doesn't have to be pre-ordained heuristic.

I think the practical challenge here is that the lead won't find enough volunteers who are domain experts and have time to devote to a thorough assessment. in other words, we may be flooded with passive observers, but they won't have much to observe...being swamped with too many volunteers would be a good problem to have :)

But to @MVrachev 's point:

Make an announcement on our team meeting and/or somewhere on the README about the upcoming security assessment which would start a week or two later.

I presume/hope that will generate more interest/volunteers. currently that is blocked by the TOC as I understand it. Falco demonstrated interest in moving forward but without clarity on whether the TOC is the gating factor or not, I think we are in a holding pattern?

[MVrachev]

In our last meeting @dshaw pointed out that's important to clarify what role do we want:

• an Observer
• an Intern
• or both?

Dan said that there is an important distinction between the terms Intern and Observer.
When you are an Observer you are "observing" and you have the freedom to look more without doing so much work but also nobody has the responsibility to give you tasks or mentor you.
An Intern on another hand has the responsibility to be more active and involved in the process and some of the security reviewers have taken the responsibility to mentor him/her.

For now, it seems like and Observer is the more practical role for our assessments because it doesn't require additional time from the other security reviewers.

[stale[bot]]

This issue has been automatically marked as inactive because it has not had recent activity.

[JustinCappos]

Based upon the meeting today, I'd like to close this as will not fix. Please re-open if anyone objects.

[MVrachev]

I didn't have the opportunity to join the meeting when you had discussed this issue.

First, to be clear: when I say inexperienced in my comments, I mean specifically inexperienced in security meaning the focus is not on the age or job position, but instead experience.

When thinking for security assessments some of the assumptions I make are:

1. The depth and scope of the security expertise for each member in sig security members differs.
2. There is some basic level of understanding needed to become a security reviewer in a security assessment.
3. We want to make this group welcoming no matter if you are a security principal engineer in a top-tier company or a student keen to learn.
4. We want to encourage a broader engagement in the security assessments no matter if the reviewers are experienced
   or inexperienced keen to learn.

If we accept that 1 and 2 are facts and strive towards 3 and 4, this will lead to the need for a Trainee role (Observer sounds too passive to me, I would prefer a term emphasizing that one should be active, not just observing).

Without a role like there are two possible scenarios:

1. the inexperienced among us won't participate in the assessments because they don't feel knowledgeable enough.
2. a couple of inexperienced developers decide to join in a specific assessment, fill the places for reviewers and we end up in a situation when only the lead reviewer has ever done something similar and has a good basic knowledge about security.

This role will allow the inexperienced among us who want to join in the assessments to be open about that and still participate.
Additionally, if people are honest and admit they are inexperienced and want to join as a Trainee, we can better balance
the reviewers vs inexperienced ratio.

[MVrachev]

Having said the above, how we should define a role like this is another question.

As we had discussed above, one requirement to become a Trainee is to find a mentor who is willing to help you find answers to your questions and if needed give you guidance.

Probably, it will be best if this mentor is another reviewer/lead reviewer from the same assessment where the trainee wants to join. That way project specific questions could be discussed between the mentor and the trainee.

[sladyn98]

@MVrachev Is there any progress on this, I would like to learn as a Trainee

[rficcaglia]

I’d like to split this off as a separate SIG or WG for a few reasons. I
would prefer to decouple training in security assessments, code auditing,
and pen testing from the SIG review (previously “assessment”) process since
the scope has changed (or rather has now been defined more precisely).

Secondly, training the community deserves its own focus and resources. I
can speak from first hand experience that finding experienced security
assessors and code auditors is near impossible. And the open community
should nurture its own community members. I see too many CNCF projects who
need security expert volunteers and not enough community experts to go
around!

Finally I think there are other communities looking at this where we can
collaborate and cross pollinate. One example is NIST which developed NICE

If anyone is interested comment here - especially if you have experience
with security training and mentoring.

On Fri, Apr 16, 2021 at 12:04 PM Sladyn @.*> wrote:

@MVrachev https://github.com/MVrachev Is there any progress on this, I
would like to learn as a Trainee

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/cncf/sig-security/issues/256#issuecomment-821493974,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGENIRT2GMRDALEKXBJCGDTJCC27ANCNFSM4IO7YLRQ
.

[MVrachev]

I’d like to split this off as a separate SIG or WG for a few reasons. I would prefer to decouple training in security assessments, code auditing, and pen testing from the SIG review (previously “assessment”) process since the scope has changed (or rather has now been defined more precisely). Secondly, training the community deserves its own focus and resources. I can speak from first hand experience that finding experienced security assessors and code auditors is near impossible. And the open community should nurture its own community members. I see too many CNCF projects who need security expert volunteers and not enough community experts to go around! Finally I think there are other communities looking at this where we can collaborate and cross pollinate. One example is NIST which developed NICE If anyone is interested comment here - especially if you have experience with security training and mentoring.
…
On Fri, Apr 16, 2021 at 12:04 PM Sladyn @.*> wrote: @MVrachev https://github.com/MVrachev Is there any progress on this, I would like to learn as a Trainee — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#256 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGENIRT2GMRDALEKXBJCGDTJCC27ANCNFSM4IO7YLRQ .

I think it does make sense to be a separate SIG or WG, but the question is are there enough experienced security reviewers ready to start such an endeavor.
The suggestion to use a Trainee role in the already happening assessments was just to simplify things.
It seems easier to mentor someone on something you are already thinking about and analyzing.

[rficcaglia]

Well to split the 2 ideas:

• experienced security assessors and pen testers and code auditors who can
  volunteer time to train the community, not for any particular review or
  assessment goal, but more generally - there are very few, especially with
  CNCF project knowledge - but one can teach 2, each who can teach 4, etc.
  there are also volunteer organizations outside our community that may be
  helpful. Also internship programs which grant money to security training,
  etc. it won’t be particularly helpful for the short term, or for this
  SIG’s needs, but it needs to start sometime otherwise we will be having
  this conversation in 5, or 10 years still

Next steps: I’ll create a PR in the proper place and link here

• trainees/training for this SIGs review process - as the current process
  is much more about documentation review, I don’t think there is much, if
  any training needed. Just participation as a volunteer on a couple, and
  attention to detail. To lead a review, a bit of project management
  experience and communication skill is helpful running things to ground.
  The reviews I have so far participated in were security focused projects,
  however - so I can see where there might be challenges if a project has
  absolutely no security architecture and no project members who know
  security. Then they may want to rely on the SIG to provide them the
  answers, instead of just asking questions, but I think the new review
  process would consider that out of scope. (As currently defined the TOC
  probably wouldn’t even request a review from this SIG for a non security
  project I think?)

Next steps: maybe create a GHI to “register” interested trainees, ie novice
volunteers, so we have a ready list for each review. Then we can curate
some lessons learned from previous reviews, or case studies/after action
reports that note how the process worked (or didn’t)

On Mon, Apr 19, 2021 at 3:27 AM Martin Vrachev @.*>
wrote:

I’d like to split this off as a separate SIG or WG for a few reasons. I
would prefer to decouple training in security assessments, code auditing,
and pen testing from the SIG review (previously “assessment”) process since
the scope has changed (or rather has now been defined more precisely).
Secondly, training the community deserves its own focus and resources. I
can speak from first hand experience that finding experienced security
assessors and code auditors is near impossible. And the open community
should nurture its own community members. I see too many CNCF projects who
need security expert volunteers and not enough community experts to go
around! Finally I think there are other communities looking at this where
we can collaborate and cross pollinate. One example is NIST which
developed NICE
https://www.nist.gov/itl/applied-cybersecurity/nice/nice-framework-resource-center/current-version
If anyone is interested comment here - especially if you have experience
with security training and mentoring.
… <#m_-1829632121454891051_>
On Fri, Apr 16, 2021 at 12:04 PM Sladyn @.*> wrote: @MVrachev
https://github.com/MVrachev https://github.com/MVrachev Is there any
progress on this, I would like to learn as a Trainee — You are receiving
this because you were mentioned. Reply to this email directly, view it on
GitHub <#256 (comment)
https://github.com/cncf/sig-security/issues/256#issuecomment-821493974>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGENIRT2GMRDALEKXBJCGDTJCC27ANCNFSM4IO7YLRQ
.

I think it does make sense to be a separate SIG or WG, but the question is
are there enough experienced security reviewers ready to start such an
endeavor.
The suggestion to use a Trainee role in the already happening assessments
was just to simplify things.
It seems easier to mentor someone on something you are already thinking
about and analyzing.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/cncf/sig-security/issues/256#issuecomment-822360932,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGENIQC4B5Q6ESYHFP3ONDTJQARHANCNFSM4IO7YLRQ
.

[sladyn98]

@rficcaglia This sounds like a good idea