Owasp-modsecurity-crs: Repository [re-]organization

[zimmerle]

I am amin to built a history on the releases and releases candidates for OWASP CRS and I am having some troubles. I would like to propose a repository organization regardless of the release cycle.

The aiming is to use: tags and branches to have three different things:

• development tree (for different major versions).
• releases code under a tag.
• branches with stable code.

Applying this concept under the repository that we have today the modification will be:

• Branches:
  
  
  
  • v2/master (development branch for v2) _[current master]_
  
  
  • v3/master (development branch for v3) _[current v3.0.0-rc3]_
  
  
• Tag:
  
  
  
  • v2.0.0
  
  
  • (..)
  
  
  • v3.0.0-rc1 _[last commit from -rc1]_
  
  
  • v3.0.0-rc2 _[last commit from -rc2]_
  
  
  • v3.0.0-rc3 _[released commit]_
  
  
  • (tags for the rest of the stuff, following the same schema)
  
  
• Branches for variations:
  
  
  
  • v2/variation/honeypot _[current owasp-honeypot]_
  
  

I think it is safe the delete the branch trunk.

I understand that the current -rc1 branch has the -rc1 plus fixes. But, I think it will be easy to understand if -rc1 has nothing else but -rc1.

If you guys agree with the modifications I will be happy to apply it on the repository. Should be no damage.

That will help me a lot to establish a compatible marking for libModSecurity regarding the OWASP CRS.

[csanders-git]

I think you might be right about this - but as final is coming up in the next few days perhaps we should wait on renaming things till the release date for that (or the day before). The 8th. I think on that date it'd make some sense to do this. Perhaps we'd still have a v3/dev also that we could then tag and merge as releases hit.

[lifeforms]

In our INSTALL we instruct users to clone the default branch, and this is also shown to visitors by default, so we should make master the stable released version. (I don't know what GitHub does if a repo is missing a master branch or if we can set another default) But we can dev in another one.

[csanders-git]

@lifeforms I think after talking to Felipe the major concern is that we're not tagging our releases. But yes, with the release coming we have some decisions to make. I think the v2/master and v2/honeypot are worth making note of @rcbarnett I know this means a slight change to the DHP. For v3 I think if we can select an alternative master and make it v3/master that'd be nice but otherwise as @lifeforms noted just master is fine and then a v3-dev branch.

[zimmerle]

Hi @lifeforms,

You can specify to GitHub which branch is the master branch of your tree [independent of the name].

I understand that there is a note on the INSTALL file, but, it will be easy for the user to figure which is which just by looking at the branches lists. Using a repo structure similar to other projects in GitHub. For instance:

• The branch v3.0.0-rc2 is neither the version -rc2 or the development branch. Correct me if I am wrong, but, technically it is a leftover for the v3.0.0 development which is now being held on the branch v3.0.0-rc3. So why we need to have it?

I want to say that ModSecurity version 3 is compatible with OWASP CRS, it was compatible with -rc1 by the time that it was released. It was not anymore. So I have to point the user to a commit hash, which is very hard to remember. Instead I will prefer to point a user to `tag' or to a branch, that would be perfect.

Let me make those modifications on my own tree, so you guys can get a better feeling on this modification... Working on it...

[csanders-git]

The RC2 branch should be what one would normally tag in a Dev branch. It should be equivalent to the RC2 available here (https://github.com/SpiderLabs/owasp-modsecurity-crs/releases). I will start using tagging in the future I think when move into our next version.

[zimmerle]

Here goes my suggestion implemented on the top of my fork:

• https://github.com/zimmerle/owasp-modsecurity-crs/branches/all

The original one:

• https://github.com/SpiderLabs/owasp-modsecurity-crs/branches/all

I would host the `gh-pages' elsewhere. There is a git repository for the wiki, like every other github project, which is: owasp-modsecurity-crs.wiki.git. I would try to use it instead of custom html pages.

Notice that i just changed the branch part, I did not performed the tag creation as it is a simple demo to illustrate what i am suggesting.

[csanders-git]

yeah this seems to make sense to do in line with our release. we'd also create a v3/dev branch but I am not opposed @lifeforms ?

[dune73]

I'm a bit late to the show, but I would like to throw in my support for @zimmerle. We need tags, we need branches for stablecode - and we need a development tree.

We have postponed the question if we will do point releases or not, but I really think we should. So in my eyes, the next releases / tags might be:

• 3.0.0 - stable release
• 3.0.1 - stable release with more FPs fixed
• 3.0.2 - stable release with more FPs fixed
• ... coming to an end some time _after_ 3.1.0 is out.
• 3.1.0 - stable release with new features
• 3.1.1 . stable release with more FPs fixed
• ...

This will mean backports, which is cumbersome. But it also forces us to cover everything via automatic testing, which will add to the quality of the ruleset.

I am not sure how this maps to the git update idea, though.

[csanders-git]

yeah - so at this point I think everyone is supporting this idea - @zimmerle we can work together to implement this on the 7th?

[zimmerle]

It will be a great pleasure ;)

Thank you for the support.

[dune73]

Could we point the update-script to the branch and then get continous updates from a stable branch until you point the update script to the next higher branch?

[csanders-git]

Can you explain that a little bit @dune73 ?

[dune73]

The upgrade.py script checks if the crs-directory is a git directory. If it is, it issues a git pull --ff-only.

So if users download from the branch initially, they will be fine with updates from that branch. Until they clone from the next higher branch.

But if users install from a tag / tar archive, there is no git repository and maybe we need to provide a way for the script to identify the correct branch (or supply via command line) and transform the local filetree into a git repo which can subsequently be updated by script.

[zimmerle]

@dune73 if they download from the github (as a .zip file) that is true. and the script will be very welcome.

I think it is safe to keep updating from v3/master, that branch should hold and only holds code that is already verified by you guys. In other words: it is stable, although it may not be released yet.

Notice also that the tag is also available like a normal branch in the git repository. You can checkout to a tag, consequently creating a new branch starting by the given tag. The big difference between a branch and a tag is the fact that _you cannot update a tag_. At least, you should not :)

[dune73]

I'm in, if v3/master get renamed to v3.0/master and gets a sibling v3.1/master eventually. I think the branch needs to make it totally clear what release line / tag series it belongs to.

[csanders-git]

never thought about that, felipe does that ruin your interchangeability image?

[zimmerle]

In ModSecurity repo we use v3/master. But that depends on your rule to make major change on the project. v3.0/master seems fine to me.

[dune73]

@lifeforms : you still around?

[lifeforms]

Sorry. I am currently spending my last RC3 moments battling a LFI bypass that I'm not understanding. :\ I don't have thought long about branching, you seem to be on top of it. My only concern is that a user coming to the default GitHub view, or doing a default git pull, will see the stable version - not a possibly insecure development version or an outdated version. The rest is internals/for developers so less critical.

[dune73]

That makes a lot of sense, @lifeform, but I am not sure how we can guarantee this.

@zimmerle: do you have an idea?

[lifeforms]

We usually use a dev -> beta -> master branches and tagging for versions so I don't know a lot about working with versions. Keeping a master branch around and merging the latest release to that would be enough I think. Or that branch could be named something else like Felipe said but I haven't tried that myself on GitHub yet.

[dune73]

Yes, that could do the trick.

[zimmerle]

Hi guys,

I can handle the "rename", won't be a problem. Check my personal tree, I already did most part of the work there (at least regarding renaming). Here:

https://github.com/zimmerle/owasp-modsecurity-crs/branches/all

(notice that it is not up-to-date)

[csanders-git]

So i moved most of the code, since we said we were doing that today. I made v2/master the default till Thursday.
@lifeforms @dune73
I still left the master there in case we wanted to merge either of the two PRs (we should decide). I can simply remove the new v2/master and rebase it from the master we have now.

Either of you know what the trunk branch is supposed to be doing?

[dune73]

No opinion on trunk. @zimmerle?

@csanders-git: I really think it should be v3.0/master as there is going to be a v3.1/master sooner or later. Or are there reasons to stick to v3/master? (This includes using v2.2/master of course. Just for clarity and to get a clean layout.)

[csanders-git]

Alright take two... :)

[dune73]

Thank you @csanders-git.

[cellscape]

If I want to fetch latest (preferably stable) changes from v3 series, what branch should I use? Currently master points to v2 series and v3.0/master will be one day replaced by v3.1/master, so can't use either of them.

[dune73]

Use v3.0/master. This will become the default in the next few hours. This will carry the stable versions for the time being.

One day, it will be replaced by v3.1/master.

[cellscape]

I understand that it will be replaced one day but this means I have to check list of branches from time to time and see if new branch was added. This is a bit inconvenient, especially if updating rules via script. I guess for other users it would be convenient as well, to be able to pull stable changes from one branch. I'm not insisting though, just hoping it would make sense for developers as well :)

[dune73]

I see the inconvenience in this. However, if you update by script and we switch the branch seamlessly, you will get new rules into your productive website automatically. False positives could kill your site. That would be inconvenient to you. And potentially disastrous for our project if we are identified as the root cause of a major outage.

However, if you are forced to switch the branch by hand, then new rules will have to be activated by hand via switching the branch on your site. That way we are sure to have your attention. You read the changelog, you are aware that testing is due, etc.

We have not totally made up our mind, but the idea is that minor releases will be bugfix releases and releases covering false positives. You should be able to update these automatically without any trouble.

Major releases (-> 3.0, 3.1, etc.) will bring new rules and new features.

You should subscribe to the mailinglist to be informed about update. Following @modsecurity or @ChrFolini on twitter should also do the trick.

[cellscape]

Thanks a lot for explanation! Such scheme makes sense and manual check for new releases is fair price for lack of false positives.

[dune73]

Thank you. Glad you agree.

CRS3 will be out later today. Please help us spread the word!

[csanders-git]

@cellscape I also 'THINK' that once we have a non-pre release navigating to https://github.com/SpiderLabs/owasp-modsecurity-crs//releases/latest will yield the latest one, we'll see :-X

[dune73]

With the successful release and the smooth transition of the src trees, I am closing this issue.