Osquery: Steps towards a osquery 4.0.1 release

[groob]

(moving some discussions from slack and querycon)

• We'd like to decide on which branch becomes the new master branch (old master vs experimental).
• We'd like to create a new stable release soon.

Proposal:

• Make experimental the new master branch.
• Add the CMake support implemented in the osql-experimental repository.
• Test/Bugfix minor changes and create a Osquery 4.0.0 release.

cc @theopolis @PoppySeedPlehzr @directionless @zwass @alessandrogario

[muffins]

+1 on making experimental the new master. A huge amount of work has been poured into that branch for stability and performance, and we've been running variants of it on macos and Windows for some time with good success.

I'd also be really in favor of bringing back CMake support. I really love buck, but for Windows there's not currently well supported ways to generate debug builds for using Visual Studio solutions. One could argue that WinDBG is _totally_ sufficient :) but I'm not that cool, and visual studio was invaluable for debugging thread deadlocks in the system service. That being said, if someone has a better way to generate debugging symbols for osqueryd on Windows using buck, please hit me up!

As for the testing frameworks a lot of things have changed I believe. CMake was the primary driver for many of our testing runs, so if we can bring that support back we _should_ be able to get "good" testing on experimental for free.

• just my 2c

[fmanco]

Really like the idea of moving experimental to master. Regarding CMake is your proposal to kill Buck completely or have both? Independently I would much rather have osql's CMake implementation than our old one from master...

[groob]

We should have buck and CMake side by side, assuming that Buck users want to maintain the buck rules moving forward. CMake will likely be the community supported build rules.

[grantseltzer]

I'm all for this, definitely willing to help move over the progress from osql as well.

[theopolis]

I agree, and this might be a nitpick, but can we compare and contrast rebasing the commits in experimental on top of master instead of renaming?

Edit: I recommend renaming experimental to master if you can fast-forward from existing checkouts of master to experimental.

[fmanco]

We froze master exactly such that we could just move commits over from experimental to master in the future. Not sure if there were commits to master meanwhile but if not I suggest we do just that.

[theopolis]

I suggest we do just that.

So who is going to do the honors? I removed the branch protection rule for master. I nominate @fmanco.

[theopolis]

@groob, your suggestion

Add the CMake support implemented in the osql-experimental repository.

This seems ideal but it doesn't seem like a drop-in replacement. I think @smjert would need to apply some refactors. I am unsure how complex this is.

[Smjert]

@groob, your suggestion

Add the CMake support implemented in the osql-experimental repository.

This seems ideal but it doesn't seem like a drop-in replacement. I think @Smjert would need to apply some refactors. I am unsure how complex this is.

It shouldn't be too difficult. Originally the CMake build system wasn't built with the submodule there, it was "inline" with the code, so when we had to change, I just introduced a couple of functions (add_osquery_library, add_osquery_executable), replaced "all" add_library/add_executable and added the OSQUERY_SOURCE_DIR variable to have the correct source path for other uses.
So it should just be a matter of some string replaces.

EDIT: Also the directory structure and CMakeLists.txt paths still follow the osquery source dirtectory structure.

[theopolis]

We also need to merge the changes for the Windows 3.4.0 release. Right? @muffins

451b5154c355da2d902d5784a48984d6b5cff88a
f1ee6844dc22ba06d85e82e0738eb90d255eac5d

[theopolis]

The above Windows 3.4.0 concern is complete with #5590.

[muffins]

Correct, thanks @theopolis!

[directionless]

FTR master was fast forwarded.

[directionless]

I've made a release milestone -- https://github.com/osquery/osquery/milestone/42

[scpeters]

I just tried updating osquery to 3.4.0 in https://github.com/Homebrew/homebrew-core/pull/41334 but stopped when I saw cmake was missing.

[theopolis]

What’s next, can we add the Azure CI badges to README and turn them on for PRs. Or do we want to build packages before CI?

[groob]

I know it's unorthodox, but we should offer an unsigned tar.gz of the osquery binary for all platforms to get some early testing from volunteers in.

CI Badges would be excellent.

[alessandrogario]

I know it's unorthodox, but we should offer an unsigned tar.gz of the osquery binary for all platforms to get some early testing from volunteers in.

CI Badges would be excellent.

That would be great; some people have offered help in testing the new release in their staging areas, so the sooner the first milestone is closed the sooner that feedback is provided.

I would be in favor of releasing as soon as the CI is working fine (badges + flaky tests disabled) and then maybe create a new milestone right away for 4.0.1 containing PRs

[directionless]

Can we close this issue? I don't think it has anything that the 4.0.1 milestone does not