Osquery: Ubuntu 16.04 Xenial repo needed

[theopolis]

While the 14.04/Trusty package will install on 16.04, the freight-generated repository complains about a weak hashing algorithm and missing/invalid Date field.

[theopolis]

Seem like a newer version of freight is needed on the build hosts: https://github.com/AtnNn/freight/commit/6da26b6575610cd667b4c2b53aeb9c31d3adc46e

I can also generate a new GPG key for Xenial, but we don't yet have Xenial build hosts so there won't be distro-built packages.

[theopolis]

For the specific errors:

W: https://osquery-packages.s3.amazonaws.com/trusty/dists/trusty/Release.gpg: Signature by key 1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B uses weak digest algorithm (SHA1)
W: Invalid 'Date' entry in Release file /var/lib/apt/lists/partial/osquery-packages.s3.amazonaws.com_trusty_dists_trusty_Release
E: Failed to fetch https://osquery-packages.s3.amazonaws.com/trusty/dists/trusty/Release  Unable to find expected entry 'main/source/Sources' in Release file (Wrong sources.list entry or malformed file)
E: Some index files failed to download. They have been ignored, or old ones used instead.

[alexlo03]

+1 thanks for doing this! please call me out if I can help.

[fuzzball5000]

I have a need to package for 16.04, how can I help?

[theopolis]

I'm about 50% finished, we have 16.04 building with the provision scripts, and there are 2 build hosts for testing PRs: https://jenkins.osquery.io/label/ubuntu16/ There's an equivalent build host on the private package generation Jenkins.

The last thing I need to wire together is the freight repo creation and auto-publishing/signing of the debs. I'm aiming for EOW but cannot make any promises. ;)

[fuzzball5000]

Brilliant, thanks :) Feel free to reach out if there is anything I can do to help!

[theopolis]

Ok, repos are created, and package build jobs are setup. The last thing we need is to cut a version and watch the magic happen. Anything less than master wont pass unit tests so it will be too much of a headache to retroactively build 1.7.4. I will update the osquery.io downloads page as soon as 1.7.5 is released.

[katherinealbany]

Hello, I note that within tools/deployment/make_linux_package.sh it supports the packaging of a systemd service unit (tools/deployment/osqueryd.service) for centos7 and rhel7. Can you confirm that a Ubuntu 16 release would include a compatible systemd service unit?

[theopolis]

We aren't including the service in Xenial, I can change that!

[katherinealbany]

:+1: Thank you.

[katherinealbany]

Thank you for your continued work towards a 16.04 release. However the systemd service unit (tools/deployment/osqueryd.service) which would now appear to be packaged with 16.04 contains RHEL specific contexts around EnvironmentFile which would not function as desired.

[theopolis]

I boot tested 16.04 using our Vagrant targets, the service started just fine. :) I don't admin any 16.04 machines, so my ability to make sure everything is sane is limited; if changes are needed I would be happy to review pull requests.

[fuzzball5000]

make_linux_package.sh seemed to be erroring but continuing (unexpected, given set -e?) over unquoted variables in single [ ], doubled up here: https://github.com/facebook/osquery/pull/2159/files

Faulting function was ahead of the one used to deploy the service unit, so possibly this was causing the problem? Built package with this change and confirmed service is starting on 16.04;

ubuntu@ip-172-30-1-51:~/osquery/tools/deployment$ sudo service osqueryd start
ubuntu@ip-172-30-1-51:~/osquery/tools/deployment$ ps -ef | grep osquery
root 31652 1 0 18:14 ? 00:00:00 /usr/bin/osqueryd --flagfile /etc/osquery/osquery.flags --config_path /etc/osquery/osquery.conf
root 31655 31652 0 18:14 ? 00:00:00 osqueryd: worker
ubuntu 31673 1167 0 18:20 pts/0 00:00:00 grep --color=auto osquery

[fuzzball5000]

Actually, set -e enforces a quit if any process exits with code !=0, so wouldn't catch bash syntax errors. Could be the fix we're looking for.

[katherinealbany]

If I have the time I may look at submitting a pull request. I the mean time, the issue is that the path /etc/sysconfig used in the systemd service is RedHat specific. On Debian/Ubuntu machines the appropriate place for this is /etc/default

[fuzzball5000]

Looking into this now.

[fuzzball5000]

Moved into new issue for better tracking: https://github.com/facebook/osquery/issues/2160

[theopolis]

Thanks @katherinealbany for the details and @fuzzball5000 for tracking that in #2160. I imagine we'll need to make more fixups/etc to the packages over time so let's call the Xenial repo finished and iterate on improvements in other issues. :)