Osquery: pkg.osquery.io is 403 forbidden as of 15:00 UTC Feb. 21

[ageis]

The repository pkg.osquery.io seems not to have an open IAM/access policy.. It's a redirect to the S3 resource: https://osquery-packages.s3.amazonaws.com. Naturally, there's no signed Release file that can be retrieved either.

E: Failed to fetch https://pkg.osquery.io/deb/dists/deb/InRelease  403  Forbidden [IP: 54.231.121.91 443]
E: The repository 'https://pkg.osquery.io/deb deb InRelease' is not signed.

[obelisk]

Heyo, can you elaborate on what you're trying to do? There is no InRelease file for that repo but there is a Release file with an accompanying Release.gpg.

[ageis]

Was just fetching via normal apt sources, but it appeared the IAM policy on
the bucket denied me (or a proxy or load balancer between me and S3). I
know the repository is valid, apt is only complaining cause it couldn't
receive signed Release file. It's some kind of S3 access or security
groups/firewall issue perhaps. Knowing nothing about your infrastructure of
course...

I also make the point that 54.231.121.91 = s3-1-w.amazonaws.com. Maybe it's a regional discrepancy (west/east)?

[muffins]

Interesting, I am having quite a bit of trouble replicating. We definitely have open IAM policies on our files, and as @obelisk mentioned there is a Release and a Release.gpg present in our repo. Could you possibly show us the relevant entries from your sources.list? Or provide more information about your environment?

~ ❯ export OSQUERY_KEY=1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys $OSQUERY_KEY
sudo add-apt-repository 'deb [arch=amd64] https://pkg.osquery.io/deb deb main'
sudo apt-get update
sudo apt-get install osquery
[sudo] password for thor: 
Executing: /tmp/tmp.CDiHtjqvUP/gpg.1.sh --keyserver
keyserver.ubuntu.com
--recv-keys
1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B
gpg: requesting key C9D8B80B from hkp server keyserver.ubuntu.com
gpg: key C9D8B80B: "osquery (osquery) <[email protected]>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Hit:1 http://us.archive.ubuntu.com/ubuntu xenial InRelease                 
Hit:2 http://security.ubuntu.com/ubuntu xenial-security InRelease          
Hit:3 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease      
Hit:4 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease
Ign:5 https://pkg.osquery.io/deb deb InRelease    
Get:6 https://pkg.osquery.io/deb deb Release [271 B]
Get:7 https://pkg.osquery.io/deb deb Release.gpg [275 B]
Get:8 https://pkg.osquery.io/deb deb/main amd64 Packages [5,816 B]
Fetched 8,063 B in 3s (2,545 B/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libllvm4.0 linux-headers-4.10.0-28 linux-headers-4.10.0-28-generic linux-image-4.10.0-28-generic linux-image-extra-4.10.0-28-generic
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
  osquery
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 17.7 MB of archives.
After this operation, 51.0 MB of additional disk space will be used.
Get:1 https://pkg.osquery.io/deb deb/main amd64 osquery amd64 2.11.0-1.linux [17.7 MB]
Fetched 17.7 MB in 8s (2,123 kB/s)                                                                                                         
Selecting previously unselected package osquery.
(Reading database ... 249211 files and directories currently installed.)
Preparing to unpack .../osquery_2.11.0-1.linux_amd64.deb ...
Unpacking osquery (2.11.0-1.linux) ...
Processing triggers for systemd (229-4ubuntu21.1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up osquery (2.11.0-1.linux) ...
1319 1098
Processing triggers for systemd (229-4ubuntu21.1) ...
Processing triggers for ureadahead (0.100.0-19) ...
~ ❯ osqueryi --version
osqueryi version 2.11.0

edit: this was also tested using an east coast IP :)

[ageis]

@muffins @obelisk I have to doubt this could be anything related to my environment; I've tested reaching the endpoint from multiple virtual machines in many regions/networks and received the same result. If you have logging enabled on your S3 bucket, you might get more info. The headers have request IDs and such. Maybe you're having trouble replicating cause your Facebook IP range is allowed? Heh... just a random thought.

curl -I https://osquery-packages.s3.amazonaws.com
HTTP/1.1 403 Forbidden
x-amz-bucket-region: us-east-1
x-amz-request-id: 24FC4E1737E9320B
x-amz-id-2: ZIFB7GJmZLyLrpuNhVsgBSL03GecdJYks/x23ph3BOawVAWGFDUk6pNCYsLe7lUl7E53PjtCulw=
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Sat, 24 Feb 2018 00:29:56 GMT
Server: AmazonS3

[thedrow]

I'm experiencing the same issue.

[muffins]

Thanks @thedrow, I'm still trying to figure out what's up. @ageis yeah the root site for osquery-packages.s3.amazonaws.com shouldn't be accessible, that's on purpose, but the apt repo structure should be fine. I've checked the IAM/access policy and it's the same as it's ever been - read perms for public, so I'm still not sure what's up. I was somewhat AFK for the weekend and I'm still not back formally yet, so my apologies for the delay on this issue, but we'll hopefully figure out what's up soon. The tests I ran were not from FB corp space, however it was mentioned that one of our internal teams might configure restrictions somewhere along the line to AWS, however that explanation felt pretty thing to me. I'll keep digging, thanks for bearing with us!

[fmanco]

Hi @ageis and @thedrow could you please provide distro, version and apt version? Also the full output of your command would be nice.

I don't think this has much to do with permissions/access control. The file InRelease actually doesn't exist and you're not supposed to be able to curl https://osquery-packages.s3.amazonaws.com. There are two ways to sign the release file (1) an InRelease file that has signatures inline and (2) a Release file with a companion Release.gpg that contains the signatures. We use the latter. Traditionally that should be fine, as you can see from @muffins output he's apt is ignoring InRelease and pulling Release and Release.gpg.

From what I can read on https://wiki.debian.org/DebianRepository/Format and https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=848279 it is however possible that Release.gpg was deprecated and apt is forcing you to fetch InRelease. I wanted to confirm this by checking the versions you're running and output.

Thanks

[ageis]

@fmanco I've run some apt repositories and as far as I can remember InRelease has always been GET requested even if it's not there. So that's just the error I'm getting. The latter scheme you with Release and Release.gpg is what I've always done too and AFAIK is still valid. Again... Here's the full error on apt-get update.

E: Failed to fetch https://pkg.osquery.io/deb/dists/deb/InRelease  403  Forbidden [IP: 52.216.97.35 443]
E: The repository 'https://pkg.osquery.io/deb deb InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
````
I uninstalled osquery (which I'd done manually with dpkg) and tried this again. did apt-get clean and autoclean.

$ apt-get install osquery
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package osquery is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'osquery' has no installation candidate
```

This is Debian buster (next-stable) apt 1.6~alpha7 Something has definitely changed. I think possibly the issue is that apt expects a 404 on InRelease but it's getting a 403?

[thedrow]

I'm using the s3 URI directly, should I change it to https://pkg.osquery.io/?
I'm on Ubuntu 17.10.

[fmanco]

Okay found the problem

fail InRelease on non-404 HTTP errorcodes

from https://github.com/Debian/apt/blob/master/debian/changelog.

So we either need to 404 or provide an InRelease file. Thanks for the help, will come back here once it's fixed.

[fmanco]

@thedrow using https://pkg.osquery.io/ won't make a difference as it is just a 301 to https://osquery-packages.s3.amazonaws.com/.

[fmanco]

@ageis @thedrow This should be fixed, feel free to reopen in case you have further problems.