Datadog-agent: Broken repository indexes for Ubuntu 14.04 "trusty"

Created on 9 Dec 2019  路  6Comments  路  Source: DataDog/datadog-agent

Describe what happened:

apt-get update is broken:

W: Failed to fetch https://apt.datadoghq.com/dists/stable/main/binary-amd64/Packages  Hash Sum mismatch

E: Some index files failed to download. They have been ignored, or old ones used instead.

with the following repository configuration:

$ cat datadog.list 
deb https://apt.datadoghq.com stable main

Repository for agent 6.x doesn't work as well:

$ cat datadog.list 
deb https://apt.datadoghq.com/ stable 6
$ sudo apt-get update >/dev/null
W: Failed to fetch https://apt.datadoghq.com/dists/stable/6/binary-amd64/Packages  Hash Sum mismatch

E: Some index files failed to download. They have been ignored, or old ones used instead.

Describe what you expected:

According to this page, agent should be available for 14.04:
https://docs.datadoghq.com/agent/basic_agent_usage/?tab=agentv6#supported-os-versions

Steps to reproduce the issue:

Simply run apt-get update with pre-configured datadog repository on 14.04.

Additional environment details (Operating System, Cloud provider, etc):

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.6 LTS
Release:    14.04
Codename:   trusty

$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="14.04.6 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.6 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
picture defanator

Most helpful comment

Thanks for all the additional details here @defanator!

Interestingly there was an InRelease file in your local apt cache (/var/lib/apt/lists/apt.datadoghq.com_dists_stable_InRelease), dated Aug 26. This matches a change we did on the APT repo on that day to start providing an InRelease file on the repo, which was subsequently reverted after we found out that older versions of Debian (7 and below) and Ubuntu (12.04 and below) disable support of the InRelease file.

For some reason even after the InRelease file was removed from the repo, it appears that apt continued to read the InRelease file it had in its cache and trusted its contents, even though it was outdated.

In any case, what you did to resolve the issue, i.e. clearing the local cache of datadog-related repository metadata file (find /var/lib/apt/lists/ -name "*datadog*" -delete) and then re-running apt-get update, is indeed a good solution 馃憤

I'll go ahead and close this issue then, but feel free to comment here if anything new comes up, thanks!

picture olivielpeau on 10 Dec 2019
🎉5 ❤3 👍2

All 6 comments

Hi @defanator, thanks for reporting this issue. We have checked the integrity of the repository on our end, is the issue still happening consistently on your end?

If you still have issues, please post the full output of sudo apt-get update (if there's anything in the output you do not wish to post publicly, feel free to remove the sensitive part or open a support case by sending an email to [email protected] and mentioning this issue)

olivielpeau picture olivielpeau on 9 Dec 2019

@olivielpeau yes, we're still having the issue on a number of hosts where we've been using datadog repository and agent 5.x for a long time:

# pwd && cat datadog.list 
/etc/apt/sources.list.d
deb https://apt.datadoghq.com stable main
# apt-get update
Ign http://us-west-2.ec2.archive.ubuntu.com trusty InRelease
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates InRelease
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports InRelease
Hit http://us-west-2.ec2.archive.ubuntu.com trusty Release.gpg                 
Hit http://us-west-2.ec2.archive.ubuntu.com trusty Release                     
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/main Sources        
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/restricted Sources  
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/universe Sources    
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/multiverse Sources  
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/main amd64 Packages 
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/restricted amd64 Packages
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/universe amd64 Packages
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/multiverse amd64 Packages
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/multiverse Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/restricted Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/universe Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/main Sources 
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/restricted Sources
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/universe Sources
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/multiverse Sources
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/main amd64 Packages
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/restricted amd64 Packages
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/universe amd64 Packages
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/multiverse amd64 Packages
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/main Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/multiverse Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/restricted Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/universe Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/main Sources            
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/restricted Sources      
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/universe Sources            
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/multiverse Sources          
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/main amd64 Packages         
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/universe amd64 Packages  
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/multiverse amd64 Packages
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/main Translation-en      
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/multiverse Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/restricted Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/universe Translation-en  
Ign http://us-west-2.ec2.archive.ubuntu.com trusty/main Translation-en_US   
Ign http://us-west-2.ec2.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us-west-2.ec2.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us-west-2.ec2.archive.ubuntu.com trusty/universe Translation-en_US
Hit https://apt.datadoghq.com stable InRelease                              
Hit http://security.ubuntu.com trusty-security InRelease                    
Hit http://security.ubuntu.com trusty-security/main Sources             
Ign https://apt.datadoghq.com stable/main Translation-en_US
Ign https://apt.datadoghq.com stable/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Sources
Hit http://security.ubuntu.com trusty-security/main amd64 Packages
Hit http://security.ubuntu.com trusty-security/universe amd64 Packages
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Fetched 26.6 kB in 2s (10.9 kB/s)            
W: Failed to fetch https://apt.datadoghq.com/dists/stable/main/binary-amd64/Packages  Hash Sum mismatch

E: Some index files failed to download. They have been ignored, or old ones used instead.

apt keys:

# apt-key list
/etc/apt/trusted.gpg
--------------------
pub   1024D/437D05B5 2004-09-12
uid                  Ubuntu Archive Automatic Signing Key <[email protected]>
sub   2048g/79164387 2004-09-12

pub   1024D/FBB75451 2004-12-30
uid                  Ubuntu CD Image Automatic Signing Key <[email protected]>

pub   4096R/C0B21F32 2012-05-11
uid                  Ubuntu Archive Automatic Signing Key (2012) <[email protected]>

pub   4096R/EFE21092 2012-05-11
uid                  Ubuntu CD Image Automatic Signing Key (2012) <[email protected]>

pub   2048R/DE57BFBE 2014-06-24
uid                  SaltStack Packaging Team <[email protected]>
sub   2048R/17928113 2014-06-24

pub   2048R/C7A7DA52 2011-03-16 [expired: 2018-03-04]
uid                  Datadog Packages <[email protected]>

pub   4096R/382E94DE 2016-06-29 [expires: 2022-06-28]
uid                  Datadog, Inc <[email protected]>
sub   4096R/F432F6E0 2016-06-29 [expires: 2022-06-28]
sub   4096R/8387EEAF 2016-06-29 [expires: 2022-06-28]
sub   4096R/BFF6291E 2016-07-11 [expires: 2022-07-10]

pub   2048R/7BD9BF62 2011-08-19 [expires: 2024-06-14]
uid                  nginx signing key <[email protected]>

pub   4096R/68576280 2014-06-13
uid                  NodeSource <[email protected]>
sub   4096R/AA01DA2C 2014-06-13

/etc/apt/trusted.gpg.d/apt.postgresql.org.gpg
---------------------------------------------
pub   4096R/ACCC4CF8 2011-10-13
uid                  PostgreSQL Debian Repository

Removing /var/cache/apt doesn't help.

However, I tried to spin up a fresh trusty VM and configure both older (5.x) and new (6.x) datadog repositories, and everything went fine.

I'm investigating what's happening on other hosts - will post an update here.

defanator picture defanator on 10 Dec 2019

Ok, perhaps this is the reason:

# find /var/lib/apt/lists/ -name "*datadog*" -exec ls -l {} \;
-rw-r--r-- 1 root root 204435 Nov  6 16:26 /var/lib/apt/lists/partial/apt.datadoghq.com_dists_stable_main_binary-amd64_Packages.decomp.FAILED
-rw-r--r-- 1 root root 26620 Nov  6 16:26 /var/lib/apt/lists/partial/apt.datadoghq.com_dists_stable_main_binary-amd64_Packages
-rw-r--r-- 1 root root 5393 Aug 26 16:26 /var/lib/apt/lists/apt.datadoghq.com_dists_stable_InRelease
-rw-r--r-- 1 root root 203425 Aug 26 16:26 /var/lib/apt/lists/apt.datadoghq.com_dists_stable_main_binary-amd64_Packages
# diff -u /var/lib/apt/lists/partial/apt.datadoghq.com_dists_stable_main_binary-amd64_Packages.decomp.FAILED /var/lib/apt/lists/apt.datadoghq.com_dists_stable_main_binary-amd64_Packages
--- /var/lib/apt/lists/partial/apt.datadoghq.com_dists_stable_main_binary-amd64_Packages.decomp.FAILED  2019-11-06 16:26:23.000000000 +0000
+++ /var/lib/apt/lists/apt.datadoghq.com_dists_stable_main_binary-amd64_Packages    2019-08-26 16:26:09.000000000 +0000
@@ -5314,29 +5314,3 @@
  forwards metrics from your applications as well as system services.
  .
  See http://www.datadoghq.com/ for more information
-
-Package: datadog-agent
-Version: 1:5.32.5-1
-License: Simplified BSD License
-Vendor: Datadog <[email protected]>
-Architecture: amd64
-Maintainer: Datadog Packages <[email protected]>
-Installed-Size: 334879
-Conflicts: datadog-agent-base (<< 5.0.0)
-Replaces: datadog-agent-base (<< 5.0.0), datadog-agent-lib (<< 5.0.0)
-Section: utils
-Priority: extra
-Homepage: http://www.datadoghq.com
-Filename: pool/d/da/datadog-agent_5.32.5-1_amd64.deb
-Size: 104802018
-SHA1: 2910d51f4f540b4fe1d7b1e6a06fcf18a227af00
-SHA256: cef68c0386d465075e7f4a46f2ed80fc9209db95b9489d51dbb227ce027a60dd
-MD5sum: 9af1ea4964187e0da36522102540a85e
-Description: Datadog Monitoring Agent
- The Datadog Monitoring Agent is a lightweight process that monitors system
- processes and services, and sends information back to your Datadog account.
- .
- This package installs and runs the advanced Agent daemon, which queues and
- forwards metrics from your applications as well as system services.
- .
- See http://www.datadoghq.com/ for more information

After removing those cached lists everything is fine:

# find /var/lib/apt/lists/ -name "*datadog*" -delete
# apt-get update
Ign http://us-west-2.ec2.archive.ubuntu.com trusty InRelease
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates InRelease
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports InRelease
Hit http://us-west-2.ec2.archive.ubuntu.com trusty Release.gpg                 
Hit http://us-west-2.ec2.archive.ubuntu.com trusty Release                     
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/main Sources        
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/restricted Sources  
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/universe Sources    
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/multiverse Sources  
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/main amd64 Packages 
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/restricted amd64 Packages
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/universe amd64 Packages
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/multiverse amd64 Packages
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/main Translation-en 
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/multiverse Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/restricted Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/universe Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/main Sources 
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/restricted Sources
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/universe Sources
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/multiverse Sources
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/main amd64 Packages
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/restricted amd64 Packages
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/universe amd64 Packages
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/multiverse amd64 Packages
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/main Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/multiverse Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/restricted Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/universe Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/main Sources            
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/restricted Sources      
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/universe Sources        
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/multiverse Sources      
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/main amd64 Packages     
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/universe amd64 Packages     
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/multiverse amd64 Packages   
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/main Translation-en      
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/multiverse Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/restricted Translation-en
Hit http://us-west-2.ec2.archive.ubuntu.com trusty/universe Translation-en  
Ign http://us-west-2.ec2.archive.ubuntu.com trusty/main Translation-en_US   
Ign http://us-west-2.ec2.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us-west-2.ec2.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us-west-2.ec2.archive.ubuntu.com trusty/universe Translation-en_US
Ign https://apt.datadoghq.com stable InRelease                              
Get:1 https://apt.datadoghq.com stable/main Translation-en_US           
Hit http://security.ubuntu.com trusty-security InRelease                       
Ign https://apt.datadoghq.com stable/main Translation-en_US                    
Ign https://apt.datadoghq.com stable/main Translation-en
Hit http://security.ubuntu.com trusty-security/main Sources
Hit http://security.ubuntu.com trusty-security/universe Sources
Hit http://security.ubuntu.com trusty-security/main amd64 Packages
Hit http://security.ubuntu.com trusty-security/universe amd64 Packages
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Fetched 32.5 kB in 2s (13.8 kB/s)            
Reading package lists... Done

Not sure whether it was a temporary issue with the repo back in Nov 6, or some bug in apt.
Let me know if you would like to get those lists, otherwise please feel free to close the issue.

Sorry for the noise. :)

defanator picture defanator on 10 Dec 2019
👍3 ❤1

Thanks for all the additional details here @defanator!

Interestingly there was an InRelease file in your local apt cache (/var/lib/apt/lists/apt.datadoghq.com_dists_stable_InRelease), dated Aug 26. This matches a change we did on the APT repo on that day to start providing an InRelease file on the repo, which was subsequently reverted after we found out that older versions of Debian (7 and below) and Ubuntu (12.04 and below) disable support of the InRelease file.

For some reason even after the InRelease file was removed from the repo, it appears that apt continued to read the InRelease file it had in its cache and trusted its contents, even though it was outdated.

In any case, what you did to resolve the issue, i.e. clearing the local cache of datadog-related repository metadata file (find /var/lib/apt/lists/ -name "*datadog*" -delete) and then re-running apt-get update, is indeed a good solution 馃憤

I'll go ahead and close this issue then, but feel free to comment here if anything new comes up, thanks!

olivielpeau picture olivielpeau on 10 Dec 2019
🎉5 ❤3 👍2

I had the same issue while trying to upgrade my Linode server from Ubuntu 14.04 to 16. Luckily, find /var/lib/apt/lists/ -name "datadog" -delete) and then re-running apt-get update fixed the issue! Thanks.

kobleistvan picture kobleistvan on 3 Dec 2020

This fix above didn't fix my problem, however, my goal was to just get this thing to work. So I updated my vagrant box to "ubuntu/focal64" which is ubuntu 20.04, so now it works

isereb picture isereb on 9 Feb 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

bhavin192 picture bhavin192  路  5Comments
agosto-calvinbehling picture agosto-calvinbehling  路  3Comments
amlwwalker picture amlwwalker  路  5Comments
pvalsecc picture pvalsecc  路  5Comments
SleepyBrett picture SleepyBrett  路  4Comments