It appears the checksums in the InRelease file have not been updated after a release.
2020-06-24T11:29:07.1014141Z Step 37/72 : RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
2020-06-24T11:29:07.1428743Z ---> Running in 8f1f1f72af1b
2020-06-24T11:29:07.4318822Z [91m [0m[91m [0m[91m%[0m[91m [0m[91mTo[0m[91mt[0m[91mal % Received % Xferd Average[0m[91m Sp[0m[91meed Time Time [0m[91m Time Current
2020-06-24T11:29:07.4324906Z [0m[91m [0m[91m Dload Upload Total Spent Left Speed
2020-06-24T11:29:07.4326186Z
2020-06-24T11:29:07.5018269Z 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0[0m[91mWarning: apt-key output should not be parsed (stdout is not a terminal)
2020-06-24T11:29:07.5511675Z [0m[91m
2020-06-24T11:29:07.5512807Z 100 983 100 983 0 0 8191 0 --:--:-- --:--:-- --:--:-- 8191
2020-06-24T11:29:08.0488990Z [0mOK
2020-06-24T11:29:09.0280731Z Removing intermediate container 8f1f1f72af1b
2020-06-24T11:29:09.0281722Z ---> 4116ce353101
2020-06-24T11:29:09.0282130Z Step 38/72 : RUN curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list | tee /etc/apt/sources.list.d/msprod.list
2020-06-24T11:29:09.0788654Z ---> Running in 7172bb67770f
2020-06-24T11:29:09.3827705Z [91m % Total % Received % Xferd Average Speed Time Time Time Current
2020-06-24T11:29:09.3828204Z Dload Upload Total Spent Left Speed
2020-06-24T11:29:09.3828460Z
2020-06-24T11:29:09.4058013Z 0 0[0m[91m 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0[0m[91m
2020-06-24T11:29:09.4059060Z 100 79 100 79 0 0 3434 0 --:--:-- --:--:-- --:--:-- 3434
2020-06-24T11:29:09.4059838Z [0mdeb [arch=amd64] https://packages.microsoft.com/ubuntu/16.04/prod xenial main
2020-06-24T11:29:10.0336367Z Removing intermediate container 7172bb67770f
2020-06-24T11:29:10.0337490Z ---> b41c61050097
2020-06-24T11:29:10.0337947Z Step 39/72 : RUN apt-get update
2020-06-24T11:29:10.0794144Z ---> Running in 21de1eb50b1e
2020-06-24T11:29:10.4557979Z Hit:1 http://deb.debian.org/debian buster InRelease
2020-06-24T11:29:10.4562997Z Hit:2 http://deb.debian.org/debian buster-updates InRelease
2020-06-24T11:29:10.4563741Z Hit:3 http://security.debian.org/debian-security buster/updates InRelease
2020-06-24T11:29:10.4611596Z Get:4 https://packages.microsoft.com/ubuntu/16.04/prod xenial InRelease [4003 B]
2020-06-24T11:29:10.9540746Z Get:5 https://packages.microsoft.com/ubuntu/16.04/prod xenial/main amd64 Packages [142 kB]
2020-06-24T11:29:10.9541253Z Err:5 https://packages.microsoft.com/ubuntu/16.04/prod xenial/main amd64 Packages
2020-06-24T11:29:10.9541718Z File has unexpected size (142137 != 142079). Mirror sync in progress? [IP: 13.90.56.68 443]
2020-06-24T11:29:10.9545512Z Hashes of expected file:
2020-06-24T11:29:10.9546659Z - Filesize:142079 [weak]
2020-06-24T11:29:10.9568588Z - SHA512:1e2d73b7082e18c33f3b4a6a584b7929344cdc87b5c96d25e2dcb3e84d72733014364d098028d9dcaddb7fd194ec11f3a74df489a9f05f67e84638cd033525e1
2020-06-24T11:29:10.9569494Z - SHA256:32a93cdfa7390dbc737271d543f7ae198d267a09aba44ca3e27beca49cc8471c
2020-06-24T11:29:10.9570119Z - SHA1:3243a484d2fa337b05e6e4be925d4ed96ba2ca38 [weak]
2020-06-24T11:29:10.9571059Z - MD5Sum:70ea695335d795cdf852fcdac44cb18e [weak]
2020-06-24T11:29:10.9573326Z Release file created at: Fri, 19 Jun 2020 08:51:18 +0000
2020-06-24T11:29:10.9619688Z Fetched 4003 B in 1s (7692 B/s)
2020-06-24T11:29:11.6849439Z Reading package lists...
2020-06-24T11:29:11.7104230Z [91mE: Failed to fetch https://packages.microsoft.com/ubuntu/16.04/prod/dists/xenial/main/binary-amd64/Packages.bz2 File has unexpected size (142137 != 142079). Mirror sync in progress? [IP: 13.90.56.68 443]
2020-06-24T11:29:11.7104962Z Hashes of expected file:
2020-06-24T11:29:11.7105626Z - Filesize:142079 [weak]
2020-06-24T11:29:11.7106397Z - SHA512:1e2d73b7082e18c33f3b4a6a584b7929344cdc87b5c96d25e2dcb3e84d72733014364d098028d9dcaddb7fd194ec11f3a74df489a9f05f67e84638cd033525e1
2020-06-24T11:29:11.7107166Z - SHA256:32a93cdfa7390dbc737271d543f7ae198d267a09aba44ca3e27beca49cc8471c
2020-06-24T11:29:11.7107756Z - SHA1:3243a484d2fa337b05e6e4be925d4ed96ba2ca38 [weak]
2020-06-24T11:29:11.7108289Z - MD5Sum:70ea695335d795cdf852fcdac44cb18e [weak]
2020-06-24T11:29:11.7108667Z Release file created at: Fri, 19 Jun 2020 08:51:18 +0000
2020-06-24T11:29:11.7109071Z E: Some index files failed to download. They have been ignored, or old ones used instead.
2020-06-24T11:29:11.9092606Z [0mThe command '/bin/sh -c apt-get update' returned a non-zero code: 100
2020-06-24T11:29:11.9218316Z ##[error]The command '/bin/sh -c apt-get update' returned a non-zero code: 100
2020-06-24T11:29:11.9237722Z ##[error]The process '/usr/bin/docker' failed with exit code 100
The same for the next packages:
https://packages.microsoft.com/debian/9/prod/dists/stretch/main/binary-amd64/Packages.bz2
Same issue here with the same package. Please let us know if we can help giving more info.
Same applies also for:
https://packages.microsoft.com/debian/10/prod/dists/buster/main/binary-amd64/Packages.bz2
Get:5 https://packages.microsoft.com/debian/10/prod buster/main amd64 Packages [43.7 kB]
Err:5 https://packages.microsoft.com/debian/10/prod buster/main amd64 Packages
File has unexpected size (43969 != 43653). Mirror sync in progress?
Hashes of expected file:
- Filesize:43653 [weak]
- SHA512:770b9079119523e27b68efe2742e196564815069321388ad9b1ce270da25b9c7f432fb57b210a6af4788d4460c1f80877c2a34de3a89e3d1584d74ed16c5f7c4
- SHA256:cd6baf1fba149709cb0a9904cb455666f295e1ceb1cef6b92dc3ba138b90a8f3
- SHA1:3cb42e61744f3cdb436f96e65d9669aa90623c88 [weak]
- MD5Sum:4864e88c059a44fe020d60d1e2a80872 [weak]
Release file created at: Wed, 17 Jun 2020 00:12:12 +0000
I'm taking a look at this (filing a ticket with the team, etc.) and will report back here with more info shortly.
(Edit: Filed https://icm.ad.msft.net/imp/v3/incidents/details/193848733/home tracking ticket.)
@dagood re: #4167, it has been significantly longer than that amount of "Expected time". Also, based on that issue, it appears the maintainers are not addressing the issue given the timeline, (nor contact-able). :(
I have the same:
[12:51:31] [Step 3/5] Preparing to unpack .../apt-transport-https_1.8.2.1_all.deb ...
[12:51:31] [Step 3/5] Unpacking apt-transport-https (1.8.2.1) ...
[12:51:31] [Step 3/5] Setting up apt-transport-https (1.8.2.1) ...
[12:51:31] [Step 3/5] Hit:1 http://security.debian.org/debian-security buster/updates InRelease
[12:51:31] [Step 3/5] Hit:2 http://deb.debian.org/debian buster InRelease
[12:51:31] [Step 3/5] Hit:3 http://deb.debian.org/debian buster-updates InRelease
[12:51:31] [Step 3/5] Get:4 https://packages.microsoft.com/debian/9/prod stretch InRelease [4009 B]
[12:51:31] [Step 3/5] Get:5 https://packages.microsoft.com/debian/9/prod stretch/main amd64 Packages [93.2 kB]
[12:51:31] [Step 3/5] Err:5 https://packages.microsoft.com/debian/9/prod stretch/main amd64 Packages
[12:51:31] [Step 3/5] File has unexpected size (93446 != 93230). Mirror sync in progress? [IP: 13.66.3.153 443]
[12:51:31] [Step 3/5] Hashes of expected file:
[12:51:31] [Step 3/5] - Filesize:93230 [weak]
[12:51:31] [Step 3/5] - SHA512:6d9952cd289566e58aad057b0c24d1d450c2dc8adfd3871b06f843c6a8ea84853f6cda10e390ffddc9621299d2dcfe516e488a6fea215928d5446a52754dfa9a
[12:51:31] [Step 3/5] - SHA256:c5c7ad271c940d9533231e61043d923d93f5bde7ab9927d8da9be3964c440bd4
[12:51:31] [Step 3/5] - SHA1:46696f97ceccbacdcdd9dda6d68fdbc1f7b76cd0 [weak]
[12:51:31] [Step 3/5] - MD5Sum:49e53481a1cbdac343dd7d54cc3c2c52 [weak]
[12:51:31] [Step 3/5] Release file created at: Wed, 17 Jun 2020 00:12:39 +0000
[12:51:31] [Step 3/5] Fetched 4009 B in 1s (6488 B/s)
[12:51:32] [Step 3/5] Reading package lists...
[12:51:32] [Step 3/5] E: Failed to fetch https://packages.microsoft.com/debian/9/prod/dists/stretch/main/binary-amd64/Packages.bz2 File has unexpected size (93446 != 93230). Mirror sync in progress? [IP: 13.66.3.153 443]
[12:51:32] [Step 3/5] Hashes of expected file:
[12:51:32] [Step 3/5] - Filesize:93230 [weak]
[12:51:32] [Step 3/5] - SHA512:6d9952cd289566e58aad057b0c24d1d450c2dc8adfd3871b06f843c6a8ea84853f6cda10e390ffddc9621299d2dcfe516e488a6fea215928d5446a52754dfa9a
[12:51:32] [Step 3/5] - SHA256:c5c7ad271c940d9533231e61043d923d93f5bde7ab9927d8da9be3964c440bd4
[12:51:32] [Step 3/5] - SHA1:46696f97ceccbacdcdd9dda6d68fdbc1f7b76cd0 [weak]
[12:51:32] [Step 3/5] - MD5Sum:49e53481a1cbdac343dd7d54cc3c2c52 [weak]
[12:51:32] [Step 3/5] Release file created at: Wed, 17 Jun 2020 00:12:39 +0000
[12:51:32] [Step 3/5] E: Some index files failed to download. They have been ignored, or old ones used instead.
[12:51:32] [Step 3/5] The command '/bin/sh -c apt-get install -y apt-transport-https && apt-get update && apt-get install -y aspnetcore-runtime-3.1 dotnet-runtime-3.1 cron' returned a non-zero code: 100
[12:51:32] [Step 3/5]
[12:51:32] [Step 3/5] Process exited with code 100
[12:51:32] [Step 3/5] Process exited with code 100 (Step: Docker build (Docker))
[12:51:32] [Step 3/5] Step Docker build (Docker) failed
same problem!
```
root@9075ffe6ea60:/# apt-get update
Hit:1 http://archive.ubuntu.com/ubuntu eoan InRelease
Hit:2 http://archive.ubuntu.com/ubuntu eoan-updates InRelease
Hit:3 http://archive.ubuntu.com/ubuntu eoan-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu eoan-security InRelease
Get:5 https://packages.microsoft.com/ubuntu/19.10/prod eoan InRelease [10.5 kB]
Get:6 https://packages.microsoft.com/ubuntu/19.10/prod eoan/main armhf Packages [504 B]
Get:7 https://packages.microsoft.com/ubuntu/19.10/prod eoan/main arm64 Packages [762 B]
Get:8 https://packages.microsoft.com/ubuntu/19.10/prod eoan/main amd64 Packages [27.7 kB]
Err:8 https://packages.microsoft.com/ubuntu/19.10/prod eoan/main amd64 Packages
File has unexpected size (27870 != 27700). Mirror sync in progress? [IP: 104.214.230.139 443]
Hashes of expected file:
@dagood , I know this team is not responsible for the msodbcsql17 package, but it is also impacting that package. Like others have said, the packages.microsoft.com ubuntu repo is hosed for 16.04 across the board.
I've opened up a support ticket with azure as it is impacting our production workloads on databricks
re: #4167, it has been significantly longer than that amount of "Expected time".
Yes, that "expected time" is just an advisory that the service might get healthy again in some situations. This is clearly an exceptional case.
On my end I've filed https://icm.ad.msft.net/imp/v3/incidents/details/193848733/home with the team. (Internal link, posting here for our own easy access in future.)
Same issue here:
apt-get update
Hit:1 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:4 http://ppa.launchpad.net/ansible/ansible/ubuntu xenial InRelease
Get:5 https://packages.microsoft.com/ubuntu/16.04/prod xenial InRelease [4,003 B]
Hit:6 http://security.ubuntu.com/ubuntu xenial-security InRelease
Get:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial/main amd64 Packages [142 kB]
Ign:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial/main amd64 Packages
Get:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial/main amd64 Packages [158 kB]
Ign:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial/main amd64 Packages
Get:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial/main amd64 Packages [791 kB]
Err:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial/main amd64 Packages
Writing more data than expected (792466 > 791391)
Fetched 4,003 B in 0s (7,632 B/s)
Reading package lists... Done
E: Failed to fetch https://packages.microsoft.com/ubuntu/16.04/prod/dists/xenial/main/binary-amd64/Packages Writing more data than expected (792466 > 791391)
E: Some index files failed to download. They have been ignored, or old ones used instead.
Is this related to unable to install netcore 2.2 on ubuntu 16.04, when doing apt-get install dotnet-sdk-2.2:
E: Unable to locate package dotnet-sdk-2.2
E: Couldn't find any package by glob 'dotnet-sdk-2.2'
E: Couldn't find any package by regex 'dotnet-sdk-2.2'
Is this related to unable to install netcore 2.2 on ubuntu 16.04, when doing apt-get install dotnet-sdk-2.2:
Yes, I expect your apt update failed earlier, and this caused the package metadata about dotnet-sdk-2.2 to not be downloaded. Ubuntu 16.04 is one of the distros where this problem is occurring. (I feel obliged to point out that 2.2 is EOL: you should try to upgrade to 3.1, or use 2.1 if the downgrade doesn't break you and it's more feasible.)
So far these repos have been reported to be affected (all Microsoft packages unavailable), and I've repro'd a few of them to confirm:
The Linux package repository team is working on applying a fix, no ETA yet.
(I feel obliged to point out that 2.2 is EOL: you should try to upgrade to 3.1, or use 2.1 if the downgrade doesn't break you and it's more feasible.)
Upgrade to 3.1 is in full progress, but until it's done, 2.2 needs to work :-)
Upgrade to 3.1 is in full progress, but until it's done, 2.2 needs to work :-)
Cool. 馃檪 And yeah, it's definitely our intention to keep EOL versions available on download endpoints and fix up issues like this as soon as we can.
@dagood what about .net core 3.1? i can't install it. So it's not about 2.2
what about .net core 3.1? i can't install it. So it's not about 2.2
You're correct, this outage applies to the entire Microsoft Linux package repository (every package) for each affected distro. The fix that the Linux package repo team is applying should restore access to all packages on these feeds, once they do it.
The reason for the side conversation is that 2.2 is EOL, and I wanted to make it clear that we will fix up both the EOL and non-EOL packages.
Hi there. Same here:
*****@**********:/# apt-get update
Hit:1 http://security.debian.org stretch/updates InRelease
Ign:2 http://deb.debian.org/debian stretch InRelease
Hit:3 http://deb.debian.org/debian stretch-updates InRelease
Hit:4 http://deb.debian.org/debian stretch Release
Get:7 https://packages.microsoft.com/debian/9/prod stretch/main amd64 Packages [103 kB]
Err:7 https://packages.microsoft.com/debian/9/prod stretch/main amd64 Packages
Writing more data than expected (103561 > 103377)
Hashes of expected file:
- Filesize:103377 [weak]
- SHA512:fa3d4bc9b2634e2c25be0c0afa362c32933f1df199213972bf7b45676afd852da61f6efc8b1ca2dd8617ec9d672425d354e528d4baf599e7ae621649c1e0ce9e
- SHA256:9fe010504f92e67c9fb22d37e6af9e02f72fff3de54edc28f5e726ece0d2a7d6
- SHA1:40ebf0eb5d1fb7bd2916d2cc2b8b334309c6daf9 [weak]
- MD5Sum:ee0f5824b41c62a11fb2dda476171a7e [weak]
Release file created at: Wed, 17 Jun 2020 00:12:39 +0000
Reading package lists... Done
E: Failed to fetch https://packages.microsoft.com/debian/9/prod/dists/stretch/main/binary-amd64/Packages.gz Writing more data than expected (103561 > 103377)
Hashes of expected file:
- Filesize:103377 [weak]
- SHA512:fa3d4bc9b2634e2c25be0c0afa362c32933f1df199213972bf7b45676afd852da61f6efc8b1ca2dd8617ec9d672425d354e528d4baf599e7ae621649c1e0ce9e
- SHA256:9fe010504f92e67c9fb22d37e6af9e02f72fff3de54edc28f5e726ece0d2a7d6
- SHA1:40ebf0eb5d1fb7bd2916d2cc2b8b334309c6daf9 [weak]
- MD5Sum:ee0f5824b41c62a11fb2dda476171a7e [weak]
Release file created at: Wed, 17 Jun 2020 00:12:39 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
Same issue here for debian 10.
Same here for both Debian 10 and 9
it works now for ubuntu 18.04
Looks like back up for 16 as well. Closing.
Just tried debian 10 and still not working
NVM, working now
Fixed in debian 9 from now.
Fixed on Ubuntu 16.04
Thanks all for checking and posting status. The repo admins haven't gotten back to us since their last update, but @rbhanda applied a fix that worked for us in the past that seems to have cleared things up.
Fixed on Debian 10
The repo admins found some more repositories that needed fixups (Ubuntu 18.10, 17.10, 19.04), and they've resolved those and closed our ticket--all repositories should be working at this point.
@dagood thank you for facilitating with the internal team!
@dagood thnx man!
@dagood the https://packages.microsoft.com/ubuntu/18.04/prod/dists/bionic/main/binary-amd64/Packages.bz2 File has unexpected size happens again in ubuntu 18.04 when apt-get update:
E: Failed to fetch https://packages.microsoft.com/ubuntu/18.04/prod/dists/bionic/main/binary-amd64/Packages.bz2 File has unexpected size (141364 != 141252). Mirror sync in progress? [IP: 13.90.56.68 443]
Hashes of expected file:
- Filesize:141252 [weak]
- SHA512:dffbaf15f537ae643df110b279248993b2828e99352dff51a5df9648cf39531ffb7022ec2f0d9e553287c9889addcb5e734d9df95f749c456db04fd328a67ef4
- SHA256:6757f7f09fb2a0491657aac899faa29bd67fe7b9dfd849844f7fafca9afcc33e
- SHA1:37be76673db70e808db91294af54fddf476aae7f [weak]
- MD5Sum:ecc0d5d8d9373eee7aeceda58d7be8f9 [weak]
Release file created at: Tue, 13 Oct 2020 22:35:00 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
Thanks for reporting, I will follow up on this and keep you posted. Is this an issue with both .NET Core 2.1 and 3.1?
It looks like the new outage was also reported at https://github.com/dotnet/core/issues/5373. It affects the entire 18.04 feed, not specific .NET packages.
I'm going to close this issue as resolved: https://github.com/dotnet/core/issues/5373 tracks the new 18.04 outage, so check there for the latest info.
Most helpful comment
The same for the next packages:
https://packages.microsoft.com/debian/9/prod/dists/stretch/main/binary-amd64/Packages.bz2