Dvc: Upgrade DVC on Ubuntu 18.04 Deb Repository Install

[melkiga]

Bug Report (Short Story)

I've installed DVC on my work server using the deb package from here, but want to be able to upgrade when a new version is released. I followed the instructions here in the repo section, and when I run sudo apt update I receive the following.

W: Failed to fetch https://dvc.org/deb/dists/stable/InRelease  Invalid response from proxy: HTTP/1.1 403 Forbidden  Server: squid/3.5.27  Mime-Version: 1.0  Date: Sat, 11 Jul 2020 19:55:43 GMT  Content-Type: text/html;charset=utf-8  Content-Length: 3506  X-Squid-Error: ERR_ACCESS_DENIED 0  Vary: Accept-Language  Content-Language: en  X-Cache: MISS from squid-deb-proxy  X-Cache-Lookup: NONE from squid-deb-proxy:8000  Via: 1.1 squid-deb-proxy (squid/3.5.27)  Connection: keep-alive     [IP: 10.224.0.67 8000]
W: Some index files failed to download. They have been ignored, or old ones used instead.

Behavior

When I run sudo apt-get install --only-upgrade dvc, I receive a response that I already have the latest version of DVC, when in fact a new version has been released (1.1.8):

$ sudo apt-get install --only-upgrade dvc
Reading package lists... Done
Building dependency tree       
Reading state information... Done
dvc is already the newest version (1.1.7).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

I then tried following your instructions here for how to install from repo, but when I run sudo apt update I receive the following error:

$ sudo apt update
.....
Reading package lists... Done                     
W: Failed to fetch https://dvc.org/deb/dists/stable/InRelease  Invalid response from proxy: HTTP/1.1 403 Forbidden  Server: squid/3.5.27  Mime-Version: 1.0  Date: Sat, 11 Jul 2020 19:55:43 GMT  Content-Type: text/html;charset=utf-8  Content-Length: 3506  X-Squid-Error: ERR_ACCESS_DENIED 0  Vary: Accept-Language  Content-Language: en  X-Cache: MISS from squid-deb-proxy  X-Cache-Lookup: NONE from squid-deb-proxy:8000  Via: 1.1 squid-deb-proxy (squid/3.5.27)  ...
W: Some index files failed to download. They have been ignored, or old ones used instead.

Please provide information about your setup

Output of dvc version:

$ dvc version
1.1.7

OS cat /etc/os-release

$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

[efiop]

Hi @melkiga !

Thank you so much for the feedback! Looks like we had some issues with our repo permissions. We've adjusted that and it should be working fine now. Could you please give it a try and let us know how it goes?

[melkiga]

Thank you for the quick response!
Unfortunately I still get the following:

W: Failed to fetch https://dvc.org/deb/dists/stable/InRelease  Invalid response from proxy: HTTP/1.1 403 Forbidden  Server: squid/3.5.27  Mime-Version: 1.0  Date: Sat, 11 Jul 2020 21:00:35 GMT  Content-Type: text/html;charset=utf-8  Content-Length: 3506  X-Squid-Error: ERR_ACCESS_DENIED 0  Vary: Accept-Language  Content-Language: en  X-Cache: MISS from squid-deb-proxy  X-Cache-Lookup: NONE from squid-deb-proxy:8000  Via: 1.1 squid-deb-proxy (squid/3.5.27)  Connection: keep-alive

[efiop]

Oops, sorry, still not quite resolved. Looking into it...

[melkiga]

Not sure if this helps:
If I open this link https://dvc.org/deb/dists/stable/InRelease in my browser, I get redirected to this https://s3-us-east-2.amazonaws.com/dvc-s3-repo/deb/dists/stable/InRelease with this as the page contents:

<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Key>deb/dists/stable/InRelease</Key>
<RequestId>BAFC7EB1008F87B5</RequestId>
<HostId>rHkvp+fPWpByruL4IQHXdrROVDcAPT1tn2PpIlRTagOyjkXbk7Q248WE2ekcik+FCmMcm8zHlMY=</HostId>
</Error>

[efiop]

@melkiga Yeah, InRelease file doesn't exist (and it never did, not sure why apt is trying to access it at all), investigating...

[efiop]

Ok, looks like this might be the cause https://github.com/krobertson/deb-s3/issues/137 . Investigating why it is not generated already...

[efiop]

@melkiga Sorry for the delay. I've added a dummy signature to make it comply for now. Could you please give it another shot? It could complain about it not being able to verify gpt signature, but it should work fine for now.

[melkiga]

No issues about the delay, this isn't a blocker for me :-)
OK so I ran the following:

$ sudo wget \
        https://dvc.org/deb/dvc.list \
        -O /etc/apt/sources.list.d/dvc.list
$ sudo apt update

and got the same result:

W: Failed to fetch https://dvc.org/deb/dists/stable/InRelease  Invalid response from proxy: HTTP/1.1 403 Forbidden  Server: squid/3.5.27  Mime-Version: 1.0  Date: Sun, 12 Jul 2020 00:35:36 GMT  Content-Type: text/html;charset=utf-8  Content-Length: 3506  X-Squid-Error: ERR_ACCESS_DENIED 0  Vary: Accept-Language  Content-Language: en  X-Cache: MISS from squid-deb-proxy  X-Cache-Lookup: NONE from squid-deb-proxy:8000  Via: 1.1 squid-deb-proxy (squid/3.5.27)  Connection: keep-alive    
W: Some index files failed to download. They have been ignored, or old ones used instead.

Could this be an issue from my end?

[efiop]

@melkiga Hm, that's strange. That file is available right now. Maybe it didn't yet update in your region, let's wait a few minutes and then try once more. Thank you for the feedback! :pray:

[melkiga]

No issues! Thanks for the quick response. I'll wait about 20 more minutes and try again. Worst case we can leave it till tomorrow and I can try then and report back.

[efiop]

@melkiga Any luck? :slightly_smiling_face:

[melkiga]

Giving it a try now. Sorry for the late reply!

[melkiga]

Still no luck, here's what I've done:

Ran the following

$ sudo wget \
       https://dvc.org/deb/dvc.list \
       -O /etc/apt/sources.list.d/dvc.list
$ sudo apt update

Here is the response of sudo apt update:

$ sudo apt update
...
Err:3 https://dvc.org/deb stable InRelease                                                                                                                                                                    
  Invalid response from proxy: HTTP/1.1 403 Forbidden  Server: squid/3.5.27  Mime-Version: 1.0  Date: Fri, 17 Jul 2020 18:58:35 GMT  Content-Type: text/html;charset=utf-8  Content-Length: 3506  X-Squid-Error: ERR_ACCESS_DENIED 0  Vary: Accept-Language  Content-Language: en  X-Cache: MISS from squid-deb-proxy  X-Cache-Lookup: NONE from squid-deb-proxy:8000  Via: 1.1 squid-deb-proxy (squid/3.5.27)  Connection: keep-alive     [IP: 10.224.0.67 8000]
......
W: Failed to fetch https://dvc.org/deb/dists/stable/InRelease  Invalid response from proxy: HTTP/1.1 403 Forbidden  Server: squid/3.5.27  Mime-Version: 1.0  Date: Fri, 17 Jul 2020 18:58:35 GMT  Content-Type: text/html;charset=utf-8  Content-Length: 3506  X-Squid-Error: ERR_ACCESS_DENIED 0  Vary: Accept-Language  Content-Language: en  X-Cache: MISS from squid-deb-proxy  X-Cache-Lookup: NONE from squid-deb-proxy:8000  Via: 1.1 squid-deb-proxy (squid/3.5.27)  Connection: keep-alive     [IP: 10.224.0.67 8000]
W: Some index files failed to download. They have been ignored, or old ones used instead.

[efiop]

@melkiga Interesting :slightly_frowning_face: Are you on company's vpn/proxy or something?

[melkiga]

Yes, I'm on a company VPN. Do you think that's the cause? Is anyone else able to update replicate the issue?

🧐

[efiop]

@melkiga Might be, actually. Could you try this on your personal machine? I'm not able to reproduce myself nor on our test machines :(

[melkiga]

Going to try that now :-) I'll report back in a few.

[melkiga]

Here's what I got on my personal machine:

Reading state information... Done
All packages are up to date.
W: GPG error: https://s3-us-east-2.amazonaws.com/dvc-s3-repo/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8F02DCF1DB7C115F

Machine Info:

~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

[efiop]

@melkiga GPG warning is expected, we are temporarily using one-off keys to sign InRelease https://github.com/iterative/dvc-s3-repo/issues/3 . So that warning could be ignored for now. I suppose other than that warning, the command worked fine, right?

I'm no longer seeing Forbidden errors in your log, so your company's VPN is indeed blocking access to our repo :slightly_frowning_face:

[melkiga]

Whoops, I mistook the warning for an error. I just completed my install so yes, it seems that the company VPN is the culprit.
Thanks again for helping out with this!

[efiop]

@melkiga Thank you so much for the feedback! So looks like you'll need to ping your admins to whitelist that url or, you could consider installing using a different package. E.g. snap or pip. Would the alternative packages work for you?

[melkiga]

I'm currently just doing deb installs manually. I'll ping our admin to whitelist the url, just because of the way our pip/python stuff is set up on our servers.

Thanks a ton for the help!