Stack: Verification error: <repo>/mirrors.json is expired

[jmitchell]

General summary/comments

The mirrors.json file expired about an hour ago, and stack is complaining about it.

Steps to reproduce

1. start with no ~/.stack directory
2. run stack setup (I'm using version 1.5.1)

Expected

Stack should have properly set up the local state necessary to create and build projects.

Actual

What actually happened.

$ stack setup
Selected mirror https://s3.amazonaws.com/hackage.fpcomplete.com/                                 
Downloading timestamp                                                                            
Downloading snapshot                                                                             
Downloading mirrors                                                                              
Verification error: <repo>/mirrors.json is expired                                               
Downloading root                                                                                 
Downloading timestamp                                                                            
Downloading snapshot                                                                             
Downloading mirrors                                                                              
Verification error: <repo>/mirrors.json is expired                                               
Downloading root                                                                                 
Downloading timestamp                                                                            
Downloading snapshot                                                                             
Downloading mirrors                                                                              
Verification error: <repo>/mirrors.json is expired                                               
Downloading root                                                                                 
Downloading timestamp                                                                            
Downloading snapshot                                                                             
Downloading mirrors                                                                              
Verification error: <repo>/mirrors.json is expired                                               
Downloading root                                                                                 
Downloading timestamp                                                                            
Downloading snapshot                                                                             
Downloading mirrors                                                                              
Verification error: <repo>/mirrors.json is expired                                               
Downloading root                                                                                 
Updating package index Hackage (mirrored at https://s3.amazonaws.com/hackage.fpcomplete.com/) ...VerificationErrorLoop [Right (VerificationErrorExpired (TargetPathRepo (Path "mirrors.json"))),Right (VerificationErrorExpired (TargetPathRepo (Path "mirrors.json"))),Right (VerificationErrorExpired (TargetPathRepo (Path "mirrors.json"))),Right (VerificationErrorExpired (TargetPathRepo (Path "mirrors.json"))),Right (VerificationErrorExpired (TargetPathRepo (Path "mirrors.json")))]

Stack version

$ stack --version
Version 1.5.1 x86_64
Compiled with:
- Cabal-1.24.2.0
- Glob-0.8.0
- HUnit-1.5.0.0
- MonadRandom-0.5.1
...

Method of installation

nix-env -i stack

[jmitchell]

The file has apparently been updated and is set to expire in a year. If this is an automated process, can whoever manages the mirror file ensure it's scheduled to update before it expires?

[snoyberg]

I can't explain the behavior here, I did nothing manually to change this. Perhaps this is just the behavior of the hackage-mirror-tool we're using (https://github.com/haskell-hvr/hackage-mirror-tool). I can't file an issue there, as I'm blocked by the author, but if you'd like to follow up there, I'd be interested in the result.

[TravisCardwell]

I just ran into this issue for the first time, installing Stack on a VPS that has not had Stack installed before.

Reading in #3738 about the possibility of clock skew causing the issue, I confirmed that my time is correct:

$ date
2019年  7月  1日 月曜日 10:04:43 JST

Checking mirrors.json, however, I see that it has an expiration of 2019-06-30T23:59:59Z, which is about an hour in the past.

$ curl -s https://s3.amazonaws.com/hackage.fpcomplete.com/mirrors.json | python3 -m json.tool
{
    "signatures": [
        {
            "keyid": "772e9f4c7db33d251d5c6e357199c819e569d130857dc225549b40845ff0890d",
            "method": "ed25519",
            "sig": "xe6JjEZlp8ueG7l0XahzG6itTVNzoYxmfF43HNcb5EdFXBIqC7iRXjcsMseIhANnhv6SgovFZx7M1RiHokRzAw=="
        }
    ],
    "signed": {
        "_type": "Mirrorlist",
        "expires": "2019-06-30T23:59:59Z",
        "mirrors": [
            {
                "urlbase": "http://hackage.fpcomplete.com/"
            },
            {
                "urlbase": "http://objects-us-east-1.dream.io/hackage-mirror/"
            }
        ],
        "version": 5
    }
}

At least in this case, it is an actual expiration.

On my desktop, which already has Stack installed, I do not receive an error, because the mirrors have already been configured. By copying the contents of ~/.stack/pantry/hackage from my working desktop to the VPS, I am able to (perhaps temporarily) get around the issue.

BTW, I tried changing my timezone to one where it is still 2019-06-30, and I confirmed that it (correctly) results in the same expiration error.

---

System information:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 9.9 (stretch)
Release:    9.9
Codename:   stretch
$ uname -a
Linux XXX 5.1.11-x86_64-XXX #1 SMP PREEMPT Mon Jun 17 21:18:26 UTC 2019 x86_64 GNU/Linux
$ stack --version
Version 2.1.1, Git revision f612ea85316bbc327a64e4ad8d9f0b150dc12d4b (7648 commits) x86_64 hpack-0.31.2

[prakhunov]

I'm currently in the middle of creating a Dockerfile for an isolated test run, and I too have just now run into the same issue.

Hopefully it will be fixed by tomorrow.

[ProofOfKeags]

Happening here too.

[snoyberg]

This is a problem on Hackage itself; the mirrors.json file there is marked as expired:

http://hackage.haskell.org/mirrors.json

We can't fix this in the FP Complete mirror, since we can't take one of their files and resign it with their keys. We can only mirror the original files, which are now expired.

So as I see it, we have two options:

1. Accept the fact that every once in a while Hackage is serving expired files and wait until it gets fixed upstream, letting all Stack users get blocked in the process
2. Change our usage of hackage-security to ignore the expiration system

Personally, I never bought the motivation around expiration files in the first place. And especially in Stack's usage of hackage-security, where we are typically setting specific snapshots which have pinned dependency versions and cryptographic hashes, I don't see an advantage.

Has anyone on this issue filed an upstream issue with the Hackage team?

[snoyberg]

As a short term workaround, you can add the following to your ~/.stack/config.yaml file to disable the expiry check:

package-indices:
- download-prefix: https://hackage.haskell.org/
  hackage-security:
    keyids:
    - 0a5c7ea47cd1b15f01f5f51a33adda7e655bc0f0b0615baa8e271f4c3351e21d
    - 1ea9ba32c526d1cc91ab5e5bd364ec5e9e8cb67179a471872f6e26f0ae773d42
    - 280b10153a522681163658cb49f632cde3f38d768b736ddbc901d99a1a772833
    - 2a96b1889dc221c17296fcc2bb34b908ca9734376f0f361660200935916ef201
    - 2c6c3627bd6c982990239487f1abd02e08a02e6cf16edb105a8012d444d870c3
    - 51f0161b906011b52c6613376b1ae937670da69322113a246a09f807c62f6921
    - 772e9f4c7db33d251d5c6e357199c819e569d130857dc225549b40845ff0890d
    - aa315286e6ad281ad61182235533c41e806e5a787e0b6d1e7eef3f09d137d2e9
    - fe331502606802feac15e514d9b9ea83fee8b6ffef71335479a2e68d84adc6b0
    key-threshold: 3 # number of keys required

    # ignore expiration date, see https://github.com/commercialhaskell/stack/pull/4614
    ignore-expiry: true

[brettgilio]

As a short term workaround, you can add the following to your ~/.stack/config.yaml file to disable the expiry check:

package-indices:
- download-prefix: https://hackage.haskell.org/
  hackage-security:
    keyids:
    - 0a5c7ea47cd1b15f01f5f51a33adda7e655bc0f0b0615baa8e271f4c3351e21d
    - 1ea9ba32c526d1cc91ab5e5bd364ec5e9e8cb67179a471872f6e26f0ae773d42
    - 280b10153a522681163658cb49f632cde3f38d768b736ddbc901d99a1a772833
    - 2a96b1889dc221c17296fcc2bb34b908ca9734376f0f361660200935916ef201
    - 2c6c3627bd6c982990239487f1abd02e08a02e6cf16edb105a8012d444d870c3
    - 51f0161b906011b52c6613376b1ae937670da69322113a246a09f807c62f6921
    - 772e9f4c7db33d251d5c6e357199c819e569d130857dc225549b40845ff0890d
    - aa315286e6ad281ad61182235533c41e806e5a787e0b6d1e7eef3f09d137d2e9
    - fe331502606802feac15e514d9b9ea83fee8b6ffef71335479a2e68d84adc6b0
    key-threshold: 3 # number of keys required

    # ignore expiration date, see https://github.com/commercialhaskell/stack/pull/4614
    ignore-expiry: true

Thanks!

[snoyberg]

I've opened up a Hackage issue for this:

https://github.com/haskell/hackage-server/issues/834

I've also opened up a discussion issue for deciding how to change Stack's behavior, if at all: #4928

[ChristophP]

Thank you

[gbaz]

Should be fixed now, if someone can verify.

[TravisCardwell]

A test confirms that it is indeed fixed now.

The new expiration date is six months in the future:

$ curl -s https://s3.amazonaws.com/hackage.fpcomplete.com/mirrors.json | jq '.signed.expires'
"2020-01-01T23:59:59Z"