Nix: etag assertions

[dhess]

Describe the bug

We've recently started hitting this assertion, fairly regularly:

Assertion failed: (request.expectedETag == res.etag), function downloadFile, file src/libfetchers/tarball.cc, line 67.

This is happening both on our Hydra and on my Mac, so the issue is independent of things like IP address or platform. It also occurs on multiple Nix projects, and on different URLs.

Steps To Reproduce

Reproducing this has been difficult, as it's intermittent and we only recently added -vv to the hydra-eval-jobs script to try to get more information. (It would be helpful if the assertion message included the URL that caused the problem!) Also, usually, you just need to retry the nix-build command a few times, or restart hydra-evaluator, and it will eventually work. But this morning, I finally hit a reproducible case that hits the assertion every time. Here I'm trying to enter nix-shell on a project. (Unfortunately, the source code for this project is not public.)

Running with nix-shell -vv:

...
evaluating file '/nix/store/8xv13mhf8vzdrhcylmnslwj5l7p83pzi-source/default.nix'
evaluating file '/nix/store/wydbs7nwx8bq8ampdxpwbd36raydym54-source/default.nix'
evaluating file '/nix/store/8xv13mhf8vzdrhcylmnslwj5l7p83pzi-source/flake.nix'
downloading 'https://api.github.com/repos/numtide/flake-utils/tarball/5021eac20303a61fafe17224c087f5519baed54d'...
Assertion failed: (request.expectedETag == res.etag), function downloadFile, file src/libfetchers/tarball.cc, line 67.

Here are the relevant contents of my ~/.cache/nix/fetcher-cache-v1.sqlite file:

 sqlite3 fetcher-cache-v1.sqlite
SQLite version 3.32.3 2020-06-18 14:16:19
Enter ".help" for usage hints.
sqlite> .dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE Cache (
    input     text not null,
    info      text not null,
    path      text not null,
    immutable integer not null,
    timestamp integer not null,
    primary key (input)
);
...
INSERT INTO Cache VALUES('{"name":"source","type":"file","url":"https://api.github.com/repos/numtide/flake-utils/tarball/5021eac20303a61fafe17224c087f5519baed54d"}','{"etag":"\"17552ff816e897217c41da34c98d7fc7fd2682fde9c63a931172d14a958303f9\"","url":"https://codeload.github.com/numtide/flake-utils/legacy.tar.gz/5021eac20303a61fafe17224c087f5519baed54d"}','/nix/store/nmczpvs1lc0p3x1bpxr454g64y3hvy28-source',0,1611270871);
INSERT INTO Cache VALUES('{"name":"source","type":"file","url":"https://codeload.github.com/numtide/flake-utils/legacy.tar.gz/5021eac20303a61fafe17224c087f5519baed54d"}','{"etag":"\"17552ff816e897217c41da34c98d7fc7fd2682fde9c63a931172d14a958303f9\"","url":"https://codeload.github.com/numtide/flake-utils/legacy.tar.gz/5021eac20303a61fafe17224c087f5519baed54d"}','/nix/store/nmczpvs1lc0p3x1bpxr454g64y3hvy28-source',0,1611270871);
INSERT INTO Cache VALUES('{"name":"source","type":"tarball","url":"https://api.github.com/repos/numtide/flake-utils/tarball/5021eac20303a61fafe17224c087f5519baed54d"}','{"etag":"\"17552ff816e897217c41da34c98d7fc7fd2682fde9c63a931172d14a958303f9\"","lastModified":1605370193}','/nix/store/if5crzwh0ziyarn0kjajd8br35i8lqa9-source',0,1611270871);

And here is the output of wget -S when I fetch this tarball:

 wget -S https://api.github.com/repos/numtide/flake-utils/tarball/5021eac20303a61fafe17224c087f5519baed54d
--2021-01-22 11:01:01--  https://api.github.com/repos/numtide/flake-utils/tarball/5021eac20303a61fafe17224c087f5519baed54d
Resolving api.github.com (api.github.com)... 140.82.121.6
Connecting to api.github.com (api.github.com)|140.82.121.6|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 302 Found
  Date: Fri, 22 Jan 2021 11:00:45 GMT
  Content-Type: text/html;charset=utf-8
  Content-Length: 0
  Server: GitHub.com
  Status: 302 Found
  X-RateLimit-Limit: 60
  X-RateLimit-Remaining: 60
  X-RateLimit-Reset: 1611316844
  x-ratelimit-used: 0
  Cache-Control: public, must-revalidate, max-age=0
  expires: Fri, 22 Jan 2021 11:00:44 GMT
  Location: https://codeload.github.com/numtide/flake-utils/legacy.tar.gz/5021eac20303a61fafe17224c087f5519baed54d
  Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
  Access-Control-Allow-Origin: *
  Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
  X-Frame-Options: deny
  X-Content-Type-Options: nosniff
  X-XSS-Protection: 1; mode=block
  Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
  Content-Security-Policy: default-src 'none'
  Vary: Accept-Encoding, Accept, X-Requested-With, Accept-Encoding
  X-GitHub-Request-Id: CA68:5F1A:65BAF0:710254:600AB06D
Location: https://codeload.github.com/numtide/flake-utils/legacy.tar.gz/5021eac20303a61fafe17224c087f5519baed54d [following]
--2021-01-22 11:01:01--  https://codeload.github.com/numtide/flake-utils/legacy.tar.gz/5021eac20303a61fafe17224c087f5519baed54d
Resolving codeload.github.com (codeload.github.com)... 140.82.121.9
Connecting to codeload.github.com (codeload.github.com)|140.82.121.9|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Fri, 22 Jan 2021 11:01:02 GMT
  Content-Type: application/x-gzip
  Content-Length: 5082
  Access-Control-Allow-Origin: https://render.githubusercontent.com
  content-disposition: attachment; filename=numtide-flake-utils-5021eac.tar.gz
  Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; sandbox
  ETag: "17552ff816e897217c41da34c98d7fc7fd2682fde9c63a931172d14a958303f9"
  Strict-Transport-Security: max-age=31536000
  Vary: Authorization,Accept-Encoding
  X-Content-Type-Options: nosniff
  X-Frame-Options: deny
  X-XSS-Protection: 1; mode=block
  Vary: Accept-Encoding
  X-GitHub-Request-Id: CA69:112C7:50C43:6F68F:600AB06E
Length: 5082 (5.0K) [application/x-gzip]
Saving to: ‘5021eac20303a61fafe17224c087f5519baed54d’

5021eac20303a61fafe17224c087f5519baed54d         100%[=========================================================================================================>]   4.96K  --.-KB/s    in 0s      

2021-01-22 11:01:02 (9.75 MB/s) - ‘5021eac20303a61fafe17224c087f5519baed54d’ saved [5082/5082]

And the corresponding curl output, if that's more helpful:

 curl -v -L -O https://api.github.com/repos/numtide/flake-utils/tarball/5021eac20303a61fafe17224c087f5519baed54d
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 140.82.121.5...
* TCP_NODELAY set
* Connected to api.github.com (140.82.121.5) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [228 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [102 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2970 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.github.com
*  start date: Jun 22 00:00:00 2020 GMT
*  expire date: Aug 17 12:00:00 2022 GMT
*  subjectAltName: host "api.github.com" matched cert's "*.github.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fdee000d600)
> GET /repos/numtide/flake-utils/tarball/5021eac20303a61fafe17224c087f5519baed54d HTTP/2
> Host: api.github.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 302 
< date: Fri, 22 Jan 2021 11:25:54 GMT
< content-type: text/html;charset=utf-8
< content-length: 0
< server: GitHub.com
< status: 302 Found
< x-ratelimit-limit: 60
< x-ratelimit-remaining: 60
< x-ratelimit-reset: 1611318354
< x-ratelimit-used: 0
< cache-control: public, must-revalidate, max-age=0
< expires: Fri, 22 Jan 2021 11:25:54 GMT
< location: https://codeload.github.com/numtide/flake-utils/legacy.tar.gz/5021eac20303a61fafe17224c087f5519baed54d
< access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
< access-control-allow-origin: *
< strict-transport-security: max-age=31536000; includeSubdomains; preload
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
< content-security-policy: default-src 'none'
< vary: Accept-Encoding, Accept, X-Requested-With, Accept-Encoding
< x-github-request-id: CBBD:6B85:17FEA02:1A819B8:600AB695
< 
{ [0 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host api.github.com left intact
* Issue another request to this URL: 'https://codeload.github.com/numtide/flake-utils/legacy.tar.gz/5021eac20303a61fafe17224c087f5519baed54d'
*   Trying 140.82.121.9...
* TCP_NODELAY set
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0* Connected to codeload.github.com (140.82.121.9) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [233 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [108 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2970 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.github.com
*  start date: Jun 22 00:00:00 2020 GMT
*  expire date: Aug 17 12:00:00 2022 GMT
*  subjectAltName: host "codeload.github.com" matched cert's "*.github.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
> GET /numtide/flake-utils/legacy.tar.gz/5021eac20303a61fafe17224c087f5519baed54d HTTP/1.1
> Host: codeload.github.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Fri, 22 Jan 2021 11:27:19 GMT
< Content-Type: application/x-gzip
< Content-Length: 5082
< Access-Control-Allow-Origin: https://render.githubusercontent.com
< content-disposition: attachment; filename=numtide-flake-utils-5021eac.tar.gz
< Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; sandbox
< ETag: "17552ff816e897217c41da34c98d7fc7fd2682fde9c63a931172d14a958303f9"
< Strict-Transport-Security: max-age=31536000
< Vary: Authorization,Accept-Encoding
< X-Content-Type-Options: nosniff
< X-Frame-Options: deny
< X-XSS-Protection: 1; mode=block
< Vary: Accept-Encoding
< X-GitHub-Request-Id: CBBE:10C71:15116E:1B591E:600AB697
< 
{ [708 bytes data]
100  5082  100  5082    0     0   2344      0  0:00:02  0:00:02 --:--:--  8807
* Connection #1 to host codeload.github.com left intact
* Closing connection 0
* Closing connection 1

nix-env --version output

On my Mac:

 nix-env --version
nix-env (Nix) 2.4pre20201205_a5d85d0

On our Hydra:

$ nix-env --version
warning: unknown setting 'gc-check-reachability'
nix-env (Nix) 2.4pre20201205_a5d85d0

[dhess]

Here's another reproducible case (they seem to be becoming more frequent):

...
building the system configuration...
downloading 'https://github.com/LnL7/nix-darwin/archive/6bd5208e7ffd141f2b2c4bd639e11ca5b96aec8c.tar.gz'...
Assertion failed: (request.expectedETag == res.etag), function downloadFile, file src/libfetchers/tarball.cc, line 67.

INSERT INTO Cache VALUES('{"name":"source","type":"file","url":"https://github.com/LnL7/nix-darwin/archive/6bd5208e7ffd141f2b2c4bd639e11ca5b96aec8c.tar.gz"}','{"etag":"\"96350991c749f73ca1703498711e1584261a2f86b8a5c24fa710bbd108b5bef1\"","url":"https://codeload.github.com/LnL7/nix-darwin/tar.gz/6bd5208e7ffd141f2b2c4bd639e11ca5b96aec8c"}','/nix/store/fm25dfbz0lq61284v9z2mxg3qd5y40lz-source',0,1611101448);
INSERT INTO Cache VALUES('{"name":"source","type":"file","url":"https://codeload.github.com/LnL7/nix-darwin/tar.gz/6bd5208e7ffd141f2b2c4bd639e11ca5b96aec8c"}','{"etag":"\"96350991c749f73ca1703498711e1584261a2f86b8a5c24fa710bbd108b5bef1\"","url":"https://codeload.github.com/LnL7/nix-darwin/tar.gz/6bd5208e7ffd141f2b2c4bd639e11ca5b96aec8c"}','/nix/store/fm25dfbz0lq61284v9z2mxg3qd5y40lz-source',0,1611101448);
INSERT INTO Cache VALUES('{"name":"source","type":"tarball","url":"https://github.com/LnL7/nix-darwin/archive/6bd5208e7ffd141f2b2c4bd639e11ca5b96aec8c.tar.gz"}','{"etag":"\"96350991c749f73ca1703498711e1584261a2f86b8a5c24fa710bbd108b5bef1\"","lastModified":1605453767}','/nix/store/yb0p9ywn77jxx3y7k1dyfbd1hfrn6ac2-source',0,1611101448);

 curl -v -L -O https://github.com/LnL7/nix-darwin/archive/6bd5208e7ffd141f2b2c4bd639e11ca5b96aec8c.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 140.82.121.4...
* TCP_NODELAY set
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0* Connected to github.com (140.82.121.4) port 443 (#0)
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [224 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [102 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2971 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=github.com
*  start date: May  5 00:00:00 2020 GMT
*  expire date: May 10 12:00:00 2022 GMT
*  subjectAltName: host "github.com" matched cert's "github.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f9e1580c600)
> GET /LnL7/nix-darwin/archive/6bd5208e7ffd141f2b2c4bd639e11ca5b96aec8c.tar.gz HTTP/2
> Host: github.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0< HTTP/2 302 
< date: Fri, 22 Jan 2021 12:43:11 GMT
< content-type: text/html; charset=utf-8
< server: GitHub.com
< status: 302 Found
< vary: X-PJAX, Accept-Encoding, Accept, X-Requested-With, Accept-Encoding
< location: https://codeload.github.com/LnL7/nix-darwin/tar.gz/6bd5208e7ffd141f2b2c4bd639e11ca5b96aec8c
< cache-control: max-age=0, private
< strict-transport-security: max-age=31536000; includeSubdomains; preload
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< referrer-policy: no-referrer-when-downgrade
< expect-ct: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
< content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events wss://alive.github.com online.visualstudio.com/api/v1/locations; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; worker-src github.com/socket-worker-5029ae85.js gist.github.com/socket-worker-5029ae85.js
< x-github-request-id: CDD9:BB23:1CA490C:23642B0:600AC85E
< 
* Ignoring the response-body
{ [157 bytes data]
100   157    0   157    0     0     58      0 --:--:--  0:00:02 --:--:--    58
* Connection #0 to host github.com left intact
* Issue another request to this URL: 'https://codeload.github.com/LnL7/nix-darwin/tar.gz/6bd5208e7ffd141f2b2c4bd639e11ca5b96aec8c'
*   Trying 140.82.121.10...
* TCP_NODELAY set
* Connected to codeload.github.com (140.82.121.10) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [233 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [108 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2970 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.github.com
*  start date: Jun 22 00:00:00 2020 GMT
*  expire date: Aug 17 12:00:00 2022 GMT
*  subjectAltName: host "codeload.github.com" matched cert's "*.github.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
> GET /LnL7/nix-darwin/tar.gz/6bd5208e7ffd141f2b2c4bd639e11ca5b96aec8c HTTP/1.1
> Host: codeload.github.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Fri, 22 Jan 2021 12:43:12 GMT
< Content-Type: application/x-gzip
< Transfer-Encoding: chunked
< Access-Control-Allow-Origin: https://render.githubusercontent.com
< content-disposition: attachment; filename=nix-darwin-6bd5208e7ffd141f2b2c4bd639e11ca5b96aec8c.tar.gz
< Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; sandbox
< ETag: "96350991c749f73ca1703498711e1584261a2f86b8a5c24fa710bbd108b5bef1"
< Strict-Transport-Security: max-age=31536000
< Vary: Authorization,Accept-Encoding
< X-Content-Type-Options: nosniff
< X-Frame-Options: deny
< X-XSS-Protection: 1; mode=block
< Vary: Accept-Encoding
< X-GitHub-Request-Id: CDDA:BA9A:15493F:1BC667:600AC85F
< 
{ [679 bytes data]
100 97592    0 97592    0     0  29404      0 --:--:--  0:00:03 --:--:-- 29404
* Connection #1 to host codeload.github.com left intact
* Closing connection 0
* Closing connection 1

I moved the old fetcher-cache-v1.sqlite file out of the way and retried this build. It succeed. Here's are the relevant lines from the newly-generated fetcher-cache-v1.sqlite:

INSERT INTO Cache VALUES('{"name":"source","type":"file","url":"https://github.com/LnL7/nix-darwin/archive/6bd5208e7ffd141f2b2c4bd639e11ca5b96aec8c.tar.gz"}','{"etag":"\"96350991c749f73ca1703498711e1584261a2f86b8a5c24fa710bbd108b5bef1\"","url":"https://codeload.github.com/LnL7/nix-darwin/tar.gz/6bd5208e7ffd141f2b2c4bd639e11ca5b96aec8c"}','/nix/store/fm25dfbz0lq61284v9z2mxg3qd5y40lz-source',0,1611319564);
INSERT INTO Cache VALUES('{"name":"source","type":"file","url":"https://codeload.github.com/LnL7/nix-darwin/tar.gz/6bd5208e7ffd141f2b2c4bd639e11ca5b96aec8c"}','{"etag":"\"96350991c749f73ca1703498711e1584261a2f86b8a5c24fa710bbd108b5bef1\"","url":"https://codeload.github.com/LnL7/nix-darwin/tar.gz/6bd5208e7ffd141f2b2c4bd639e11ca5b96aec8c"}','/nix/store/fm25dfbz0lq61284v9z2mxg3qd5y40lz-source',0,1611319564);
INSERT INTO Cache VALUES('{"name":"source","type":"tarball","url":"https://github.com/LnL7/nix-darwin/archive/6bd5208e7ffd141f2b2c4bd639e11ca5b96aec8c.tar.gz"}','{"etag":"\"96350991c749f73ca1703498711e1584261a2f86b8a5c24fa710bbd108b5bef1\"","lastModified":1605453767}','/nix/store/yb0p9ywn77jxx3y7k1dyfbd1hfrn6ac2-source',0,1611319564);

The database values are the same except for the last field, which I assume is the time at which they were inserted.

[zimbatm]

I started seeing that as well on my machine.

Deleting ~/.cache/nix allowed me to work around the issue.

[dhess]

@zimbatm Yes, that works for awhile. On our Hydra, it doesn't work for very long!

[matthewbauer]

It looks like GitHub is now correctly responding to If-Not-Modified, but no longer includes ETag. Try:

curl -I https://github.com/edolstra/flake-compat/archive/master.tar.gz -H "If-None-Match: \"fb8ab964c56e8790f657e22def167ecd511a2ccfdcd5a928daea4decd33fb434\""

https://github.com/NixOS/nix/pull/4470 should fix it.

BTW: You only have to delete ~/.cache/nix/fetcher-cache-v1.sqlite & ~/.cache/nix/fetcher-cache-v1.sqlite-journal.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-shell-assertion-request-expectedetag-res-etag-failed/11119/7

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/cant-set-nixflakes-as-nix-package/11151/2