Go: support/historyarchive: Make sure all http requests bypass cache for files that do not have unique names

[nyetwurk]

What version are you using?

stellar-archivist master

What did you do?

Attempt to run archivist against a bucket that does not mark files whose name does not change with no-cache (e.g. stellar-history.json)

What did you expect to see?

Up to date stellar-history.json

What did you see instead?

Bucket returned cached stellar-history.json

As a workaround, I have a gswrapper for google buckets:

#!/bin/bash

if [ $# -ne 2 ]; then
    exit 1
fi

dest=$(basename $2)
if [ "$dest" == "stellar-history.json" ]; then
    header="Cache-Control:no-cache, max-age=0, no-transform"
    exec gsutil -h "$header" cp "$1" "$2"
else
    exec gsutil cp "$1" "$2"
fi

A possible solution is to append ?<datestamp> to the stellar-history.json URL when fetching it.

[bartekn]

Is it a public history archive? The problem with adding a random string is it doesn't solve the core issue which is a lack of Cache-Control header.

[nyetwurk]

This would be hitting the http endpoint of a google storage backed archive. Convincing the google bucket to send the right cache control headers natively requires something like the wrapper I included above.

[bartekn]

Wouldn't this solve the issue in Google Cloud Storage: https://cloud.google.com/storage/docs/metadata#cache-control ?

[nyetwurk]

Yes, the wrapper above solves the issue, but it is an extra step that a node runner would have to set up.

[nyetwurk]

Note that the cache-control header has to be set every time the stellar-horizon.json file is uploaded.

[bartekn]

I understand. What I'm trying to say is that adding unique string to the request doesn't solve the root issue. Even if we add this change to all of the SDKs and you request stellar-horizon.json in a browser it's still possible you'll get a cached version.

I think a solution could be to have a simple bash script in your Stellar-Core config that adds cache headers when uploading .well-known/stellar-history.json file.

[nyetwurk]

I think this should be documented somewhere then, along with providing an example wrapper script.

[bartekn]

Good point, created a new issue here: https://github.com/stellar/docs/issues/557

[nyetwurk]

Thanks. I believe you are correct about this not solving the root cause, nor the issue with other HTTP clients.