Dpl: Fine-grained Cache-Control/Expires headers for S3

[ghost]

Issue #167 Requests the original feature
PR #170 Added support in for fine-grained cache-control

  - provider: s3
    ...
    cache_control:
      - "max-age=300": "*.js"
      - "max-age=3000":
        - "*.css"
        - "*.png"
      - "no-cache"

The yml above should set a cache_control: max-age=[value] header for JS files to 300, CSS and PNG files to 3000, all other files should have a cache_control: no-cache header

The result is that all files have their cache_control header set to no-cache

Either the YML is incorrect, or not being passed down correctly to dpl/provider/s3

The linter does not validate this as expected:

$ travis lint
Warnings for .travis.yml:
[x] in deploy section: dropping cache_control section: unexpected sequence

Documentation also needs to be updated
https://docs.travis-ci.com/user/deployment/s3

[danielwhatmuff]

Any idea when this will be looked at?

[hkdobrev]

@BanzaiMan @Joshua-Anderson @rkh Hi guys, sorry to ping you like that, but since you have reviewed #170, could you please shed some light on what's the status of this? Is it deployed? Does it work and if yes, how? Many thanks!

[bulyshko]

It seems travis-yaml was changed since PR #170 was merged into master, because right now yaml parser doesn't expects "cache_control" value to be a "sequence". Will try to investigate how to fix it.

[BanzaiMan]

I apologize for this oversight while reviewing the previous PR. It is not clear from the history if it was tested, or ever worked.

The way the data structures in .travis.yml translates to the command line flags to dpl, which are then passed to the deployment provider code is not very well-documented, and for this, I am sorry. I'll take a look at this S3 feature, as well as better documenting the data flow in the near future.

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[kmjennison]

If this hasn't been resolved, could we reopen this?

[benedikt-budig]

@BanzaiMan @bulyshko any news on this? Would be great to have that feature working. Many thanks guys!

[AdamPD]

Such a necessary feature. Seems like the work is 95% done but just missing the final touch.

[bl4ck5un]

This is indeed a necessary feature.

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[tvald]

keepalive

[chriselsen]

I fixed this. Give it a try with the following .travis.yml config and please provide feedback

deploy:
  - provider: s3
    edge:
      source: chriselsen/dpl
      branch: patch-2
    cache_control: "\"public, max-age=31536000\": *.jpg, *.gif, *.png, *.webp, *.css, *.js, *.ico"
    expires: "\"2030-12-31 00:00:00 -0000\": *.jpg, *.gif, *.png, *.webp, *.css, *.js, *.ico"

I know that my config differs from the above example. But to get that example .travis.yml config working would require major work in travis-yml:
Currently travis-yml passes most of the lines from .travis.yml to 'dpl' as string-based key value pairs. It doesn't have the capability to pass an entire YAML substructure.
Even if we fix this the question would be how 'dpl' should consume it. Because the the .travis.yml setting for e.g. "cache_control: setting" is passed like this to 'dpl':
dpl --cache_control=setting
That YAML structure from above would need to be translated into some kind of a string.
The s3_spec provides the following as the spec:
["max-age=99999999", "no-cache" => ["foo.html", "bar.txt"], "max-age=9999" => "*.txt"]
That's neither YAML nor JSON, but a string representation of a ruby array of hashes.

And that's one more reason why the previous code would have never worked. Because we would have had to configure this in .travis.yml as
cache_control: "[\"max-age=99999999\", \"no-cache\" => [\"foo.html\", \"bar.txt\"], \"max-age=9999\" => \"*.txt\"]"
This would at least pass the value from travis-yml to dpl. But that passed value would now still be a string, not an actual array of hashes.
But the s3 provider was expecting this input to be an array of hashes.

With that here's what I did:

• Correct the s3_spec to represent the test with the actual string max-age=99999999, no-cache: foo.html, bar.txt, max-age=9999: *.txt"
• Correct the s3_provider to parse through the provided string and build an internal hash table. That hash table is then used with fnmatch.

For the above example the hash table would look like this:

[*] -> [max-age=99999999]     <--- That's basically the default value
[foo.html] -> [no-cache]
[bar.txt] -> [no-cache]
[*.txt] -> [max-age=9999]

Here is an example of a successful deployment in Travis-CI.

[svenfuchs]

I have ported the PR https://github.com/travis-ci/dpl/pull/958 to the new master (dpl v2) here with some minor adjustments, mostly making the format more tight: https://github.com/travis-ci/dpl/pull/1062

it would be amazing if some of you could try this out.

I am going to close this issue here. Let's continue the conversation on https://github.com/travis-ci/dpl/pull/1062