Silverstripe-framework: Make behat failure screen shot publics in S3

[maxime-rainville]

Our Behat builds take screenshots when a scenario fails. The screen shots are then uploaded to an S3 bucket along with other _artifacts_. There's a URL listed on the behat logs however those offuscate the actual URL to fetch the artifacts.

This makes it impossible for people outside of SilverStripe to see the screen shots and makes it cumbersome for people inside SilverStripe to do so.

My suggestion would be to:

• make the Behat screen shot public,
• publicise the URL to see them in the Travis logs,
• and add a simple index file to make it simple to browse in S3.

[ScopeyNZ]

Can you provide an example of a travis build that has the obfuscated URL? I think that when it's set up correctly it _does_ have public URLs?

[maxime-rainville]

https://travis-ci.org/silverstripe/silverstripe-admin/jobs/466238930

That's the output you have to go through.

php ./vendor/silverstripe/framework/tests/behat/travis-upload-artifacts.php --if-env BEHAT_TEST,ARTIFACTS_BUCKET,ARTIFACTS_KEY,ARTIFACTS_SECRET --target-path $TRAVIS_REPO_SLUG/$TRAVIS_BUILD_ID/$TRAVIS_JOB_ID --artifacts-base-url https://s3.amazonaws.com/$ARTIFACTS_BUCKET/ --artifacts-path ./artifacts/
Installing artifacts script to ~/bin/artifacts
+ curl -sL https://raw.githubusercontent.com/travis-ci/artifacts/master/install | bash
artifacts version v0.7.9-3-geef78ca revision=eef78ca2da49a8783a32d4293c24b7025b52b097
Creating ./artifacts/index.html...
+ ~/bin/artifacts upload --permissions public-read --target-paths silverstripe/silverstripe-admin/466238922/466238930 ./artifacts/
time="2018-12-10T23:11:14Z" level="info" msg="uploading with settings" bucket="[secure]" cache_control="private" permissions="public-read" 
time="2018-12-10T23:11:14Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/composer-show.json (size: 16KB)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/composer-show.json" 
time="2018-12-10T23:11:14Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/composer-show.txt (size: 6.9KB)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/composer-show.txt" 
time="2018-12-10T23:11:14Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/composer.lock (size: 180KB)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/composer.lock" 
time="2018-12-10T23:11:14Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/chromedriver.log (size: 80B)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/chromedriver.log" 
time="2018-12-10T23:11:14Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/index.html (size: 1.6KB)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/index.html" 
time="2018-12-10T23:11:14Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/screenshots/duplicate-a-page.feature_17.png (size: 30KB)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/screenshots/duplicate-a-page.feature_17.png" 
time="2018-12-10T23:11:14Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/screenshots/insert-a-link.feature_33.png (size: 45KB)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/screenshots/insert-a-link.feature_33.png" 
time="2018-12-10T23:11:14Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/screenshots/publish-a-page.feature_67.png (size: 54KB)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/screenshots/publish-a-page.feature_67.png" 
time="2018-12-10T23:11:14Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/screenshots/publish-a-page.feature_83.png (size: 56KB)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/screenshots/publish-a-page.feature_83.png" 
time="2018-12-10T23:11:14Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/screenshots/search-for-a-page.feature_104.png (size: 41KB)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/screenshots/search-for-a-page.feature_104.png" 
time="2018-12-10T23:11:14Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/screenshots/search-for-a-page.feature_123.png (size: 45KB)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/screenshots/search-for-a-page.feature_123.png" 
time="2018-12-10T23:11:14Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/screenshots/search-for-a-page.feature_18.png (size: 37KB)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/screenshots/search-for-a-page.feature_18.png" 
time="2018-12-10T23:11:14Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/screenshots/search-for-a-page.feature_26.png (size: 43KB)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/screenshots/search-for-a-page.feature_26.png" 
time="2018-12-10T23:11:15Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/screenshots/search-for-a-page.feature_54.png (size: 38KB)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/screenshots/search-for-a-page.feature_54.png" 
time="2018-12-10T23:11:15Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/screenshots/search-for-a-page.feature_65.png (size: 38KB)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/screenshots/search-for-a-page.feature_65.png" 
time="2018-12-10T23:11:15Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/screenshots/search-for-a-page.feature_76.png (size: 41KB)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/screenshots/search-for-a-page.feature_76.png" 
time="2018-12-10T23:11:15Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/screenshots/search-for-a-page.feature_96.png (size: 41KB)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/screenshots/search-for-a-page.feature_96.png" 
time="2018-12-10T23:11:15Z" level="info" msg="uploading: /home/travis/build/silverstripe/silverstripe-admin/artifacts/serve.log (size: 152KB)" download_url="https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/serve.log" 
Uploaded artifacts to https://s3.amazonaws.com/[secure]/silverstripe/silverstripe-admin/466238922/466238930/index.html

[chillu]

Yeah the intent was always to have them public. If they aren't we should fix that.

[maxime-rainville]

They are but you need to know what to replace [secure] with.

[dnsl48]

The current index file implementation seemed to be usable to me. Do you suggest to improve it in a particular way?

[ScopeyNZ]

The bug here is that variable is set up in travis as protected when it doesn't need to be. In this case it's $ARTIFACTS_BUCKET. I've updated silverstripe/admin on travis so that's visible in build logs now.

[robbieaverill]

Sounds like this is a bug rather than an RFC =)

[maxime-rainville]

Yeah, I thought the URL was hidden on purpose.

[chillu]

This should work on pull requests as well (which will be hard because of Travis secrets like S3 upload tokens not being available on PR builds)

[ScopeyNZ]

Yeah we can't do that without giving any random person the ability to upload to our S3 buckets. Even if we obfuscated the credentials somehow that wouldn't stop a nefarious user from just PRing a script that dumps the credentials to stdout.