Sentry-ruby: Heroku release version sometimes inaccurate

[nfm]

Related to #377.

It looks like ENV['HEROKU_SLUG_COMMIT'] is only available for git-push based deploys. We used to use git-push to deploy, and switched to Heroku's Pipelines feature several months ago. The HEROKU_SLUG_COMMIT environment variable remains present, but is set to a very old value - the last SHA that we deployed via git-push.

I couldn't find another environment variable that Heroku supplies that contains the SHA from a Pipeline deploy. However, heroku labs:enable runtime-dyno-metadata writes this metadata to /etc/heroku/dyno (see https://github.com/heroku/fix/blob/6c8ab7a/lib/heroku_dyno_metadata.rb).

There might be a better way to do this, I'm not sure - this was the only solution I could find!

Would you be open to a PR that read the metadata from this file first, then fell back to HEROKU_SLUG_COMMIT, and finally to git rev-parse --short HEAD?

[nateberkopec]

Weird, I'll take a look.

[nateberkopec]

Opened up a ticket with Heroku support, we'll see what's up.

[nfm]

Hopefully they can do something on their end, reading it from an environment variable is much nicer!

[nateberkopec]

Well I basically got a non-answer from Heroku. We could just eliminate the 'HEROKU_SLUG_COMMIT' check and just shell out to git immediately.

[nateberkopec]

@nfm To clarify - HEROKU_SLUG_COMMIT the environment variable is _incorrect_, but in the dyno metadata it _is_ correct?

[nfm]

@nateberkopec That's right, for us, HEROKU_SLUG_COMMIT is present and incorrect.

The dyno metadata is correct, but it's an opt-in feature. The file /etc/heroku/dyno doesn't exist unless you enable the runtime-dyno-metadata labs feature.

I just made a new Heroku app, and did a Pipeline deploy (rather than a git-push deploy), and can confirm that the HEROKU_SLUG_COMMIT environment variable isn't set at all. So I think it would only be present and incorrect if you were initially deploying via git-push, and then switched to Pipelines for deploying. If you always deploy with git-push, it should be present and correct. And if you never deploy with git-push, it seems to be absent.

Unfortunately, we can't shell out to git on Heroku, as it removes the .git directory from the compiled slug.

[atmos]

Well I basically got a non-answer from Heroku

I work at heroku and could poke around with some people possibly.

I think the culprit is that pipeline promotions take an existing slug and move it from one application to another. Depending on how you got it into the original application git-push, builds api, docker, github sync, etc, these are all gonna differ.

Nothing immediately comes to mind as a catch all for all of the entry points we have for getting code into a slug, and requiring people to enable features is kind of questionable too.

Maybe we could:

• check for existence of /etc/heroku/dyno
• use if present, ignore otherwise

Then you could at least document that teams who want revision tracking need to enable a feature, instead of trying to accommodate all of the variations that are possible.

[dcramer]

I wouldn't be opposed to checking for the file. It's a bit of a pita to implement in each SDK, but not out of the question.

[nateberkopec]

Just to update, I've gotten through to a higher level of Heroku support and they seem to think this may be a bug.

[gudmundur]

Hi @nfm.

I'm on the team behind Heroku Pipelines and I've been trying to figure out the cause for this issue. I've attempted to reproduce this myself, with no luck. The flag you mention, runtime-dyno-metadata, does inject a env var called HEROKU_SLUG_COMMIT upon creating a release. As far as I can see, it should be set to commit of the we're promoting. Can you tell me how the promotable release is being made? Git push, GitHub deploy, manually constructed, etc? Also, to further debug this, can you tell me the names of the apps on either side of the equation? Since this is all in the public here, feel free to open up a ticket, and mention that I sent you and request the ticket to be sent over to DevEx.

Thanks,
Guðmundur Bjarni

[nfm]

@gudmundur Thanks for chiming in.

I've done some more investigating and think I've figured this out. Sorry for the wall of text!

Initially when I was trying to set up Sentry's release tracking, I did heroku run bash to check if HEROKU_SLUG_COMMIT was set, and it was. Then I realized it was set to a very old commit (from about 12 months ago), and assumed that was when we switched from git-push to using Pipelines (it actually wasn't - this was a red herring).

When I realized the environment variable was pointing at an old commit, I started looking around for other ways to get the SHA, and came across the runtime-dyno-metadata labs option. I ran heroku labs:info for our app, and it didn't have runtime-dyno-metadata enabled, so I enabled it. Then I wrote some code to retrieve the SHA from /etc/heroku/dyno. This worked as expected.

It turns out I was mistaken about when HEROKU_SLUG_COMMIT is present. I thought it was _always_ available, but it looks like you have to explicitly enable runtime-dyno-metadata for the environment variable to be set.

To figure out what was going on, I made a new app, did a git-push deploy, and ran a console, and HEROKU_SLUG_COMMIT was not present. Then I enabled runtime-dyno-metadata and ran a console, and /etc/heroku/dyno was present and correct, but HEROKU_SLUG_COMMIT was still not present. Then I added a new commit (#2), did another git-push deploy, and ran a console, and HEROKU_SLUG_COMMIT was then present, and correct. So it looks like you need to do a deploy after enabling labs for the environment variable to be set, but the /etc/heroku/dyno file is available as soon as you enable runtime-dyno-metadata.

Then I disabled runtime-dyno-metadata and ran a console, and HEROKU_SLUG_COMMIT was still present. I added a new commit (#3), did a git-push deploy, and ran a console, and HEROKU_SLUG_COMMIT remained present, and the value had incorrectly remained set to the SHA of commit #2, from when runtime-dyno-metadata was previously enabled.

So it turns out this was nothing to do with git-push vs Pipeline deploys. We must have had runtime-dyno-metadata enabled in the past, which set HEROKU_SLUG_COMMIT correctly until we disabled it about 12 months ago, and the environment variable HEROKU_SLUG_COMMIT remained present and set to an SHA from 12 months ago.

To corroborate this, HEROKU_SLUG_COMMIT is now set to the correct SHA on our main app. I checked it after enabling runtime-dyno-metadata and it was still incorrect, but that was before we did an actual deploy.

I'm not sure where that leaves things. I guess for raven-ruby, it's good to be aware that HEROKU_SLUG_COMMIT can be present and incorrect, but that /etc/heroku/dyno is always correct, if it's present. Heroku may also be able to remove the environment variable if runtime-dyno-metadata is disabled, as it will definitely be incorrect from that point onwards, and can cause significant confusion!

[gudmundur]

@nfm Nice! Thanks for the detective work on this one. I'm glad you got to the bottom of this. I agree that we shouldn't leave the HEROKU_SLUG_COMMIT hanging there if you disable the flag. I'm sending this issue and initial support ticket to the team that owns the runtime-dyno-metadata flag. Hopefully they can address the issue or give some insight into why it is like it is. Cheers!

[nateberkopec]

@nfm Cheers for that debugging. Definitely think we could improve our Heroku release detection based on that.

[nateberkopec]

See #566

[nfm]

Late to the party, but #566 looks great - more or less what we were doing. Thanks @nateberkopec!