Cirrus-ci-docs: Login and encrypted variable flakiness

[connorbrinton]

I'm having a lot of issues with Cirrus CI today. Specifically:

• Intermittent authentication issues: From time to time, I'll try to open a task page and see "Page not found". I'll then click the link to log in, but it says something about a 500 error and an "io exception" (?)
• Encrypted variable substitution: An encrypted variable which has worked well for a while is no longer being decrypted. The full ENCRYPTED[...] identifier is being substituted in its place

My questions are:

• Are these known problems?
• Are these problems specific to me? and
• What can I do to avoid these problems? 😅

Thanks for making Cirrus CI!

[fkorotkov]

Hey @connorbrinton,

there was some rare issues this morning with decryption of variables. There were two other reports so far but it should be fixed as of noon EST.

Will take a look into your log out issue. Haven't seen similar reports yet. Will update you with my findings shortly. Sorry for the inconveniences!

[connorbrinton]

Hi @fkorotkov, I'm still running into issues with variable decryption (I ran the two pipelines three times each to make sure 😅). I'll try creating a new variable to see if it can be decrypted, but I'm not aware of any changes in the old variable that would result in it not being able to be decrypted

[fkorotkov]

Interesting. Could you please provide task ids for some of the failing tasks?

[connorbrinton]

Yup, 5658849370963968 is having this issue

[fkorotkov]

I think I found the root cause of the flakiness and weirdness. Transitive dependency of a dependency introduced a flaky regression which starts appearing only on internal cache invalidation.

Fixing it now and will fully rollout in about an hour. Sorry for the inconveniences!

[connorbrinton]

No worries, thanks for digging to find the issue! 😁

[fkorotkov]

@connorbrinton not sure if failure of 5658849370963968 is related to the flakiness. There should've been a warning on the task in that case. Something like "Failed to decrypt some varibales."

Encrypted variable are decrypted only if they appear under environment field (or env as alias). If you want you can send .cirrus.yml to [email protected] so we can troubleshoot there.

[connorbrinton]

The variable is under the env field, and the .cirrus.yml is the same on the master branch (where this task succeeds) and the feature branch (where it fails). Strangely, on the master branch I've also seen variables not even being substituted. I'll send my .cirrus.yml to [email protected] to see if I'm doing anything unusual 🙂

[fkorotkov]

The issue was resolved via support. The issue with the encrypted variable was related to a bug in GitHub API where it returns wrong permissions for thrid party GitHub Apps like dependabot.

I did reach to GitHub support about the issue a while ago, pinged them again but also created a public issue for a reference: https://github.community/t5/GitHub-API-Development-and/Repository-permissions-for-GitHub-App-Fake-User/m-p/30633#M2883