Screwdriver: Bitbucket Cloud REST APIs are changing to improve user privacy

[shaochieh]

What happened:
https://developer.atlassian.com/cloud/bitbucket/bitbucket-api-changes-gdpr/
"In order to improve user privacy in accordance with GDPR regulations, Bitbucket and other Atlassian Cloud products are updating our product APIs to consolidate how we manage access to personal data.
...
The short summary is that username endpoints and username fields will stop being available on 29 April 2019, and we're introducing some new data points that are available immediately to replace them.
...
In all cases where Bitbucket APIs return user details, we are introducing two new fields to the object body, Atlassian account ID account_id and nickname."

https://github.com/screwdriver-cd/data-schema/blob/master/core/scm.js#L19
the SCHEMA_USER username key has to be updated to reflect that change. from username to nickname for starter. Thus dataSchema.core.scm.user is updated and won't complain about missing username like the following:

190601/152919.213, (1559402953119:screwdriver-api-6fbfb4c4c4-httrm:23:jwdjakcb:10866) [request,server,error] data: ValidationError: child "Username" fails because ["Username" is required]
    at Object.exports.process (/usr/src/app/node_modules/joi/lib/errors.js:196:19)
    at internals.Object._validateWithOptions (/usr/src/app/node_modules/joi/lib/types/any/index.js:675:31)
    at module.exports.internals.Any.root.validate (/usr/src/app/node_modules/joi/lib/index.js:146:23)
    at validate (/usr/src/app/node_modules/screwdriver-scm-bitbucket/node_modules/screwdriver-scm-base/index.js:18:24)
    at validate.then.then.decoratedAuthor (/usr/src/app/node_modules/screwdriver-scm-bitbucket/node_modules/screwdriver-scm-base/index.js:244:38)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7)
190601/152913.119, [response,api,events] http://screwdriver.default.svc.cluster.local/api: post /v4/events {} 500 (6099ms) 

What you expected to happen:
builds kicked off as usual.

How to reproduce it:
simply kick off any bitbucket scm based builds where it might issue api calls to bitbucket

[tkyi]

I don't think we can just change the data-schema, since scm-github also uses the same schema (https://github.com/screwdriver-cd/scm-github/blob/master/index.js#L829-L849). I also don't think username should be changed to nickname, since according to Bitbucket's description, nickname is not unique:
...
A new user identifier which is easily customized and always publicly visible. This user identifier will not be required to be unique to that user.
...

Instead, I think we can map it like this:

| Bitbucket | data-schema | description |
| --- | --- | --- |
| account_id | username | unique identifier |
| nickname | name | display name |

_Note: If we do the change according to the table above, might need to double check where we display username in Screwdriver UI, since the Bitbucket username is not user-friendly._

[shaochieh]

agreed. we found that you could fetch username with account_id, which is unique. our workaround is to update the scm-bitbucket to normalize the username. We may have to make an extra call to bitbucket for it to work, but worth a look. (ie. make extra API call using account_id to grab username). This will only work if the account_id can be use to grab the username.

[shaochieh]

looks like BB rolled back so username field and endpoint are back, so are other affected fields so we're good now. will let you know once they roll forward again somehow.

[shaochieh]

I just checked they rolled it forward again thus pipeline won't run now...
https://api.bitbucket.org/2.0/users/{username}
we might have to proceed with that workaround here to stitch username back

[xangxiong]

Looks like Bitbucket removed all reference to username. This field is no longer available, and the only API that still has reference to it is https://api.bitbucket.org/2.0/user/, which will return only the current login user's info.

With this in mind, we cannot easily make the extra calls to Bitbucket for the username to be use in stitching the username back into the Bitbucket SCM decorations.

Bitbucket is moving towards just using the account_id and nickname. The account_id is unique, but there is no guarantee that the nickname is unique.

@tkyi , do you know if the username field in Screwdriver SCM Base is used for anything else other than just display?

[tkyi]

The only two places in the Screwdriver UI I see username getting displayed is:
1.) for the login info in the upper right corner
_I think we should stick to showing a unique identifier in this case._
2.) for job disable/enable toggle metadata on the Options tab
_We can easily change this field to use name instead of username._

For all other purposes we just display name in the UI.

I think the mapping I suggested in my comment above still stands:
https://github.com/screwdriver-cd/screwdriver/issues/1662#issuecomment-498451788

[xangxiong]

Using account_id in place of username for Bitbucket should be fine with us as well. Until the UI gets update to display the name instead of username, we will just see the long UUID for the account_id. That is fine for our use case.

As for name, can we keep it as display_name instead of mapping it to nickname? This will reduce the amount of changes.

If the changes are fine, I can help patch the bitbucket scm to use account_id instead of username.

[tkyi]

The only thing that concerns me about display_name vs nickname is Bitbucket's description for display_name:

The value of this field will change depending on the privacy settings of the user account and the relationship between the requesting user and the target user. This field will always have some value.

Seems a little more uncertain than nickname.

A new user identifier which is easily customized and always publicly visible. This user identifier will not be required to be unique to that user.

But I'm okay with leaving display_name. 👍

---

Another thing to think about with the Bitbucket change in general is that your user data will be off moving forward. So with the account_id as username, Screwdriver will add you as a new user. You might need to do some DB migration if you really need to preserve that data.

[xangxiong]

Sound good with the nickname to be use as name.

As for DB migration, it should not impact us since we are in the evaluation phase for screwdriver.

What are the process to introduce a breaking change like this that may affect existing users of the Bitbucket SCM? Are there any documentation on DB migration?

[tkyi]

We usually just write up a detailed blog post with any instructions for DB migration if necessary.
For example, see the "Database Migration" section: https://blog.screwdriver.cd/post/175385505042/introducing-template-namespaces

[xangxiong]

After some development, looks like account_id does not exists for all user. During updating of the unit test, the batman account on Bitbucket returns with a null account_id. Though the unit tests does not make actual API request to Bitbucket, just seeing that account_id can be null raise concerns.

Since uuid can be use in place of both username and account_id, I opted to use the uuid to replace username. Initial testing and deployment on my local test environment looks good.

One thing to note, looks like changing the username to uuid in the decoration functions does not affect the username used to login to Bitbucket. Users are still able to login to Bitbucket through Screwdriver using their traditional username, which means that no DB migration is needed.

I'm guessing the username used in decorating commit and author are simply there for generating the proper links to the user's profile and not used for authenticating with Bitbucket or account generation on Screwdriver.

[xangxiong]

Pull request created.

https://github.com/screwdriver-cd/scm-bitbucket/pull/61

[jithine]

Closing based on https://github.com/screwdriver-cd/scm-bitbucket/pull/61