Appcenter: Support private git submodule from bitbucket

[FoodlyEric]

Describe the solution you'd like
Support private git submodule from Bitbucket.
Currently Netlify support private git submodule from Bitbucket. Netlify will generate one SSH key for every project that can then be transfered to an account with access to all submodules.

Describe alternatives you've considered
There is no way to get this working unless I move all my repos to Github, which is a problem due to financial reasons.

[abrunner73]

We would like to access private submodules using SSH with Azure DevOps as well.

[bramski]

This is blocking me. I will have to move away from appcenter as our app is well down this road already. Very annoying.

[teekirol]

Here's the workaround my team uses.

Say you have an npm dependency that is in a private BB repo:

"@foo/repo": "git+https://bitbucket.org/user/repo.git"

In our appcenter-post-clone.sh and appcenter-pre-build.sh we install that package as a separate step:

npm install git+https://mybbuser:${BB_PASSWORD}@bitbucket.org/user/repo.git

where BB_PASSWORD is an environment variable within the build config.

[nilofer]

Thanks for this feature request. A build script would be the workaround for this now. We'll keep this open for tracking further interest.

@teekirol if you've got a working build script, we'd love for you to contribute to the build script examples for others to use!

[IslamSalah]

Our use case was a bit different because we are using a private git repo as an npm dependency. This blog post helped us a lot.

I'll explain the general approach independently of npm. As a work around, we created an account and gave it read-only access to the concerned submodules. Then we generated an access token for this account using github access tokens. I believe you can also generate token on bitbucket but I haven't tried it.

Then we used on AppCenter something like this:
git clone https://<Token>@github.com/my-user-name/my-repo-name.git

[bramski]

Right, we solved this by using bitbucket's SSH keys for private NPM bitbucket git+ssh access. You go to bitbucket and add create a new SSH key for AppCenter.
You have it in your package json as:
"my-lib": "git+ssh://[email protected]/myuser/myrepo#mytag"
Then you base64 encode the private key as such:
base64 -i ~/.ssh/my_private_key | pbcopy
And you put that into a build variable BITBUCKET_SSH_KEY
Then you add the following appcenter-post-clone.sh file:

#!/usr/bin/env bash
#ADD Bitbucket fingerprint to known_hosts
mkdir -p ~/.ssh
echo "Adding bitbucket to ssh known hosts"
ssh-keyscan -t rsa bitbucket.org >> ~/.ssh/known_hosts
#ADD SSH key to the image
echo "Adding bitbucket SSH key"
echo $BITBUCKET_SSH_KEY | base64 -D > ~/.ssh/bitbucket-ssh
chmod 600 ~/.ssh/bitbucket-ssh
ssh-add ~/.ssh/bitbucket-ssh

Add the script, commit, then resave your build so it picks up the new script. Boom you're off to the races.

[jbstewart]

I would like to see submodule support for GitLab added as well.

[chrisparton1991]

Thanks for your script @bramski!

I was getting the following error:

Error loading key "/Users/runner/.ssh/bitbucket-ssh": invalid format

Turns out I had carriage returns in my SSH key (Copy+pasted it from a remote computer into a Windows text file). I verified this by adding od -xc ~/.ssh/bitbucket-ssh before the ssh-add call in the post clone script.

Long story short, the fix was to strip carriage returns from the private key prior to converting to base64:

cat ~/.ssh/my_private_key | tr -d '\r' | base64 | pbcopy

[ianthetechie]

This is an issue with GitLab as well. The initial fetch works, but submodules fail with an auth error (naturally). You supply proper auth for the main repo, but fail to address submodules... this makes no sense, Microsoft.

[bramski]

Did you say git submodules? I think I found your problem.

[Nickolas-]

Right, we solved this by using bitbucket's SSH keys for private NPM bitbucket git+ssh access. You go to bitbucket and add create a new SSH key for AppCenter.
You have it in your package json as:
"my-lib": "git+ssh://[email protected]/myuser/myrepo#mytag"
Then you base64 encode the private key as such:
base64 -i ~/.ssh/my_private_key | pbcopy
And you put that into a build variable BITBUCKET_SSH_KEY
Then you add the following appcenter-post-clone.sh file:

#!/usr/bin/env bash
#ADD Bitbucket fingerprint to known_hosts
mkdir -p ~/.ssh
echo "Adding bitbucket to ssh known hosts"
ssh-keyscan -t rsa bitbucket.org >> ~/.ssh/known_hosts
#ADD SSH key to the image
echo "Adding bitbucket SSH key"
echo $BITBUCKET_SSH_KEY | base64 -D > ~/.ssh/bitbucket-ssh
chmod 600 ~/.ssh/bitbucket-ssh
ssh-add ~/.ssh/bitbucket-ssh

Add the script, commit, then resave your build so it picks up the new script. Boom you're off to the races.
Hi @bramski
Is it still valid ? Because Im feeling that this post clone script invoke after submodule update ...

[alexeystrakh]

Switch from SSH to HTTPS because AppCenter is fetching using HTTPS:

##[command]git remote set-url origin https://***:***@gitlab.com/group/project-repo.git

So in your .gitsubmodules file change submodule url from

url = [email protected]:group/submodule-repo.git

to

url = https://gitlab.com/com:group/submodule-repo.git

[hannesjung]

Thanks @alexeystrakh it also works with BitBucket app passwords. You can create an app password by accessing your BitBucket profile settings:
Settings > App passwords > Create app password > Give Repositories / Read permission

Then you can change the urls of your submodules in the .gitmodules file to:
url = https://bb_username:[email protected]/organization/submodule-repo.git

[iamtesch]

A :+1: here for this issue; this doesn't work if you have private ssh submodules in your respository itself, b/c the post clone script is invoked after the submodules try to update, as mentioned by @Nickolas

If you are installing the ssh keys to be used later (e.g., w/ NPM), then this would be fine.

The https workaround mentioned by @alexeystrakh does address this problem, but requires a change to the repository that may not be possible depending on other constraints a project has.

[hoopoe]

Any plans to add pre-clone script? Switching chain of git submodules to use https looks very painful for us

[robertLichtnow]

Right, we solved this by using bitbucket's SSH keys for private NPM bitbucket git+ssh access. You go to bitbucket and add create a new SSH key for AppCenter.
You have it in your package json as:
"my-lib": "git+ssh://[email protected]/myuser/myrepo#mytag"
Then you base64 encode the private key as such:
base64 -i ~/.ssh/my_private_key | pbcopy
And you put that into a build variable BITBUCKET_SSH_KEY
Then you add the following appcenter-post-clone.sh file:

#!/usr/bin/env bash
#ADD Bitbucket fingerprint to known_hosts
mkdir -p ~/.ssh
echo "Adding bitbucket to ssh known hosts"
ssh-keyscan -t rsa bitbucket.org >> ~/.ssh/known_hosts
#ADD SSH key to the image
echo "Adding bitbucket SSH key"
echo $BITBUCKET_SSH_KEY | base64 -D > ~/.ssh/bitbucket-ssh
chmod 600 ~/.ssh/bitbucket-ssh
ssh-add ~/.ssh/bitbucket-ssh

Add the script, commit, then resave your build so it picks up the new script. Boom you're off to the races.

This is a great workaround, but I feel that appcenter should provide an ssh, so it gets easier to handle these kinds of situations

[johnnyzen]

This is still an issue for me. We have submodules in bitbucket that cannot get access. As iamtesch mentioned, the scripts do not help as access is required before post clone is fired.

Any update on this?

[msftbot[bot]]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[msftbot[bot]]

This issue will now be closed because it hasn't had any activity for 15 days after stale. Please feel free to open a new issue if you still have a question/issue or suggestion.