Renovate: Gitlab support

[guumaster]

This seems very useful. Would it work with gitlab instead of github? If not, any plans to support that?

[rarkins]

Unfortunately it only supports the GitHub API for now.

You can see the GitHub API commands that are used here: https://github.com/singapore/renovate/blob/master/lib/api/github.js

Most are "generic" (create branch, get file contents, etc) but a few are quite GitHub-specific. It would probably be easily ported to support Gitlab if Gitlab API supports equivalent capabilities.

I'll rename this issue to "Gitlab support" and keep it open.

[hutson]

I myself moved all my projects to GitLab, for various reasons, and though that was a good decision in my case, I greatly miss the benefits of leveraging tools like GreenKeeper, and now renovate.

[hutson]

I went through the GitHub API file you have and looked for GitLab equivalents. As you noted, there are GitLab equivalents for most end-points (The URL and response body may be a little different.).

There are two shortcomings with the GitLab API. GitLab does not have a convenient search end-point, so all files and merge requests would need to be fetched, with filtering done on client side.

Lastly, there is no equivalent to the end-points used by the commitFile function (and the functions it uses), but there is a GitLab end-point to create commits with arbitrary files.

Each function is listed below, with the first API end-point being what's used in the GitHub file, and the second being what GitLab supports.

initRepo

• GET repos/${repoName}
  
  
  
  • res.body.owner.login
  
  
  • res.body.default_branch
  
  
• GET /projects/:id
  
  
  
  • res.body.owner.name
  
  
  • res.body.default_branch
  
  

getBranchCommit

• GET repos/${config.repoName}/git/refs/heads/${branchName}
  
  
  
  • res.body.object.sha
  
  
• GET /projects/:id/repository/commits/:sha
  
  
  
  • res.body.id
  
  

getCommitTree

• GET repos/${config.repoName}/git/commits/${commit}
  
  
  
  • res.body.tree.sha
  
  
• GET /projects/:id/repository/commits/:sha
  
  
  
  • res.body.parent_ids[0]
  
  

getRateLimit

• GET rate_limit
  
  
  
  • res.body
  
  
• No GitLab rate limit end-point, and no documented rate limit for their API (though they do have a rate limit)

findFilePaths

• GET search/code?q=repo:${config.repoName}+filename:${fileName}
• No known equivalent in GitLab. It's possible to retrieve a list of all files, and filter on that (https://docs.gitlab.com/ee/api/repository_files.html#get-file-from-repository)

branchExists

• GET repos/${config.repoName}/git/refs/heads/${branchName}
• GET /projects/:id/repository/commits/:sha

createBranch

• POST repos/${config.repoName}/git/refs
• POST /projects/:id/repository/branches
  
  
  
  • branch_name
  
  
  • ref (commit sha)
  
  

deleteBranch

• DELETE repos/${config.repoName}/git/refs/heads/${branchName}
• DELETE /projects/:id/repository/branches/:branch

updateBranch

• PATCH repos/${config.repoName}/git/refs/heads/${branchName}
• No known equivalent, but the create branch end-point may work (no documentation though)

getBranchPr

• GET repos/${config.repoName}/pulls?+ state=open&base=${config.defaultBranch}&head=${config.owner}:${branchName}
• No known equivalent. It's possible to retrieve a list of all merge requests, and filter on that (https://docs.gitlab.com/ee/api/merge_requests.html#list-merge-requests)

addAssignee

• POST repos/${config.repoName}/issues/${issueNo}/assignees
• PUT /projects/:id/issues/:issue_id
  
  
  
  • assignee_id
  
  

addLabels

• POST repos/${config.repoName}/issues/${issueNo}/labels
• PUT /projects/:id/issues/:issue_id
  
  
  
  • labels
  
  

findPr

• GET repos/${config.repoName}/pulls?head=${config.owner}:${branchName}
• No known equivalent. It's possible to retrieve a list of all merge requests, and filter on that (https://docs.gitlab.com/ee/api/merge_requests.html#list-merge-requests)

checkForClosedPr

• GET repos/${config.repoName}/pulls?state=closed&head=${config.owner}:${branchName}
• No known equivalent. It's possible to retrieve a list of all merge requests, and filter on that (https://docs.gitlab.com/ee/api/merge_requests.html#list-merge-requests)

createPr

• POST repos/${config.repoName}/pulls
• POST /projects/:id/merge_requests

getPr

• PATCH repos/${config.repoName}/pulls/${prNo}
• PUT /projects/:id/merge_requests/:merge_request_id
  
  
  
  • Requires branchName, which is not already passed to this function.
  
  

getFile/getFileContent/getFileJson

• GET repos/${config.repoName}/contents/${filePath}?ref=${branchName}
• GET /projects/:id/repository/files?file_path=${filePath}&ref=${branchName}

writeFile

• PUT repos/${config.repoName}/contents/${filePath}
• GitLab - To Create File - POST /projects/:id/repository/files
• GitLab - To Update File - PUT /projects/:id/repository/files

commitFile -> createBlob -> createTree -> createCommit

• No known equivalent end-points for these functions. However, you may be able to create the required commit, and file for that commit, using the commit end-point (https://docs.gitlab.com/ee/api/commits.html#create-a-commit-with-multiple-files-and-actions)

[rarkins]

@hbetts that's awesome help - thanks.

I think that the two main shortcomings you mention are not blockers. In fact they are both essentially convenience functions.

For the "find file" one, that's used for convenience with monorepos to automatically discover all package.json files in a repo. It can be overridden if you manually configure all file locations instead. So in an initial release we could leave this off.

For "commit file", that is a low-level hack I recently added that enables two things:

• When previously using the high level API to first "create branch" and then "commit file", it was resulting in two webhook notifications and hence two builds to CI systems. Now it's one
• This allows me to do the equivalent of a "rebase" whenever I detect that a PR has a merge conflict, and again only triggering one build in CI

Hence, I think that it's possible to get by without this low-level, and just deal with the implications.

[rarkins]

I'm taking first steps to implement this

[rarkins]

Getting there..

[rarkins]

So.. Gitlab deleted themselves so work cannot resume yet.

[SebastienElet]

They are transparent on the incident : https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik/pub

[hutson]

@rarkins wanted to say thank you for gl-got.

I've begun refactoring semantic-release-gitlab to take advantage of gl-got - https://gitlab.com/hyper-expanse/semantic-release-gitlab-notifier/merge_requests/3/diffs

Please let me know if there's anything I can do to help you with gl-got.

[rarkins]

@hbetts I'm glad to hear it's of use! I thought it would make the GitLab API for renovate easier to keep consistent. I've added you as a Collaborator for gl-got in case you find anything missing. Let's bump it to 1.0.0 once we've tested it a bit and using it in production for either renovate or semantic-release-gitlab.

[rarkins]

@hbetts after discussing with the gh-got author, I pushed some changes to GitHub

• Now keeping the git history from gh-got
• Need to use a version number higher than existing gh-got latest

Hence this branch now uses gl-got version 6.0.0-rc0 and will update to 6.0.0 once I feel it's been validated enough. Feel free to let me know if you think it's ready.

[rarkins]

GitLab support is now added in PR #83 and is ready for testing. Cc @guumaster @hbetts

As I'm not a regular GitLab user, I won't merge this PR until someone else thinks it's ready. Also feel free to comment on any of the implementation details.

[rarkins]

Check the contributing.md for instructions on how to run locally (you'll also need to check out the relevant branch).

FYI here is how I am running it locally to test:

LOG_LEVEL=debug npm run start-babel -- --platform=gitlab <username/repo>

[guumaster]

Thanks for all the effort guys! Great job. I'll try it asap and let you know

[hutson]

I ran renovate with the following command:

LOG_LEVEL=debug npm run start-babel -- --platform=gitlab --token=TOKEN --commitMessage="chore(package): update dependency {{depName}} to version {{newVersion}}" --labels=refactor,operations --onboarding=false hyper-expanse/test-project

Here's a section of the verbose output for the outdated semver dependency.

I'm including it here because the output ends with an _error_. That error appears for every single dependency that is updated. It should, however, be noted that merge requests do get created as expected - https://gitlab.com/hyper-expanse/test-project/merge_requests

debug: semver: Checking for closed PR
debug: findPr(renovate/semver-5.x, Pin dependency semver to version 5.3.0, closed)
debug: Closed PR existed: false
debug: Checking if branch exists: renovate/semver-5.x
debug: Branch doesn't exist
verbose: semver: creating new branch renovate/semver-5.x
debug: setNewValue: dependencies.semver = 5.3.0
debug: Starting search at index 544
debug: Found match at index 957
debug: Replacing "^5.1.0" with "5.3.0" at index 957
debug: commitFileToBranch(renovate/semver-5.x, package.json, fileContents, message, master)
debug: Checking if branch exists: renovate/semver-5.x
debug: Branch doesn't exist
debug: Creating branch renovate/semver-5.x
debug: package.json exists - updating it
debug: Ensuring PR
debug: getBranchPr(renovate/semver-5.x)
debug: Got res with 4 results
debug: Didn't find existing PR for branch renovate/semver-5.x
debug: Creating Merge Request: Pin dependency semver to version 5.3.0
debug: Adding labels refactor,operations to #2217526
error: semver failed to ensure PR: {"message":"Response code 404 (Not Found)","host":"gitlab.com","hostname":"gitlab.com","method":"PUT","path":"/api/v3/projects/hyper-expanse%2Ftest-project/merge_requests/prNo?labels=refactor,operations","statusCode":404,"statusMessage":"Not Found"}

So in addition to not understanding what the error is from, I also see two instances of Creating branch renovate/semver-5.x.

[hutson]

This may be out of the scope of this particular change, but upon figuring out what was causing the label error to occur, I re-ran renovate, mostly in hopes that it would go back and update all the pending pull requests with the two labels I specified.

Apparently if a pull request already exists, it only checks whether the body or title our out of date, and updates them if they are.

Would it make sense to update labels as well?

[rarkins]

I didn't think it does. For example I add the label "ready" with renovate but remove it manually and replace with "waiting" if I need to add more code or perform manual tests. So I wouldn't want that ready label being added back again. Same logic for assignees.

[hutson]

@rarkins, that description of how you would interact with the pull requests makes sense, and I agree that automatically updating labels would be problematic. I retract my suggestion.

[rarkins]

On the same topic, GitHub will automatically create labels you specify to add. Ie no such thing as a "label doesn't exist" error. Is GitLab the same?

[hutson]

gl-got 6.0.0-rc0 is now used by the latest version of semantic-release-gitlab-notifier, which has been released in version 3.0.0 of semantic-release-gitlab.

[rarkins]

@hbetts In that case, I've pushed stable version 6.0.0 to npm

[hutson]

Apparently GitLab will create a label if it doesn't already exist:

Using --labels=doesntexist hyper-expanse/test-project