Netlify-cms: Gitlab as backend.

[muhajirdev]

As a gitlab user, it would be nice if you support gitlab too as the backend :)

[biilmann]

Right now only GitHub has a "Data API" that allows us the level of control over commits that we need for the CMS. Our plan is to make netlify-git-api capable of synching from one git repository to another, so you can spin up a git API instance for the CMS, and have all commits made from there synched to your git repo in GitLab/BitBucket, Gogs, etc...

[muhajirdev]

did you mean this API http://docs.gitlab.com/ce/api/repository_files.html ?

[b-long]

Any update non-github git support?

[calavera]

@b-long this is not a priority for us at the moment. However, the cms is designed to be able to plug different backends if they support the right API. I don't think it would be hard to copy GitHub's implementation and adapt it for GitLab if someone is motivated enough 😉

https://github.com/netlify/netlify-cms/tree/master/src/backends/github

[tech4him1]

I've been trying to work on creating a GitLab implementation, but I've run into one big difference in the GitHub and GitLab APIs: as far as I can tell, GitLab does not have any low-level APIs for adding blobs and trees directly. So the metadata would need to be stored under a regular branch. Unfortunately, there is no current way to create an orphan branch either. If anybody can correct me on this, that would be great.

[biilmann]

From what I've seen you're right about that. The backend would have to behave a bit differently than the GitHub backend. You could consider using a special file in the main branch as an alternative to an orphaned metadata branch in GitLab...

[tech4him1]

Yes, that would work. Also, I was wondering if you could just store the metadata in the pull request itself. Or did you guys already try that and have problems?

[Benaiah]

I suspect a single file would end up using a lot more bandwidth over time as the file grew - that approach might work better with a special directory from which the CMS could request individual metadata files - it'd also likely be less work, since it's already designed to use multiple files.

[b-long]

It seems there's momentum, so I think we ought to get started. I've done minimal research, but it seems a project called Prose is similarly positioned to Netlify. Prose is also written in NodeJS and it also has users asking for GitLab API support. Even better, @dereklieu seems to have forked and developed the ability to work with GitLab.

Perhaps we can explore this with @dereklieu , find the functions that are needed in both Netlify and Prose, and distill them into their own module that can be used by both projects?

[tech4him1]

I have been looking at the APIs, and it looks like for the standard workflow (non editorial), the APIs are entirely compatible. It is only when you try to start adding in metadata in the editorial workflow that you run into differences.

[tech4him1]

@b-long Does Prose have the ability to save as drafts? If not, that would still be something we would have to work out above and beyond @dereklieu's implementation if we wanted to support the editorial workflow on GitLab as well.

[b-long]

@tech4him1 I don't think the draft feature is included in Prose (it's focus seems to be immediate publication and editing Markdown). What about simply using the GitLab capabilities that have been built into Prose for now (as a baseline), and disabling saving drafts if a project's source is hosted in GitLab?

[tech4him1]

That's an option, although I don't think it would be too hard to do what @biilmann and @Benaiah were suggesting and create a separate folder to hold the drafts.

EDIT: I'm not saying it wouldn't be good to just start as it is, just test it should be easy to add later.

[dereklieu]

Hey @b-long I'm unfortunately not too familiar with your endeavor here happy to help where I can.

Prose has a mostly working beta that interacts with gitlab. Currently I'm hosting it off my fork via Github pages. Link (http://lieu.io/prose/).

Prose is actually all client-side javascript, aside from a small oauth server that is.

Prose does have the concept of a draft. You put drafts in _drafts and posts in _posts, and there is a UI toggle to move a file from one folder to another, or vice versa. This follows Jekyll naming conventions.

Prose is always looking for maintainers or contributors, but it's core "mission statement" is to be a great interface to editing content on Github (and now Gitlab!) that doesn't require tech savvy.

I would love to work on a Github/Gitlab API abstraction as well, as you say the two are quite similar, at least in how Prose uses them.

[tech4him1]

@b-long How are you thinking of building the authentication? It looks like from what @dereklieu said that Prose has a separate Oauth server to handle that part, so would you want to try to get Gitlab support into netlify-auth, or some other way?

[biilmann]

@tech4him1 the netlify auth provider plays the same role in our case as the oauth server Prose uses (and I would love to see some integration with their Oauth Server as well for auth).

Using the same auth system as for GitHub, the only change needed to get a GitLab access token is to change provider to gitlab here:

https://github.com/netlify/netlify-cms/blob/master/src/backends/github/AuthenticationPage.js#L23

[tech4him1]

I could very well be mistaken, but I thought netlify-auth only supported GitHub and BitBucket:

1. The docs: https://www.netlify.com/docs/authentication-providers/
   > ##Supported Providers
   >
   > Right now Netlify only supports GitHub and BitBucket as authentication providers, but we’re working on adding more.
   >
   > If you’re building a single page app and need to speak with a specific API, let us know and we’ll help you out.
2. The simple implementation I made (https://github.com/tech4him1/netlify-cms/tree/gitlabSupport/src/backends/gitlab) was not able to authenticate because netlify-auth could not find an access token. However, when I try to add an access token to my Netlify app, it only shows GitHub and BitBucket as options.
   

[b-long]

I've not yet had an opportunity to wrap my head around what's needed for Netlify to support GitLab as a backend, but I'm so appreciative of this discussion since it will be helpful to me or anyone in beginning the work.

@dereklieu Sorry to pull you in without any context, I just saw your work and thought you may be interested in this discussion! :)

@tech4him1 I'm unsure (re: your last comment), perhaps that's just a bug in the documentation? The signup page for Netlify has GitHub, GitLab and Bitbucket: https://app.netlify.com/signup .

[biilmann]

Yes, we do support GitLab, but do lack GitLab in the UI for our authentication providers right now. I'll make sure we get that fixed - and for testing meanwhile just ping us in the Intercom widget within the app and we can help getting it set up...

[tech4him1]

@b-long I'd still be glad to help with a shared module if you want, but here is a quick implementation I put together by copying the GitHub implementation. It doesn't have Editorial Workflow support.
https://github.com/tech4him1/netlify-cms/tree/gitlabSupport/src/backends/gitlab

The only thing that I had problems with is the persistFiles function, used to upload the post and any media attached to it. On the GitHub implementation, it uploads each file individually, then advances the branch with a single commit using the low-level API. With GitLab (which doesn't have a low-level API), you can upload all the files in a single commit, but I didn't think that would be a good idea since if you had an image-heavy post, that would be a huge request. Instead, I commited each file and the post separately, but using the same commit message. The only bad thing about that is that if one upload fails, the ones before that are already commited, possibly leaving a post without its images/media. Suggestions welcome.

If you want, I can add you to the GitLab repo so you can try/test the demo site that I made (https://supervisor-ann-84734.netlify.com/admin).

[Benaiah]

@tech4him1 Personally, I think a single large request is better than risking inconsistent repository state. I'm in favor of just uploading everything for an entry in a single commit, if those two options are our choices.

[tech4him1]

@Benaiah Yes, you are right about avoiding inconsistent repository state. My only concern was if a request was too large. What would you think of uploading all the images/assets first, then the post? That way, if something broke, you could have assets uploaded, but the post wouldn't show up. Or do you think the fact that netlify-cms is built to run on any cms should make us stick to one request, in case assets would be shown individually?

[tech4him1]

@b-long @Benaiah It looks like prose uploads each media file individually when you add them to the page, and uploads the page when you save it. That way, if anything breaks with an image upload, the post update fails and there is no incosistent state. For our implementation, what do you think about something like this, where each media file is uploaded fist, and then if there are no problems, the main post is uploaded:

...
const files = mediaFiles.forEach(mediaFile => {
  return this.uploadAndCommit(`Created ${ mediaFile.name }.`, mediaFile);
});
return Promise.all(files).then(() => {
  this.uploadAndCommit(options.commitMessage, post);
})

[calavera]

@tech4him1 it looks like GitLab actually has an API to create branches:
https://docs.gitlab.com/ce/api/branches.html#create-repository-branch

[tech4him1]

@calavera What are you suggesting the use of the branches for? If it is for the Simple workflow, I'd be glad to get your ideas, but if for the Editorial Workflow, the problem we were running into is that GitLab does not have any low-level APIs for adding blobs and trees directly. So the metadata would need to be stored under a regular branch. But as far as I can tell, the GitLab API doesn't allow you to create an orphan branch either, which means we would need to store metadata in a file under master.

[calavera]

GitLab's api is open source, maybe they are interested in having those endpoints.

[tech4him1]

Related Issue on GitLab: https://gitlab.com/gitlab-org/gitlab-ce/issues/28993

[timwood]

If finished will this work with a self-hosted Gitlab CE edition?

[tech4him1]

@timwood Good question, the APIs are the same, so we would just need to make the URL a variable. In my understanding that should be really easy.

[caiofbpa]

Any news on this? I'm interested in using Netlify-CMS with GitLab, how can I help?

[tech4him1]

@caiofbpa Here is our issue on GitLab regarding the APIs we need: https://gitlab.com/gitlab-org/gitlab-ce/issues/28993. If you can contribute to that or (even better) find a way to use their existing APIs for what we need, that would be great!

[tech4him1]

Here is another GitLab issue that would help as a workaround for some APIs if necessary: https://gitlab.com/gitlab-org/gitlab-ce/issues/31538

[erquhart]

@caiofbpa any work toward this would be awesome! I haven't looked into it myself, but I do remember hearing that GitLab's current API should work for the CMS, but that needs to be definitively verified. If you're up for confirming that, it's a good first step toward integration.

[ShalokShalom]

I am also very interested in that topic. So, this repo here have to get ported, in order to provide it somewhere else, like in Gitea?

https://github.com/netlify/netlify-cms/tree/master/src/backends/github

[tech4him1]

If anybody wants to help contribute, we have a working PR at #517.

[erquhart]

Thank you for killing it on this @tech4him1, can't wait to see it land!

[suprafly]

Would love to know when this is working!

[ShalokShalom]

Then subscribe this; https://github.com/netlify/netlify-cms/pull/517

[erquhart]

Backends will be a big focus after 1.0 drops in early December - in the meantime, feel free to grab the branch for #517 and test it out! Reviews and comments on the PR are super helpful.

[tech4him1]

We should add a note to the docs that self-hosted GitLab instances are (should be, I haven't tested it yet) supported by setting an api_root in your backend config:

backend:
  name: gitlab
  repo: username/repo
  api_root: https://my-hosted-gitlab-instance.com/api/v4
  auth_type: implicit
  appid: my-app-id-from-creating-oauth-app
  auth_url: https://my-hosted-gitlab-instance.com/oauth/authorize

cc/ @verythorough @timwood

EDIT: Updated to add implicit OAuth settings (https://github.com/netlify/netlify-cms/pull/1314).

[verythorough]

If you're following this issue, check out the latest PR, due to merge soon: https://github.com/netlify/netlify-cms/pull/1343

[tech4him1]

This was merged way back in v1.9, closing!