Git-lfs: The Git init & fetch approach breaks Git LFS

[strich]

I'm currently involved in setting up a Jenkins-based build for our project and I've found that the popular Git plugin for Jenkins takes the slightly weird approach of git init+fetch instead of git clone when setting up the repo on agent machines.

This appears to break Git LFS as all the binary files are still only the reference pointer files.

The plugin does not currently appear to have any advanced options to force git cloning and there are a number of aged active tickets for the plugin to support it.

The author has stated that adding git clone as a problem:

".... Git fetch was chosen because there are fewer ways it can hang (prompting for authentication information) in the various authentication scenarios supported by the plugin. It is even more challenging because there are currently no unit tests for the authentication scenarios, so they must be tested interactively (or not tested) at each plugin release. Adding the option to use "clone" instead of "fetch" would effectively double the cases we need to test in an already very complicated portion of the code."

I'm no expert to Git and don't propose any best solution to this problem, nor do I imply that the solution is one to be solved by Git LFS. However I'd like to seek advice here on what the best practice solution might be from the Git LFS point of view.

[technoweenie]

In the upcoming Git LFS v0.6.0, you can run git lfs pull to download the lfs objects and replace the pointers. Does the git plugin let you add another command like that?

[strich]

I believe that can be done yes. I'm also willing to contribute to the plugin to more officially support Git LFS by way of actual flags/options.

We could potentially have the plugin use lfs wrapper commands such as git lfs init/fetch. Would that be a better approach in your opinion?

[strich]

@technoweenie Doesn't look like there is a simple way for a user to add custom git commands as part of the Git plugin process sadly. I can add a custom build step, for now. But I'm currently struggling to have any git-lfs command (Windows here) perform non-interactive auth.

Am I right in assuming that if I manually clone the Git LFS active repo the first time it will be fine with git fetch commands in future?

[technoweenie]

I'm not sure the git plugin should have _explicit_ Git LFS support. But it should be possible to change or add on to it's fetching routine.

Since CI servers typically don't commit, you don't actually need git lfs init. You can just run git lfs pull.

Note: the pull command doesn't exist in Git LFS v0.5.x. I'm hoping to push out v0.6.0 within the next week though. It's a huge improvement.

[strich]

Normally I would agree, but the problem is that the Git plugin fully manages the Git login credentials during the build process and wipes them once its own build step is complete. If I try to add any git commands outside of that plugin then I'll have to go through a lot just to get auth working.

If you're confident git lfs pull is due within that time frame then that could be the best solution. I can write an additive sub-build step for the plugin that only those who want LFS can use.

[strich]

So after downloading Git LFS v0.60 I tried adding Git LFS pull to the commands I run as part of the batch however I get errors. See issue #660.

[juddgaddie]

I'm seeing a similar problem with Jenkins I tried to do a git lfs pull, but get the following error: http://pastebin.com/c5h1yEgF

[matthauck]

Since this thread is mostly about jenkins usage of git lfs, and since it has such google presence for search terms "git lfs jenkins", I wanted to just note here I filed an issue with JIRA to add explicit support for git lfs, with some pull requests of how it could basically work. https://issues.jenkins-ci.org/browse/JENKINS-35687.

[strich]

Hi @matthauck. If you haven't seen it already, there is an existing Jenkins issue I made many months ago with some discussion that might be relevant to your work: https://issues.jenkins-ci.org/browse/JENKINS-30318

[matthauck]

Yeah, saw that - I think I linked the issues in jira. Decided to not pursue that discussion since the git plugin seems pretty committed to init+fetch, and also for that to magically work with lfs, we would have have to run git lfs install as the jenkins user on all the jenkins nodes -- more setup steps == bad. So, decided to start a new issue for explicit support.

[qrilka]

@strich this issue seems to be almost a year old already - did you find any decent workaround maybe?

[strich]

Nope! I use a second script step to run the correct commands after Git has gone.

[technoweenie]

Closing this, as it's really about getting Jenkins to recognize Git LFS. As @strich notes, it can be worked around.

[pwr22]

Does anyone have an example workaround available somewhere?

[qrilka]

@pwr22 we've just added installation of git lfs and git lfs pull into job scripts, nothing fancy

[matthauck]

I referenced some PR's I created to get this working on a fork of the github plugin: https://issues.jenkins-ci.org/browse/JENKINS-35687

[vlseattlebot]

This is completely broken behavior for Jenkins not to support Git-LFS. It's incredible to me that the most popular build harness doesn't support such near-standard functionality. Anyone working on a project of any reasonable size, and particularly projects that do have a number of binaries in the repo, should be using LFS.

[lgreenlee]

@vlseattlebot in their defense, git lfs is not really that easy to use. If git pull worked liked they said it would then we wouldn't have these problems.

[qrilka]

@lgreenlee Jenkins uses init+fetch, why do you talk about git pull?

[creste]

I created two new PRs that fixed a few issues with @matthauck's PRs:

• jenkinsci/git-client-plugin#232
• jenkinsci/git-plugin#463

[MarkEWaite]

I've started the evaluation of the two pull requests for the git client plugin and the git plugin. Notes are being added to https://github.com/jenkinsci/git-plugin/pull/463

[maxter2323]

Hello @MarkEWaite . First of all I would like to say that I really appreciate your work and effort for the LFS support and Jenkins development. You are helping a lot of people.

At my work place I tried to set up Jenkins on Mac Os Sierra and encountered this LFS issue("error code 128" and "unknown command git-lfs").
To fix it, I did many things, including setting up a git path in Jenkins configuration, downloading your docker plugin snaphots, using experimental plugin versions and etc. All that time "git lfs" was working in the terminal for jenkins user and my account as well but failed in Jenkins pipeline building.

It even worked with:

sudo su jenkins
git lfs

For multiple times I was checking "usr/bin" and "usr/local/bin" to ensure that git-lfs is located there.
Everything seemed fine but the error was constantly occurring.

So when I decided to do this:
git --exec-path

I was surprised that instead of "/usr/bin/" or "/usr/local/bin" I see:
_/Applications/Xcode.app/Contents/Developer/usr/libexec/git-core_

So after this I just dragged "git-lfs" unix executable to this folder and now everything works.
I hope it will help somebody as it helped me.

[MarkEWaite]

The Jenkins git plugin 3.1.0 release, 4 Mar 2017, includes git large file support.

[aufano]

Now works!, thanks to @maxter2323 I had the same issue with jenkins installed in mac os , with git plugin 3.3.1

[ludddd]

@maxter2323 , great thanks!!!
For those who stumble upon this thread:
This fix (3 messages up) solves the problem with "git-lfs filter-process: git-lfs: command not found" also.

[alexdmotoc]

EDIT (fixed):
Homebrew installs git-lfs in /usr/local/bin; Jenkins must know about this.

I added the PATH environment variable to Manage Jenkins -> Nodes -> Configure -> Environment Vars and it works now

Deleted the modifications I mentioned below

---

Hi @maxter2323,

I also have Jenkins running on a macOS High Sierra, and had this issue, so tried your proposed solution.

My git --exec-path returned /Library/Developer/CommandLineTools/usr/libexec/git-core so I copied an alias to git-lfs there.

Jenkins does see git-lfs now, but I get this error:
stderr: git-lfs filter-process: /Library/Developer/CommandLineTools/usr/libexec/git-core/git-lfs: Permission denied fatal: The remote end hung up unexpectedly

Any suggestions what can I do in this case?

[matthauck]

Is there a way to lock this issue?

[ttaylorr]

As @matthauck notes, this conversation is getting off-topic, since the issue was closed some time ago. If anyone is having issues with Git LFS that are different to the one described in the original body of the issue, then please do not hesitate to open a new issue stating so.