Gitea: Repo mirroring doesn't mirror LFS data.

[damomurf]

• Gitea version (or commit ref): 1.0.0+220-g027591a3
• Git version: 2.11.0
• Operating system: Ubuntu 16.04 (Docker)
• Database (use [x]):
  
  
  
  • [x] PostgreSQL
  
  
  • [ ] MySQL
  
  
  • [ ] SQLite
  
  
• Can you reproduce the bug at https://try.gitea.io:
  
  
  
  • [ ] Yes (provide example URL)
  
  
  • [ ] No
  
  
  • [x] Not relevant
  
  
• Log gist:

Description

In a clean install of Gitea I have attempted to mirror a Bitbucket server repository that I know has LFS enabled and stores LFS data. The LFS data appears to not being mirrored in Gitea. (The configured LFS data directory remains empty, even though the repository mirroring appears to be successful).

[lunny]

This is the same reason with #732. Let's close this and discuss there.

[fabian-z]

I'm afraid I have to disagree: #732 is not related. Gitea cannot currently retrieve LFS objects from another server. This would basically require adding a complete LFS client on top of the current implementation.

@damomurf: Please see GitHub documentation regarding repository mirroring with LFS

[lunny]

Seems you are right.

[fabian-z]

I'd suggest to tag this as a feature request. Current behaviour is basically expected imho and supporting LFS mirrors will be challenging in some regards:

• Need LFS client implementation
  
  
  
  • CLI client not developed for interactive use
  
  
  • Reusing source would require rewriting to some extent
  
  
• Using CLI client
  
  
  
  • Would use at least double the disk space for the LFS objects during mirror
  
  
  • Continuous mirror pulling would likely need double the disk space all the time
  
  
• The operation will need to scan all files in every commit for LFS meta files, potentially taking very long for large repositories

Just out of interest, does somebody know whether GitHub support automated LFS object mirroring?

For future reference:
GitLab issue, git-lfs issue

[lunny]

Maybe put a notice on migrating UI at first.

[fabian-z]

I'll look into adding a note to the UI when LFS support is enabled.

[boskiv]

What is steps for mirroring LFS now? Is there some hook to make LFS pull with cmd ?
Or may be I need to setup some cron task ?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

[fabian-z]

@boskiv
You will currently have to manually mirror the repository if you wish to transfer the LFS objects, too.
It may or may not be (easily) possible to setup a custom mirror using cronjobs and shell scripts, but you would need to plan for double the disk space usage (permanently) and possibly use a separate local user account.

Please also see the GitHub docs for manual steps.

[mloskot]

BTW, the GitHub docs is what currently Git LFS recommends as per https://github.com/git-lfs/git-lfs/issues/2342#issuecomment-310323647

---

@fabian-z Forgive me if I'm hijacking the issue, but could you explain your last comment in more details, in the context explained below?

I have a GitHub repo with files in Git LFS.
I'm looking for solution to have Gitea serve mirror that repo (just Git repo).

TL;TR: Is it possible?

Would something along these lines work?

1. Mirror GitHub repo with LFS

   git clone --bare <GITHUB-URL> <GITHUB-DIR>
   git lfs fetch origin --all
   git push --mirror <GITEA-URL>
   git lfs push --all <GITEA-URL>
   

2. Attach Gitea to serve the files pushed to <GITEA-URL>

3. Frequently run

   cd <GITHUB-DIR>
   git fetch origin --all
   git lfs fetch origin --all
   git push --all <GITEA-URL>
   git lfs push --all <GITEA-URL>
   

[GlassedSilver]

Is there anyone kinda sorta working on it? Or is there anyone who figured out a way how to do this manually/scripted externally for the time being?

I'm using Gitea to keep backups of software I use or just find interesting, just in case... Obviously having the LFS objects as well is often very handy, many times necessary.

[zeripath]

No one is working on this. A bounty on bountysource might spur someone to look at it.

From a quick glance we would need to implement an LFS client - possibly through vendoring git-lfs/git-lfs. There would need to be significant changes to the mirroring UI too.

[GlassedSilver]

Chipped in 70 bucks into the bounty: https://www.bountysource.com/issues/41700547-repo-mirroring-doesn-t-mirror-lfs-data

[GlassedSilver]

Is it rude for me to ask for a maintainer of the repo to add the bounty tag and maybe declare the bounty value in the title?

Cheers and overall many many thanks for this lovely project to everyone who is working on it. It's of great use to me. :)

[techknowlogick]

@GlassedSilver thanks for your support :) I've just applied the bounty label to make this easier to find.