Gitea: LFS: file is undownloadable with EOF error, `git lfs push --all origin master` from source repo (where the full file is still present) doesn't fix it

[etc0de]

• Gitea version (or commit ref): 9e627af built with: bindata, sqlite
• Git version: 2.11.2
• Operating system: whatever the default docker image uses
• Database (use [x]):
  
  
  
  • [X] PostgreSQL
  
  
  • [ ] MySQL
  
  
  • [ ] MSSQL
  
  
  • [ ] SQLite
  
  
• Can you reproduce the bug at https://try.gitea.io:
  
  
  
  • [ ] Yes (provide example URL)
  
  
  • [X] No - don't know how to reproduce!
  
  
  • [ ] Not relevant
  
  
• Log gist:

Description

I have an LFS file which cannot be downloaded: every pull that involves it will give an EOF error as shown below. git lfs push --all origin master from source repo (where the full file is still present) doesn't fix it, the lfs push doesn't return any error but the problem simply doesn't go away.

[etc0de]

Also please note that the git-lfs client is built to absolutely rely on the server to have successfully stored the file when it gets a positive HTTP response at the end of the upload. Therefore if gitea's LFS server doesn't do this yet, it would probably be a good idea to have some sort of hook verify that the whole file arrived before sending a positive HTTP response code. (There is also the "verify" protocol feature I believe where you can require the client to check on some URL itself if the whole file is available)

[threadproc]

+1 on this being required, using Git LFS for any number of large files frequently results in them not being able to be pulled anymore.

@JonasT have you found a way to re-upload the files manually in this circumstance?

[technoweenie]

Hey :wave:, I'm one of the Git LFS authors. I don't know how Gitea works, but the Git LFS client will attempt to verify uploads if the server desires.

https://github.com/git-lfs/git-lfs/blob/master/docs/api/basic-transfers.md#verification

This is useful in cases where an LFS server uses external cloud storage (like S3) without any proxying. By adding a verify action to the Batch API response, Git LFS can make another call to the LFS server, giving it a chance to verify the file has been successfully uploaded.

If you can delete the file from the server, you can then run git lfs push --all on a clone with a valid .git/lfs/objects/03/4c/034c77f1f05ec89421e4a63f0e3a4ca1ecf852cc6d2bf611f126f275728e017d file to re-upload to the server.

[threadproc]

@technoweenie Interestingly enough, I've now deleted and repushed the particular file that I'm running into issues with to the server and I'm still getting the same error. I actually reuploaded all of the LFS objects, and it's still the same file. I've checked the MD5 sum of all of the files (the original file, the file in .git/lfs/objects/{OID_PATH} on the original repo, the LFS object on the server, and the incomplete file on the broken clone .git/lfs/objects/incomplete/{OID}.tmp) and they all actually match. I'm going to poke around some more and see why it might be getting an EOF error on this.

I will also look into adding the verify action, thanks for that hint!

Edit: just did a fresh clone, md5sums still match and now it's giving me several files that aren't quite right. I'll update with anything new I find out while I poke at this.

[lafriks]

@technoweenie is it common behaviour for LFS servers to return verify in response even if LFS files are stored localy on server?
Just for info gitea LFS implementation is mostly based on LFS test server code

[threadproc]

Submitted a PR that fixes this issue for me. The problem was that it was returning a 206 response (Partial Content) but providing the full file size as the Content-Length instead of the truncated size.

[technoweenie]

@lafriks I don't know. GitHub.com currently uses verify, while GitHub Enterprise does not (due to a custom upload server). So, we test both.

If you're comfortable that your LFS server can verify that the _exact_ contents have been saved to the storage layer, adding the verify action will slow things down. But, I suppose double checking doesn't hurt much :) This only happens the first time a unique LFS OID is uploaded, so it's not a big perf hit.

[lafriks]

@techknowlogick thanks for your answer. So probably we need to add that verify until we implement proper server-side verification

[techknowlogick]

lol. wrong tech

ping @technoweenie

[lafriks]

@techknowlogick sorry :)

[lafriks]

@technoweenie what status should /verify handler return if verification fails and should it remove lfs object from server store if for example size differs?

[technoweenie]

I'd say a 422. LFS will follow any 3xx redirects, and treats 4xx and 5xx statuses as errors. A 4xx error is more appropriate in this case, as a 5xx would imply some kind of server issue (DB down, disk full, etc).