Three.js: Suggestion: use git lfs to store binary files

[garyo]

There's a bunch of large binary files checked into the three.js repo (examples mostly). It would help keep the repo trim if the maintainers used git lfs (https://git-lfs.github.com/) to store large binaries -- it's much more efficient, because when cloning, you don't have to pull down all versions of those binaries, only the most recent. Also you don't run into github's pack size limitations with lfs files.

I'm not suggesting rewriting all of three.js's history to LFSize all the existing binaries, btw -- that can be done but it's pretty invasive and changes all the SHAs. Just do it going forward.

One objection is that users have to have git lfs installed, so it's an extra dependency. But IMHO the benefits outweigh that.

[gkjohnson]

I think one main benefit to storing all the example files in the main repo is that it lets us preview examples from PRs using githack, which I don't think would work immediately when using git lfs: https://raw.githack.com/mrdoob/three.js/dev/examples/webgl2_loader_gltf.html

Maybe there are other options, though?

[garyo]

I don't know if githack.com has a problem with git-lfs; I'd expect it to be fine, since it's just proxying the requests over to github. But indeed one would have to test, and you're right that there's real value in having the assets right there.

[mrdoob]

On top of that. I heard bad experiences when moving things to git lfs (repos breaking) from @jsantell. So I'm suuuuper hesitant about it. More clarity will definitely help.

[garyo]

It's up to you of course. I've used it with no problems for years, in consulting gigs and for my company's work, but YMMV as they say.

[garyo]

One note: github does enforce strict limits on file and pack sizes (non-LFS single file size < 100MB, repo size recommended <1GB, 2GB hard limit), which have bitten me before. Three.js is currently around 690MB, and the largest single file is examples/models/gltf/MetalRoughSpheres/glTF-Embedded/MetalRoughSpheres.gltf, at 15MB.

[jsantell]

On top of that. I heard bad experiences when moving things to git lfs (repos breaking) from @jsantell. So I'm suuuuper hesitant about it. More clarity will definitely help.

My issue was, similar to this scenario, adding git+lfs not from the begining. I wanted to blanket select all assets, lets say ".gltf", moving forward, to use git+lfs via .gitattributes. This resulted in existing glTFs being in limbo. I think there's a way to properly deregister and reregister under git+lfs. I don't think that retroactively does anything with the non-LFS'd versions of the assets however. I'll have to find my post mortem, because I definitely broke a repo -- but maybe some git+lfs experts know a way of handling this nicely retroactively

[garyo]

@jsantell makes a good point -- it has to be done carefully. There are two ways: (1) don't rewrite history, and keep the old versions of binaries in the repo, or (2) rewrite all and push to new repo. (2) is cleaner, but three.js has too many forks for it to make sense, so doing (1) would be the only sensible way IMHO. The process for (1) is to enable LFS for all desired file types, commit, then remove and re-add all the binaries to "LFS-ize" them and commit and push. Otherwise as you say, you get in a weird state. (You can always back out though.)

[mrdoob]

I don't know if githack.com has a problem with git-lfs; I'd expect it to be fine, since it's just proxying the requests over to github. But indeed one would have to test, and you're right that there's real value in having the assets right there.

We should get some clarity on that first.

[garyo]

It seems githack.com does _not_ work with git lfs, at least at first glance. Here's a test repo with an LFS file: https://github.com/Schoonology/git-lfs-test - the file binary.jpg is stored with LFS. If I understand how githack is supposed to work, I should be able to wget that file straight from githack.com:

wget https://raw.githack.com/Schoonology/git-lfs-test/master/binary.jpg

But that just returns the pointer file:

% head binary.jpg
version https://git-lfs.github.com/spec/v1
oid sha256:6fe2e48ef494b7855e6d8f93c59cf38d38350274e304a9ac35102d5c365ee002
size 620773

which is not good.

Downloading straight from github raw works fine:

% wget https://github.com/Schoonology/git-lfs-test/raw/master/binary.jpg
% file binary.jpg.2
binary.jpg.2: JPEG image data, JFIF standard 1.01

but downloading using raw.githubusercontent.com (which may be where githack.com is proxying from?) does not:

% wget https://raw.githubusercontent.com/Schoonology/git-lfs-test/master/binary.jpg 
% file binary.jpg.3
binary.jpg.3: ASCII text

So it would appear that at this point, if githack.com is important, then LFS won't work.
In the future, if three.js moved to publishing CI builds, those would of course work fine.

[looeee]

githack is a based on the now defunct rawgit, and there's an old issue about lfs here:

https://github.com/rgrove/rawgit/issues/128

You could try opening a new issue about this on githack's repo:

https://github.com/neoascetic/rawgithack/issues

[garyo]

Issue opened: https://github.com/neoascetic/rawgithack/issues/38

[mrdoob]

Lets close this issue for now...