Bfg-repo-cleaner: Convert to git lfs by size

[dpopiashvili]

Hi

I wonder if there's a way to convert binary files to git lfs by size and not extension. I couldn't find anything in help section. something like --strip-blobs-bigger but instead of stripping it from history, to convert them to git lfs.

Thanks

[javabrett]

You might want to review how Git LFS tracks files and replaces them with pointers, then later resolves those pointers to the original large file. That is all done with filters, and the filters are triggered by file-pattern, not size.

So I think it makes little sense to convert-by-size, since you won't then have consistent file-name based filters for Git LFS to operate-on, but you might be able to explain how you hope to use this.

I think you would be better-off running analysis to find the location and filename patterns of your large files, then apply those patterns.

[dpopiashvili]

Hi javabrett.

Thank you for your reply. I'm not sure if I understand it correctly, you suggest that we do an analyzes first to find large files that we want to convert to git-lfs. If so, I have 2 questions:

1. is it possible to provide a list of files to bfg --convert-to-git-lfs instead of file extensions?
2. if it is, is it possible to list files larger than n MB by BFG? I couldn't find anything in command line output, only stip files larger than n MB...

Thank you again :)

[TobiahZ]

I believe you pass into bfg --convert-to-git-lfs the same line you would put into a .gitattributes file.

Which is to say, often a "*.", but could also be a certain folder, or even a specific file in a specific folder.

If you had a list of certain files, you could make a script that does bfg --convert-to-git-lfs 'path/to/file.type' for each one.

(Unless I'm mistaken on how it works)

[javabrett]

My point was:

• Git LFS uses .gitattributes files to apply filters to export/import large files from side-storage.
• .gitattributes supports certain file patterns etc., but not files-by-size.
• Therefore, unless all your files that would be picked-out by a size-filter happen to match a location/name-matching spec, you don't want to export-by-size.

So you are better off locating the pattern for large files, and converting based on that.

[estyrke]

Actually, there seems to be a size check already, which affects me in the opposite manner: I have *.png in my LFS patterns, but some PNG files are smaller than 512 bytes, which means BFG will ignore them during conversion. This gives me trouble later when git-lfs expects these files to be LFS pointers when they are actually still real files.

To be clear, I agree that converting to LFS by size is a bad idea since there is no way to match this with the git attributes in the resulting repository. But making this 512 byte limit configurable would solve both my and @dpopiashvili 's problems (although for my part, I'd really prefer to get rid of the limit entirely).