Shrine: Detect duplicate files

[brunowego]

Maybe you can give me an orientation on how to detect duplicate files. Imagine a use case where the user sends a remittance file and if the file has already been sent the user will be notified.

Maybe save the md5 code of file and compare it when saving. Thanks.

[brunowego]

Trying something like this https://github.com/carrierwaveuploader/carrierwave/wiki/How-to:-Use-file%60s-MD5-as-filename

[brunowego]

Works perfectly!

before_validation do
  self.md5 = Digest::MD5.hexdigest file.read.to_s
end

validates_uniqueness_of :md5

Thanks!

[janko]

Great that you figured it out! However, this solution uses a lot of RAM, because it loads the whole file at once. This will probably not scale well, depending on the size of your files. I would recommend the following:

before_validation do
  digest = Digest::MD5.new
  file.open { |io| digest.update(io.read(16384), buffer ||= "") until io.eof? }
  self.md5 = digest.hexdigest 
end

Now only chunks of files are loaded into memory, which is a memory-friendly way of reading the file. The chunk size is arbitrary, I just copied it from the implementation of Digest::MD5.file. The buffer variable makes it so that each new chunk replaces the old chunk in-place, instead of creating a new string for each chunk.

Also, file.open { ... } will close the underlying opened file after you've calculated the digest. It's important to close file descriptors after using them (that's why Shrine closes files after uploading), but also many storages implement reading the file in terms of a Tempfile, and some Ruby implementations don't properly clean up Tempfiles after their reference is lost (MRI), so not closing the Shrine::UploadedFile could fill your temporary storage with unused downloaded files over time.

[brunowego]

Great! Thanks a lot!