Extensions: [BUG] Multiple thumbnails folder creation (critical)

[marvinsxtr]

[READ] Step 1: Are you in the right place?

yes

[REQUIRED] Step 2: Describe your configuration

• Extension name: storage-resize-images

[REQUIRED] Step 3: Describe the problem

When uploading an image to the storage bucket, seemingly infinite thumbnails folders are created, which exceeded my quota for SocketConnectNonbillable within seconds.

Steps to reproduce:

Upload an image to storage bucket.

Expected result

Image should be resized inside one lowRes folder.

Actual result

The folder was created multiple times, each containing the resized image, which probably lead to my quota exceeding.

Logs i managed to get before my quota was exceeded:
(shortened paths with ... in order to make it more readable)

Function execution started
Billing account not configured. External network is not accessible and quotas are severely limited. Configure billing account to remove these restrictions
Started execution of extension with configuration { bucket: '[...]', cacheControlHeader: undefined, imageSizes: [ '200x200' ], resizedImagesPath: 'lowRes' }
Creating temporary directory: '/tmp/[...]/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes'
Created temporary directory: '/tmp/[...]/lowRes/.../lowRes'
Downloading image file: '[...]/lowRes.../lowRes/_200x200'
Downloaded image file: '[...]/lowRes/.../lowRes/_200x200' to '/tmp/[...]/lowRes/.../lowRes/_200x200'
Resizing image at path '/tmp/_200x200' to size: 200x200
Resized image created at '/tmp/_200x200'
Uploading resized image to '[...]/lowRes/.../lowRes/_200x200'
Uploaded resized image to '[...]/lowRes/.../lowRes/_200x200'
Deleting temporary resized file: '[...]/lowRes/.../lowRes/_200x200'
Deleted temporary resized file: '[...]/lowRes/.../lowRes/_200x200'
Completed execution of extension
Deleting temporary original file: '[...]/lowRes/.../lowRes/_200x200'
Deleted temporary original file: '[...]/lowRes/.../lowRes/_200x200'
Function execution took 465 ms, finished with status: 'ok'
Error: quota exceeded (Quota exceeded for quota group 'SocketConnectNonbillable' and limit 'CLIENT_PROJECT-100s' of service 'cloudfunctions.googleapis.com' for consumer 'project_number:[...]'.);

In conclusion, the cloud function was called ~500 times which can be seen in my statistics.
(I never used cloud functions before and only tried to upload an image two times)
Later i also got this error, probably as a result:

Error when resizing image { Error: The maximum object length is 1024 characters, but got a name with 1025 characters: ''[...]/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRe...'' at Util.parseHttpRespBody (/srv/node_modules/@google-cloud/common/build/src/util.js:191:38) at Util.handleResp (/srv/node_modules/@google-cloud/common/build/src/util.js:135:117) at retryRequest (/srv/node_modules/@google-cloud/common/build/src/util.js:423:22) at onResponse (/srv/node_modules/retry-request/index.js:206:7) at /srv/node_modules/teeny-request/build/src/index.js:158:17 at <anonymous> at process._tickDomainCallback (internal/process/next_tick.js:229:7) code: 400, errors: [ { domain: 'global', reason: 'invalid', message: 'The maximum object length is 1024 characters, but got a name with 1025 characters: \'\'[...]/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRe...\'\'' } ], response: undefined, message: 'The maximum object length is 1024 characters, but got a name with 1025 characters: \'\'[...]/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRe...\'\'' }

[Salakar]

Hey @marvinsxtr, looks like this is the regex failing here to pickup that it's already a resized image.

I think we can look at removing this regex and instead use a custom metadata field to infer that an image has already been resized.

[codediodeio]

It looks like this issue is caused when an image file is uploaded without an extension. The isResizedImage should be be expanded to check for the extension, or the function should auto-assign an extension if missing, or as @Salakar mentioned a custom metadata field may be the best option.

Here is a test case for the current implementation. In prod, the result is an infinite loop that uploads resized images with an empty filename. This will cause runaway billing if not caught early.

const path = require('path')

const suffixRegex = /_\d+x\d+$/;
const isResizedImageValidator = (fileName) => suffixRegex.test(fileName);

const filePath = 'thumbs/someImageWithoutExt_200x200' 
const fileName = path.basename(filePath);
const fileExtension = path.extname(filePath); // ""
const fileNameWithoutExtension = fileName.slice(0, -fileExtension.length);  // ""
const isResizedImage = isResizedImageValidator(fileNameWithoutExtension);

console.log(isResizedImage) // false :(

[laurenzlong]

Hi @codediodeio what do you mean by "image file is uploaded without an extension"? I want to make sure our fix that is in flight is going to solve this issue, and i'm having trouble reproducing the issue in the first place.

@marvinsxtr can you tell me exactly what your installation parameters are? What did you put for "Cloud Storage path for resized images", and what is the format and name of the image you then uploaded? Was it uploaded to the root of the bucket or a sub folder, if so, what was the name of the sub folder? As much detail as possible would be appreciated. Thanks!

[michel-p]

@laurenzlong Hello LaurenWiong, just stumbled upon the exact same issue.
Extension parameters:
Deployment location : europe-west2
Cloud Storage bucket for imagesDescription : comando-78b26.appspot.com
Sizes of resized imagesDescription : 300x300,600x450
Cloud Storage path for resized images (Optional): thumbnails
Cache-Control header for resized images (Optional): Parameter not set

Name of images set without the MYME type at the end like follow ("profile-USERID") :

```// Update images
const storageRef = storage().ref();
let storedProfileImageUrl = "";
let storedCoverImageUrl = "";

if (profilePicture && !typeof coverPicture === "string") {
  const profileImageRef = storageRef.child(
    `profile-${userID}.${profilePicture.type}`
  );
  const storedProfileImage = yield profileImageRef.put(profilePicture);

  storedProfileImageUrl = yield storedProfileImage.ref.getDownloadURL();
}```

[codediodeio]

I mean when you upload a file like ref('my-image').put(img), where the file mime type is an image, but the path is missing .png.

Setting a metadata property on the results, like use_resizer: false is the best solution IMO. It has the added benefit of allowing developers to opt-out of the extension for specific files.

[laurenzlong]

The fix is now live in v.0.1.1 of the extension. You can update via Firebase Console or CLI by running:
firebase ext:update storage-resize-images --project <projectId>

@codediodeio If you would like to opt out of resizing some images, my suggestion would be to put those images inside of a different bucket. The reason is that if they are in the same bucket as the one that the extension is listening to, a Cloud Function will still be triggered. This will lead to higher quota usage and potential cost.