Rocket.chat: RC bug: FileSystem Avatars not available

[soundstorm]

Rocket.Chat Version: 0.57.0.rc-1
Running Instances: 1

Probably this issue exists because of #6788 ?
We used FileSystem as storage type, but it seems that the storage engine was changed to GridFS. Doesn't matter if I switch between both, if I upload a new avatar the file date doesn't alter, so the storage type defaults to GridFS and is not changeable.

[sampaiodiego]

before 0.57-rc we have two different storage types: one just for avatars and another one just for file uploads. but now we just have one configuration for both features, which is the previous option for file uploads. when you upgraded your rocket.chat, your avatars should have been migrated from the FileSystem to the file uploads storage option.

with that said, are you saying you cannot change the file uploads storage type option from GridFS to FileSystem? or are you saying if you change between them the files are kept on the old option?

[soundstorm]

Okay:

So the option where to store the avatars is currently remaining in the admin panel but has no influence on the storage as it is set by the main storage settings? So the deprecated menu is confusing.

When upgrading to the rc the avatars are not migrated. Is there a way to do a automatic migration instead of forcing the users to reupload their images?

They are saved in the same cryptic version as ordinary uploads in e.g. uploads/guTGt472jjHldth instead of avatars/username? And emojis get the same change?

[Majkl578]

After upgrading from 0.55.1-rcX to 0.57.1-rc1 all of our avatars and files now return 404.

Avatar storage type is (was) FileSystem.
File Upload storage type is (was) FileSystem.

After upgrade, it's still FileSystem for both, but actual avatars/files do not load.

[lucasvanhalst]

Had the same problem as @Majkl578 so I guess the migration isn't working properly. Avatars were stored in gridFS, other images on disk.

[soundstorm]

For me they are simply reset to the colored letters.

[Majkl578]

For me they are simply reset to the colored letters.

Same here for avatars, altough they still exist physically in original location.
Files just simply result in a 404.

[Scream85]

My logs are full with this since 0.57.0-rc1, is it related to this problem?

Jun 10 23:46:25 nodejs-RocketChat2[47837]: rocketchat_logger rocketchat_logger.js:377 #033[34mLDAPSync ➔ info#033[39m Syncing user avatar Jun 10 23:46:25 nodejs-RocketChat2[47837]: rocketchat_logger rocketchat_logger.js:377 #033[31mLDAPSync ➔ error#033[39m { [Error: Forbidden [forbidden]] Jun 10 23:46:25 nodejs-RocketChat2[47837]: error: 'forbidden', Jun 10 23:46:25 nodejs-RocketChat2[47837]: reason: 'Forbidden', Jun 10 23:46:25 nodejs-RocketChat2[47837]: details: undefined, Jun 10 23:46:25 nodejs-RocketChat2[47837]: message: 'Forbidden [forbidden]', Jun 10 23:46:25 nodejs-RocketChat2[47837]: errorType: 'Meteor.Error' }

[Scream85]

It seems this LDAP avatar sync problem also breaks the LDAP authentication and login.
After disabling avatar sync, I managed to login again.

[admin-crowd-iugo]

same here with ldap enabled and filesystem for avatar and file uploads. please fix. we have over a 100gb of files.

[soundstorm]

@sampaiodiego is working on this: #7352

[sampaiodiego]

@soundstorm thx for referecing it.

so #7352 should have fixed most of the problems with avatar migration, but unfortunately it does not repair broken migrations already ran. :disappointed:

looks like your rocket.chat is on an undesired state.. so let me ask: are avatar files still on the old configured path? what is your configured file upload storage type (regular files not avatar)?

[sampaiodiego]

@Scream85 can you please fill a new issue describing your LDAP avatar sync issues? I dind't test it on #7352 so it might still be broken

[Majkl578]

Confirmed with RC3, avatars still missing.

[sampaiodiego]

@Majkl578 as I said rc.3 will not fix broken migrations..

since you have said the original files are still on the old location, can you please confirm if you still have the old configs on your database? please run the following on mongo shell:

db['rocketchat_settings'].find({ 
  _id: { $in: ['Accounts_AvatarStoreType', 'Accounts_AvatarStorePath'] }
});

[Majkl578]

root@7b3d2c86df5a:/# mongo 'mongodb://127.0.0.1:27017/rocketchat'
MongoDB shell version v3.4.5
connecting to: mongodb://127.0.0.1:27017/rocketchat
MongoDB server version: 3.4.5
Server has startup warnings: 
2017-06-30T03:34:16.599+0000 I STORAGE  [initandlisten] 
2017-06-30T03:34:16.600+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-06-30T03:34:16.601+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-06-30T03:34:20.625+0000 I CONTROL  [initandlisten] 
2017-06-30T03:34:20.625+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-06-30T03:34:20.625+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2017-06-30T03:34:20.625+0000 I CONTROL  [initandlisten] 
> db['rocketchat_settings'].find({ 
...   _id: { $in: ['Accounts_AvatarStoreType', 'Accounts_AvatarStorePath'] }
... });
{ "_id" : "Accounts_AvatarStorePath", "type" : "string", "enableQuery" : "{\"_id\":\"Accounts_AvatarStoreType\",\"value\":\"FileSystem\"}", "group" : "Accounts", "section" : "Avatar", "packageValue" : "", "valueSource" : "processEnvValue", "hidden" : false, "blocked" : false, "sorter" : 33, "processEnvValue" : "/app/uploads", "i18nLabel" : "Accounts_AvatarStorePath", "i18nDescription" : "Accounts_AvatarStorePath_Description", "ts" : ISODate("2017-06-11T01:10:20.888Z"), "_updatedAt" : ISODate("2017-06-11T01:10:20.888Z"), "createdAt" : ISODate("2016-12-07T01:06:53.515Z"), "value" : "/avatars" }
{ "_id" : "Accounts_AvatarStoreType", "type" : "select", "values" : [ { "key" : "GridFS", "i18nLabel" : "GridFS" }, { "key" : "FileSystem", "i18nLabel" : "FileSystem" } ], "group" : "Accounts", "section" : "Avatar", "packageValue" : "GridFS", "valueSource" : "packageValue", "hidden" : false, "blocked" : false, "sorter" : 32, "i18nLabel" : "Accounts_AvatarStoreType", "i18nDescription" : "Accounts_AvatarStoreType_Description", "ts" : ISODate("2017-06-11T01:10:20.872Z"), "_updatedAt" : ISODate("2017-06-11T01:10:20.872Z"), "createdAt" : ISODate("2016-12-07T01:06:53.497Z"), "value" : "FileSystem" }

[Majkl578]

So you released stable version without actually fixing this... Good job.

[JSzaszvari]

It looks to be fixed in the next minor release.

Instead of being rude and unappreciative, try and add something constructive.

[Majkl578]

Well, I did right in the previous post. I posted the settings I was asked for (for which I had to google how to execute query in mongo etc.) and it was left without any response - can't do more as I don't code in Node/JS. So now what's being rude... ☹️

[geekgonecrazy]

@Majkl578 this is why we strongly caution moving production systems to release candidates. They are unfinished and often contain bugs.

In this case for people coming directly from 0.56.0 to 0.57.0 it seems to work fine. This is why a release went on out.

Its cases where someone tried out 0.57.0-rc.1 / 0.57.0-rc.2 that we're having to work through

[ImageEngineIT]

We have a copy of our production system that went directly from 0.56 -> 0.57 (never installed a release candidate) and has this same issue with LDAP avatars. Logins are also broken until the "Sync User Avatar" option is set to false.

Is there a manual query or other work around we can use to fix the issue?

[Scream85]

@sampaiodiego I made the mistake to go from 0.56.0 to 0.57-rc1 with a prod system with 280 users and in use 24 hrs a day. I'm still running 0.57-rc1. So doing any test other than applying a working fix is currently not possible, unfortunately. Is there anything else I can do which won't require downtime and possibly won't break anything else? Thank you!

[sampaiodiego]

@Scream85 we just released a patch last night which should fix any avatar issue caused either by an rc or 0.57.0 version. please give it a try.

if you were using file system as your avatar storage option, you have to set an environmente variable AVATARS_PATH with the path you used before you update.

@Majkl578 we're sorry for that.. please update your server to 0.57.1, since you have the previous settings still on DB, it should be just a matter of update. If anything goes wrong, you can try restarting with the folowing env vars set: AVATARS_PATH=/avatars (I got /avatars path from the settings you posted before) and MIGRATION_VERSION=99,rerun

[cbanal]

Installed 0.57.1 on a test machine with a snapshot of the mongo database from when 0.56 was installed and ldap avatars are still missing.

[soundstorm]

@sampaiodiego worked for me - at least for FS migration. Seems like LDAP is another construction side?

[sampaiodiego]

yes @soundstorm , LDAP is a different issue. thx