Rocket.chat: Avatar migration still not working properly

Created on 2 Jul 2017 · 33Comments · Source: RocketChat/Rocket.Chat

The improved avatar migration (#7352) still isn't working properly, and the error messages aren't really saying much. See the log here: https://pastebin.com/kN7F0Npz
Those avatars are currently stored in the database so I assume it's trying to migrate them to place them on disk but it's not working. I'm using the docker image from dockerhub if that makes a difference. The migration is also marked as being executed successfully while it's clearly not.

Rocket.Chat Version: 010a60ac7d1e550e0323c037b7006f7336821600
Running Instances: 1
DB Replicaset OpLog: true
Node Version: 4.6

docker-compose.yml:

services:
  mongo:
    image: mongo
    restart: unless-stopped
    volumes:
      - ./data/runtime/db:/data/db
    command: mongod --smallfiles --oplogSize 128 --replSet rs0
  rocketchat:
    image: rocketchat/rocket.chat:develop
    restart: unless-stopped
    volumes:
      - ./uploads:/app/uploads
    environment:
      - PORT=3000
      - ROOT_URL=https://example.com
      - MONGO_URL=mongodb://mongo:27017/rocketchat
      - MONGO_OPLOG_URL=mongodb://mongo:27017/local
      - DDP_DEFAULT_CONNECTION_URL=https://example.com
    ports:
      - 3000:3000
    depends_on:
      - mongo

release candidate bug

Source

lucasvanhalst

All 33 comments

Probably related to #7172

MartinSchoeler on 3 Jul 2017

I'm using the docker rocketchat image, and and i just had a failed migration.
I tried to rerun the migration by lowering the migration version in the db, but it skipped it saying the migration had already ran.

Newly uploaded avatars appear to work, but none of the old ones do.

My log file looked similar to the above one.

My avatar storage was gridfs, my upload storage s3.

khobbits on 4 Jul 2017

@lucasvanhalst / @khobbits had you guys tried any release candidate prior to rc.3 ? If so your migrations are in a partially functioning state. Please see #7172 where trying to walk through and get instances stuck in a limbo state repaired.

geekgonecrazy on 4 Jul 2017

No, no rc builds here. Upgraded from .56 to .57.
I'll look though that thread tomorrow.

On 4 Jul 2017 22:44, "Aaron Ogle" notifications@github.com wrote:

@lucasvanhalst https://github.com/lucasvanhalst / @khobbits
https://github.com/khobbits had you guys tried any release candidate
prior to rc.3 ? If so your migrations are in a partially functioning state.
Please see #7172 https://github.com/RocketChat/Rocket.Chat/issues/7172
where trying to walk through and get instances stuck repaired.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/RocketChat/Rocket.Chat/issues/7385#issuecomment-312958933,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAaLC9lgedFPUPYzMJ0dMOni9xBOdKN6ks5sKrJFgaJpZM4OLrc2
.

khobbits on 4 Jul 2017

We don't use docker and hat 5 instances set up on a single 6 core server.
After upgrading from 0.56.0 to 0.57.0 we encountered the problem that people are not able to login via LDAP when the avatars will be synced. We disabled the avatar sync but now some, not all, avatars wen't missing.

In version 0.56.0 we had the problem that avatar went missing random too. We just clean the the avatarchunks from the database (db.avatars.chunks.remove({})) and resynced.
Now the avatars seem to be in another collection (db.rocketchat_avatars) with only 79 entires (we got 104 user total).

KervyN on 5 Jul 2017

guys we just released 0.57.1 which should fix any avatar migration issue.

if you were using file system as your avatar storage previous the update, you'll need to set a new environmente variable on first restart to be able to fix the issue. set an env var: AVATARS_PATH with the path used for avatar storage.

sampaiodiego on 6 Jul 2017

Does anyone know what will happen if multiple instances have different versions?

Edit: comparing the 0.57.1 and 0.57.2 milestones I think I will wait a bit. o.o

KervyN on 6 Jul 2017

I can confirm that upgrading from 0.57.0 to 0.57.1 does fix my avatars.
(after previously having a broken 0.56.0 -> 0.57.0 migration using gridfs->s3)

khobbits on 6 Jul 2017

🎉1

I can also confirm 0.57.1 sort off works, but not completely. I have 58 users, it migrated 48 avatars. Those remaining 10 users now have a blank avatar. Requests for the avatar return status code 200 but with no content.

Whatever, not gonna waste any more time on this, they'll just have to upload a new one.

lucasvanhalst on 6 Jul 2017

@sampaiodiego 0.57.1 had also resolved my issues with migrating avatars. But its @lucasvanhalst mentioned, 2 of 12 users are now missing their avatar. There are no errors in the log after first start of the instance and running the migration.

TheReal1604 on 7 Jul 2017

@sampaiodiego upgraded another env, it seems that there is an issue with jpegs?

2017-07-11T07:09:02.750535783Z { [Error: Command failed: convert: no decode delegate for this image format `' @ error/constitute.c/ReadImage/501.
2017-07-11T07:09:02.750535783Z convert: no images defined `jpeg:/tmp/ufs/EdcrMCcim46yB9L5v' @ error/convert.c/ConvertImageCommand/3210.
2017-07-11T07:09:02.750535783Z ] code: 1, signal: null }

TheReal1604 on 11 Jul 2017

@TheReal1604 are you getting this on regular uploads or just during the migration?

sampaiodiego on 11 Jul 2017

Doesn't tested uploads, this one was during the migration. @sampaiodiego

TheReal1604 on 11 Jul 2017

we have a fresh install of 0.57.1 and the login problem persists - meaning: as soon as we have the avatar sync option turned on, users cannot log in. users that are already logged in get all avatars displayed as they are stored in the LDAP directory. But logging out such a client and trying to re-login fails.
From the logs:

LDAPSync ➔ info Syncing user avatar

After that, no more info on this login process.

Any ideas on that?

herste on 11 Jul 2017

@herste please see #7472

sampaiodiego on 12 Jul 2017

@TheReal1604 which was the avatar file storage and the file upload storage?

sampaiodiego on 12 Jul 2017

@sampaiodiego @TheReal1604 - we have the same problem while upgrading from 0.56.0 to 0.57.1 - we see the same errors like
Command failed: convert: no decode delegate for this image format
for some users. We use GridFS

I have reverted back to 0.56.0 until this is solved.

danpospisil on 13 Jul 2017

👍1

@danpospisil The problem seams to be related with your imagemagick

Which OS are you using?

Can you run identify -list format and check if JPEG is in the list?

rodrigok on 14 Jul 2017

@rodrigok - I am using the official Docker image for running Rocket.Chat and I see JPEG there:

     JPEG* JPEG      rw-   Joint Photographic Experts Group JFIF format (62)
      JPG* JPEG      rw-   Joint Photographic Experts Group JFIF format (62)

danpospisil on 17 Jul 2017

Can confirm, using the official Docker image in 0.57.1 (rocketchat/rocket.chat:0.57.1), and after having upgraded from 0.56.0 to 0.57.0 to 0.57.1, the problem of avatars migration is not solved.

{"line":"147","file":"rocketchat_migrations.js","message":"Migrations: Migrating from version 98 -> 99","time":{"$date":1500904312651},"level":"info"}
{"line":"147","file":"rocketchat_migrations.js","message":"Migrations: Running up() on version 99","time":{"$date":1500904312659},"level":"info"}
[AVATAR] Migrating avatars. This might take a while.
{"line":"147","file":"rocketchat_migrations.js","message":"Migrations: Finished migrating.","time":{"$date":1500904312700},"level":"info"}
Not migrating, control is locked. Attempt 1/30. Trying again in 10 seconds.
{"line":"147","file":"rocketchat_migrations.js","message":"Migrations: Not migrating, already at version 99","time":{"$date":1500904322721},"level":"info"}
Updating process.env.MAIL_URL
Using FileSystem for custom sounds storage
Using FileSystem for custom emoji storage
ufs: temp directory created at "/tmp/ufs"
➔ System ➔ startup
➔ +----------------------------------------------------+
➔ |                   SERVER RUNNING                   |
➔ +----------------------------------------------------+
➔ |                                                    |
➔ |  Rocket.Chat Version: 0.57.1                       |
➔ |       NodeJS Version: 4.8.1 - x64                  |
➔ |             Platform: linux                        |
[...]
➔ |          Commit Hash: 0c0fa2a8af                   |
➔ |        Commit Branch: HEAD                         |
[...]
[AVATAR] Total users to migrate avatars -> 1
[AVATAR] Migrating 1 of 1

Horgix on 24 Jul 2017

👍1

@Horgix please try 0.57.2 as it contains an additional bug fix specifically for avatars

geekgonecrazy on 24 Jul 2017

@geekgonecrazy : just updated to 0.57.2, and it looks like it actually made it worse:

{"line":"147","file":"rocketchat_migrations.js","message":"Migrations: Not migrating, already at version 99","time":{"$date":1500965564043},"level":"info"}

So nothing has been migrated anyway, but now it's not even possible to update avatars:

 [Error: FileNotFound: no file with id yG2Q9ZmgvFmF4yz64 found]
 ufs: cannot write file "yG2Q9ZmgvFmF4yz64" (EACCES: permission denied, open '/yG2Q9ZmgvFmF4yz64') { [Error: EACCES: permission denied, open '/yG2Q9ZmgvFmF4yz64']
   errno: -13,
   code: 'EACCES',
   syscall: 'open',
   path: '/yG2Q9ZmgvFmF4yz64' }
 ufs: cannot delete temp file "/tmp/ufs/yG2Q9ZmgvFmF4yz64" (ENOENT: no such file or directory, unlink '/tmp/ufs/yG2Q9ZmgvFmF4yz64')

No idea why it's not trying to look for files directly in /, but it didn't do that before and I have trouble understanding why user avatars would ever be stored in the root directory

Horgix on 25 Jul 2017

@sampaiodiego @rodrigok you guys have ideas here?

geekgonecrazy on 26 Jul 2017

@Horgix are you using filesystem to store avatars? Can you show me the configuration?

rodrigok on 27 Jul 2017

@rodrigok yes I am!

I'm using the official Docker image with the default configuration that is provided inside it. For reference, here is the docker-compose part that I'm using to run the container:

services:
  rocketchat:
    image:          rocketchat/rocket.chat:0.57.2
    restart:        unless-stopped
    volumes:
      - /srv/docker_volumes/rocketchat-uploads:/app/uploads
    environment:
      - PORT=3000
      - ROOT_URL=<my_root_url>
      - MONGO_URL=mongodb://mongo:27017/rocketchat
      - MONGO_OPLOG_URL=mongodb://mongo:27017/local
    depends_on:
      - mongo
    labels:
      traefik.docker.network: "traefik_default"
      traefik.port: "3000"
      traefik.backend: "rocketchat"
      traefik.frontend.rule: "Host:<my_root_url>"
    networks:
      - default
      - traefik

Horgix on 1 Aug 2017

We just updated our instance from 0.56 to 0.57.2. I can see some migration steps taking place in the logs. But afterwards for users who had an uploaded avatar it shows a blank image:

Startup Logs: https://pastebin.com/raw/vagNu3sH

bbrauns on 7 Aug 2017

@rodrigok - any update on this?

danpospisil on 21 Aug 2017

using avatars in LDAP seems to be still broken on 0.58.2. Whenever the avatar sync option is enabled, users cant login anymore. Anything we can do to help fixing this?

herste on 2 Sep 2017

@herste For this particular issue check https://github.com/RocketChat/Rocket.Chat/issues/7773

KervyN on 4 Sep 2017

It looks like that it works already.
There was a problem upgrading from 0.56 to 0.58.2, but upgrading from 0.58.2 to 0.58.4 did not give any error.