Zulip: KeyError after deleting user (result.append((recipient.id, [user_profiles[recipient.type_id]])))

[7lym]

Hello!
Running on-premise, DigitalOcean-stock Zulip droplet, version 2.1.3.
I wanted to delete(not deactivate) a user and followed the instructions from here, that means running:

./manage.py shell with UserProfile.objects.get(email="[email protected]").delete()

The user was succesfully deleted, here is the shell output

(94,
 {'zerver.UserProfile_groups': 0,
  'zerver.UserProfile_user_permissions': 0,
  'zerver.UserGroupMembership': 0,
  'zerver.EmailChangeStatus': 0,
  'zerver.PushDeviceToken': 0,
  'zerver.MutedTopic': 0,
  'zerver.SubMessage': 0,
  'zerver.Reaction': 0,
  'zerver.UserMessage': 35,
  'zerver.Attachment_messages': 1,
  'zerver.MissedMessageEmailAddress': 0,
  'zerver.ArchivedSubMessage': 0,
  'zerver.ArchivedReaction': 0,
  'zerver.ArchivedUserMessage': 0,
  'zerver.Subscription': 3,
  'zerver.UserActivity': 28,
  'zerver.UserActivityInterval': 6,
  'zerver.UserPresence': 2,
  'zerver.UserStatus': 0,
  'zerver.ScheduledEmail_users': 1,
  'zerver.ScheduledMessage': 0,
  'zerver.RealmAuditLog': 4,
  'zerver.UserHotspot': 2,
  'zerver.CustomProfileFieldValue': 0,
  'zerver.Service': 0,
  'zerver.BotStorageData': 0,
  'zerver.BotConfigData': 0,
  'social_django.UserSocialAuth': 0,
  'otp_totp.TOTPDevice': 0,
  'two_factor.PhoneDevice': 0,
  'analytics.UserCount': 7,
  'zerver.Message': 3,
  'zerver.Attachment': 1,
  'zerver.UserProfile': 1})

Following this memcached related issue, i executed ./scripts/setup/flush-memcached and the user seemed completely erased from the UI too.

However, i experienced problems:

At a close inspection, it seems that this API call

Request info:
- path: /json/messages
- GET: {'anchor': ['204'], 'num_before': ['200'], 'num_after': ['200'], 'client_gravatar': ['true']}
- QUERY_STRING: "['anchor=******&num_before=******&num_after=******&client_gravatar=******"
- SERVER_NAME: "['']"

results in internal server error:

Logger root, from module zerver.middleware line 291:

Traceback (most recent call last):
  File "/srv/zulip-venv-cache/6176a52fab8737c09656fc8ad0352547e9801c07/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "./zerver/lib/rest.py", line 35, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/srv/zulip-venv-cache/6176a52fab8737c09656fc8ad0352547e9801c07/zulip-py3-venv/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "./zerver/lib/rest.py", line 172, in rest_dispatch
    return target_function(request, **kwargs)
  File "/srv/zulip-venv-cache/6176a52fab8737c09656fc8ad0352547e9801c07/zulip-py3-venv/lib/python3.6/site-packages/django/utils/decorators.py", line 149, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "./zerver/decorator.py", line 710, in _wrapped_view_func
    return authenticate_log_and_execute_json(request, view_func, *args, **kwargs)
  File "./zerver/decorator.py", line 689, in authenticate_log_and_execute_json
    return limited_view_func(request, user_profile, *args, **kwargs)
  File "./zerver/decorator.py", line 829, in wrapped_func
    return func(request, *args, **kwargs)
  File "./zerver/lib/request.py", line 367, in _wrapped_view_func
    return view_func(request, *args, **kwargs)
  File "./zerver/views/messages.py", line 927, in get_messages_backend
    allow_edit_history=user_profile.realm.allow_edit_history,
  File "./zerver/lib/message.py", line 115, in messages_for_ids
    MessageDict.post_process_dicts(message_list, apply_markdown, client_gravatar)
  File "./zerver/lib/message.py", line 194, in post_process_dicts
    MessageDict.bulk_hydrate_recipient_info(objs)
  File "./zerver/lib/message.py", line 485, in bulk_hydrate_recipient_info
    display_recipients = bulk_fetch_display_recipients(recipient_tuples)
  File "./zerver/lib/display_recipient.py", line 165, in bulk_fetch_display_recipients
    cache_transformer=personal_and_huddle_cache_transformer
  File "./zerver/lib/cache.py", line 395, in generic_bulk_cached_fetch
    db_objects = query_function(needed_ids)
  File "./zerver/lib/display_recipient.py", line 135, in personal_and_huddle_query_function
    result.append((recipient.id, [user_profiles[recipient.type_id]]))
KeyError: 15

I tried everything, from restarting services to clearing queues, restarting workers and finally rebooting the whole VM, nothing helped.
Digging into the db, i discovered that the deleted user had id=15 (exactly as the number in the error above). I have located this number in a row in the table named zerver_recipient

but in order to delete it, i had to first delete all entries in the zerver_message table that had recipiend_id = 20

which also meant that i first had to delete all entries from the table zerver_usermessage that had their message_id equal with the primary keys listed above 😭

Doing all these steps helped, as the API call is returning 200 now, and the server seems to run fine. But the problem is(besides the fact that _deleting a user does not correctly cascade via models, in my opinion_) that the browser console now outputs:

clearly in relation to the deleted account with id=15 and i really have no idea how to get rid of any deletion leftovers.
Any help would be appreciated :)

[7lym]

Seems like the user id is embedded in zerver_message entries that have the subject field set to signups, in the form @_**user_name|user_id**:

removing these rows resolves the frontend problem, the browser console is now clean.

However, some leftovers still remain (e.g. there are zerver_message entries about a user signing up or accepting the invitation, there might be even more cases).

Is there a clean way to remove them from the feed? Thanks!

[timabbott]

I think the browser console issue is fixed in master; we no longer assume that user IDs mentioned in the syntax still exist in that code path.

The traceback indeed has to do with an undeleted Recipient object for a private message that had been sent to that user. If you'd deleted the Recipient object from the Django manage.py shell, it would have taken care of the cascading for you.

Those two are the only complications that'll result in errors I'm aware of here. There's no clean way to do this; the idea of making one is precisely the reason why #10533 is still open :). The good news for you is that we're planning to resolve that issue in the somewhat near future.

@mateuszmandera FYI; mostly this is a reminder that when we implement a do_delete_user, we should update #10533 to recommend it. I don't think we'd remove notifications about signups when a user is deleted (etc.), as that information might still be useful after the user is gone. Though you could of course script it yourself in your own tooling.

[7lym]

Oke, nice .. maybe it would make sense to update #10533 with this information?

The traceback indeed has to do with an undeleted Recipient object for a private message that had been sent to that user. If you'd deleted the Recipient object from the Django manage.py shell, it would have taken care of the cascading for you.

Because in the current state, it literally causes disruption, as stated above, so one MUST delete Recipient objects in order for things to work again :)

[timabbott]

Added a note to #10533, thanks for the reminder (I'd meant to do it yesterday).