Kolibri: Sync does not work after deprovision

[intelliant01]

Observed behavior

Tried a manage sync after manage deprovision and am consistently getting this error -

$ kolibri manage sync --baseurl http://192.168.2.104 --username admin --password password --facility 33126ed37618e68b541368ff7828035c --no-push
WARNING:root:No C Extensions available for this platform.

/usr/lib/python3/dist-packages/kolibri/dist/rest_framework/utils/serializer_helpers.py:107: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if value is None or value is '':
INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
WARNING  Redis is configured without a maximum memory policy. Using Redis with Kolibri, the following is suggested: maxmemory-policy allkeys-lru
WARNING  Redis is configured without a maximum memory size.
WARNING  Problematic Redis settings detected, please see Redis configuration documentation for details: https://redis.io/topics/config
INFO     Invoking command sync --baseurl http://192.168.2.104 --username admin --password password --facility 33126ed37618e68b541368ff7828035c --no-push
INFO     Attempting connections to variations of the URL: http://192.168.2.104
INFO     Attempting connection to: http://192.168.2.104/
INFO     Unable to connect: HTTPConnectionPool(host='192.168.2.104', port=80): Max retries exceeded with url: /api/public/info/ (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0xffffa2570eb0>, 'Connection to 192.168.2.104 timed out. (connect timeout=5)'))
INFO     Attempting connection to: http://192.168.2.104:8080/
INFO     Success! We connected to: http://192.168.2.104:8080/api/public/info/
INFO     Syncing has been initiated (this may take a while)...
INFO     Creating pull transfer session
INFO     There are no records to transfer
INFO     Completed pull transfer session
Traceback (most recent call last):
  File "/usr/bin/kolibri", line 11, in <module>
    load_entry_point('kolibri==0.14.3', 'console_scripts', 'kolibri')()
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 277, in invoke
    return super(KolibriDjangoCommand, self).invoke(ctx)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 727, in manage
    execute_from_command_line(["kolibri manage"] + ctx.args)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/kolibri/core/tasks/management/commands/base.py", line 113, in handle
    return self.handle_async(*args, **options)
  File "/usr/lib/python3/dist-packages/kolibri/core/auth/management/commands/sync.py", line 195, in handle_async
    create_superuser_and_provision_device(
  File "/usr/lib/python3/dist-packages/kolibri/core/auth/management/utils.py", line 209, in create_superuser_and_provision_device
    facility = Facility.objects.get(dataset_id=dataset_id)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/query.py", line 378, in get
    raise self.model.DoesNotExist(
kolibri.core.auth.models.DoesNotExist: Facility matching query does not exist.

Seems that deprovision may have cleaned up more than needed. :)

Only way as of now to get working again is by using a new KOLIBRI_HOME.

Expected behavior

kolibri manage sync should work after starting kolibri after a kolibri manager deprovision

User-facing consequences

Inconsistent system behaviour while using system prescribed tools and commands.

Errors and logs

Shared above.

Steps to reproduce

1. Stop a kolibri instance where all facilities and related data may be cleaned.
2. Execute kolibri manage deprovision
3. Start kolibri
4. Attempt a facility import via kolibri manage sync

Context

Version: 0.14.3
OS: Linux-5.4.0-1015-raspi-aarch64-with-glibc2.29
Python: 3.8.2
Installer: kolibri(apt) with kolibri-server Version: 0.3.8~beta2-0ubuntu1
Server: nginx/1.18.0 (Ubuntu)
Database: postgresql

[intelliant01]

Noticed that kolibri manage sync worked again after a importchannel and importcontent. So whatever "inconsistency" was created due to deprovision seems to be fixed as part or post import* execution.

Could not consistently reproduce this behaviour that seemed to fix the problem with deprovision hence striking it out.

[pateljj04]

If it helps here is the error I get when trying to run the sync from cli on a device that has been deprovisioned.

INFO     Syncing has been initiated (this may take a while)...
INFO     Creating pull transfer session
INFO     There are no records to transfer
INFO     Completed pull transfer session
INFO     Creating push transfer session
INFO     There are no records to transfer
INFO     Completed push transfer session
Traceback (most recent call last):
  File "/usr/bin/kolibri", line 11, in <module>
    load_entry_point('kolibri==0.14.3', 'console_scripts', 'kolibri')()
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 277, in invoke
    return super(KolibriDjangoCommand, self).invoke(ctx)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 727, in manage
    execute_from_command_line(["kolibri manage"] + ctx.args)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/kolibri/core/tasks/management/commands/base.py", line 113, in handle
    return self.handle_async(*args, **options)
  File "/usr/lib/python3/dist-packages/kolibri/core/auth/management/commands/sync.py", line 196, in handle_async
    username, dataset_id, noninteractive=noninteractive
  File "/usr/lib/python3/dist-packages/kolibri/core/auth/management/utils.py", line 209, in create_superuser_and_provision_device
    facility = Facility.objects.get(dataset_id=dataset_id)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/query.py", line 380, in get
    self.model._meta.object_name
kolibri.core.auth.models.DoesNotExist: Facility matching query does not exist.

[pateljj04]

So I have been doing consistent testing, if I setup a fresh kolibri instance, sync it with a facility, deprovision it, I can no longer import that facility.

[jonboiser]

@pateljj04 Are you using the deprovision management command mainly to remove facility and user data from your server? There is another command kolibri manage deletefacility that I think works better if you want to delete a facility and then re-import it again later on.

I do think it is a bug that deprovision doesn't have the same behavior, but deletefacility may be a workaround if you need to just remove facilities, but not content.

[pateljj04]

@jonboiser Yes, we need to remove facility and user data so we can make copies of the SD cards for the Pi. The workaround right now I am using is using importchannel and importconnect command to pull the content in on the main Pi image. Once the content is imported, then I can make copies of the SD card and do an import of the facility on each device. I will give deletefacility a try and see if that works.

[pateljj04]

@jonboiser Thanks! I gave that a try, looks like I have to run both deletefacility and then deprovision otherwise, the setup wizard GUI doesn't launch after the restart. It does let me re-import the facility so that is good news for the workaround, I am guessing deprovision sets a flag to bring up the provisioning whereas deletefacility only removes the facility data, is that correct?

[rtibbles]

I am guessing deprovision sets a flag to bring up the provisioning whereas deletefacility only removes the facility data, is that correct

Yes - deprovision will additionally remove any device settings.

[jonboiser]

@intelliant01 Does this workaround work for you?

1. Use kolibri manage deletefacility to remove the facility data + any associated syncing data (that which may have been causing the problems when re-importing the facility)
2. Use kolibri manage deprovision to delete the device settings and reset the server

It seems that the root cause for this issue is that deprovision was initially implemented when it was not the norm to initialize facilities remotely using the new sync functionality, so how it deletes facilities may differ slightly from deletefacility which was written to complement the sync command.

So @rtibbles suggestion to fix this is to leverage deletefacility within deprovision so that the result of deprovisioning your device are the same irrespective of whether you created your Facility in the traditional way or via syncing.

[intelliant01]

@jonboiser yes let's please go ahead with the suggestion from @rtibbles because system should be in a consistently functional state irrespective of the command having been executed.