Kolibri: Version 0.10 Server SSL Errors

[nobicycle]

Hi. I noticed v0.10 was out and did a clean install, deleting the old kolibri directory.
I selected Khan Academy for import. "Generating channel listing. This could take a few minutes…" took so long I looked in the logs, attached. Seems like there are various issues.

server.log
kolibri.log

[indirectlylit]

@nobicycle thanks for the report!

Could you please provide:

• install method (pex, pypi, etc)
• OS and version of OS

[nobicycle]

Sorry.
Arch( Manjaro) Linux
$ uname -a
Linux tower 4.14.52-1-MANJARO #1 SMP PREEMPT Tue Jun 26 15:24:31 UTC 2018 x86_64 GNU/Linux

Method: pip install in a vitualenv

[benjaoming]

It seems that the error occurs in the middle of downloading of channel id 1ceff53605e55bef987d88e0908658c5

@nobicycle thanks a lot for filing this. I suspect this is related to some kind of missing error-retry. So probably a sub task failed, and there is no user feedback in the UI unfortunately. You could potentially retry this, it doesn't seem like something that would persist.

INFO:kolibri.content.management.commands.importchannel:Downloading data for channel id 1ceff53605e55bef987d88e0908658c5

  0%|          | 0/162202624 [00:00<?, ?it/s]
  1%|▏         | 2097152/162202624 [00:05<07:10, 371926.49it/s]venv/lib/python3.6/site-packages/kolibri/dist/tqdm/_monitor.py:89: TqdmSynchronisationWarning: Set changed size during iteration (see https://github.com/tqdm/tqdm/issues/481)
  TqdmSynchronisationWarning)

  3%|▎         | 4194304/162202624 [00:12<07:35, 346983.86it/s]
  4%|▍         | 6291456/162202624 [00:17<07:10, 361964.45it/s]
  5%|▌         | 8388608/162202624 [00:22<06:35, 388820.85it/s]
  6%|▋         | 10485760/162202624 [00:26<06:02, 418468.50it/s]
  8%|▊         | 12582912/162202624 [00:30<05:42, 437342.04it/s]
  9%|▉         | 14680064/162202624 [00:34<05:16, 465434.39it/s]
 10%|█         | 16777216/162202624 [00:37<04:47, 506111.89it/s]
 12%|█▏        | 18874368/162202624 [00:40<04:18, 555307.01it/s]
 13%|█▎        | 20971520/162202624 [00:43<03:52, 606634.40it/s]
 14%|█▍        | 23068672/162202624 [00:47<04:00, 577483.49it/s]
 16%|█▌        | 25165824/162202624 [00:51<04:12, 542155.34it/s]
 17%|█▋        | 27262976/162202624 [00:58<04:52, 460935.41it/s]
 18%|█▊        | 29360128/162202624 [01:05<05:36, 394798.92it/s]
 19%|█▉        | 31457280/162202624 [01:11<05:47, 375937.87it/s]
 21%|██        | 33554432/162202624 [01:15<05:24, 396163.39it/s]
 22%|██▏       | 35651584/162202624 [01:20<05:09, 409239.64it/s]
 23%|██▎       | 37748736/162202624 [01:26<05:14, 395452.06it/s]
 25%|██▍       | 39845888/162202624 [01:31<05:04, 401876.25it/s]
 26%|██▌       | 41943040/162202624 [01:35<04:35, 436459.79it/s]
 27%|██▋       | 44040192/162202624 [01:38<04:05, 480402.22it/s]
 28%|██▊       | 46137344/162202624 [01:41<03:44, 516070.79it/s]
WARNING  Job 2c37ebf9f9e1466c88dbbf1e4796478f raised an exception: Traceback (most recent call last):
  File "venv/lib/python3.6/site-packages/kolibri/dist/urllib3/contrib/pyopenssl.py", line 280, in recv_into
    return self.connection.recv_into(*args, **kwargs)
  File "venv/lib/python3.6/site-packages/kolibri/dist/OpenSSL/SSL.py", line 1625, in recv_into
    self._raise_ssl_error(self._ssl, result)
  File "venv/lib/python3.6/site-packages/kolibri/dist/OpenSSL/SSL.py", line 1431, in _raise_ssl_error
    raise WantReadError()
OpenSSL.SSL.WantReadError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "venv/lib/python3.6/site-packages/kolibri/dist/iceqube/worker/backends/inmem.py", line 75, in handle_finished_future
    result = future.result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "venv/lib/python3.6/site-packages/kolibri/dist/iceqube/worker/backends/inmem.py", line 149, in wrap
    raise e
  File "venv/lib/python3.6/site-packages/kolibri/dist/iceqube/worker/backends/inmem.py", line 145, in wrap
    return f(*args, **kwargs)
  File "venv/lib/python3.6/site-packages/kolibri/dist/iceqube/common/classes.py", line 119, in y
    return func(*args, **kwargs)
  File "venv/lib/python3.6/site-packages/kolibri/dist/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "venv/lib/python3.6/site-packages/kolibri/dist/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "venv/lib/python3.6/site-packages/kolibri/tasks/management/commands/base.py", line 99, in handle
    return self.handle_async(*args, **options)
  File "venv/lib/python3.6/site-packages/kolibri/content/management/commands/importchannel.py", line 125, in handle_async
    self.download_channel(options["channel_id"], options["baseurl"])
  File "venv/lib/python3.6/site-packages/kolibri/content/management/commands/importchannel.py", line 76, in download_channel
    self._transfer(DOWNLOAD_METHOD, channel_id, baseurl)
  File "venv/lib/python3.6/site-packages/kolibri/content/management/commands/importchannel.py", line 105, in _transfer
    for chunk in filetransfer:
  File "venv/lib/python3.6/site-packages/kolibri/content/utils/transfer.py", line 74, in __next__
    return self.next()
  File "venv/lib/python3.6/site-packages/kolibri/content/utils/transfer.py", line 163, in next
    return super(FileDownload, self).next()
  File "venv/lib/python3.6/site-packages/kolibri/content/utils/transfer.py", line 78, in next
    chunk = next(self._content_iterator)
  File "venv/lib/python3.6/site-packages/kolibri/dist/requests/models.py", line 745, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "venv/lib/python3.6/site-packages/kolibri/dist/urllib3/response.py", line 436, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "venv/lib/python3.6/site-packages/kolibri/dist/urllib3/response.py", line 384, in read
    data = self._fp.read(amt)
  File "/usr/lib64/python3.6/http/client.py", line 449, in read
    n = self.readinto(b)
  File "/usr/lib64/python3.6/http/client.py", line 493, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib64/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "venv/lib/python3.6/site-packages/kolibri/dist/urllib3/contrib/pyopenssl.py", line 296, in recv_into
    return self.recv_into(*args, **kwargs)
  File "venv/lib/python3.6/site-packages/kolibri/dist/urllib3/contrib/pyopenssl.py", line 280, in recv_into
    return self.connection.recv_into(*args, **kwargs)
  File "venv/lib/python3.6/site-packages/kolibri/dist/OpenSSL/SSL.py", line 1625, in recv_into
    self._raise_ssl_error(self._ssl, result)
  File "venv/lib/python3.6/site-packages/kolibri/dist/OpenSSL/SSL.py", line 1456, in _raise_ssl_error
    _raise_current_error()
  File "venv/lib/python3.6/site-packages/kolibri/dist/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'ssl3_get_record', 'decryption failed or bad record mac')]

WARNING:iceqube.worker.backends.base:Job 2c37ebf9f9e1466c88dbbf1e4796478f raised an exception: Traceback (most recent call last):
  File "venv/lib/python3.6/site-packages/kolibri/dist/urllib3/contrib/pyopenssl.py", line 280, in recv_into
    return self.connection.recv_into(*args, **kwargs)
  File "venv/lib/python3.6/site-packages/kolibri/dist/OpenSSL/SSL.py", line 1625, in recv_into
    self._raise_ssl_error(self._ssl, result)
  File "venv/lib/python3.6/site-packages/kolibri/dist/OpenSSL/SSL.py", line 1431, in _raise_ssl_error
    raise WantReadError()
OpenSSL.SSL.WantReadError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "venv/lib/python3.6/site-packages/kolibri/dist/iceqube/worker/backends/inmem.py", line 75, in handle_finished_future
    result = future.result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "venv/lib/python3.6/site-packages/kolibri/dist/iceqube/worker/backends/inmem.py", line 149, in wrap
    raise e
  File "venv/lib/python3.6/site-packages/kolibri/dist/iceqube/worker/backends/inmem.py", line 145, in wrap
    return f(*args, **kwargs)
  File "venv/lib/python3.6/site-packages/kolibri/dist/iceqube/common/classes.py", line 119, in y
    return func(*args, **kwargs)
  File "venv/lib/python3.6/site-packages/kolibri/dist/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "venv/lib/python3.6/site-packages/kolibri/dist/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "venv/lib/python3.6/site-packages/kolibri/tasks/management/commands/base.py", line 99, in handle
    return self.handle_async(*args, **options)
  File "venv/lib/python3.6/site-packages/kolibri/content/management/commands/importchannel.py", line 125, in handle_async
    self.download_channel(options["channel_id"], options["baseurl"])
  File "venv/lib/python3.6/site-packages/kolibri/content/management/commands/importchannel.py", line 76, in download_channel
    self._transfer(DOWNLOAD_METHOD, channel_id, baseurl)
  File "venv/lib/python3.6/site-packages/kolibri/content/management/commands/importchannel.py", line 105, in _transfer
    for chunk in filetransfer:
  File "venv/lib/python3.6/site-packages/kolibri/content/utils/transfer.py", line 74, in __next__
    return self.next()
  File "venv/lib/python3.6/site-packages/kolibri/content/utils/transfer.py", line 163, in next
    return super(FileDownload, self).next()
  File "venv/lib/python3.6/site-packages/kolibri/content/utils/transfer.py", line 78, in next
    chunk = next(self._content_iterator)
  File "venv/lib/python3.6/site-packages/kolibri/dist/requests/models.py", line 745, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "venv/lib/python3.6/site-packages/kolibri/dist/urllib3/response.py", line 436, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "venv/lib/python3.6/site-packages/kolibri/dist/urllib3/response.py", line 384, in read
    data = self._fp.read(amt)
  File "/usr/lib64/python3.6/http/client.py", line 449, in read
    n = self.readinto(b)
  File "/usr/lib64/python3.6/http/client.py", line 493, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib64/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "venv/lib/python3.6/site-packages/kolibri/dist/urllib3/contrib/pyopenssl.py", line 296, in recv_into
    return self.recv_into(*args, **kwargs)
  File "venv/lib/python3.6/site-packages/kolibri/dist/urllib3/contrib/pyopenssl.py", line 280, in recv_into
    return self.connection.recv_into(*args, **kwargs)
  File "venv/lib/python3.6/site-packages/kolibri/dist/OpenSSL/SSL.py", line 1625, in recv_into
    self._raise_ssl_error(self._ssl, result)
  File "venv/lib/python3.6/site-packages/kolibri/dist/OpenSSL/SSL.py", line 1456, in _raise_ssl_error
    _raise_current_error()
  File "venv/lib/python3.6/site-packages/kolibri/dist/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'ssl3_get_record', 'decryption failed or bad record mac')]

[benjaoming]

I think this issue has 3 levels:

1. Investigating and fixing the immediate SSL problem
2. Adding proper user feedback for this exception, since it seems that the job has failed and the user isn't informed
3. #3494

[indirectlylit]

yes, @nobicycle I'm very curious whether this recurs after restarting Kolibri?

Assuming that it does not, I think our short-term fix is simply to make sure we're catching this and any other unhandled errors correctly and flagging the outstanding iceqube transfer as 'failed' when they happen happens.

[lyw07]

Hi @nobicycle ,
I tried to reproduce the error on a Manjaro Linux 17.1.6 but failed to do so. Do you mind telling us what is the version of your openssl by entering openssl version in your terminal? Thank you!

[benjaoming]

@lyw07 it's okay if the problem cannot be reproduced, I think the logs show enough traceback to simulate it and create the necessary error handling.

The problem looks like a network connection error occurred.

[nobicycle]

$ openssl version
OpenSSL 1.1.0h 27 Mar 2018

Here are the fresh logs (after truncation) following the same procedure.

kolibri.log.txt
server.log.txt

[benjaoming]

Thanks, @nobicycle, pretty convincing that it's a more common issue, since it's now happening already after 1% progress.

[nobicycle]

Hi,

I believe it is a an external package (openssl) problem, since I am getting similar issues building packages:

==> Making package: laverna 0.7.51-10 (Sun 08 Jul 2018 12:24:31 PM WITA)
==> Retrieving sources...
-> Found laverna.desktop
-> Downloading laverna-0.7.51-linux-x64.zip...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 618 0 618 0 0 839 0 --:--:-- --:--:-- --:--:-- 838
21 49.5M 21 10.6M 0 0 33735 0 0:25:41 0:05:31 0:20:10 0
curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
==> ERROR: Failure while downloading https://github.com/Laverna/laverna/releases/download/0.7.51/laverna-0.7.51-linux-x64.zip
Aborting...
Error downloading sources: laverna

[benjaoming]

Seems your network connections might be interrupted from a local or intermediate issue? Try downloading big files w/o SSL connections?

Here's a big file, no SSL: http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip

Also, try downloading big files on another device, on the same network? Do you have IP conflicts on the network? Weird stuff like that could cause it...

[nobicycle]

@benjaoming
Thanks.
Compiling an opensmtpd package today I obtained:

checking OpenSSL header version... 1010008f (OpenSSL 1.1.0h 27 Mar 2018)
checking OpenSSL library version... 1010008f (OpenSSL 1.1.0h 27 Mar 2018)
checking whether OpenSSL's headers match the library... no
configure: error: Your OpenSSL headers do not match your
library. Check config.log for details.

Give the error has not been reproduced given the external ssl issues raised shall I close?

[benjaoming]

@nobicycle that does sound as the likely cause!

I will create another issue about the missing user feedback.

[nobicycle]

I tested downloading the smaller channels.

Global Digital Library - Book Catalog
Open Stax
Osmosis.org
PhET Interactive Simulations (en)
Pratham Books' StoryWeaver
Touchable Earth (en)
Sikana

These were all OK. There were no errors in the server.log

Then I tried a "bigger" channel, CK12. There were errors.

So the problem seems to be only the bigger channels such as Khan, MIT Blossoms and CK12

Could this mean "big files" is not the issue - since I was able to download 30MB of content in one go. I suppose the channel listings are smaller than that.

Incidentally, there is a visual indication when an error occurs. A red exclamation mark appears next to "Generating channel listing. This could take a few minutes…"

[benjaoming]

Incidentally, there is a visual indication when an error occurs. A red exclamation mark appears next to "Generating channel listing. This could take a few minutes…"

Will add this info to the other issue. Thanks!

[benjaoming]

The larger the file, the longer the download, the more likely that a randomly occurring error pattern is reproduced!

Could you try downloading the large file that I linked w/o SSL and see if that's successful? It seems that you have general system-wide SSL library issues?

I suppose the channel listings are smaller than that.

Am not sure of their size!

If you are a very curious person, you could install Wireshark on the failing machine and monitor the network traffic to see what's generally happening :)

[nobicycle]

I downgraded openssl

openssl version

OpenSSL 1.1.0e 16 Feb 2017

And MIT Blossom channel was imported without error.

However Khan did not

INFO     Downloading data for channel id 1ceff53605e55bef987d88e0908658c5
INFO:kolibri.content.management.commands.importchannel:Downloading data for channel id 1ceff53605e55bef987d88e0908658c5
  6%|▋         | 10485760/162202624 [07:10<1:37:54, 25828.38it/s][root@tower kolibri]# tail  /opt/kolibri/server.log 
  File "/opt/kolibri/kolibri-venv/lib/python3.6/site-packages/kolibri/dist/urllib3/contrib/pyopenssl.py", line 280, in recv_into
    return self.connection.recv_into(*args, **kwargs)
  File "/opt/kolibri/kolibri-venv/lib/python3.6/site-packages/kolibri/dist/OpenSSL/SSL.py", line 1625, in recv_into
    self._raise_ssl_error(self._ssl, result)
  File "/opt/kolibri/kolibri-venv/lib/python3.6/site-packages/kolibri/dist/OpenSSL/SSL.py", line 1456, in _raise_ssl_error
    _raise_current_error()
  File "/opt/kolibri/kolibri-venv/lib/python3.6/site-packages/kolibri/dist/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'ssl3_get_record', 'decryption failed or bad record mac')]

BTW there is a github issue referenced in the log:

# grep -i tqdm  /opt/kolibri/server.log 
  0%|          | 0/3977216 [00:00<?, ?it/s]/opt/kolibri/kolibri-venv/lib/python3.6/site-packages/kolibri/dist/tqdm/_monitor.py:89: TqdmSynchronisationWarning: Set changed size during iteration (see https://github.com/tqdm/tqdm/issues/481)
  TqdmSynchronisationWarning)

Anything to do with the problem?

[benjaoming]

No that seems unrelated, the problem must be due to SSL or network issues.

How does this command run? (it downloads a large file w/o SSL)

curl -S http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip > /dev/null

[nobicycle]

Hi benjaoming, I don't want to download nearly 1GB - we're' on very limited quota here. I download ISO's as big as that when necessary, but as you are suggesting, there may be some intermittent network issue at play.

Today SSL was updated to $ openssl version
OpenSSL 1.1.0h 27 Mar 2018

and ...

all channel indexes are imported except Khan, but even then, it reached 96%

BTW is there a manual way to obtain the 160 MB Khan index? It is actually the most important channel for us.

Perhaps a resume partial download would be interesting for these quite big index downloads.

[nobicycle]

Just poking around for similar issues ... here they say OpenSSL.SSL.WantReadError connectivity interruptions are inevitable and recommend a wrapper to trap the error:

https://stackoverflow.com/questions/49530034/tweepy-openssl-ssl-wantreaderror

We know the issue is core SSL. Here is is again merely installing a package on Arch:

==> Retrieving sources...
-> Downloading android_amd64.img...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 354 100 354 0 0 114 0 0:00:03 0:00:03 --:--:-- 114
28 310M 28 87.0M 0 0 592k 0 0:08:56 0:02:30 0:06:26 685k
> curl: (56) OpenSSL SSL_read: error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac, errno 0
==> ERROR: Failure while downloading http://build.anbox.io/android-images/2018/06/12/android_amd64.img
Aborting...
Error downloading sources: anbox-image

But at least there, I can restart and nothing is lost. A wrapper seems inevitable for now and in the future.

[nobicycle]

The solution - install trickle bandwidth throttling. I limited 50 KB/s download and used the cli:

trickle -d50 -u10 ./kolibri-venv/bin/kolibri manage importchannel network 1ceff53605e55bef987d88e0908658c5

100%|█████████████████████████| 162202624/162202624 [03:34<00:00, 554272.63it/s]INFO     Importing ContentTag data
INFO:kolibri.content.utils.channel_import:Importing ContentTag data
INFO     Importing ContentNode_has_prerequisite data
INFO:kolibri.content.utils.channel_import:Importing ContentNode_has_prerequisite data
INFO     Importing ContentNode_related data
INFO:kolibri.content.utils.channel_import:Importing ContentNode_related data
INFO     Importing ContentNode_tags data
INFO:kolibri.content.utils.channel_import:Importing ContentNode_tags data
INFO     Importing ContentNode data
INFO:kolibri.content.utils.channel_import:Importing ContentNode data
INFO     Importing Language data
INFO:kolibri.content.utils.channel_import:Importing Language data
INFO     Importing File data
INFO:kolibri.content.utils.channel_import:Importing File data
INFO     Importing LocalFile data
INFO:kolibri.content.utils.channel_import:Importing LocalFile data
INFO     Importing AssessmentMetaData data
INFO:kolibri.content.utils.channel_import:Importing AssessmentMetaData data
INFO     Importing ChannelMetadata data
INFO:kolibri.content.utils.channel_import:Importing ChannelMetadata data
INFO     Setting availability of File objects based on LocalFile availability
INFO:kolibri.content.utils.annotation:Setting availability of File objects based on LocalFile availability
INFO     Setting availability of non-topic ContentNode objects based on File availability
INFO:kolibri.content.utils.annotation:Setting availability of non-topic ContentNode objects based on File availability
INFO     Setting availability of ContentNode objects with children for 5 levels
INFO:kolibri.content.utils.annotation:Setting availability of ContentNode objects with children for 5 levels
INFO     Setting availability of ContentNode objects with children for level 5
INFO:kolibri.content.utils.annotation:Setting availability of ContentNode objects with children for level 5
INFO     Setting availability of ContentNode objects with children for level 4
INFO:kolibri.content.utils.annotation:Setting availability of ContentNode objects with children for level 4
INFO     Setting availability of ContentNode objects with children for level 3
INFO:kolibri.content.utils.annotation:Setting availability of ContentNode objects with children for level 3
INFO     Setting availability of ContentNode objects with children for level 2
INFO:kolibri.content.utils.annotation:Setting availability of ContentNode objects with children for level 2
INFO     Setting availability of ContentNode objects with children for level 1
INFO:kolibri.content.utils.annotation:Setting availability of ContentNode objects with children for level 1

There are a ton of references on the subject of SSL link failures. Here is one that lists 6 experiments to debug the problem: https://bbs.archlinux.org/viewtopic.php?id=201625

[nobicycle]

Actually I spoke too soon. Now I have a /opt/kolibri/content/databases
$ ls

000409f81dbe5d1ba67101cb9fed4530.sqlite3
0e173fca6e9052f8a474a2fb84055faf.sqlite3
131e543dbecf5776bb13cfcfddf05605.sqlite3
197934f144305350b5820c7c4dd8e194.sqlite3
1ceff53605e55bef987d88e0908658c5.sqlite3
1d8f6d84618153c18c695d85074952a7.sqlite3
3e464ee12f6a50a781cddf59147b48b1.sqlite3
8b28761bac075deeb66adc6c80ef119c.sqlite3
913efe9f14c65cb1b23402f21f056e99.sqlite3
d6e3b856125f5e6aa5fb40c8b112d5e9.sqlite3
fc47aee82e0153e2a30197d3fdee1128.sqlite3

and a folder /opt/kolibri/.kolibri/content/databases/

$ ls -lh
total 155M
-rw-r--r-- 1 kolibri kolibri 155M Jul 11 05:53 1ceff53605e55bef987d88e0908658c5.sqlite3
-rw-r--r-- 1 kolibri kolibri    0 Jul 11 05:57 1ceff53605e55bef987d88e0908658c5.sqlite3.transfer

Can I just move the latter sqlite3 into the former directory?

[benjaoming]

Hi @nobicycle - I just hit the same issues on a Raspberry Pi. It also has a very fast connection, so I think your idea of throttling is interesting.

[nobicycle]

Hi benjaoming
But also a wrapper to trap the error (see two messages up in the thread) - then 96% downloads can resume. :-)

[benjaoming]

@nobicycle - yes, I think that @lyw07 is revisiting the retry logic and will have a fix ready for 0.10.1, thanks a lot for guiding through the cause of this error!

[nobicycle]

Note: throttling is not a panacea:

sudo -u kolibri KOLIBRI_USER=kolibri KOLIBRI_HOME=/opt/kolibri trickle -d50 -u10 /opt/kolibri/kolibri-venv/bin/kolibri manage importchannel network 1ceff53605e55bef987d88e0908658c5
trickle: Could not reach trickled, working independently: No such file or directory
INFO Running Kolibri with the following settings: kolibri.deployment.default.settings.base
INFO Downloading data for channel id 1ceff53605e55bef987d88e0908658c5
4%|█ | 4194304/105951232 [00:09<03:54, 434225.29it/s]/opt/kolibri/kolibri-venv/lib/python3.6/site-packages/kolibri/dist/tqdm/_monitor.py:89: TqdmSynchronisationWarning: Set changed size during iteration (see https://github.com/tqdm/tqdm/issues/481)
TqdmSynchronisationWarning)
75%|███████████████████▌ | 79691776/105951232 [04:30<01:50, 238322.21it/s]
Traceback (most recent call last):

File "/opt/kolibri/kolibri-venv/lib/python3.6/site-packages/kolibri/dist/urllib3/contrib/pyopenssl.py", line 280, in recv_into
return self.connection.recv_into(args, *kwargs)
File "/opt/kolibri/kolibri-venv/lib/python3.6/site-packages/kolibri/dist/OpenSSL/SSL.py", line 1625, in recv_into
self._raise_ssl_error(self._ssl, result)
File "/opt/kolibri/kolibri-venv/lib/python3.6/site-packages/kolibri/dist/OpenSSL/SSL.py", line 1431, in _raise_ssl_error
raise WantReadError()
OpenSSL.SSL.WantReadError

[benjaoming]

Fixed in #4059 and released in 0.10.1

@nobicycle the installers for this release are here:

PEX, .whl and Windows installer: https://github.com/learningequality/kolibri/releases/tag/v0.10.1

Debian installers currently from the Kolibri proposed repository: https://launchpad.net/~learningequality/+archive/ubuntu/kolibri-proposed

[benjaoming]

@nobicycle also big thanks for doing so much work to help getting this fixed! I _hope_ that it works now, and I will try to do some testing on Raspberry Pi.

[nobicycle]

Much obliged. will test asap.

[nobicycle]

Just downloaded the Kahn Academy channel listings!
No error in server.log!
Great stuff.
Thanks

[benjaoming]

I observe the same -- have been importing for 24 hours on my Raspberry Pi without error messages :)