Kolibri: 500 Internal Server error on import channel

Created on 2 Oct 2017 · 15Comments · Source: learningequality/kolibri

Summary

While trying to import CK12 channel in a fresh development repo:
fullscreen_02_10_17_01_58

System information

Version: develop branch
Operating system: OSX
Browser: Chrome

Traceback or relevant snippet from server.log or browser console

[02/Oct/2017 01:56:39] "GET /api/tasks/?1506902199429=1506902199429 HTTP/1.1" 200 2
ERROR    Internal Server Error: /api/tasks/startremoteimport/
Traceback (most recent call last):
  File "/Users/mac/.venvs/kolibri/lib/python2.7/site-packages/django/core/handlers/base.py", line 149, in get_response
    response = self.process_exception_by_middleware(e, request)
  File "/Users/mac/.venvs/kolibri/lib/python2.7/site-packages/django/core/handlers/base.py", line 147, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/mac/.venvs/kolibri/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/Users/mac/.venvs/kolibri/lib/python2.7/site-packages/rest_framework/viewsets.py", line 87, in view
    return self.dispatch(request, *args, **kwargs)
  File "/Users/mac/.venvs/kolibri/lib/python2.7/site-packages/rest_framework/views.py", line 466, in dispatch
    response = self.handle_exception(exc)
  File "/Users/mac/.venvs/kolibri/lib/python2.7/site-packages/rest_framework/views.py", line 463, in dispatch
    response = handler(request, *args, **kwargs)
  File "kolibri/tasks/api.py", line 102, in startremoteimport
    get_content_database_file_url(channel_id)).status_code
  File "/Users/mac/.venvs/kolibri/lib/python2.7/site-packages/requests/api.py", line 97, in head
    return request('head', url, **kwargs)
  File "/Users/mac/.venvs/kolibri/lib/python2.7/site-packages/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/mac/.venvs/kolibri/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/mac/.venvs/kolibri/lib/python2.7/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/Users/mac/.venvs/kolibri/lib/python2.7/site-packages/requests/adapters.py", line 477, in send
    raise SSLError(e, request=request)
SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:661)
[02/Oct/2017 01:56:40] "POST /api/tasks/startremoteimport/?1506902199198=1506902199198 HTTP/1.1" 500 109686

After setting "http" instead of "https" here, as suggested by @jamalex, no more 500 error, but transfer still fails, with no apparent errors in console/terminal:

manage_device_content_-_kolibri

[02/Oct/2017 02:43:15] "GET /api/tasks/?1506904995392=1506904995392 HTTP/1.1" 200 151
INFO     Downloading data for channel id 95a52b386f2c485cb97dd60901674a98
INFO:kolibri.content.management.commands.importchannel:Downloading data for channel id 95a52b386f2c485cb97dd60901674a98
DEBUG    URL to fetch: http://studio.learningequality.org/content/databases/95a52b386f2c485cb97dd60901674a98.sqlite3
DEBUG:kolibri.content.management.commands.importchannel:URL to fetch: http://studio.learningequality.org/content/databases/95a52b386f2c485cb97dd60901674a98.sqlite3
DEBUG    Destination: /Users/mac/.kolibri/content/databases/95a52b386f2c485cb97dd60901674a98.sqlite3
DEBUG:kolibri.content.management.commands.importchannel:Destination: /Users/mac/.kolibri/content/databases/95a52b386f2c485cb97dd60901674a98.sqlite3
[02/Oct/2017 02:43:16] "GET /api/tasks/?1506904996392=1506904996392 HTTP/1.1" 200 957
[02/Oct/2017 02:43:17] "GET /api/tasks/?1506904997392=1506904997392 HTTP/1.1" 200 957

Will try installing packages suggested at SO next, to see if it solves something...

needs gherkin update bug

Source

radinamatic

All 15 comments

After installing pyopenssl and ndg-httpsclient (pyasn1 requirement was already satisfied), import is working again:

fullscreen_02_10_17_12_25

(kolibri) macs-iMac:kolibri mac$ pip install pyopenssl
Collecting pyopenssl
  Downloading pyOpenSSL-17.3.0-py2.py3-none-any.whl (51kB)
    100% |████████████████████████████████| 51kB 308kB/s 
Collecting cryptography>=1.9 (from pyopenssl)
  Downloading cryptography-2.0.3-cp27-cp27m-macosx_10_6_intel.whl (1.5MB)
    100% |████████████████████████████████| 1.5MB 639kB/s 
Requirement already satisfied: six>=1.5.2 in /Users/mac/.venvs/kolibri/lib/python2.7/site-packages (from pyopenssl)
Requirement already satisfied: enum34 in /Users/mac/.venvs/kolibri/lib/python2.7/site-packages (from cryptography>=1.9->pyopenssl)
Collecting cffi>=1.7 (from cryptography>=1.9->pyopenssl)
  Downloading cffi-1.11.0-cp27-cp27m-macosx_10_6_intel.whl (237kB)
    100% |████████████████████████████████| 245kB 3.2MB/s 
Collecting ipaddress (from cryptography>=1.9->pyopenssl)
  Downloading ipaddress-1.0.18-py2-none-any.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.9->pyopenssl)
  Downloading asn1crypto-0.23.0-py2.py3-none-any.whl (99kB)
    100% |████████████████████████████████| 102kB 4.4MB/s 
Collecting idna>=2.1 (from cryptography>=1.9->pyopenssl)
  Downloading idna-2.6-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 6.5MB/s 
Collecting pycparser (from cffi>=1.7->cryptography>=1.9->pyopenssl)
  Downloading pycparser-2.18.tar.gz (245kB)
    100% |████████████████████████████████| 256kB 2.2MB/s 
Building wheels for collected packages: pycparser
  Running setup.py bdist_wheel for pycparser ... done
  Stored in directory: /Users/mac/Library/Caches/pip/wheels/95/14/9a/5e7b9024459d2a6600aaa64e0ba485325aff7a9ac7489db1b6
Successfully built pycparser
Installing collected packages: pycparser, cffi, ipaddress, asn1crypto, idna, cryptography, pyopenssl
Successfully installed asn1crypto-0.23.0 cffi-1.11.0 cryptography-2.0.3 idna-2.6 ipaddress-1.0.18 pycparser-2.18 pyopenssl-17.3.0
(kolibri) macs-iMac:kolibri mac$ pip install pyasn1
Requirement already satisfied: pyasn1 in /Users/mac/.venvs/kolibri/lib/python2.7/site-packages
(kolibri) macs-iMac:kolibri mac$ pip install ndg_httpsclient
Collecting ndg_httpsclient
  Downloading ndg_httpsclient-0.4.3-py2-none-any.whl
Requirement already satisfied: PyOpenSSL in /Users/mac/.venvs/kolibri/lib/python2.7/site-packages (from ndg_httpsclient)
Requirement already satisfied: pyasn1>=0.1.1 in /Users/mac/.venvs/kolibri/lib/python2.7/site-packages (from ndg_httpsclient)
Requirement already satisfied: cryptography>=1.9 in /Users/mac/.venvs/kolibri/lib/python2.7/site-packages (from PyOpenSSL->ndg_httpsclient)
Requirement already satisfied: six>=1.5.2 in /Users/mac/.venvs/kolibri/lib/python2.7/site-packages (from PyOpenSSL->ndg_httpsclient)
Requirement already satisfied: enum34 in /Users/mac/.venvs/kolibri/lib/python2.7/site-packages (from cryptography>=1.9->PyOpenSSL->ndg_httpsclient)
Requirement already satisfied: cffi>=1.7 in /Users/mac/.venvs/kolibri/lib/python2.7/site-packages (from cryptography>=1.9->PyOpenSSL->ndg_httpsclient)
Requirement already satisfied: ipaddress in /Users/mac/.venvs/kolibri/lib/python2.7/site-packages (from cryptography>=1.9->PyOpenSSL->ndg_httpsclient)
Requirement already satisfied: asn1crypto>=0.21.0 in /Users/mac/.venvs/kolibri/lib/python2.7/site-packages (from cryptography>=1.9->PyOpenSSL->ndg_httpsclient)
Requirement already satisfied: idna>=2.1 in /Users/mac/.venvs/kolibri/lib/python2.7/site-packages (from cryptography>=1.9->PyOpenSSL->ndg_httpsclient)
Requirement already satisfied: pycparser in /Users/mac/.venvs/kolibri/lib/python2.7/site-packages (from cffi>=1.7->cryptography>=1.9->PyOpenSSL->ndg_httpsclient)
Installing collected packages: ndg-httpsclient
Successfully installed ndg-httpsclient-0.4.3
(kolibri) macs-iMac:kolibri mac$

@aronasorman @jamalex

radinamatic on 2 Oct 2017

looks like we may need to add these to our requirements?

indirectlylit on 2 Oct 2017

Unfortunately, they depend on cryptography, for which there is no pure python alternative.

The resolution suggested by @jamalex was to revert to http rather than https. Given that this is just downloading files, this doesn't seem like a huge concern to me?

rtibbles on 2 Oct 2017

At least for 0.6 that sounds fine. However I assume we'll want to revisit in the future and potentially use https by default when we're on a platform that supports it. (Content download should be 'private' in the same way that browser and search history is.)

indirectlylit on 2 Oct 2017

There was also a suggestion to change the certificate we use to one that requests will handle natively, so longer run fix should be possible.

rtibbles on 2 Oct 2017

change the certificate we use to one that requests will handle natively

requests meaning the python http client library?

what's wrong with our current certificate?

indirectlylit on 3 Oct 2017

I submitted the PR above (#2339), but @radinamatic mentioned that

After setting "http" instead of "https" here, as suggested by @jamalex, no more 500 error, but transfer still fails, with no apparent errors in console/terminal:

so I'm actually not sure whether this addresses the issue

indirectlylit on 3 Oct 2017

Switching to http alone did not solve the problem, at least on my repo. Only after installing the two packages above was I able to import channels.

radinamatic on 3 Oct 2017

@rtibbles what do you think is the answer here?

Initially I thought this might have something to do with our DNS redirecting to HTTPS, but it doesn't seem to be doing that as far as I could tell.

indirectlylit on 8 Oct 2017

@aronasorman had mentioned that he had tweaked the cloud flare settings to avoid this.

Best would be to try to replicate it without the two python packages installed, @radinamatic.

rtibbles on 8 Oct 2017

So the way I was going about this was to fetch the latest develop, create a new venv and install the reqs from scratch. If that was not the best way to go about, let me know...

At first yarn install step failed:

(kolibri-sans2) Dmac:kolibri mac$ yarn install
yarn install v1.1.0
$ node ./frontend_build/src/npm_deprecation_warning.js
[1/5] �  Validating package.json...
[2/5] �  Resolving packages...
success Already up-to-date.
$ node ./frontend_build/src/install_dependencies.js
WARNING:root:No C Extensions available for this platform.

INFO     Version was 0.6.dev020171008165556-git, new version: 0.6.dev020171009215917-git
INFO:kolibri.utils.cli:Version was 0.6.dev020171008165556-git, new version: 0.6.dev020171009215917-git
INFO     Running update routines for new version...
INFO:kolibri.utils.cli:Running update routines for new version...
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/mac/LE/kolibri/kolibri/__main__.py", line 8, in <module>
    main(args=sys.argv[1:])
  File "kolibri/utils/cli.py", line 607, in main
    initialize(debug=debug)
  File "kolibri/utils/cli.py", line 203, in initialize
    update()
  File "kolibri/utils/cli.py", line 264, in update
    call_command("migrate", interactive=False, database="default")
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/core/management/__init__.py", line 119, in call_command
    return command.execute(*args, **defaults)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/core/management/base.py", line 399, in execute
    output = self.handle(*args, **options)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 200, in handle
    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/migrations/executor.py", line 92, in migrate
    self._migrate_all_forwards(plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/migrations/executor.py", line 121, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/migrations/executor.py", line 198, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/migrations/migration.py", line 123, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/migrations/operations/models.py", line 115, in database_forwards
    schema_editor.delete_model(model)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/backends/sqlite3/schema.py", line 213, in delete_model
    super(DatabaseSchemaEditor, self).delete_model(model)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 305, in delete_model
    "table": self.quote_name(model._meta.db_table),
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 110, in execute
    cursor.execute(sql, params)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/utils.py", line 95, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 323, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: content_license
child_process.js:515
    throw err;
    ^

Error: Command failed: python -m kolibri manage webpack_json -- --outputfile /var/folders/9g/_0n3kc0j15z4jt4bnfxxf_th0000gn/T/117910-1801-l3uj8g.l9ogtd42t9.json
WARNING:root:No C Extensions available for this platform.

INFO     Version was 0.6.dev020171008165556-git, new version: 0.6.dev020171009215917-git
INFO:kolibri.utils.cli:Version was 0.6.dev020171008165556-git, new version: 0.6.dev020171009215917-git
INFO     Running update routines for new version...
INFO:kolibri.utils.cli:Running update routines for new version...
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/mac/LE/kolibri/kolibri/__main__.py", line 8, in <module>
    main(args=sys.argv[1:])
  File "kolibri/utils/cli.py", line 607, in main
    initialize(debug=debug)
  File "kolibri/utils/cli.py", line 203, in initialize
    update()
  File "kolibri/utils/cli.py", line 264, in update
    call_command("migrate", interactive=False, database="default")
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/core/management/__init__.py", line 119, in call_command
    return command.execute(*args, **defaults)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/core/management/base.py", line 399, in execute
    output = self.handle(*args, **options)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 200, in handle
    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/migrations/executor.py", line 92, in migrate
    self._migrate_all_forwards(plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/migrations/executor.py", line 121, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/migrations/executor.py", line 198, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/migrations/migration.py", line 123, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/migrations/operations/models.py", line 115, in database_forwards
    schema_editor.delete_model(model)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/backends/sqlite3/schema.py", line 213, in delete_model
    super(DatabaseSchemaEditor, self).delete_model(model)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 305, in delete_model
    "table": self.quote_name(model._meta.db_table),
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 110, in execute
    cursor.execute(sql, params)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/utils.py", line 95, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/Users/mac/.venvs/kolibri-sans2/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 323, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: content_license

    at checkExecSyncError (child_process.js:472:13)
    at execSync (child_process.js:512:13)
    at module.exports (/Users/mac/LE/kolibri/frontend_build/src/read_webpack_json.js:10:3)
    at Object.<anonymous> (/Users/mac/LE/kolibri/frontend_build/src/install_dependencies.js:6:15)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

Once I deleted the ./kolibri folder, error was no more, server started OK, and import works:

fullscreen_10_10_17_01_39

So, the original error here is not extant anymore, and I'm going to close this issue.
@rtibbles If the yarn error requires a separate issue, let me know and I'll file it :slightly_smiling_face:

radinamatic on 10 Oct 2017

No, the yarn error is because you had a database created by the version of 0.6.x that did not properly import licenses, and you are running it with the version that does now. The migration histories for these do not mix.

rtibbles on 10 Oct 2017

👍1

Note that this was initially tested on develop, but happened on 0.6 for a different reason :) I'm reopening it for the issue on develop.

I changed our SSL cert for studio.learningequality.org @radinamatic, can you try and see if you can still replicate the issue?

aronasorman on 17 Oct 2017

👍1

Not able to replicate this on latest develop @aronasorman, import proceeding correctly over here...

fullscreen_17_10_17_01_18

radinamatic on 17 Oct 2017

Yey!

aronasorman on 17 Oct 2017

😄1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Update profile modal displays to super admin account created with setup wizard

radinamatic · 4Comments

Users with a Facility/Class Coach Role should be redirected to the Coach page after logon

jonboiser · 5Comments

migrate testing info to docs

indirectlylit · 4Comments

Missing images in Perseus exercises when the mastery model is changed on Studio

radinamatic · 5Comments

Create linting rules to check for unused props, data, computed props, and vuex mappers

rtibbles · 6Comments