Spyder: Kite call-to-action consistently causes a Spyder hard crash

[CAM-Gerlach]

Description

What steps will reproduce the problem?

I was able to reproduce a hard crash to desktop/unrecoverable hang consistantly (4 out of 4 attempts) when triggering completions on any code that triggers the Kite advertisement. Otherwise, completion works fine. I have not experienced this problem prior to Beta 6, either with dev builds from before #10332 was merged or Beta 5.

Furthermore from a UX perspective, since showing the ads for Kite that trigger this bug is opt-out rather than opt-in, if other users run into this they will have no idea that this issue is due to Kite ads being enabled by default (particularly if they already affirmatively clicked dismiss already on the popup ad for Kite that appears on Spyder launch, as I did) and the only way to disable them otherwise is buried at the very bottom of the Advanced tab of the completion pane of the preferences dialog.

Easiest way to repro:

1. Open a new file
2. Type any code that would trigger the Kite advertisement, e.g. (as in #10332 ):

import matplotlib.pyplot as plt
fig = plt.figure()
fig.add

1. Press tab to trigger completions, and Spyder crashes to desktop or hangs forever and must be force-killed.

When I tested it with on the fly completion enabled (as is default), merely steps 1 and 2 were sufficient to trigger the issue, though often I only got this error dialog instead of a hard crash (which is why I'm reporting it now), whereas manually triggering completion seemed to always cause a hard crash. I will restest further with the latest master and confirm it is still though non of the commits seem obviously related.

Traceback

  File "C:\Miniconda3\envs\spyder-4\lib\site-packages\spyder\plugins\editor\widgets\codeeditor.py", line 1091, in process_completion
    self.kite_call_to_action.handle_processed_completions(completions)
  File "C:\Miniconda3\envs\spyder-4\lib\site-packages\spyder\plugins\completion\kite\widgets\calltoaction.py", line 106, in handle_processed_completions
    if not KiteBloomFilter.is_valid_path(fn_path):
  File "C:\Miniconda3\envs\spyder-4\lib\site-packages\spyder\plugins\completion\kite\bloomfilter\__init__.py", line 28, in is_valid_path
    cls._filter = BloomFilter.fromfile(data_path)
  File "C:\Miniconda3\envs\spyder-4\lib\site-packages\spyder\utils\external\pybloom_pyqt\pybloom.py", line 215, in fromfile
    raise ValueError("unable to open file " + path)
ValueError: unable to open file C:\Miniconda3\envs\spyder-4\lib\site-packages\spyder\plugins\completion\kite\bloomfilter\kite.bloom

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Miniconda3\envs\spyder-4\lib\logging\__init__.py", line 1025, in emit
    msg = self.format(record)
  File "C:\Miniconda3\envs\spyder-4\lib\logging\__init__.py", line 869, in format
    return fmt.format(record)
  File "C:\Miniconda3\envs\spyder-4\lib\logging\__init__.py", line 608, in format
    record.message = record.getMessage()
  File "C:\Miniconda3\envs\spyder-4\lib\logging\__init__.py", line 369, in getMessage
    msg = msg % self.args
ValueError: incomplete format
Call stack:
  File "C:\Miniconda3\envs\spyder-4\Scripts\spyder-script.py", line 10, in <module>
    sys.exit(main())
  File "C:\Miniconda3\envs\spyder-4\lib\site-packages\spyder\app\start.py", line 196, in main
    mainwindow.main()
  File "C:\Miniconda3\envs\spyder-4\lib\site-packages\spyder\app\mainwindow.py", line 3646, in main
    mainwindow = run_spyder(app, options, args)
  File "C:\Miniconda3\envs\spyder-4\lib\site-packages\spyder\app\mainwindow.py", line 3529, in run_spyder
    app.exec_()
  File "C:\Miniconda3\envs\spyder-4\lib\site-packages\spyder\plugins\completion\languageserver\client.py", line 343, in on_msg_received
    handler(resp['result'], req_id)
  File "C:\Miniconda3\envs\spyder-4\lib\site-packages\spyder\plugins\completion\languageserver\providers\document.py", line 109, in process_document_completion
    LSPRequestTypes.DOCUMENT_COMPLETION, {'params': response})
  File "C:\Miniconda3\envs\spyder-4\lib\site-packages\spyder\plugins\completion\languageserver\plugin.py", line 279, in receive_response
    self.COMPLETION_CLIENT_NAME, req_id, response)
  File "C:\Miniconda3\envs\spyder-4\lib\site-packages\spyder\plugins\completion\plugin.py", line 138, in receive_response
    self.gather_and_send(request_responses)
  File "C:\Miniconda3\envs\spyder-4\lib\site-packages\spyder\plugins\completion\plugin.py", line 189, in gather_and_send
    response_instance.handle_response(req_type, responses)
  File "C:\Miniconda3\envs\spyder-4\lib\site-packages\spyder\plugins\editor\widgets\codeeditor.py", line 915, in handle_response
    handler(params)
  File "C:\Miniconda3\envs\spyder-4\lib\site-packages\spyder\plugins\editor\widgets\codeeditor.py", line 1098, in process_completion
    self.log_lsp_handle_errors('Error when processing completions')
  File "C:\Miniconda3\envs\spyder-4\lib\site-packages\spyder\plugins\editor\widgets\codeeditor.py", line 946, in log_lsp_handle_errors
    logger.error('%', 1, stack_info=True)
Message: '%'
Arguments: (1,)

Versions

• Spyder version: 4.0.0b6
• Python version: 3.7.4
• Qt version: 5.9.6
• PyQt5 version: 5.9.2
• Operating System: Windows 8.1

Dependencies

cloudpickle >=0.5.0          :  0.7.0 (OK)
pygments >=2.0               :  2.3.1 (OK)
qtconsole >=4.5.5            :  4.5.5 (OK)
nbconvert >=4.0              :  5.4.0 (OK)
sphinx >=0.6.6               :  1.8.4 (OK)
pylint >=0.25                :  2.2.2 (OK)
psutil >=0.3                 :  5.6.3 (OK)
qtawesome >=0.5.7            :  0.6.0 (OK)
qtpy >=1.5.0                 :  1.9.0 (OK)
pickleshare >=0.4            :  0.7.5 (OK)
zmq >=17                     :  17.1.2 (OK)
chardet >=2.0.0              :  3.0.4 (OK)
numpydoc >=0.6.0             :  0.8.0 (OK)
spyder_kernels >=1.5.0;<2.0.0:  1.5.0 (OK)
qdarkstyle >=2.7             :  2.7 (OK)
atomicwrites >=1.2.0         :  1.3.0 (OK)
diff_match_patch >=20181111  :  20181111 (OK)
watchdog                     :  None (OK)
keyring                      :  None (OK)
pexpect >=4.4.0              :  4.7.0 (OK)
pympler                      :  None (OK)
sympy >=0.7.3                :  1.3 (OK)
cython >=0.21                :  0.29.4 (OK)
IPython >=4.0                :  7.2.0 (OK)
matplotlib >=2.0.0           :  3.0.2 (OK)
pandas >=0.13.1              :  0.24.2 (OK)
numpy >=1.7                  :  1.15.4 (OK)
scipy >=0.17.0               :  1.2.0 (OK)
pyls >=0.28.2;<0.29.0        :  0.28.3 (OK)
rtree >=0.8.3                :  0.8.3 (OK)

[CAM-Gerlach]

At the risk of doing the Kite company's work for them which I am not particularly inclined to do, some additional investigation has apparently narrowed down what seems to be going on.

I tested with the latest master and the issue did not occur, but neither did it when I checked out the beta 6 release tag. Combined with the initial error message from the traceback, it seems possible the kite binary blob isn't being opened properly and instead relying on false assumptions about package structure that only hold if and when running from source rather than from a normal install. If so, then I would imagine the solution would be to follow basic packaging guidelines and access the data file via something like pkg_resources rather than whatever is being used currently.

BTW, the "During handling..." error is the same as in #10376 , but this appears otherwise unrelated.

Also, to note, in my testing this continues to happen even if the option to enabled the advertisements is unchecked in preferences, until Spyder is restarted completely (or crashes). Similarly, it requires another restart to re-enable the ads and the crashes using the preferences checkbox. Is that really the intended behavior?

[ccordoba12]

@CAM-Gerlach, thanks a lot for reporting this problem! This is caused because I forgot to package the new bloom file your talking about in our tarball.

This is a usual problem with new file types: they need to be explicitly declared in our setup.py to be added to our tarball, wheel and conda package.

I'll release a new beta (beta7) as soon as possible with a fix for this.

[ccordoba12]

particularly if they already affirmatively clicked dismiss already on the popup ad for Kite that appears on Spyder launch, as I did

It'll be possible to also dismiss this ad in the ad itself. However, if users find it too intrusive, we'll disable it before the final release.

[CAM-Gerlach]

Great, thanks! Glad I could be of help. In the meantime you can just tell users to disable the ads if they don't have Kite. Might also be a good idea to try/except the error in case there is any problem loading the file and just skip showing the ad so if it does stop getting packaged or something else goes wrong there aren't serious consequences for users (since your tests can't catch that).

Just FYI, in addition to not being included in the packages at all, the relevant code here uses a crude and hacky method using __file__ to load the data file that will break under some packaging methods. It would be best to use pkg_resources (or eventually importlib.resources) but if Spyder's already loading binary resources from inside the package using this hack without issue under the packaging types you distribute then I figure it isn't a big deal.

[ccordoba12]

In the meantime you can just tell users to disable the ads if they don't have Kite

The purpose of those ads is to show the relevance of Kite in cases where completions can't be obtained through the PyLS. But it'll be very easy to dismiss them.

[CAM-Gerlach]

The purpose of those ads is to show the relevance of Kite in cases where completions can't be obtained through the PyLS. But it'll be very easy to dismiss them.

I meant to avoid this bug, sorry, if any other users have this issue before you can get Beta 7 out.

Once I actually got the ads working switching back to the dev build disabling them did work, and unlike the preferences option it disabled them immediately without a restart.

[CAM-Gerlach]

However, if users find it too intrusive, we'll disable it before the final release.

As you are probably aware uses have often reacted swiftly and violently to Kite's ads in other similar open-source packages they've sponsored in the past, and while the situation is much more subtle and much better handled here (due in large part to you and @goanpeca 's careful attention to the UX and the UI text, based on what I noticed in other PRs), the lighter the better as far as keeping users content is concerned though that has to be balanced with any contractual obligations to Kite.

As a compromise between disabling them completely and the status quo, you could have the dismiss button (perhaps renamed "Don't show again") on the first page of the Kite install UI that users see on first launch also disable the Kite completion ads, since the user already read about Kite and decided they didn't want it, whereas simply closing the dialog or clicking another button keeps them enabled until the user sees and optionally disables them. But either they are very easy to perma-disable the first time a user sees them, its more about optics and impressions than practical UX.

[ccordoba12]

As a compromise between disabling them completely and the status quo, you could have the dismiss button (perhaps renamed "Don't show again")

The problem is users wouldn't be able to see the real added value Kite provides when facing a completion that only Kite can return.

The ads require just an additional dismiss besides the initial one (which I think is not too much to ask).

[CAM-Gerlach]

Yeah, from a practical standpoint its a very minimal UX impact. Its just that some users can get all up in arms about it so you'll have to play it by ear. Hopefully you can get good feedback during the beta (or no feedback being the best feedback) so you can make the best call by the final release. As much as I have an idealistic disdain for plugging a commercial non-free product, pragmatically speaking better completions is the number 1 thing Spyder's users have asked for so you are really just giving them what they want.

I just hope PyLS and Spyder's open-source completion system doesn't suffer from neglect because of it especially considering Spyder is also the maintainers of that now.

[ccordoba12]

Yeah, from a practical standpoint its a very minimal UX impact. Its just that some users can get all up in arms about it so you'll have to play it by ear

If our users reaction is really bad then we'll remove it.

[rgommers]

Technical comment: this kind of packaging issue happened to me all the time with NumPy and SciPy, a good way to avoid it is to have one CI entry go via sdist creation and isolated install from tarball (e.g. https://github.com/scipy/scipy/blob/master/.travis.yml#L223)

[ccordoba12]

That's a good idea, thanks @rgommers! An additional problem in this case is that we didn't add a test for the call-to-action, which would have allowed us to really catch this bug.

[CAM-Gerlach]

Yeah, especially for an end-user application like Spyder that has a nontrivial packaging setup and has had wheels break in the past as I recall, it might make your job easier to have something like that and avoid having to put you through emergency releases like this.

Just to clarify, a test would only have caught this if it had been running on a CI that actually installed from a packaged distribution rather than from source, where it worked just fine.

[NaderNazemi]

I hope this is the right place to post this.
I have upgraded to spyder 4.0 and installed kite.
I am experiencing severe slowdowns.
I would appreciate some guidance.
Thank you

[rgommers]

@NaderNazemi that's different from a crash, which is what this issue is about. So please open a new issue, and add as much detail as you can. "I am experiencing severe slowdowns" isn't enough to go on, please add:

• exactly what do the slowdowns look like, and under what circumstances
• how did you install Kite and Spyder
• add a list of other things installed (e.g. conda list or pip list)
• if possible, a screen capture of what is happening

[ccordoba12]

Thanks @rgommers for your input!

@NaderNazemi, most serious slowdowns were already fixed and they'll be part of 4.0.1, to be released shortly.

[NaderNazemi]

Thanks @rgommers for your input!

@NaderNazemi, most serious slowdowns were already fixed and they'll be part of 4.0.1, to be released shortly.

I appreciate the update and all the efforts you have put into this. I keep getting the "Code Completion and ..." message. Looking forward to the release.

[ccordoba12]

@NaderNazemi, that's a duplicate of #11026 and we're working on a fix for it right now.