Che: Support latest VS Code Python Extension (with Jupyter)

[l0rd]

Is your enhancement related to a problem? Please describe.

Latest python extension includes jupyter support. I have tried it out but loading a jupyter notebook failed.

To try it out on che.osio -->

Describe the solution you'd like

This is the devfile I have been using:

devfile.yaml
---
apiVersion: 1.0.0
metadata:
  name: python-workspace
projects:
  - name: jupyter-hello-world1
    source:
      location: 'https://github.com/chasbecker/TextAnalysis.git'
      type: git
components:
  - type: chePlugin
    alias: ms-python/python/latest
    reference: https://gist.github.com/l0rd/7190c65773a23f393a839598b22f3c1a/raw/meta.yaml
    memoryLimit: 512Mi
  - type: cheEditor
    alias: che-theia
    reference: https://gist.github.com/l0rd/7190c65773a23f393a839598b22f3c1a/raw/theia.yaml
    memoryLimit: 512Mi
  - type: chePlugin
    alias: machine-exec
    id: eclipse/che-machine-exec-plugin/latest

vscode python meta.yaml

vscode-python-meta.yaml
---
apiVersion: v2
publisher: ms-python
name: python
version: 2020.4.76186
type: VS Code extension
displayName: Python
title: Python extension
description: Linting, Debugging (multi-threaded, remote), Intellisense, code formatting,
  refactoring, unit tests, snippets, and more.
icon: https://www.eclipse.org/che/images/logo-eclipseche.svg
repository: https://github.com/Microsoft/vscode-python
category: Language
firstPublicationDate: '2020-04-27'
spec:
  containers:
  - image: "quay.io/eclipse/che-sidecar-python:3.7.3-8f39348"
    name: vscode-python
    memoryLimit: '512Mi'
  extensions:
  - https://github.com/microsoft/vscode-python/releases/download/2020.2.62710/ms-python-release.vsix

Describe alternatives you've considered

I have also tried to add python in che-theia container:

FROM quay.io/eclipse/che-theia:7.12.0

USER root

RUN apk add --no-cache python3

# make some useful symlinks that are expected to exist
RUN cd /usr/bin \
    && ln -s idle3 idle \
    && ln -s pydoc3 pydoc \
    && ln -s python3 python \
    && ln -s python3-config python-config

USER theia

But it didn't help

Sub-tasks

• [x] vscode.workspace.fs API https://github.com/eclipse-theia/theia/pull/7908
• [x] Adapt Che Theia to the new Theia FS API https://github.com/eclipse/che/issues/17357
• [ ] Che Theia Web View doesn't work properly when a plugin is running in its own sidecar https://github.com/eclipse/che/issues/16870

[ericwill]

I was working on upgrading the Python plugin today (it's almost a year out of date) as part of #16723. Thanks for doing some more investigation, once I get the Python extension upgraded (there are some bugs at the moment) it will be quite helpful.

[JPinkney]

Just wanted to note, we might be missing some plugin api: https://github.com/eclipse-theia/theia/issues/3186#issuecomment-610356821 for the newest version of the python plugin. There seems to be a few other Jupyter issues on the theia repo as well: https://github.com/eclipse-theia/theia/issues?q=is%3Aissue+is%3Aopen+Jupyter

[ericwill]

Just wanted to note, we might be missing some plugin api: eclipse-theia/theia#3186 (comment) for the newest version of the python plugin. There seems to be a few other Jupyter issues on the theia repo as well: https://github.com/eclipse-theia/theia/issues?q=is%3Aissue+is%3Aopen+Jupyter

Yes, jupyter issues aside the latest python extension also complains about Python not being installed, even though it's present in the container and running python code still works. Lots of weirdness going on there.

[azatsarynnyy]

Yes, jupyter issues aside the latest python extension also complains about Python not being installed, even though it's present in the container and running python code still works.

It looks like a known issue https://github.com/Microsoft/vscode-python/issues/11870

[azatsarynnyy]

The missing parts of upstream Theia are tracked here:

• https://github.com/eclipse-theia/theia/issues/8022
• https://github.com/eclipse-theia/theia/issues/3186#issuecomment-610356821

[azatsarynnyy]

Editors Team will work on adding CustomEditor API support (https://github.com/eclipse-theia/theia/issues/6636)
as Python extension relies on it for rendering Jupyter Notebooks.

[azatsarynnyy]

Several updates:

1. After a more detailed investigation, it turned out that Python VS Code extension needs support for CustomEditor Plugin API only as an experimental feature that is not used by default. So, it's not a blocker for Jupyter Notebooks support.

2. There's WIP PR https://github.com/eclipse-theia/theia/pull/7908 (FS Plugin API) in Theia upstream that fixes running Jupyter Notebooks in vanilla Theia. Che Theia isn't compatible with these changes and needs to be adapted https://github.com/eclipse/che/issues/17357.
   I fixed the compilation errors in Che Theia (https://github.com/eclipse/che-theia/pull/798) and tried to build Che Theia based on Theia from upstream Theia WIP PR branch, but it doesn't fix this issue - ... is not a valid file name

So, we will postpone work on CustomEditor Plugin API and start investigating why new FS Plugin API fixes Jupyter Notebooks support in vanilla Theia but not in Che Theia.

[azatsarynnyy]

After adapting Che Theia to the latest Theia FS Plugin API https://github.com/eclipse/che-theia/pull/823 there's some progress with the Jupyter Notebooks support :tada:

It looks like working well when the Python VS Code extension is running in the Theia container

But it doesn't when it's running in a separate sidecar. I think it requires adapting the WebView API to work remotely https://github.com/eclipse/che/issues/16870
So, it will be the next step.

[azatsarynnyy]

I'm converting it into an epic since multiple steps are required.
See the sub-tasks in the description.

[sub-mod]

Just saw this Introducing the Jupyter Extension for VS Code from microsoft and I wanted to check if it changes anything for us or if there is anything new which might break for us ? @azatsarynnyy

[azatsarynnyy]

Thanks @sub-mod for the link!

They split Python extension by extracting Jupyter related functionality into a separate extension.
For Che, it means both extensions have to be packed into a single Che Plug-in. So, actually, nothing should be changed for us :slightly_smiling_face:
We continue adapting Che Theia to support loading WebView from a sidecar.

What will really require extra effort is adding VS Code compatible Native Notebook API into upstream Theia.
It's not required yet. But I saw they're experimenting with it in Jupyter extension, so we'll need the support of it as well in order to use the future versions.