Vscode: 📢 Notebook API announcements

[rebornix]

We introduced a proposed API for Notebook but currently the API is majorly two managed object: NotebookDocument and NotebookCell. We create them for extensions and listen to their properties changes. However this doesn't follow the principal of TextEditor/Document where TextDocument is always readonly and TextEditor is the API for applying changes to the document.

If we try to follow TextEditor/Document, the API can be shaped as below

export interface NotebookEditor {
    readonly document: NotebookDocument;
    viewColumn?: ViewColumn;
    /**
     * Fired when the output hosting webview posts a message.
     */
    readonly onDidReceiveMessage: Event<any>;
    /**
     * Post a message to the output hosting webview.
     *
     * Messages are only delivered if the editor is live.
     *
     * @param message Body of the message. This must be a string or other json serilizable object.
     */
    postMessage(message: any): Thenable<boolean>;

    /**
     * Create a notebook cell. The cell is not inserted into current document when created. Extensions should insert the cell into the document by [TextDocument.cells](#TextDocument.cells)
     */
    createCell(content: string, language: string, type: CellKind, outputs: CellOutput[], metadata: NotebookCellMetadata): NotebookCell;

    /**
     * Insert/Delete cells from the document
     */
    spliceCells(index: number, deleteCnt: number, insertedCells: NotebookCell[]): Promise<void>;

    /**
     * Make changes to individual cells
     */
    applyCellEdits(cell: NotebookCell, changes: { language?: string, outputs?: CellOutput[], metadata?: NotebookCellMetadata }): Promise<void>;
}

export interface NotebookDocument {
    readonly uri: Uri;
    readonly fileName: string;
    readonly isDirty: boolean;
    readonly languages: string[];
    readonly cells: NotebookCell[];
    readonly displayOrder?: GlobPattern[];
    readonly metadata?: NotebookDocumentMetadata;
}

export interface NotebookCell {
    readonly uri: Uri;
    readonly handle: number;
    readonly language: string;
    readonly cellKind: CellKind;
    readonly outputs: CellOutput[];
    readonly metadata?: NotebookCellMetadata;
    getContent(): string;
}

[roblourens]

Does the edit API need to support reordering? Like could there any transient state in webview cell outputs that would not be represented in CellOutput?

[roblourens]

If I set changes.metadata to something, am I overwriting all of metadata or merging in the properties that are set?

And I think metadata should be non-optional, even if all its properties are, it makes lots of things simpler.

[rebornix]

Does the edit API need to support reordering?

Extensions already use splice to do reordering since Cell are created through NotebookEditor.createCell so Cells are managed objects. We need to make sure they are not disposed during splice.

Like could there any transient state in webview cell outputs that would not be represented in CellOutput?

This should probably be done by output in the webview sending its state through webview api to extensions.

If I set changes.metadata to something, am I overwriting all of metadata or merging in the properties that are set?

It should be override IMHO.

And I think metadata should be non-optional, even if all its properties are, it makes lots of things simpler.

I agree making metadata non-optional is clear to both us (for implementation) and extensions author (maybe) but it will make cell-default-metadata on document impossible, unless we use inherit | enable | disable enum instead of boolean.

[jrieken]

We could make this even more similar to TextEditor#edit by also having an edit-callback which in the only place to create new cell, delete cell etc. Something like this

export interface NotebookEditorCellEdit {
    insert(content: string, language: string, type: CellKind, outputs: CellOutput[], metadata: NotebookCellMetadata | undefined): void;
    delete(cell: NotebookCell): void;
    // more
}

export interface NotebookEditor {
    readonly document: NotebookDocument;
    viewColumn?: ViewColumn;

    readonly onDidReceiveMessage: Event<any>;

    postMessage(message: any): Thenable<boolean>;

    editCells(callback: (cellEdit: NotebookEditorCellEdit) => any): any
}

[roblourens]

I agree making metadata non-optional is clear to both us (for implementation) and extensions author (maybe) but it will make cell-default-metadata on document impossible, unless we use inherit | enable | disable enum instead of boolean.

Right, I mean having metadata non-optional but its properties can be optional, where undefined means "inherit"

[rebornix]

Current state of the Notebook API: we added Document Edit API and make document.cells readonly. All changes to the cells should be performed through the Edit API

export interface NotebookEditorCellEdit {
    insert(index: number, content: string, language: string, type: CellKind, outputs: CellOutput[], metadata: NotebookCellMetadata | undefined): void;
    delete(index: number): void;
}

export interface NotebookEditor {
    readonly document: NotebookDocument;
    viewColumn?: ViewColumn;
    readonly onDidReceiveMessage: Event<any>;
    postMessage(message: any): Thenable<boolean>;
    edit(callback: (editBuilder: NotebookEditorCellEdit) => void): Thenable<boolean>;
}

The renderer API is also simplified as

export interface NotebookOutputRenderer {
    render(document: NotebookDocument, output: CellOutput, mimeType: string): string;
    preloads?: Uri[];
}

Note that we didn't pass NotebookEditor to renderers, instead they can only get the NotebookDocument. This is because The same document can be opened in two editors but the document is shared. My current thinking is custom renderers don't care about which editor the transformed output (html) will be rendered in and it also doesn't talk with the output through webview api because it doesn't know which editor the document will be loaded in.

[connor4312]

When implementing it the NotebookOutputRenderer felt a little awkward. The types indicated that I need check the output kind and check that rich output had data matching the mime type I care about... and throw an error if it's not right? Or return an empty string? Hard to know. It's also a little confusing to me because it seems like render will only be called for my own mime types, not CellStreamOutput/CellStreamInput. This was my boilerplate:

render(document: vscode.NotebookDocument, output: vscode.CellOutput): string {
  if (output.outputKind !== vscode.CellOutputKind.Rich) {
    return '';
  }

  if (output.data[Constants.TableMimeType] === undefined) {
    return '';
  }

  // actually do stuff...
}

Maybe the NotebookOutputRenderer's render function could be render(document: NotebookDocument, source: string, mimeType: string): CellOutput, where vscode would only call the renderer and pass in the source when the cell mime type matches. I also like the idea proposed of returning a CellOutputKind--gives me an easy way to deal with errors and higher-order renderers might be interesting.

[rebornix]

@connor4312 thanks for the feedback. you are right that customs renderers should not check cellKind as we are invoking custom renderers only for CellDisplayOutput.

The API is changed to

export interface NotebookOutputRenderer {
  render(document: NotebookDocument, output: CellDisplayOutput, mimeType: string): string;
  preloads?: Uri[];
}

No functionality changes under the hood, just making the type safe. We can discuss about whether we should return CellDisplayOutput in next API meeting.

[rebornix]

Discussion points for coming API sync (April 14th)

• ~NotebookCell.source should be string[] other than string. Firstly we are storing contents as string[] thus returning string means we need to do concatenation but it can be overhead as extensions might need to split the NotebookCell.source into individual lines again before saving to disk.~
• NotebookProvider.save(document: NotebookDocument): Promise<boolean>; should the first argument be editor: NotebookEditor just in case the notebook provider wants to talk to the webview?
• Cell content update API

export interface NotebookEditorCellEdit {
    insert(index: number, content: string | string[], language: string, type: CellKind, outputs: CellOutput[], metadata: NotebookCellMetadata | undefined): void;
    delete(index: number): void;
+   replace(index: number, value: string): void;
}

[jrieken]

NotebookCell.source should be string[] other than string. Firstly we are storing contents as string[] thus returning string means we need

Not a fan, we should not bleed implementation details into the API and there is no obvious reason why cells should be kept as lines. Also, this would be different from the text document content provider API which returns a string. If performance is of outmost matter here then we should consider talking bytes instead of strings

[sguillia]

I am wondering if you are planning to have all cells one below the other, just like in a notebook.

Or if we will be able to have one cell occupy 50% of the webview width, as customizable as an HTML textarea.

I believe extensions can benefit from having native editors and HTML content side-by-side, even if having native editors is already a great win.

[rebornix]

@sguillia the current UX for the native notebook will be a list of cells, or in your words, one cell below the other.

[JeffreyCA]

With the stable API, Is it currently possible to create a webview like the Interactive Playground, with Markdown interleaved with editable code areas?

[rebornix]

@JeffreyCA short answer is no. We plan to move the core Interactive Playground to Notebook API in the future, extensions should be able to do that as well.

[rebornix]

@colombod @davidanthoff @DonJayamanne @sbatten notebook enthusiasts: we will use this meta issue to track the changes/proposals in the Notebook API. Discussions around details/individual changes will still be in separate issues but will be linked here.

[davidanthoff]

@rebornix I kept the Julia extension PR for notebooks in line with all the API changes over time, but now I have one question: is there some discussion of the whole kernel as a separate provider idea? I think I don't fully understand that part.

[rebornix]

@davidanthoff we didn't have a discussion tracked online for the kernel concept. While @connor4312 building a jupyter/xeus notebook sample, we found that we don't have a way to allow you to choose which kernel to use. Thus we separated them and allow multiple kernel contributions. But we still need more thoughts on this one as it's not clear (meaning we don't have concrete answer yet) for how the kernel picker should look like, how to manage the lifecycle of a kernel (connect, initialize, disconnect, etc).

[jrieken]

fyi these changes have been made

• _(breaking)_ removed NotebookCell.source in favor of NotebookCell.document.getText()
• _(breaking)_ removed notebook.activeNotebookDocument in favour of notebook.activeNotebookEditor.document
• _(added)_ notebook.notebookDocuments

[rebornix]

fyi API changes for tomorrow's Insiders

• (breaking) NotebookOutputSelector { type, subType } is now changed to NotebookOutputSelector { mimeTypes: string[] }
• (breaking) NotebookOutputRenderer.render is now changed to `render(document: NotebookDocument, request: vscode.NotebookRenderRequest) (https://github.com/microsoft/vscode/pull/100032)
• NotebookContentProvider now supports hot exit

revertNotebook(document: NotebookDocument, cancellation: CancellationToken): Promise<void>;
backupNotebook(document: NotebookDocument, context: NotebookDocumentBackupContext, cancellation: CancellationToken): Promise<NotebookDocumentBackup>;

[TylerLeonhardt]

Hi all - just found out about vscode.NotebookKernel.

function vscode.notebook.registerNotebookKernel(id: string, selectors: vscode.GlobPattern[], kernel: vscode.NotebookKernel): vscode.Disposable

Is there a reason this requires a selectors: vscode.GlobPattern[] when (if you're using a vscode.NotebookProvider as well) that glob is already in the package.json:

{
"notebookProvider": [
      {
        "viewType": "PowerShellNotebookMode",
        "displayName": "Powershell Notebook",
        "selector": [
          {
            "filenamePattern": "*.ps1"
          }
        ],
        "priority": "option"
      }
    ],
}

[DonJayamanne]

To my understanding this is because a kernel can be provided by another extension.
One extension can read/write notebooks, and another can provide the ability to execute it. Thus when providing a kernel, we need to let VSC know what files (notebooks) should be handled when executing.

[TylerLeonhardt]

Makes sense. Still would be nice to not have to write it twice if they are the same class.

[connor4312]

If you have a kernel and a content provider side by side, you can provide the kernel implementation directly in the content provider via

https://github.com/microsoft/vscode/blob/a5f071760497b4bd7cc01f5f36ede8b0ca8555e5/src/vs/vscode.proposed.d.ts#L1768

[TylerLeonhardt]

Yep - that's what I'm doing today but also have heard that there's still discussion on that mechanism.

[rebornix]

@TylerLeonhardt created a github action which pulls in the changes of notebook API in vscode.proposed.api and checks if the build is broken or not https://github.com/TylerLeonhardt/vscode-powershell/blob/notebook-ui-support/.github/workflows/updateNotebookApi.yml . It can help with checking if existing API has breaking changes and can be used as a good hint for picking up API updates.

cc @DonJayamanne @colombod

[rebornix]

If you are writing integration tests for the notebook feature, you can now follow this sample
https://github.com/rebornix/vscode-extension-samples/tree/rebornix/notebook-test/notebook-test-sample to get it set up.

[roblourens]

Will also mention it here for any other people. Discussed a bit in https://github.com/microsoft/vscode/issues/99203, I am making these changes

• Removing CancellationToken from the execute APIs
• Adding cancelCellExecution and cancelAllCellsExecution
• Adding runState to the notebook document metadata, which is intended to be set to Running by the extension after executeAllCells has been called and the full notebook is running. That runState will control whether the document-level cancel button is visible

[rebornix]

@DonJayamanne @colombod @TylerLeonhardt

FYI changes to the notebook API, the extension can run successfully in Insiders today but if you try to build against latest proposed API, you might see an error so please update them.

Major changes:

• NotebookContentProvider#kernel is removed. The current recommended way of contributing kernels is through kernel provider. Feedbacks are welcome (for example, https://github.com/microsoft/vscode/issues/105376)
• registerNotebookContentProvider now takes a new parameter options which describes if the content provider will save outputs or a specific metadata to disk when saving. By sharing this information with the core, the core knows if it should mark the document dirty or push an undo/redo element in to the undo stack. Also this information can be used in enhanced text diff (https://github.com/microsoft/vscode/issues/99877#issuecomment-678857384)

export function registerNotebookContentProvider(
    notebookType: string,
    provider: NotebookContentProvider,
    options?: {
        /**
         * Controls if outputs change will trigger notebook document content change and if it will be used in the diff editor
         * Default to false. If the content provider doesn't persisit the outputs in the file document, this should be set to true.
         */
        transientOutputs: boolean;
        /**
         * Controls if a meetadata property change will trigger notebook document content change and if it will be used in the diff editor
         * Default to false. If the content provider doesn't persisit a metadata property in the file document, it should be set to true.
         */
        transientMetadata: { [K in keyof NotebookCellMetadata]?: boolean }
    }
): Disposable;

For example, you won't save outputs into disk and also don't care about execution related metadata, you can filter them out by

context.subscriptions.push(vscode.notebook.registerNotebookContentProvider('github-issues', notebookProvider, {
    transientOutputs: true,
    transientMetadata: {
        inputCollapsed: true,
        outputCollapsed: true,
        runState: true,
        runStartTime: true,
        executionOrder: true,
        lastRunDuration: true,
        statusMessage: true
    }
}));

With this approach, as a content provider, it doesn't need to handle metadata undo/redo or trigger additional content change events. @TylerLeonhardt @ colombod this might be useful in ps1/dib notebooks as they don't persist outputs or metadata.

[jrieken]

@DonJayamanne @colombod @TylerLeonhardt

Also, FYI that we are reworking/extending the notebook edit API. Cells, output, and metadata should only be changed via declared edits that are then applied by the core. See https://github.com/microsoft/vscode/issues/105283 for all the details.

[jrieken]

Plan to rename the activation event from onNotebookEditor:<viewType> to onNotebook:<viewType>. Next insider will support both for a while and then remove the old variant. Discussion and progress: https://github.com/microsoft/vscode/issues/105496

[jrieken]

/cc @renkun-ken

[jrieken]

fyi - removing NotebookDocument#displayOrder because there also is NotebookDocumentMetadata#displayOrder, https://github.com/microsoft/vscode/issues/106305

[jrieken]

fyi - we will continue to use this issue to announce notebook API changes, for the API evolution we use https://github.com/microsoft/vscode/issues/106744. Both issues are locked to guide discussions into separate issues.

[rebornix]

fyi, we moved notebook editor related properties and event listeners from vscode.notebook to vscode.window namespace

    export namespace window {
        export const visibleNotebookEditors: NotebookEditor[];
        export const onDidChangeVisibleNotebookEditors: Event<NotebookEditor[]>;
        export const activeNotebookEditor: NotebookEditor | undefined;
        export const onDidChangeActiveNotebookEditor: Event<NotebookEditor | undefined>;
        export const onDidChangeNotebookEditorSelection: Event<NotebookEditorSelectionChangeEvent>;
        export const onDidChangeNotebookEditorVisibleRanges: Event<NotebookEditorVisibleRangesChangeEvent>;
    }

[connor4312]

1. The postMessage and onDidReceiveMessage on the acquireNotebookRendererApi will be removed tomorrow (these have been non-functional since the move to pure renderers, so this should not be breaking.
2. I would like to make a change so that acquireVsCodeApi is exposed only to kernel preloads synchronously when they load in, and not accessible outside of that point. Please let me know if this change will cause issues for you. See here for the recommended path forwards: https://github.com/microsoft/vscode-docs/blob/vnext/api/extension-guides/notebook.md#interactive-notebooks (formalizing the approach that the Python team is already using)