Delta: How I can run this is jupyter notebook?

[pyMixin]

I want to save imported CSV files in delta lake format? How can I do this in jupyter? Has anybody tried?

[bachwehbi]

With Spark in standalone mode, the following workes for me:

instruct pyspark to launch Jupyter

export PYSPARK_DRIVER_PYTHON="jupyter"
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"

Then run pyspark:

pyspark --packages io.delta:delta-core_2.12:0.1.0

Then I can work with delta in the notebook.

[pyMixin]

I am getting error

Py4JJavaError: An error occurred while calling o39.save.
: java.lang.ClassNotFoundException: Failed to find data source: delta. Please find packages at http://spark.apache.org/third-party-projects.html

Where you are running this?
pyspark --packages io.delta:delta-core_2.12:0.1.0

I am trying to put in my bash profile

export PYSPARK_SUBMIT_ARGS= "--packages io.delta:delta-core_2.12:0.1.0 pyspark-shell"

[pyMixin]

Here is my profile:

export SPARK_PATH=~/spark-2.4.2-bin-hadoop2.7
export PYSPARK_DRIVER_PYTHON="jupyter"
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
export PACKAGES="io.delta:delta-core_2.12:0.1.0"
export PYSPARK_SUBMIT_ARGS="--packages ${PACKAGES} pyspark-shell"
#For python 3, You have to add the line below or you will get an error
export PYSPARK_PYTHON=python3
alias snotebook='$SPARK_PATH/bin/pyspark --master local[2]'

[pyMixin]

Figured it out.. here is the solution:

export SPARK_PATH=~/spark-2.4.2-bin-hadoop2.7
export PYSPARK_DRIVER_PYTHON="jupyter"
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
export PACKAGES="io.delta:delta-core_2.12:0.1.0"
export PYSPARK_SUBMIT_ARGS="--packages ${PACKAGES} pyspark-shell"
#For python 3, You have to add the line below or you will get an error
export PYSPARK_PYTHON=python3
alias snotebook='$SPARK_PATH/bin/pyspark --packages io.delta:delta-core_2.12:0.1.0 --master local[2]'

not sure if you need these two lines;

export PACKAGES="io.delta:delta-core_2.12:0.1.0"
export PYSPARK_SUBMIT_ARGS="--packages ${PACKAGES} pyspark-shell"

Complete instructions here: https://github.com/pyMixin/DeltaLake/blob/master/Delta%20Lake%20on%20Jupyter%20Notebooks.ipynb

[CyborgDroid]

Here is another way:

from pyspark.sql import SQLContext
from pyspark import SparkContext
import os

os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages io.delta:delta-core_2.12:0.1.0 pyspark-shell'

sc = SparkContext()

spark = SQLContext(sc)