Delta: Create a PyPI release for Python

[gwax]

Could you please release the Python API via the Python Package Index (PyPI)

It would help adoption and development flows to be able to install via pip instead of having to copy the source tree into projects.

Since the API seems compatible, this would be especially useful for developers testing and linting locally while running against the Databricks API.

Thank you.

[abij]

+1
I would this as well for the same reasons.

[DdMad]

+1 I would like to have this for the same reason.
Also, for those who is suffering copying the python source code, you can use sparkContext.addPyFile(/path/to/delta-core.jar) to load the python code.

[pedro-bralv]

+1
Does anyone know if someone is working on this?

[tdas]

We folks at databricks are not working on this yet, we are currently swamped with Spark 3.0 migration for Delta Lake. It would be awesome if someone from the community starts working on this.

[tdas]

@gwax you said that "It would help adoption and development flows to be able to install via pip instead of having to copy the source tree into projects." Can you explain why do you have to copy the source tree into the project? Essentially why isnt pyspark/spark-submit --packages io.delta:delta-core_2.11:0.5.0 enough?

Also, @abij, if you can chime in as well.

[nicktgr15]

As mentioned in the PR (https://github.com/delta-io/delta/pull/353), a workaround for this is the following

Apart from the above though, I realised there's a workaround to solve this problem, without releasing a pypi package. e.g. https://github.com/Azure/mmlspark

spark = pyspark.sql.SparkSession.builder.appName("MyApp") \
            .config("spark.jars.packages", "com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc1") \
            .config("spark.jars.repositories", "https://mmlspark.azureedge.net/maven") \
            .getOrCreate()
import mmlspark

were the session is created first and afterwards you can import mmlspark.

In my case with delta (when using spark.jars.packages) the following doesn't work because delta is not installed

from delta.tables import DeltaTable

def delta_table_exists(path):  
    return DeltaTable.isDeltaTable(identifier=path, sparkSession=get_spark_session())

however, the following works fine (because the delta import is executed inside the spark session).

def delta_table_exists(path):
    from delta.tables import DeltaTable
    return DeltaTable.isDeltaTable(identifier=path, sparkSession=get_spark_session())

Of course static code analysis doesn't work very well with this approach.

[gwax]

@tdas the issue is not the limitation of being able execute our code within a spark environment. As you say, and I have done, adding io.delta:delta-core_2.11:0.5.0 as a package to the executor makes things run just fine.

The issue is with my development environment. I use quite a few static analysis tools and linters (pylint, mypy, etc.) that depend on analyzing library code. These tools exist both to improve my code quality / decrease bugs and to power my code completion / intelligence workflow. This code is run in a local virtual environment (in my case managed by pyenv and VSCode) and does not have access to all of the components added to a pyspark session.

The workaround, for now, is to copy the source tree into a "vendor" path and add that path to my PYTHONPATH; this works but it quite hacky on my end and hard to share with other developers on my team. Additionally, it means that I have to manage version changes manually instead of depending on the tooling that pip and the general Python ecosystem provides.

The specific ask in this case is to follow the Packaging Python Projects guide, add setup.py as delta/python/setup.py, create a PyPI account, and publish releases as they occur.

I, and I suspect others, would be happy to contribute to the details of managing the setup but I cannot publish an "official" package and would feel gauche publishing an unofficial package without first exhausting attempts to encourage an official release.

[SpaceRangerWes]

+1. Would be a huge dev unlock.

[tdas]

@gwax I understand your argument for Pypi now. Thank you very much for clarifying that static code stuff will not work. Unfortunately, as discussed in more details here, the pyspark community hasnt quite figured how to do this cleanly, that does not require manual fiddling of versions. The main challenge that there is no well-defined way to package jar files inside wheels such that when the wheel is installed via pip, the jars are automatically discovered by the jvm of spark. Some python-based libraries with jar dependencies have released pip containing only python files and then rely on sparkContext.addJar(xxx.version) to explicitly add the jars of the correct version to the jvm. Obviously, this manual syncing of the versions in pip and in code is a management nightmare.

So in summary, there is a well-defined solution for this which brings us to this conundrum whether to make a pip release that enables some things (static analysis) but also enables a whole plethora extremely-hard-to-debug problems.

[SpaceRangerWes]

@tdas is there at least an intermediate solution that allows developers to create some kind of python package locally for better IDE integration and testing during pyspark development? I think even a list of complicated instructions is better than nothing.

[Fokko]

+1

Would be great to have this available. We're working with Databricks connect, and having this would give us Python autocomplete on the development machine.

[alextmeyer]

+1 this would also be useful to us.

[saravp123]

I tried the workaround given here and it works for
DeltaTable.isDeltaTable(identifier=path, sparkSession=get_spark_session())

But when I use DeltaTable.forPath(spark,'/path/to/delta-file'), I am getting below error:
py4j.protocol.Py4JJavaError: An error occurred while calling z:io.delta.tables.DeltaTable.forPath.
: java.lang.UnsupportedOperationException

Has anyone used DeltaTable.forPath() from PyCharm? If so, can you please share the steps you followed to use this?

[prasannakumar2012]

I just created an example showing how to use it in Google Colab/Jupyter Notebook.

https://github.com/prasannakumar2012/spark_experiments/blob/master/examples/Delta_Lake.ipynb

[kalbes]

I also had to waste time doing hack... I couldn't believe there wasn't a library! This makes the development experience a lot worse.
This is basic stuff, not a nice to have. I had never seen a library from a major project that is not published to a package indexer. And delta has already been around for a while...