Koalas: How to convert back from Koala dataframe to Databricks dataframe

[rodneyjoyce]

Hi, this maybe a silly question - I can convert from my Databricks dataframe to Koalas using
cleanKoalaDf = ks.from_pandas(cleanDf.toPandas())

I then do some processing (E.g. get_dummies). Now I want to convert back to a Databricks dataframe but I can't find the appropriate method?

[garawalid]

I don't understand what do you mean by Databricks dataframe. But, from the example you gave, I can guess that cleanDf is a spark DataFrame.

Here is a small example of how to move between Koalas DataFrame, Pandas DataFrame and Spark DataFrame. Hope it helps.

pdf means Pandas DataFrame
kdf means Koalas DataFrame
sdf means Spark DataFrame

import databricks.koalas as ks
import pandas as pd
from pyspark.sql import SparkSession

pdf = pd.DataFrame({
    'a': [1, 2, 3, 4, 5, 6, 7, 8, 9],
    'b': [4, 5, 6, 3, 2, 1, 0, 0, 0],
}, index=[0, 1, 3, 5, 6, 8, 9, 9, 9])

# Pandas DataFrame <=> Koalas DataFrame 

kdf = ks.from_pandas(pdf)

pdf = kdf.toPandas()

print(pdf.head())


# Spark DataFrame => Koalas DataFrame => Pandas DataFrame

spark = SparkSession.builder.getOrCreate()

sdf = spark.createDataFrame(pdf)

kdf = ks.DataFrame(sdf)

pdf = kdf.toPandas()

print(pdf.head())

[rodneyjoyce]

Yes, thanks - that is what I was looking for - so I first convert to Pandas and then to a Spark dataframe.

[richardagalvez]

Hi, this maybe a silly question - I can convert from my Databricks dataframe to Koalas using
cleanKoalaDf = ks.from_pandas(cleanDf.toPandas())

I then do some processing (E.g. get_dummies). Now I want to convert back to a Databricks dataframe but I can't find the appropriate method?

You need not go from koalas -> pandas -> spark dataframe. You can go directly from a koalas dataframe to spark via the to_spark() method: (https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.to_spark.html).

[HyukjinKwon]

I wrote some documentations for this :-) https://koalas.readthedocs.io/en/latest/user_guide/pandas_pyspark.html