Koalas: String concatenation does not work correctly

[sadikovi]

When concatenating columns and assigning to another column works for numeric columns, it does not work for strings:

Works for numeric columns:

df["a_b"] = df["a_b"] + df["a_b"]
df.toPandas()

# Out[29]: 
#    id  a.b  a_b
# 0   0    1    4

Does not work for strings:

from pyspark.sql.functions import lit
spark.range(1).withColumn("a", lit("abc")).withColumn("b", lit("def")).withColumn("c", lit("+")).coalesce(1).write.mode("overwrite").parquet("/mnt/ivan/sample2.parquet")

df = ks.read_parquet("/mnt/ivan/sample2.parquet")
df["c"] = df["a"] + df["b"]
df.toPandas()

# Out[36]: 
#    id    a    b     c
# 0   0  abc  def  None

However, this works in Pandas:

df = ks.read_parquet("/mnt/ivan/sample2.parquet").toPandas()
df["c"] = df["a"] + df["b"]
df

# Out[38]: 
#    id    a    b       c
# 0   0  abc  def  abcdef

[rxin]

Want to submit a PR?

[sadikovi]

Yes, I will do.