Delta: [DML] Scala/Java API for OPTIMIZE

[tdas]

This tracks the ongoing work and discussions for building the Scala/Java APIs for optimizing the data layout of Delta tables by coalescing small files into larger files.

[marmbrus]

At this point, there are no plans to open-source the OPTIMIZE command, as the actual implementation is pretty deeply tied to other functionality that is only present in Databricks Runtime.

That said, it is really easy to coalesce small files into bigger, more efficient ones as long as you have the power of ACID transaction. The real hard part of this operation is making sure you do not corrupt the table while updating it in-place, and this comes out of the box with Delta!

Example:

val table = "..."
val partition = "year = 2019"
val numFiles = 10

spark.read
  .format("delta")
  .load(table)
  .where(partition)
  .repartition(numFiles)
  .write
  .format("delta")
  .mode("overwrite")
  .option("replaceWhere", partition)
  .save(table)

[btbbass]

Does that imply that if I am using the delta table as a stream input, the rewritten files are not taken into consideration by the stream job as the original file where already processed ?

EDIT: I did some test, and that seem not to be the case, that is, the streaming job will either:

• Fail, if 'ignoreChanges' option is set to False (default value)
• Reproccess data, if 'ignoreChanges' option is set to True

[marmbrus]

@btbbass that is an excellent point and I am missing something in that example. Thanks for pointing that out!

We need to expose the ability for writes to be marked as "rearranging data only" in order to avoid breaking streams that are querying this data. This is the API that OPTIMIZE uses internally and we should expose that in OSS too.

I opened #146 to track, should be pretty easy to implement as all the protocol support is already there.

[MrPowers]

You need to add single quotes to val partition = "year = '2019'" to make Michael's answer work (for anyone else silly enough to get caught on this).

Thanks for the great answer @marmbrus and happy Delta makes this so easy!