Delta: no support for dynamic partition overwrite

[ivoson]

There is no support for dynamic partition overwrite for now. It seems we can only overwrite with replaceWhere option to specify the overwrite part, which is not convenient.
I am not sure why we don't have dynamic overwrite support in delta, is there any limits about it?

[tdas]

Fundamentally, dynamic partition overwrite is not a great API because it is quite scary; you can accidentally overwrite a partition with bad records falling in partitions that shouldnt have been touched. So it is always safer to use replaceWhere to provide guard rails on what partitions should get overwritten.

In the future, we can always add support for dynamic partition overwrite. But I would still recommend using replaceWhere.

[ivoson]

Got it and Thanks for your explain @tdas

[koertkuipers]

note that the risk of bad records overwriting partitions is zero if you do something like

  .filter($"x" == 1)
  .write
  .format("delta")
  .option("partitionOverwriteMode", "dynamic")
  .partitionBy("x")
  .save("/some/location")

now you might argue that in such a case i can use replaceWhere and you are right.
but what is the logic is split up across jobs?
i can have one job doing the filtering and writing somewhere, and another job reading this data back in and doing the dynamic partition overwrite into delta. now the replaceWhere is suddenly cumbersome (the filter logic sits in another job, i now have to go repeat it), and this is still safe isn't it? we have this situation pretty frequently.

yes you can do dumb things with dynamic partition overwrite, but also very useful things.
removing it hurts adoption in my opinion.
(we added it back in internally because rewriting logic in many places wasnt an option)

[marmbrus]

It is not really that you cannot do dumb things with replaceWhere. Its that the action being taken is explicit rather than implicit.

More over, up until now, every one who has asked for the feature was happy with replaceWhere. As such its never been a priority to add it.

If you got an implementation you should consider opening a PR.

[koertkuipers]

opened PR here:
https://github.com/delta-io/delta/pull/371

[gdoron]

@tdas to be honest, breaking the normal API of spark 2, instead of .mode("overwrite") only overwrites the dynamic partitions in the df, it deletes the entire table. That's scary.

It's so easy to write overwrite when you are used to working with parquets table and forgetting adding the replaceWhere and boom, the table is gone.

Maybe that's why you added the time machine... because you knew it would happen ;)