Koalas: Clear Plots

[brookewenig]

When I plot a DataFrame, then in a separate cell plot again, the first plot is not cleared.

I think the plots should be auto-cleared (e.g. plt.clf()).

[harupy]

@brookewenig

I failed to reproduce this issue. Did I miss something?

Environment

• Databricks Community Edition
• Runtime version: 6.0 beta
• koalas version: 0.17.0
• notebook: click here

[harupy]

I digged into how display of 6.0 betaon Databricks works with the code below.

# print the source code of `display`
import inspect
lines = inspect.getsource(display)
print(lines)

The output says:

...

elif type(input).__module__ == 'matplotlib.figure' and type(input).__name__ == 'Figure':
    _format = 'png'
    extension = '.' + _format
    figureFile = NamedTemporaryFile(delete=False, suffix = extension)
    input.savefig(figureFile, format = _format, transparent = True)
    figureFile.close()
    # for pandasDF.plot to work well, need to close the figure, otherwise the same figure
    # will continued to be use across calls to pandasDF.plot.
    mpl.pyplot.close(input)
    self.figures.append(figureFile.name)
...

The full output is included in this notebook.

[harupy]

I also checked display of 5.5 conda beta (which I think is older than 6.0 beta)

...

elif type(input).__module__ == 'matplotlib.figure' and type(input).__name__ == 'Figure':
    _format = 'png'
    extension = '.' + _format
    figureFile = NamedTemporaryFile(delete=False, suffix = extension)
    input.savefig(figureFile, format = _format, transparent = True)
    self.figure = figureFile.name
    figureFile.close()

...

As you can see, display of 6.0 beta has a couple of new lines to prevent this issue.

[harupy]

I can reproduce almost the same result as @brookewenig on 5.5 conda beta.

[harupy]

This issue doesn't happen on Jupyter (notebook).

[ueshin]

@harupy Thanks for the investigation!
Sounds like it's related to how the display function handles matplotlib.
Does that mean it happens even with pandas DataFrame?

[harupy]

Yes, plots interfere even with pandas DF.

[harupy]

Found why plots on different cells don't interfere on Juypter.

https://github.com/ipython/ipython/blob/5bcd244b377b0c3dd626eba08b5a1776552039ca/IPython/core/display.py#L1466-L1490

By default, the inline backend used in the IPython Notebook will close all
matplotlib figures automatically after each cell is run. This means that
plots in different cells won't interfere. Sometimes, you may want to make
a plot in one cell and then refine it in later cells.

[harupy]

Quick workaround on Databricks runtime older than 6.0 is

import matplotlib.pyplot as plt

# call plt.close("all") after each cell is run
get_ipython().events.register('post_run_cell', lambda: plt.close('all'))

Ref.

https://ipython.readthedocs.io/en/stable/config/callbacks.html#ipython-events

Notebook

https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/7872419575180801/3144542009007501/4764411236566651/latest.html

[HyukjinKwon]

Perfect analysis @harupy! Thank you so much.

[HyukjinKwon]

@brookewenig, I think workaround was provided properly and seems not an issue in koalas. Please let me know if you think further action is required for this ticket.