Pandas-profiling: Multiple request to pandas profiling at the same time is failing

[haripriyatv01]

Describe the bug

I am using pandas profiling (version=2.9.0) with a python django application. When ever i am giving multiple requests to pandas profiling at the same time , Python django application itself is getting crashed.

report = ProfileReport(table, title='Profile Report', minimal=True,pool_size=3)
html_string = report.to_html()

its crashing when executing the line html_string = report.to_html()

Tried changing pool size also, but couldn't find any change.

To Reproduce

Give multiple requests to pandas profiling at the same time.

Version information:

• _Python version_: 3.6.9.
• _Environment_: IDE (PyCharm)

Could you please help me to fix this issue.

[loopyme]

As far as I know, PP 2.9 is unable to execute concurrently (there are a lot of hurdles), so before a new version is released, I recommend that you can add a global queue to Django, and let it generate reports one by one.

[haripriyatv01]

@loopyme Thank you so much for your reply. So can we expect this functionality for the new version ?

[loopyme]

I'm not working on the new version, so I don't know exactly if it supports concurrent execution. I would say this is a big challenge.

[cdeil]

I have a JupyterLab session out of which I created multiple profiles, one after the other from different cells.

I see weird behaviour as well, e.g. when I used the minimal=True option in the first ProfileReport, but then in the second cell an hour later minimal=False, I do get a new profile report, but still a minimal one.

Is it not possible to avoid global state and make the ProfileReport objects independent?

Except for this buggy behaviour, and #663, I'm really loving pandas-profile. I just discovered it today and it's super useful. Thank you!

[BenDavisson]

Adding on to this thread.

I have ran into this same "crashing" behavior whenever I try to generate a pandas profile before the previous profile is finished.

I'm sending a request from a Web App hosted by Flask to generate a pandas profile. Basically, when a user clicks a button on the web app, a report is generated. If a user clicks the button before the previous profile is generated, the entire web app will crash.

Has there been any work done in this area? Is the best solution to execute a queue of some kind to handle profile generation?