Papermill: Getting KeyError: 'kernelspec' when running from Airflow using S3 as input and output paths

[kebab-mai-haddi]

While running the python client command from Airflow and using the S3 paths as input and output, Kerncel spec error raises which is not understandable.

Reference: https://stackoverflow.com/questions/56024237/how-to-avoid-keyerror-named-kernelspec-in-papermill

[MSeal]

I think you might have an old ipython or other jupyter dependency. What's your pip list look like? Try upgrading ipython and it's dependencies to latest versions and the issue will probably go away based on the stack trace.