Kedro: [KED-1455] kedro run pdb flag

[d-chambers]

Description

When prototyping a pipeline not everything goes right the first time. I often find myself digging through tracebacks so I can determine where to drop a breakpoint the I run the node/pipeline again. It would be great if kedro run supported a pdb flag to drop the user into an interactive debugging session when an un-handled exception surfaces.

pytest implements this exact feature.

I know kedro supports pytest for testing pipelines but in some single-use workflows spending time writing tests seems overkill as the pipeline will only ever see one dataset.

Possible Implementation

Usage would be as follows:

kedro run --pdb

Possible Alternatives

I am not aware of any.

[d-chambers]

I just discovered a usable workaround. kedro can be run like this to achieve the desired effect:

python -m pdb -m kedro run

[lorenabalan]

Hi @d-chambers apologies, this somehow got lost in the inbox. We have an item in our backlog to have a look at our interactive/debug workflow with Kedro, we'll make sure to keep your observation in mind as well. It might end up just being a documentation exercise, or it might be that some changes should be done to the framework to better facilitate that workflow. We ask that you're a little patient with us, we're working through a long series of improvements for our next release. 🙏
Possibly related to https://github.com/quantumblacklabs/kedro/pull/225 .

[d-chambers]

Hi @lorenabalan, thanks for the update. I am certainly in no rush, especially since there is a workaround. A big thanks to you and your team making a great open-source tool!

[lorenabalan]

Hi @d-chambers , good news! This has become significantly easier since we introduced Hooks in 0.16.*. @limdauto and @mzjp2 went ahead and included an example in our docs that you might find useful: https://github.com/quantumblacklabs/kedro/commit/f227d4dc2ec22e89fa519457b690111c8a8827b0. Bear in mind this will only be available on develop, so on the latest readthedocs url.

[lorenabalan]

Closing this as resolved as per https://kedro.readthedocs.io/en/latest/04_user_guide/16_debugging.html

[WaylonWalker]

Hi @d-chambers , good news! This has become significantly easier since we introduced Hooks in 0.16.*. @limdauto and @mzjp2 went ahead and included an example in our docs that you might find useful: f227d4d. Bear in mind this will only be available on develop, so on the latest readthedocs url.

@lorenabalan can hooks reference cli arguments? It would be nice to be able to have a --pdb option similar to frameworks like pytest even if its not native.

[lorenabalan]

If I understood your question right, you can have hooks that inspect click.get_current_context to look for a debugging flag and start a pdb session based on that. :)
Example in Kedro: https://github.com/quantumblacklabs/kedro/blob/25fd242faafd1740f37e5094cd2e51f2120a0313/kedro/framework/session/session.py#L157