Kedro: How to let the after_node_run hook start after the output of the node is written?

[widovanheemstra]

What are you trying to do?

I have a node that creates a reporting string that will be written to txt file as output. I also have reporting hooks that start and stop mlflow runs. But I want the hook after_node_run to log the report file as artifact but this file is then still in progress and can't be found.
I assume that the hook is called as the node is stopped but that the process for storing the output is running in parallel and not part of the node run time.

In short how can I make the hook start when the output of the node is saved?

Preferred channel

We will do our best to help you with your query but in future, head to Stack Overflow for questions like this. Tag your questions with kedro and we'll get a notification.

[deepyaman]

@widovanheemstra While there isn't an after_dataset_save in the currently-implemented set of hooks, you could use transformers to modify save operations. Said transformer could then be registered in the after_catalog_created hook (see https://kedro.readthedocs.io/en/stable/04_user_guide/04_data_catalog.html#applying-built-in-transformers).

[widovanheemstra]

@deepyaman thanks for this great solution. I will try this soon as I'm not able to code it now. But from the docs I think this will work.