Machinelearningnotebooks: Pipeline step allow_reuse flag is set to True but steps are not being reused

[clmccart]

I am creating a multistep pipeline in azureml and all of the steps have the allow_reuse flag set to true. However, none of the steps are being reused.

Included are two runs that have no changes between them but there is no reuse:
Run ID: 9aa9bee5-d27e-48b3-b03e-72703f0e4c10
Step ID: f3fe9958-a650-44c2-8ece-6e42f9d287dc

Run ID: d45f5f9d-df6b-449e-8855-1e340043eb32
Step ID: 7207f375-d908-4562-94bf-02f022f28047

Specifically, I am looking at the 1st step in the pipeline as I am still working on the subsequent steps.
Does the entire pipeline have to complete successfully in order for the steps to be eligible for reuse?

[YutongTie-MSFT]

@clmccart
Thanks for the feedback! I have assigned the issue to the content author to investigate further and update the document as appropriate.

[sanpil]

We will certainly take a look, but can you check if you are doing this:

The best practice for working with PythonScriptStep is to use a separate folder for scripts and any dependent files associated with the step, and specify that folder with the source_directory parameter. Following this best practice has two benefits. First, it helps reduce the size of the snapshot created for the step because only what is needed for the step is snapshotted. Second, the step's output from a previous run can be reused if there are no changes to the source_directory that would trigger a re-upload of the snapshot.

[clmccart]

I am following that best practice. My snapshot is relatively large but both runs listed above have no changes.

[sanpil]

Can you provide the RunID?

[clmccart]

see original comment for the Run ID and Step ID of both runs

[purnesh42H]

@clmccart which version of sdk are you using? Can you upgrade to latest version of sdk? If that doesn't solve the issue, could you share a repro?

[sanpil]

We took a look at this, and sees that reuse is working fine. Can you tell us why you think the steps are not reused?

[2020-01-13T22:25:21.0836373Z] [d45f5f9d-df6b-449e-8855-1e340043eb32] Successfully reused Job d45f5f9d-df6b-449e-8855-1e340043eb32@@@prepare_dutchf3_step@@@b1af9181, original job 9aa9bee5-d27e-48b3-b03e-72703f0e4c10@@@prepare_dutchf3_step@@@5eba5dff, node hash 193cafbd-bb34-4b8f-9722-b3295702ab7b-/9az8VMthaNzoBr6yTyOuuLz6omupgn2yvM6KqXHVVI=, dmtEtag 19f2c6cd-dfd8-4fa6-b6ba-605b1970061a

[clmccart]

I am using 1.0.83

[clmccart]

I am not using a standalone notebook. This pipeline is part of a larger system. i can add you to our repo if you would like access to the source code to investigate this

[clmccart]

@sanpil I know that is not getting reused because I see it recomputing as I monitor the run and the reuse column in the portal indicates it is not being reused. the step takes a similar amount of time to complete as it does when there are changes.

[clmccart]

@sanpil @YutongTie-MSFT any update on this?

[purnesh42H]

@clmccart could you check in the studio portal -https://ml.azure.com/ ?

[sanpil]

Looks like there is a bug in portal.azure.com. Can you check the new portal at https://ml.azure.com/ as @purnesh42H mentioned?

[clmccart]

I see in the new portal that it shows as reused but the duration indicates that it was not reused and i am seeing the output being re-generated in blob storage each time. the step is definitely NOT getting reused but I see that the new portal says that is is being reused.

[purnesh42H]

@clmccart are you saying the output of step which is reused is generating the new output? How are you checking that?

[clmccart]

no steps are getting reused. the pythonscriptstep which i would expect to get reused is running everytime, no matter what. I am checking this because part of that pythonscript writes something out to blob and i can see it getting writing out everytime. if the step was being reused, my understanding is that, the python script would not be executed and it would, therefore, not regenerate its write-outs.

[purnesh42H]

is there a parent step to that? is that being reused?

[clmccart]

no parent step currently but in previous runs there was and it also was not getting reused.

[purnesh42H]

Can you share a repro?

[clmccart]

@swanderz produced a really clean repro for this issue in #740

[sanpil]

We are looking at a reuse bug that got introduced recently. Please stay tuned.

[purnesh42H]

@clmccart #740 is unrelated to this. In that case, there is a change in hyperdrive setting which is preventing the reuse. In your case, the reuse happened, but as you mentioned it still running in spite of reuse. Hence, could you please provide repro for your case?

[nemasobhani]

Any update on this issue? Reuse in the classic UI is always "No" even if it does occur, and hyperdrive never reuses the step outputs, even though it supports the allow_reuse parameter.

CC: @swanderz

[sonnypark]

Fix has been ready. We expect to deliver the fix on next release.

[YutongTie-MSFT]

@clmccart
We will now proceed to close this thread. If there are further questions regarding this matter, please respond here and @YutongTie-MSFT and we will gladly continue the discussion.