Machinelearningnotebooks: PipelineParameter name case in DatabricksStep is not preserved in Databricks job run

[jarandaf]

If I use a PipelineParameter in a DatabricksStep, the name case is not preserved later on in Databricks job arguments (parameter names are always in uppercase while I define them in lowercase).

[balapv]

Can you please provide a sample?

[jarandaf]

What do you need exactly? I think the description is quite clear.

[shbijlan]

@jarandaf I have opened a bug for this issue. The findings will be shared here.

[purnesh42H]

@jarandaf we verified the workflow for pipeline parameters in DatabricksStep with intro notebook. We don't see any upper case conversion

Are you talking about the _AML_PARAMETER_ prefix? If yes, then that is something internal to implementation and not set by user. Here is the screenshot from Databricks UI where name and value casing is preserved

[jarandaf]

Hi @purnesh42H I did not try it with a Databricks notebook but with a Databricks job (python script instead of a notebook).

[purnesh42H]

For python script and jar params, parameters are passed as sequence rather than dictionary as in the case of notebook params, so only the value of the parameter is preserved

pipeline parameter adds two values in the python_script_params, a name followed by value. the name will be in this format --MY_PIPELINE_PARAM.

For example, if you have a pipeline parameter
pipeline_param = PipelineParameter(name="my_pipeline_param", default_value="pipeline_param1")

python_script_params={'arg1', pipeline_param, 'arg2'} will be transformed to ["arg1", "--MY_PIPELINE_PARAM", "pipeline_param1", "arg2"]

Please check Sample Notebook

[jarandaf]

Yes @purnesh42H this is exactly the behavior I am reporting. Why is the parameter name case not preserved? (i.e. --my_pipeline_param instead of --MY_PIPELINE_PARAM). I find it very confusing.

[purnesh42H]

Agree but as I said this limitation is due to the fact that params are passed as sequence rather than a dict which means there is no notion of parameter name. In order to make it possible to fetch this value by name instead of arg index, in the script, we provide this distinct formatting otherwise there won't be a way of identifying if the parameter is a value or a name.

[purnesh42H]

We are working on improving this behavior as a whole and will update the sample and respective docs, once done. Thanks for the feedback!

[jarandaf]

Agree but as I said this limitation is due to the fact that params are passed as sequence rather than a dict which means there is no notion of parameter name. In order to make it possible to fetch this value by name instead of arg index, in the script, we provide this distinct formatting otherwise there won't be a way of identifying if the parameter is a value or a name.

I am not sure I follow. Python's built-in ArgumentParser allows exactly that.

[purnesh42H]

Sorry if it was unclear. Basically when you are parsing, you would ideally want to distinguish between pipeline parameter and constants as pipeline parameters values can change. Providing --MY_PIPELINE_PARAM just before the actual value allows you to not bound by the index. You can simply assume any value followed by -- and remove the case folding but at this point that would be breaking change because as per the sample the existing users will be expecting the case folding.

[yanrez]

@purnesh42H the fact that behavior is different from other computes (like regular script run) is unfortunate, and clearly not something users would expect. Maybe we should find a way to fix this and deprecate and eventually retire old buggy behavior ?

[purnesh42H]

Yes, we have this feedback documented and will work on unifying the behavior across steps.