Conductor: Question: Wishlist #8 Validating Task inputKeys and outputKeys

[cclulu]

I was hoping I would be able to contribute to this open source project, and I wanted to ask if what I had planned for the validation would be enough or was there more to the validation that was intended for this. What I had planned is simply adding a @NotEmpty(message = "") On the inputkeys and outputKeys fields on TaskDef.java file.My reasoning for doing something so simple is because in your documentation on the fourth bullet point it states that

In the future, this can be extended to be a strict template that all task implementations must adhere to ...

My understanding was that users creating tasks would need to have these fields populated to be able to document what is needed for a task to be run and what you can expect it to output once finished, making it easier for re-use. Looking forward to your response, Thank you.

[naveenchlsn]

We are looking for this feature as well. @cclulu I think we need to think through some details.

• How do we ensure backward compatibility when we introduce validation?
• How to have strict validation? For example, the data structure of an input/output key.

@cyzhao @apanicker-nflx @mashurex @kishorebanala @mudit3774 Thoughts?

[cclulu]

What do you mean the data structure of the input/output key? Do you mean change the input/output keys to be something similar to what the input field is for StartWorkflowRequest is? Was there something else that you meant? From what I understand it doesn't seem that is much we can validate, just that they have something there. Since what the task can input is different depending on what the task does, I am not sure how much validation we can make on whats there, but only that they have something there. Am I missing understanding something here? If I am, i would be glad to learn more.

As far as backward compatibility goes, I had not thought about about. If we just add the validation that I had mentioned, then backwards compatibility wouldn't be as simple since it changes the model of the taskdef. I would have to think more about this one. Any ideas would be great.

[apanicker-nflx]

@cclulu Thanks for the proposal. As @naveenchlsn mentioned and you quoted in your proposal, the eventual goal would be to enforce a strict template for both input and output of a task. This would mean, the task inputs and outputs would need to be typed, so that re-use of tasks can allow for type-safety constraints.

To ensure backward compatibility for existing tasks, this could be rolled out in phases.
Phase 1:

• Add a new field to TaskDef , say "inputVariables" and "outputVariables" which could be a Map where the key in the map is the inputKey/outputKey and the value is the type (must be either a primitive type or JsonNode(?)[This part needs to be hashed out more]).
• This field can be enforced for all new task definitions.
• Deprecate the inputKeys and outputKeys fields.
• Minor version bump of Conductor release.

Phase 2:

• Next major version release of Conductor would remove the inputKeys and outputKeys fields and enforce this typed contract.

@cclulu @naveenchlsn Please let me know your thoughts regarding this.

[naveenchlsn]

This looks good to me @apanicker-nflx .

We can consider using JsonSchema to let the users input the schema definition of task inputs and outputs. Please let me know your thoughts @cclulu @apanicker-nflx

[apanicker-nflx]

JsonSchema is a good approach towards implementing this. It additionally allows the task definition owner to specify if certain fields are required/optional, while not compromising on type-safety.

[cclulu]

sounds like there is a plan, I would be excited to try to implements this, I have never used JsonSchema before so I look forward to learning something new. If this is the implementation that everyone agrees with I would like to get started on this is that is alright with everyone? Also if I have any more question should I just post them on the gitter or post to this thread?

[apanicker-nflx]

@cclulu Please feel free to get started with on this, and continue to update here with regards to questions and design choices. Thank you very much for the contribution.

[andrea11]

Hello @cclulu,
Me and my team, we are currently using JSON schema to make validation possible on some inputs. And my feedback is really positive. Allow to describe workflow inputs and task inputs with JSON schema, can give the possibility to perform a wide and deep validation, on types, constraints and even conditional dependencies among inputs.

[mashurex]

I would think that using something more flexible than an annotation on the classes would be required for input/output keys, as they're dynamic to each task defined in the system. Using JSON Schema or having a way to describe validation requirements on the task def (or in the workflow def) would make this much more flexible.

For our custom implemented tasks, we validate task inputs in the start and/or execute methods. A few times we've created a data validation tasks that sit between two 'real' tasks in the workflow for this very purpose as well.