Syndesis: [api provider - collections] introduce foreach step in syndesis integration runtime

[lburgazzoli]

At back-end side we do already have a SplitStepHandler which is injected when a connector declares that if produces a collection.

We should create two additional steps:

• foreach
• endForeach

The foreach step should support:

• setting the split strategy using some predefined values
• setting the script used to split the exchange (javascript, groovy)
• setting the aggregation strategy using some predefined values
• setting the aggregation strategy using a script (javascript, groovy)

[christophd]

Things that we should to take into account during implementation:

• Deprecate/remove current SplitStepHandler impl
• Make sure we do not break current behavior acting as a split step without ending
• Make sure that activity logging is working with repeated steps inside of a foreach
• DataMappers placed after an endForeach step should not provide access to the inside shapes of the foreach but only to the endForeach overall outcome data shape

[dongniwang]

@lburgazzoli @christophd - Hi! I just learned about the new forEach step and I'm wondering if there's a need for UX design to show the flow and possible changes in the IVP?

Also, are we treating the "foreach" and "endForeach" as one single option when users add the step from the step list ( e.g. basic filter, advanced filter, log, etc.) ?

I can open a separate issue to track the UX designs.

cc: @gaughan @amysueg @TovaCohen

[lburgazzoli]

I would say that yes, UX design would be very welcome.

About the second question I would say that having a single option would help visualizing what happens immediately and it would also avoid the user to forget to add the endForeach but I definitively not an expert on the subject.

[christophd]

@dongniwang UX design would be awesome, thanks!

I agree with Luca and also would vote for having that as a single option.

[gaughan]

+1 on single option

[zregvart]

Perhaps the first implementation could only support collections and split the exchange into separate exchanges for each collection element remembering the size of the collection, aggregation strategy can then be just to aggregate up to the collection size of exchanges.

I think we might go far with this simplified approach and then implement more complex expression support in subsequent iterations.

[lburgazzoli]

Perhaps the first implementation could only support collections and split the exchange into separate exchanges for each collection element remembering the size of the collection, aggregation strategy can then be just to aggregate up to the collection size of exchanges.

Well, if you have a filter inside the foreach block, the size does not match any more :)

I think we might go far with this simplified approach and then implement more complex expression support in subsequent iterations.

We do have already implemented all the options at back-end size as everything boils down to:

.split(expression, aggregation)
    ....
.end()

And a simple split / collect is just a specialization of such pattern.

[zregvart]

Well, if you have a filter inside the foreach block, the size does not match any more :)

Perhaps we should not support that in first go.

Or perhaps the filter should emit a null body on the exchange if within a split/aggregate.

Not sure how citizen integrators will manage to get the expression right.

[christophd]

We can hide the expression in a first go, so the expression is a split on the collection body. Just like it is done right now with the implicit split.

Defining custom expression and aggregation options are supported in backend but may not be visible to the user in a first version.

[lburgazzoli]

expression on ui side is not supposed to land in this first step, it is implemented at back-end side because it is simpler to add now that refactor it later

[mastepan]

Split/aggregate steps are in place, issues describing specific problems have been created.