Syndesis: Add UI page to describe data shape for shapeless connectors

[dhirajsb]

UXD has designed an extra page for shapeless connectors in #1194. Shapeless connectors are identified as ones with input/output shape or both of the form:

            "inputDataShape": {
              "type": "*"
            },
            "outputDataShape": {
              "type": "*"
            }

UI needs to add page to allow user to specify input and output shapes from json/xml schema or instance documents.

[gashcrumb]

Question, why can't this be tacked on to the existing plumbing we have in place for multi-page configuration for connections? The descriptor returned from that call also has the input/output data shapes, we can update this component to ensure those are set appropriately.

[gashcrumb]

Ah, it's a whole new step type then -> https://github.com/syndesisio/syndesis/blob/master/ux/designs/describedatastep/describedatastep.md

[gashcrumb]

@sjcox-rh @dongniwang @dhirajsb Question on the design, how does the user indicate the schema/instance document should be set for the input or output data shape?

[gashcrumb]

Ah, silly me, it's this one -> https://github.com/syndesisio/syndesis/blob/master/ux/designs/describe-data/describe-data.md

So my question still stands on this one...

[zregvart]

@gashcrumb I don't think this would work by adding an action property, if we do that we would just add a property, the data shape of the action would remain as *. This would then require us to configure the mapper by looking a the data shapes and at the configured action properties which feels wrong to me.

[gashcrumb]

I wonder though then, how exactly is the UI going to set these data shapes on a step? Currently it's all driven off of the action descriptor. If we add an additional page after the data shape input that's effectively a no-op the UI then has a chance to post the data shape properties, which could then replace what's set on the action descriptor, then I guess we'd have to set that on the step.

But then, what's the other idea of how this is going to be implemented, are there endpoints for the UI to invoke currently to post these data shapes?

[zregvart]

The big dirty secret is that the UI can mutate any part of the integration and the back-end will store it as is, the back-end stores the JSON as received. So mutating the action descriptor's shape should be possible.
We can certainly think about adding endpoints to make this _proper_. I'm adding a discussion label and inviting @syndesisio/backend for comments on this.

[gashcrumb]

Ah, of course, I see. So for the file input we'll just read it client-side and replace the action's data shape property locally. I think that'll work fine.

Now so I don't forget or if someone else picks this up, in the block where we check for the last page of properties we'll have to check the input and output data shapes in the action descriptor and prompt the user for each of those then.

[zregvart]

Yeah, and on the adding endpoint argument, one downside of that is that this could be still on a unsaved integration, and as back-end is stateless (i.e. no HTTP session) there is no where to put that change in to apply it when we save the integration.

[gashcrumb]

Absolutely...

[gashcrumb]

So based on the design it'd make sense when it's the start connection, the user would be specifying the output data shape.

And if it's the end connection, the user will be specifying the input data shape.

Then if it's a connection in the middle of the flow, the user would potentially need to specify an input and an output, I think I can cater for that easily enough.

Just to double-check though, what attributes would I be setting on the data shape object? There's inputs for the schema type and the schema/instance document.

[zregvart]

There is a user story in #1134 that goes into this.

[dhirajsb]

@gashcrumb connector's shape properties tell you whether input/output shapes MAYBE specified. Some actions in the middle might not have an output shape, but simply pass along the message from the step before in the Camel route.

[dhirajsb]

@gashcrumb has anyone had a chance to work on this feature on the UI side?

[gashcrumb]

No, we're pretty backlogged.

[dongniwang]

@gashcrumb - do you have any questions on the UX designs? Please post here and tag me.

FYI - Latest PR is merged and added fields for specify data type display name and display description. See latest UX design here.

[gashcrumb]

@zregvart @dhirajsb question on this one, does the user's schema/data or whatever go into the exemplar or specification attribute on the data shape?

[igarashitm]

From datamapper POV it should be in specification. I don't know the original intention, but datamapper don't use exemplar ATM.

[gashcrumb]

@igarashitm cool thanks!, is there any particular interface the stringified object should follow? Was looking at what winds up in there for twitter...

[igarashitm]

@gashcrumb it's really a plain schema or example documents, but I think we just want to keep it as a string so it's transferred to AtlasMap design runtime as-is.

• XML Schema: W3C XML Schema, or SchemaSet defined in AtlasMap here - https://github.com/atlasmap/atlasmap/blob/master/runtime/modules/xml/model/src/main/resources/atlas-xml-schemaset-model-v2.xsd
• XML Instance: Arbitrary XML document example (build a definition from an example)
• JSON Schema: json-schema.org JSON Schema
• JSON Instance: Arbitary JSON document example (build a definition from an example)

For example when we pass the XML Schema in the inspection request (= JSON), we put the whole XML payload into specification and then it's transferred to the inspection service directly.

[igarashitm]

Oh BTW the java kind is special. For java we carry an inspection result in specification, then AtlasMap UI parse it. Even in this case the specification is just a string until AtlasMap parse it here:
https://github.com/atlasmap/atlasmap/blob/383c8bf797c63275db38304bbb01c8e4766bf9bb/ui/src/app/lib/atlasmap-data-mapper/services/document.management.service.ts#L79

[gashcrumb]

Okay, so that's perfect, I'll just plop the user input as-is for now into the specification attribute, thanks!

[gashcrumb]

What's the difference between none and *?

Also we currently allow From actions as middle steps, is this behavior correct?

[igarashitm]

I think those are not well defined yet, but I propose:

• none: nothing (void)
• any: something, but the type is not defined (for so called shapeless connector)

And migrate * to any.

[igarashitm]

And I'm going to use any as an input kind of datamapper step, as it accepts all the preceding inputs.

[gashcrumb]

@dongniwang in the user story and in the app it's possible that the user may have to specify both the input and output data shape for a step, i.e. if I have AMQ request/response I think the input and output are shapeless.

Will I just make the user basically go through the same form page twice then and toggle the title with 'Input' and 'Output'?

@igarashitm:

I think those are not well defined yet, but I propose:

none: nothing (void)
any: something, but the type is not defined (for so called shapeless connector)

And migrate * to any.

That sounds good, do we need an issue for this? I suspect we'll want to get this ironed out in the deployment.json etc.

[gashcrumb]

I've started to knock in a page but have found a couple things that have held me up:

kind vs type vs any vs * = ???? -> https://github.com/syndesisio/syndesis/issues/1551
once I set the kind or type client side I won't know again that it's user set -> https://github.com/syndesisio/syndesis/issues/1552

[dhirajsb]

@gashcrumb do we need to update current shapeless connectors to use the metadata tag you are looking for? Or is that done already?

[gashcrumb]

I can set the tag client-side when the user changes the kind, the data shape property itself though needs to be updated to support the metadata attribute, see #1552

[gashcrumb]

PR there adds an initial implementation with a textarea that can be used to copy/paste a document.

So on file uploading or fetching via URL there's a couple things/issues. I can read in a file the user chooses and set it for the specification attribute, but how exactly is validation supposed to work? Per the design after uploading the file it should be validated but we've no backend to upload the file to as it's stateless.

The other issue is fetching a document from a URL, I can't do that in the frontend (CORS), we'd need to post the URL to the backend and fetch it in the backend.

Actually, even for the text area there should be some kind of validation, do we have an endpoint for that?

[zregvart]

@kcbabo what we currently have is shown here:

I've created AMQ broker/connection on staging if you'd like to try it out.

It does fulfill the story as described in #1134.

In order to comply with the UXD there is still more to do with uploading and validating the supplied specification.

In the interest of schedule perhaps we can leave it at this and refine it later? Or do you feel that there is still work left to be done here?

[kcbabo]

Looking good! A few comments/questions:

• Will we still support uploading a schema vs. copy/paste?
• We need to support not specifying a data type. Does that happen prior to this screen or on this screen?
• If it happens on this screen, the no data type option should be the default and specifying JSON or XML schema should be options the user selects.
• If the no data type option is selected, the other inputs (Schema, Data Type Name, Data Type Description) should be hidden or disabled IMO.

[paoloantinori]

Crosslinking screenshots:

https://github.com/syndesisio/syndesis/issues/1229#issuecomment-369536659

[kcbabo]

One quick clarification on my comment on uploading a schema. I meant to say "uploading a schema in addition to copy/paste". Not vs.

[paoloantinori]

replying to @kcbabo

Will we still support uploading a schema vs. copy/paste?

Not for this Sprint. Additional work required:

• backend endpoint implementation
• validation serverside
• proxy mode (provide a url to fetch a remote resource, needs to be implemented as a server side operation)

[igarashitm]

Is there any case in Syndesis, that user would not describe data type, but it's consumed by the next step? (i.e. consuming shapeless data like byte[]) If not I'd like to have it marked as DataShape.kind=none. This helps data type mismatch indicator not to suggest to go to "describe data type" page of previous step.

If yes, maybe we just need to check if it's user's decision leaving it as any. If it's user's decision, then instead it suggests adding datamapper step and consume from the other steps. A bit tangled logic is required in UI :-)

cf. #1756

[gashcrumb]

If the no data type option is selected, the other inputs (Schema, Data Type Name, Data Type Description) should be hidden or disabled IMO.

Once we get #897 in place this should be straightforward.

If yes, maybe we just need to check if it's user's decision leaving it as any. If it's user's decision, then instead it suggests adding datamapper step and consume from the other steps. A bit tangled logic is required in UI :-)

I'm sure we'll continue to come up with edge cases :-)

[zregvart]

Will we still support uploading a schema vs. copy/paste?

So this is the question, we can work on that or leave it as is, how badly is it needed? Would require server side and UI work, probably day or two for those combined.

We need to support not specifying a data type. Does that happen prior to this screen or on this screen?

On this screen, default is any type (e.q. un-typed), see:

If it happens on this screen, the no data type option should be the default and specifying JSON or XML schema should be options the user selects.

We know about _Any Type_ (un-typed) and _No Type_ (without payload), I think this is exactly as required, I could be miss-understanding.

[zregvart]

What we agreed on:

• [x] replace _No Type_ and _Any Type_ with _Don't specify type_
• [x] remove _Java_ type
• [x] rename _Select Schema Type_ label to _Select type_
• [x] rename _Schema_ label to _Definition_
• [x] when _Don't specify type_ is selected other fields are grayed out

[gashcrumb]

4 out of 5 tasks agree with the linked PR, waiting on #897 for the last bit