Conductor: JSON_JQ_TRANSFORM usage details

[wknd-deevpos]

Hello, I have been trying to use JSON_JQ_TRANSFORM task type but it is not working. Is there any documentation for the usage? One of our API returns following JSON

{
    "hits": [
        {
            "_index": "mes$i$v2$lid_infor.daf.daf$mds_file$1.0$1561444341392",
            "_type": "mds_file",
            "_source": {
                "LASTCHANGEDTS": "2018-11-06T10:01:10Z"
            },
            "_id": "3_ax1",
            "_score": 4.491742
        },
        {
            "_index": "mes$i$v2$lid_infor.daf.daf$mds_file$1.0$1567159125621",
            "_type": "mds_file",
            "_source": {
                "LASTCHANGEDTS": "2019-04-25T07:07:14.223Z"
            },
            "_id": "3_ax2",
            "_score": 2.2275105
        }
    ]
}

I used following task definition, but the workflow results in error. I tested JQ on jqplay.org and they work as expected.

{
    "type": "JSON_JQ_TRANSFORM",
    "name": "transform",
    "taskReferenceName": "TransformSerchResult",
    "inputParameters": {
        "hits": "${search.output.response.body}",
        "queryExpression": ".hits[] |=  { id: ._id, index: ._index, source: {created:._source.LASTCHANGEDTS} }"
    }
}

Any suggestions ?

[elisherer]

I tried running the following dynamically:

{
  "schemaVersion": 2,
  "name": "test_jq",
  "version": 1,
  "description": "Test JSON_JQ_TRANSFORM",
  "ownerEmail": "[email protected]",
  "tasks": [
    {
      "name": "lambda_1",
      "taskReferenceName": "lambda_1",
      "type": "LAMBDA",
      "inputParameters": {
        "input": "${workflow.input}",
        "scriptExpression": "return {\n    \"hits\": [\n        {\n            \"_index\": \"mes$i$v2$lid_infor.daf.daf$mds_file$1.0$1561444341392\",\n            \"_type\": \"mds_file\",\n            \"_source\": {\n                \"LASTCHANGEDTS\": \"2018-11-06T10:01:10Z\"\n            },\n            \"_id\": \"3_ax1\",\n            \"_score\": 4.491742\n        },\n        {\n            \"_index\": \"mes$i$v2$lid_infor.daf.daf$mds_file$1.0$1567159125621\",\n            \"_type\": \"mds_file\",\n            \"_source\": {\n                \"LASTCHANGEDTS\": \"2019-04-25T07:07:14.223Z\"\n            },\n            \"_id\": \"3_ax2\",\n            \"_score\": 2.2275105\n        }\n    ]\n};"
      }
    },
    {
      "name": "jq_1",
      "taskReferenceName": "jq_1",
      "type": "JSON_JQ_TRANSFORM",
      "inputParameters": {
        "hits": "${lambda_1.output.result.hits}",
        "queryExpression": ".hits[] |=  { id: ._id, index: ._index, source: {created:._source.LASTCHANGEDTS} }"
      },
      "taskDefinition": {
        "name": "jq_1",
        "ownerEmail": "[email protected]"
      }
    }
  ],
  "outputParameters": {
    "result": "${jq_1.output.result}"
  }
}

I did manage to get the result. The only difference I could see is that array are translated as objects with indices as keys, which I think should be fixed.

[elisherer]

Changing the queryExpression to:

[.hits[] | { id: ._id, index: ._index, source: {created:._source.LASTCHANGEDTS} }]

did the trick (notice the surrounding brackets)