Activiti: Add sequence to Events emitted by Runtime Bundle

[salaboy]

Each event sent by the Runtime Bundle should include a sequence number which helps to identify the order of the event inside the transaction. Each event is providing a timestamp, but this is not enough to understand which event happened first from the consumer side (ie audit, query).

[miguelruizdev]

We need to clearly define the usage and presence of this new field.

@salaboy is the scope of the field sequenceNumber attached to the entity it contains?
The same way that timestamp scope is the calendar itself, we need a definition to base the sequenceNumber upon.
If we get a list of events based on entityId like this:

GET /audit/v1/events?sort=timestamp,desc&sort=id,desc&search=entityId:2b3e0dbb-19db-11e9-b5dd-0a580a18004b

We get this:

"events": [
            {
                "id": "6f76b594-15fc-4edb-82fd-16444433b3dc",
                "timestamp": 1547676480762,
                "processInstanceId": "2b3de6a8-19db-11e9-b5dd-0a580a18004b",
                "processDefinitionId": "SingleTaskProcess:1:ee023d30-199b-11e9-b5dd-0a580a18004b",
                "processDefinitionKey": "SingleTaskProcess",
                "entity": {
                    "id": "2b3e0dbb-19db-11e9-b5dd-0a580a18004b",
                    "name": "new-task-name",
                    "status": "CANCELLED",
                    "assignee": "testuser",
                    "createdDate": "2019-01-16T22:07:52.553+0000",
                    "dueDate": "2019-01-17T22:07:51.941+0000",
                    "priority": 3,
                    "processDefinitionId": "SingleTaskProcess:1:ee023d30-199b-11e9-b5dd-0a580a18004b",
                    "processInstanceId": "2b3de6a8-19db-11e9-b5dd-0a580a18004b",
                    "formKey": "new-task-form-key"
                },
                "appName": "default-app",
                "serviceFullName": "rb-my-app",
                "appVersion": "",
                "serviceName": "rb-my-app",
                "serviceVersion": "",
                "serviceType": "runtime-bundle",
                "entityId": "2b3e0dbb-19db-11e9-b5dd-0a580a18004b",
                "eventType": "TASK_CANCELLED",
                "_links": {
                    "self": {
                        "href": "http://audit/v1/events/{eventId}",
                        "templated": true
                    }
                }
            },
            {
                "id": "426d588e-463c-4ea5-8319-e3d225960c09",
                "timestamp": 1547676473546,
                "processInstanceId": "2b3de6a8-19db-11e9-b5dd-0a580a18004b",
                "processDefinitionId": "SingleTaskProcess:1:ee023d30-199b-11e9-b5dd-0a580a18004b",
                "processDefinitionKey": "SingleTaskProcess",
                "entity": {
                    "id": "2b3e0dbb-19db-11e9-b5dd-0a580a18004b",
                    "name": "new-task-name",
                    "status": "ASSIGNED",
                    "assignee": "testuser",
                    "createdDate": "2019-01-16T22:07:52.553+0000",
                    "priority": 50,
                    "processDefinitionId": "SingleTaskProcess:1:ee023d30-199b-11e9-b5dd-0a580a18004b",
                    "processInstanceId": "2b3de6a8-19db-11e9-b5dd-0a580a18004b",
                    "formKey": "taskForm"
                },
                "appName": "default-app",
                "serviceFullName": "rb-my-app",
                "appVersion": "",
                "serviceName": "rb-my-app",
                "serviceVersion": "",
                "serviceType": "runtime-bundle",
                "entityId": "2b3e0dbb-19db-11e9-b5dd-0a580a18004b",
                "eventType": "TASK_UPDATED",
                "_links": {
                    "self": {
                        "href": "http://audit/v1/events/{eventId}",
                        "templated": true
                    }
                }
            },
            {
                "id": "0a6f5174-d364-407c-858b-808c42ff6e4a",
                "timestamp": 1547676472553,
                "processInstanceId": "2b3de6a8-19db-11e9-b5dd-0a580a18004b",
                "processDefinitionId": "SingleTaskProcess:1:ee023d30-199b-11e9-b5dd-0a580a18004b",
                "processDefinitionKey": "SingleTaskProcess",
                "entity": {
                    "id": "2b3e0dbb-19db-11e9-b5dd-0a580a18004b",
                    "name": "my-task",
                    "status": "ASSIGNED",
                    "assignee": "testuser",
                    "createdDate": "2019-01-16T22:07:52.553+0000",
                    "priority": 50,
                    "processDefinitionId": "SingleTaskProcess:1:ee023d30-199b-11e9-b5dd-0a580a18004b",
                    "processInstanceId": "2b3de6a8-19db-11e9-b5dd-0a580a18004b",
                    "formKey": "taskForm"
                },
                "appName": "default-app",
                "serviceFullName": "rb-my-app",
                "appVersion": "",
                "serviceName": "rb-my-app",
                "serviceVersion": "",
                "serviceType": "runtime-bundle",
                "entityId": "2b3e0dbb-19db-11e9-b5dd-0a580a18004b",
                "eventType": "TASK_ASSIGNED",
                "_links": {
                    "self": {
                        "href": "http://audit/v1/events/{eventId}",
                        "templated": true
                    }
                }
            },
            {
                "id": "8484b58f-01df-46c1-a43d-402748a6631f",
                "timestamp": 1547676472553,
                "processInstanceId": "2b3de6a8-19db-11e9-b5dd-0a580a18004b",
                "processDefinitionId": "SingleTaskProcess:1:ee023d30-199b-11e9-b5dd-0a580a18004b",
                "processDefinitionKey": "SingleTaskProcess",
                "entity": {
                    "id": "2b3e0dbb-19db-11e9-b5dd-0a580a18004b",
                    "name": "my-task",
                    "status": "ASSIGNED",
                    "assignee": "testuser",
                    "createdDate": "2019-01-16T22:07:52.553+0000",
                    "priority": 50,
                    "processDefinitionId": "SingleTaskProcess:1:ee023d30-199b-11e9-b5dd-0a580a18004b",
                    "processInstanceId": "2b3de6a8-19db-11e9-b5dd-0a580a18004b",
                    "formKey": "taskForm"
                },
                "appName": "default-app",
                "serviceFullName": "rb-my-app",
                "appVersion": "",
                "serviceName": "rb-my-app",
                "serviceVersion": "",
                "serviceType": "runtime-bundle",
                "entityId": "2b3e0dbb-19db-11e9-b5dd-0a580a18004b",
                "eventType": "TASK_CREATED",
                "_links": {
                    "self": {
                        "href": "http://audit/v1/events/{eventId}",
                        "templated": true
                    }
                }
            }
        ]
    }

The intention behind this issue is to have a field in each of the events that establishes a sequence number based on chronological order, but without relying in the timestamp, right?

[miguelruizdev]

@salaboy A different approach would be populating that field once is persisted in the database on the consumer end, without the need for establishing in the runtime bundle.

[salaboy]

@miguelruizdev we need to generate from the producer side.. in the same way that we do with the timestamp.
So sequence should be there.. I am starting to think that we need a unique (transaction/)message identifier as well. Remember that we batch all the events of the transaction together.

[miguelruizdev]

@salaboy we set the timestamp at the moment of instantiation in every event:
https://github.com/Activiti/activiti-api/blob/develop/activiti-api-model-shared-impl/src/main/java/org/activiti/api/runtime/event/impl/RuntimeEventImpl.java#L37
Along the same lines, what should be best the place for the SequenceNumber to be set?

The value of sequenceNumber will depend on the existence of other events, so that place needs to have that information.

Also, the lifespan of this sequence should be bound to the execution of a process, right?

[miguelruizdev]

Regarding the transaction unique identifier, do we have model for transactions?
Also, what is the criteria to start and end one of those batches of events?

[salaboy]

@miguelruizdev after discussing this with @erdemedeiros and @ryandawsonuk I think that we agreed that we should:
1) pick up the message from the Stream message containing the Event. There are some examples on how to inject the @Header of the message received from a @StreamListener. we can use that as the unique message identifier.
2) We can create and store the sequence of the events based on the array of events that we receive in Audit. We just need to store that into the Event itself.

I would start with that basic approach first.. then we can see if we need to change Query as well.

[miguelruizdev]

Then, I guess that the place where to start in RB is the MessageProducerCommandContextCloseListener.

Regarding the changes in the api layer, I think that the addition of a method in CloudRuntimeEvent and its implementation will suffice.

[salaboy]

@miguelruizdev you need to start in query.. both of my points were in query service. No need to touch RB

[miguelruizdev]

@salaboy @erdemedeiros @ryandawsonuk regarding the value of the Header the message carries, will it be enough to have a UUID or do we need a specific value already present? Will it be a value we extract from the execution itself?

[miguelruizdev]

Regarding the id of the batch of events processed by audit, the logical place for those changes to take place seems to be AuditConsumerChannelHandlerImpl.

But then, one question arises: Audit is a dumb consumer since we don't process the data we get from the outside world, we just store it.
With changes in AuditConsumerChannelHandlerImpl to set the trasanctionId, we're kinda breaking that definition cause we'd be actually modifying the objects we receive.

Is this right from a design point of view?

[salaboy]

@miguelruizdev it is totally right as soon as we decorate the data that we are getting. So it is ok to do it there.

[salaboy]

merged