Cvat: Unable to see 'task' info in Analytics

[stefannilsson-github]

I have turned on the Analytics/Kibana module and it seems to work fine.
I can see timing information for 'users'.
However, I cannot see any information based on 'tasks', 'List of tasks' just returns 'No results found'.

I have asked about this on gitter, here is a short summary :
"""
Q: The view is empty for any period of time (e.g. 1 week)?
A: Yes it's empty no matter what time interval I ask about

Q: Visualize / List of tasks. Is it empty as well?
A: Visualize / List of tasks also returns 'No results found'

Q: Is task.keyword aggregatable?
A: Yes task.keyword is aggregatable

Q: go to Discover page and in 'Selected fields' choose 'task' (with t prefix). It should show you top 5 values. Could you please paste here a screenshot?
A: The output I get is
Top 5 values in 500 / 500 records
173
and one bar with 100% (this seems to be correct at the time)

[stefannilsson-github]

This is what I get internally when trying "List of tasks". The 'buckets' list is empty

[stefannilsson-github]

This is a typical entry in 'Discover'

[IuliiaSaveleva]

Is it any way to get 'task' info?

[grochefort]

I fear that analytics have been partially broken. As far as I can see, some restful api refactoring have broken the logs used by the ELK stack. For example, I was looking for job/task change of status (mainly when a task change to "completed")
https://github.com/opencv/cvat/pull/389/files?file-filters%5B%5D=.py#diff-bf1630f93d02478d85113174c790def2L206

[stefannilsson-github]

@grochefort is this still broken in the repo? I'm running a fairly old version of cvat (late 2019), so I am curious to know if upgrading might fix this or not.

[stefannilsson-github]

This is probably related to that 'List of tasks' searches for 'task.keyword'
As a workaround for this I have edited 'components/analytics/kibana/export.json' so that it looks for 'task' instead.
This might not be the correct way of solving this, but seems to work

[thomasstats]

@stefannilsson-github could you elaborate on the modification you made?

[stefannilsson-github]

@thomasstats I have not tested this very carefully and I'm still not sure if it's the correct way. I know very little about the detailed workings of kibana.
That said, what I have tested is just to change from 'task.keyword' -> 'task' in export.json (2 places). I also changed the type of the field from 'string' to 'number'.
There are now two fields with the same name ('task') but with different attributes, I'm investigating how this affects things.

[s3298230]

I am facing the same issue with blank "list of tasks". Wondering if anyone has any workaround? since I am not familiar with Kibana

[stefannilsson-github]

@s3298230 I did the following hack to 'components/analytics/kibana/export.json' . It is most likely not the correct way to fix this, but it seems to work:

diff --git a/components/analytics/kibana/export.json b/components/analytics/kibana/export.json
index 477b1f4..a553af8 100644
--- a/components/analytics/kibana/export.json
+++ b/components/analytics/kibana/export.json
@@ -99,7 +99,7 @@
"_id": "ec510550-c238-11e8-8e1b-758ef07f6de8",
"_type": "index-pattern",
"_source": {
- "fields": "[{\"name\":\"@timestamp\",\"type\":\"date\",\"count\":2,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"@version\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"@version.keyword\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"_id\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_index\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_score\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_source\",\"type\":\"_source\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_type\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"application\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"application.keyword\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"box count\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"duration\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"event\",\"type\":\"string\",\"count\":2,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"event.keyword\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"frame count\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"object count\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"points count\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"polygon count\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"polyline count\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"task\",\"type\":\"string\",\"count\":2,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"task.keyword\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"timestamp\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"track count\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"userid\",\"type\":\"string\",\"count\":2,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"userid.keyword\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"working time\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true}]",
+ "fields": "[{\"name\":\"@timestamp\",\"type\":\"date\",\"count\":2,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"@version\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"@version.keyword\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"_id\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_index\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_score\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_source\",\"type\":\"_source\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_type\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"application\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"application.keyword\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"box count\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"duration\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"event\",\"type\":\"string\",\"count\":2,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"event.keyword\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"frame count\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"object count\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"points count\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"polygon count\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"polyline count\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"task\",\"type\":\"number\",\"count\":2,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"task\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"timestamp\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"track count\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"userid\",\"type\":\"string\",\"count\":2,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"userid.keyword\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"working time\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true}]",
"title": "cvat*",
"timeFieldName": "@timestamp",
"fieldFormatMap": "{\"duration\":{\"id\":\"duration\",\"params\":{\"inputFormat\":\"milliseconds\",\"outputFormat\":\"asSeconds\"}},\"working time\":{\"id\":\"duration\",\"params\":{\"inputFormat\":\"milliseconds\",\"outputFormat\":\"asHours\"}}}"
@@ -164,7 +164,7 @@
"_type": "visualization",
"_source": {
"title": "List of tasks",
- "visState": "{\"title\":\"List of tasks\",\"type\":\"table\",\"params\":{\"perPage\":20,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":2,\"direction\":\"desc\"},\"showTotal\":false,\"totalFunc\":\"sum\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"sum\",\"schema\":\"metric\",\"params\":{\"field\":\"working time\",\"customLabel\":\"Working time (h)\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"task.keyword\",\"size\":1000,\"order\":\"desc\",\"orderBy\":\"1\",\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Task\"}},{\"id\":\"4\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"userid.keyword\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"_key\",\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"User\"}}]}",
+ "visState": "{\"title\":\"List of tasks\",\"type\":\"table\",\"params\":{\"perPage\":20,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":2,\"direction\":\"desc\"},\"showTotal\":false,\"totalFunc\":\"sum\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"sum\",\"schema\":\"metric\",\"params\":{\"field\":\"working time\",\"customLabel\":\"Working time (h)\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"task\",\"size\":1000,\"order\":\"desc\",\"orderBy\":\"1\",\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Task\"}},{\"id\":\"4\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"userid.keyword\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"_key\",\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"User\"}}]}",
"uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":2,\"direction\":\"desc\"}}}}",
"description": "",
"version": 1,
@@ -195,4 +195,4 @@
"savedObjectVersion": 2
}
}
-]

Basically, the only thing I've done is to change from 'task.keyword' -> 'task' in 2 places.

[s3298230]

Your workaround works for me 👍 The list appears as expected.
Quick questions are:
1/How is the "working time" calculated?
2/Can you advise how to display the start and end time of that particular working time?

[s3298230]

Your workaround works for me 👍 The list appears as expected.
Quick questions are:
1/How is the "working time" calculated?
2/Can you advise how to display the start and end time of that particular working time?

1/From that I know, Working time is calculated based on user's login time while the duration is the actual time user spent on annotating (Draw, resize...)
2/Updated the List of Tasks adding additional buckets :)

PS: Look like we are not able to retrieve the task/job status from Kibana... any suggestion would be appreciated

[azhavoro]

@stefannilsson-github Hi, could you please submit a PR to fix this issue?

[azhavoro]

Your workaround works for me 👍 The list appears as expected.
Quick questions are:
1/How is the "working time" calculated?
2/Can you advise how to display the start and end time of that particular working time?

1/From that I know, Working time is calculated based on user's login time while the duration is the actual time user spent on annotating (Draw, resize...)
2/Updated the List of Tasks adding additional buckets :)

PS: Look like we are not able to retrieve the task/job status from Kibana... any suggestion would be appreciated

Working time is calculated based on the user interactions with the UI (draw, resize, frame navigations, etc) with some time threshold. Implementation details: https://github.com/openvinotoolkit/cvat/blob/develop/cvat-core/src/logger-storage.js#L50-L57
Yep, currently there is no information inside kibana about job/task status. Please submit a new feature request.

[stefannilsson-github]

@azhavoro I could probably do that. I'll see if I can find the time to do it next week

[s3298230]

Your workaround works for me 👍 The list appears as expected.

Quick questions are:

1/How is the "working time" calculated?

2/Can you advise how to display the start and end time of that particular working time?

1/From that I know, Working time is calculated based on user's login time while the duration is the actual time user spent on annotating (Draw, resize...)

2/Updated the List of Tasks adding additional buckets :)

PS: Look like we are not able to retrieve the task/job status from Kibana... any suggestion would be appreciated

Working time is calculated based on the user interactions with the UI (draw, resize, frame navigations, etc) with some time threshold. Implementation details: https://github.com/openvinotoolkit/cvat/blob/develop/cvat-core/src/logger-storage.js#L50-L57

Yep, currently there is no information inside kibana about job/task status. Please submit a new feature request.

Thanks for your quick response :)

As for the working time , I noticed that in Kibana it only logs the user activity (generic info). However, Duration,on the other hand, contains more specific info such as drawing,loading activities etc...

So, correct me if I wrong that if an admin want to track the time user interacting with the system (including annotation and non-annotations tasks) then Working Time will work. On the other hand, if the admin wants to specifically track the actual time spent on annotation task then DURATION should fit the case

[s3298230]

@azhavoro I could probably do that. I'll see if I can find the time to do it next week

I've created a feature request at #2253
Please have a look :)