Cvat: Automatic annotation 500 ServerError

[EldarKurbanov]

My actions before raising this issue

• [ ] Wrote in the Gitter to the @nmanovic

Expected Behaviour

Automatic annotation for the task

Current Behaviour

Failed.requests.exceptions.HTTPError: 500 Server Error: Internal server error for url: http://nuclio:8070/api/function_invocations for all openvino and tensorflow example models from deploy.sh (for example YOLO v3) when click to the automatic annotation button.

Possible Solution

Fix import models to the nuclio

Steps to Reproduce (for bugs)

1. Installed Ubuntu 18.04 on Amazon (also I tried on Ubuntu 20.04 on new virtual machine);
2. Installed CVAT by this instructions;
3. cd ~/cvat
4. ./deploy.sh

Context

Can't use automatic annotation but I installed cvat and nuctl according to the instructions.

Your Environment

• Git hash commit: 8f323cfd5c579473e24818e004708dfbdfed829b
• Docker version docker version: Docker 19.03.12
• Are you using Docker Swarm or Kubernetes? Docker Swarm
• Operating System and version: Ubuntu 18.04 on Amazon and Ubuntu 20.04 on ParallersDesktop
• Output command ./nuctl get function:
  
  
  

  Screenshot: https://prnt.sc/uih1qq

  
  
• Output command docker logs cvat: Docker logs.txt

• Output of command docker ps:
  
  
  

  Screenshot: https://prnt.sc/uih5x6

  
  

Next steps

Waiting for any help.

[bsekachev]

I am not sure what is a reason, but each model on the screenshot https://prnt.sc/uih1qq has error state. Do these models work immediately after the script ./deploy.sh is performed? I get similar error after rebooting a PC, maybe your instance did reboot too..?

[EldarKurbanov]

I tried on virtual machine and models doesn't work without rebooting too.

[bugfloyd]

I'm experiencing the same problem.
I've deployed some functions using nuctl as described in the docs.

nuctl get function reports function status as ready:

  NAMESPACE |               NAME               | PROJECT | STATE | NODE PORT | REPLICAS  
  nuclio    | openvino.dextr                   | cvat    | ready |     42503 | 1/1       
  nuclio    | openvino.omz.public.yolo-v3-tf   | cvat    | ready |     36835 | 1/1       
  nuclio    | tf.faster_rcnn_inception_v2_coco | cvat    | ready |     35819 | 1/1 

When I try auto annotating a task via CVAT dashboard, This error appears:

Fetching inference status for the task 1
Error: Inference status for the task 1 is failed. requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations

I get this log in nuclio container:

20.09.26 11:20:09.356 dashboard.platform.docker (D) Container is healthy {"containerID": "4b7f5cc22a8f00b4d81fe2e6bd5dc60a46d6bda8e02b8c0ff0d807fe743952b1"}
20.09.26 11:20:11.133 .api/function_invocations (W) Failed to invoke function {"err": "Failed to send HTTP request", "errVerbose": "\nError - Post \"http://172.17.0.1:35819\": dial tcp 172.17.0.1:35819: i/o timeout\n    /nuclio/pkg/platform/abstract/invoker.go:119\n\nCall stack:\nFailed to send HTTP request\n    /nuclio/pkg/platform/abstract/invoker.go:119\nFailed to send HTTP request", "errCauses": [{"error": "Post \"http://172.17.0.1:35819\": dial tcp 172.17.0.1:35819: i/o timeout"}]}
20.09.26 11:20:11.133          dashboard.server (D) Handled request {"requestMethod": "POST", "requestPath": "/api/function_invocations", "requestHeaders": {"Accept":["*/*"],"Accept-Encoding":["gzip, deflate"],"Connection":["close"],"Content-Length":["177225"],"Content-Type":["application/json"],"User-Agent":["python-requests/2.24.0"],"X-Nuclio-Function-Name":["tf.faster_rcnn_inception_v2_coco"],"X-Nuclio-Function-Namespace":["nuclio"],"X-Nuclio-Log-Level":[""],"X-Nuclio-Path":["/"],"X-Nuclio-Project-Name":["cvat"],"X-Nuclio-Target":["tf.faster_rcnn_inception_v2_coco"]}, "requestBody": "{\"image\": \"iVBOgoA..........4AkJggg==\"}", "responseStatus": 500, "responseBody": "{\"error\": \"Failed to invoke function\"}", "responseTime": "30.174545501s"}

cvat container logs:

2020-09-26 13:09:37,674 DEBG 'rqworker_low' stderr output:
DEBUG:rq.worker:Handling failed execution of job 1ed2e996-87fc-4a8f-8445-0e0a61aef933

2020-09-26 13:09:37,676 DEBG 'rqworker_low' stderr output:
ERROR:rq.worker:Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.5/dist-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.5/dist-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 473, in __call__
    kwargs.get("threshold"), kwargs.get("mapping"))
  File "/home/django/cvat/apps/lambda_manager/views.py", line 342, in _call_detector
    "threshold": threshold})
  File "/home/django/cvat/apps/lambda_manager/views.py", line 175, in invoke
    response = self.gateway.invoke(self, payload)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 81, in invoke
    'x-nuclio-path': '/'
  File "/home/django/cvat/apps/lambda_manager/views.py", line 56, in _http
    reply.raise_for_status()
  File "/usr/local/lib/python3.5/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.5/dist-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.5/dist-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 473, in __call__
    kwargs.get("threshold"), kwargs.get("mapping"))
  File "/home/django/cvat/apps/lambda_manager/views.py", line 342, in _call_detector
    "threshold": threshold})
  File "/home/django/cvat/apps/lambda_manager/views.py", line 175, in invoke
    response = self.gateway.invoke(self, payload)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 81, in invoke
    'x-nuclio-path': '/'
  File "/home/django/cvat/apps/lambda_manager/views.py", line 56, in _http
    reply.raise_for_status()
  File "/usr/local/lib/python3.5/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations

2020-09-26 13:09:37,677 DEBG 'rqworker_low' stderr output:
DEBUG:rq.worker:Invoking exception handler <function rq_handler at 0x7f3ddca79e18>

Also there is no log for nuclio-nuclio-tf.faster_rcnn_inception_v2_coco container during the process.

All three functions fail to be invoked and I get same error message.

My CVAT env details:

Server version: 1.1
Core version: 3.5.0
Canvas version: 2.0.2
UI version: 1.8.4

[nmanovic]

@EldarKurbanov , I will look at the issue today. Sorry for the delay with my reply.

[cstamatopoulos]

I'm also experiencing the same problem.

[Becktor]

I'm experiencing the same problem.

[prateek09101996]

I am facing the same problem.
My system is windows 10. I had been using cvat for a couple of months and had been using auto annotation. Now, the auto annotation stopped working while using a video annotation with a mask. Now, it completely stopped working showing the above-mentioned results. I had been using auto annotation and now it completely stopped working with any new or past task. Can anyone help me?

[nmanovic]

@prateek09101996 , could you please post output for a couple of commands (it is better to attach output of these commands as text files):

• docker logs ps
• nuctl get function
• docker logs cvat
• nuctl version

Let's start the investigation process together.

[GoGoPen]

@prateek09101996 , could you please post output for a couple of commands (it is better to attach output of these commands as text files):

• docker logs ps
• nuctl get function
• docker logs cvat
• nuctl version

Let's start the investigation process together.

@nmanovic I encountered the same error and created an issue here : https://github.com/openvinotoolkit/cvat/issues/2280#issue-717078553. I have tried to use the same version of nuctl as the docker (1.4.8) and the same error occurred.

[ilouzl]

I also experience the same issue, but in my case it happens only with certain models.

  NAMESPACE |                     NAME                     | PROJECT | STATE | NODE PORT | REPLICAS  
  nuclio    | openvino.omz.public.yolo-v3-tf               | cvat    | ready |     32768 | 1/1       
  nuclio    | openvino.omz.semantic-segmentation-adas-0001 | cvat    | ready |     32808 | 1/1  

yolo-v3-tf works great but semantic-segmentation-adas-0001 raises that invoke error.

[prateek09101996]

@prateek09101996 , could you please post output for a couple of commands (it is better to attach output of these commands as text files):

• docker logs ps
• nuctl get function
• docker logs cvat
• nuctl version

Let's start the investigation process together.

@nmanovic I encountered the same error and created an issue here : #2280 (comment). I have tried to use the same version of nuctl as the docker (1.4.8) and the same error occurred.

While trying to fix the problem, I somehow tried updating the docker and lost every image I had on the machine. Therefore, I could not provide you all with the necessary logs. While reinstalling from scratch, I found the models is missing in my new id.

[belorenz]

@prateek09101996 , could you please post output for a couple of commands (it is better to attach output of these commands as text files):

* `docker logs ps`

* `nuctl get function`

* `docker logs cvat`

* `nuctl version`

Having the same issue on both ubuntu 16 and 18.
Here are my logs:

https://pastebin.com/btHkaWSp

[borgarpa]

Hi!

I left a comment in a related issue with some info regarding the same error: 2280#issuecomment

This error prompted when using f-BRS:

Interaction error occured
Error: Request failed with status code 500.
"500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations".

[belorenz]

This is still an issue. Is there any way to use an older release in which automatic annotation is working?

[fkreh]

I got the same issue.

Setup:

Cvat and nuclio is running on system 1. The annotations should be done from several machines in the network.

• Git hash commit: f3c336cf26088f5b8e68d775b5c7e5bc17d69316

• Docker version docker version: Docker 19.03.13

• Are you using Docker Swarm or Kubernetes? -

• Operating System and version: Ubuntu 18.04

• Output command ./nuctl get function:

  NAMESPACE |       NAME        | PROJECT | STATE | NODE PORT | REPLICAS
  nuclio    | pth-saic-vul-fbrs | cvat    | error |     32768 | 1/1

• Output of command docker ps:

CONTAINER ID        IMAGE                                  COMMAND                  CREATED             STATUS              PORTS                            NAMES
1fc839003437        alpine:3.11                            "/bin/sh -c '/bin/sl…"   12 minutes ago      Up 12 minutes                                        nuclio-local-storage-reader
84812e17c692        nginx:stable-alpine                    "/bin/sh -c 'envsubs…"   22 minutes ago      Up 12 minutes       0.0.0.0:8080->80/tcp             cvat_proxy
08f16e9e42b4        cvat/ui                                "nginx -g 'daemon of…"   22 minutes ago      Up 12 minutes       80/tcp                           cvat_ui
461bb7772978        cvat/server                            "/usr/bin/supervisord"   22 minutes ago      Up 12 minutes       8080/tcp, 8443/tcp               cvat
79b050f30220        postgres:10-alpine                     "docker-entrypoint.s…"   22 minutes ago      Up 12 minutes       5432/tcp                         cvat_db
f834f7a5660b        quay.io/nuclio/dashboard:1.4.8-amd64   "sh -c ./runner.sh"      22 minutes ago      Up 12 minutes       80/tcp, 0.0.0.0:8070->8070/tcp   nuclio
31a9a0215b00        redis:4.0-alpine                       "docker-entrypoint.s…"   22 minutes ago      Up 12 minutes       6379/tcp                         cvat_redis

• docker-compose-override.yml:

version: '3.3'

services:
  cvat_proxy:
    environment:
      CVAT_HOST: ubuntu-ki

When I try ping nucleo, I get the message ping: nuclio: The name or service is not known. It doesn't matter if i try to ping from the cvat-machine or any other system in the network.

The nucleo dashboard is reachable with the same url like the cvat dashboard. For my setup it's ubuntu-ki.

[turowicz]

I'm having this problem with CVAT but the discussion is being held here:

https://github.com/nuclio/nuclio/issues/2031

[gachiemchiep]

I also have this issue.
I tried "develop" and "v1.2.0" branch but both have the same error.

The annotation for the first image is generated, but it is failed for the rest.
I guess some code after draw annotation to image failed to execute.