Imageai: Issue with using imageai with multithreading

Created on 17 Dec 2018  路  14Comments  路  Source: OlafenwaMoses/ImageAI

Hi,
The object detection function works well when used in while loop.
However, when i try to run the same function using threading in python, it does not works.
The error led me to things like session ends,invalid input image path..... but I could not resolve it.
Is it because we cannot use functions of imageai in multithreading in python or is there some more parameters to set.
Errors i got:
Exception in thread Thread-2:
Traceback (most recent call last):
File "/home/pi/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1050, in _run
subfeed, allow_tensor=True, allow_operation=False)
File "/home/pi/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3488, in as_graph_element
return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
File "/home/pi/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3567, in _as_graph_element_locked
raise ValueError("Tensor %s is not an element of this graph." % obj)
ValueError: Tensor Tensor("keras_learning_phase:0", shape=(), dtype=bool) is not an element of this graph.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/pi/tensorflow/lib/python3.5/site-packages/imageai/Detection/__init__.py", line 777, in detectCustomObjectsFromImage
K.learning_phase(): 0
File "/home/pi/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 887, in run
run_metadata_ptr)
File "/home/pi/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1053, in _run
'Cannot interpret feed_dict key as Tensor: ' + e.args[0])
TypeError: Cannot interpret feed_dict key as Tensor: Tensor Tensor("keras_learning_phase:0", shape=(), dtype=bool) is not an element of this graph.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(self._args, *self._kwargs)
File "/home/pi/tensorflow/lib/python3.5/site-packages/threadfinal.py", line 53, in read
objectDetect(imageInProcess)
File "/home/pi/tensorflow/lib/python3.5/site-packages/singularObjectDetection.py", line 13, in objectDetect
detections = detector.detectCustomObjectsFromImage(custom_objects=custom, input_image=os.path.join(execution_path,filename), output_image_path=os.path.join(execution_path, "image3new-custom.jpg"), minimum_percentage_probability=30)
File "/home/pi/tensorflow/lib/python3.5/site-packages/imageai/Detection/__init__.py", line 860, in detectCustomObjectsFromImage
"Ensure you specified correct input image, input type, output type and/or output image path ")
ValueError: Ensure you specified correct input image, input type, output type and/or output image path

My function
execution_path = os.getcwd()
detector = ObjectDetection()
detector.setModelTypeAsTinyYOLOv3()
detector.setModelPath(os.path.join(execution_path, "yolo-tiny.h5"))
detector.loadModel(detection_speed="flash")
custom = detector.CustomObjects(person=True, dog=True)
def objectDetect(filename):
humanDetected=False
detections = detector.detectCustomObjectsFromImage(custom_objects=custom, input_image=os.path.join(execution_path,filename), output_image_path=os.path.join(execution_path, "image3new-custom.jpg"), minimum_percentage_probability=30)
for eachObject in detections:
print(eachObject["name"], " : ", eachObject["percentage_probability"], " : ", eachObject["box_points"])
print("--------------------------------")
if(eachObject["name"]=="person"):
if(humanDetected==False):
humanDetected=True
break

print("Finish detecting photo "+ str(humanDetected))

Thread calls the above function

q=queue.Queue()
t1=threading.Thread(target=write,args=(q,),daemon=True)

the write function calls the object detection function

Most helpful comment

Greetings.
After spending a couple of days on trying to get imageAI to work with Flask, it was made obvious that the threading issue is not resolved.
To make a long story short if the prediction is executed on a different thread that the model was loaded it fails with the following error

ValueError: Tensor Tensor("predictions/Softmax:0", shape=(?, 56), dtype=float32) is not an element of this graph.

which is the same problem described in this issue and is related to this issue in Keras

I have tried many different solution proposed but nothing worked.
The only way that "worked" is to force Flask not to use threading
i.e
app.run(host='0.0.0.0',threaded = False)

I have not looked into the ImageAI code extensively but for custom image detection using InceptionV3 its 100% sure it does not work with threads.

All 14 comments

When you are using multithreading, ensure you load the model [ detector.loadModel() ]in the same thread that you want to use for the detection.聽

Or better still, you can load the model outside the thread and parse the "detector" object into the thread, from where you will call the detection function.聽

I already try calling the detector.loadModel() in the same thread and even outside the thread yet no positive result was obtained.
I print the path of where the thread was being executed which was same as the path of the main program but yet i got the same error
I did not parse the detector object though,"how to do that"???

here is part of my script
this is my function that the thread t2 calls:
def read():
print("read "+ os.getcwd())
execution_path = os.getcwd()
detector = ObjectDetection()
detector.setModelTypeAsTinyYOLOv3()
detector.setModelPath(os.path.join(execution_path, "yolo-tiny.h5"))
detector.loadModel(detection_speed="flash")
custom = detector.CustomObjects(person=True, dog=True)
while True:
objectDetect("image1.jpg")

And in main:

t2=threading.Thread(target=read,daemon=True)
t2.start()

I use django and your library works perfectly when i use the --nothreading option when launching the server. But when i remove that option, the first time when an http request for image analysis is sent the analysis works perfectly.
But then every time when an http request is sent to the django server for image analyzing i get the following errors.
Errors:
web_1 | File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1075, in _run web_1 | subfeed, allow_tensor=True, allow_operation=False) web_1 | File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3590, in as_graph_element web_1 | return self._as_graph_element_locked(obj, allow_tensor, allow_operation) web_1 | File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3669, in _as_graph_element_locked web_1 | raise ValueError("Tensor %s is not an element of this graph." % obj) web_1 | ValueError: Tensor Tensor("Placeholder:0", shape=(3, 3, 3, 16), dtype=float32) is not an element of this graph. web_1 | web_1 | During handling of the above exception, another exception occurred: web_1 | web_1 | Traceback (most recent call last): web_1 | File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner web_1 | response = get_response(request) web_1 | File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 128, in _get_response web_1 | response = self.process_exception_by_middleware(e, request) web_1 | File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response web_1 | response = wrapped_callback(request, *callback_args, **callback_kwargs) web_1 | File "/code/website/upload/views.py", line 299, in analyze_photo web_1 | result = image_ai.analyzeImage(algo_id,photo_ref.google_image_path,output_fname) web_1 | File "/code/website/misc/object_detection/image_ai.py", line 36, in analyzeImage web_1 | self.detector.loadModel() web_1 | File "/usr/local/lib/python3.6/site-packages/imageai/Detection/__init__.py", line 213, in loadModel web_1 | model.load_weights(self.modelPath) web_1 | File "/usr/local/lib/python3.6/site-packages/keras/engine/network.py", line 1166, in load_weights web_1 | f, self.layers, reshape=reshape) web_1 | File "/usr/local/lib/python3.6/site-packages/keras/engine/saving.py", line 1058, in load_weights_from_hdf5_group web_1 | K.batch_set_value(weight_value_tuples) web_1 | File "/usr/local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2470, in batch_set_value web_1 | get_session().run(assign_ops, feed_dict=feed_dict) web_1 | File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run web_1 | run_metadata_ptr) web_1 | File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1078, in _run web_1 | 'Cannot interpret feed_dict key as Tensor: ' + e.args[0]) web_1 | TypeError: Cannot interpret feed_dict key as Tensor: Tensor Tensor("Placeholder:0", shape=(3, 3, 3, 16), dtype=float32) is not an element of this graph. web_1 | File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner web_1 | response = get_response(request) web_1 | return self._as_graph_element_locked(obj, allow_tensor, allow_operation) web_1 | File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3669, in _as_graph_element_locked web_1 | File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 128, in _get_response web_1 | raise ValueError("Tensor %s is not an element of this graph." % obj) web_1 | response = self.process_exception_by_middleware(e, request) web_1 | ValueError: Tensor Tensor("Placeholder:0", shape=(3, 3, 3, 16), dtype=float32) is not an element of this graph. web_1 | web_1 | File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response web_1 | During handling of the above exception, another exception occurred: web_1 | web_1 | response = wrapped_callback(request, *callback_args, **callback_kwargs) web_1 | File "/code/website/upload/views.py", line 299, in analyze_photo web_1 | Traceback (most recent call last): web_1 | result = image_ai.analyzeImage(algo_id,photo_ref.google_image_path,output_fname) web_1 | File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner web_1 | response = get_response(request) web_1 | File "/code/website/misc/object_detection/image_ai.py", line 36, in analyzeImage web_1 | self.detector.loadModel() web_1 | File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 128, in _get_response web_1 | File "/usr/local/lib/python3.6/site-packages/imageai/Detection/__init__.py", line 213, in loadModel web_1 | model.load_weights(self.modelPath) web_1 | response = self.process_exception_by_middleware(e, request) web_1 | File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response web_1 | response = wrapped_callback(request, *callback_args, **callback_kwargs) web_1 | File "/usr/local/lib/python3.6/site-packages/keras/engine/network.py", line 1166, in load_weights web_1 | f, self.layers, reshape=reshape) web_1 | File "/usr/local/lib/python3.6/site-packages/keras/engine/saving.py", line 1058, in load_weights_from_hdf5_group web_1 | File "/code/website/upload/views.py", line 299, in analyze_photo web_1 | K.batch_set_value(weight_value_tuples) web_1 | File "/usr/local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2470, in batch_set_value web_1 | result = image_ai.analyzeImage(algo_id,photo_ref.google_image_path,output_fname) web_1 | get_session().run(assign_ops, feed_dict=feed_dict) web_1 | File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run web_1 | run_metadata_ptr) web_1 | File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1078, in _run web_1 | File "/code/website/misc/object_detection/image_ai.py", line 36, in analyzeImage web_1 | self.detector.loadModel() web_1 | File "/usr/local/lib/python3.6/site-packages/imageai/Detection/__init__.py", line 213, in loadModel web_1 | model.load_weights(self.modelPath) web_1 | File "/usr/local/lib/python3.6/site-packages/keras/engine/network.py", line 1166, in load_weights web_1 | f, self.layers, reshape=reshape) web_1 | 'Cannot interpret feed_dict key as Tensor: ' + e.args[0]) web_1 | File "/usr/local/lib/python3.6/site-packages/keras/engine/saving.py", line 1058, in load_weights_from_hdf5_group web_1 | K.batch_set_value(weight_value_tuples) web_1 | File "/usr/local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2470, in batch_set_value web_1 | get_session().run(assign_ops, feed_dict=feed_dict) web_1 | TypeError: Cannot interpret feed_dict key as Tensor: Tensor Tensor("Placeholder:0", shape=(3, 3, 3, 16), dtype=float32) is not an element of this graph. web_1 | File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run web_1 | run_metadata_ptr) web_1 | File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1078, in _run web_1 | 'Cannot interpret feed_dict key as Tensor: ' + e.args[0]) web_1 | TypeError: Cannot interpret feed_dict key as Tensor: Tensor Tensor("Placeholder:0", shape=(3, 3, 3, 16), dtype=float32) is not an element of this graph.

A summary of my code:

def analyze(self):
self.detector = ObjectDetection()
self.detector.setModelTypeAsRetinaNet()
self.detector.setModelPath( os.path.join(execution_path,"resnet50_coco_best_v2.0.1.h5"))
self.detector.loadModel()
detections = self.detector.detectObjectsFromImage(input_image=self.input_image,
output_image_path=self.output_image)

Please note that i don't use anywhere in my code threads, i don't know how django organizes my code in threads, i've tried putting the whole function in a thread but still the same error occurs

Hi, is there any news about this issue? I have the same problem.

Here is my thread class:
```
class RetinaNetCaptureThread(Thread):
detection_speed = "normal" # possible values: 'fast', 'faster', 'fastest' and 'flash'
custom_objects = "person"

def __init__(self, i_capture_handler, srcfile='test3.mp4', src_url=''):
    Thread.__init__(self)

    self.detector = VideoObjectDetection()
    self.detector.setModelTypeAsRetinaNet()
    self.detector.setModelPath(os.path.join(os.getcwd(), "resnet50_coco_best_v2.1.0.h5"))
    self.detector.loadModel(detection_speed=self.detection_speed)

    self.cap = cv2.VideoCapture(srcfile)

    self.keep_tracking = False

def run(self):
    self.keep_tracking = True
    while self.keep_tracking:
        detections = self.detector.detectCustomObjectsFromVideo(
            custom_objects=self.detector.CustomObjects(person=True),
            camera_input=self.cap,
            return_detected_frame=False,
            output_file_path=os.path.join(os.getcwd(), "tmp_output_video.mp4"),
            save_detected_video=False)
        print('Found detections:', detections)

    print('Capturing thread has stopped.')

def stop_tracking(self):
    print('Capturing thread was asked to stop.')
    self.keep_tracking = False

````

I get this error:
````
Exception in thread Thread-1:
Traceback (most recent call last):
File "E:\Dev\AI\Labs\DevoAgencyControl\venven\imageai\Detection__init__.py", line 1894, in detectCustomObjectsFromVideo
_, _, detections = model.predict_on_batch(np.expand_dims(frame, axis=0))
File "C:\Python\Python36\lib\site-packages\keras\engine\training.py", line 1273, in predict_on_batch
self._make_predict_function()
File "C:\Python\Python36\lib\site-packages\keras\engine\training.py", line 554, in _make_predict_function
*kwargs)
File "C:\Python\Python36\lib\site-packages\keras\backend\tensorflow_backend.py", line 2744, in function
return Function(inputs, outputs, updates=updates, *
kwargs)
File "C:\Python\Python36\lib\site-packages\keras\backend\tensorflow_backend.py", line 2546, in __init__
with tf.control_dependencies(self.outputs):
File "C:\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 5004, in control_dependencies
return get_default_graph().control_dependencies(control_inputs)
File "C:\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 4543, in control_dependencies
c = self.as_graph_element(c)
File "C:\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 3490, in as_graph_element
return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
File "C:\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 3569, in _as_graph_element_locked
raise ValueError("Tensor %s is not an element of this graph." % obj)
ValueError: Tensor Tensor("regression/concat:0", shape=(?, ?, 4), dtype=float32) is not an element of this graph.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Python\Python36\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "E:\Dev\AI\Labs\DevoAgencyControl\venven\CaptureThread.py", line 82, in run
save_detected_video=False)
File "E:\Dev\AI\Labs\DevoAgencyControl\venven\imageai\Detection__init__.py", line 2297, in detectCustomObjectsFromVideo
"An error occured. It may be that your input video is invalid. Ensure you specified a proper string value for 'output_file_path' is 'save_detected_video' is not False. "
ValueError: An error occured. It may be that your input video is invalid. Ensure you specified a proper string value for 'output_file_path' is 'save_detected_video' is not False. Also ensure your per_frame, per_second, per_minute or video_complete_analysis function is properly configured to receive the right parameters.
````

The same code works fine when it is inside a while true loop. All my files are in the same folder and I have checked the video and model path are correct.

Hi,
i am sure about it but maybe that the size allocated for each thread(bearing in mind that all the threads same the same memory space) is not big enough to load the model iand maybe that's what is causing the error.(something to do with threading.stack_size())

I did get around my problem by using threads to do the other functions but call the object detection model from the main thread itself and works fine until now.

Tried to give them an email to reply ... looks like their email server is even jacked. Opps.

```An error occurred while trying to deliver the mail to the following recipients:
[email protected]
[email protected]

Technical report:

Reporting-MTA: dsn; a11-78.smtp-out.amazonses.com

Action: failed
Final-Recipient: rfc822; [email protected]
Diagnostic-Code: smtp; 550 5.1.1 moses@deepquest.com: Recipient address rejected: User unknown in relay recipient table
Status: 5.1.1

Action: failed
Final-Recipient: rfc822; [email protected]
Diagnostic-Code: smtp; 550 5.1.1 moses@deepquest.com: Recipient address rejected: User unknown in relay recipient table
Status: 5.1.1
```

I'm experiencing a similar issue while trying to run capture in one thread and imageai processing in another thread:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/root/miniconda3/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "detect.py", line 68, in func2
    results_array = prediction.predictMultipleImages(all_images_array, result_count_per_image=1, input_type="array")
  File "/root/miniconda3/lib/python3.7/site-packages/imageai/Prediction/Custom/__init__.py", line 721, in predictMultipleImages
    prediction = model.predict(image_to_predict, steps=1)
  File "/root/miniconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 1113, in predict
    self, x, batch_size=batch_size, verbose=verbose, steps=steps)
  File "/root/miniconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 195, in model_iteration
    f = _make_execution_function(model, mode)
  File "/root/miniconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 122, in _make_execution_function
    return model._make_execution_function(mode)
  File "/root/miniconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 1989, in _make_execution_function
    self._make_predict_function()
  File "/root/miniconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 1979, in _make_predict_function
    **kwargs)
  File "/root/miniconda3/lib/python3.7/site-packages/tensorflow/python/keras/backend.py", line 3201, in function
    return GraphExecutionFunction(inputs, outputs, updates=updates, **kwargs)
  File "/root/miniconda3/lib/python3.7/site-packages/tensorflow/python/keras/backend.py", line 2939, in __init__
    with ops.control_dependencies(self.outputs):
  File "/root/miniconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 5028, in control_dependencies
    return get_default_graph().control_dependencies(control_inputs)
  File "/root/miniconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 4528, in control_dependencies
    c = self.as_graph_element(c)
  File "/root/miniconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3478, in as_graph_element
    return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "/root/miniconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3557, in _as_graph_element_locked
    raise ValueError("Tensor %s is not an element of this graph." % obj)
ValueError: Tensor Tensor("activation_26/Softmax:0", shape=(?, 3), dtype=float32) is not an element of this graph.

As soon as I take the same code out of the thread/function, it processes as it should. I've tried passing the object and also used a global declaration and both produce the same error above. It doesn't seem to want to play.

When I was using the imageai in my pyqt5 project, I met the same problem.

"Ensure you specified correct input image, input type, output type and/or output image path ")
ValueError: Ensure you specified correct input image, input type, output type and/or output image path

However I solved that by laoding the model in the function "run()" of my thread. A part of code is like below. Hoping this can be helpful.

class run_thread(QThread):
    def __init__(self, parent=None):
        super(run_thread, self).__init__(parent)
        self.isRunning = True
        self.tag = True
        self.cap = cv2.VideoCapture(videoname)
    def run(self):
        execution_path = os.getcwd()
        detector = ObjectDetection()
        detector.setModelTypeAsYOLOv3()
        detector.setModelPath( os.path.join(execution_path , ./yolo.h5"))
        detector.loadModel()
        custom_objects = detector.CustomObjects(bus=True, car=True, truck=True)
        while self.tag:
            if(self.cap.isOpened()==True and self.isRunning == True):
                ret, frame = self.cap.read()
            if ret:
                _, contours = detector.detectCustomObjectsFromImage(custom_objects=custom_objects, input_type="array", input_image=frame, output_type="array", minimum_percentage_probability=30, display_percentage_probability=False, display_object_name=True)

I think #311 should solve this issue.

Please, upgrade ImageAI (pip install imageai==2.1.5) and check if it is solved now.

If this is the case, please close this issue

thanks for the comment @rola93

This issue has been solved in the latest version.

Greetings.
After spending a couple of days on trying to get imageAI to work with Flask, it was made obvious that the threading issue is not resolved.
To make a long story short if the prediction is executed on a different thread that the model was loaded it fails with the following error

ValueError: Tensor Tensor("predictions/Softmax:0", shape=(?, 56), dtype=float32) is not an element of this graph.

which is the same problem described in this issue and is related to this issue in Keras

I have tried many different solution proposed but nothing worked.
The only way that "worked" is to force Flask not to use threading
i.e
app.run(host='0.0.0.0',threaded = False)

I have not looked into the ImageAI code extensively but for custom image detection using InceptionV3 its 100% sure it does not work with threads.

Same issue as @skoroneos. I'm using ImageAI 2.1.5.

This issue has been solved in the latest version.

Not solved in latest version still getting error. I'm using ImageAI 2.1.5.

Was this page helpful?
0 / 5 - 0 ratings