Flask-socketio: SocketIO: Video Streaming from server to client

[Fizmath]

Dear Miguel:

Hello

This is from your video streaming page + emit , server side (i removed the Return):

@app.route('/video_feed')
def video_feed():
       socketio.emit('from_flask', Response(gen(), mimetype='multipart/x-mixed-replace; 
                                                boundary=frame'),namespace='/test')

here the problems are Response and the Generator mixing with the event-driven SocketIO

client

 socket.on('from_flask', function (data) { 
          $("#image").attr("src",data) ;
           });

html

<img id="image" src="video_feed">

Would you please help me ? i know something is wrong here but i cant't figure it

Thanks a million for your great work

[miguelgrinberg]

Generators are a Flask feature, they are not supported for Socket.IO events. If you want to send the video frames over Socket.IO you have to send them one by one as independent events, and then you need to implement the display of these frames in JavaScript for the client.

If you want to take advantage of the Motion JPEG streaming support in browsers, then you have to do it as I do it in my articles, using HTTP.

[Fizmath]

So, i should do something with the generator before feeding into the SocketIO , right ? for example passing it to the Redis then emitting one by one from Redis list ...
Any other possible solutions ?

[miguelgrinberg]

You can just treat the generator as an iterable, and then in each iteration emit an event with the frame payload.

for video_frame in gen():
    emit('my_video_frame', {'data': video_frame})

You will also need to change the generator to not include all the boundary stuff, which is not part of the video frame and is only necessary when streaming via HTTP.

[Fizmath]

Oh! greetings ! , thank you very much for your instant response .. I used this

https://github.com/dxue2012/python-webcam-flask

If you open it's demo you will see the mirrored image from the server , but if you or someone else simultaneously opens another browser you will see that the returned image from the other clients got mixed and shuffled , that is because the returned video is not sending through the SocketIO , also look how simply the video has been sent from the browser tho the SERVER through the SocketIO

I am struggling to find a way to send the returned video 1 - through the SocketIO , 2 - individually to the specific client ( you know : request.sid and room = id )

I will be very happy if i'd implent this task

Thank you again for your kind replies though thousand of miles away ....

[miguelgrinberg]

I don't think the example that you pointed out sends video through Socket.IO. This project is heavily based on my articles, and it uses HTTP to stream: https://github.com/dxue2012/python-webcam-flask/blob/master/app.py#L46-L49.

[Fizmath]

Yes , it does not send video from the server to the client throuth Socket.IO , but it sends from the browser to the server through Socket.IO : https://github.com/dxue2012/python-webcam-flask/blob/master/static/js/main.js#L19

am i right ?

[miguelgrinberg]

Yes, it sends a single frame back to the server, encoded as a data url. It is actually very similar to what I mentioned above, just put the jpeg data in an event.

[Fizmath]

OK )) thank you very much and also note that this is an good example showing how your protocol can be implemented on HEROKU .

[Fizmath]

This works fine and machine-guns base64 encoding from the flask server to the browser through SocketIO :

@app.route('/video_feed')
def video_feed():
    for video_frame in gen():
        socketio.emit('from_flask',{'data': video_frame} ,namespace='/test')

The base64 encoded JPEG video_frame starts with :

 /9j/4AAQSkZJRgABAQAAAQABAAD/2.....

the browser receives the encoding by :

  socket.on('from_flask', function (data) {
          $("#imageElement").attr("src","data:image/jpeg;base64,"+data);

it seems to me that something is missing in the above code , further on ,in the HTML :

<img id="imageElement" src="video_feed">

we have the route src="video_feed" and {"src","data:image/jpeg;base64,"+data} , they might conflict. The browser shows errors :

note the streaming frames 698 in seconds ! and [object ] ....

Thank you very much for your attentions

[miguelgrinberg]

I see two problems.

You seem to be mixing two different methods of streaming. If you use the browser support for motion jpeg like I did using HTTP, then you have to use the generator response. If you want to stream over Socket.IO, then you have to create your image without a URL, and then each time your socket.io client receives a frame it can refresh the src attribute.

The second, and most important, problem is that you are not handling the jpeg data correctly. The server sends a payload with the format {data: video_frame}, so on the client, you have to do this:

socket.on('from_flask', function (payload) {
    $("#imageElement").attr("src","data:image/jpeg;base64,"+payload['data']);
}

[Fizmath]

i describe and update new problem in the next comment :

[miguelgrinberg]

Sorry I don't understand what you want to do. If you want to emit to a single client you can, just set the room argument to the sid value of that client.

[Fizmath]

the client opens his browser , webcam captures his video and sends through Socket.IO to the server , the server handles the incoming images and reflexes the modified images through the SocketIO :

```
users = []

@socketio.on('connect', namespace='/test')
def test_connect():

users[:] = []
users.append(request.sid)
app.logger.info("client connected")

..

@app.route('/video_feed')
def video_feed():
for video_frame in gen():
print(users[0])
socketio.emit('from_flask',{'data': video_frame} , namespace='/test' , room = users[0] )
` every time a new client connects, his webcam sends video , the single element users list updates and thissid get fed into the from_flaskevent , room = user[0] ``` , but things got messed up : while the first client smoothly sees his reflected video ,suddenly the second client connects and he receives not his reflected images from the server but the reflected images of the first client , and even the reflected images of both clients got mixed as if there were no individual channels for them

[miguelgrinberg]

You need separate background threads to handle the video for each client, and you need a way for clients to authenticate, so that the server knows which video stream to send when the client connects to the video feed route.

[Fizmath]

Thank you very very much . By your advice now i know what to do : i just start learning the subjects background threads and authentication

[Fizmath]

Is it possible to use threaded=True to handle this problem ?

[miguelgrinberg]

The threaded option handles requests in their own threads. This is different, since you have background threads for the video streams, running separately from client requests.

[Fizmath]

in camera class we have thread daemon

class Camera(object):
    def __init__(self, makeup_artist):
        self.to_process = []
        self.to_output = []
        self.makeup_artist = makeup_artist

        thread = threading.Thread(target=self.keep_processing, args=())
        thread.daemon = True
        thread.start()

    def process_one(self):
        if not self.to_process:
            # return to exit whole function
            return

        # input is an ascii string. 
        input_str = self.to_process.pop(0)

        # convert it to a pil image
        input_img = base64_to_pil_image(input_str)

        ################## where the hard work is done ############
        # output_img is an PIL image
        output_img = self.makeup_artist.apply_makeup(input_img)

        # output_str is a base64 string in ascii
        output_str = pil_image_to_base64(output_img)

        # convert eh base64 string in ascii to base64 string in _bytes_

        self.to_output.append(output_str)
        #self.to_output.append(binascii.a2b_base64(output_str))

    def keep_processing(self):
        while True:
            self.process_one()
            sleep(0.01)

    def enqueue_input(self, input):
        self.to_process.append(input)

    def get_frame(self):
        while not self.to_output:
            sleep(0.05)
        return self.to_output.pop(0)

you mean that i should separate threads in this class ? or should write threading code in initiation connect event ?

@socketio.on('connect', namespace='/test')
def test_connect():
    app.logger.info("client connected")

i really got perplexed for a few days finding the solution ... thank you very much for you advises .. i am waiting for your guides ....

[miguelgrinberg]

I cannot give you specific details, as I don't know the requirements of your application. But based on what you told me so far, you need a separate Camera instance, with its corresponding background thread, for each client.

[Fizmath]

OK , thank you . I'll try .
Let's assume that the Camera class defined in the above comment receives frames from a client , the thread.daemon starts and then suddenly second client sends his frames , where and how should i code the background thread to separate the processing ?

[miguelgrinberg]

You are asking me to provide a solution that I never intended to implement myself, so I really don't know. There are many different ways to implement multiple streams, I would need to know the particulars of your application. Without knowing details, I expect you would have a different Camera instance for each client, but how to implement that depends on your needs.

[Fizmath]

Here is a fundamental question :
Assuming that we have imported theCamera module and make a single instance of it

````python
from sys import stdout
from makeup_artist import Makeup_artist
import logging
from flask import Flask, render_template, Response , request, session
from flask_socketio import SocketIO, emit
from camera import Camera
from utils import base64_to_pil_image, pil_image_to_base64

app = Flask(__name__)
app.logger.addHandler(logging.StreamHandler(stdout))
app.config['SECRET_KEY'] = 'secret!'
app.config['DEBUG'] = True
socketio = SocketIO(app)

users = []
camera = Camera(Makeup_artist())

@socketio.on('input image', namespace='/test')
def test_message(input):
input = input.split(",")[1]
camera.enqueue_input(input)

@socketio.on('connect', namespace='/test')
def test_connect():

users..append(request.sid)
print(users)
app.logger.info("client connected")

@app.route('/')
def index():
"""Video streaming home page."""
return render_template('index.html')

def gen():
app.logger.info("starting to generate frames!")
while True:
frame = camera.get_frame() #pil_image_to_base64(camera.get_frame())
yield frame

    # yield (b'--frame\r\n'
    #        b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')

@app.route('/video_feed')
def video_feed():
"""Video streaming route. Put this in the src attribute of an img tag."""
for video_frame in gen():
socketio.emit('from_flask',{'data': video_frame.decode()} , namespace='/test' )

if __name__ == '__main__':
socketio.run(app)
````

the global camera instance appears twice : inside input image event and inside gen() method . Once the server runs the camera get uploaded just a single time no matter how many clients connect ,any idea how we can create new instances of Cameraevery time a client connects ?

[miguelgrinberg]

This is what I suggested a few days ago:

You need separate background threads to handle the video for each client, and you need a way for clients to authenticate, so that the server knows which video stream to send when the client connects to the video feed route.

Once you know who each user is, you can maintain a camera object for each, or if you prefer, use a single camera object with one background thread per client.

[Fizmath]

Can i do this ?:

...
from makeup_artist import Makeup_artist
from camera import Camera
.
.

@socketio.on('connect', namespace='/test')
def test_connect():
    session['instance'] = Camera(Makeup_artist())     # pass instance of Camera to flask session
    app.logger.info("client connected")

@socketio.on('input image', namespace='/test')   
def test_message(input):
    input = input.split(",")[1]
    session['instance'].enqueue_input(input)             #  then use that
.
.

is it legal ?

[miguelgrinberg]

Yes, nothing wrong with that.

[Fizmath]

Yes, it works fine inside the socketio.on and gives camera object but inside app.route gives an error:

```python

here put the socketio code from the above comment

@app.route('/')
def index():
"""Video streaming home page."""
return render_template('index.html')

def gen():
"""Video streaming generator function."""
app.logger.info("starting to generate frames!")
while True:
frame = session['instance'].get_frame() # gives error , see below
yield frame

````

[miguelgrinberg]

Did you read the limitations with the session? Changes made in a socket.io handler are not visible in HTTP handlers. See https://flask-socketio.readthedocs.io/en/latest/#access-to-flask-s-context-globals for details.

[Fizmath]

I delayed to find the answer and the issue closed , would you please kindly give me some guidance :

Here is a fundamental question :
Assuming that we have imported theCamera module and make a single instance of it

from sys import stdout
from makeup_artist import Makeup_artist
import logging
from flask import Flask, render_template, Response , request, session
from flask_socketio import SocketIO, emit
from camera import Camera
from utils import base64_to_pil_image, pil_image_to_base64


app = Flask(__name__)
app.logger.addHandler(logging.StreamHandler(stdout))
app.config['SECRET_KEY'] = 'secret!'
app.config['DEBUG'] = True
socketio = SocketIO(app)

users = []
camera = Camera(Makeup_artist())




@socketio.on('input image', namespace='/test')
def test_message(input):
    input = input.split(",")[1]
     camera.enqueue_input(input)

@socketio.on('connect', namespace='/test')
def test_connect():

    users..append(request.sid)
    print(users)
    app.logger.info("client connected")



@app.route('/')
def index():
    """Video streaming home page."""
    return render_template('index.html')



def gen():
    app.logger.info("starting to generate frames!")
    while True:
        frame = camera.get_frame() #pil_image_to_base64(camera.get_frame())
        yield frame

        # yield (b'--frame\r\n'
        #        b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')



@app.route('/video_feed')
def video_feed():
    """Video streaming route. Put this in the src attribute of an img tag."""
    for video_frame in gen():
        socketio.emit('from_flask',{'data': video_frame.decode()} , namespace='/test' )




if __name__ == '__main__':
    socketio.run(app)

the global camera instance appears twice : inside input image event and inside gen() method . Once the server runs the camera get uploaded just a single time no matter how many clients connect ,any idea how we can create new instances of Cameraevery time a client connects ?

1 -the camera instance is imported once , how can i create a new instance for every new client , knowing his sid from inside socketio ?
2 - the thread starts inside camera class:
https://github.com/miguelgrinberg/Flask-SocketIO/issues/778#issuecomment-422047892

how can i create a new thread for a newly connected client?

Thank you very much for your kind remarks

[miguelgrinberg]

This is really a situation that I have never thought about in detail, so it is hard for me to give you advice. My suggestion was that you add authentication, so that you know who each user is. This is necessary so that you can recognize the same user when it contacts the server via socket.io and HTTP. Without authentication you have no way to match requests and socket events as coming from the same user.

Once you have that, one way to keep track of all the camera instances is to use a global dictionary:

cameras = {}

# to add a camera
cameras[username] = Camera()

Sorry I can't be more specific. You keep asking me about a specific solution, but unfortunately I have no code to show you that does what you want.

[Fizmath]

Thank you very much , i tried the above method but the desired result not achieved
I am racking my brains out of tune solving this problem , i will inform you as soon as a get the solution

[Fizmath]

Here is a general question : first i post a session value :

````python
cameras = {}

@app.route('/session', methods=['POST'])
def session_access():
global camera
if 'session' in data:
session['value'] = data['session']
username = session.get('value', '')
cameras[username] = Camera()
return '', 204

````

after the above Post i want the below route to be uploaded ! because i first set the camera object in the above code . should i use redirect?
should i use Login Required Decorator http://flask.pocoo.org/docs/1.0/patterns/viewdecorators/ ?

````python

@app.route('/video_feed')
def video_feed():
global camera
for video_frame in gen(cameras[username]):
socketio.emit('from_flask',{'data': video_frame.decode()} , namespace='/test', room = sidd )
```

[miguelgrinberg]

I don't understand what you are trying to do, sorry. Your first function references a data variable that is undefined. Your second function references username which is also undefined. I'm really confused, does this code run at all?

I don't understand your question either. The /video_feed route is likely going to be invoked from a <img> tag in a web page, right? So you will never redirect to it.

[Fizmath]

so , any idea ? i am still stuck in this quagmire

[miguelgrinberg]

@Fizmath I don't know, sorry. I don't understand your code. I tried to explain what you need to do to the best of my ability. As I said before, I don't have sample code to show you, I have never implemented what you need.

[Fizmath]

Ok. thanks

[LazyerIJ]

hello. did u solve this problem? I'm trying perfectly same thing but i can't know how to do it. thanks.

[thisisashukla]

Hi @Fizmath did you get good enough latency from socketio.emit method for video streaming. i am doing the same but getting high latency. Link to my question https://stackoverflow.com/questions/57088967/how-to-speed-up-flask-socketio-emit/57207738#57207738

[thisisashukla]

You can just treat the generator as an iterable, and then in each iteration emit an event with the frame payload.

for video_frame in gen():
    emit('my_video_frame', {'data': video_frame})

You will also need to change the generator to not include all the boundary stuff, which is not part of the video frame and is only necessary when streaming via HTTP.

This method gave me poor FPS while streaming a recorded video to browser from flask server.

[rajdip34]

Hey anyone solves this problem ?

[AhmedBhati]

I have just done live video-streaming with flask-socketio, uploaded on the github just check, might help...https://github.com/AhmedBhati/video-streaming-with-flask-socketio

[whikwon]

@thisisashukla I also have high latency when using socketio.emit method did you find the solution?

[AhmedBhati]

@whikwon if you are using eventlet as the server for sending frames put broadcast=True in emit

for video_frame in gen():
    emit('my_video_frame', {'data': video_frame}, broadcast=True)

[whikwon]

@AhmedBhati I can't find the difference with, without broadcast argument.

Is there anything wrong with my code?

app.py

import base64
import time
import cv2
from flask import Flask, render_template, request
from flask_socketio import SocketIO, emit
import eventlet

eventlet.monkey_patch()
app = Flask(__name__)
socketio = SocketIO(app, logger=True, async_mode='eventlet')


@socketio.on('connect', namespace='/web')
def connect_web():
    print('[INFO] Web client connected: {}'.format(request.sid))


@socketio.on('disconnect', namespace='/web')
def disconnect_web():
    print('[INFO] Web client disconnected: {}'.format(request.sid))


@socketio.on('connect', namespace='/local')
def connect_cv():
    print('[INFO] CV client connected: {}'.format(request.sid))


@socketio.on('disconnect', namespace='/local')
def disconnect_cv():
    print('[INFO] CV client disconnected: {}'.format(request.sid))


@app.route('/')
def index():
    return render_template('index.html', async_mode=socketio.async_mode)


@socketio.on('stream_request')
def stream_video(message):
    socketio.emit('stream_response', message, namespace='/web', broadcast=True)
    return 


if __name__ == '__main__':
    socketio.run(app, host="127.0.0.1", port=5000, debug=True)

camera.py

import datetime
from threading import Thread
import base64
import time
import cv2
import socketio

sio = socketio.Client(logger=True)
sio.connect('http://127.0.0.1:5000',
            transports=['websocket'],
            namespaces=['/local'])

@sio.event
def connect():
    print('[INFO] Successfully connected to server')


@sio.event
def connect_error():
    print('[INFO] Failed to connect to server.')


@sio.event
def disconnect():
    print('[INFO] Disconnected from server.')


def encode_image(image):
    image = cv2.imencode('.jpg', image)[1].tobytes()
    image = base64.b64encode(image).decode('utf-8')
    image = f"data:image/jpeg;base64,{image}"
    return image


class FPS:
    def __init__(self):
        # store the start time, end time and total number of frames
        # that were examined between the start and end intervals
        self._start = None
        self._end = None
        self._numFrames = 0

    def start(self):
        # start the timer
        self._start = datetime.datetime.now()
        return self

    def stop(self):
        # stop the timer
        self._end = datetime.datetime.now()

    def update(self):
        # increment the total number of frames examined during the
        # start and end intervals
        self._numFrames += 1

    def elapsed(self):
        # return the total number of seconds between the start and 
        # end interval
        return (self._end - self._start).total_seconds()

    def fps(self):
        # compute the (approximate) frames per second
        return self._numFrames / self.elapsed()

class WebCamVideoStream:
    def __init__(self, src=0):
        # initialize the video camera stream and read the first frame 
        # from the stream
        self.stream = cv2.VideoCapture(src)
        (self.grabbed, self.frame) = self.stream.read()

        # initialize the variable used to inidicate if the thread 
        # should be stopped
        self.stopped = False

    def start(self):
        # start the thread to read frames from the video stream
        Thread(target=self.update, args=()).start()
        return self

    def update(self):
        # keep looping infinitely until the thread is stopped
        while True:
            # if the thread indicator variable is set, stop the thread
            if self.stopped:
                return
            # otherwise read the next frame from the stream
            (self.grabbed, self.frame) = self.stream.read()

    def read(self):
        # return the frame most recently read
        return self.frame

    def stop(self):
        # indicate that the thread should be stopped
        self.stopped = True


def main():
    # SLOW VERSION
    # cap = cv2.VideoCapture(0)
    # cap.set(cv2.CAP_PROP_FRAME_WIDTH, 600)
    # cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 600)
    stream = WebCamVideoStream(src=0).start()
    i = 0
    while True:
        # ret, frame = cap.read()
        frame = stream.read()
        frame = cv2.resize(frame, (600, 600))
        sio.emit('stream_request', {'image': encode_image(frame)})
        k = cv2.waitKey(1) 
        if k == 27: 
            break
        i += 1
        time.sleep(0.05)

    stream.stop()
    # cap.release()
    cv2.destroyAllWindows()


if __name__ == "__main__":
    main()

[AhmedBhati]

@whikwon can you send your index.html so i would run this code up and you can also refer the code written by me on video streaming using flask-socketio

[whikwon]

@AhmedBhati Here's my index.html file.

<html>
<head>
    <title>SocketIO</title>
    <script src="//cdnjs.cloudflare.com/ajax/libs/socket.io/2.2.0/socket.io.js" integrity="sha256-yr4fRk/GU1ehYJPAs8P4JlTgu0Hdsp4ZKrx8bDEDC3I=" crossorigin="anonymous"></script>
    <style>
        .container {
            display: grid;
            grid-template-rows: repeat(2, 1fr);
            grid-template-columns: repeat(2, 1fr);
        }
        .contents {
            background: white;
        }
        .images {
            background: #665c9c;
        }
    </style>
</head>
<body>
    <div class="container">
        <div class="images">
            <img id="streamed-image" src="">
        </div>
        <div class="contents">
            <button id='play-btn'>PLAY</button>
        </div>
    </div>
    <script type="text/javascript" charset="utf-8">
        const namespace = '/web';
        const socket = io(namespace);
        socket.on('stream_response', (msg) => {
            document.querySelector('#streamed-image').src = msg.image;
        });
    </script>
</body>
</html>

[whikwon]

@AhmedBhati I've ran your source code and it is faster than mine. What is the difference between them?

[AhmedBhati]

i have executed your code, @whikwon Firstly reduce the frame size the smaller the better, secondly remove the delay time.sleep(0.05) from the code.

[whikwon]

@AhmedBhati If I remove the delay, lagging are getting more and more. If I use flask Response instead like the code below, I found socketio between javascript and python doesn't work. Is it normal?

def video_generator():
    import imagezmq
    subscriber = imagezmq.ImageHub('tcp://127.0.0.1:5555', REQ_REP=False)
    while True:
        camera_id, frame = subscriber.recv_image()
        frame = cv2.imencode('.jpg', frame)[1].tobytes()
        yield (b'--frame\r\n'
               b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')


@app.route('/video_feed/', methods=['POST', 'GET'])
def stream_imagezmq():
    return Response(video_generator(), mimetype='multipart/x-mixed-replace; boundary=frame')

[AhmedBhati]

@whikwon yeah but this is based on http protocol not on websockets, I'll send your modified code also. run it once an see

[whikwon]

@AhmedBhati Thank you so much, What is the big difference between http and websocket? Is there impact on performance? or else?

[AhmedBhati]

Performance wise i found websockets faster and better than http when using multiple clients because for every request there would a three way handshake in http whereas it only in the start in websocket.

If you want to see the difference I have added a repository in which I have used http as the protocol, i have attached a demo video of the output there. So you can compare between the either

# app.py
import base64
import time
import cv2
from time import sleep
from flask import Flask, render_template, request
from flask_socketio import SocketIO, emit
import eventlet

eventlet.monkey_patch()
app = Flask(__name__)
socketio = SocketIO(app, logger=True, async_mode='eventlet')


@socketio.on('connect', namespace='/web')
def connect_web():
    print('[INFO] Web client connected: {}'.format(request.sid))


@socketio.on('disconnect', namespace='/web')
def disconnect_web():
    print('[INFO] Web client disconnected: {}'.format(request.sid))


@socketio.on('connect', namespace='/local')
def connect_cv():
    print('[INFO] CV client connected: {}'.format(request.sid))


@socketio.on('disconnect', namespace='/local')
def disconnect_cv():
    print('[INFO] CV client disconnected: {}'.format(request.sid))


@app.route('/')
def index():
    return render_template('index.html', async_mode=socketio.async_mode)


@socketio.on('stream_request')
def stream_video(message):
    socketio.emit('stream_response', message, namespace='/web', broadcast=True)

if __name__ == '__main__':
    socketio.run(app,host="127.0.0.1", port=5000)

# camera.py
import datetime
from threading import Thread
import base64
import time
import cv2
import socketio

sio = socketio.Client(logger=True)

@sio.event
def connect():
    print('[INFO] Successfully connected to server')

@sio.event
def connect_error():
    print('[INFO] Failed to connect to server.')

@sio.event
def disconnect():
    print('[INFO] Disconnected from server.')

def encode_image(image):
    image = cv2.imencode('.jpg', image)[1].tobytes()
    image = base64.b64encode(image).decode('utf-8')
    image = f"data:image/jpeg;base64,{image}"

    return image

class FPS:
    def __init__(self):
        # store the start time, end time and total number of frames
        # that were examined between the start and end intervals
        self._start = None
        self._end = None
        self._numFrames = 0

    def start(self):
        # start the timer
        self._start = datetime.datetime.now()
        return self

    def stop(self):
        # stop the timer
        self._end = datetime.datetime.now()

    def update(self):
        # increment the total number of frames examined during the
        # start and end intervals
        self._numFrames += 1

    def elapsed(self):
        # return the total number of seconds between the start and 
        # end interval
        return (self._end - self._start).total_seconds()

    def fps(self):
        # compute the (approximate) frames per second
        return self._numFrames / self.elapsed()

class WebCamVideoStream:
    def __init__(self, src=0):
        # initialize the video camera stream and read the first frame 
        # from the stream
        self.stream = cv2.VideoCapture(src)
        (self.grabbed, self.frame) = self.stream.read()

        # initialize the variable used to inidicate if the thread 
        # should be stopped
        self.stopped = False

    def start(self):
        # start the thread to read frames from the video stream
        Thread(target=self.update, args=()).start()
        return self

    def update(self):
        # keep looping infinitely until the thread is stopped
        while True:
            # if the thread indicator variable is set, stop the thread
            if self.stopped:
                return
            # otherwise read the next frame from the stream
            (self.grabbed, self.frame) = self.stream.read()

    def read(self):
        # return the frame most recently read
        return self.frame

    def stop(self):
        # indicate that the thread should be stopped
        self.stopped = True


def main():
    # SLOW VERSION
    cap = cv2.VideoCapture(0)
    #cap.set(cv2.CAP_PROP_FRAME_WIDTH, 600)
    #cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 600)
    # stream = WebCamVideoStream(src=0).start()
    i = 0
    while True:
        ret, frame = cap.read()
        #frame = stream.read()
        frame = cv2.resize(frame, (320,280))
        sio.emit('stream_request', {'image': encode_image(frame)})
    #stream.stop()
    cap.release()
    #cv2.destroyAllWindows()


if __name__ == "__main__":
    sio.connect('http://127.0.0.1:5000',namespaces=['/local'])
    main()

[whikwon]

I'm sorry but I can't find the difference in code you shared compared to mine. Could you let me know?

By the way, I found the way for real-time streaming based on your repository.

app.py

import base64
import time
import cv2
from flask import Flask, render_template, request, Response
from flask_socketio import SocketIO, emit
import eventlet

eventlet.monkey_patch()
app = Flask(__name__)
socketio = SocketIO(app, logger=True, async_mode='eventlet')


@socketio.on('connect', namespace='/web')
def connect_web():
    print('[INFO] Web client connected: {}'.format(request.sid))


@socketio.on('disconnect', namespace='/web')
def disconnect_web():
    print('[INFO] Web client disconnected: {}'.format(request.sid))


@socketio.on('connect', namespace='/local')
def connect_cv():
    print('[INFO] CV client connected: {}'.format(request.sid))


@socketio.on('disconnect', namespace='/local')
def disconnect_cv():
    print('[INFO] CV client disconnected: {}'.format(request.sid))


@app.route('/')
def index():
    return render_template('index.html', async_mode=socketio.async_mode)


@socketio.on('inc_request', namespace='/web')
def inc_request(message):
    message += 'a'
    socketio.emit('inc_response', message, namespace='/web')


@socketio.on('stream_request', namespace='/web')
def video_generator():
    import imagezmq
    subscriber = imagezmq.ImageHub('tcp://127.0.0.1:5555', REQ_REP=False)
    while True:
        camera_id, frame = subscriber.recv_image()
        frame = cv2.imencode('.jpg', frame)[1].tobytes()
        frame = base64.encodebytes(frame).decode('utf-8')
        frame = f"data:image/jpeg;base64,{frame}"
        socketio.emit('stream_response', {'image': frame}, namespace='/web')
        socketio.sleep(0)


if __name__ == '__main__':
    socketio.run(app, host="127.0.0.1", port=5000, debug=True)

index.html

<html>
<head>
    <title>SocketIO</title>
    <script src="//cdnjs.cloudflare.com/ajax/libs/socket.io/2.2.0/socket.io.js" integrity="sha256-yr4fRk/GU1ehYJPAs8P4JlTgu0Hdsp4ZKrx8bDEDC3I=" crossorigin="anonymous"></script>
    <style>
        .container {
            display: grid;
            grid-template-rows: repeat(2, 1fr);
            grid-template-columns: repeat(2, 1fr);
        }
        .contents {
            background: white;
        }
        .images {
            background: #665c9c;
        }
    </style>
</head>
<body>
    <div class="container">
        <div class="images">
            <img id="streamed-image" src="">
        </div>
        <div class="contents">
            <button id='btn'>a</button>
        </div>
    </div>
    <script type="text/javascript" charset="utf-8">
        const namespace = '/web';
        const socket = io(namespace);

        document.addEventListener('DOMContentLoaded', () => {
            socket.on('stream_response', (msg) => {
                document.querySelector('#streamed-image').src = msg.image;
            });
            socket.on('inc_response', (msg) => {
                document.querySelector('#btn').textContent = msg; 
            });
            socket.emit('stream_request');
        });
        document.querySelector('#btn').addEventListener('click', () => {
            socket.emit('inc_request', document.querySelector('#btn').textContent);
        });
    </script>
</body>
</html>

camera.py

import cv2
import imagezmq


def main():
    publisher = imagezmq.ImageSender(connect_to='tcp://127.0.0.1:5555', REQ_REP=False)
    cap = cv2.VideoCapture(0)
    cap.set(cv2.CAP_PROP_FRAME_WIDTH, 600)
    cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 600)
    i = 0
    while True:
        ret, frame = cap.read()
        publisher.send_image('camera', frame)
        # cv2.imshow('imagezmq', frame)
        if i % 10:
            print(i)
        i += 1

    cap.release()
    cv2.destroyAllWindows()


if __name__ == "__main__":
    main()

[AhmedBhati]

@whikwon I have just reduced the frame size and the delay and have attached the video of your code's output. Check it out
videostreaming.zip

and bascially in the above attached code in camera.py you are sending frames to the 5555 port and then taking the frames from that port and sending over to 5000 which will create more delay

[whikwon]

I needed to separate camera for other usage. I've checked your video and works very well.

Thank you for your help. I'll try the experiment with your advise!

[Zarvoira]

hello guys im trying similiar thing,i want the client (raspberry pi) to send video to the remote server (aws), what should i do with @whikwon @AhmedBhati example?, im not quite sure, should i put the camera.py and run it on pi and put the flask server in aws??

[Zarvoira]

i want that my camera.py run on pi and send it over socket somehow and the server(aws) receive it on flask and

right now i have

camera.py where i get the picture from webcam and do some processing

#Modified by
#Date: 27.06.20
#Desc: This scrtipt is running a face recongition of a live webcam stream. This is a modifed
#code of the orginal Ageitgey (GitHub) face recognition demo to include multiple faces.
#Simply add the your desired 'passport-style' face to the 'profiles' folder.

import face_recognition
import cv2
import numpy as np
import os
face_cascade=cv2.CascadeClassifier("haarcascade_frontalface_alt2.xml")
ds_factor=0.6

#Store objects in array
known_person=[] #Name of person string
known_image=[] #Image object
known_face_encodings=[] #Encoding object

# Initialize some variables
face_locations = []
face_encodings = []
face_names = []
process_this_frame = True

#Loop to add images in friends folder
for file in os.listdir("profiles"):
    try:
        #Extracting person name from the image filename eg: david.jpg
        known_person.append(file.replace(".jpg", ""))
        file=os.path.join("profiles/", file)
        known_image = face_recognition.load_image_file(file)
        #print("test")
        #print(face_recognition.face_encodings(known_image)[0])
        known_face_encodings.append(face_recognition.face_encodings(known_image)[0])
        #print(known_face_encodings)

    except Exception as e:
        pass

#print(len(known_face_encodings))
#print(known_person)


class VideoCamera(object):
    def __init__(self):
        self.video = cv2.VideoCapture(0)

    def __del__(self):
        self.video.release()

    def get_frame(self):
        success, image = self.video.read()

        process_this_frame = True

            # Resize frame of video to 1/4 size for faster face recognition processing
        small_frame = cv2.resize(image, (0, 0), fx=0.25, fy=0.25)

        # Convert the image from BGR color (which OpenCV uses) to RGB color (which face_recognition uses)
        rgb_small_frame = small_frame[:, :, ::-1]

       # Only process every other frame of video to save time
        if process_this_frame:
            # Find all the faces and face encodings in the current frame of video
            face_locations = face_recognition.face_locations(rgb_small_frame)
            face_encodings = face_recognition.face_encodings(rgb_small_frame, face_locations)

            global name_gui;
            #face_names = []
            for face_encoding in face_encodings:
                # See if the face is a match for the known face(s)
                matches = face_recognition.compare_faces(known_face_encodings, face_encoding)
                name = "Unknown"

                #print(face_encoding)
                print(matches)

                face_distances = face_recognition.face_distance(known_face_encodings, face_encoding)
                best_match_index = np.argmin(face_distances)
                if matches[best_match_index]:
                    name = known_person[best_match_index]

                print(name)
                #print(face_locations)
                face_names.append(name)

                name_gui = name

        process_this_frame = not process_this_frame

# Display the results
        for (top, right, bottom, left), name in zip(face_locations, face_names):
            # Scale back up face locations since the frame we detected in was scaled to 1/4 size
            top *= 4
            right *= 4
            bottom *= 4
            left *= 4

            # Draw a box around the face
            cv2.rectangle(image, (left, top), (right, bottom), (255, 255, 255), 2)

            # Draw a label with a name below the face
            cv2.rectangle(image, (left, bottom - 35), (right, bottom), (255, 255, 255), cv2.FILLED)
            font = cv2.FONT_HERSHEY_DUPLEX
            cv2.putText(image, name_gui, (left + 10, bottom - 10), font, 1.0, (0, 0, 0), 1)


        ret, jpeg = cv2.imencode('.jpg', image)
        return jpeg.tobytes()

main.py where my flask server run

from flask import Flask, render_template, Response, request
from camera import VideoCamera
import time
import os

app = Flask(__name__)
#app = Flask(__name__, template_folder='/var/www/html/templates')

#background process happening without any refreshing


@app.route('/', methods=['GET', 'POST'])
def move():
    result = ""
    if request.method == 'POST':

        return render_template('index.html', res_str=result)

    return render_template('index.html')


def gen(camera):
    while True:
        frame = camera.get_frame()
        yield (b'--frame\r\n'
               b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n\r\n')

@app.route('/video_feed')
def video_feed():
    return Response(gen(VideoCamera()),
                    mimetype='multipart/x-mixed-replace; boundary=frame')

if __name__ == '__main__':
    app.run(host='192.168.1.15', debug=True, threaded=True)

and the index.html where i display the video

<div class="main" id="newpost">
  <img  class="camera-bg" style="width: 100%; height:80%; background-attachment: fixed;" id="bg" class="center" src="{{ url_for('video_feed') }}">
  <!--<img  class="camera-bg" style="width: 100%; height:80%; background-attachment: fixed;" id="bg" class="center" src="https://www.psdbox.com/wp-content/uploads/2011/01/security-camera-photoshop-effect.jpg">-->

</div>

[AhmedBhati]

@Zarvoira I haven't used flask with amazon aws but if you want to send a video from client to remote server try using python-socketio, wherein the client will take the video feed and send it to the server and server will display the video, I have tried this and it worked for only live video feeds it wasn't able to send saved video feeds.

[Zarvoira]

hey can u give me a sample code how to do that @AhmedBhati

[AhmedBhati]

I'll upload the code in Github repo by this week end, will share the link of the repository @Zarvoira

[AhmedBhati]

@Zarvoira go through the Flask SocketIO repository, the demo video as well as the code is present in it.

[Zarvoira]

thank you @AhmedBhati you should make youtube videos about this i think there is alot of people want to see these stuff!

[AhmedBhati]

@Zarvoira thanks for the suggestion, if you face any difficulty ping me back or raise an issue, would be happy to help.

[slowpoison752]

@miguelgrinberg i have a flask app running, it takes video stream from client webcam, processes it and creates a log file at server end. It works fine for a single client. Now i want to be able to server 1000's of client if and when needed, what will be the most efficient way to go about this? i am not sending processed video to client end, client system will only have its own live stream. I tried it with 2 client and the video streams got scrambled. Why is this so? Is this an issue with flask?

[miguelgrinberg]

@slowpoison752 The only reason I can think of for the video from the two clients getting mixed up is that this is a bug in your application. Nothing in Flask or Flask-SocketIO can cause that.