Howdy: HP Spectre x360 13" IR camera support

Created on 1 Dec 2018 · 32Comments · Source: boltgolt/howdy

OpenCV does not currently support my IR camera (/dev/video2), although it does support my regular camera (/dev/video0).

I get this error message from OpenCV. While I was able to download and compile a newer version (3.4.4 and 4.0.0) of OpenCV, I could not figure out how to force Howdy to use it.

VIDEOIO ERROR: V4L2: Pixel format of incoming image is unsupported by OpenCV

However, FFMPEG does support the IR camera. I was able to locally modify the add.py and compare.py files to use the FFMPEG python library, using a config variable set by "sudo howdy config" and an IF statement in the code that determines whether to use opencv or ffmpeg. The only issues I'm running into right now is the quality of the IR image is poor in bright daylight or just heavy brightness. This is also happening in Windows as well. I was in an airport yesterday sitting near a window and neither Windows nor the newly modified Howdy would detect my face. When I saved the photo, it appeared that half of my face was super dark, so that kind of makes sense for detection issues. I'm going to work on a dual solution, where it uses both the IR cameras and the regular camera to either merge the images together to increase quality or just attempt to use the normal camera image if the IR image fails to detect due to brightness or other issues. Also, I'm going to attempt to speed up the detection process a little bit. It seems to take a few seconds longer in Howdy than on Windows, likely my own code causing slowness. :-)

What's the best path for me to get these changes merged?
Maybe a better question is, would this be something you would be interested in merging? If not, I'll just fork the repo and use it privately (unless/until openCv works with my IR camera.)

It seems to me like this could open up Howdy to more people, if openCv doesn't detect or work properly, they could possibly use/try ffmpeg detection instead. :)

enhancement

Source

timwelch

All 32 comments

Basically, the changes are thus:

In both files, add these imports...

import ffmpeg
import numpy
from skimage import color

In add.py

# Open the camera
if config.get("video", "recording_plugin") == "opencv":
    video_capture = cv2.VideoCapture(config.get("video", "device_path"))

    # Force MJPEG decoding if true
    if config.get("video", "force_mjpeg") == "true":
        video_capture.set(cv2.CAP_PROP_FOURCC, 1196444237)

    # Set the frame width and height if requested
    if int(config.get("video", "frame_width")) != -1:
        video_capture.set(cv2.CAP_PROP_FRAME_WIDTH, int(config.get("video", "frame_width")))

    if int(config.get("video", "frame_height")) != -1:
        video_capture.set(cv2.CAP_PROP_FRAME_HEIGHT, int(config.get("video", "frame_height")))

    # Request a frame to wake the camera up
    video_capture.read()
else:
    # Request a frame to wake the camera up
    video_capture = (
        ffmpeg
        .input(config.get("video", "device_name"), format=config.get("video", "device_format"))
        .output('pipe:', vframes=1, format='image2', vcodec='mjpeg')
        .run(capture_stdout=True)
    )
    probe = ffmpeg.probe(config.get("video", "device_name"))

    if int(config.get("video", "frame_width")) != -1:
        frame_width = int(config.get("video", "frame_width"))
    else:
        frame_width = probe['streams'][0]['width']

    if int(config.get("video", "frame_height")) != -1:
        frame_height = int(config.get("video", "frame_height"))
    else:
        frame_height = probe['streams'][0]['height']



...
...
# Loop through frames till we hit a timeout
while frames < 60:
    frames += 1

    # Grab a single frame of video
    # Don't remove ret, it doesn't work without it
    if config.get("video", "recording_plugin") == "opencv":
        ret, frame = video_capture.read()
    else:
        stream, ret = (
            ffmpeg
            .input(config.get("video", "device_name"), format=config.get("video", "device_format"))
            .filter('select', 'gte(n,{})'.format(24))
            .output('pipe:', vframes=1, format='image2', vcodec='mjpeg')
            .run(capture_stdout=True)
        )
        nframe = numpy.frombuffer(stream, numpy.uint8)
        cframe = cv2.imdecode(nframe, 0)
        frame = color.gray2rgb(cframe, alpha=None)

And then in compare, I started adding a way to increase brightness over the course of multiple tries, since in very dark places, my IR camera was really dark. Maybe something with the way I saved the image, not sure...

compare.py

def increase_brightness(img, value=30):
    hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
    h, s, v = cv2.split(hsv)

    lim = 255 - value
    v[v > lim] = 255
    v[v <= lim] += value

    final_hsv = cv2.merge((h, s, v))
    img = cv2.cvtColor(final_hsv, cv2.COLOR_HSV2BGR)
    return img

...
...


if config.get("video", "recording_plugin") == "opencv":

    # Start video capture on the IR camera
    video_capture = cv2.VideoCapture(config.get("video", "device_path"))

    # Force MJPEG decoding if true
    if config.get("video", "force_mjpeg") == "true":
        # Set a magic number, will enable MJPEG but is badly documentated
        video_capture.set(cv2.CAP_PROP_FOURCC, 1196444237)

    # Set the frame width and height if requested
    if int(config.get("video", "frame_width")) != -1:
        video_capture.set(cv2.CAP_PROP_FRAME_WIDTH, int(config.get("video", "frame_width")))

    if int(config.get("video", "frame_height")) != -1:
        video_capture.set(cv2.CAP_PROP_FRAME_HEIGHT, int(config.get("video", "frame_height")))

    # Capture a single frame so the camera becomes active
    # This will let the camera adjust its light levels while we're importing for faster scanning
    video_capture.read()

else:
    # Request a frame to wake the camera up
    video_capture = (
        ffmpeg
        .input(config.get("video", "device_name"), format=config.get("video", "device_format"))
        .output('pipe:', vframes=1, format='image2', vcodec='mjpeg')
        .run(capture_stdout=True)
    )
    probe = ffmpeg.probe(config.get("video", "device_name"))

    if int(config.get("video", "frame_width")) != -1:
        frame_width = int(config.get("video", "frame_width"))
    else:
        frame_width = probe['streams'][0]['width']

    if int(config.get("video", "frame_height")) != -1:
        frame_height = int(config.get("video", "frame_height"))
    else:
        frame_height = probe['streams'][0]['height']

...
...

# Start the read loop
frames = 0
brightness_boost = 0

while True:
    # Increment the frame count every loop
    frames += 1

    # Stop if we've exceded the time limit
    if time.time() - timings[3] > int(config.get("video", "timout")):
        stop(11)

    # Grab a single frame of video
    # Don't remove ret, it doesn't work without it
    if config.get("video", "recording_plugin") == "opencv":
        ret, frame = (video_capture.read())
    else:
        stream, ret = (
            ffmpeg
            .input(config.get("video", "device_name"), format=config.get("video", "device_format"))
            .filter('select', 'gte(n,{})'.format(24))
            .output('pipe:', vframes=1, format='image2', vcodec='mjpeg')
            .run(capture_stdout=True)
        )
        nframe = numpy.frombuffer(stream, numpy.uint8)
        cframe = cv2.imdecode(nframe, 0)
        frame = color.gray2rgb(cframe, alpha=None)
        if brightness_boost > 0:
            frame = increase_brightness(frame, value=brightness_boost)
            print("brightness_boost: " + str(brightness_boost))

...
...
    # Scrip the frame if it exceeds the threshold
    if float(hist[0]) / hist_total * 100 > float(config.get("video", "dark_threshold")):
        dark_tries += 1
        brightness_boost += 10
        continue

And finally, in the [video] section of the config ...

[video]
#
# opencv or ffmpeg.  Choose ffmpeg if your IR device is a grayscale only.
# Run this: v4l2-ctl --list-devices --all
#
recording_plugin = ffmpeg

#
# format to use for ffmpeg. Could be vfwcap or v4l2 or some other thing...
#
device_format = v4l2

#
# device name if using ffmpeg
#
device_name = /dev/video2

timwelch on 1 Dec 2018

Final note, I'm not quite sure yet how to get test.py working with ffmpeg. Seems to be a much larger code modification for that piece.

timwelch on 1 Dec 2018

The FFmpeg option isn't hard to support for face addition or authentication, but the test command heavily depends on OpenCV to show the output. You can add the config value to use FFmpeg in those 2 commands. Be sure to reuse as much code as possible between OpenCV and FFmpeg though.

For now it is sufficient to just disable the test command when FFmpeg is used. This limitation will be gone once we have an actual UI.

As for the speed improvements, take a look at #99. Multiple people are working really hard over there to cut startup times and improve recognition.

boltgolt on 2 Dec 2018

I found my issue with speed of detection. It was actually this line:

            .filter('select', 'gte(n,{})'.format(24))

This tells ffmpeg which frame to save. Since the IR emitters turn on and off, frame 24 for me is nearly dark. I just chose it at random when I was testing. frame 30 is also fairly dark as well. The lower the frame number you can use, the quicker detection becomes. Frame 30 on a camera that is 30fps is at the tail end of 1 second. (I should have realized that sooner. lol) The IR lights flashing off is why I had to add the increase brightness function. When I changed the capture number to 3, it began working perfectly and very quickly every time. (i.e frame 3 is 1/10 of a second)

So I added one more option to the config, to set the frame number to capture.

add.py and compare.py

            .filter('select', 'gte(n,{})'.format(int(config.get("video", "image_number"))))

config

#
# When saving an image from the camera, which image number to use. 0 will almost always be black
# and when using an IR camera, it depends on how the lights are flashing as to which image number
# is black or darker. Experiment if the add function doesn't pick your face out quickly. 
#
image_number = 3

I'm wondering if during initialization, Howdy could just capture a group of 20 frames, and then determine which frame is the brightest? Ask if the user knows which frame they want to use, if not, do that test.

I'll run through my code and make sure it's lean and clean before submitting a pull request. I still want to run it for a few more days to make sure it's stable for me.

timwelch on 2 Dec 2018

The OpenCV version circumvents this by detecting if a frame seems really dark and skipping it if it is, you could probably do the same for FFmpeg. I think a config value for this is very unfriendly to users, who not only need to understand the concept of dark frames but also randomly guess the right one for their camera.

boltgolt on 2 Dec 2018

Agreed, the video frames are coming in in such a way that they are identical to the openCV (numpy array) so they can be treated identically.

There's no way (to my knowledge) to read a single frame at a time through ffmpeg, so what I did was created a class for the functions needed. And on first read(), it runs an initialization which simply probes the device. On subsequent reads, it pulls in 10 frames into a buffer, that is then read as an array on each iteration of the while loop. If the while loop needs to continue past 10 frames, the read() function will then record() another set of 10 frames and continue on indefinitely. By only recording 10 frames, we limit the camera use time to roughly 1/3 of a second.

ffmpeg_reader class file

import ffmpeg
import numpy
from cv2 import CAP_PROP_FRAME_WIDTH
from cv2 import CAP_PROP_FRAME_HEIGHT

#
# This class was created to look as similar to the openCV features used
# in Howdy as possible for overall code cleanliness. 
#
class ffmpeg_reader:

    #
    # Initialization
    #
    def __init__(self, device_name, device_format, numframes):
        self.device_name = device_name
        self.device_format = device_format
        self.numframes = numframes
        self.video = ()
        self.num_frames_read = 0
        self.height = 0
        self.width = 0
        self.init_camera = True

    #
    # Setter method for height and width
    #
    def set(self, prop, setting):
        if prop == CAP_PROP_FRAME_WIDTH:
            self.width = setting
        elif prop == CAP_PROP_FRAME_HEIGHT:
            self.height = setting


    #
    # Getter method for height and width
    #
    def get(self, prop):
        if prop == CAP_PROP_FRAME_WIDTH:
            return self.width
        elif prop == CAP_PROP_FRAME_HEIGHT:
            return self.height

    #
    # Probe the video device to get and set height and width info
    #
    def probe(self):
        if self.get(CAP_PROP_FRAME_WIDTH) == 0 or self.get(CAP_PROP_FRAME_HEIGHT) == 0:
            probe = ffmpeg.probe(self.device_name)

        if self.get(CAP_PROP_FRAME_WIDTH) == 0:
            self.set(CAP_PROP_FRAME_WIDTH, probe['streams'][0]['width'])
        if self.get(CAP_PROP_FRAME_HEIGHT) == 0:
            self.set(CAP_PROP_FRAME_HEIGHT, probe['streams'][0]['height'])

    #
    # Record a video, saving it to self.video for processing later
    #
    def record(self, numframes):
        #
        # In case we forget to call probe, ensure we have set our width and height
        #
        self.probe()
        self.num_frames_read = 0

        stream, ret = (
            ffmpeg
            .input(self.device_name, format=self.device_format)
            .output('pipe:', format='rawvideo', pix_fmt='rgb24', vframes=numframes)
            .run(capture_stdout=True, quiet=True)
        )
        self.video = (
            numpy
            .frombuffer(stream, numpy.uint8)
            .reshape([-1, self.width, self.height, 3])
        )

    #
    # Read a frame from the video
    #
    def read(self):
        #
        # First time we are called, we want to initialize the camera by probing it and ensure self.video is empty.
        #
        if self.init_camera:
            self.init_camera = False
            self.probe
            self.video = ()
            return 0, self.video

        #
        # If we are called and self.video is empty, we should record self.numframes to fill the video buffer
        #
        if self.video == ():
            self.record(self.numframes)

        #
        # If we've read max frames, but still are being requested to read more, we simply record another batch.
        # Note, the video array is 0 based, so if numframes is 10, we must subtract 1 or run into an array index
        # error.
        #
        if self.num_frames_read >= (self.numframes - 1):
            self.record(self.numframes)

        #
        # Add one to the count. If we were at 0, that's fine as frame 0 is almost 100% going to be black
        # as the IR lights aren't fully active yet anyways. Saves us some trouble in the long run.
        #
        self.num_frames_read += 1
        return 0, self.video[self.num_frames_read]

and then the code to instantiate the new video_capture variable inside of add.py and compare.py is the only piece of the code that actually needs an IF statement. The rest of the original code can stay the same.

I placed the ffmpeg_reader.py in the same folder as the compare.py.

from ffmpeg_reader import ffmpeg_reader
...
...
...
if config.get("video", "recording_plugin") == "opencv":

    # Start video capture on the IR camera
    video_capture = cv2.VideoCapture(config.get("video", "device_path"))

    # Force MJPEG decoding if true
    if config.get("video", "force_mjpeg") == "true":
        # Set a magic number, will enable MJPEG but is badly documentated
        video_capture.set(cv2.CAP_PROP_FOURCC, 1196444237)

else:
    # Set the capture source for ffmpeg
    video_capture = ffmpeg_reader(config.get("video", "device_name"), config.get("video", "device_format"), 10)


# Set the frame width and height if requested
if int(config.get("video", "frame_width")) != -1:
    video_capture.set(cv2.CAP_PROP_FRAME_WIDTH, int(config.get("video", "frame_width")))

if int(config.get("video", "frame_height")) != -1:
    video_capture.set(cv2.CAP_PROP_FRAME_HEIGHT, int(config.get("video", "frame_height")))

# Capture a single frame so the camera becomes active
# This will let the camera adjust its light levels while we're importing for faster scanning
# Note: For FFMPEG this just runs a probe to ensure height and width are set properly. 
video_capture.read()

...
...
    # Grab a single frame of video
    # Don't remove ret, it doesn't work without it
    ret, frame = (video_capture.read())

timwelch on 4 Dec 2018

Looks good, but cv2.CAP_PROP_FRAME_WIDTH will not exist without a OpenCV import. Besides that, i think OpenCV should be the default and FFmpeg explicitly set. (reverse the if else with config.get("video", "recording_plugin") == "ffmpeg")

Could you also take a look at the documentation style in your class? Take a look at the style of the other files in the repo and maybe look into docstrings.

boltgolt on 4 Dec 2018

No problem! I reversed the if/else as requested, and I added docstrings into ffmpeg_reader class.

Help on module ffmpeg_reader:

NAME
    ffmpeg_reader

CLASSES
    builtins.object
        ffmpeg_reader

    class ffmpeg_reader(builtins.object)
     |  This class was created to look as similar to the openCV features used in Howdy as possible for overall code cleanliness.
     |  
     |  Methods defined here:
     |  
     |  __init__(self, device_name, device_format, numframes)
     |      Initialize self.  See help(type(self)) for accurate signature.
     |  
     |  get(self, prop)
     |      Getter method for height and width
     |  
     |  probe(self)
     |      Probe the video device to get height and width info
     |  
     |  read(self)
     |      Read a sigle frame from the self.video array. Will record a video if array is empty.
     |  
     |  record(self, numframes)
     |      Record a video, saving it to self.video array for processing later
     |  
     |  set(self, prop, setting)
     |      Setter method for height and width
     |  
     |  ----------------------------------------------------------------------

timwelch on 5 Dec 2018

Changes are committed in my forked repo here.

timwelch/howdy

timwelch on 5 Dec 2018

👍1

Great! You can open a pull request against the dev branch on this repo.

boltgolt on 5 Dec 2018

👍1

So, I'm not 100% happy with the code. Specifically, the PROBE. It takes 800ms to probe. We must probe to grab the resolution (in my case 352x352) of the IR camera before running the numpy resize to make the video compatible with the facial recog code.

Would it be easy to do a probe on initial configure and set height/width in the config file? My code will ignore the probe if those values are set.

Here's a run with probe...

Time spent
  Starting up: 118ms
  Opening the camera: 1836ms
  Importing face_recognition: 958ms
  Searching for known face: 94ms

Resolution
  Native: 352x352
  Used: 320x320

Frames searched: 4 (42.72 fps)
Dark frames ignored: 3 
Certainty of winning frame: 3.369
Winning model: 0 ("Initial model")
Identified face as tim

And here's a run without probe (hard coded 352x352). Note the time on Opening the camera, that's where probe is called.

Time spent
  Starting up: 115ms
  Opening the camera: 1071ms
  Importing face_recognition: 954ms
  Searching for known face: 92ms

Resolution
  Native: 352x352
  Used: 320x320

Frames searched: 1 (10.91 fps)
Dark frames ignored: 0 
Certainty of winning frame: 3.460
Winning model: 0 ("Initial model")
Identified face as tim

timwelch on 11 Dec 2018

Ok, I figured it out... The call to ffmpeg returns the output of the actual command line command in the value of ret as a byte object.

This looks like ...

ffmpeg version 3.4.4-0ubuntu0.18.04.1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
  configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Input #0, video4linux2,v4l2, from '/dev/video2':
  Duration: N/A, start: 62106.710514, bitrate: 29736 kb/s
    Stream #0:0: Video: rawvideo (Y800 / 0x30303859), gray, 352x352, 29736 kb/s, 30 fps, 30 tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> rawvideo (native))
Press [q] to stop, [?] for help
Output #0, rawvideo, to 'pipe:':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 352x352, q=2-31, 89210 kb/s, 30 fps, 30 tbn, 30 tbc
    Metadata:
      encoder         : Lavc57.107.100 rawvideo
frame=   10 fps=0.0 q=-0.0 Lsize=    3630kB time=00:00:00.33 bitrate=89211.0kbits/s dup=5 drop=0 speed=2.45x    
video:3630kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%

And in testing, we can easily pull out the values for height and width

>>> tim = str(ret.decode("utf-8"))
>>> for line in tim.splitlines():
...    if ", rgb24, " in line:
...       (height, width) = [x.strip() for x in [y.strip() for y in line.split(',')][2].split('x')]
...       print(height)
...       print(width)
... 
352
352

So we can parse from there. I used the output to pipe line that included ", rgb24, " in the line. It only occurs once, and we know it will be there, since the call to record forces the output to rgb24.

So all we have to do is move the probe after the record call and the parse, and before the numpy in case the parse fails. This will ensure the probe runs as a backup, but since it's expensive (800ms) we only run it if we absolutely need to.

    def record(self, numframes):
        """ Record a video, saving it to self.video array for processing later """


        # Ensure num_frames_read is reset to 0
        self.num_frames_read = 0

        stream, ret = (
            ffmpeg
            .input(self.device_name, format=self.device_format)
            .output('pipe:', format='rawvideo', pix_fmt='rgb24', vframes=numframes)
            .run(capture_stdout=True, quiet=True)
        )

        # We can detect the video resolution from the ret output of the video. But it takes a little parsing.
        # We are looking for this line:  
        #       Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 352x352, q=2-31, 89210 kb/s, 30 fps, 30 tbn, 30 tbc
        parse_resolution = str(ret.decode("utf-8"))
        for line in parse_resolution.splitlines():
            if ", rgb24, " in line:
                (height, width) = [x.strip() for x in [y.strip() for y in line.split(',')][2].split('x')]
                if height.isdigit() and width.isdigit():
                    self.set(CAP_PROP_FRAME_HEIGHT, int(height))
                    self.set(CAP_PROP_FRAME_WIDTH, int(width))

        # Eensure we have set our width and height before we attempt to call numpy so the call to numpy does not fail
        if self.get(CAP_PROP_FRAME_WIDTH) == 0 or self.get(CAP_PROP_FRAME_HEIGHT) == 0:
            self.probe()

        self.video = (
            numpy
            .frombuffer(stream, numpy.uint8)
            .reshape([-1, self.width, self.height, 3])
        )

This decreases the "Opening the camera" step by the ~800ms ...

Time spent
  Starting up: 114ms
  Opening the camera: 1073ms
  Importing face_recognition: 951ms
  Searching for known face: 97ms

Resolution
  Native: 352x352
  Used: 320x320

Frames searched: 1 (10.36 fps)
Dark frames ignored: 0 
Certainty of winning frame: 2.765
Winning model: 0 ("Initial model")
Identified face as tim

timwelch on 11 Dec 2018

I'm a bit worried about fetching the resolution directly from the FFmpeg output, as this is not meant as input and could change in later versions. I'd propose using v4l2-ctrl if possible or another machine usable tool.

boltgolt on 11 Dec 2018

That does make sense. I agree with you. Looking at v4l2-ctrl, it would only work on cams that support it, correct? In my case, that works fine, but there could be a case where a webcam uses vfwcap driver instead. I don't know if the v4l2-ctrl command would still work in that scenario?

Should be easy to get a python function written to pull in the info, it will need to be parsed.

v4l2-ctl -d /dev/video2 --list-formats-ext | grep Size| awk '{print $3}'

352x352

timwelch on 11 Dec 2018

Here's what I have so far in ffmpeg_reader class for probe:

    def probe(self):
        """ Probe the video device to get height and width info """

        args = ["v4l2-ctl", "-d", self.device_name, "--list-formats-ext"]
        process = Popen(args, stdout=PIPE, stderr=PIPE)
        out, err = process.communicate()

        if err:
            print("Error opening subprocess for v4l2-ctl.\n\n%s" % err)
            sys.exit(1)

        regex = re.compile('\s\d+x\d+')

        probe = regex.findall(str(out.decode("utf-8")))
        if len(probe) < 1: 
            # Could not determine the resolution from v4l2-ctl call. Reverting to ffmpeg.probe()
            probe = ffmpeg.probe(self.device_name)
            height = probe['streams'][0]['height']
            width = probe['streams'][0]['width']
        else:
            (height, width) = [x.strip() for x in probe[0].split('x')]


        if height.isdigit() and self.get(CAP_PROP_FRAME_HEIGHT) == 0:
            self.set(CAP_PROP_FRAME_HEIGHT, int(height))

        if width.isdigit() and self.get(CAP_PROP_FRAME_WIDTH) == 0:
            self.set(CAP_PROP_FRAME_WIDTH, int(width))

This will attempt to use v4l2-ctl, and revert back to using ffmpeg.probe() if the command fails.

On my laptop, this brings the probe function down to between 1100-1200ms, from 1800ms.

  Opening the camera: 1145ms

Note, there's also this command in ffmpeg itself that I found would return the resolution as well... but that command takes an additional 50-60ms. So, it would be less dependencies at the cost of 50-60ms.

 ffmpeg -f v4l2 -list_formats all -i /dev/video2

timwelch on 12 Dec 2018

I switched to using ffmpeg, so as not to increase dependencies. It does add a little more time, since the ffmpeg process itself is beefy, but 100-150ms is still much better than 800ms total for the ffmpeg.probe() to run.

  Opening the camera: 1164ms

timwelch on 12 Dec 2018

no need for ffmpeg for the spectre ?

Just adding
if len(frame.shape) == 2: frame = cv2.cvtColor(frame,cv2.COLOR_GRAY2BGR)
after each ret, frame = video_capture.read() worked here ( from https://github.com/boltgolt/howdy/issues/70#issuecomment-439123621 )

Maybe I missed something ?

gled-rs on 3 Jan 2019

What model are you using?

On my model (13t-something), frame.shape was empty. OpenCV is not currently capable of handling the IR camera. If I use the normal camera, it works fine. But that's not desirable.

I have since found a better way than using FFMPEG, it has been merged in. See the link below.

https://github.com/boltgolt/howdy/pull/116

timwelch on 4 Jan 2019

I have an ap0xxxx ( 2018 Gem cut ) 13" i7.

Hardware is UVC 1.50 device HP Wide Vision FHD Camera (0408:5251).

Creates 4 /dev/video. the one to use for the IR is /dev/video2.

gled-rs on 5 Jan 2019

Release 2.5.0 includes both alternative recorders and grayscale support, so this issue should be fixed.

boltgolt on 6 Jan 2019

🎉1

@boltgolt I just patched the AUR build config and compiled 2.5.0 for Manjaro on my 2018 13" Spectre x360 and can confirm it works like a charm!

I was struggling with the current (outdated, and flagged as such) AUR build of 2.3.1; sudo howdy test would launch the IR camera and my face would be circled in black but no recognition stats would be shown (which it would if I configured it to use the colour camera) but compiling this new release and doing the permissions patch on the models directory along with adding auth sufficient pam_python.so /lib/security/howdy/pam.py into the top line of /etc/pam.d/sudo and /etc/pam.d/gdm-password has got everything working exactly as desired.

Huge thanks to you and the other contributors for this excellent work! 👏👏👏

deepybee on 7 Jan 2019

Is there anything that needs to be edited to get it to work on an x360? I have a 13t-ae000 and just following the instructions in ReadMe, I still get the original error plus more?

VIDEOIO ERROR: V4L2: Pixel format of incoming image is unsupported by OpenCV

Please look straight into the camera
Traceback (most recent call last):
  File "/usr/local/bin/howdy", line 89, in <module>
    import cli.add
  File "/lib/security/howdy/cli/add.py", line 147, in <module>
    gsframe = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
cv2.error: OpenCV(3.4.4) /io/opencv/modules/imgproc/src/color.cpp:181: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

mjlabe on 7 Jan 2019

@mjlabe which camera is selected? What happens when you run sudo howdy test? And which distro are you using? Are you running a distribution packaged version or did you build it yourself?

It might be you need to run sudo howdy config and point howdy at a different numbered device in /dev/videoX; on Arch the camera that's auto detected is /dev/video0 - this needs to be changed to /dev/video2 to use the IR camera.

deepybee on 7 Jan 2019

I'm not sure which camera, how can I tell?
sudo howdy test results in:

```VIDEOIO ERROR: V4L2: Pixel format of incoming image is unsupported by OpenCV