Tfjs: Feature Request: PoseNet Video API

[Jurge92]

Hi,

I really enjoy using PoseNet here:

https://github.com/tensorflow/tfjs-models/tree/master/posenet

However, one issue I'm having is that I cannot load in a local/remote video file. I can only use a real-time video stream. I know there's some hacky workaround to make it somewhat work, but those aren't optimal and reduces the simpleness of the code.

Would it be possible to add a simple API in a similar way as loading an image for a video file?

It would make my life (and probably a few others) much easier!

Cheers

[rthadur]

@tafsiri please let us know if you have any thoughts on this feature request ?

[tafsiri]

@Jurge92 you should be able to pass in an HTMLVideoElement instead of an image. If you have any trouble with that let us know more about what kind of video you want to load and where using a video element falls short.

[Jurge92]

@Jurge92 you should be able to pass in an HTMLVideoElement instead of an image. If you have any trouble with that let us know more about what kind of video you want to load and where using a video element falls short.

I have been trying to load a local .mp4 file instead of a .jpg here:

https://github.com/tensorflow/tfjs-models/blob/d836d5f89ec5fb49a3213bc1e14c847b42a61bb5/posenet/demos/coco.js#L33

When I do, nothing happens. That is basically what I want, just use any local video (be it .mp4 or whatnot) and then it can be processed accordingly, showing the video running with detection on each frame.

[tafsiri]

@Jurge92 can you get your video playing in a

[Jurge92]

I'm currently trying to load the video file into the canvas the same way as an image. And that doesn't give any result.

Could you make an example of a simple approach for how to load a local video file and displaying it (after being processed)?

[Deunkel]

Being able to process local/remote video files (like .mp4) would be great for "non browser environments" like on node.js. I tried adding canvas libs into node.js, but it felt hacky and had also a bad performance regarding processing time. So I agree with @Jurge92 this feature would be a great help.

[carlottaCC]

@Jurge92 and @Deunkel , have you found out how to process local video files into the canvas? ( Or any other way to use prerecorded videos rather than the webcam?). Would love to hear from you!

[tafsiri]

Take a look at https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video for an example of how to load video into an HTMLVideoElement.

You should be able to pass a JavaScript reference to that DOM element to the posenet estimate* methods

That JavaScript reference should also allow you to control advancing the video etc.

[ThaoDoann]

I have the same issue using Tensorflow.js to detect pose on a local mp4 video. Apparently it works perfectly when I use its original size, however, if I resize the video, the output skeleton doesn't match the pose on the video the predictions seem to be wrong.

[tafsiri]

@ThaoDoann you might just be resizing the 'preview' of the video on your browser rather than the video itself. Feel free to file a new issue with sample code/way to reproduce. I'm going to close this issue since the original question of using it with video tags seems solved.