Ml5-library: Add support for PoseNet `.singlePose()` & `.multiPose()` callback

[joeyklee]

Hello!

Related to https://github.com/ml5js/ml5-examples/pull/79 in using callbacks to structure the js.

In trying to run the PoseNet .singlePose() function using a callback, it doesn't seem to run the callback function. e.g.:

poseNet.singlePose(img, function(err, res) {
        if(err) return err
        // when we get a response, store the results to our poses[] array
        // then draw will do it's thing
        console.log(res)
        poses = res;
    });

However, when using promises, we get the expected results:

poseNet.singlePose(img).then( (res) => {
        // when we get a response, store the results to our poses[] array
        // then draw will do it's thing
        poses = res;
    }).catch( (err) => {
        return err;
    });

Could this be an issue with ML5 or with TFJS core?

Below are the images of the differing results:

Hope this helps. Thanks!

[shiffman]

I used the new GitHub beta feature to move the issue here, I think this is something to fix / look at in the library then we can update the example! @oveddan, can you take a look?

[cvalenzuela]

PoseNet works as an EventEmitter. So you will need to listen to the pose event.

This should work:

// Create the model without a video element
poseNet = ml5.poseNet(modelReady);

// This sets up an event that listens to 'pose' events
poseNet.on('pose', function (results) {
   poses = results;
});

poseNet.singlePose(img)

[joeyklee]

@cvalenzuela Ah! Interesting. Should I update https://github.com/ml5js/ml5-examples/pull/79#issuecomment-442969484 with the suggestion above + submit a new PR? Thanks a bunch for the feedback!

[cvalenzuela]

That would be great!

On Thu, Nov 29, 2018 at 15:49 Joey Lee [e-mail] wrote:

@cvalenzuela
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_cvalenzuela&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=iz4Iqm-iIde9PSPTj-dshNGWoOwH8s4BYl-WBzNnd0o&m=1JZwpSHE0ELdnCKxOpQDtv6LT0HBHWaVTeUF36i7Y1U&s=609XA1UctQ8slKEMg_-hNth0wOXhqJHx6SKL6ZenOLE&e=
Ah! Interesting. Should I update ml5js/ml5-examples#79 (comment)
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ml5js_ml5-2Dexamples_pull_79-23issuecomment-2D442969484&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=iz4Iqm-iIde9PSPTj-dshNGWoOwH8s4BYl-WBzNnd0o&m=1JZwpSHE0ELdnCKxOpQDtv6LT0HBHWaVTeUF36i7Y1U&s=UPE7kG_YbifkEtvmRu_JrKXaH4E0_TAsOFMZQLV4VCM&e=
with the suggestion above + submit a new PR? Thanks a bunch for the
feedback!

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ml5js_ml5-2Dlibrary_issues_244-23issuecomment-2D442987830&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=iz4Iqm-iIde9PSPTj-dshNGWoOwH8s4BYl-WBzNnd0o&m=1JZwpSHE0ELdnCKxOpQDtv6LT0HBHWaVTeUF36i7Y1U&s=Cfm6mw68I5-QwD6T-t3SxKRUaBhCe5ophGpg1aYAlOw&e=,
or mute the thread
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AKHU-5FdknelkQ1gu3n1CVDLSAq83eYgP0ks5u0EhJgaJpZM4Y6ZQj&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=iz4Iqm-iIde9PSPTj-dshNGWoOwH8s4BYl-WBzNnd0o&m=1JZwpSHE0ELdnCKxOpQDtv6LT0HBHWaVTeUF36i7Y1U&s=tiJmgG8kCxnBfvAOW_WCdC2s6d8HVnn4FnMwHJev1_0&e=
.

[joeyklee]

@cvalenzuela cool! Will do. I'll make sure any of the next submissions also follow the same style. Many thanks ❤️!

[shiffman]

This can be closed, however, I wonder if supporting callbacks for these "single use" functions would be helpful in the cases where it's not continuously being run on a webcam?

[joeyklee]

@shiffman If I understand how the poseNet event listener works is that regardless if you're streaming images or not, poseNet is just waiting for some kind of detection to come in. My first instinct would be to use callbacks or a promise (as in the first PR) so if it is not crazy effort to add this, then maybe it's good for consistency with the other image detection methods? Otherwise, using the poseNet event listener is also fine I think if we can point to these examples.

[shiffman]

Yes, I was thinking that for an example where you are maybe loading an array of images it might be tricky to pair the pose data with the correct image if the event listener function is separate?

[cvalenzuela]

Having a callback as an options should be straightforward to implement and makes sense. I guess the initial idea of using poseNet in ml5 was more focused on video/interactive but we should support single images.

[joeyklee]

@cvalenzuela @shiffman - I'd be happy to look into this (unless someone else wants to jump in first). Maybe in the next couple weeks I'll try to cross this off this list :) It'll be a nice opportunity to look into the ml5 lib! Thanks!

[cvalenzuela]

@joeyklee go for it! happy to help if necessary.

[joeyklee]

@cvalenzuela Sure thing! Will keep ya posted. Thanks!

[joeyklee]

Hello there! I just made a PR regarding this thread here https://github.com/ml5js/ml5-library/pull/254

TODO: I just want to make a note to update the documentation on the website if/when the changes are integrated so we don't forget :) - Thanks + Happy New Year + Feliz año nuevo!

[joeyklee]

Thanks all! Closing this now! 🙏