Ml5-library: Simple PoseNet

[shiffman]

I'm working on a new PoseNet example showing how to use just a few keypoints. Here's one with just the nose:

https://editor.p5js.org/codingtrain/sketches/By8YLbXyE

This leads me to some thinking about simplifying the API in the very common case of just wanting one pose and one or two keypoints (like the hands). What do you think about:

let video;
let poseNet;
let nose;

function setup() {
  createCanvas(640, 480);
  video = createCapture(VIDEO);
  let poseNet = ml5.poseNet(video);
  poseNet.on('singlePose', gotPose);
}

function gotPose(pose) {
  nose = pose.getPosition('nose');
}

function draw() {
  image(video, 0, 0, width, height);
  if (nose) {
    stroke(255, 0, 0);
    strokeWeight(16);
    point(nose.x, nose.y);
  }
}

Alternative:

function gotPose(pose) {
  // could also include the confidence?
  nose = pose.get(ml5.NOSE);
}

Also, is it too p5 specific to return a p5.Vector from getPosition()?

Related to: https://github.com/ml5js/ml5-website/issues/94

[cvalenzuela]

Makes sense!

Should we keep the position names consistent with poseNet API (ie: 'right_shoulder')?

[shiffman]

Should we keep the position names consistent with poseNet API (ie: 'right_shoulder')?

Yes, I think so! I am going to do a tutorial with this on Thursday (https://youtu.be/EA3-k9mnLHs) is it worth it / realistic for me to trying to add something to support this by then?

[cvalenzuela]

Perhaps as an experimental feature? Not sure if will have it as a npm version. Are you thinking of a parser that will take pose results (pose.getPosition('nose')) and just return x and y coordiantes?

[shiffman]

Yes, exactly. For what I'm showing having what is returned be ap5.Vector object would be super useful, but maybe that is too p5 specific? It's probably a good idea to have it also return the confidence. So it could be:

let nose = pose.getPosition('nose');
console.log(nose.x, nose.y, nose.confidence);

I can do the tutorial with the way it is now and we can revisit this later in a more major update.

[mayaman]

Continuing this conversation, I'm going to work on implementing an easier way to access key points using the part names given by posenet. There are a few decisions to make in terms of syntax and after discussing today with @shiffman, there are initial thoughts we have as to which way to go, but I would love to hear other people's thoughts!

Convention to access specific part data

Leaning toward: pose['nose'].x && pose['nose'].confidence
Here are the options currently in consideration (open to other suggestions!):

1. pose.getPosition(‘nose’) // As suggested above
   Issue here is that it wouldn't make sense to include the confidence score which people may want to access.
2. pose['nose'].position.x && pose.nose.position.x && pose.nose.confidence
   This keeps with the current data structure returned by posenet, where each keypoint contains a position and score (confidence) key, and the x and y values are nested under the position key.
3. pose['nose'].x && pose.nose.x && pose.nose.confidence
   This is the one we think is the best option, given it flattens the x and y coordinates and the score (confidence) value onto the same plane.

"rightShoulder" vs "right_shoulder"

Leaning toward: "rightShoulder"
In the pose returned by posenet, each part is specified using the "rightShoulder" convention, so I think for consistency we should just keep with that. That way we can also create this new data structure by mapping the existing part labelings to their respective values.

"confidence" vs "score"

Leaning toward: "confidence"
Posenet labels the confidence value of the overall pose and specific keypoints using the keyword "score". I think confidence may be more intuitive to beginners, but we also want to make sure we are consistent across ml5, so I'm going to look into what is currently used now as well and follow up. But would be interested to hear thoughts on which word people prefer!

Also, still thinking about potential for p5.Vector to be incorporated somehow! But not sure if it would be appropriate because like Dan mentioned above it may be "too p5 specific"...

[shiffman]

Thanks @mayaman, this sounds great! Yay! 💯 After we get this new API feature implemented, we can also revisit whether / how we want to default to a single pose (or multiple poses) and how we might better align the API with other models that operate continuously on video (see the discussion about face segmentation for example: #274). I like how PoseNet uses an event listener but we don't do this for any other ml5 classes. . . maybe we should?

[sguzmanolmos]

Hi! I'm super new to this and I have one question, how do I set this up to work on multiple people at the same time? Because it's only working on one pose. Thanks :)

PS: Wonderful tutorial!

[joeyklee]

Closing this as @mayaman - made the wonderful update to posenet back a Feb! Thanks! 🙏