Openpose: Any plan to add a common body ID between video frames?

[kidapu]

Type of issue

• Enhancement / offering possible extensions

Issue summary

Any plan to add a common body ID between video frames like below?

frame 0

{
    "version":0.1,
    "people":[
        {"body0":[1114.15,160.396,0.846207,...]},
        {"body1":[...]},
    ]
}

frame 1

{
    "version":0.1,
    "people":[
        {"body0":[1116.15,162.396,0.856207,...]},
        {"body1":[...]},
    ]
}

Sometimes I want to extract only a specific body.

[tsimk]

There are no current plans for this, so I suggest you do this as a post-processing step. For example, association based on the distance between keypoints in successive frames.

Handling more complex cases such as re-identification or cross-overs is unlikely to happen soon, as the approaches we have looked at are currently far from realtime and very experimental.

[kidapu]

@tsimk
Thank you for your reply m(_ _)m
I will try to calc distance in successive frame on myself.

[wbadart]

Just wanted to share this snippet since my team has recently had to introduce this post-processing step for our project. There are a number of ways to do this; I originally kept the traces in a big numpy array, but found the dictionary to be more amenable to changing numbers of skeletons (even though the implementation is a little less concise):

from itertools import chain, count
from sklearn.neighbors import NearestNeighbors

def push(traces, new_skels):
    """Add the keypoints from a new frame into the buffer."""
    dists, neighbors = nearest_neighbors(traces, new_skels)

    # New skeletons which aren't close to a previously observed skeleton:
    unslotted = []
    # Previously observed skeletons which aren't close to a new one:
    unseen = set(self.data.keys())

    for skel, dist, neighbor in zip(new_skels, dists, neighbors):
        if dist <= THRESHOLD:
            traces[neighbor].append(skel)
            if len(traces[neighbor]) > TRACE_SIZE:
                traces[neighbor].pop(0)
            unseen.discard(neighbor)
        else:
            unslotted.append(skel)

    for i in unseen:
        del traces[i]

    # Indices we didn't match, and the rest of the numbers are fair game
    availible_slots = chain(sorted(unseen), count(len(traces)))
    for slot, skel in zip(availible_slots, unslotted):
        traces[slot].append(skel)

def nearest_neighbors(traces, skels):
    if not traces:  # First pass
        return np.zeros(len(skels)), np.arange(len(skels))
    prev = np.array([  # Pull the most recent location of each skeleton
        coords[-1][TRACE_IDX] for _, coords in sorted(traces.items())])
    curr = skels[:, TRACE_IDX]
    # N is typically small (< 40) so brute force is fast
    nn_model = NearestNeighbors(n_neighbors=1, algorithm='brute')
    nn_model.fit(prev)
    dist, nn = nn_model.kneighbors(curr, return_distance=True)
    return dist.flatten(), nn.flatten()

# Usage:
from collections import defaultdict
my_traces = defaultdict(list)
for frame in my_frame_source():  # e.g. using cv2.VideoCapture
    keypoints, _ = openpose.forward(frame)
    keypoints = keypoints[:, :, :-1]  # Drop confidence scores
    push(my_traces, keypoints)

Here, traces is a dictionary of lists of numpy arrays, where each of those arrays is on of the skeletons returned by openpose.forward for a particular frame. For example, trace[1] is the list of full skeletons for body1 in chronological order. TRACE_IDX is simply the index of the joint you want to track. My team uses 1, which is the chest/ neck point on the COCO model. THRESHOLD is the maximum distance you could consider to skeletons to be contiguous, and TRACE_SIZE is the maximum number of previous positions you'd like to store.

This is almost certainly NOT perfect, but it's been working for us so far with a small number of people in the frame and seems to be robust to people leaving the frame (haven't fully tested on new people entering the frame, but it should work in theory).

[taraprasad73]

@wbadart
Hi, could you point me to some article or research paper, which explains the algorithm you used. Is it different from the popular tracking algorithms implemented in OpenCV?

[wbadart]

Hey taraprasad73, unfortunately I can't since I didn't base this implementation on any existing research. It's just a janky, home-grown tracing algorithm I rolled for my team's own purposes.

If it helps, the underlying intuition here is pair each skeleton in the current frame with the closest one from the previous frame, unless its nearest neighbor is far away, in which case, start a new trace.

[soulslicer]

Hi all, we will be providing some tracking support in the coming months, including a faster and temporally consistent version of Openpose. We are evaluating against the Posetrack metric as well to have good scores before we release it. Thank you

[HONGJUNRL]

@soulslicer
Hi, have you tried to train on posetrack dataset? I tried it but only get mAP about 59.3%. could you please give me some advise?

[soulslicer]

Hi we have published some work for keeping temporal consistency across people in real time:

https://cmu-perceptual-computing-lab.github.io/spatio-temporal-affinity-fields/

We will release this to Openpose by June

[HONGJUNRL]

Thanks for your reply，so could you please tell me the scale search and threshold you used while evaluating on posetrack dataset？

发自我的iPhone

------------------ Original ------------------
From: Raaj [e-mail]
Date: Fri,Apr 19,2019 11:13 PM
To: CMU-Perceptual-Computing-Lab/openpose [e-mail]
Cc: HONGJUNRL [e-mail], Comment [e-mail]
Subject: Re: [CMU-Perceptual-Computing-Lab/openpose] Any plan to add a common body ID between video frames? (#15)

Hi we have published some work for keeping temporal consistency across people in real time:

https://cmu-perceptual-computing-lab.github.io/spatio-temporal-affinity-fields/

We will release this to Openpose by June

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[PawelFaron]

Hi @wbadart ,
Thanks you for sharing the code, it looks great! I have few question to it if you don't mind:

1. "unseen = set(self.data.keys())" - sholud this be just "unseen = set(traces.keys())"?
2. Can you recommend some value for THRESHOLD?