Tf-pose-estimation: Speed up data preprocessing?

[Yuliang-Zou]

I found that the data loading bottleneck seems to be the get_heatmap and get_vectormap operations, while these operations are the same for each channel. Is it possible to use multithreading to do these operations in parallel to speed up?

[mrT23]

if transformed these functions into vectoric form instead of direct loops. got x10 acceleration in my training time

[Yuliang-Zou]

@mrT23 Can you share how you did this? Thanks!

[mrT23]

here is an example for my fast version of "put_heatmap".
the main idea - use python vectoric function, never (ever!) use direct
loops over matrices

def put_heatmap(heatmap, plane_idx, center, sigma):
center_x, center_y = center
_, height, width = heatmap.shape[:3]

    th = 4.6052
    delta = math.sqrt(th * 2)

    x0 = int(max(0, center_x - delta * sigma + 0.5))
    y0 = int(max(0, center_y - delta * sigma + 0.5))

    x1 = int(min(width - 1, center_x + delta * sigma + 0.5))
    y1 = int(min(height - 1, center_y + delta * sigma + 0.5))

    exp_factor = 1 / 2.0 / sigma / sigma

    ## fast - vectorize
    arr_heatmap = heatmap[plane_idx, y0:y1 + 1, x0:x1 + 1]
    y_vec = (np.arange(y0, y1 + 1) - center_y) ** 2  # y1 included
    x_vec = (np.arange(x0, x1 + 1) - center_x) ** 2
    xv, yv = np.meshgrid(x_vec, y_vec)
    arr_sum = exp_factor * (xv + yv)
    arr_exp = np.exp(-arr_sum)
    arr_exp[arr_sum > th] = 0
    heatmap[plane_idx, y0:y1 + 1, x0:x1 + 1] = np.maximum(arr_heatmap, arr_exp)

    ## slow - loops
    # for y in range(y0, y1 + 1):  # y0 to y1 include
    #     y_factor = (y - center_y) ** 2
    #     for x in range(x0, x1 + 1):
    #         # d = (x - center_x) ** 2 + (y - center_y) ** 2
    #         d = (x - center_x) ** 2 + y_factor
    #         # exp = d / 2.0 / sigma / sigma
    #         exp = d * exp_factor
    #         if exp > th:  # math.exp(-exp))
    #             continue
    #         val1 = math.exp(-exp)
    #         mat_val = heatmap[plane_idx, y, x]
    #         val2 = max(mat_val, val1)  # heatmap initilized to zero, cant be bigger then 1
    #         heatmap[plane_idx, y, x] = val2
    #         # heatmap[plane_idx][y][x] = max(heatmap[plane_idx][y][x], math.exp(-exp))
    #         # heatmap[plane_idx][y][x] = min(heatmap[plane_idx][y][x], 1.0)
    # arr_heatmap2 = heatmap[plane_idx, y0:y1 + 1, x0:x1 + 1]

[Yuliang-Zou]

@mrT23 Thanks!

[tomercohen11]

Does anybody have a similar implementation of "put_vectormap"?
Thanks

[rawandeheliah]

could I ask "put_heatmap" where does it exist in which file ?

[tomercohen11]

It is in tf_pose/post_dataset.py

[liangkezhao]

Could any one provide me an answer why th=4.6052? What's the meaning of that?

[hafiz031]

@liangkezhao for this th = 4.6052 the delta = math.sqrt(th * 2) is approximately 3 and it is a popular choice to take 3 * sigma i.e, 3 standard deviation from the mean point of a normal distribution. As it covers 99.7 % of the distribution. If it is the case, then, they could have taken delta = 3 directly. What's the point of taking it in this way is unknown to me as well 🤔😕