Tf-pose-estimation: the Train of MobileNet is too slow

[andyqian2015]

Hi @ildoonet sorry to intrupt you for another time. when I try to train the MobileNet from scratch.

[0109 11:50:12 @prefetch.py:170] [PrefetchData] Will fork a dataflow more than one times. This assumes the datapoints are i.i.d

when run this ,it take too much time, Have I made something wrong ?
I would appreciate if you can reply.

[elmuz]

To me that part works quite fast (Intel i7, Nvidia 1070). You can evince it from time-stamps.

[2018-01-12 10:34:05,236] [train] [INFO] define model+
[2018-01-12 10:34:05,239] [pose_dataset] [INFO] dataflow img_path=/home/alessio/Sandbox/COCO/
loading annotations into memory...
Done (t=7.11s)
creating index...
index created!
[2018-01-12 10:34:12,581] [pose_dataset] [INFO] /home/alessio/Sandbox/COCO/annotations dataset 118287
[0112 10:34:12 @prefetch.py:169] [PrefetchData] Will fork a dataflow more than one times. This assumes the datapoints are i.i.d.
[0112 10:34:13 @prefetch.py:169] [PrefetchData] Will fork a dataflow more than one times. This assumes the datapoints are i.i.d.
[2018-01-12 10:34:13,964] [pose_dataset] [INFO] dataflow img_path=/home/alessio/Sandbox/COCO/
loading annotations into memory...
Done (t=1.41s)
creating index...
index created!
[2018-01-12 10:34:15,432] [pose_dataset] [INFO] /home/alessio/Sandbox/COCO/annotations dataset 5000
[0112 10:34:15 @prefetch.py:169] [PrefetchData] Will fork a dataflow more than one times. This assumes the datapoints are i.i.d.
[0112 10:34:15 @prefetch.py:169] [PrefetchData] Will fork a dataflow more than one times. This assumes the datapoints are i.i.d.
[2018-01-12 10:34:23,542] [train] [INFO] tensorboard val image: 12
[2018-01-12 10:34:23,547] [train] [INFO] Tensor("fifo_queue_Dequeue:0", shape=(8, 288, 368, 3), dtype=float32, device=/device:GPU:0)
[2018-01-12 10:34:23,547] [train] [INFO] Tensor("fifo_queue_Dequeue:1", shape=(8, 72, 92, 19), dtype=float32, device=/device:GPU:0)
[2018-01-12 10:34:23,547] [train] [INFO] Tensor("fifo_queue_Dequeue:2", shape=(8, 72, 92, 38), dtype=float32, device=/device:GPU:0)
[2018-01-12 10:34:50,229] [train] [INFO] define model-
2018-01-12 10:34:57.854218: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful...

However, I noticed it uses a lot of RAM. So it may be that it swap to disk, which is really slow.

[tranorrepository]

I also find the training is slow. It seems dataflow is the bottleneck. "queue_size" is quite small, it is close to 0. Would you share your training log? @ildoonet

[andyqian2015]

@elmu thx for your reply, I check the code and find that the load data is too slow and I decrease the queue_size,and the speed is much better. During training, I find another question, the loss doesn't converage and still to 1000 more or less. Have you encounter this problem? thx if you can reply.

[elmuz]

the loss doesn't converage

Yes, many users found the same issue, included me. Maybe a problem of learning rate (or batch size). Let's see if @ildoonet shares some info more.

[ouceduxzk]

basically, his change of architecture is dramatic, and it is hard to train. I managed to train a mobilenet based 1 stage network with loss like this, the axis is the average of 200 iterations.

The l2 loss converges at around 35 in constrast to cmu-pose 20, l1 loss is around 130 vs 100 (cmu-pose), but I did use the original mobilenet architecture without any change and sgd as optimizer instead

[andyqian2015]

Hi @ouceduxzk , I am very appreciated of your reply. what do you mean of "based 1 stage network "(one stage of openpose?),what lr ,batchsize and other parameter do you choose ? the same as the auther? thx ouceduxzk

[andyqian2015]

@ouceduxzk , another question is you mean you use the original mobilenet architecture,is it the net in tf_pose_estimation or mobilenet paper(if the architecture, the output should be revised ? ) thx if you can reply.

[ouceduxzk]

@andyqian2015 for the first question, yes, only one stage of openpose, lr batchsize is same to the training of openpose, but I am using caffe instead of tensorflow in this case. The mobilenet arch is from the original paper and using first 12 layers as input

[andyqian2015]

@ouceduxzk thx, I am trying !

[ouceduxzk]

with batchsize 96, my loss now looks similar to what the author has

[elmuz]

So which are the general rules to train on our own a mobilenet (I wanted to try mobilenet_accurate)? I am suffering like many others of high loss values (1.0k~1.5k) and it would be nice if we could understand why.

• Considering the architecture: @ouceduxzk said "his change of architecture is dramatic". In what sense? Can you provide a pull request to add models in 'networks.py'?

Training options

• Both @ouceduxzk and @ildoonet used a 96 batch size. Is it vital to get convergence with little loss? If we have just a single 8GB GPU is there any chance to get good results?
• Which optimizer (adam? rmsprop?) you used in the end? Is there particular learning step program to set (if you used exp decay, which paramters you applied?)

[ouceduxzk]

@elmuz in the sense of concat different layers in the feat_concat , which are different type of information, then rmsprop is crucial in my experience to make the training stable. I am not sure how vital is the batchsize is, but now I am training on a 4-gpu or 8-gpu machine while before I usually use 1 or 2 gpu for experiments. I will try to reproduce the problem and figure out the most sensitive variable. lr decay is same as @ildoonet in the code

[ouceduxzk]

@elmuz the feat_concat in this mobilenet_thin is called Hyperfeature which works well for small object discovery, which is explained in this paper https://arxiv.org/pdf/1604.00600.pdf for object detection. Thanks @ildoonet for his work to show that hyper feature are great also for pose estimation

[ouceduxzk]

@elmuz I found big batch size, like 96 with 8 gpu is indeed better (not from the loss point of view) from the visualization point of view (more acc prediction and less false positives). I still did not get a great model that does not have obvious false positive though.

[elmuz]

Ok, thank you for the updates!

[FoolishMao]

The weight initialization method，I choose normal distribution instead of Xavier and have a good result.

[chuck0518]

@andyqian2015 for the first question, yes, only one stage of openpose, lr batchsize is same to the training of openpose, but I am using caffe instead of tensorflow in this case. The mobilenet arch is from the original paper and using first 12 layers as input

Can you show your prototxt file? thanks