3Q for your tf code.But when i train the mobilenet model, the memory use 80g,it is so big.How much memory do you need when training?
my batchsize is 32
@gavinzhang1995 Since proprocessed input batch is large, it isn't inevitable.
But I'm trying to squeeze memory footprint, using lower precision variables.
Also It might have some duplicated memory, I will check that too.
Thanks.
Thanks for you answer. I also meet another problem. During training, memory usage is increasing until out of memory. The memory is not GPU Memory
@gavinzhang1995 Okay, I will look into it. In my machines, It is increased but doesn't get me a OOM(Out of Memory) Error. I guess my machines hava larger mems...
@gavinzhang1995 Mem usage is decreased now, but it will consume some amount. This is inevitable since input features and labels are basically large.
I recommend you to run training on a machine with large memory.
Close due to inacitivity.
But I will optimize codes further...
Most helpful comment
@gavinzhang1995 Mem usage is decreased now, but it will consume some amount. This is inevitable since input features and labels are basically large.
I recommend you to run training on a machine with large memory.