Incubator-mxnet: Hi, how to create .rec?

[alecwangcq]

I have two .txt files(maybe train.txt and validation.txt, both have been shuffled already) which contains all the address and label of each sample:
address_of_sample label

So, I hope to create train.rec and validation.rec according to these two .txt files. And I googled it, but found nothing.

Maybe I should generate .lst files manually?
Such as:
train.lst
0 label address_of_sample
1 label address_of_sample
.....
.....
n label address_of_sample

---

validation.lst
n+1 label address_of_sample
n+2 label address_of_sample
......
.....

---

Note:: Why this website now is not accessible?(https://mxnet.readthedocs.io/en/stable/python/io.html)

[hariag]

Hi,

I think the .txt file should work also.
try following command

~/mxnet/bin/im2rec  train.txt  /media/data/ train.rec resize=32
~/mxnet/bin/im2rec  val.txt    /media//data/ vali.rec  resize=32

which /media/data/ is the base folder of your dataset resize=32 means the input shape of your images
for base folder , ex
assume you have two folder under /media/data/ named category1, category2, then in the train.txt should looks like
0 labelforcategory1 category1/00001.jpg
1 labelforcategory2 category2/00001.jpg
....
use relative path instead of full path here if you specify basefolder.

good luck!

[alecwangcq]

Err...
Resize=32 means output shape? Since my input images' shape are varied....
No mean.bin(mean image) will be generated? (In caffe, mean image can be generated after got .lmdb file...)
Is it true that during the training process, mxnet will subtract the training sample with mean image automatically?
And how about when predicting?

Thanks anyway!!

[hariag]

what I mean input shape is for the network, you don't need to generate mean data mxnet will generate it if not exists.I don't think mxnet will subtract mean automarically, you need do it mannually.

[alecwangcq]

Okay, when it will generate mean data? The time I generate .rec file? or at the beginning during training stage?

[hariag]

at the beginning of training。

[alecwangcq]

So, what about when I applied my trained network to predict a new input image. Will it subtract the mean data automatically? I do not have mean data... (Maybe at the beginning of training, mxnet generates the mean data and stores it physically?

[hariag]

you need manually subtract mean. mxnet generate mean data at the begining of teaining.

[alecwangcq]

So, that's the problem. After I finished my training, and I got my trained network, but no mean data(if mxnet did not save the automatically-generated mean data at the beginning of training), I am not quite sure about how to subtract mean manually... sad...

[hariag]

do you specify mean option to your dataiter?

[alecwangcq]

ah, could you be a bit more specific...

[alecwangcq]

`train = mx.io.ImageRecordIter(
path_imgrec = args.data_dir + "_train.rec",
data_shape = data_shape,
batch_size = args.batch_size,
rand_crop = True,
rand_mirror = False,
num_parts = kv.num_workers,
part_index = kv.rank)

val = mx.io.ImageRecordIter(
    path_imgrec = args.data_dir + "_val.rec",
    rand_crop   = False,
    rand_mirror = False,
    data_shape  = data_shape,
    batch_size  = args.batch_size,
    num_parts   = kv.num_workers,
    part_index  = kv.rank)`

[hariag]

train = mx.io.ImageRecordIter(
path_imgrec = os.path.join(args.data_dir, args.train_dataset),
mean_r = 123.68,
mean_g = 116.779,
mean_b = 103.939,
data_shape = data_shape,
batch_size = args.batch_size,
rand_crop = True,
rand_mirror = True,
num_parts = kv.num_workers,
part_index = kv.rank)
by default , it use fixed value, replace mean_r,g,b to mean=os.path.join(args.data_dir, "mean.bin")

sorry i reply you through mobilephone, i can't say more detailed.

[hariag]

specify such option, and mxnet will generate mean.bin for you. have a try.

[alecwangcq]

Oh, thank you! Appreciate!!