Tflearn: Copy-paste error in dnn.py and network with multiple outputs (multi-task learning)

[mubastan]

There is (I guess) a copy-paste bug in dnn.py

self.targets = tf.get_collection(tf.GraphKeys.TARGETS)
if len(self.inputs) == 0:

The second line should be if len(self.targets) == 0:

One question: is it possible to build a network with two outputs to do multi-task learning using tflearn?
Something like this:
VGG network with 2 outputs, one for 256 classes to be trained on one dataset; another with 45 classes to be trained on another dataset. The training will be interleaved; I do not want to train on one dataset, save model to disk, reload and fine tune on the other dataset.
I am trying something like this (see below), but it gives an error:

feed_dict[net_targets[i]] = y
IndexError: list index out of range

Because, self.targets = tf.get_collection(tf.GraphKeys.TARGETS) in dnn.py (70) returns 2 targets (network has two outputs), but there is actually one target at a time.

network = vggnet.VGG12()

task1 = tasks.get_caltech256_task()
task2 = tasks.get_mvod45_task()

network1 = fully_connected(network, task1.num_classes, activation='softmax', regularizer='L2', name='fc_1_8')
network2 = fully_connected(network, task2.num_classes, activation='softmax', regularizer='L2', name='fc_2_8')

regress1 = regression(network1, optimizer='adam', loss='categorical_crossentropy', learning_rate=task1.learning_rate, batch_size=task1.batch_size)
regress2 = regression(network2, optimizer='adam', loss='categorical_crossentropy', learning_rate=task2.learning_rate, batch_size=task2.batch_size)

tf.get_variable_scope().reuse_variables()

model1 = tflearn.DNN(regress1, checkpoint_path=task1.ckptfile, max_checkpoints=1)
model2 = tflearn.DNN(regress2, checkpoint_path=task2.ckptfile, max_checkpoints=1)

ds1 = datasets.load_caltech256_dataset()
ds2 = datasets.load_mvod45_dataset()

ITERATIONS = 100
for i in range(ITERATIONS):
    print "\n**** TASK 1 ****"
    model1.fit(ds1.Xtrain, ds1.Ytrain, n_epoch=1, shuffle=True, validation_set=(ds1.Xtest, ds1.Ytest), show_metric=True, batch_size=task1.batch_size, run_id=task1.runid)

    print "\n**** TASK 2 ****"
    model2.fit(ds2.Xtrain, ds2.Ytrain, n_epoch=1, shuffle=True, validation_set=(ds2.Xtest, ds2.Ytest), show_metric=True, batch_size=task2.batch_size, run_id=task2.runid)

[aymericdamien]

Yes, it is possible. You can use any number of input/target and optimizers. In your case, you need to feed 2 times targets (if you have 2 regressions layers => so 2 placeholders to feed), so it gives: model1.fit(ds1.X, [ds1.Ytrain, ds1.Ytrain])

You may also use the 'Merge' layer and train the model all in one:

merge = tflearn.Merge([regress1, regress2], 1)
model = tflearn.DNN(merge, ...)
model.fit(ds1.X, [ds1.Ytrain, ds1.Ytrain], ...)

[mubastan]

Thanks for the reply. I tried the first option, because I am not sure if the second option will do what I am trying to do:
I have 2 tasks, each task has its own data set, task 1: (X1, Y1); task 2: (X2, Y2).
I would like to train task 1 on (X1, Y1) for some epochs.
Then I would like to train task 2 on (X2, Y2) for some epochs.
And this will be repeated.

Therefore, when the input is X1, the network should train only task 1 based on labels Y1; and same for task 2.
But if I do the following, I think it will try to train both output layers based on Y1 and Y2 labels, which is wrong:

model1.fit(ds1.Xtrain, [ds1.Ytrain, ds2.Ytrain], n_epoch=1, shuffle=True, validation_set=(ds1.Xtest, [ds1.Ytest, ds2.Ytest]), show_metric=True, batch_size=task1.batch_size, run_id=task1.runid)
model2.fit(ds2.Xtrain, [ds1.Ytrain, ds2.Ytrain], n_epoch=1, shuffle=True, validation_set=(ds2.Xtest, [ds1.Ytest, ds2.Ytest]), show_metric=True, batch_size=task2.batch_size, run_id=task2.runid)

Anyway, the above code gives the following error message and program hangs:

Log directory: /tmp/tflearn_logs/
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/xyz/libs/tf09/local/lib/python2.7/site-packages/tflearn/data_flow.py", line 183, in fill_feed_dict_queue
    data = self.retrieve_data(batch_ids)
  File "/home/xyz/libs/tf09/local/lib/python2.7/site-packages/tflearn/data_flow.py", line 218, in retrieve_data
    utils.slice_array(self.feed_dict[key], batch_ids)
  File "/home/xyz/libs/tf09/local/lib/python2.7/site-packages/tflearn/utils.py", line 166, in slice_array
    return X[start]
IndexError: index 5294 is out of bounds for axis 0 with size 5000

---------------------------------
Training samples: 10720
Validation samples: 2680
--
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/xyz/libs/tf09/local/lib/python2.7/site-packages/tflearn/data_flow.py", line 183, in fill_feed_dict_queue
    data = self.retrieve_data(batch_ids)
  File "/home/xyz/libs/tf09/local/lib/python2.7/site-packages/tflearn/data_flow.py", line 218, in retrieve_data
    utils.slice_array(self.feed_dict[key], batch_ids)
  File "/home/xyz/libs/tf09/local/lib/python2.7/site-packages/tflearn/utils.py", line 166, in slice_array
    return X[start]
IndexError: index 5275 is out of bounds for axis 0 with size 5000

The inputs, targets and datasets are like this:

Inputs and targets:
[<tf.Tensor 'input/X:0' shape=(?, 256, 256, 3) dtype=float32>]
[<tf.Tensor 'TargetsData/Y:0' shape=(?, 67) dtype=float32>, <tf.Tensor 'TargetsData_1/Y:0' shape=(?, 45) dtype=float32>]
Y1: Tensor("TargetsData/Y:0", shape=(?, 67), dtype=float32)
Y2: Tensor("TargetsData_1/Y:0", shape=(?, 45), dtype=float32)

Task 1 dataset:
Training data: (5360, 256, 256, 3)
Training data labels: (5360, 67)
Test data: (1340, 256, 256, 3)
Test data labels: (1340, 67)

Task 2 dataset:
Training data: (5000, 256, 256, 3)
Training data labels: (5000, 45)
Test data: (203, 256, 256, 3)
Test data labels: (203, 45)

I also tried the following and it resulted in a similar error message and hang:

model1.fit( {Xp: ds1.Xtrain}, [{Y1p: ds1.Ytrain}, {Y2p: ds2.Ytrain}], n_epoch=1, shuffle=True, validation_set=({Xp: ds1.Xtest}, [{Y1p: ds1.Ytest}, {Y2p: ds2.Ytest}]), show_metric=True, batch_size=task1.batch_size, run_id=task1.runid)
model2.fit( {Xp: ds2.Xtrain}, [{Y1p: ds1.Ytrain}, {Y2p: ds2.Ytrain}], n_epoch=1, shuffle=True, validation_set=({Xp: ds2.Xtest}, [{Y1p: ds1.Ytest}, {Y2p: ds2.Ytest}]), show_metric=True, batch_size=task2.batch_size, run_id=task2.runid)

Log directory: /tmp/tflearn_logs/
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/xyz/libs/tf09/local/lib/python2.7/site-packages/tflearn/data_flow.py", line 183, in fill_feed_dict_queue
    data = self.retrieve_data(batch_ids)
  File "/home/xyz/libs/tf09/local/lib/python2.7/site-packages/tflearn/data_flow.py", line 218, in retrieve_data
    utils.slice_array(self.feed_dict[key], batch_ids)
  File "/home/xyz/libs/tf09/local/lib/python2.7/site-packages/tflearn/utils.py", line 166, in slice_array
    return X[start]
TypeError: unhashable type: 'numpy.ndarray'

Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/mubastan/libs/tf09/local/lib/python2.7/site-packages/tflearn/data_flow.py", line 183, in fill_feed_dict_queue
    data = self.retrieve_data(batch_ids)
  File "/home/mubastan/libs/tf09/local/lib/python2.7/site-packages/tflearn/data_flow.py", line 218, in retrieve_data
    utils.slice_array(self.feed_dict[key], batch_ids)
  File "/home/mubastan/libs/tf09/local/lib/python2.7/site-packages/tflearn/utils.py", line 166, in slice_array
    return X[start]
TypeError: unhashable type: 'numpy.ndarray'

---------------------------------
Training samples: 2
Validation samples: 2
--

Maybe I need to write the training loop myself, if tflearn cannot handle such a case.

[aymericdamien]

I think the reason is that your two data input should have the same input data length.
Task1: 5000 and Task2: 5000. It seems we actually have a bug here, as it TFLearn should be able to handle different data with not the same length.

[bhavikajalli]

I would like to perform joint learning using tflearn where I combine different losses from a single fully connected layer and perform a joint optimization. How can I go about doing that using tflearn?