Tflearn: How to use tflearn with native session of tensorflow

[shixiazuike]

Sometimes we have to use tf.Session explicitly like distributed tensorflow. Example foleder gives an example layer.py but not work correctly because some layers like tflearn.dropout and tflearn.batch_normalization behave differently at training time and testing time. These layers use variable "is_training" and tf.cond to distinguish between training mode and testing mode, but the defalut value of "is_traing" is False, so the dropout layer does not work.

Here is an example:

import tensorflow as tf
import tflearn
from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

x = tf.placeholder(shape=(None, 784), dtype=tf.float32)
y_ = tf.placeholder(shape=(None, 10, dtype=tf.float32)

net = tflearn.input_data(placeholder=x)
net = tflearn.fully_connected(net, 128, activation='relu', weigits_init='zeros')
net = tflearn.dropout(net, 0.5)
net = tflearn.fully_connected(net, 256, activation='relu', weights_init='zeros')
net = tflearn.dropout(net, 0.5)
net = tflearn.fully_connected(net, 10, activation='softmax', weights_init='zeros')

loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(net, y_))
train_op = tf.train.AdamOptimizer(learning_rate=0.01).minimize(loss)

sess = tf.Session()
sess.run(tf.initialize_all_variables())

for i in range(100):
     batch_xs, batch_ys = mnist.train.next_batch(100)
     sess.run(train_op, feed_dict={x: batch_xs, y_: batch_ys})

correct_prediction = tf.equal(tf.argmax(net, 1), tf.argmax(y_, 1))
acc_val = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print sess.run(acc_val)

The result is exactly the same even if comment out part of droput code (Only the first 10,000 examles are used so no shuffle on data). Using tflearn.is_trianing(True) to set training mode but still not work

sess = tf.Session()
sess.run(tf.initialize_all_variables()) 
print sess.run(tflearn.get_training_mode()) #False
tflearn.is_training(True, session=sess)
print sess.run(tflear.get_training_mode())  #now True

Is a correct way to switch between training mode and testing mode?

[aymericdamien]

Yes, it is the correct way, you simply need to run:

# Before training, run this:
tflearn.is_training(True, session=sess)
# Before testing, run this:
tflearn.is_training(False, session=sess)

[aymericdamien]

So you should do something like that:

# Training
tflearn.is_training(True, session=sess)
sess.run([train_op])
# Testing/Predicting
tflearn.is_training(False, session=sess)
sess.run([accuracy_op])

[amobiny]

I'm having a very similar question, so I didn't create a new title.
I'm trying to use tflearn with native tensorflow session and I also need to add image preprocessing and augmentation. Here's the code I'm using now:

# Real-time data preprocessing
img_prep = ImagePreprocessing()
img_prep.add_featurewise_zero_center()
img_prep.add_featurewise_stdnorm()

# Real-time data augmentation
img_aug = ImageAugmentation()
img_aug.add_random_rotation(max_angle=180.)

# Input data
with tf.name_scope('Input'):
    X = tf.placeholder(tf.float32, shape=(None, image_size, image_size, image_size, num_channels), name='x-input')
    Y = tf.placeholder(tf.float32, shape=(None, label_cnt),name='yinput')

# Convolutional network building
network = input_data(shape=[None, 32, 32, 32, 1],
                     placeholder = X,
                     data_preprocessing=img_prep,
                     data_augmentation=img_aug)

# First Conv layer + Max Pooling
network = conv_3d(network, 8, 4, strides=2 , padding='valid',
                  regularizer=reg_type, weight_decay=lamda,
                  activation='relu', name='Conv1')
network = batch_normalization(network)
network = max_pool_3d(network, 3, strides=2, name='MaxPool1')
.
.
.
sess = tf.Session()
sess.run(tf.global_variables_initializer())
.
.
.

However, I found that when I'm running the session, it's not applying the data augmentation as such! I'm getting the following warning

WARNING:tensorflow:Error encountered when serializing data_augmentation. Type is unsupported, or the types of the items don't match field type in CollectionDef. 'ImageAugmentation' object has no attribute 'name'

WARNING:tensorflow:Error encountered when serializing data_preprocessing. Type is unsupported, or the types of the items don't match field type in CollectionDef. 'NoneType' object has no attribute 'name'

I think it must be because of the way I fed my tensor X in "input_data".

[sagarreddypatil]

Could you pass a tensorflow placeholder for the tflearn.is_training instead of a bool value? I want to control it depending on training and testing and that too in different scripts.