Keras: class_weight

[ErnstTmp]

Hello,
I am trying to add a class weight to a graph model that is fitted by a generator. I have 3 classes, and the occurrence of the classes are in a ratio of 1:1:10. Whatever class_weight I chose, result is identical: only class 2 is chosen (the one occurring the most). Also, losses and final loss of the optimization are identical, and seem not take into account the class_weight. Am I doing something wrong or is this a bug?

The number of classes is 3, and the class_weight looks like this:

class_weight = {'output': {0:1000., 1:1000., 2:1.}}

The compile looks like this:

loss="categorical_crossentropy"
model.compile(optimizer=optimizer, loss={'output': loss})

The fit_generator looks like this:

    history = model.fit_generator(generator(X, y),
                      nb_worker=1,
                      samples_per_epoch=2,
                      verbose=1,
                      class_weight=class_weight,
                      nb_epoch=epoch)

Thanks a lot for your help,
Ernst

[ErnstTmp]

I looked through the Keras - Code, i found in models.py that the sample_weights are forced to be a dict:

  def _check_generator_output(self, generator_output, stop):
.....
        assert type(sample_weight) == dict

lateron, in models.py, however, the class_weights are only enforced when sample_weight is None:

This can not be the case, however, since sample_weights is an empty dict as set before.

def standardize_weights(y, sample_weight=None, class_weight=None,
                        sample_weight_mode=None):
....
   if sample_weight is not None:
        assert len(sample_weight.shape) <= len(y.shape)
        assert y.shape[:len(sample_weight.shape)] == sample_weight.shape
        return sample_weight
    elif isinstance(class_weight, dict):
        weights = np.asarray([class_weight[cls] for cls in y_classes])
  .... 

So, class_weights are never used, even when they are set as input.

What can I do to help to fix this - my knowledge of Keras (which is GREAT!!!) is too limited to fix this myself.

Thanks a lot for creating Keras,
Ernst

[EderSantana]

Did you try class_weights = {"Ouput1": weights1, "Output2": weights2}? For graphs things usually have to be dicts so we can match things by name.

[ErnstTmp]

Eder,
thanks for your message. I have only one output named 'output', so I had
{'output': {0:100000000., 1:1000., 2:1.},}, but this did not help.

However, I managed to patch the problem temporarily (by removing the block:

if sample_weight is not None:
     assert len(sample_weight.shape) <= len(y.shape)
     assert y.shape[:len(sample_weight.shape)] == sample_weight.shape
     return sample_weight
 el

from model.py (in standardize_weights)

but then I get the exception: 'class_weight not supported for 3+
dimensional targets.'

Is this a hard limit or could we extend the program to make this work
with 3 or more dimensions?

Thanks,
Ernst

On 03/26/2016 05:16 AM, Eder Santana wrote:

Did you try |class_weights = {"Ouput1": weights1, "Output2":
weights2}|? For graphs things usually have to be dicts so we can match
things by name.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/2071#issuecomment-201702746

[EderSantana]

I don't think you should use a dict of dicts there. make your class_weight into a singled 3 column matrix with the same shape as your final output and do {"output": my_3d_matrix}

[kepler]

class_weights is never used even when y has less than 3 dimensions.

In models.py:1366 there's this call to standardize_weights:

sample_weight_list = [standardize_weights(y[i],
                      sample_weight=sample_weight.get(self.output_order[i]),
                      sample_weight_mode=self.sample_weight_modes.get(self.output_order[i])) for i in range(len(self.output_order))]

standardize_weights:129-149 contains the following code:

    if sample_weight is not None:
        assert len(sample_weight.shape) <= len(y.shape)
        assert y.shape[:len(sample_weight.shape)] == sample_weight.shape
        return sample_weight
    elif isinstance(class_weight, dict):
        if len(y.shape) > 2:
            raise Exception('class_weight not supported for '
                            '3+ dimensional targets.')
        if y.shape[1] > 1:
            y_classes = y.argmax(axis=1)
        elif y.shape[1] == 1:
            y_classes = np.reshape(y, y.shape[0])
        else:
            y_classes = y
        weights = np.asarray([class_weight[cls] for cls in y_classes])
        return weights
    else:
        if sample_weight_mode is None:
            return np.ones((y.shape[0],))
        else:
            return np.ones((y.shape[0], y.shape[1]))

Having sample_weight and sample_weight_mode set to None makes line 147 to be run (3rd from the bottom in the above snippet), thus setting sample_weight_list to an array of ones. Further down in models.py:1397, standardize_weights is called again, this time with class_weight_list passed as parameter (class_weight_list is built from class_weight, which yes, should be a dict of dicts like {'output': {0:100000000., 1:1000., 2:1.}}). However, sample_weight_list is also passed as parameter, but this time it is an array of ones, so in the above code snippet, lines 130-132 are run (2-4).

A workaround is to use sample_weight: y[y.argmax(axis=1) == 0] = 100000000, y[y.argmax(axis=1) == 1] = 1000, y[y.argmax(axis=1) == 2] = 1, etc.

We should probably wait for Keras-1 before trying to fix this.

[ErnstTmp]

Hello,
thank you very much for your help, class_weights now work in Keras 1.0!!

Thanks again for making Keras such a good tool for Neural Net research!

Kind regards
Ernst

[ghost]

Hello, @ErnstTmp. Could I ask a sample code how you solved the issue?
As suggested above, did you use sample_weight, instead of class_weight?

[newbiesitl]

Hi ghost,
this is my sample code that uses class_weights, as mentioned before, class_weights only works when sample_weights is None.
But in some cases we need both sample_weights and class_weights to be set, like off-policy update in Reinforcement learning

from keras.models import Sequential
from keras.layers import Dense
from keras import optimizers
import numpy as np
model = Sequential()
model.add(Dense(4, input_shape=(2,)))
model.add(Dense(4, activation='softmax'))
model.compile(optimizer=optimizers.Adagrad(), loss='categorical_crossentropy')


x = np.array([[1,1], [1,1]])
y = np.array([[0,1,0,0], [0,0,0,1]])
weights_mask = np.array([1, 1])
class_weights = {
    0:0,
    1:0,
    2:0,
    3:1
}
# weights_mask = np.array([1])
# model.fit(x,y, epochs=1000, sample_weight=weights_mask, class_weight=class_weights, validation_data=((x,y)))
model.fit(x,y, epochs=1000, class_weight=class_weights, validation_data=((x,y)))

ret = model.predict(x)

print(ret)
print(sum(ret[0]))