Captum: Question: Integrated Gradient w/ Embedded Categorical Data

[reggievick]

Hi Everyone,

Question:

How can I apply integrated gradient to a dataset with numerical and embedded categorical data?

I am somewhat of a beginner with pytorch and the available resources are just not clicking with my use case. The ultimate goal is for me to plot the feature importance of a model, but I am stuck on calculating the attribution. Any help or guidance would be much appreciated.

What I've reviewed:

• Multimodal_VQA_Captum_Insights tutorial
• BERT tutorials
• https://github.com/pytorch/captum/issues/282

_(These resources all have very different data structures(images/sentences) and are confusing for a beginner to translate to an easier tabular numerical/categorical dataset)_

My Problem:

Tutorial/Full Code
Dataset

Model:
```Model(
(all_embeddings): ModuleList(
(0): Embedding(3, 2)
(1): Embedding(2, 1)
(2): Embedding(2, 1)
(3): Embedding(2, 1)
)
(embedding_dropout): Dropout(p=0.4, inplace=False)
(batch_norm_num): BatchNorm1d(6, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(layers): Sequential(
(0): Linear(in_features=11, out_features=200, bias=True)
(1): ReLU(inplace=True)
(2): BatchNorm1d(200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(3): Dropout(p=0.4, inplace=False)
(4): Linear(in_features=200, out_features=100, bias=True)
(5): ReLU(inplace=True)
(6): BatchNorm1d(100, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(7): Dropout(p=0.4, inplace=False)
(8): Linear(in_features=100, out_features=50, bias=True)
(9): ReLU(inplace=True)
(10): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(11): Dropout(p=0.4, inplace=False)
(12): Linear(in_features=50, out_features=2, bias=True)
)
)

**Categorical Data Example:**

tensor([[0, 0, 1, 1],
[2, 0, 0, 1],
[0, 0, 1, 0],
[0, 0, 0, 0],
[2, 0, 1, 1]])

**Numerical Data Example**

tensor([[6.1900e+02, 4.2000e+01, 2.0000e+00, 0.0000e+00, 1.0000e+00, 1.0135e+05],
[6.0800e+02, 4.1000e+01, 1.0000e+00, 8.3808e+04, 1.0000e+00, 1.1254e+05],
[5.0200e+02, 4.2000e+01, 8.0000e+00, 1.5966e+05, 3.0000e+00, 1.1393e+05],
[6.9900e+02, 3.9000e+01, 1.0000e+00, 0.0000e+00, 2.0000e+00, 9.3827e+04],
[8.5000e+02, 4.3000e+01, 2.0000e+00, 1.2551e+05, 1.0000e+00, 7.9084e+04]])

**Output Data Example**

tensor([1, 0, 1, 0, 0])

**My Failing Attempt at Attribution**

interpretable_embedding = configure_interpretable_embedding_layer(model, 'all_embeddings')

cat_input_embedding = interpretable_embedding.indices_to_embeddings(categorical_train_data).unsqueeze(0)

I received an error here "NotImplementedError"

ig = IntegratedGradients(model)

ig_attr_train = ig.attribute(inputs=(numerical_train_data, categorical_train_data), baselines=(numerical_train_data * 0.0, cat_input_embedding), target=train_outputs, n_steps=50)
```

[NarineK]

Thank you for the question, @reggievick ! Do you mind posting the entire stacktrace of the NotImplementedError ?
I'd be curious to know if you get similar error if you hook each embedding separately, similar to this?
https://captum.ai/tutorials/Bert_SQUAD_Interpret

[reggievick]

Thank you for the question, @reggievick ! Do you mind posting the entire stacktrace of the NotImplementedError ?
I'd be curious to know if you get similar error if you hook each embedding separately, similar to this?
https://captum.ai/tutorials/Bert_SQUAD_Interpret

Thanks for the quick response @NarineK! Sure, the full trace is below. I will take a look at the Bert_SQUAD_Interpret tutorial again, but I had a difficult time understanding how to make the construct_bert_sub_embedding function work with my data.

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
 in 
----> 1 cat_input_embedding = interpretable_embedding.indices_to_embeddings(categorical_train_data)

~/GitHub/torched/torched/lib/python3.7/site-packages/captum/attr/_models/base.py in indices_to_embeddings(self, *input, **kwargs)
     89             indices specified in the input
     90         """
---> 91         return self.embedding(*input, **kwargs)
     92 
     93 

~/GitHub/torched/torched/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/GitHub/torched/torched/lib/python3.7/site-packages/torch/nn/modules/module.py in forward(self, *input)
     94             registered hooks while the latter silently ignores them.
     95         """
---> 96         raise NotImplementedError
     97 
     98     def register_buffer(self, name, tensor):

NotImplementedError: 

[NarineK]

Hi @reggievick ! I was able to reproduce the error. In this specific example embeddings are wrapped in the ModuleList which is not a torch.nn.Module. We can only make torch.nn.Module's interpretable.
For that reason we need to hook the individual embedding layers. This works.

interpretable_embedding0 = configure_interpretable_embedding_layer(model, 'all_embeddings.0')
interpretable_embedding1 = configure_interpretable_embedding_layer(model, 'all_embeddings.1')
interpretable_embedding2 = configure_interpretable_embedding_layer(model, 'all_embeddings.2')
interpretable_embedding3 = configure_interpretable_embedding_layer(model, 'all_embeddings.3')

When you are done with everything you can remove them with:

remove_interpretable_embedding_layer(model, interpretable_embedding0)
remove_interpretable_embedding_layer(model, interpretable_embedding1)
remove_interpretable_embedding_layer(model, interpretable_embedding2)
remove_interpretable_embedding_layer(model, interpretable_embedding3)

For more details you can look into Bert tutorial.

[reggievick]

Thanks @NarineK i was just in the middle of trying the individual hooks out. I was able to successfully hook each embedding layer like your example, but still trying to figure out what the correct inputs and baselines should be. My current assumption is that I should apply the indices_to_embeddings function to each column's values and embedding indexes and add each as a seperate input/baseline. Am I on the right track?

[NarineK]

indices_to_embeddings method will give you the embedding representation for each batch. You need to create that same representation also for baselines using all 0-indices or whichever indice(s) are the best for baselines. You need to pass these to the attribute method of the attribution algorithm along with the real-valued input features and baselines. Does this make sense ?
Let me know.

Update:
I think it is a bit tricky with the concats of the embeddings.
What you can do is this:

emb0 = interpretable_embedding0.indices_to_embeddings(categorical_test_data[:,0])
emb1 = interpretable_embedding1.indices_to_embeddings(categorical_test_data[:,1])
emb2 = interpretable_embedding2.indices_to_embeddings(categorical_test_data[:,2])
emb3 = interpretable_embedding3.indices_to_embeddings(categorical_test_data[:,3])

cat_emb = torch.cat([emb0, emb1, emb2, emb3], axis=1)

So you can pass cat_emb instead of categorical_test_data as the inputs to attribute method.

[reggievick]

Thanks again @NarineK I think i'm close here, but am getting an additional error. Not sure if it's the way i did the baseline, or maybe another issue like needing a additional_forward_args

interpretable_embedding0 = configure_interpretable_embedding_layer(model, 'all_embeddings.0')
interpretable_embedding1 = configure_interpretable_embedding_layer(model, 'all_embeddings.1')
interpretable_embedding2 = configure_interpretable_embedding_layer(model, 'all_embeddings.2')
interpretable_embedding3 = configure_interpretable_embedding_layer(model, 'all_embeddings.3')

emb0 = interpretable_embedding0.indices_to_embeddings(categorical_test_data[:,0])
emb1 = interpretable_embedding0.indices_to_embeddings(categorical_test_data[:,1])
emb2 = interpretable_embedding0.indices_to_embeddings(categorical_test_data[:,2])
emb3 = interpretable_embedding0.indices_to_embeddings(categorical_test_data[:,3])

cat_emb = torch.stack((emb0, emb1, emb2, emb3), axis=1)
cat_base = torch.stack((emb0 * 0, emb1 * 0, emb2 * 0, emb3 * 0), axis=1)

ig = IntegratedGradients(model)
ig.attribute(inputs=(cat_emb, numerical_test_data), baselines=(cat_base, numerical_test_data * 0.0), target=test_outputs, n_steps=50)

Error

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
 in 
      1 ig = IntegratedGradients(model)
----> 2 ig.attribute(inputs=(cat_emb, numerical_test_data), baselines=(cat_base, numerical_test_data * 0.0), target=test_outputs, n_steps=50)

~/GitHub/torched/torched/lib/python3.7/site-packages/captum/attr/_core/integrated_gradients.py in attribute(self, inputs, baselines, target, additional_forward_args, n_steps, method, internal_batch_size, return_convergence_delta)
    282             internal_batch_size=internal_batch_size,
    283             forward_fn=self.forward_func,
--> 284             target_ind=expanded_target,
    285         )
    286 

~/GitHub/torched/torched/lib/python3.7/site-packages/captum/attr/_utils/batching.py in _batched_operator(operator, inputs, additional_forward_args, target_ind, internal_batch_size, **kwargs)
    162         )
    163         for input, additional, target in _batched_generator(
--> 164             inputs, additional_forward_args, target_ind, internal_batch_size
    165         )
    166     ]

~/GitHub/torched/torched/lib/python3.7/site-packages/captum/attr/_utils/batching.py in (.0)
    161             **kwargs
    162         )
--> 163         for input, additional, target in _batched_generator(
    164             inputs, additional_forward_args, target_ind, internal_batch_size
    165         )

~/GitHub/torched/torched/lib/python3.7/site-packages/captum/attr/_utils/gradient.py in compute_gradients(forward_fn, inputs, target_ind, additional_forward_args)
     94     with torch.autograd.set_grad_enabled(True):
     95         # runs forward pass
---> 96         outputs = _run_forward(forward_fn, inputs, target_ind, additional_forward_args)
     97         assert outputs[0].numel() == 1, (
     98             "Target not provided when necessary, cannot"

~/GitHub/torched/torched/lib/python3.7/site-packages/captum/attr/_utils/common.py in _run_forward(forward_func, inputs, target, additional_forward_args)
    501         *(*inputs, *additional_forward_args)
    502         if additional_forward_args is not None
--> 503         else inputs
    504     )
    505     return _select_targets(output, target)

~/GitHub/torched/torched/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

 in forward(self, x_categorical, x_numerical)
     31         x_numerical = self.batch_norm_num(x_numerical)
     32         x = torch.cat([x, x_numerical], 1)
---> 33         x = self.layers(x)
     34         return x

~/GitHub/torched/torched/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/GitHub/torched/torched/lib/python3.7/site-packages/torch/nn/modules/container.py in forward(self, input)
     98     def forward(self, input):
     99         for module in self:
--> 100             input = module(input)
    101         return input
    102 

~/GitHub/torched/torched/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/GitHub/torched/torched/lib/python3.7/site-packages/torch/nn/modules/linear.py in forward(self, input)
     85 
     86     def forward(self, input):
---> 87         return F.linear(input, self.weight, self.bias)
     88 
     89     def extra_repr(self):

~/GitHub/torched/torched/lib/python3.7/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1368     if input.dim() == 2 and bias is not None:
   1369         # fused op is marginally faster
-> 1370         ret = torch.addmm(bias, input, weight.t())
   1371     else:
   1372         output = input.matmul(weight.t())

RuntimeError: size mismatch, m1: [100000 x 14], m2: [11 x 200] at ../aten/src/TH/generic/THTensorMath.cpp:136

[NarineK]

yeah, I think we can clean things up and make more modular with something like this:

class CombinedEmbedding(nn.Module):
    def __init__(self, embedding_size):
        super().__init__()
        self.all_embeddings = nn.ModuleList([nn.Embedding(ni, nf) for ni, nf in embedding_size])

    def forward(self, x_categorical):
        embeddings = []
        for i,e in enumerate(self.all_embeddings):
            print(e(x_categorical[:,i]).shape)
            embeddings.append(e(x_categorical[:,i])) 
        x = torch.cat(embeddings, 1)    
        return x

class Model(nn.Module):

    def __init__(self, embedding_size, num_numerical_cols, output_size, layers, p=0.4):
        super().__init__()
        #self.all_embeddings = nn.ModuleList([nn.Embedding(ni, nf) for ni, nf in embedding_size])
        self.all_embedding = CombinedEmbedding(embedding_size)

        self.embedding_dropout = nn.Dropout(p)
        self.batch_norm_num = nn.BatchNorm1d(num_numerical_cols)

        all_layers = []
        num_categorical_cols = sum((nf for ni, nf in embedding_size))
        input_size = num_categorical_cols + num_numerical_cols

        for i in layers:
            all_layers.append(nn.Linear(input_size, i))
            all_layers.append(nn.ReLU(inplace=True))
            all_layers.append(nn.BatchNorm1d(i))
            all_layers.append(nn.Dropout(p))
            input_size = i

        all_layers.append(nn.Linear(layers[-1], output_size))

        self.layers = nn.Sequential(*all_layers)

    def forward(self, x_categorical, x_numerical):
        x = self.all_embedding(x_categorical)
        x = self.embedding_dropout(x)

        x_numerical = self.batch_norm_num(x_numerical)
        x = torch.cat([x, x_numerical], 1)
        x = self.layers(x)
        return x

Here is all you need for interpretability:

from captum.attr import IntegratedGradients
from captum.attr import configure_interpretable_embedding_layer, remove_interpretable_embedding_layer

interpretable_embedding = configure_interpretable_embedding_layer(model, 'all_embedding')

emb = interpretable_embedding.indices_to_embeddings(categorical_test_data)


ig = IntegratedGradients(model)
ig.attribute((emb, numerical_test_data), target=0)


remove_interpretable_embedding_layer(model, interpretable_embedding)

I didn't specify baselines. Feel free to specify it too.

[reggievick]

Awesome, that is much cleaner. I was planning on refactoring once i understood it, but you've nailed it here. Thanks so so much @NarineK!

[NarineK]

Closing for now since the problem got resolved. Feel free to open a new issue if you have more questions.