Captum: Extension to graph model

[tchaton]

Dear people of Captum,

I would like to add explanability within https://github.com/nicolas-chaulet/deeppointcloud-benchmarks
How complex would it be to extend Captum to support at least Pytorch Geometric (https://github.com/rusty1s/pytorch_geometric)

Best,
Thomas Chaton

[NarineK]

Hi @tchaton,

I've tried IG on the tutorial from pytorch_geometric and I got it working with a small trick.
I modified original network to look like something like this:

class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = GCNConv(dataset.num_node_features, 16)
        self.conv2 = GCNConv(16, dataset.num_classes)

    def forward(self, x, edge_index):
        #x, edge_index = data.x, data.edge_index
        print(self.conv1(x, edge_index))
        x = self.conv1(x, edge_index)
        x = F.relu(x)
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index)

        return F.log_softmax(x, dim=1)

Then I attributed to the dataset[0].x, I'm not sure if that's what you want to attribute to but it looks like all values in the x are 0. We might need to think of a different representation.

from captum.attr import IntegratedGradients

def custom_forward(x, edge_index):
    return model(x.squeeze(), edge_index[0])

ig = IntegratedGradients(custom_forward)
attr = ig.attribute(dataset[0].x, additional_forward_args=((dataset[0].edge_index.unsqueeze(0))), target=0)

Let me know if this helps. I'm not very familiar with Pytorch Geometric let me know if this makes sense to you.

[tchaton]

Hey @NarineK

dataset[0].x. In PyG, they use a Data wrapper which uses the python setattr function to put tensor into attributes as x for the features, y for labels, etc.
So here, you are accessing the first indexed sample feature.

Ok sounds great ! So basically, as long as there is an forward and backward, even for custom modules, it should works.

The IntegreatedGradients should work, as it is going to call those functions.

Do you know if there are any methods which couldn't work properly due to custom modules ?
I had this trouble with ResNet when I implemented gradients https://arxiv.org/abs/1611.06440. Residual Blocks were a bit complex to handle properly.
I needed to create wrapper for each layer, and therefore couldn't be easily adapted to new layer.

Best,
Thomas Chaton.

[NarineK]

Hi @tchaton,
Yes, that's right as long as we have forward and backward even with custom modules it should work fine. It usually should work without problems. You can try to apply on those custom modules and let us know if you see any problems.

[tchaton]

Hey @Narine,

Last question: What would be your intuition about choosing a goof baseline for pointclouds for classification, segmentation ?

Next, we need to define simple input and baseline tensors. Baselines belong to the input space and
often carry no predictive signal. Zero tensor can serve as a baseline for many tasks. Some
interpretability algorithms such as Integrated Gradients, Deeplift and GradientShap are designed to
attribute the change between the input and baseline to a predictive class or a value that the neural
network outputs.

I could take random noise, or just points collapse in zero, or the centroid of a given class ?

[NarineK]

HI @tchaton,
it depends on the dataset and task. There is also an option to have a distribution of baselines and sample from that distribution and average the results: GradientShap does something similar. Some papers have shown that the careful you choose the baseline the more meaningful your attribution will become.

Here they study the choice of baselines but there are also many other papers too:
https://arxiv.org/pdf/1811.06471.pdf

[CM-BF]

Hi @NarineK,
I guess it's a good idea to wrap the model to make those inputs acceptable form to IG, because it only cares about inputs here. But for a Layer Method such as GradCAM, it cannot work, because the inner layer's features don't have the proper shape the attribute method need. Thus, I have to inherit the GradCAM, then rewrite a part of the attribute method like this:

        layer_gradients, layer_evals, is_layer_tuple = compute_layer_gradients_and_eval(
            self.forward_func,
            self.layer,
            inputs,
            target,
            additional_forward_args,
            device_ids=self.device_ids,
            attribute_to_layer_input=attribute_to_layer_input,
        )
        undo_gradient_requirements(inputs, gradient_mask)

        # what I add
        layer_gradients = tuple(layer_grad.transpose(0, 1).unsqueeze(0)
                           for layer_grad in layer_gradients)

        layer_evals = tuple(layer_eval.transpose(0, 1).unsqueeze(0)
                       for layer_eval in layer_evals)
        # end

        summed_grads = tuple(
            torch.mean(
                layer_grad,
                dim=tuple(x for x in range(2, len(layer_grad.shape))),
                keepdim=True,
            )
            for layer_grad in layer_gradients
        )

I think it is not friendly to us if we always have to hack in your code and modify it for using it.

[Saadman]

Hi @NarineK,
Is there any code and explanation specifically for graph convolutional networks which I can look at? The captum paper mentions that it can handle graph models but I couldn't find any examples for it. It will be really helpful to know how captum is used for such models.
Thanks in advance!

[joaquincabezas]

Hi @Saadman,

have you seen the Google Colab by Amin (@m30m)? He is applying feature attribution to graphs with the Mutagenicity dataset from TUDatasets (from @chrsmrrs)

Check it out at: https://colab.research.google.com/drive/1fLJbFPz0yMCQg81DdCP5I8jXw9LoggKO?usp=sharing

It has been just added to the list of colabs of Pytorch Geometric: https://pytorch-geometric.readthedocs.io/en/latest/notes/colabs.html

Regards,
Joaquín.

[Saadman]

Hello Joaquin,

This is fantastic! I'll look into it a bit more and see if I can use it
with my example.

Thank you!

Best,
Rashid

On Sun, Nov 29, 2020 at 3:41 PM Joaquin Cabezas [e-mail]
wrote:

Hi @Saadman https://github.com/Saadman,

have you seen the Google Colab by Amin (@m30m https://github.com/m30m)?
He is applying feature attribution to graphs with the Mutagenicity dataset
from TUDatasets (from @chrsmrrs https://github.com/chrsmrrs)

Check it out at:
https://colab.research.google.com/drive/1fLJbFPz0yMCQg81DdCP5I8jXw9LoggKO?usp=sharing

It has been just added to the list of colabs of Pytorch Geometric:
https://pytorch-geometric.readthedocs.io/en/latest/notes/colabs.html

Regards,
Joaquín.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pytorch/captum/issues/246#issuecomment-735450808, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AARWBR63HDNZK77E5OOFXHTSSKWWNANCNFSM4KJSGJPQ
.