Captum: Question: Why the model's forward method is called multiple times for a single lig.attribute() call?

[mainulquraishi]

Hi, I am new to Captum. I was trying to understand from this tutorial from the website's tutorial section.
I noticed for each call of the following line the model's forward method is called 6 times:
attributions_ig, delta = lig.attribute(input_indices, reference_indices, n_steps=500, return_convergence_delta=True)

Can anybody explain why it is called multiple times (6 times in this case)?

Question 2:
I wrote a print statement (inspection purpose) in the forward method to see the shape of the embedding inputs:
print(embedded.shape)
I get the following outputs for running for one sentence:

interpret_sentence(model, 'Best film ever', label=1)

output:

torch.Size([1, 1, 7, 50])
torch.Size([1, 1, 7, 50])
torch.Size([1, 1, 7, 50])
torch.Size([500, 1, 7, 50])
torch.Size([1, 1, 7, 50])
torch.Size([1, 1, 7, 50])

Heer, as I mentioned, the forward method has been called 6 times for a single sentence. Why the 4th one's embedding shape [500, 1, 7, 50]?
one clue is 500 is the n_step in lig.attribute .

I am hoping to some easy and intuitive explanation. Besides, pointing towards corresponding literature will also help.

[vivekmig]

Hi @mainulquraishi , of the 6 times the forward pass is called, the first time is as a result of the prediction call in interpret_sentence before actually calling Layer Integrated Gradients (pred = forward_with_sigmoid(input_indices).item()). The remaining 5 times are within Layer Integrated Gradients.

The first two calls evaluate the (embedding) layer at the input (input_indices) and baseline (reference_indices). Once the embeddings are evaluated at these points, we compute intermediate points between these two embeddings and evaluate the gradients at each of these points. This is controlled by num_steps, which is 500. The input is evaluated with 500 examples in order to compute the gradient at 500 points between the embeddings of the input_indices and reference_indices. The integrated gradients paper provides more details on this https://arxiv.org/pdf/1703.01365.pdf, in this case, the path being integrated is between the embedding baselines and inputs.

The last two calls are as a result of the option return_convergence_delta=True. This evaluates the output at the baseline and input to compute the convergence delta based on the completeness property (sum of attributions should match the difference between the model output at baseline and input). Setting this to False will avoid the last two model evaluations. Hope this helps!

[mainulquraishi]

Great, thanks for the explanation. I will go through the papers.

[vivekmig]

Closing this issue, feel free to create a new issue if you have further questions!