Tfjs: Implement gradient for dl.gather

[nsthorat]

_From @tafsiri on March 5, 2018 15:35_

_Copied from original issue: tensorflow/tfjs-core#813_

[BusbyActual]

This issue link is broken or internal? I'm open to taking this just want to read about it more : ^)

[manrajgrover]

Hi @BusbyActual :)

All of the issues and discussions have been moved to this repository to centralize management of issues, hence the link is not working. Feel free to discuss the issue here and take this one up.

[BusbyActual]

Would this feature be similar to gradients? I checked the docs for gather && gatherrd and they don't reference gradients. I'm trying to scope out the issue and map it out before I dive in : ^)

[jgartman]

@BusbyActual Just to give a little background (sorry if you knew this already), you can see here that tf.gather in TensorflowJS doesn't have it's gradient defined. The gradient for gather in Tensorflow is here so you could use that as a guide if you wanted to implement it. One difficulty that I've noticed is that in Tensorflow the gradient uses IndexedSlices which are used to represent sparse tensors. Since sparse tensors don't yet exist in TensorflowJS I think you would need to find a way to convert that sparse representation to a dense tensor. Hope that helps you out a little bit.

[nsthorat]

Just took a look at the TF implementation, it looks non-trivial. As @jgartman says, the gradient for tf.gather is sparse, making this an interesting problem. It looks like TF is on GatherV2: https://github.com/tensorflow/tensorflow/blob/r1.7/tensorflow/python/ops/array_grad.py#L411-L472

The interesting here is we actually need to do something similar to unsorted_segment_sum: https://www.tensorflow.org/api_docs/python/tf/unsorted_segment_sum

That happens in the grad at line 464.

Conceptually this shouldn't be too difficult, but requires a bit of work to write an unsorted_segment_sum op as well as do the work that TF does in PY to recreate the gradient.

If somebody wants to take this one, I would be more than happy to guide in a code review.

[jgartman]

@nsthorat I'll give it a try.

[zaidalyafeai]

thanks for your efforts. Any alternative for the time being? I can't use embedding layers.

[BusbyActual]

About to be on holiday for a few days. Happy to take another issue when I return

[nsthorat]

@zaidalyafeai I'd rather not rush on this since it's a tricky gradient and we want to get it right. Will ping you when it's done :)

[tafsiri]

closed by https://github.com/tensorflow/tfjs-core/pull/981