Theano: CTC for theano

[trungnt13]

Is it possible to integrate this CUDA-CTC into theano computational graph ?
https://github.com/baidu-research/warp-ctc

[nouiz]

This can be done, but we need to find someone to do it.

Mostly, this need one new op, it should cover the CPU and GPU
implementation. That information would be an input of the op. To compute or
not the gradient would also be an input.

Then an optimization would be needed to make it optimal for the case where
the gradient isn't needed.

If someone want to do it, tell us, we can guide him.

On Thu, Jan 14, 2016 at 4:28 PM, TrungNT [email protected] wrote:

Is it possible to integrate this CUDA-CTC into theano computational graph ?
https://githubcom/baidu-research/warp-ctc

—
Reply to this email directly or view it on GitHub
https://github.com/Theano/Theano/issues/3871.

[sherjilozair]

I wonder how this can be achieved. In my understanding, to create an Op for GPU computation, one needs to implement the c_code function, which requires that the code be in a string in Theano code and use the Python C-API. Warp-ctc is a third-party library. Is there any way Theano code can just call a function in a third-party library?

@nouiz, it will be useful if you could put down some of your preliminary thoughts here, on how warp-ctc could be ported to Theano.

[kastnerkyle]

I see a few options.

You could put the C code from warp-ctc (with associated license, attribution, and pointer back to the implementation) into an Op then have a version in Theano directly (ideal, accessible to many users), and make a PR back to warp-ctc with a separate copy of the Theano Op. But if it gets into Theano core (not a given, but it would be great) I don't see a need for 2 versions floating around when you could add a pointer from warp-ctc to the version in the Theano documentation. Given the use of Apache (vs. BSD 3-clause) for warp-ctc I think license compatibility is not a problem - that particular section of the codebase would just be under a _slightly_ more restricted license. See the discussion here.

There is a maintenance overhead associated with not sharing the same code files, but this also ensures the Theano version is in working order as other changes are made in the Theano codebase, rather than relying on a 3rd party to handle this/getting random bug reports about the boundary between warp-ctc and Theano.

Alternatively you could make warp-ctc a fixed dependency that gets updated periodically by developers, either through something like a git submodule or a simple copy-paste with commit versioning (best) somewhere in the Theano codebase. You could then compile it on install and the Op could work with the compiled .so. It would be tricky to ensure that it doesn't add install complexity for Theano users, though.

The more onerous way would be to require the user to download warp-ctc, compile it, and put it somewhere particular (maybe along LD_LIBRARY_PATH or something), then the Op would work, but that doesn't seem like a good general solution to me.

[ZhangAustin]

The first option from kastnerkyle makes more sense to me :+1:

BTW, it seems warp-ctc does not include decoding/inference, we have to do this part in Theano anyway

[nouiz]

From memory, the CTC code is accepting a NULL paramater to don,t compute
everything. So I think it support inference only. Baidu need that in
production, it would be a big missing piece if they didn't got that.

The simplest is to copy the file. To make it fast, I would make a
theano-ctc repo, that include the original file and have the op read that
file and use it as needed. So there should be no need for Theano
modification of the original file. We would just need to copy it to update
it.

On Tue, Jan 19, 2016 at 4:48 AM, Austin Zhang [email protected]
wrote:

The first option from kastnerkyle makes more sense to me [image: :+1:]

BTW, it seems warp-ctc does not include inference ...

—
Reply to this email directly or view it on GitHub
https://github.com/Theano/Theano/issues/3871#issuecomment-172795006.

[sherjilozair]

FWIW, I hacked together Numpy bindings for warp-ctc here: https://github.com/sherjilozair/warp-ctc/tree/interface/python

It uses ctypes, and is fairly straightforward. It's still rough around the edges, but may be a good starting point for someone working on porting to Theano.

[sherjilozair]

@nouiz, Theano chooses to keep the implementations of perform and grad separate. However, in this case, the same computation actually gets us both the cost and grad, and doing the computation two times is inefficient. What's the right design pattern for this case? Should one compute both, return only cost, and put the gradient in a node member variable, and then when grad is called, return whatever was saved in that variable? This doesn't feel very clean to me. Is there something like perform_and_grad for a Theano Op?

[nouiz]

I would go to have the forward call do both. So the make_node would return
2 values and add the attribute default_output=0 in the op. So the second
(grad) output of the node won't be returned to the user.

Then use this in the grad. The op need a prop compute_grad.

some code to make this clear:

class CTC(Op):
default_output=0
props = ("compute_grad",)
def init(self, compute_grad=True):
self.compute_grad = True
def make_node(self, inputs...):
...
cost = ...
grad = ...
if self.compute_grad:
return Apply(self, inputs, [cost, grad])
return Apply(self, inputs, [cost])
def grad(self, inp, grads):
out = self(inp)
return out.owner.outputs[1]
def c_code(...):
# use self.compute_grad to compute the grad only when needed

so this would reuse the output of computed during the forward

To don't compute the grad when not needed (when only testing, not
training), then just add this optimization

@register_canonicalize
@register_stabilize
@gof.local_optimizer([CTC])
def local_ctc_no_grad(node):
if not isinstance(node.op, CTC):
return
if len(node.outputs[1].clients) == 0:
return [CTC(compute_grad=False)(node.inputs)]

On Wed, Jan 20, 2016 at 8:20 PM, Sherjil Ozair [email protected]
wrote:

@nouiz https://github.com/nouiz, Theano chooses to keep the
implementations of perform and grad separate. However, in this case, the
same computation actually gets us both the cost and grad, and doing the
computation two times is inefficient. What's the right design pattern for
this case? Should one compute both, return only cost, and put the gradient
in a node member variable, and then when grad is called, return whatever
was saved in that variable? This doesn't feel very clean to me. Is there
something like perform_and_grad for a Theano Op?

—
Reply to this email directly or view it on GitHub
https://github.com/Theano/Theano/issues/3871#issuecomment-173421092.

[sherjilozair]

Here's a basic implementation of Theano bindings for warp-ctc: https://github.com/sherjilozair/ctc

[mcf06]

I've published an initial c_code()-based implementation of bindings for warp-ctc at https://github.com/mcf06/theano_ctc

CPU and GPU computation are supported.

This is set up as a separate repository, but folding it into Theano would be welcome.

[nouiz]

Thanks for that and sorry for the delay, the email got list in my inbox.

I'm not sure how to handle the license. Here is the option I see:

• If you are ready to make your code under BSD 3 clauses, we could move
  this to Theano and it would only work when user install baidu-ctc separatly
  as it is under Apache 2.0
• Keep this in a separate repo under Apache license, then you could copy
  baidu-ctc in that repo to only request 1 extra repo and not 2.
• Put the Theano binding into baidu-ctc repo.

In all cases, we could document that in Theano.

Are you read to change your license to the same as Theano to include it in
Theano?

On Fri, Apr 8, 2016 at 1:54 PM, mcf06 [email protected] wrote:

I've published an initial c_code()-based implementation of bindings for
warp-ctc at https://github.com/mcf06/theano_ctc

CPU and GPU computation are supported.

This is set up as a separate repository, but folding it into Theano would
be welcome.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/Theano/Theano/issues/3871#issuecomment-207536539

[mcf06]

I've updated github to our latest internal version, and I've changed the license to BSD 3-clause.

The current code looks at the environment variable CTC_LIB to find warp-ctc headers and libraries. If the user downloads and compiles warp-ctc and sets CTC_LIB, then it will get linked in when the theano ops get compiled. If the theano_ctc code is merged into theano, this behavior could be maintained. Alternatively / additionally, one could add some code to try to guess a default path for warp-ctc, but I doubt it's worth the effort.

[saseptim]

I have a question about your implementation: why do you not take as an input the label lengths of the outputs?

[mcf06]

Numpy and Theano are optimized around rectangular matrices instead of variable-length data structures. The typical design pattern for SGD training in Theano is to load one's training data into large matrices and then carve out fixed-sized submatrices for each minibatch (e.g. miniBatchfeatures = features[b * batchSize : (b+1) * batchSize, :] for b in range(0, miniBatchN)). To accommodate variable-length labels, any negative target values are ignored.

Note that the negative target values do not need to be at the end. For speech rec, where one may have existing frame-level alignments, the target vector could contain a phone target at the start / middle / end of each phone (negative values at other frames), which makes debugging and resegmentation easier.

[saseptim]

Thanks for your reply.

I was not aware that negative values play a role in using the CTC. Until now, I would simply use a 'PAD' label to pad the input (for speech recognition) with this value, and then give a vector of input lengths. I have been using the theano wrapper at https://github.com/sherjilozair/ctc because that is the one I found first, and he takes the input lengths as a vector of length minibatchsize x 1. My problem until now has been that my system learns with minibatch size 1 but not with > 1.

Is this idea of using negative values for "ignore" something unique to your theano wrapper, or all wrappers to warp-ctc would use this convention?

[mcf06]

Ignoring negative values is part of the wrapper code. The warp-ctc C code takes a different type of data structure. I'm not sure about other wrappers.

[saseptim]

Thanks, I have got it working now.