Gluon-nlp: [Good First Issue] Add test cases for Truncated Gumbel

[sxjscience]

Well, Gumbel Distribution is magical. Basically, given a sequence of K logits, i.e., "\log a_1, \log a_2, ..., \log a_K" and K independent gumbel random variables, i.e., "g_1, g_2, ..., g_K". We have

\argmax_i \log a_i + g_i ~ Categorical({a_i / sum(a)})

This gives you a very simple way to sample from categorical distribution:

• Just draw a bunch of gumble variables with the same shape as the input logits. You may do that with np.random.gumble() as implemented in the source code:
  https://github.com/dmlc/gluon-nlp/blob/32e87d4d4aa20a6eb658ee90d765ccffbd160571/src/gluonnlp/op.py#L187
• Add the gumbels to the logits
• Perform an argmax on the correct axis (i.e., the one that you will need to enumerate the classes).

Now, let's understand the meaning of TruncatedGumbel. The definition is that you need to sample from a distribution that has the same cumulative density function as Gumbel, but is truncated at the given threshold \tau, meaning that it will never exceed \tau.

Here, we implement the TruncatedGumbel via the inverse CDF technique, which is also used in A* sampling (See footnote 1 on Page 3):
https://github.com/dmlc/gluon-nlp/blob/32e87d4d4aa20a6eb658ee90d765ccffbd160571/src/gluonnlp/op.py#L200-L228

The question is how would you verify that the implementation is correct? Basically, it relates to how you may conduct statistical testing and the author of this piece of codes has not implemented the test case well (he's not sure about how to do it well...):
https://github.com/dmlc/gluon-nlp/blob/32e87d4d4aa20a6eb658ee90d765ccffbd160571/tests/test_op.py#L106-L113

We will need your help and this will be a great contribution to GluonNLP!

Just FYI @xidulu @aashiqmuhamed @GomuGomuGo @liangliannie

[xidulu]

Recall that truncated Gumbel has a closed form CDF and inverse CDF, so you could simply use the random sample testing infra from MXNet, see https://github.com/apache/incubator-mxnet/blob/master/tests/nightly/https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/test_utils.py#L81 for examples.

Of course, a simpler solution would be KS test (empirical CDF v.s. analytical CDF): https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kstest.html , which only involves a CDF function and an array of samples.

[sxjscience]

Yeah, feel free to add the test case for that!

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Aditya Mandke [e-mail]
Sent: Tuesday, September 1, 2020 6:32:01 AM
To: dmlc/gluon-nlp [e-mail]
Cc: Xingjian SHI [e-mail]; Author [e-mail]
Subject: Re: [dmlc/gluon-nlp] [Good First Issue] Add test cases for Truncated Gumbel (#1299)

Can I do this?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/dmlc/gluon-nlp/issues/1299#issuecomment-684855544, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABHQH3RGHQ4DJIPMDV4C62TSDTZVDANCNFSM4P7K3I7A.