Chatterbot: SyntaxNet

[gunthercox]

SyntaxNet is an open source project released by Google for analysing the parts of a statement.
I haven't had a chance to look fully into it yet but this might be a usefull feature to add to ChatterBot. Essentially it could make it so that ChatterBot can learn to understand what statements mean.

• Google Research Blog article: https://research.googleblog.com/2016/05/announcing-syntaxnet-worlds-most.html
• Penn Treebank Project: http://www.cis.upenn.edu/~treebank/

[vkosuri]

@gunthercox interested to work on this feature? Any guidelines?

[gunthercox]

I think the best way to implement this would be as a comparison method, similar to the ones found in chatterbot/conversation/comparisons.py.

At this point, I'm not sure if this will be possible to do, but it would be amazing if the SyntaxNet library could be used to compare the similarity of two statements.

[vkosuri]

Thanks @gunthercox some useful links i have found

on ubuntu

• https://github.com/tensorflow/models/pull/115
• Installation instrctuion https://github.com/JoshData/models/blob/b72274d38f169f77e6a15e54834f463f627dc82a/syntaxnet/build/ubuntu-14.04_x64.sh

GitHub syntaxnet https://github.com/tensorflow/models/tree/master/syntaxnet

On Windows it's very difficulty https://github.com/tensorflow/models/issues/77

Stack-overflow syntaxnet tags http://stackoverflow.com/questions/tagged/syntaxnet

[narvind2003]

Hello,

I wrote a custom comparison method in chatterbot using the WMD (word mover's algorithm) implementation in the gensim python package.

WMD takes 2 sentences(or docs) and compares them semantically. Even if the 2 sentences have no common words, they get a high similarity score if they 'mean' the same thing.

Example - these 2 sentences get a high similarity score:

1. The king and queen paid a visit to the victims in the hospital last night
2. The patients got a royal visit yesterday

By combining the leveshtein distance with WMD distance, I am able to get chatterbot to perform a lot better - _Yet to start benchmarking_

Please let me know if you're interested in seeing the preliminary code.

_note: My intention is to really solve issue for my bot: "context based reply. #616"_

[vkosuri]

@narvind2003 Sure feel free to submit a PR what you have.

And also I was planning do this task using Dependencies and Dependency Grammar 5th topic in this chapter.

[kokenjr]

@narvind2003 Do you have an example of this custom comparison method? I'd like to implement it as well.

[narvind2003]

@kokenjr : I'm still working on publishing my code...some more testing to be done.

But a combination of levenshtein (for typos and word swaps) and WMD using gensim library (for semantic relationships) seems to work best. But the challenge is getting the comparisons done reasonably fast.

Please take a look at wmd_distance in gensim library using gloVe or fasstext embedings. I'll share updates soon.

[gunthercox]

I'm going to close this ticket off. It's been quite a while since I opened it and I don't think there is anything actionable here as of the present time.