Chatterbot: The list trainer is slowly

[wilddylan]

I have 1 million conversation, when train with ListTrainer, it's slowly. how to speed up it.

[gunthercox]

Hi @WildDylan This is something I will have to look into. That's a pretty large data set and some changes to ChatterBot may be required to speed that up.

[wilddylan]

yep, Does ChatterBot used processor or other multiple thread when training?

[anilneeluri]

What are the formats used to train the chatbot?

[wilddylan]

799

[anilneeluri]

Thanks for responding. But, I want to know the default formats used to train the chatbot.

[wilddylan]

from chatterbot.trainers import ListTrainer

...

chatbot.set_trainer(ListTrainer)
trainingSet = [
"Hello",
"Hi"
]

chatbot.train(trainingSet)
chatbot.trainer.export_for_training('./train_export.json')

[wilddylan]

@gunthercox When I use SQLite as the storage adapter, get this error:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/chatterbot/storage/sql_storage.py", line 292, in _session_finish
    session.commit()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 906, in commit
    self.transaction.commit()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 461, in commit
    self._prepare_impl()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 441, in _prepare_impl
    self.session.flush()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2171, in flush
    self._flush(objects)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2291, in _flush
    transaction.rollback(_capture_exception=True)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
    raise value
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2255, in _flush
    flush_context.execute()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
    rec.execute(self)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
    uow
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
    mapper, table, insert)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 835, in _emit_insert_statements
    execute(statement, params)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
    context)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
    exc_info
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: ResponseTable.text, ResponseTable.text_search [SQL: 'INSERT INTO "ResponseTable" (id, text, occurrence, statement_text, text_search) VALUES (?, ?, ?, ?, ?)'] [parameters: (None, '矫情爹夹的女人我要悄悄对你说:好想一脚踹死你[可爱]', 1, '同感!', '{"id": null, "text": "\\u77eb\\u60c5\\u7239\\u5939\\u7684\\u5973\\u4eba\\u6211\\u8981\\u6084\\u6084\\u5bf9\\u4f60\\u8bf4:\\u597d\\u60f3\\u4e00\\u811a\\u8e39\\u6b7b\\u4f60[\\u53ef\\u7231]", "occurrence": 1, "statement_text": "\\u540c\\u611f!"}')]
None

[vkosuri]

@WildDylan Which version of chatterbot are you using, I believe this error fixed in https://github.com/gunthercox/ChatterBot/pull/850,

Could you please try to upgrade chatterbot to latest version to resolve this issue.

pip install --upgrade chatterbot

[wilddylan]

@vkosuri Thx, I upgrade it Fixed that problem.
another question I need help: how to speed up the get_response when I have 1 million corpus in my database

[vkosuri]

There any many other workarounds you could do with this great project. Thanks to Master @gunthercox made such a beautiful project.

You could consider this is a workaround, There are much more improvements on training and retrieval responses are in pipeline in upcoming release.

Coming to speed up, you could write your own training class with list comprehension example are located in this PR https://github.com/gunthercox/ChatterBot/pull/823

The reason this PR not consider into account due to lack of readability

[wilddylan]

Okey, I will give a try.

[Hemanth2396]

thanks for providing the code for the chatter bot ,i have two questions .
one is regarding can NLP be included for the coding to give the appropriate response for the unknown input data without training?
second one is regarding the to get a response we must have to add response of unknown question or else is there any possibility to get GOOD RESPONSE for the unknown input.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.