Rasa: HTTP server throws "ValueError: bad value(s) in fds_to_keep"

[parthsharma1996]

Rasa NLU version: 0.13.5

Operating system (windows, osx, ...): Arch Linux

Content of model configuration file:

language: "en"

pipeline:
- name: "tokenizer_whitespace"
- name: "ner_crf"
- name: "ner_duckling"
  dimensions: ["time"]
- name: "intent_featurizer_count_vectors"
- name: "intent_classifier_tensorflow_embedding"
  intent_tokenization_flag: true
  intent_split_symbol: "+"

Issue:
I recently made a fresh clone of my repository on a new machine and ran rasa_nlu.server on it.
When I send any parse request to the server, it seems to throw an error "bad value(s) in fds_to_keep" (full trace below)

(rasa_env) ❯ curl -XPOST localhost:5000/parse -d '{"q":"hello there", "project": "mood_tracker_nlu", "model": "mood_tracker_nlu"}'
2018-09-30 02:33:28+0530 [-] 2018-09-30 02:33:28 ERROR    __main__ - bad value(s) in fds_to_keep
2018-09-30 02:33:28+0530 [-] Traceback (most recent call last):
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/rasa_nlu/server.py", line 245, in parse
2018-09-30 02:33:28+0530 [-]     self.data_router.parse, data))
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/twisted/python/threadpool.py", line 250, in inContext
2018-09-30 02:33:28+0530 [-]     result = inContext.theWork()
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
2018-09-30 02:33:28+0530 [-]     inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
2018-09-30 02:33:28+0530 [-]     return self.currentContext().callWithContext(ctx, func, *args, **kw)
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
2018-09-30 02:33:28+0530 [-]     return func(*args,**kw)
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/rasa_nlu/data_router.py", line 264, in parse
2018-09-30 02:33:28+0530 [-]     model)
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/rasa_nlu/project.py", line 261, in parse
2018-09-30 02:33:28+0530 [-]     interpreter = self._interpreter_for_model(model_name)
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/rasa_nlu/project.py", line 367, in _interpreter_for_model
2018-09-30 02:33:28+0530 [-]     return Interpreter.create(metadata, self._component_builder)
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/rasa_nlu/model.py", line 315, in create
2018-09-30 02:33:28+0530 [-]     components.validate_requirements(model_metadata.component_classes)
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/rasa_nlu/components.py", line 56, in validate_requirements
2018-09-30 02:33:28+0530 [-]     component_class.required_packages()))
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/rasa_nlu/components.py", line 39, in find_unavailable_packages
2018-09-30 02:33:28+0530 [-]     importlib.import_module(package)
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/importlib/__init__.py", line 126, in import_module
2018-09-30 02:33:28+0530 [-]     return _bootstrap._gcd_import(name[level:], package, level)
2018-09-30 02:33:28+0530 [-]   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
2018-09-30 02:33:28+0530 [-]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2018-09-30 02:33:28+0530 [-]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2018-09-30 02:33:28+0530 [-]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
2018-09-30 02:33:28+0530 [-]   File "<frozen importlib._bootstrap_external>", line 678, in exec_module
2018-09-30 02:33:28+0530 [-]   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/sklearn_crfsuite/__init__.py", line 2, in <module>
2018-09-30 02:33:28+0530 [-]     from .estimator import CRF
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/sklearn_crfsuite/estimator.py", line 10, in <module>
2018-09-30 02:33:28+0530 [-]     from sklearn_crfsuite.compat import BaseEstimator
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/sklearn_crfsuite/compat.py", line 3, in <module>
2018-09-30 02:33:28+0530 [-]     from sklearn.base import BaseEstimator
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/sklearn/__init__.py", line 64, in <module>
2018-09-30 02:33:28+0530 [-]     from .base import clone
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/sklearn/base.py", line 13, in <module>
2018-09-30 02:33:28+0530 [-]     from .utils.fixes import signature
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/sklearn/utils/__init__.py", line 13, in <module>
2018-09-30 02:33:28+0530 [-]     from .validation import (as_float_array,
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/sklearn/utils/validation.py", line 27, in <module>
2018-09-30 02:33:28+0530 [-]     from ..utils._joblib import Memory
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/sklearn/utils/_joblib.py", line 18, in <module>
2018-09-30 02:33:28+0530 [-]     from ..externals.joblib import __all__   # noqa
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/sklearn/externals/joblib/__init__.py", line 119, in <module>
2018-09-30 02:33:28+0530 [-]     from .parallel import Parallel
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py", line 22, in <module>
2018-09-30 02:33:28+0530 [-]     from ._multiprocessing_helpers import mp
2018-09-30 02:33:28+0530 [-]   File "/home/parth/rasa_env/lib/python3.6/site-packages/sklearn/externals/joblib/_multiprocessing_helpers.py", line 34, in <module>
2018-09-30 02:33:28+0530 [-]     _sem = Semaphore()
2018-09-30 02:33:28+0530 [-]   File "/usr/lib64/python3.6/multiprocessing/context.py", line 82, in Semaphore
2018-09-30 02:33:28+0530 [-]     return Semaphore(value, ctx=self.get_context())
2018-09-30 02:33:28+0530 [-]   File "/usr/lib64/python3.6/multiprocessing/synchronize.py", line 127, in __init__
2018-09-30 02:33:28+0530 [-]     SemLock.__init__(self, SEMAPHORE, value, SEM_VALUE_MAX, ctx=ctx)
2018-09-30 02:33:28+0530 [-]   File "/usr/lib64/python3.6/multiprocessing/synchronize.py", line 81, in __init__
2018-09-30 02:33:28+0530 [-]     register(self._semlock.name)
2018-09-30 02:33:28+0530 [-]   File "/usr/lib64/python3.6/multiprocessing/semaphore_tracker.py", line 83, in register
2018-09-30 02:33:28+0530 [-]     self._send('REGISTER', name)
2018-09-30 02:33:28+0530 [-]   File "/usr/lib64/python3.6/multiprocessing/semaphore_tracker.py", line 90, in _send
2018-09-30 02:33:28+0530 [-]     self.ensure_running()
2018-09-30 02:33:28+0530 [-]   File "/usr/lib64/python3.6/multiprocessing/semaphore_tracker.py", line 71, in ensure_running
2018-09-30 02:33:28+0530 [-]     pid = util.spawnv_passfds(exe, args, fds_to_pass)
2018-09-30 02:33:28+0530 [-]   File "/usr/lib64/python3.6/multiprocessing/util.py", line 417, in spawnv_passfds
2018-09-30 02:33:28+0530 [-]     False, False, None)
2018-09-30 02:33:28+0530 [-] ValueError: bad value(s) in fds_to_keep
2018-09-30 02:33:28+0530 [-] "127.0.0.1" - - [29/Sep/2018:21:03:28 +0000] "POST /parse HTTP/1.1" 500 44 "-" "curl/7.61.1"
{
  "error": "bad value(s) in fds_to_keep"
}%                               

I have already done the following things:-

1. Re-installed rasa_nlu and re-trained the models.
2. Re-installed python and re-created the virtual environment.
3. Tried using the python API (using the Interpreter object). It seems to work perfectly fine in python. Therefore, the problem seems to be related to the HTTP server.
4. Tried running the server command with sudo, just in case there was a permissions problem (saw some calls to files in /usr/lib64/python3.6 folder). Same error.

Can anyone help me out with this? I can't understand what's causing this and how to fix it.

P.S: The calls being made to files in /usr/lib64/python3.6/multiprocessing caught my eye. I was under the impression that a virtualenv used a completely different python installation kept in the virtual environment folder.

[akelad]

Thanks for raising this issue, @ricwo will get back to you about it soon.

[MikiGrit]

Hi, I've received the same error.

When I requested localhost:5000/parse?q=who is john

The docker container returned

2018-10-01 13:52:59+0000 [-] Log opened.
2018-10-01 13:52:59+0000 [-] Site starting on 5000
2018-10-01 13:52:59+0000 [-] Starting factory <twisted.web.server.Site object at 0x7f87163dff28>
2018-10-01 13:53:53+0000 [-] 2018-10-01 13:53:53 ERROR    __main__  - bad value(s) in fds_to_keep
2018-10-01 13:53:53+0000 [-] Traceback (most recent call last):
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/rasa_nlu/server.py", line 245, in parse
2018-10-01 13:53:53+0000 [-]     self.data_router.parse, data))
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 250, in inContext
2018-10-01 13:53:53+0000 [-]     result = inContext.theWork()
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
2018-10-01 13:53:53+0000 [-]     inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
2018-10-01 13:53:53+0000 [-]     return self.currentContext().callWithContext(ctx, func, *args, **kw)
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
2018-10-01 13:53:53+0000 [-]     return func(*args,**kw)
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/rasa_nlu/data_router.py", line 264, in parse
2018-10-01 13:53:53+0000 [-]     model)
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/rasa_nlu/project.py", line 261, in parse
2018-10-01 13:53:53+0000 [-]     interpreter = self._interpreter_for_model(model_name)
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/rasa_nlu/project.py", line 367, in _interpreter_for_model
2018-10-01 13:53:53+0000 [-]     return Interpreter.create(metadata, self._component_builder)
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/rasa_nlu/model.py", line 315, in create
2018-10-01 13:53:53+0000 [-]     components.validate_requirements(model_metadata.component_classes)
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/rasa_nlu/components.py", line 56, in validate_requirements
2018-10-01 13:53:53+0000 [-]     component_class.required_packages()))
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/rasa_nlu/components.py", line 39, in find_unavailable_packages
2018-10-01 13:53:53+0000 [-]     importlib.import_module(package)
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
2018-10-01 13:53:53+0000 [-]     return _bootstrap._gcd_import(name[level:], package, level)
2018-10-01 13:53:53+0000 [-]   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
2018-10-01 13:53:53+0000 [-]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2018-10-01 13:53:53+0000 [-]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2018-10-01 13:53:53+0000 [-]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
2018-10-01 13:53:53+0000 [-]   File "<frozen importlib._bootstrap_external>", line 678, in exec_module
2018-10-01 13:53:53+0000 [-]   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/sklearn/__init__.py", line 64, in <module>
2018-10-01 13:53:53+0000 [-]     from .base import clone
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/sklearn/base.py", line 13, in <module>
2018-10-01 13:53:53+0000 [-]     from .utils.fixes import signature
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/sklearn/utils/__init__.py", line 13, in <module>
2018-10-01 13:53:53+0000 [-]     from .validation import (as_float_array,
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/sklearn/utils/validation.py", line 27, in <module>
2018-10-01 13:53:53+0000 [-]     from ..utils._joblib import Memory
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/sklearn/utils/_joblib.py", line 18, in <module>
2018-10-01 13:53:53+0000 [-]     from ..externals.joblib import __all__   # noqa
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/sklearn/externals/joblib/__init__.py", line 119, in <module>
2018-10-01 13:53:53+0000 [-]     from .parallel import Parallel
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py", line 22, in <module>
2018-10-01 13:53:53+0000 [-]     from ._multiprocessing_helpers import mp
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/site-packages/sklearn/externals/joblib/_multiprocessing_helpers.py", line 34, in <module>
2018-10-01 13:53:53+0000 [-]     _sem = Semaphore()
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/multiprocessing/context.py", line 82, in Semaphore
2018-10-01 13:53:53+0000 [-]     return Semaphore(value, ctx=self.get_context())
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/multiprocessing/synchronize.py", line 127, in __init__
2018-10-01 13:53:53+0000 [-]     SemLock.__init__(self, SEMAPHORE, value, SEM_VALUE_MAX, ctx=ctx)
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/multiprocessing/synchronize.py", line 81, in __init__
2018-10-01 13:53:53+0000 [-]     register(self._semlock.name)
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/multiprocessing/semaphore_tracker.py", line 83, in register
2018-10-01 13:53:53+0000 [-]     self._send('REGISTER', name)
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/multiprocessing/semaphore_tracker.py", line 90, in _send
2018-10-01 13:53:53+0000 [-]     self.ensure_running()
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/multiprocessing/semaphore_tracker.py", line 71, in ensure_running
2018-10-01 13:53:53+0000 [-]     pid = util.spawnv_passfds(exe, args, fds_to_pass)
2018-10-01 13:53:53+0000 [-]   File "/usr/local/lib/python3.6/multiprocessing/util.py", line 417, in spawnv_passfds
2018-10-01 13:53:53+0000 [-]     False, False, None)
2018-10-01 13:53:53+0000 [-] ValueError: bad value(s) in fds_to_keep
2018-10-01 13:53:53+0000 [-] "172.21.0.1" - - [01/Oct/2018:13:53:52 +0000] "GET /parse?q=who%20is%20john HTTP/1.1" 500 44 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"

[parthsharma1996]

Hi guys,
So I was able to (sort-of) able to resolve the issue.

There is an issue with the new `sklearn 0.20.01 release (which came out on 26th September) which is causing this error.

Rolling back to 0.19.1 solved the issue for me.

Perhaps someone can inspect the error in detail and fix the bug (if it's in rasa_nlu) or report the issue to scikit-learn if there's some bug there.

[ricwo]

@parthsharma1996 @WestFlame Thanks for pointing out this issue. Please make sure you have scikit-learn==0.19.1 installed, as you suggested. I'm not sure at this point what's causing the problem. The error comes up when scikit-learn is imported inside the Klein http server

[MikiGrit]

Well, unfortunately, the 0.19.1 version was already installed in my environment when the issue occured.

[b1xian]

Hi guys,
So I was able to (sort-of) able to resolve the issue.

There is an issue with the new `sklearn 0.20.01 release (which came out on 26th September) which is causing this error.

Rolling back to 0.19.1 solved the issue for me.

Perhaps someone can inspect the error in detail and fix the bug (if it's in rasa_nlu) or report the issue to scikit-learn if there's some bug there.

thanks,it does works

[akelad]

Since the original issue seems to have been resolved, I'll close this

[yangh1368]

sudo pip3 install scikit-learn==0.19.2 (the version number should be check on the official website),
and retrain you model with python3 -m rasa_nlu.train ...