Rasa: TypeError: Cannot interpret feed_dict key as Tensor: Tensor Tensor("masking_1_input:0", shape=(?, 3, 24), dtype=float32) is not an element of this graph.

[ray-man]

Rasa Core version : 0.7.0

Operating system: OSX 10.11.6

Issue:
Using rasa_core as part of Django app while requesting for training to be done with provided data everything goes smooth but if i try to re train without changing anything from data or any any other related file am getting TypeError: Cannot interpret feed_dict key as Tensor: Tensor Tensor("masking_1_input:0", shape=(?, 3, 24), dtype=float32) is not an element of this graph.

Traceback:

Internal Server Error: /3/process_training/
Traceback (most recent call last):
File "/Users/rayman/virtualenvs/RT/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 930, in _run
    allow_operation=False)
  File "/Users/rayman/virtualenvs/RT/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2414, in as_graph_element
    return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "/Users/rayman/virtualenvs/RT/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2493, in _as_graph_element_locked
    raise ValueError("Tensor %s is not an element of this graph." % obj)
ValueError: Tensor Tensor("masking_1_input:0", shape=(?, 3, 24), dtype=float32) is not an element of this graph.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/rayman/virtualenvs/RT/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/Users/rayman/virtualenvs/RT/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/Users/rayman/virtualenvs/RT/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/rayman/Sites/rasa_trainer/thumbcrowdai/webapps/version_3.0/core/rasa_trainer/views.py", line 407, in process_training
    train_dm.train()
  File "/Users/rayman/Sites/rasa_trainer/thumbcrowdai/webapps/version_3.0/modules/rasa_dm/bots/weather/train_dm.py", line 28, in train
    validation_split=0.2
  File "/Users/rayman/virtualenvs/RT/lib/python3.6/site-packages/rasa_dm-0.7.0a1-py3.6.egg/rasa_dm/agent.py", line 101, in train
    trainer.train(filename, **kwargs)
  File "/Users/rayman/virtualenvs/RT/lib/python3.6/site-packages/rasa_dm-0.7.0a1-py3.6.egg/rasa_dm/policies/trainer.py", line 44, in train
    self.ensemble.train(X, y, self.domain, self.featurizer, **kwargs)
  File "/Users/rayman/virtualenvs/RT/lib/python3.6/site-packages/rasa_dm-0.7.0a1-py3.6.egg/rasa_dm/policies/ensemble.py", line 29, in train
    policy.train(X, y, domain, **kwargs)
  File "/Users/rayman/virtualenvs/RT/lib/python3.6/site-packages/rasa_dm-0.7.0a1-py3.6.egg/rasa_dm/policies/keras_policy.py", line 92, in train
    self.model.fit(shuffled_X, shuffled_y, **kwargs)
  File "/Users/rayman/virtualenvs/RT/lib/python3.6/site-packages/keras/models.py", line 856, in fit
    initial_epoch=initial_epoch)
  File "/Users/rayman/virtualenvs/RT/lib/python3.6/site-packages/keras/engine/training.py", line 1498, in fit
    initial_epoch=initial_epoch)
  File "/Users/rayman/virtualenvs/RT/lib/python3.6/site-packages/keras/engine/training.py", line 1152, in _fit_loop
    outs = f(ins_batch)
  File "/Users/rayman/virtualenvs/RT/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2229, in __call__
    feed_dict=feed_dict)
  File "/Users/rayman/virtualenvs/RT/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 778, in run
    run_metadata_ptr)
  File "/Users/rayman/virtualenvs/RT/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 933, in _run
    + e.args[0])
TypeError: Cannot interpret feed_dict key as Tensor: Tensor Tensor("masking_1_input:0", shape=(?, 3, 24), dtype=float32) is not an element of this graph.
[08/Nov/2017 09:03:30] "GET /3/process_training/ HTTP/1.1" 500 175964

P.S : Raising issue in rasa_core after closing in rasa_nlu because it felt more relevant to core.

[ray-man]

@npx @benshippee @normanrz @tmbo @eliseboyd guys any help would be appreciated.

[tmbo]

@ray-man you specified that you are using 0.7.0 mind trying the latest release (0.7.4)? Please let me know how that goes and please also note your tensorflow / keras version.

[ray-man]

@tmbo I've tried using latest 0.7.4 no luck with it too, I was on Tensor-flow 1.1.0 which i tried upgrading to 1.4.0 and Keras 2.0.8 to 2.1.0 but no success...

[tmbo]

Using 1.10 locally as well. This sounds a bit like an issue with the tensorflow session. Is that using multiple processes?

[ray-man]

@tmbo No it is not using multiple processes, and i have tried clearing any existing session through keras.backend.clear_session() but no luck so far.

[tmbo]

I tried to reproduce this but I can't.

Can you provide a list of the installed python dependencies so I can have a look if something looks off the usual setup?

[ray-man]

Here is a list of all dependencies installed

alabaster==0.7.10
apipkg==1.4
appnope==0.1.0
APScheduler==3.3.1
astroid==1.5.3
attrs==17.2.0
Automat==0.6.0
Babel==2.5.1
backports.functools-lru-cache==1.4
backports.shutil-get-terminal-size==1.0.0
bleach==2.1.1
boto3==1.4.7
botocore==1.7.36
certifi==2017.7.27.1
chardet==3.0.4
click==6.7
cloudpickle==0.4.1
ConfigArgParse==0.12.0
configparser==3.5.0
constantly==15.1.0
coverage==4.4.1
cycler==0.10.0
cymem==1.31.2
cytoolz==0.8.2
decorator==4.1.2
dill==0.2.7.1
Django==1.11.6
django-cors-headers==2.1.0
djangorestframework==3.7.3
docopt==0.6.2
docutils==0.14
en-core-web-sm==1.2.0
entrypoints==0.2.3
enum34==1.1.6
execnet==1.5.0
fakeredis==0.8.2
first==2.0.1
Flask==0.12.2
foursquare==1!2016.9.12
ftfy==4.4.3
future==0.16.0
geopy==1.11.0
gevent==1.2.2
graphviz==0.7.1
greenlet==0.4.12
h5py==2.7.0
html5lib==1.0b10
hyperlink==17.3.1
idna==2.6
imagesize==0.7.1
incremental==17.5.0
inflect==0.2.5
ipykernel==4.6.1
ipython==5.5.0
ipython-genutils==0.2.0
ipywidgets==7.0.3
isort==4.2.15
itsdangerous==0.24
jaraco.itertools==2.0.1
Jinja2==2.9.6
jmespath==0.9.3
jsonfield==2.0.2
jsonpickle==0.9.4
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.1.0
jupyter-console==5.2.0
jupyter-core==4.3.0
Keras==2.0.4
klein==17.10.0
lazy-object-proxy==1.3.1
Markdown==2.6.9
MarkupSafe==1.0
matplotlib==2.1.0
mccabe==0.6.1
mistune==0.7.4
more-itertools==3.2.0
murmurhash==0.26.4
nbconvert==5.3.1
nbformat==4.4.0
nbsphinx==0.2.13
networkx==1.11
notebook==5.2.0
numpy==1.13.3
pandoc==1.0.0b2
pandocfilters==1.4.2
pathlib==1.0.1
pathlib2==2.3.0
pbr==2.0.0
pep8==1.7.1
pexpect==4.2.1
pickleshare==0.7.4
pip-tools==1.9.0
plac==0.9.6
ply==3.10
preshed==1.0.0
prompt-toolkit==1.0.15
protobuf==3.4.0
psutil==5.4.0
psycopg2==2.7.3.2
ptyprocess==0.5.2
py==1.4.34
Pygments==2.2.0
pykwalify==1.6.0
pylint==1.7.2
pymessenger==0.0.7.0
pyparsing==2.2.0
pytest==3.2.3
pytest-cache==1.0
pytest-cov==2.5.1
pytest-flask==0.10.0
pytest-pep8==1.0.6
pytest-services==1.2.1
pytest-xdist==1.15.0
python-crfsuite==0.9.5
python-dateutil==2.6.1
pytz==2017.3
PyYAML==3.12
pyzmq==16.0.2
qtconsole==4.3.1
rasa-dm==0.7.0a1
rasa-nlu==0.9.0
redis==2.10.5
regex==2017.9.23
requests==2.18.4
requests-toolbelt==0.8.0
s3transfer==0.1.11
scandir==1.6
scikit-learn==0.19.1
scipy==1.0.0
simplegeneric==0.8.1
simplejson==3.11.1
singledispatch==3.4.0.3
six==1.11.0
sklearn-crfsuite==0.3.6
snowballstemmer==1.2.1
spacy==1.9.0
Sphinx==1.6.5
sphinxcontrib-websupport==1.0.1
stevedore==1.21.0
tabulate==0.8.1
tensorflow==1.1.0
tensorflow-tensorboard==0.4.0rc2
termcolor==1.1.0
terminado==0.6
testpath==0.3.1
Theano==0.9.0
thinc==6.5.2
toolz==0.8.2
tornado==4.5.2
tqdm==4.19.4
traitlets==4.3.2
Twisted==17.9.0
typing==3.6.2
tzlocal==1.4
ujson==1.35
urllib3==1.22
virtualenv==15.1.0
virtualenv-clone==0.2.6
virtualenvwrapper==4.7.2
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.12.2
widgetsnbextension==3.0.6
wolframalpha==3.0
wrapt==1.10.11
xmltodict==0.11.0
yahooweather==0.8
yweather==0.1.1
zope.interface==4.4.3`

[aidanSoles]

Same issue here. I am up to the keras.backend.clear_session() step as well with no luck. I can provide dependencies also if needed.

[aidanSoles]

@ray-man : I'm not sure how related our issues our (mine was for model.load_weights() and not training) but one solution that worked for me was to switch from a TensorFlow backend to a Theano backend (directions here: https://keras.io/backend/). I'm also unsure of whether this is a Django issue or a TensorFlow issue (who's at fault). Hopefully this helps.

[bitraten]

Having the same issue with rasa_core 0.7.9 and tensorflow 1.4.1, without Django being involved.

[tmbo]

@bitraten can you try with the latest rasa_core (0.8.0) - I think the referenced tensorflow is 1.1.0 though

[tmbo]

but should work with 1.4.1 as well

[bitraten]

@tmbo Error still appears for me with rasa_core 0.8.1 and tensorflow 1.1.0 as well as 1.4.1.

[tmbo]

I'd love to help on this one, but I couldn't reproduce this yet.

@bitraten which command are you running?

[simaoluizjr]

I'm trying to use Django and Keras (not using RasaHQ) and have exactly the same issue. If I restart django server, the first training goes ok. But when try to retrain even creating a new empty keras model this error appears.

Although if I access the same method using manage.py shell it works perfectly.

Weird.

[tmbo]

Can you post a minimal non working django app? That would be great and i could use it for debugging.

[Tyell14]

I'm having the same issue with Keras and Tensorflow. Although I'm using Flask instead of Django. I expose a training model service: the first request goes smoothly, but if I send another request I get the "Cannot interpret feed_dict key as Tensor" error.

EDIT: seems that the problem is related to threading mechanism. If I run the app single-threaded (threaded=False in flask), subsequent requestes work as expected

[PvsManiyan]

hi, I have experienced the same error if we apply following code to release the model session from memory keras.backend.clear_session() and you can load it again to get the model back on the consecutive request.

[jdhurwitz]

same issue

[PvsManiyan]

u can solve the issue by doing the above command

[ray-man]

@PvsManiyan as mentioned above it doesn't help.

[oscalarr]

same issue

Possible solution?
https://stackoverflow.com/questions/40785224/tensorflow-cannot-interpret-feed-dict-key-as-tensor

[mohammedyunus009]

I was using flask,
This worked for me

from keras import backend as K
and after predicting my data i inserted this part of code then i had again loaded the model.

K.clear_session()

[tmbo]

Yes so the issue here is, that tensorflow doesn't like it if the graph is created in a different thread and not set properly. I'll take a look at this, but clearing the session might do the trick as well :+1:

[Peresh]

I got the same error when I upgraded my Flask from 0.12.0 to 1.0.2. It's strange.
But keras.backend.clear_session() works for me.

[bibhu107]

same error!
TypeError: Cannot interpret feed_dict key as Tensor: Tensor Tensor("inputTensor:0", shape=(?, 88), dtype=int32) is not an element of this graph.

[asharsid]

same error...can someone help out please..
Tensor Tensor("Placeholder:0", shape=(3, 3, 3, 64), dtype=float32) is not an element of this graph.

App runs fine for the first time, but second run and it gives this error.
I have tried default graph and clear session, nothing working for me.

Help please

[bibhu107]

@asharsid check the input size you are providing in your app script
classifier.inputSize = __;

[itachi0071998]

I am getting same error

TypeError: Cannot interpret feed_dict key as Tensor: Tensor Tensor("Placeholder:0", shape=(7, 7, 3, 64), dtype=float32) is not an element of this graph.

I am using socket and multiple threading and it gives good result to first client connected only for every another client it throws this error

[asharsid]

@bibhu107 .. Hi Bibhu, I forgot to mention that I am using Keras on top of tensorflow...and passing images for CNN, for which I am fixing the input size. However, I was able to fix my problem using your answer as a hint :)
@aggarwalaakash007 .. In my case, I was using loop while calling in CNN multiple times, I fixed my problem by doing the following:

Declare this as global:

global graph
graph = tf.get_default_graph()

Then just before you call in your model, use this

with graph.as_default():
#call you models here

Note: In my case too, the app ran fine for the first time and then gave the error above. Using the above fix solved the problem.

Hope that helps.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[oscalarr]

Estaré ausente de la oficina hasta el 19/09/2018.

Responderé a su mensaje cuando regrese.

Nota: Esta es una respuesta automática a su mensaje "[EXT] Re:
[RasaHQ/rasa_core] TypeError: Cannot interpret feed_dict key as Tensor:
Tensor Tensor("masking_1_input:0", shape=(?, 3, 24), dtype=float32) is not
an element of this graph. (#80)" enviado el: 18/09/2018 8:17:46.

Esta es la única notificación que recibirá mientras esta persona está
ausente.

[afghani-iitkgp]

I'm having the same issue with Keras and Tensorflow. Although I'm using Flask instead of Django. I expose a training model service: the first request goes smoothly, but if I send another request I get the "Cannot interpret feed_dict key as Tensor" error.

EDIT: seems that the problem is related to threading mechanism. If I run the app single-threaded (threaded=False in flask), subsequent requests work as expected

It occurred same with me too. May I know the reasons?
what threaded=False does here?