Keras: Calling model.predict inside the generator that feeds model.fit throws ValueError: [...] is not an element of this graph

[lucasdavid]

I found this issue while trying to implement the triplet loss from FaceNet: A Unified Embedding for Face Recognition and Clustering.

In the paper, input triplets (a, p, n) are selected from the batch s.t. they minimize ||f(n)-f(a)||, where f(x) is the network transformation of the sample x. Therefore, I need to call model.predict() inside the generator that feeds model.fit(), which throws the error:

ValueError: Tensor Tensor("Relu_1:0", shape=(?, 2048), dtype=float32) is not an element of this graph.

Link to gist that reproduces the error (you can also confirm this error goes away when not calling predict by commenting lines 59-60 and uncommenting 61-62): https://gist.github.com/lucasdavid/4cd49f68e5a4002a2611c18c876a1998

[unrealwill]

Hello,
Probably unrelated but I think there is a bug
line 21 should be :
K.expand_dims(_x[2])

[lucasdavid]

That's true, thanks for the catch. Still, the reported problem persists.

Note: the easy explanation on why this is unrelated is: the triplets were supposed to be (anchor, positive, negative), but with K.expand_dims(_x[1]) I was ending up with (anchor, positive, positive), which is indeed a logical problem, but not what I'm complaining with this issue (an exception).

[EderSantana]

I got that same problem yeah. I just gave up using tensorflow models inside generators :)
also, a solution I'd suggestion for your triplet loss is to create a new cost function, calculate all pairwise distances inside and match triplets afterwards using sorting (which I think you can do using tf.nn.top_k). This way you avoid evaluating your model twice.
If you ever write that, let me know how it goes

[lucasdavid]

@EderSantana bummer. When I think I had a reasonably pretty solution it turns out it doesn't work. Anyways, I'll try your way out, cheers!

[perone]

Also facing this same issue when calling model.predict() inside a generator, it seems to be something related to the threads being used for data augmentation. Does anyone found any workaround for it ?

[manojsukhavasi]

@lucasdavid I am also facing the same issue. Did you find the fix for this?

Edit: Found the fix for this issue. The model which was being used inside the generator will run in a different thread than the one in which it was declared. Following steps fixed it for me.

```model = Model(inp,outp)
model._make_predict_function()
graph = tf.get_default_graph()
model.compile()

Using the below code when calling model.predict() :

with graph.as_default():
preds = model.predict()

[dschwertfeger]

I came across this post recently of someone who had the same issue.

He found that calling model.predict(x) once before you call model.fit_generator(…) solves this issue.

Not clean but seems to work.

[socathie]

use model.fit_generator(..., workers=0)

[jliebana]

I faced the same problem, however the @socathie workaround worked for me. Thank you!

[collinsimon]

use model.fit_generator(..., workers=0)

Works here too. CPU used at 50% (instead of 100%) - GPU 50% (like always) - I guess the drop in CPU usage is due to the fact that it feeds slower the model but the bottle neck is most probably the predictions in the generator anyway

edit : I think a better way to do it if it is possible in the application would be to make the predictions before hand and store them on the disk.

Thanks Cathie!

[Barnonewdm]

@socathie I have tried your skill and successfully get results. Could you explain the reason behind this?

[dbulhosa]

@Barnonewdm I believe when one sets workers=0 both the generator and the model run in the same thread. When we set workers=1 they run in separate threads, as @manojsukhavasi mentioned. When the model and generator are in the same thread presumably they are also on the same graph by default.

Using the solution suggested by @manojsukhavasi with Keras version 2.2.4 worked for me. Great solution because the alternative is setting workers to zero and then your sequence can't prepare mini-batches in parallel with model backpropagation.