Cntk: Evaluate models in Python Docs Example leads to ValueError

[hellosaumil]

I am running CNTK v2.3 on Docker macOS. And I wanted to evaluate ResNet-18 based on instructions from How do I evaluate models in Python.

I'm struggling to replicate results as per the Evaluate a saved convolutional network from the above docs link.

My code is as follows: (exactly as described in above webpage)

from cntk.ops.functions import load_model
from PIL import Image 
import numpy as np

z = load_model("ResNet18_ImageNet_CNTK.model")
rgb_image = np.asarray(Image.open("temp.jpg"), dtype=np.float32) - 128
bgr_image = rgb_image[..., [2, 1, 0]]
pic = np.ascontiguousarray(np.rollaxis(bgr_image, 2))

predictions = np.squeeze(z.eval({z.arguments[0]:[pic]}))          # ERROR OCCURS HERE
top_class = np.argmax(predictions)

I get following error:
ValueError: Values for 1 required arguments 'Input('labels', [#, ], [1000])', that the requested output(s) 'Output('ce', [], []), Output('errs', [], []), Output('top5Errs', [], []), Output('z', [#, ], [1000])' depend on, have not been provided.

What am I missing here? Please help.
@KeDengMS @cha-zhang @sayanpa @tangyuq @spandantiwari @dongyu888 @frankibem
@chrisbasoglu @wolfma61 @mahilleb-msft @Maggsl @rotycenh

Thank You.

PS - temp.jpg is 224x224 - RGB

[hellosaumil]

Full error is as follows:

ValueError                                Traceback (most recent call last)
<ipython-input-50-8ac533b2a908> in <module>()
      8 pic = np.ascontiguousarray(np.rollaxis(bgr_image, 2))
      9 
---> 10 predictions = np.squeeze(z.eval({z.arguments[0]:[pic]}))
     11 top_class = np.argmax(predictions)

~/anaconda3/envs/cntk-py35/lib/python3.5/site-packages/cntk/ops/functions.py in eval(self, arguments, outputs, device, as_numpy)
    685             outputs = self.outputs
    686 
--> 687         _, output_map = self.forward(arguments, outputs, device=device, as_numpy=as_numpy)
    688         return sanitize_variable_value_dict(output_map)
    689 

~/anaconda3/envs/cntk-py35/lib/python3.5/site-packages/cntk/internal/swig_helper.py in wrapper(*args, **kwds)
     67     @wraps(f)
     68     def wrapper(*args, **kwds):
---> 69         result = f(*args, **kwds)
     70         map_if_possible(result)
     71         return result

~/anaconda3/envs/cntk-py35/lib/python3.5/site-packages/cntk/ops/functions.py in forward(self, arguments, outputs, keep_for_backward, device, as_numpy)
    819 
    820         state = super(Function, self)._forward(in_var_map, output_map, device,
--> 821                                                keep_for_backward)
    822         if as_numpy:
    823             for k, v in output_map.items():

~/anaconda3/envs/cntk-py35/lib/python3.5/site-packages/cntk/cntk_py.py in _forward(self, *args)
   1890 
   1891     def _forward(self, *args):
-> 1892         return _cntk_py.Function__forward(self, *args)
   1893 
   1894     def _infer_outputs(self, outputs):

ValueError: Values for 1 required arguments 'Input('labels', [#, ], [1000])', that the requested output(s) 'Output('ce', [], []), Output('errs', [], []), Output('top5Errs', [], []), Output('z', [#, ], [1000])' depend on, have not been provided.

When I replace the lines based on a stackoverflow answer

z_out = cntk.combine( z.find_all_with_name('z.x') )
predictions = np.squeeze(z_out.eval({z_out.arguments[0]:[pic]}))
top_class = np.argmax(predictions)
print (top_class)

I get output as 261. Can anybody explain this result?

[KeDengMS]

This is because your loaded model is a combine:

Composite(Combine): Input('features', [#, ], [3 x 224 x 224]), Input('labels', [#, ], [1000]) -> Output('ce', [], []), Output('errs', [], []), Output('top5Errs', [], []), Output('z', [#, ], [1000])

In evaluation of ce/err/top5Errs, you need to provide label. If that's not needed, you can evaluate only z:

import cntk as C
combined = C.load_model(r'x:\ResNet18_ImageNet_CNTK.model')
z = C.as_composite(combined[3].owner)

Here, combined[3] means taking the output variable z, and then get it's owner function (which is a primitive function, which only shows immediate input) and wrap it as composite function. Then you'll get z without the need to provide label as input:

Composite(Plus): Input('features', [#, ], [3 x 224 x 224]) -> Output('z', [#, ], [1000])

[hellosaumil]

Thank you very much @KeDengMS !!!

Besides, can you direct me a resource where I can find labels or class names of ResNet?

I found a file Labels of ResNet...are these valid?

Thanks again.

[KeDengMS]

It has nothing to do with ResNet, but rather semantics of ImageNet label. I found a link here that might be useful.

[hellosaumil]

@KeDengMS Thank you for your help and resourcefulness! 😄

[mx-iao]

@hellosaumil FYI here is a pickled dictionary mapping ImageNet class IDs to the labels: https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a

[hellosaumil]

Thanks @mx-iao. I also created a json file for ImageNet objects https://gist.github.com/hellosaumil/e1d9cb933cb0511ed6d2ac0a5cb78f37