Ludwig: Cannot feed value of shape for Tensor when predicting model with image

[abogoyavlensky]

I get this error when trying to perform prediction with a new image on trained model:

Traceback (most recent call last):
  File "/home/andrey/.venvs/ludwig-learn/bin/ludwig", line 11, in <module>
    sys.exit(main())
  File "/home/andrey/.venvs/ludwig-learn/lib/python3.6/site-packages/ludwig/cli.py", line 86, in main
    CLI()
  File "/home/andrey/.venvs/ludwig-learn/lib/python3.6/site-packages/ludwig/cli.py", line 64, in __init__
    getattr(self, args.command)()
  File "/home/andrey/.venvs/ludwig-learn/lib/python3.6/site-packages/ludwig/cli.py", line 73, in predict
    predict.cli(sys.argv[2:])
  File "/home/andrey/.venvs/ludwig-learn/lib/python3.6/site-packages/ludwig/predict.py", line 379, in cli
    full_predict(**vars(args))
  File "/home/andrey/.venvs/ludwig-learn/lib/python3.6/site-packages/ludwig/predict.py", line 104, in full_predict
    debug
  File "/home/andrey/.venvs/ludwig-learn/lib/python3.6/site-packages/ludwig/predict.py", line 173, in predict
    gpu_fraction=gpu_fraction
  File "/home/andrey/.venvs/ludwig-learn/lib/python3.6/site-packages/ludwig/models/model.py", line 1182, in predict
    only_predictions=only_predictions
  File "/home/andrey/.venvs/ludwig-learn/lib/python3.6/site-packages/ludwig/models/model.py", line 756, in batch_evaluation
    is_training=is_training
  File "/home/andrey/.venvs/ludwig-learn/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/andrey/.venvs/ludwig-learn/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1128, in _run
    str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (1,) for Tensor 'image_path/image_path:0', which has shape '(?, 100, 100, 3)'

The model definition I use:

input_features:
    -
        name: image_path
        type: image
        encoder: stacked_cnn
        in_memory: false
        height: 100
        width: 100

output_features:
    -
        name: tags
        type: set`

Example of testing csv-file:

image_path,tags
testdata/cat.101.jpg

I've tried output features with category and set types but eventually get that error.

[w4nderlust]

Can you past the command you are using for prediction?
Are you using the --only_predict option?
Can you try removing the tags column as it is empty?
Thanks

[us]

maybe check the input somewhere and added this np.expand_dims(img, axis=0)

[abogoyavlensky]

@w4nderlust I use command from the example:

ludwig predict \
  --data_csv test.csv \
  --model_path results/experiment_run_0/model

--only_predictions option produces the same error.

About the tags column is a bit more interesting. If I remove the tags column in the test.csv and use the command without --only_predictions option the following error appears:

Loading metadata from: results/experiment_run_0/model/train_set_metadata.json
Traceback (most recent call last):
  File "/home/andrey/.venvs/ludwig-learn/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2656, in get_loc
    return self._engine.get_loc(key)
  File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1601, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1608, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'tags'

If I add --only_predictions option

ludwig predict \
  --only_predictions \
  --data_csv test.csv \
  --model_path results/experiment_run_0/model

then I see the same error from the issue.
Thank you for the fast reply!

[crocki134]

Just a wild guess:

You feed uncropped/padded images into predict, but your model is trained on 100x100 size.
Predict is not resizing or in your case cropping/padding the images.

Edit: Same issue for me at the moment and that's what came to my mind

[abogoyavlensky]

@crocki134 After a couple attempts with uncropped image I tried to crop image exactly 100x100 size but the result was the same.

[allenkao]

Same issue for me. But when I predicted using cached hdf5 of original training data, it works without problems. Any new CSV files w/wo labels were still NOT working for prediction

[w4nderlust]

Got it, it may be the case that I'm not correctly applying the preprocessing to image features in prediction. Will look into it. @ydudin3 can you also try to check this out?

[ydudin3]

@allenkao - not able to reproduce, are you providing the --only_predictions flag?

[w4nderlust]

@abogoyavlensky this PR #141 should have solved the issue. Can you please doublecheck? (remove the hdf5 and json files from your data dir before please).

[abogoyavlensky]

@w4nderlust Sorry for the delay but I think that I probably do something wrong and I still can't get a prediction. I've completely cleaned dir from old computed data. All input data is the same as previous. Then retrained model with new code from PR. Next, when I try to predict I receive the error:

Traceback (most recent call last):
  File "/home/andrey/.venvs/ludwig-learn-pr/bin/ludwig", line 11, in <module>
    load_entry_point('ludwig', 'console_scripts', 'ludwig')()
  File "/home/andrey/Projects/ludwig-learn-pr/ludwig/ludwig/cli.py", line 86, in main
    CLI()
  File "/home/andrey/Projects/ludwig-learn-pr/ludwig/ludwig/cli.py", line 64, in __init__
    getattr(self, args.command)()
  File "/home/andrey/Projects/ludwig-learn-pr/ludwig/ludwig/cli.py", line 73, in predict
    predict.cli(sys.argv[2:])
  File "/home/andrey/Projects/ludwig-learn-pr/ludwig/ludwig/predict.py", line 379, in cli
    full_predict(**vars(args))
  File "/home/andrey/Projects/ludwig-learn-pr/ludwig/ludwig/predict.py", line 86, in full_predict
    only_predictions
  File "/home/andrey/Projects/ludwig-learn-pr/ludwig/ludwig/data/preprocessing.py", line 684, in preprocess_for_prediction
    train_set_metadata=train_set_metadata
  File "/home/andrey/Projects/ludwig-learn-pr/ludwig/ludwig/data/preprocessing.py", line 61, in build_dataset
    **kwargs
  File "/home/andrey/Projects/ludwig-learn-pr/ludwig/ludwig/data/preprocessing.py", line 89, in build_dataset_df
    global_preprocessing_parameters
  File "/home/andrey/Projects/ludwig-learn-pr/ludwig/ludwig/data/preprocessing.py", line 164, in build_data
    preprocessing_parameters
  File "/home/andrey/Projects/ludwig-learn-pr/ludwig/ludwig/features/image_feature.py", line 163, in add_feature_data
    image_dataset[i, :height, :width, :] = img
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/andrey/.venvs/ludwig-learn-pr/lib/python3.6/site-packages/h5py/_hl/dataset.py", line 708, in __setitem__
    self.id.write(mspace, fspace, val, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 221, in h5py.h5d.DatasetID.write
  File "h5py/_proxy.pyx", line 132, in h5py._proxy.dset_rw
  File "h5py/_proxy.pyx", line 93, in h5py._proxy.H5PY_H5Dwrite
OSError: Can't write data (no appropriate function for conversion path)

[w4nderlust]

@abogoyavlensky thanks for reporting.
This is a different issue from the original one. It's probably stemming from the fact that you specified in_memory: false in the model definition, so the image preprocessing is trying to do the same thing at prediction time too. This is likely a different bug, @ydudin3 can you please look into this?

In the meantime, as a workaround, @abogoyavlensky can you please trythe following.
First thing: we changed a bit the syntax for the in_memory, width and height parameters, no you should specify them in your YAML file this way:

input_features:
    -
        name: image_path
        type: image
        encoder: stacked_cnn
        preprocessing:
            in_memory: false
            height: 100
            width: 100

so train a model with the updated yaml definition.
Then open the model directory, open the hyperaparameters json fine, remove the line that looks like "in_memory": truefrom it, and try to use the model to predict.
This time it should work, please let me know.

[abogoyavlensky]

@w4nderlust I've tried your suggestion but I get the same error. Anyway, I think that the current issue is resolved and we could close it. I’m pretty sure that is something wrong with my dataset or there is a similar cause with misconfiguration and should fix it by myself. Thanks!

[w4nderlust]

@abogoyavlensky I'd like to investigate this further, so I'm reopening it.
If you could please send me and @ydudin3 the command you use for training, the yaml file and a small sample of your data in one zip file that would be great and we can try to solve it ;)

[abogoyavlensky]

@w4nderlust Ok, thanks, here the link to my dataset sample and commands I use ludwig-prediction-example.zip

[ydudin3]

Thanks, I can reproduce, will have a look.

[w4nderlust]

@abogoyavlensky the issue should be solved in master now, can you please confirm that?

[abogoyavlensky]

@w4nderlust Yes, it works just as expected, thank you.

[w4nderlust]

Great, thanks!