Sagemaker-python-sdk: No h5py libraries in the Tensorflow container

[chinazm]

no h5py library in the Tensorflow container, such that tf.keras.model.load_weights does not work!!!

[nadiaya]

Sorry for that!
Temporary work around would be:

from subprocess import call
call("pip install h5py".split(" "))

We have a pull request open for installing dependencies issue

We are also considering adding h5py library to the Tensorflow container.

[chinazm]

Thanks @nadiaya , yes, i did this thing and I also ran the h5py.run_tests() successfully, however, when we load the weights from h5 file:
model = tf.keras.models.model_from_json(d)
model.load_weights(model_weight_path)

it still reported the error:
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/_impl/keras/models.py", line 685, in load_weights
raise ImportError('load_weights requires h5py.')
ImportError: load_weights requires h5py.

[chinazm]

I saw the new release of sagemaker 1.1.1 supporting the requirments.txt packages. However, even we install the package, it still reports the error:

2018-03-07 00:07:28,648 INFO - root - running container entrypoint
2018-03-07 00:07:28,648 INFO - root - starting train task
2018-03-07 00:07:31,203 INFO - botocore.vendored.requests.packages.urllib3.connectionpool -
2018-03-07 00:07:32,781 INFO - botocore.vendored.requests.packages.urllib3.connectionpool - Starting new HTTPS connection (1): **.s3.amazonaws.com
2018-03-07 00:07:33,739 INFO - container_support.environment - current Python environment:
absl-py==0.1.10
awscli==1.14.50
backports.weakref==1.0.post1
bleach==1.5.0
boto3==1.6.3
botocore==1.9.3
click==6.7
colorama==0.3.7
docutils==0.14
enum34==1.1.6
Flask==0.11
funcsigs==1.0.2
futures==3.2.0
gevent==1.2.2
greenlet==0.4.13
grpcio==1.10.0
gunicorn==19.7.1
html5lib==0.9999999
itsdangerous==0.24
Jinja2==2.9
jmespath==0.9.3
Markdown==2.6.11
MarkupSafe==1.0
mock==2.0.0
numpy==1.14.1
pandas==0.22.0
pbr==3.1.1
protobuf==3.5.1
pyasn1==0.4.2
python-dateutil==2.6.1
pytz==2018.3
PyYAML==3.12
rsa==3.4.2
s3transfer==0.1.13
SageMakerContainerSupport==1.0
SageMakerTensorflowContainer==1.0
scikit-learn==0.19.1
scipy==1.0.0
six==1.11.0
sklearn==0.0
tensorflow==1.5.0
tensorflow-serving-api==1.5.0
tensorflow-tensorboard==1.5.1
Werkzeug==0.14.1

2018-03-07 00:07:33,739 INFO - container_support.environment - installing requirements in /opt/ml/code/requirements.txt via pip
2018-03-07 00:07:35,480 INFO - container_support.environment - Collecting hkdf (from -r /opt/ml/code/requirements.txt (line 1))
Downloading hkdf-0.0.3.tar.gz
Collecting h5py (from -r /opt/ml/code/requirements.txt (line 2))
Downloading h5py-2.7.1-cp27-cp27mu-manylinux1_x86_64.whl (4.8MB)
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages (from h5py->-r /opt/ml/code/requirements.txt (line 2))
Requirement already satisfied: numpy>=1.7 in /usr/local/lib/python2.7/dist-packages (from h5py->-r /opt/ml/code/requirements.txt (line 2))
Building wheels for collected packages: hkdf
Running setup.py bdist_wheel for hkdf: started
Running setup.py bdist_wheel for hkdf: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/72/05/12/d7bc286c228af2baeb8e63ad86626d5e046a4ff8e9a36d92ed
Successfully built hkdf
Installing collected packages: hkdf, h5py
Successfully installed h5py-2.7.1 hkdf-0.0.3

/usr/local/lib/python2.7/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
2018-03-07 00:07:35,512 INFO - botocore.vendored.requests.packages.urllib3.connectionpool - Starting new HTTPS connection (1): s3.amazonaws.com
2018-03-07 00:07:35,568 INFO - tf_container - ----------------------TF_CONFIG--------------------------
2018-03-07 00:07:35,568 INFO - tf_container - {"environment": "cloud", "cluster": {"master": ["algo-1:2222"]}, "task": {"index": 0, "type": "master"}}
2018-03-07 00:07:35,568 INFO - tf_container - ---------------------------------------------------------
2018-03-07 00:07:35,568 INFO - tf_container - going to training
2018-03-07 00:07:35,568 INFO - tf_container - creating RunConfig:
2018-03-07 00:07:35,568 INFO - tf_container - {u'save_checkpoints_secs': None, u'save_checkpoints_steps': 1000}
2018-03-07 00:07:35,568 INFO - tf_container - involing keras_model_fn
2018-03-07 00:07:35,799 ERROR - tf_container - writing error
2018-03-07 00:07:35,799 ERROR - tf_container - error file is
2018-03-07 00:07:35,799 ERROR - tf_container -
Exception during training:
load_weights requires h5py.
Traceback (most recent call last):
File "/opt/amazon/lib/python2.7/site-packages/tf_container/run.py", line 21, in train_and_log_exceptions
test_wrapper.train()
File "/opt/amazon/lib/python2.7/site-packages/tf_container/trainer.py", line 109, in train
experiment_fn = self._generate_experiment_fn()
File "/opt/amazon/lib/python2.7/site-packages/tf_container/trainer.py", line 113, in _generate_experiment_fn
estimator = self._build_estimator()
File "/opt/amazon/lib/python2.7/site-packages/tf_container/trainer.py", line 194, in _build_estimator
model = self.customer_script.keras_model_fn(self.customer_params)
File "/opt/ml/code/model.py", line 80, in keras_model_fn
model.load_weights(model_path)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/_impl/keras/models.py", line 698, in load_weights
raise ImportError('load_weights requires h5py.')
ImportError: load_weights requires h5py.

[andremoeller]

Hi @chinazm,

I'm looking into this issue now. Installing h5py via subprocess.call seems to be failing because of validation in this module, where if h5py can't be imported when the module is initially imported, an ImportError is raised:

https://github.com/tensorflow/tensorflow/blob/5aee07fd0462d00c52efb5d3c86bfb955a9d976e/tensorflow/python/keras/_impl/keras/models.py#L44-L47

https://github.com/tensorflow/tensorflow/blob/5aee07fd0462d00c52efb5d3c86bfb955a9d976e/tensorflow/python/keras/_impl/keras/models.py#L697-L698

In other words, if we import tensorflow, then install h5py at runtime, load_weights doesn't know that we can actually import h5py, so it throws this error.

And unfortunately, reloading the tensorflow module is inadvisable: https://github.com/tensorflow/tensorflow/issues/15741

About using the requirements file: similarly: it seems that h5py is installed, but again, it's possible that we're importing TensorFlow before the h5py module is installed, so the validation fails later on, though I haven't confirmed this. I'll create an item for us to handle this case correctly.

We should add h5py to our TensorFlow container so that installing it through pip isn't necessary. I am adding a work item for this too.

In the meantime, as an immediate work-around, I made this gist, which doesn't do the import validation, and instead installs / imports h5py when load_weights is called. I believe you can just drop this function instead of tf.keras.model.load_weights:

https://gist.github.com/andremoeller/c09c8a3bc3eaf0b873a2a3238b0a1657

Thanks, and please let me know how things go, or if something's unclear.

[chinazm]

Thank you Andre @andremoeller ,

your answer is very helpful. However, sometime we need to use the model weight from the imagenet, like
tf.keras.applications.inception_v3.InceptionV3(weights='imagenet')

tf.keras.applications.resnet50.ResNet50(weights='imagenet')

then that will raise the ImportError('load_weights requires h5py.')

Thank you!
Best,

[chinazm]

In addition, @andremoeller , I think I was using the requirements.txt in the sagemaker 1.1.1 (new feature), not Installing h5py via subprocess.call. I think this will install the dependencies first prior to launch the tensorflow...

[andremoeller]

Hi @chinazm ,

Got it. We'll be incorporating h5py into our image so this isn't necessary. And we're also looking into why installing it with the requirements_file didn't work in this case. Thanks!

[ChoiByungWook]

Hello,

We have added h5py within the Tensorflow container and promoted those images.

Thank you for using SageMaker!