Serving: Fail to load the models from HDFS

[tobegit3hub]

We have tested the integration of TensorFlow and HDFS. It works for training and we can use TensorFlow APIs to access HDFS data.

However, I use that version of TensorFlow to build TensorFlow Serving and the server could not load the files from HDFS. I think TensorFlow Serving use the native TensorFlow Filesystem APIs and it should support HDFS well.

We pass the path of HDFS like this.

--model_base_path=hdfs://namenode:10000/user/u_chendihao/deep_recommend_system/model/

Here is the error log.

E tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:304] FileSystemStoragePathSource encountered a file-system access error: Could not find base path hdfs://namenode:10000/user/u_chendihao/deep_recommend_system/model/ for servable dense

[chrisolston]

When you run 'configure' on tensorflow, did you say "Y" to including hadoop support? (By default it is not included.)

[tobegit3hub]

Thanks @chrisolston . Yes, we have set "Y" for hadoop support when ./configure.

I also set all the environment variables for Java and hadoop although it's not documented for TensorFlow Serving. It may be the same as TensorFlow itself. But it doesn't work and throw those errors.

[terryKing1992]

@tobegit3hub did you find the approach on loading model from HDFS？if not perhaps i will download the model to local, and serving it.

[tobegit3hub]

Thanks @terryKing1992 and downloading model files into local filesystem can always work without doubt.

[manuzhang]

The error is from the following codes.

  if (!Env::Default()->FileExists(servable.base_path()).ok()) {
    return errors::InvalidArgument("Could not find base path ",
                                   servable.base_path(), " for servable ",
                                   servable.servable_name());
  }

After diving into Env::FileExists I found null is returned with error Unimplemented: File system scheme hdfs not implemented although I'd set Y for Hadoop support when ./configure.

[tobegit3hub]

@chrisolston Can you have a look at this which is easy to re-produced?

[llhe]

Adding @org_tensorflow//tensorflow/core/platform/hadoop:hadoop_file_system here should solve this.

[llhe]

Another option (after proper ./configure):

bazel build --define with_hdfs_support=true tensorflow_serving/model_servers:tensorflow_model_server

[tobegit3hub]

This works like a charm and thanks @llhe

[JohnHerry]

Again, the tensorflow_model_server built can not listen on hdfs address. tensorflow serving branch r1.5

Execute command like this:
./tensorflow_model_server --port=10023 --model_base_path=hdfs://192.168.12.129:9000/tensorflow_models/compute_tensor --model_name="example"

Error info like this:
FileSystemStoragePathSource encountered a file-system access error: Could not find base path hdfs://192.168.12.129:9000/tensorflow_models/compute_tensor for servable default

The model file can be accessed with hdfs command directly:
$hdfs dfs -ls hdfs://192.168.12.129:9000/tensorflow_models/compute_tensor

out put is:
18/02/09 16:29:53 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 1 items drwxrwxrwx - pyenv supergroup 0 2018-02-07 16:12 hdfs://192.168.12.129:9000/tensorflow_models/compute_tensor/1

[chentianyou]

I have this problem too.
@JohnHerry Have you solved the problem？

[JohnHerry]

Yes, I missed some environment variable about hadoop before.
If you still have such problem, try as follows:

in $HOME/.bash_profile.

export HADOOP_HOME=/home/john/tests/hdfs_tensorflow/hadoop/hadoop-2.8.0 export HADOOP_INSTALL=$HADOOP_HOME export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin