Serving: Package recently broken on ubuntu 16.04 ?

Created on 19 Mar 2018  路  11Comments  路  Source: tensorflow/serving

Hello,

On an ubuntu 16.04, when I install the latest tensorflow serving package, it does not start with:

~$ tensorflow_model_server --help
tensorflow_model_server: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by tensorflow_model_server)
tensorflow_model_server: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by tensorflow_model_server)

Idem when I really try to serve models.

My system is up to date.

It was working perfectly before an apt update + upgrade this week end (~03/17, not sure of exact date).

Current package state:

~(1)$ apt show tensorflow-model-server
Package: tensorflow-model-server
Version: 1.6.0
Built-Using: Bazel
Priority: optional
Section: contrib/devel
Maintainer: TensorFlow Serving team
Installed-Size: inconnu
Homepage: https://github.com/tensorflow/serving
Download-Size: 88,8 MB
APT-Manual-Installed: yes
APT-Sources: http://storage.googleapis.com/tensorflow-serving-apt stable/tensorflow-model-server amd64 Packages
Description: TensorFlow Serving ModelServer

~$ apt-cache show tensorflow-model-server
Package: tensorflow-model-server
Version: 1.6.0
Architecture: all
Maintainer: TensorFlow Serving team
Priority: optional
Section: contrib/devel
Filename: pool/tensorflow-model-server/t/tensorflow-model-server/tensorflow-model-server_1.6.0_all.deb
Size: 88812004
SHA256: 04f7e0937ffd6554e04d95eb850a7740ad81ad3c39d594b4d213dc6b14d775d7
SHA1: 4bc7445661aeacc03fa373474c3be4ba86eee8dd
MD5sum: ceabe0a8fa3a3809015c5cd203714488
Description: TensorFlow Serving ModelServer
Description-md5: 9b7e03f5296f318009581d6e285e2f89
Homepage: https://github.com/tensorflow/serving
Built-Using: Bazel

~$ uname -a
Linux Lenovo 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Do you have any clue ? Is it an issue with the package or do I need another dependency ?

Best,

picture taynaud

Most helpful comment

From how I understood several AskUbuntu and Stackoverflow threads, you have to enable the newest gcc version, by adding a certain repository to your apt-get package sources.

apt-get install -y software-properties-common # if not already installed
add-apt-repository ppa:ubuntu-toolchain-r/test -y

And then do the following commands (until it works, I had to do a dist-upgrade, but it could work earlier if I understood it correctly)

apt-get update
apt-get upgrade
apt-get dist-upgrade

Note that I usually don't use Ubuntu/apt-get and only did this to fix my docker.

Credits to https://askubuntu.com/a/582910

picture klockeph on 20 Mar 2018
👍22

All 11 comments

Hi!

Same issue here with Ubuntu LTS 16.04, the 1.5.0 package is not listed anymore in the Release file, but you can download it via a wget.

root@0a96112b4fac:~# file 1.5.0/usr/bin/tensorflow_model_server 
1.5.0/usr/bin/tensorflow_model_server: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=d644ec743c5c224deea8fbad3378d9e0733769d4, not stripped
root@0a96112b4fac:~# file 1.6.0/usr/bin/tensorflow_model_server 
1.6.0/usr/bin/tensorflow_model_server: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=22568349c0aa5e934bb606ac117af06f4bc41be3, not stripped
sbadia picture sbadia on 19 Mar 2018

From how I understood several AskUbuntu and Stackoverflow threads, you have to enable the newest gcc version, by adding a certain repository to your apt-get package sources.

apt-get install -y software-properties-common # if not already installed
add-apt-repository ppa:ubuntu-toolchain-r/test -y

And then do the following commands (until it works, I had to do a dist-upgrade, but it could work earlier if I understood it correctly)

apt-get update
apt-get upgrade
apt-get dist-upgrade

Note that I usually don't use Ubuntu/apt-get and only did this to fix my docker.

Credits to https://askubuntu.com/a/582910

klockeph picture klockeph on 20 Mar 2018
👍22

Seems like we would need to update https://www.tensorflow.org/serving/setup#installing_using_apt-get

lc0 picture lc0 on 20 Mar 2018
👍1

Seeing the same. Here's a Travis build that uses the model server with error logs:
https://travis-ci.org/rsepassi/tensor2tensor/jobs/357041726#L2254

rsepassi picture rsepassi on 22 Mar 2018

I had the same problem, and it worked perfectly when I used the ubuntu:18.04 docker image instead of 16.04.

nuchi picture nuchi on 26 Mar 2018

As @sbadia mentioned above, pulling the packages directly seemed to work. Here is an updated dockerfile that worked for me.

FROM ubuntu:16.04
# Install general packages
RUN apt-get update && apt-get install -y \
        curl \
        libcurl3-dev \
        unzip \
        wget \
        && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Previous Installation of tensorflow-model-server (BROKEN RECENTLY)
#RUN echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | tee /etc/apt/sources.list.d/tensorflow-serving.list \
#    && curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | apt-key add - \
#    && apt-get update && apt-get install tensorflow-model-server

# New installation of tensorflow-model-server
RUN TEMP_DEB="$(mktemp)" \
    && wget -O "$TEMP_DEB" 'http://storage.googleapis.com/tensorflow-serving-apt/pool/tensorflow-model-server-1.5.0/t/tensorflow-model-server/tensorflow-model-server_1.5.0_all.deb' \
    && dpkg -i "$TEMP_DEB" \
    && rm -f "$TEMP_DEB" 

# Download the trained model for serving
RUN wget SOME_HOSTED_URL/model-export.zip -O model_export.zip \
    && unzip model_export.zip -d model-export \
    && mv model-export/ /tmp \
    && rm model_export.zip

EXPOSE 9000

# Serve the model when the container starts
ENTRYPOINT ["tensorflow_model_server"]
CMD ["--port=9000", "--model_name=model", "--model_base_path=/tmp/model-export"]
dcyoung picture dcyoung on 26 Mar 2018
👍7 🎉2

@dcyoung Does this update work on GPU enabled models?

strategywise picture strategywise on 26 Mar 2018

@strategywise Whenever I create docker container intended for GPU support, I have them build tensorflow serving from source. I haven't tested the package in that scenario.

dcyoung picture dcyoung on 27 Mar 2018

While the solution suggested by @KillPinguin (thx!) above works fine (on ubuntu 16.04), it suffices to just update libstdc++6, as suggested by the actual error:

add-apt-repository ppa:ubuntu-toolchain-r/test -y
apt-get update
apt-get install libstdc++6
chleibig picture chleibig on 1 Apr 2018
👍17

Hi @KillPinguin,

Thank you for the suggestion. I solve the same issue with update/upgrade command.
It's odd that the update/upgrade command didn't change my GLIBC/CXXAB version, upgrade my tensorflow-model-server version from 1.7.0 to 1.8.0 instead. I think maybe use tensorflow-model-server 1.8.0 directly is another workaround to this issue.

apt-get update
apt-get upgrade
apt-get dist-upgrade

System Information:

root@59dbb35082c6:/# strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.16
GLIBCXX_3.4.17
GLIBCXX_3.4.18
GLIBCXX_3.4.19
GLIBCXX_3.4.20
GLIBCXX_3.4.21
GLIBCXX_DEBUG_MESSAGE_LENGTH

root@59dbb35082c6:/notebooks/downloads/installation_file# strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep CXXAB
CXXABI_1.3
CXXABI_1.3.1
CXXABI_1.3.2
CXXABI_1.3.3
CXXABI_1.3.4
CXXABI_1.3.5
CXXABI_1.3.6
CXXABI_1.3.7
CXXABI_1.3.8
CXXABI_1.3.9
CXXABI_TM_1
CXXABI_FLOAT128

root@59dbb35082c6:/# apt show tensorflow-model-server
Package: tensorflow-model-server
Version: 1.8.0
Built-Using: Bazel
Priority: optional
Section: contrib/devel
Maintainer: TensorFlow Serving team
Installed-Size: unknown
Homepage: https://github.com/tensorflow/serving
Download-Size: 98.0 MB
APT-Manual-Installed: yes
APT-Sources: http://storage.googleapis.com/tensorflow-serving-apt stable/tensorflow-model-server amd64 Packages
Description: TensorFlow Serving ModelServer


root@59dbb35082c6:/# apt-cache show tensorflow-model-server
Package: tensorflow-model-server
Version: 1.8.0
Architecture: all
Maintainer: TensorFlow Serving team
Priority: optional
Section: contrib/devel
Filename: pool/tensorflow-model-server/t/tensorflow-model-server/tensorflow-model-server_1.8.0_all.deb
Size: 97987044
SHA256: 3a3288a964402a5c7f1217ff26d3f971adfbd8a2a2206087f0ea61662cca1d6b
SHA1: 6e446cbdd6fe63c661899c2b14ebf8c0ab3a8df9
MD5sum: 79730a9f53b26e231e08e89b5fe664b9
Description: TensorFlow Serving ModelServer
Description-md5: 9b7e03f5296f318009581d6e285e2f89
Homepage: https://github.com/tensorflow/serving
Built-Using: Bazel
cchung100m picture cchung100m on 21 Jun 2018

The latest docker images have all the instructions needed to build on Ubuntu 16.04. Closing this issue.

gautamvasudevan picture gautamvasudevan on 11 Aug 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

abcfy2 picture abcfy2  路  4Comments
daikankan picture daikankan  路  4Comments
atwj picture atwj  路  4Comments
waichee picture waichee  路  4Comments
cchung100m picture cchung100m  路  4Comments