Caffe: Provide a Caffe package in Debian

[cdluminate]

Status

Caffe packages are available for Debian/unstable.
Caffe packages are failing to build for Ubuntu-devel and need to be patched.

Last update: Dec.20 2016

Draft guide

Deploy Caffe with merely one command.

Brief Guide for Debian/unstable users

Only experienced linux users are recommended to try Debian/unstable (Sid).
To install caffe, first make sure you have something like the follows in file /etc/apt/sources.list:
(Uncomment the second line if you want to re-compile caffe locally.)

deb http://ftp.cn.debian.org/debian sid main contrib non-free
#deb-src http://ftp.cn.debian.org/debian sid main contrib non-free

Then update apt cache and install it. Note, you cannot install both the cpu version and the cuda version.

# apt update
# apt install [ caffe-cpu | caffe-cuda ]
# caffe

It should work out of box. I hope this work is helpful since there are many people struggling at the Caffe compiling process.

Here are some notes:

• Please re-compile OpenBLAS locally with optimization flags for sake of performance. This is highly recommended if you are writing a paper. The way to re-compile OpenBLAS from Debian source is very similar with the next subsection.
• If you are going to install caffe-cuda, it will automatically pull the CUDA package and the nvidia driver packages. The installation procress may fail if any part of the caffe dependency chain gets into trouble. That is to say, please take care if you have manually installed or significantly modified nvidia driver or CUDA toolkit or protobuf or any other related stuff.
• if you encountered any problem when installing caffe-cuda on a clean Debian system, report bug to me (via Debian's bug tracking system) please.
• If you encountered any problem when installing caffe-cpu, please report bug to me via Debian's bug tracking system.
• Both of caffe-cpu and caffe-cuda contain a manpage (man caffe) and a bash complementation script (caffe <TAB><TAB>, caffe train <TAB><TAB>). Both of them are still not merged into caffe master.
• The python interface is Python3 version: python3-caffe-{cpu,cuda}. No plan to support python2.

Compiling your custom caffe package on Debian/unstable

There is no promise for the content in this subsection. If you just want to compile again from the source without any change, the following should work as expected. If you want to compile it with e.g. CUDNN support, you should at least be able to read and hack the file debian/rules under the source tree (It's a Makefile).

First make sure you have a correct deb-src line in your apt source list file. Then we compile caffe with several simple commands.

# apt update
# apt install build-essential debhelper devscripts    # These are standard package building tools
# apt build-dep [ caffe-cpu | caffe-cuda ]    # the most elegant way to pull caffe build dependencies
# apt source [ caffe-cpu | caffe-cuda ]    # download the source tarball
# cd caffe-XXXX    # now we enter into the source tree
[ ... optional, make your custom changes at your own risk ... ]
# debuild -B -j4    # build caffe with 4 parallel jobs (similar to make -j4)
[ ... building ...]
# debc    # optional, if you want to check the package contents
# debi    # install the generated packages

FAQ

1. where is caffe-cudnn?
   Due to legal reason the cudnn library cannot be redistributed. I'll be happy to make this package when CUDNN becomes re-distributable. The workaround is to install cudnn by yourself, and hack at least the debian/rules file if you really want the caffe *.deb packages with CUDNN support.

2. how to report bug via Debian bug tracking system?
   See https://www.debian.org/Bugs/ .

3. I installed the CPU version, what should I do if I want to switch to CUDA verison?
   sudo apt install caffe-cuda, apt's dependency resolver is smart enough for this.

4. Where is the examples, the models and other documentation stuff?
   sudo apt install caffe-doc; dpkg -L caffe-doc

[bhack]

Probably https://github.com/BVLC/caffe/pull/2523 could help you.

[cdluminate]

@bhack Thank you for this info.
I'm now struggling with caffe's Makefile and Debian packaging rules.
Maybe I will have to look into compiler issues/etc someday in the future.

[bhack]

Actually caffe maintain two build systems: Makefile and Cmake.

[flx42]

Is there really a need to provide both Atlas and OpenBLAS versions? I don't think it makes a big difference performance-wise so I won't be upset if you were to settle on one BLAS library and only provide caffe-cpu and caffe-gpu.
Of course that's only my personal opinion.

[cdluminate]

@bhack Yes, and I prefer to use Makefile :-)

[cdluminate]

@flx42 Initially I don't want to force users to use one of the BLASes.
However thank to your reminder I realized that, maybe it's really not necessary to provide both atlas version and openblas version, because those who really need extraordinary performance will build caffe by themselves, linking with some proprietary software/libraries (cuDNN, MKL).

These blogs are found in order to make a choice between atlas and openblas:
this one , this one,
this one,
and this one

Well, seems that I should take openblas ?

[talda]

I completely agree with @flx42 's comment.
Those who want a specific configuration will compile for themselves anyway, and for those who just want to try it out on a reasonable configuration it wouldn't matter.

Sometimes too much choice is not that good.

I think that OpenBLAS is a better choice

[cdluminate]

@talda OK, thank you for feedback :-)

[bhack]

See also https://wiki.debian.org/DebianScience/LinearAlgebraLibraries

[cdluminate]

@bhack Gotcha.
This is inspiring:

The optimal way of using these two packages is to recompile them locally on the machines on which they are to be used. 

and README.Debian of openblas says:

 34 Compile the package:
 35 
 36 $ cd openblas-<VERSION>
 37 $ fakeroot debian/rules custom
 38 
 39 It should produce a package that you can install with:
 40 
 41 $ sudo dpkg -i ../libopenblas-base_<DEBVERSION>.deb

Then IMHO

• it seems to be of less and less need to provide many versions of compiled caffe
• it could be better if I provide similar custom target like atlas/openblas does

[cdluminate]

By the way, the alternative link libblas.so.3 will not work for caffe, because e.g. the reault of readelf -d against openblas libcaffe.so is

 0x0000000000000001 (NEEDED)             Shared library: [libopenblas.so.0]
[...]
 0x000000000000000e (SONAME)             Library soname: [libcaffe.so.0]

and the result of readelf -d caffe.bin is

 0x0000000000000001 (NEEDED)             Shared library: [libcaffe.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libglog.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgflags.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

We see libcaffe.so is directly linked to openblas, so alternating libblas.so makes no difference.

---

Note that, I added SONAME and fixed RPATH issue for distro, so the result on your local caffe may differ from mine.

[cdluminate]

The packaging work is available at the temp repo:
http://anonscm.debian.org/cgit/users/cdluminate-guest/caffe.git
it passes my local build and the core component works, but is still under heavy development.

---

These packages are provided:

Package: caffe
Package: libcaffe
Package: libcaffe-dev
Package: caffe-cuda
Package: libcaffe-cuda
Package: libcaffe-cuda-dev
Package: caffe-doc

And maybe python-caffe packages will be added in the future.

[cdluminate]

The repo http://anonscm.debian.org/cgit/users/cdluminate-guest/caffe.git/
has trouble to be cloned.

So I just made a mirror for it:
https://github.com/CDLuminate/caffe.git

[cdluminate]

Packaging work was moved to Debian-Science:
http://anonscm.debian.org/cgit/debian-science/packages/caffe.git

[cdluminate]

Current summary:
https://lists.debian.org/debian-science/2015/08/msg00030.html

[shelhamer]

Thank you for working on packaging. This has been requested a lot.
On Fri, Aug 7, 2015 at 01:52 Lumin Zhou [email protected] wrote:

Current summary:

https://lists.debian.org/debian-science/2015/08/msg00030.html

—
Reply to this email directly or view it on GitHub
https://github.com/BVLC/caffe/issues/2601#issuecomment-128644323.

[cdluminate]

Good news, the Debian Packaging of Caffe is done.
http://anonscm.debian.org/cgit/debian-science/packages/caffe.git/

Current packaging scheme provided these Debian package:
Suite 1: caffe-cpu:

• caffe-cpu
• libcaffe-cpu0
• libcaffe-cpu-dev
• python3-caffe-cpu

Suite 2: caffe-cuda

• caffe-cuda
• libcaffe-cuda0
• libcaffe-cuda-dev
• python3-caffe-cuda

For pyhton2 caffe package please have a look at debian/README.Debian in the packaging repo.

Currently I'm running the last build test on my machine,
and will soon find sponsorship to upload it to debian/experimental.

If you can't wait for Debian Official Archive to accept this package,
you can build package by yourself.
(you can do simply debuild or do custom build following
instructions from debian/README.Debian)

I'll provide a new installation guide once the caffe was uploaded to
debian/experimental, for Debian and possibly Ubuntu users.
(well, installing Debian packages on Ubuntu is BAD IDEA but I hope it works when there is e.g. urgent need.)

P.S. Did you Caffe Devs plan any release date for caffe-1.0 ?
Please tell me if there is, so I can schedule my working time for importing caffe-1.0 into Debian Packaging repo.

[bhack]

Do you plan to re-organize the modularization after https://github.com/BVLC/caffe/pull/2523?

[cdluminate]

Well, I'm not going to change the modularization after that merger.
And I think providing all IO compatibility is a good choice for Debian package.
:-)
@bhack

[cdluminate]

Oops, seems that there are some troubles with python3-* packages.
By the way which version of python is recommended? 2 or 3?

[shelhamer]

@CDLuminate thanks for your packaging work! For pycaffe I think Python 2 is the most common choice but some use Python 3 and both are included in our continuous integration.

[cdluminate]

wishlist: more cmake flags -

+ USE_PYTHON2
+ USE_PYTHON3
- python_version

which means, if I turn USE_PYTHON2 and USE_PYTHON3 on, py2caffe and py3caffe will both be built. Then I can provide both python2-caffe* and python3-caffe* packages at the same time, without hacking cmake system too much, or compiling caffe 4 times.

currently there's no python3 version of google.protobuf in Debian, so I‘d better to bump caffe's python interface version to python2.

[cdluminate]

This is a packaging status ping, as said in https://github.com/BVLC/caffe/pull/3311

Caffe itself currently doesn't block packaging at all (it was completed). The real trouble and the real blocker is CUDA from Debian Unstable/Experimental, which is quite out of date (6.5.14 currently) so that it refuses to work with GCC-5. And that is what I'm looking into recently.

[micahcc]

Hmm. So will the package show up in experimental now that there is a package in? http://anonscm.debian.org/cgit/debian-science/packages/caffe.git/

[cdluminate]

@MicahChambers I plan to upload Caffe onto experimental once Debian's CUDA version is working with GCC-5. This may take several more months.

http://anonscm.debian.org/cgit/debian-science/packages/caffe.git/

is my working repo. You may see there is a release version of caffe for experimental in debian/changelog but actually it was blocked by cuda and hence not uploaded.

[micahcc]

Thanks for the reply. Thats kind of a bummer. Debian packages are a pain in the ass.

[cdluminate]

Debian NVIDIA team is planning to upload CUDA 7.5 (likely within half a month),
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=807579
which unblocks Caffe's upload.

[cdluminate]

@shelhamer Are there significant bug fixes or important updates for Caffe since the release of 1.0.0~rc3? Or 1.0.0~rc3 and master, which one is recommended to upload?

Since CUDA 7.5 is about to be uploaded, I start to refresh the packaging repo.

[cdluminate]

CUDA 7.5 is ready in NEW, and my packaging for debian-science/caffe is ready.

In my local build test nothing breaks, so I think caffe is nearly ready to be uploaded.

[naibaf7]

@CDLuminate
Do you think it would be possible to also package libcaffe-opencl? I think the necessary OpenCL packages are also available in Debian.

[cdluminate]

@naibaf7 Well... I'm not familiar with that, but would like to try.
However there is no OpenCL related option in the Cmake compiling system, so how can I achieve that ?

[cdluminate]

@naibaf7 Do you mean this?
https://github.com/amd/OpenCL-caffe

Seems that it is not merged with BVLC/caffe, so it is another upstream....

[naibaf7]

@CDLuminate
No, this: https://github.com/BVLC/caffe/tree/opencl
Does not even depend on clBLAS, so it should be a fairly easy package to make.

[cdluminate]

As wrote in the heading lines It is a work-in-progress.
We can introduce it later when it becomes stable enough, and I'm afraid it'd introduce trouble to package that too early.
Besides, I use caffe_1.0.0~rc3 as the original source, and it contains no opencl codes, if I want to package caffe-opencl, I'll have to import another master version of caffe and then cherry-pick all the changes from the opencl branch. Not worthwhile to do so.

---

@naibaf7

[cdluminate]

Is there any important update or bug fix (commit) after the release of 1.0.0~rc3? so I can include that into the Caffe packages to be uploaded.
I'll wait for responce until several days after ECCV16 submission deadline.

If got no response w.r.t this question, finaly the rc3 version would be uploaded.

[cdluminate]

@lukeyeager please see the post above

[shelhamer]

@CDLuminate there have been changes since 1.0.0-rc3 of course but there are plenty of users staying with that tag for now so packaging it sounds helpful to me. Thanks!

[cdluminate]

@shelhamer Ack. 1.0.0-rc3 will be uploaded.

[dbrouard]

Hi @CDLuminate, I'm trying to compile your package on Jessie and I get this error:

"/usr/bin/ld: cannot find -lpthreads"

I think it's related to g++ version ( Jessie: g++ 4.9.2-10 ). Have you been able to succesfully compile on Jessie?

Thanks, Diego

[cdluminate]

@dbrouard see README.Debian, does $ debian/rules custom-cpu work for you? compiling CUDA version may fail without cuda7.5 being backported.
All packaging things should work on Sid(unstable).

[cdluminate]

My working machine runs jessie, and I compiled caffe-cpu in the way i said above.

[cdluminate]

If you have manually installed CUDA7.5 directly from nvidia, compiling caffe-cuda is also possible but requires user to make some changes on the packaging scripts.

[dbrouard]

Hi, I was trying first a general ( debian/rules ) build ( I should have read the docs first, my fault ).

Now, while doing a debian/custom-cpu build, what fails is a patch ( branch "master" )

patching file CMakeLists.txt
Hunk #1 FAILED at 3.
1 out of 1 hunk FAILED
dpkg-source: info: the patch has fuzz which is not allowed, or is malformed
dpkg-source: info: if patch 'fix-cmake-add-target-property-version-soversion.patch' is correctly applied by quilt, use 'quilt refresh' to update it
dpkg-source: info: restoring quilt backup files for fix-cmake-add-target-property-version-soversion.patch
dpkg-source: error: LC_ALL=C patch -t -F 0 -N -p1 -u -V never -g0 -E -b -B .pc/fix-cmake-add-target-property-version-soversion.patch/ --reject-file=- < caffe/debian/patches/fix-cmake-add-target-property-version-soversion.patch gave error exit status 1
debian/rules:166: recipe for target 'custom-cpu' failed
make: *** [custom-cpu] Error 2

[cdluminate]

Oops, I'm sorry that I'm working on branch debian/experimental, and the master branch is not in use currently....

[cdluminate]

@dbrouard

[cdluminate]

going to upload

[cdluminate]

@bhack @flx42 @shelhamer @lukeyeager

I was wrong on some aspect of this Debian Caffe package.

I'm fixing caffe's cmake system for Debian package, as suggested by some experienced Debian Developers.

Actually for Ubuntu/Debian users, one can compile caffe against the basic BLAS library, and switch the runtime BLAS library with update-alternatives. That is to say, switch BLAS library as you like after only one build. To achieve this, I added FindBLAS module in Caffe's cmake system, and things seem to be working.

[lukeyeager]

Neat. So you would have the deb package depend on the virtual package libblas.so.3?
http://packages.ubuntu.com/trusty/libblas.so.3

Note that not all of those packages are actually interchangeable, though. Caffe probably won't care, but numpy does.
http://stackoverflow.com/a/34956540/2404152

[cdluminate]

@lukeyeager

Caffe package in the git repo depends on libblas.so.3 already.

Numpy's dependency:

Depends: python (>= 2.7), python (<< 2.8), python2.7:any, libblas3 | libblas.so.3, libc6 (>= 2.14), libgcc1 (>= 1:4.1.1), libgfortran3 (>= 4.3), liblapack3 | liblapack.so.3, libquadmath0 (>= 4.6)

so I don't need to change current dependency of caffe. Missing parts will be pulled by numpy.

However this is a notable aspect, as alternatives are controled by users and admins, I should mention it in the package readme file. Thanks!

[cdluminate]

@shelhamer @lukeyeager

Bad news, in the last stage of Debian caffe package review, the upload abort again.
Current caffe package uses python2 interface, but python2 is not recommend for next Debian release. Now some python3 dependency packages blocked it. The caffe package doen't meet all Debian's requirments, so this time of upload aborted. We have to solve more problems before it can be uploaded.

My potential users please don't be sad, the packaging script is currently in a wonderful shape. If you can't wait to use these packages, you can download it and compile it by yourself. Instructions are included in README.Debian. Feel free to report any problems with it.

DETAIL:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=823140

PACKAGING SCRIPTS:
http://anonscm.debian.org/cgit/debian-science/packages/caffe.git/
note: branch "master"

[cdluminate]

How many Caffe users use caffe's python interface?

[flx42]

A large majority I would say.

[cdluminate]

So I should not temporarily disable building python packages. @flx42

[cdluminate]

At the end of CVPR2016 some experimental caffe packages was uploaded to Debian experimental. Note they are for debugging purpose, I'm sure these experimental packages are still problematic but they still bring useful information to us:

The first upload of caffe-1.0.0~rc3 was successfully built[1] by several architectures amd64
ppc64el and s390x (and kfreebsd-amd64, sparc64), while it fails to build on arm*, i386, mips*, powerpc, and so on.
So good news is amd64 and ppc64el are safe.

Arm64 failed because

[ RUN      ] DataLayerTest/1.TestReadCropTestLMDB
F0701 18:55:54.704546 10958 db_lmdb.hpp:14] Check failed: mdb_status == 0 (12 vs. 0) Cannot allocate memory
*** Check failure stack trace: ***
    @       0x7f81fd7280  google::LogMessage::Fail()
    @       0x7f81fd917c  google::LogMessage::SendToLog()
    @       0x7f81fd6df8  google::LogMessage::Flush()
    @       0x7f81fd9a1c  google::LogMessageFatal::~LogMessageFatal()
    @       0x7f822f763c  caffe::db::LMDB::Open()
    @       0x55600cf1ec  caffe::DataLayerTest<>::Fill()
    @       0x55600dd460  caffe::DataLayerTest_TestReadCropTestLMDB_Test<>::TestBody()
    @       0x55603005dc  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @       0x55602f853c  testing::Test::Run()
    @       0x55602f8678  testing::TestInfo::Run()
    @       0x55602f8784  testing::TestCase::Run()
    @       0x55602fa1f0  testing::internal::UnitTestImpl::RunAllTests()
    @       0x55602fa4fc  testing::UnitTest::Run()
    @       0x556003cd94  main
    @       0x7f81421724  __libc_start_main

armel failed because

[ RUN      ] DBTest/1.TestWrite
F0701 20:07:22.396028 13666 db_lmdb.hpp:14] Check failed: mdb_status == 0 (-30792 vs. 0) MDB_MAP_FULL: Environment mapsize limit reached
*** Check failure stack trace: ***
    @ 0xb64beda4  google::LogMessage::Fail()
    @ 0xb64c0f1c  google::LogMessage::SendToLog()
    @ 0xb64be8bc  google::LogMessage::Flush()
    @ 0xb64c17c4  google::LogMessageFatal::~LogMessageFatal()
    @ 0xb67d5b4c  caffe::db::LMDBTransaction::Put()
    @ 0xb6c0cbe8  caffe::DBTest_TestWrite_Test<>::TestBody()
    @ 0xb6e34f20  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @ 0xb6e2dbd8  testing::Test::Run()
    @ 0xb6e2dd78  testing::TestInfo::Run()
    @ 0xb6e2de94  testing::TestCase::Run()
    @ 0xb6e2fbb8  testing::internal::UnitTestImpl::RunAllTests()
    @ 0xb6e2ff70  testing::UnitTest::Run()
    @ 0xb6b4b1bc  main
    @ 0xb58b3c2c  __libc_start_main

armhf failed because

[ RUN      ] DBTest/1.TestWrite
F0701 21:11:28.346155 23381 db_lmdb.hpp:14] Check failed: mdb_status == 0 (-30792 vs. 0) MDB_MAP_FULL: Environment mapsize limit reached
*** Check failure stack trace: ***
    @ 0xb67a9e2e  google::LogMessage::Fail()
    @ 0xb67ab42a  google::LogMessage::SendToLog()
    @ 0xb67a9adc  google::LogMessage::Flush()
    @ 0xb67aba4c  google::LogMessageFatal::~LogMessageFatal()
    @ 0xb6a053fc  caffe::db::LMDBTransaction::Put()
    @ 0xb6dc01ba  caffe::DBTest_TestWrite_Test<>::TestBody()
    @ 0xb6f2e164  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @ 0xb6f29638  testing::Test::Run()
    @ 0xb6f29744  testing::TestInfo::Run()
    @ 0xb6f297f2  testing::TestCase::Run()
    @ 0xb6f2aa6e  testing::internal::UnitTestImpl::RunAllTests()
    @ 0xb6f2acc6  testing::UnitTest::Run()
    @ 0xb6d40688  main
    @ 0xb5ece7aa  __libc_start_main

i386 failed because

[ RUN      ] DBTest/1.TestWrite
F0701 19:53:48.369148  6432 db_lmdb.hpp:14] Check failed: mdb_status == 0 (-30792 vs. 0) MDB_MAP_FULL: Environment mapsize limit reached
*** Check failure stack trace: ***
    @ 0xf6ca8985  google::LogMessage::Fail()
    @ 0xf6caabe4  google::LogMessage::SendToLog()
    @ 0xf6ca84be  google::LogMessage::Flush()
    @ 0xf6cab76d  google::LogMessageFatal::~LogMessageFatal()
    @ 0xf6f7db0b  caffe::db::LMDBTransaction::Put()
    @ 0xf73fa74f  caffe::DBTest_TestWrite_Test<>::TestBody()
    @ 0xf7667294  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @ 0xf765f7d9  testing::Test::Run()
    @ 0xf765f93c  testing::TestInfo::Run()
    @ 0xf765fa45  testing::TestCase::Run()
    @ 0xf7661e4e  testing::internal::UnitTestImpl::RunAllTests()
    @ 0xf766219d  testing::UnitTest::Run()
    @ 0xf7326956  main
    @ 0xf6002517  __libc_start_main
    @ 0xf732ba1f  (unknown)

mips failed because

[ RUN      ] DBTest/1.TestWrite
F0701 20:18:26.746393  2562 db_lmdb.hpp:14] Check failed: mdb_status == 0 (-30792 vs. 0) MDB_MAP_FULL: Environment mapsize limit reached
*** Check failure stack trace: ***
    @ 0x76ffcd34  google::LogMessage::Fail()
    @ 0x76fff9f8  google::LogMessage::SendToLog()
    @ 0x76ffc78c  google::LogMessage::Flush()
    @ 0x770004e4  google::LogMessageFatal::~LogMessageFatal()
    @ 0x7735b398  caffe::db::LMDBTransaction::Put()
    @ 0x77850fc8  caffe::DBTest_TestWrite_Test<>::TestBody()
    @ 0x77b2e86c  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @ 0x77b24584  testing::Test::Run()
    @ 0x77b247a0  testing::TestInfo::Run()
    @ 0x77b24938  testing::TestCase::Run()
    @ 0x77b27880  testing::internal::UnitTestImpl::RunAllTests()
    @ 0x77b27c70  testing::UnitTest::Run()
    @ 0x777540a0  main
    @ 0x761009ec  __libc_start_main

mipsel

[ RUN      ] DBTest/1.TestKeyValue
F0701 22:55:32.162811 13376 db_lmdb.hpp:14] Check failed: mdb_status == 0 (-30792 vs. 0) MDB_MAP_FULL: Environment mapsize limit reached
*** Check failure stack trace: ***
    @ 0x76dd9d38  google::LogMessage::Fail()
    @ 0x76ddc984  google::LogMessage::SendToLog()
    @ 0x76dd9790  google::LogMessage::Flush()
    @ 0x76ddd4f0  google::LogMessageFatal::~LogMessageFatal()
    @ 0x771427a8  caffe::db::LMDBTransaction::Put()
    @ 0x77634808  caffe::DBTest<>::SetUp()
    @ 0x7791b788  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @ 0x77911388  testing::Test::Run()
    @ 0x77911654  testing::TestInfo::Run()
    @ 0x779117ec  testing::TestCase::Run()
    @ 0x77914738  testing::internal::UnitTestImpl::RunAllTests()
    @ 0x77914b38  testing::UnitTest::Run()
    @ 0x7753d0a0  main
    @ 0x75eb19fc  __libc_start_main

powerpc

[ RUN      ] DBTest/1.TestWrite
F0701 18:55:30.709589 27882 db_lmdb.hpp:14] Check failed: mdb_status == 0 (-30792 vs. 0) MDB_MAP_FULL: Environment mapsize limit reached
*** Check failure stack trace: ***
    @ 0x1ffea8f0  google::LogMessage::Fail()
    @ 0x1ffecdfc  google::LogMessage::SendToLog()
    @ 0x1ffea340  google::LogMessage::Flush()
    @ 0x1ffed7e0  google::LogMessageFatal::~LogMessageFatal()
    @ 0x203e4290  caffe::db::LMDBTransaction::Put()
    @ 0x208c5bbc  caffe::DBTest<>::SetUp()
    @ 0x20b31034  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @ 0x20b28168  testing::Test::Run()
    @ 0x20b283c8  testing::TestInfo::Run()
    @ 0x20b28534  testing::TestCase::Run()
    @ 0x20b2ade4  testing::internal::UnitTestImpl::RunAllTests()
    @ 0x20b2b0cc  testing::UnitTest::Run()
    @ 0x207f736c  main
    @ 0x1f141674  (unknown)
    @ 0x1f14187c  __libc_start_main
    @      (nil)  (unknown)

[1] https://buildd.debian.org/status/package.php?p=caffe&suite=experimental

[cdluminate]

Well all 32-bit architectures mentioned above failed here

db_lmdb.hpp:
 10 const size_t LMDB_MAP_SIZE = 1099511627776;  // 1 TB
 11 
 12 void LMDB::Open(const string& source, Mode mode) {
 13   MDB_CHECK(mdb_env_create(&mdb_env_));
 14   MDB_CHECK(mdb_env_set_mapsize(mdb_env_, LMDB_MAP_SIZE));

Seems that the arm64 build failed because of the same number.

I found these

• http://jetsonhacks.com/2015/01/20/nvidia-jetson-tk1-cudnn-install-caffe-example/
• https://github.com/BVLC/caffe/issues/1861
• https://github.com/BVLC/caffe/issues/2293
• https://github.com/BVLC/caffe/issues/2709

And it seems that this change fixes all the failures above:

• https://github.com/BVLC/caffe/pull/3731

@lukeyeager @shelhamer

[cdluminate]

This is the final package scheme after revised again and again:

source caffe only builds cpu version
* caffe-cpu (meta package, pulls command line interface and the python3 interface)
* caffe-tools-cpu (command line interface)
* python3-caffe-cpu (python3 interface)
* octave-caffe-cpu (octave interface)
* libcaffe-cpu1 (libcaffe)
* libcaffe-cpu-dev (headers)
* caffe-doc (Doxygen document and some other documents)

[cdluminate]

Similarly, these will be made for the CUDA version

* caffe-cuda (metapackage)
* caffe-tools-cuda
* python3-caffe-cuda
* octave-caffe-cuda
* libcaffe-cuda1
* libcaffe-cuda-dev

[cdluminate]

Concerning the cpu version of caffe in experimental:
https://tracker.debian.org/pkg/caffe

Conmmand line tools should be working nicely, and the python3 interface should be working
if you had python3-protobuf manually installed. The missing of python3-protobuf is actually
the reason why that package goes into experimental instead of unstable.

The cuda version will soon emerge in experimental too.

[lukeyeager]

And it seems that this change fixes all the failures above: #3731

Glad to hear it. Did you change the source tarball or include those commits as patches? Just curious what the preferred Debian workflow is.

[cdluminate]

Did you change the source tarball or include those commits as patches?

Will cherry-pick the patches for the next upload 1.0.0~rc3-4 for CPU version and 1.0.0~rc3-2 for cuda version.

Just curious what the preferred Debian workflow is.

The release experimental can be used for debugging. I build package on amd64 locally, and then upload it to experimetal, which triggers the Debian automatic build machine to build the uploaded source, so I can see on which architecture it fails to build. Now have the result and fix is needed.

CUDA version was uploaded into experimental.

[cdluminate]

@shelhamer @lukeyeager

Updated the post on the top of this page with a brief and informal guide. However only Debian/unstable users are recommended to have a try (Do production research machines use Debian/Sid? I guess no way. The administrator must be very cool to tame that beast).

[cdluminate]

@lukeyeager @shelhamer

I'd like to bump the caffe debian package version to the current master branch, since some fixes like

1. lmdb map size
2. float comparison [1]
   are introduced after the release of rc3, which solves build failures on many architectures. How do you like?

[1] https://github.com/BVLC/caffe/commit/f1a8470aa21e35a5b2bb83007f8fb7680a354815

[shelhamer]

@CDLuminate sure, I think the current master is a reasonable point to make a package. Fixing these failures, which do show up in the wild, would be nice. Thanks!

[cdluminate]

Uploaded caffe (cpu version) 1.0.0~rc3+20160930-ga7f950b-1 and caffe (cuda version)1.0.0~rc3+20160930-ga7f950b-1 onto Debian/experimental.

[cdluminate]

@shelhamer I'm trying to get caffe-cpu package into the next Debian release (its freeze date is comming soon -- Jan 5 2017). Do you have any release plan within this period?

As mentioned before, the next Debian release is not possible to ship caffe-cuda since CUDA8.0 doesn't support its compiler (GCC-6).

[shelhamer]

@CDLuminate Thanks for the heads-up about packaging. It has been quite a while since a release at this point so we'll be meeting this week to make a plan for the next release. No promises, but with any luck there'll be a release for the end of the year.

[cdluminate]

@shelhamer Thanks. For me the only blocker is python3-protobuf
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844608

That means I can go ahead uploading caffe to unstable (then it will automatically enter the next release) as long as some Debian Developer is willing to have the protobuf package fixed.

[cdluminate]

Python3-protobuf (protobuf_3.0.0-8) is pending in Debian's upload queue.
https://tracker.debian.org/pkg/protobuf

[cdluminate]

I'm preparing the experimental -> unstable migration for both caffe-cpu and caffe-cuda.
The cuda version is considered because I finally compiled caffe-cuda with CUDA-8.0 and Clang-3.8, which means I have a good chance to close this issue before the end of this year.

If any Debian 9 Stretch user complains about the GCC-6's failure working with CUDA-8.0 when compiling caffe in the future, you can suggest him/her try Clang/LLVM.

$ sudo apt install clang-3.8
$ cd caffe-master
$ mkdir build; cd build
$ cmake .. -DCMAKE_CXX_COMPILER=clang++-3.8 -DCMAKE_C_COMPILER=clang-3.8 <YOUR_FLAGS>

---

Edit: Maybe I'm wrong on Stretch's default clang version.

[cdluminate]

Caffe cpu version[1] and cuda version[2] both available for Debian/unstable.
These sources were pulled into Ubuntu devel by bot, see [3][4].

It remains to be seen whether caffe will be shipped in the next release of Debian.
Besides, Ubuntu-devel is failing to build caffe because their protobuf version
is still staying at 3.0.0-7 instead of >=3.0.0-8 which yields python3-protobuf.

Well... What I can do now is just waiting.

[1] https://tracker.debian.org/pkg/caffe
[2] https://tracker.debian.org/pkg/caffe-contrib
[3] https://launchpad.net/ubuntu/+source/caffe
[4] https://launchpad.net/ubuntu/+source/caffe-contrib

[cdluminate]

Replaced the first post of this issue with a draft guide. FAQ is open to anyone.

[cdluminate]

@shelhamer @lukeyeager

I'm thinking of closing this bug, since the Debian packages are working well.
There is a draft guide for these packages on the top of this page.

Currently the caffe packages are only available for Debian/Sid, but if you
would like to merge the guide as a part of your document (e.g. extending install_apt.md),
I'll keep updating that guide by PR's. If not, maybe I'll keep updating the top post
here.

BTW, I think install_apt.md should be renamed to install_apt_ubuntu.md, so
it will not conflict with the Debian guide install_apt_debian.md.

https://tracker.debian.org/pkg/caffe
https://tracker.debian.org/pkg/caffe-contrib

[cdluminate]

P.S. Today I verified that caffe-cuda package is working with TitanX Pascal card.

[naibaf7]

@CDLuminate Would there be a possibility or interest to add OpenCL-Caffe packages?

[shelhamer]

@CDLuminate If you close the issue because the packages are working I'd vote for including a debian install guide. Otherwise your guide is unlikely to be easily discoverable. If you could make a PR to the documentation when you close we can merge.

BTW, I think install_apt.md should be renamed to install_apt_ubuntu.md, so
it will not conflict with the Debian guide install_apt_debian.md.

I hear you, but i'd rather not change the URL of the existing guide. Could you just add the new guide instead?

[cdluminate]

@naibaf7 Providing caffe-opencl is possible, as long as it has a number of users, so that the packaging work will be valuable. Of course, I'd be happy to do that if OpenCL is obviously faster than CUDA in some cases. However https://github.com/soumith/convnet-benchmarks .

I'm also working on Torch and Tensorflow packaging.

[cdluminate]

@shelhamer Please look forward to the PR.

[cdluminate]

https://tracker.debian.org/pkg/caffe
https://tracker.debian.org/pkg/caffe-contrib

Both the caffe-cpu and the caffe-cuda package has entered Debian/testing. Although caffe-cuda will be removed from testing soon however I cannot find a reason to keep this issue open.

Marking this bug as done. LOL

[cdluminate]

Update:

While Stretch/Sid is still shipping Caffe-{cpu,gpu}_rc4 , the rc5 version is available in experimental.

Ubuntu 17.04 has been released several days ago, and this is the first time that a Ubuntu release ships Caffe. Ubuntu 17.04 ships the rc4 version (automatically pulled from Debian), however with a hacky patch applied by Ubuntu guys.

http://launchpadlibrarian.net/305712477/caffe_1.0.0~rc4-1_1.0.0~rc4-1ubuntu1.diff.gz (canceling unit test)

This ubuntu1 patch above is exactly why I don't want to mention the caffe package existence in Ubuntu. It disabled unit test to bypass the unit test failure so that the package can be built successfully. In this case I cannot make sure whether the resulting package is working at all. However, eventually some users will install and use this package ...

The bypassed unit test failure looks like the follows, and it is a problem of Ubuntu itself:

[----------] 5 tests from ImageDataLayerTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN      ] ImageDataLayerTest/0.TestResize
*** Aborted at 1485019567 (unix time) try "date -d @1485019567" if you are using GNU date ***
PC: @     0x7facc82c00ba (unknown)
*** SIGSEGV (@0xfffffffffffffff7) received by PID 20719 (TID 0x7facb38bf700) from PID 18446744073709551607; stack trace: ***
    @     0x7facd0d42620 (unknown)
    @     0x7facc82c00ba (unknown)
    @     0x7facc82c018b (unknown)
    @     0x7facc82c1ce8 (unknown)
    @     0x7facc82c0692 (unknown)
    @     0x7facc82bb020 (unknown)
    @     0x7facc82b9165 tbb::internal::allocate_root_with_context_proxy::allocate()
    @     0x7faccf7fae22 cv::parallel_for_()
    @     0x7faccfbbeb2a (unknown)
    @     0x7faccfbbbbb7 cv::resize()
    @     0x7facd18240f7 caffe::ReadImageToCVMat()
    @     0x7facd173b82b caffe::ImageDataLayer<>::load_batch()
    @     0x7facd16c9d46 caffe::BasePrefetchingDataLayer<>::InternalThreadEntry()
    @     0x7facd16a58eb boost::detail::thread_data<>::run()
    @     0x7facd1179576 (unknown)
    @     0x7facd0d386ca start_thread
    @     0x7facceb100ff clone
    @                0x0 (unknown)
Segmentation fault (core dumped)

Full build log (unit test not canceled) https://launchpadlibrarian.net/303350189/buildlog_ubuntu-zesty-amd64.caffe_1.0.0~rc4-1_BUILDING.txt.gz

Ubuntu (>= 16.10) users are suffering from the same problem, see e.g. https://github.com/BVLC/caffe/issues/5077

@shelhamer @lukeyeager

[lukeyeager]

I have thoughts (probably an OpenCV issue rather than "a problem of Ubuntu itself"), but neither this bug nor #5077 seems appropriate for the discussion. Will you make a new issue for this test error?

[cdluminate]

@lukeyeager https://github.com/BVLC/caffe/issues/5540

[cdluminate]

Uploaded caffe-cpu 1.0 to experimental. CUDA version will be updated soon.

[cdluminate]

I sensed an odor of new release since you guys are ridiculously active these days. hhh