Incubator-mxnet: Discussion and troubleshooting on PyPI (pip) / Scala Maven (Nexus) installation

[szha]

PyPI for Apache MXNet installation has been available for a while and has been the most important channel for distribution. I'm opening this issue to collect feedbacks, suggestions and issue reports for releases and nightly pre-releases.

pip Status

Supported variants:

• mxnet on Windows, Linux, and Mac OSX, with CPU-only without MKL-DNN support.
• mxnet-cu75 on Linux, supports CUDA-7.5. (discontinued in 1.2)
• mxnet-cu80 on Windows and Linux, supports CUDA-8.0.
• mxnet-cu90 on Windows and Linux, supports CUDA-9.0.
• mxnet-cu91 on Windows and Linux, supports CUDA-9.1. (will discontinue in 1.3)
• mxnet-cu92 on Windows and Linux, supports CUDA-9.2.
• mxnet-cu100 on Windows and Linux, supports CUDA-10.0.
• mxnet-mkl on Windows, Linux, and Mac OSX, with CPU-only MKLDNN support.
• mxnet-cu75mkl on Linux, supports CUDA-7.5 and MKLDNN support. (discontinued in 1.2)
• mxnet-cu80mkl on Windows and Linux, supports CUDA-8.0 and MKLDNN support.
• mxnet-cu90mkl on Windows and Linux, supports CUDA-9.0 and MKLDNN support.
• mxnet-cu91mkl on Windows and Linux, supports CUDA-9.1 and MKLDNN support. (will discontinue in 1.3)
• mxnet-cu92mkl on Windows and Linux, supports CUDA-9.2 and MKLDNN support.
• mxnet-cu100mkl on Windows and Linux, supports CUDA-10.0 and MKLDNN support.

Scala Maven (Nexus) Status

Packages can be found on Nexus

• mxnet-full_2.11-linux-x86_64-cpu (Linux-CPU, same features as pip mxnet variant on Linux)
• mxnet-full_2.11-linux-x86_64-gpu (Linux-GPU, CUDA 9.0, same features as pip mxnet-cu90 variant on Linux)
• mxnet-full_2.11-darwin-x86_64-cpu (OSX-CPU, OSX 10.13+, same features as pip mxnet variant on OSX)

Compilation Flags

| Flags | mxnet | mxnet-mkl | mxnet-cu75 | mxnet-cu75mkl | mxnet-cu80 | mxnet-cu80mkl | mxnet-cu90 | mxnet-cu90mkl | mxnet-cu91 | mxnet-cu91mkl | mxnet-cu92 | mxnet-cu92mkl | mxnet-cu100 | mxnet-cu100mkl |
|--------------------|-----------------------------------|-----------|------------|---------------|------------|---------------|------------|---------------|------------|---------------|------------|---------------|------------|---------------|
| USE_OPENMP* | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| USE_DIST_KVSTORE | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| USE_MKLDNN | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 |
| USE_BLAS | apple for OSX, openblas for Linux
| USE_LAPACK | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| USE_NVRTC | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |1 | 1 |1 | 1 |
| USE_NCCL** | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| USE_CUDNN | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| USE_S3 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| USE_HDFS | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| USE_OPENCV | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| USE_SIGNAL_HANDLER | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| USE_PROFILER | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| DEV | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |0 | 0 |0 | 0 |
| DEBUG | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |

Notes:

* `USE_OPENMP` only applies to Linux

** USE_NCCL is supported since release version 1.0.0.post2, nightly since 1.0.1b20171228. Version: 2.1.2-1

Usage

Release

For release, users can install it via

MXNET_VARIANT=mxnet-cu90 MXNET_VERSION=1.0.0.post2 pip install --upgrade ${MXNET_VARIANT}==${MXNET_VERSION}

Be sure to update pip (setuptools) to the most recent version, by pip install --upgrade pip.

Nightly

For nightly pre-release, users can install it via pip install --pre --upgrade. Be sure to update pip (setuptools) to the most recent version, by pip install --upgrade pip.

Report issue

For bug submissions on issues found in the pre-releases, please attach:

• MXNet pip package version
• Environment report:

What to do:
1. Download the diagnosis script from https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py
2. Run the script using `python diagnose.py` and paste its output here.

• Problem code
  
  
  
  • Code snippet to reproduce the issue
  
  
  • Code should use dummy data (e.g. mx.nd.random.uniform), unless bug is only triggered with specific data value, in which case that specific value should be attached.
  
  

[szha]

8903

[rahul003]

Is there a way to see the build flags used to build the pip packages?

[szha]

I will add a section for the flags

[szha]

9048

[adaaaaaa]

头像换了，不好看，还是原来的猪头像好看。。。蛤蛤。。。

[adaaaaaa]

https://pypi.python.org/pypi/mxnet-cu91/
Not Found
Sorry, the page you're looking for couldn't be found.
Perhaps check the URL?

[szha]

@adaaaaaa cu91 versions are WIP.

[adaaaaaa]

cu90和cu91有没有benchmark，对比一下？

[adaaaaaa]

求cu91的更一下win版本的？

[szha]

pinging @yajiedesign

[adaaaaaa]

你@了一个不存在的人？

[yzzymt]

realese date for cu91 for win? Xiexie

[ghost]

@szha when can we get cu91 for win ?

[szha]

@yajiedesign is working on it.

[rahul003]

@szha Can we turn on the USE_LIBJPEG_TURBO flag. I find that it helps improve the speed of IO pipeline significantly?

Results for Resnet50 v1 Imagenet, 480px resized data, batch size 1920, float16, symbolic, p3.16x
Current package: 3600samples/sec
With libjpeg-turbo: 4600samples/sec

As an example, Here's how I used LIBJPEG_TURBO on ubuntu.

sudo apt-get install autoconf automake libtool nasm
JPEG_TURBO_VERSION=1.5.2 && \
wget -q -O - https://github.com/libjpeg-turbo/libjpeg-turbo/archive/${JPEG_TURBO_VERSION}.tar.gz | tar -xzf - && \
cd libjpeg-turbo-${JPEG_TURBO_VERSION} && \
autoreconf -fiv && \
./configure --enable-shared --prefix=/usr 2>&1 >/dev/null && \
sudo make -j"$(nproc)" install 2>&1 >/dev/null && \
rm -rf libjpeg-turbo-${JPEG_TURBO_VERSION}

Flags
USE_LIBJPEG_TURBO=1 USE_LIBJPEG_TURBO_PATH=/usr

[szha]

@rahul003 thanks for the suggestion. I will certainly take a look. For these dependencies, my approach have been to statically link them, so some steps will be different.

[xinyu-intel]

Is mxnet-mkl 1.2.0 ready for windows?

[fhieber]

Are the pip distributions compiled with USE_DIST_KVSTORE = 1 and if not, why?

[szha]

@fhieber Yes, they should have USE_DIST_KVSTORE=1 already. Did you have trouble using the distributed kvstore feature?

[fhieber]

@szha Oh great, thanks! I haven't tried it in a while but remembered that last time I checked the distributions were compiled without it. Glad to hear its enabled by default now!

[szha]

@fhieber no problem. I added the corresponding row in the table.

[eric-haibin-lin]

@szha what version of opencv is used for pip?

[szha]

@eric-haibin-lin 3.3.0

[yajiedesign]

@szha the windows mkl is compiled,is update?

[MoritzMaxeiner]

@szha What malloc are you using for pip? tcmalloc, jemalloc, or default?

[szha]

@MoritzMaxeiner default malloc so far. I intended to use jemalloc by building it from source but our makefile logic for jemalloc prevents a static link. https://github.com/apache/incubator-mxnet/blob/master/Makefile#L254-L278 (e.g. it forces to look in specific paths for libjemalloc.so).

[MoritzMaxeiner]

@szha Thank you very for the info, I suspected as much.

OT: I was just making sure, since I use MXNet from a shared object (dynamically loaded at runtime by another executable that I can't change to tcmalloc or jemalloc, which links dynamically against OpenCV), and using tcmalloc in MXNet leads to these issues with the LD_PRELOAD workaround not working (freezing the executable), so I was wondering if the prebuilt binary versions had some workaround for that.

[cconvey]

@szha : I'm trying to locally reproduce the mxnet and mxnet-mkl wheels that end up on PyPI. (Linux and OS X only.) Regarding the process you guys use:

1) Is there anything noteworthy about the platforms and/or steps used to prepare the wheels?

2) What tests (if any) do you run to verify that a wheel is good prior to upload?

Thanks!

[szha]

@cconvey I use ubuntu 14.04 as the environment because I want the GLIBC to be old enough for portability. I statically link all the dependencies too. For verification, if I changed any dependency, I do a manual verification of the new wheel. For all wheels, I run unit-test as sanity test before any upload.

[cconvey]

@szha When building the wheels, do you use the provided Makefiles or CMake?

edit for clarification: Do you use the script tools/pip_package/make_pip_package.sh to build the wheels that end up on PyPI?

[szha]

I'm using Makefile along with a separate set of scripts.

[perdasilva]

@szha I noticed USE_LIBJPEG_TURBO=1 (e.g.) but that's not described on the table.

Cheers!