One: Use pip cache while one-compiler install

[cgbahk]

While installing one-compiler with apt, I found several occurances of these line in install log.

The directory '/home/chbahk/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/chbahk/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

It seems failed to use pip cache. As I understand, one-compiler install many big python packages like TF/torch. It would be much faster if we can use pip cache. (I felt like pip download time was more than 70% of all install time)

Why it fails to use pip cache is not clear. My best guess is that install user(i.e. root) has no write permission to the cache directory. I may fix permission setting locally (not tested though), but it would be better if we find global fix for this :smile:

Log says to try sudo -H, but I'm not sure this is applicable or resolve this.

[cgbahk]

cc @seanshpark @mhs4670go

[seanshpark]

7218 is related and patch #7219 has landed. please check with nightly build debian package.

I've pinged you in our internal setup repo.

[cgbahk]

I've installed one-compiler 1.16.1-21071519 from nightly, but the symptom was same.

Progress bar means it did download, not using cache

$ sudo apt update && sudo apt install -y one-compiler

Hit:1 XXXXXXXXXXXXXXXXXXXXX bionic InRelease
Hit:2 XXXXXXXXXXXXXXXXXXXXX bionic-dev InRelease
Get:3 XXXXXXXXXXXXXXXXXXXXX bionic-nightly InRelease [4,357 B]
Get:6 XXXXXXXXXXXXXXXXXXXXX bionic-nightly/universe amd64 Packages [22.3 kB]
(Some removed)
Fetched 91.1 kB in 1s (61.7 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
26 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  python3-click python3-colorama
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
  one-compiler
0 upgraded, 1 newly installed, 0 to remove and 26 not upgraded.
Need to get 2,522 kB of archives.
After this operation, 12.1 MB of additional disk space will be used.
Get:1 XXXXXXXXXXXXXXXXXXXXX bionic-nightly/universe amd64 one-compiler amd64 1.16.1-21071519 [2,522 kB]
Fetched 2,522 kB in 0s (45.1 MB/s)
Selecting previously unselected package one-compiler.
(Reading database ... 284643 files and directories currently installed.)
Preparing to unpack .../one-compiler_1.16.1-21071519_amd64.deb ...
Unpacking one-compiler (1.16.1-21071519) ...
Setting up one-compiler (1.16.1-21071519) ...
The directory '/home/chbahk/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/chbahk/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting pip==20.2.1
  Downloading https://files.pythonhosted.org/packages/bd/b1/56a834acdbe23b486dea16aaf4c27ed28eb292695b90d01dff96c96597de/pip-20.2.1-py2.py3-none-any.whl (1.5MB)
    100% |████████████████████████████████| 1.5MB 1.3MB/s
Collecting setuptools==49.3.0
  Downloading https://files.pythonhosted.org/packages/f9/d8/89848d7f341de2d2acf18e9c9b30ee69dcf6d87f839121c0564c0cf013bc/setuptools-49.3.0-py3-none-any.whl (790kB)
    100% |████████████████████████████████| 798kB 2.6MB/s
Installing collected packages: pip, setuptools
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
  Found existing installation: setuptools 39.0.1
    Uninstalling setuptools-39.0.1:
      Successfully uninstalled setuptools-39.0.1
Successfully installed pip-20.2.1 setuptools-49.3.0
WARNING: The directory '/home/chbahk/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting tensorflow-cpu==2.3.0
  Downloading tensorflow_cpu-2.3.0-cp36-cp36m-manylinux2010_x86_64.whl (137.3 MB)
     |████████████████████████████████| 137.3 MB 16.9 MB/s
Collecting tensorboard<3,>=2.3.0
  Downloading tensorboard-2.5.0-py3-none-any.whl (6.0 MB)
     |████████████████████████████████| 6.0 MB 21.5 MB/s
Collecting termcolor>=1.1.0
  Downloading termcolor-1.1.0.tar.gz (3.9 kB)
Collecting wheel>=0.26
  Downloading wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Collecting grpcio>=1.8.6
  Downloading grpcio-1.38.1-cp36-cp36m-manylinux2014_x86_64.whl (4.2 MB)
     |████████████████████████████████| 4.2 MB 42.4 MB/s
Collecting six>=1.12.0
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting h5py<2.11.0,>=2.10.0
  Downloading h5py-2.10.0-cp36-cp36m-manylinux1_x86_64.whl (2.9 MB)
     |████████████████████████████████| 2.9 MB 25.3 MB/s
Collecting protobuf>=3.9.2
  Downloading protobuf-3.17.3-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 8.4 MB/s
Collecting tensorflow-estimator<2.4.0,>=2.3.0
  Downloading tensorflow_estimator-2.3.0-py2.py3-none-any.whl (459 kB)
     |████████████████████████████████| 459 kB 20.3 MB/s
Collecting absl-py>=0.7.0
  Downloading absl_py-0.13.0-py3-none-any.whl (132 kB)
     |████████████████████████████████| 132 kB 25.0 MB/s
Collecting numpy<1.19.0,>=1.16.0
  Downloading numpy-1.18.5-cp36-cp36m-manylinux1_x86_64.whl (20.1 MB)
     |████████████████████████████████| 20.1 MB 10.1 MB/s
Collecting wrapt>=1.11.1
  Downloading wrapt-1.12.1.tar.gz (27 kB)
Collecting scipy==1.4.1
  Downloading scipy-1.4.1-cp36-cp36m-manylinux1_x86_64.whl (26.1 MB)
     |████████████████████████████████| 26.1 MB 51.1 MB/s
Collecting google-pasta>=0.1.8
  Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
     |████████████████████████████████| 57 kB 39.5 MB/s
Collecting opt-einsum>=2.3.2
  Downloading opt_einsum-3.3.0-py3-none-any.whl (65 kB)
     |████████████████████████████████| 65 kB 24.8 MB/s
Collecting gast==0.3.3
  Downloading gast-0.3.3-py2.py3-none-any.whl (9.7 kB)
Collecting astunparse==1.6.3
  Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting keras-preprocessing<1.2,>=1.1.1
  Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 24.4 MB/s
Requirement already satisfied: setuptools>=41.0.0 in /usr/share/one/bin/venv/lib/python3.6/site-packages (from tensorboard<3,>=2.3.0->tensorflow-cpu==2.3.0) (49.3.0)
Collecting google-auth<2,>=1.6.3
  Downloading google_auth-1.33.0-py2.py3-none-any.whl (151 kB)
     |████████████████████████████████| 151 kB 63.2 MB/s
Collecting tensorboard-data-server<0.7.0,>=0.6.0
  Downloading tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB)
     |████████████████████████████████| 4.9 MB 29.7 MB/s
Collecting google-auth-oauthlib<0.5,>=0.4.1
  Downloading google_auth_oauthlib-0.4.4-py2.py3-none-any.whl (18 kB)
Collecting werkzeug>=0.11.15
  Downloading Werkzeug-2.0.1-py3-none-any.whl (288 kB)
     |████████████████████████████████| 288 kB 10.4 MB/s
Collecting markdown>=2.6.8
  Downloading Markdown-3.3.4-py3-none-any.whl (97 kB)
     |████████████████████████████████| 97 kB 24.7 MB/s
Collecting requests<3,>=2.21.0
  Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
     |████████████████████████████████| 62 kB 28.4 MB/s
Collecting tensorboard-plugin-wit>=1.6.0
  Downloading tensorboard_plugin_wit-1.8.0-py3-none-any.whl (781 kB)
     |████████████████████████████████| 781 kB 26.7 MB/s
Collecting cachetools<5.0,>=2.0.0
  Downloading cachetools-4.2.2-py3-none-any.whl (11 kB)
Collecting rsa<5,>=3.1.4; python_version >= "3.6"
  Downloading rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
     |████████████████████████████████| 155 kB 9.4 MB/s
Collecting requests-oauthlib>=0.7.0
  Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting dataclasses; python_version < "3.7"
  Downloading dataclasses-0.8-py3-none-any.whl (19 kB)
Collecting importlib-metadata; python_version < "3.8"
  Downloading importlib_metadata-4.6.1-py3-none-any.whl (17 kB)
Collecting charset-normalizer~=2.0.0; python_version >= "3"
  Downloading charset_normalizer-2.0.2-py3-none-any.whl (35 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
     |████████████████████████████████| 138 kB 86.6 MB/s
Collecting idna<4,>=2.5; python_version >= "3"
  Downloading idna-3.2-py3-none-any.whl (59 kB)
     |████████████████████████████████| 59 kB 6.2 MB/s
Collecting certifi>=2017.4.17
  Downloading certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
     |████████████████████████████████| 145 kB 85.3 MB/s
Collecting pyasn1>=0.1.3
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
     |████████████████████████████████| 77 kB 29.8 MB/s
Collecting oauthlib>=3.0.0
  Downloading oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
     |████████████████████████████████| 146 kB 64.6 MB/s
Collecting typing-extensions>=3.6.4; python_version < "3.8"
  Downloading typing_extensions-3.10.0.0-py3-none-any.whl (26 kB)
Collecting zipp>=0.5
  Downloading zipp-3.5.0-py3-none-any.whl (5.7 kB)
Using legacy 'setup.py install' for termcolor, since package 'wheel' is not installed.
Using legacy 'setup.py install' for wrapt, since package 'wheel' is not installed.
Installing collected packages: wheel, six, protobuf, grpcio, cachetools, pyasn1, rsa, pyasn1-modules, google-auth, tensorboard-data-server, oauthlib, charset-normalizer, urllib3, idna, certifi, requests, requests-oauthlib, google-auth-oauthlib, dataclasses, werkzeug, typing-extensions, zipp, importlib-metadata, markdown, numpy, absl-py, tensorboard-plugin-wit, tensorboard, termcolor, h5py, tensorflow-estimator, wrapt, scipy, google-pasta, opt-einsum, gast, astunparse, keras-preprocessing, tensorflow-cpu
    Running setup.py install for termcolor ... done
    Running setup.py install for wrapt ... done
Successfully installed absl-py-0.13.0 astunparse-1.6.3 cachetools-4.2.2 certifi-2021.5.30 charset-normalizer-2.0.2 dataclasses-0.8 gast-0.3.3 google-auth-1.33.0 google-auth-oauthlib-0.4.4 google-pasta-0.2.0 grpcio-1.38.1 h5py-2.10.0 idna-3.2 importlib-metadata-4.6.1 keras-preprocessing-1.1.2 markdown-3.3.4 numpy-1.18.5 oauthlib-3.1.1 opt-einsum-3.3.0 protobuf-3.17.3 pyasn1-0.4.8 pyasn1-modules-0.2.8 requests-2.26.0 requests-oauthlib-1.3.0 rsa-4.7.2 scipy-1.4.1 six-1.16.0 tensorboard-2.5.0 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.0 tensorflow-cpu-2.3.0 tensorflow-estimator-2.3.0 termcolor-1.1.0 typing-extensions-3.10.0.0 urllib3-1.26.6 werkzeug-2.0.1 wheel-0.36.2 wrapt-1.12.1 zipp-3.5.0
WARNING: You are using pip version 20.2.1; however, version 21.1.3 is available.
You should consider upgrading via the '/usr/share/one/bin/venv/bin/python -m pip install --upgrade pip' command.
WARNING: The directory '/home/chbahk/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting Pillow==6.2.2
  Downloading Pillow-6.2.2-cp36-cp36m-manylinux1_x86_64.whl (2.1 MB)
     |████████████████████████████████| 2.1 MB 1.5 MB/s
Installing collected packages: Pillow
Successfully installed Pillow-6.2.2
WARNING: You are using pip version 20.2.1; however, version 21.1.3 is available.
You should consider upgrading via the '/usr/share/one/bin/venv/bin/python -m pip install --upgrade pip' command.
WARNING: The directory '/home/chbahk/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Looking in links: https://download.pytorch.org/whl/torch_stable.html
Collecting torch==1.7.0+cpu
  Downloading https://download.pytorch.org/whl/cpu/torch-1.7.0%2Bcpu-cp36-cp36m-linux_x86_64.whl (159.3 MB)
     |████████████████████████████████| 159.3 MB 190.1 MB/s
Requirement already satisfied: numpy in /usr/share/one/bin/venv/lib/python3.6/site-packages (from torch==1.7.0+cpu) (1.18.5)
Collecting future
  Downloading future-0.18.2.tar.gz (829 kB)
     |████████████████████████████████| 829 kB 1.2 MB/s
Requirement already satisfied: typing-extensions in /usr/share/one/bin/venv/lib/python3.6/site-packages (from torch==1.7.0+cpu) (3.10.0.0)
Requirement already satisfied: dataclasses in /usr/share/one/bin/venv/lib/python3.6/site-packages (from torch==1.7.0+cpu) (0.8)
Building wheels for collected packages: future
  Building wheel for future (setup.py) ... done
  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491059 sha256=22f8d8cc32aaecdd320caf18841c23a2710828a226ea9ac21cd4735d59df2366
  Stored in directory: /tmp/pip-ephem-wheel-cache-2ie4sb6x/wheels/6e/9c/ed/4499c9865ac1002697793e0ae05ba6be33553d098f3347fb94
Successfully built future
Installing collected packages: future, torch
Successfully installed future-0.18.2 torch-1.7.0+cpu
WARNING: You are using pip version 20.2.1; however, version 21.1.3 is available.
You should consider upgrading via the '/usr/share/one/bin/venv/bin/python -m pip install --upgrade pip' command.
WARNING: The directory '/home/chbahk/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting onnx==1.8.0
  Downloading onnx-1.8.0-cp36-cp36m-manylinux2010_x86_64.whl (7.7 MB)
     |████████████████████████████████| 7.7 MB 8.1 MB/s
Collecting onnx-tf==1.8.0
  Downloading onnx_tf-1.8.0-py3-none-any.whl (219 kB)
     |████████████████████████████████| 219 kB 17.0 MB/s
Requirement already satisfied: numpy in /usr/share/one/bin/venv/lib/python3.6/site-packages (from onnx==1.8.0) (1.18.5)
Requirement already satisfied: typing-extensions>=3.6.2.1 in /usr/share/one/bin/venv/lib/python3.6/site-packages (from onnx==1.8.0) (3.10.0.0)
Requirement already satisfied: six in /usr/share/one/bin/venv/lib/python3.6/site-packages (from onnx==1.8.0) (1.16.0)
Requirement already satisfied: protobuf in /usr/share/one/bin/venv/lib/python3.6/site-packages (from onnx==1.8.0) (3.17.3)
Collecting tensorflow-addons
  Downloading tensorflow_addons-0.13.0-cp36-cp36m-manylinux2010_x86_64.whl (679 kB)
     |████████████████████████████████| 679 kB 4.6 MB/s
Collecting PyYAML
  Downloading PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl (640 kB)
     |████████████████████████████████| 640 kB 966 kB/s
Collecting typeguard>=2.7
  Downloading typeguard-2.12.1-py3-none-any.whl (17 kB)
Installing collected packages: onnx, typeguard, tensorflow-addons, PyYAML, onnx-tf
Successfully installed PyYAML-5.4.1 onnx-1.8.0 onnx-tf-1.8.0 tensorflow-addons-0.13.0 typeguard-2.12.1
WARNING: You are using pip version 20.2.1; however, version 21.1.3 is available.
You should consider upgrading via the '/usr/share/one/bin/venv/bin/python -m pip install --upgrade pip' command.
patching file onnx_tf/handlers/backend/conv_mixin.py
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...

$ bash my/driver/show_env.sh

one-compiler  1.16.1-21071519

[mhs4670go]

@cgbahk Your best guess is right:)

FYI,
There are (bash) scripts named Package maintainer scripts that are run when the package is installed, upgraded or removed. With this, the debian package developer can do pre-process, post-process and etc.

one-compiler package uses one of them named postinst that is called after the package is unpacked(flowchart). The reason is because one-compiler binaries are run under python virtual environment, which should be created before user uses our package. Here is the scirpt.
https://github.com/Samsung/ONE/blob/2dc8f730a5184f96a59b0cf57833d0228f8e1741/infra/debian/compiler/postinst#L1-L8

postinst script simply runs one-prepare-venv. one-prepare-venv script creates the python virtual environment and installs required packages(tensorflow, onnx, etc.).

But, the thing is maintainer scripts are invoked as root.

https://wiki.debian.org/UntrustedDebs#Purpose

It runs all the maintainer scripts as root and installs files anywhere in the hierarchy with any set of permissions

Which is not what I want but it actually is. And let's see the log again.

The directory '/home/chbahk/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/chbahk/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

Since one-prepare-venv is called as root, installing python packages is processed under root privilege. As warning message said, pip doesn't use .cache unless current user is the owner of that directory. And it looks its $HOME directory as its hint.

This is why sudo -H works out okay that changes its $HOME to /root. Then, pip can see /root/.cache/pip.

Well, then, what can we do? Here are things that I came up with.

• Call postinst as user permission rather than root.

-> I don't know it is possible T_T

• Before installation, change the owner of the .cache directory to root and restore it after installaion.

-> Seems not natural.

• Remove postinst and make users run one-prepare-venv script manually.

->I've never seen the debain package that user must run some script manually after installation.

Hope we can get a better idea later.

[seanshpark]

Call postinst as user permission rather than root.

Found something with su command...

su -c [command] (useraccount)

@cgbahk, @mhs4670go can this help?

su -c one-prepare-venv chbahk

--> I think won't work but can try something like this with pip comamnd..

[cgbahk]

We may 'somehow' use

$ pip --help
...
  --cache-dir <dir>           Store the cache data in <dir>.

around here
https://github.com/Samsung/ONE/blob/dbcde5c688b56449f4614740ae14cecb1b0462ce/compiler/one-cmds/one-prepare-venv#L48

Maybe like

PIP_CACHE_DIR="/tmp/one-compiler-pip-cache"  # Not sure where is good place to be
mkdir -p ${PIP_CACHE_DIR}  # Made by this user, so ensure having permission on this

PIP_OPTIONS="${PIP_TIMEOUT} ${PIP_TRUSTED_HOST} --cache-dir ${PIP_CACHE_DIR}"

[mhs4670go]

@cgbahk But, I thought you were saying about using .cache directory that an user already has. Since user normally installs our package once, there is no need to cache it.

[mhs4670go]

@seanshpark

I think won't work but can try something like this with pip comamnd..

Surprisingly, it worked!!

I thought su command needs password of the user to run but in debian package it didn't.

su - $(whoami) -c /usr/share/one/bin/one-prepare-venv

I don't know it is all okay but seems okay,,

[mhs4670go]

@cgbahk Could you install this file?
one-compiler_1.16.1_amd64.zip

[cgbahk]

@cgbahk Could you install this file?
one-compiler_1.16.1_amd64.zip

It's using cache on second run! :tada:
It seems using cache /root/.cache/pip as you expected

[cgbahk]

I would close this when this available with apt :)

[cgbahk]

I would close this when this available with apt :)

Checked resolved, thanks !