Rasa: Permission denied error on model upload

[dhwanils95]

Rasa version: 1.2.2
Rasa X version (if used & relevant): 0.20.1
Python version: 3.6
Operating system (windows, osx, ...): Ubuntu 18.04 and CentOS 7.6

Issue:
I am trying to upload a model via Rasa-x on two different installations, one on Ubuntu and the other on CentOS, both are throwing the same error of permission denied.

Error:

rasa-x_1           | Exception occurred while handling uri: 'http://localhost/api/projects/default/models?api_token=3518637d2fc37ac6eb504894e749e64105f5ca1e'
rasa-x_1           | Traceback (most recent call last):
rasa-x_1           |   File "/usr/local/lib/python3.6/shutil.py", line 550, in move
rasa-x_1           |     os.rename(src, real_dst)
rasa-x_1           | OSError: [Errno 18] Invalid cross-device link: '/tmp/tmpkpa18idb/model.tar.gz' -> '/app/models/model.tar.gz'
rasa-x_1           | 
rasa-x_1           | During handling of the above exception, another exception occurred:
rasa-x_1           | 
rasa-x_1           | Traceback (most recent call last):
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 917, in handle_request
rasa-x_1           |     response = await response
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/rasax/community/api/decorators.py", line 176, in decorated_function
rasa-x_1           |     return await await_and_return_response(args, kwargs, request)
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/rasax/community/api/decorators.py", line 106, in await_and_return_response
rasa-x_1           |     response = await response
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/rasax/community/api/blueprints/models.py", line 55, in upload_model
rasa-x_1           |     saved = await model_service.save(project_id, model_name, tpath)
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service.py", line 164, in save
rasa-x_1           |     stored_path = self._store_zipped_model(model_name, path)
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service.py", line 264, in _store_zipped_model
rasa-x_1           |     shutil.move(file_path, zpath)
rasa-x_1           |   File "/usr/local/lib/python3.6/shutil.py", line 564, in move
rasa-x_1           |     copy_function(src, real_dst)
rasa-x_1           |   File "/usr/local/lib/python3.6/shutil.py", line 263, in copy2
rasa-x_1           |     copyfile(src, dst, follow_symlinks=follow_symlinks)
rasa-x_1           |   File "/usr/local/lib/python3.6/shutil.py", line 121, in copyfile
rasa-x_1           |     with open(dst, 'wb') as fdst:
rasa-x_1           | PermissionError: [Errno 13] Permission denied: '/app/models/model.tar.gz'

Command or request that led to error:

Rasa X - Model Upload UI

Content of configuration file (config.yml) (if relevant):

# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: "en"

pipeline:
- name: "SpacyNLP"
- name: "SpacyTokenizer"
- name: "SpacyFeaturizer"
- name: "RegexFeaturizer"
- name: "CRFEntityExtractor"
- name: "EntitySynonymMapper"
- name: "SklearnIntentClassifier"

# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
  - name: MemoizationPolicy
  - name: KerasPolicy
  - name: MappingPolicy
  - name: TwoStageFallbackPolicy
    nlu_threshold: 0.3
    core_threshold: 0.3
    fallback_core_action_name: "action_default_fallback"
    fallback_nlu_action_name: "action_default_fallback"
    deny_suggestion_intent_name: "out_of_scope"

Content of domain file (domain.yml) (if relevant):

intents:
- impact
- inform
- greet
entities:
- above_below
- operation
- period
templates:
...
slots:
...

[sara-tagger]

Thanks for the issue, @tmbo will get back to you about it soon!

You may find help in the docs and the forum, too 🤗

[tmbo]

@erohmensing I think you have been looking into this right?

[erohmensing]

Yeah i've seen the error before -- @dhwanils95 what does your deployment setup look like? Does mounting a /tmp volume help at all?

[dhwanils95]

The rasa_x container already has a /tmp/ directory in it with some files. Would mounting a directory explicitly help? If you want I will try and check as soon as possible.

Thanks!

[erohmensing]

Hm I'm not quite sure, we're trying to debug the problem currently. How did you go about installing (manual, script, openshift instructions, etc.) and what is your deployment setup? I haven't reproduced it yet, that information could help.

[dhwanils95]

I used both the script and manual installation. Since there is not automated script for CentOS, I did a manual installation using my custom action server which was replaced in the app service. I am using docker containers (Following are the versions I am using right now):

bitnami/rabbitmq    3.7.17              c79793bf81b2        3 days ago          202MB
rasa/rasa           1.2.3-full          1705d8edd332        11 days ago         2.7GB <--
python              3.6-slim            547f7a923796        12 days ago         174MB
rasa/rasa-x         stable              1cb17941f09d        2 weeks ago         1.45GB <--
rasa/nginx          stable              fe1f85d1cda2        2 weeks ago         87.1MB
rasa/duckling       latest              b7785de7a33a        3 months ago       5.55GB <--
rasa/logger         stable              515e477dcfc7        3 months ago        93.4MB

I did mount an empty directory to /tmp/ but there is no change. The issue is still there and I am not able to upload my model file.

Also in an unrelated context, where can I get your dockerfiles? I think I will be able to drastically reduce your image size since right now rasa, rasa-x and duckling are very bulky (as you can see in the above output), it will become a problem for a lot of people in the coming future for production deployments as everyone wants to reduce their resource consumption.
Let me know how I can help.

Thanks and regards.

[erohmensing]

Great, thank you for the info! As for the image size, there's already a PR about reducing the rasa size via multi-stage docker images. Feel free to give your input here: https://github.com/RasaHQ/rasa/pull/3332

The duckling dockerfile can be found here: https://github.com/RasaHQ/duckling/blob/master/Dockerfile

[dhwanils95]

Great, I will look into it and see how I can help out.

[erohmensing]

@dhwanils95 are you sure that you ran the permission commands in step 10 (also 11?)

So for e.g. the models folder, you would run

sudo chgrp -R root /etc/rasa/models && sudo chmod -R 770 /etc/rasa/models

Can you check the permissions of the directory and post them here?

[dhwanils95]

Hey @erohmensing ,

So, I had changed the permission of the directory but the docker user which ran the containers was different and was not in the group with the same permissions and hence it was throwing that error. Your solution pointed me to the right direction to check out that and I was able to solve it.

Closing the issue.
Thanks.

[tpinju00]

Hi @dhwanils95,

can you please specify what line and where did you add it to solve this issue?

[dhwanils95]

Hi @dhwanils95,

can you please specify what line and where did you add it to solve this issue?

Hello @tpinju00,

It was not a line that I had to edit. The docker user which runs my docker containers did not have sufficient permissions even though I had changed the permissions of the directory as pointed out by @erohmensing. You will need to check to which groups does your docker user belongs to and whether that group has the required (770) rights on those directories.

Add your docker user to the group and you should be good to go.

[tpinju00]

@dhwanils95 , I did this:

1) sudo passwd root
2) added my pw
3) docker exec -u root -t -i CONTAINER_ID /bin/bash
4) su - (typed in pw)
5) apt-get install sudo -y
6) sudo groupadd docker
7) sudo usermod -aG docker $USER

But the error still persists, any advice?