Salt: Install multiple requirements files via pip and virtualenv states

[OddBloke]

We have some projects where we have more than one requirements file (a core one which everywhere should install, and a site-specific one). At the moment, we have to write out a full state for each file:

main_requirements:
  pip.installed:
    - name:
    - requirements: ...

other_requirements:
  pip.installed:
    - name:
    - requirements: ...

It is also painful in a virtualenv definition:

my_virtualenv:
  virtualenv.managed:
    - name: ...
    - requirements: ...

extra_requirements:
  pip.installed:
    - name:
    - requirements: ...

And, of course, we then either have to introduce dependencies between the requirements states or depend on all of the requirements states where we need them.

Preferable would be a list of requirements files:

main_requirements:
  pip.installed:
    - name:
    - requirements:
      - main_requirements
      - other_requirements

This would boil down to pip install -r main_requirements -r other_requirements.

[thatch45]

I am going to approve this for the future, thanks for the request

[yellottyellott]

It'd be pretty sweet if we could get a cwd on virtualenv.managed like there is on pip.installed.

I typically have my requirements files split out by environment like this:

Base file:

# base.txt
Django==1.5.1

Environment file:

# dev.txt
-r base.txt

django-debug-toolbar==0.9.4

The blank - name: is kind of weird, but works for now.

[copelco]

@yellottyellott Agreed. Just ran into this issue too.

[gravyboat]

I would like to see this as well.

[JasonSwindle]

It looks like we are getting hit by this as well.

.......
  virtualenv.managed:
    - name: {{ pillar['document_root'] }}/.virtualenvs/{{ pillar['project_name'] }}-python-2.7
    - no_site_packages: True
    - user: {{ pillar['project_name'] }}
    - cwd: {{ pillar['document_root'] }}/{{ pillar['project_name'] }}
    - requirements: {{ pillar['document_root'] }}/{{ pillar['project_name'] }}/requirements.txt
.......

    State: - virtualenv
    Name:      /opt/apps/.virtualenvs/moo-python-2.7
    Function:  managed
        Result:    False
        Comment:   Created new virtualenv
Could not open requirements file: [Errno 2] No such file or directory: '/tmp/requirements/production.txt'
Storing complete log in /opt/apps/.pip/pip.log

        Changes:   new: Python 2.7.3

requirements.txt

# This file is here because many Platforms as a Service look for
#       requirements.txt in the root directory of a project.
-r requirements/production.txt

requirements/production.txt

# Pro-tip: Try not to put anything here. There should be no dependency in
#   production that isn't in development.
-r base.txt

psycopg2

requirements/base.txt

Django>=1.5.5
bpython>=0.12
django-braces>=1.2.2
django-model-utils>=1.5.0
logutils>=0.3.3
South>=0.8.2
django_auth_ldap>=1.1.3
ipython>=1.1.0
django-extensions>=1.2.5
pytz
django-crispy-forms>=1.4.0
python-memcached>=1.53

           Salt: 0.17.4
         Python: 2.7.3 (default, Apr 10 2013, 06:20:15)
         Jinja2: 2.6
       M2Crypto: 0.21.1
 msgpack-python: 0.1.10
   msgpack-pure: Not Installed
       pycrypto: 2.4.1
         PyYAML: 3.10
          PyZMQ: 13.0.0
            ZMQ: 3.2.2

Even with -cwd: and chaining requirements.txt, it keeps using /tmp and not the correct path.

Jason Swindle

[garethgreenaway]

Going to look into fixing this.

[garethgreenaway]

Confirmed that the cwd option for the virtualenv state does work using the example state above with pillar data. Can someone include an example state where its not working?

[garethgreenaway]

Multiple requirements appears to work as well. Simply separate them by a comma on in the state.

[gifflen]

I'm working with Jason Swindle above on the same project. When we attempt to run anything that chains requirement files with the -r option in a requirements file the cwd location is not being respected though it is for any file explicitly called from salt.

We have requirements.txt in the root of our project directory: /opt/apps/django-project/requirements.txt Its contents contains -r requirements/production.txt which in turn has -r base.txt. Using this method we are chaining the various requirements for different environments keeping common requirements in base.txt and secific environment requirements in each spcecific environment file.

When attempting to use virtualenv.managed as seen in Jason's comment above it finds /opt/apps/django-project/requirements.txt file appropriately and attempts to install the requirements in the file. Since that file is present to really only to act as a proxy for PAAS installs it should then hand it off to requirements/production.txt locatied in /opt/apps/django-project/requirements/production.txt. Instead the logs show it is searching /tmp/requirements/production.txt

When using virtualenv directly this functionality works as expected it is only when using salt that we encounter this problem.

If you need further information let me know. I can provide any amount of debug needed.

[garethgreenaway]

That helps make the situation clearer :) I'll replicate that scenario and see what happens. Thanks.

[garethgreenaway]

Can you provide the state file and pillar files you're using? I tested with the following and it worked without error:

State:
foo:
virtualenv.managed:
- name: {{ pillar['document_root'] }}/.virtualenvs/{{ pillar['project_name'] }}-python-2.7
- no_site_packages: True
- cwd: {{ pillar['document_root'] }}/{{ pillar['project_name'] }}
- requirements: requirements.txt

Pillar:
document_root: /opt/apps
project_name: moo

/opt/apps/moo/requirements:
-rw-r--r-- 1 root root 19 Dec 31 09:03 base.txt
-rw-r--r-- 1 root root 12 Dec 31 09:20 production.txt

/opt/apps/moo/requirements.txt:
-r requirements/production.txt

/opt/apps/moo/requirements/production.txt:
-r base.txt

/opt/apps/moo/requirements/base.txt:
Django>=1.5.5
pytz

[JasonSwindle]

Here you go, I hope this helps. :)

pillar

## Project name
project_name: 'appname'

## Application's home
document_root: '/opt/apps'

sls

app_install:
  file.directory:
    - name: {{ pillar['document_root'] }}/.virtualenvs/
    - owner: {{ pillar['project_name'] }}
    - group: www-data
    - require:
      - user: app_user
  git.latest:
    - name: {{ pillar['git_url'] }}
    - rev: {{ pillar['branch_or_tag'] }}
    - target: {{ pillar['document_root'] }}/{{ pillar['project_name'] }}/{{ pillar['project_name'] }}-repo
    - user: {{ pillar['project_name'] }}
    - force: True
    - require:
      - file: app_install
      - pkg: git_install
  virtualenv.managed:
    - name: {{ pillar['document_root'] }}/.virtualenvs/{{ pillar['project_name'] }}
    - no_site_packages: True
    - user: {{ pillar['project_name'] }}
    - cwd: {{ pillar['document_root'] }}/{{ pillar['project_name'] }}/{{ pillar['project_name'] }}-repo
    - requirements: {{ pillar['document_root'] }}/{{ pillar['project_name'] }}/{{ pillar['project_name'] }}-repo/requirements.txt
    - require:
      - git: app_install
      - pip: python-virtualenv_install
      - pkg: python-dev_install
      - pkg: libldap2-dev_install
      - pkg: libsasl2-dev_install
      - pkg: libssl-dev_install
      - pkg: python-pip_install
      - pkg: python-ldap_install
      - pkg: libpq-dev_install
      - file: newrelic_app_config
      - file: ldap_conf
    - require_in:
      - pip: uwsgi_install

Error

    State: - virtualenv
    Name:      /opt/apps/.virtualenvs/appname
    Function:  managed
        Result:    False
        Comment:   virtualenv exists
Could not open requirements file: [Errno 2] No such file or directory: '/tmp/requirements/production.txt'
Storing complete log in /opt/apps/appname/.pip/pip.log

        Changes:
----------
    State: - pip
    Name:      uwsgi
    Function:  installed
        Result:    False
        Comment:   One or more requisite failed
        Changes:

[basepi]

So unless I'm mistaken, this thread has gone fairly off-topic. Can one of you create a new thread for the cwd stuff? This thread should only be used for the multiple requirements files feature request.

[garethgreenaway]

From my testing with both scenarios, both the cwd option and being able to specify multiple requirement files either by chaining them or specifying multiple requirement files inside the state file, works.

[basepi]

Can someone else verify on the latest develop branch?

[JasonSwindle]

Howdy,

I am trying to but #9587 sadly is stopping me.

Jason Swindle

[rallytime]

Looks like #9587 has been resolved. Can anyone else reproduce these issues? Looks like this has been fixed on develop.

[rallytime]

Since this has been fixed on develop, I am closing this one. If this issue comes up again, leave a comment and we can definitely re-open it and address any problems.

[chris-martin]

@garethgreenaway:

being able to specify multiple requirement files either by chaining them or specifying multiple requirement files inside the state file, works

Can you please clarify what this means?

• What does it means to "chain" requirements files?
• "specifying multiple requirement files" - How?

The documentation for this state still seems to indicate that only one requirements file may be given.

[gravyboat]

@chris-martin For specifying multiple requirements files the code over here: https://github.com/saltstack/salt/blob/develop/salt/modules/pip.py#L200 seems to indicate you'd want to use:

- requirements:
  - requirement_one
  - requirement_two

For the chaining of requirements files I believe @garethgreenaway is referring to something like this:

https://stackoverflow.com/questions/11704287/split-requirements-files-in-pip

Where a file contains two additional file references. (which is disgusting by the way @garethgreenaway ;) )

Docs probably need to be updated @jfindlay. This issue should either be re-opened and noted as docs, or a new one created to confirm functionality and document it.

[bashu]

+1 still an issue...

[gladiatr72]

For those that are watching this thread, I've got a patch to modules/pip.py to address the requirements chaining bug. My work is against develop--I haven't verified that the multiple requirements feature has been taken care of, but it looks like it! I ran into the issue with the chained requirements file and decided to fix it. After I give it another once-over (post-holiday) I'll submit it against this issue.

Summary: With the default method of copy + chown for handling requirements files and a non-root user argument, the existing code takes the referenced requirements file, copies it to a temp file, does its user-switch and then runs against the temp file.

The patched module, instead, uses a temp directory filtering on -r lines with the state provided requirements file as a base. The end result is a list of requirements files that are copied to the temp directory, chown'd and then given to the dropped-privilege pip install process.

@baseapi, I know this is an ancient thread. If you'd prefer a new issue (and you are available to even see this), I'd be happy to make the pull req against a new issue.

[basepi]

Looks like your pull request was submitted and accepted! No need for a new issue as far as I'm concerned. Thanks @gladiatr72 !

(I should also note to be careful of username typos..... @basepi vs @baseapi, or I might miss something in the future. :) )

[anlutro]

Just going to chime in and say that while the pip module now supports multiple requirements file, the virtualenv.managed state function still insists that the requirements argument be a string here.

Would a fix for this be appropriate against current release branches, or would it have to go into develop?

[thatch45]

This is typically something that we would put in develop, since it changes the behavior of a state instead of being a bugfix. Thanks for asking :)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.