St2: Packs.setup_virtualenv edge case when a sensor is running already.

[NikosVlagoidis]

SUMMARY

There is an edge case when you use packs.setup_virtualenv and a sensor from the pack is running and using a file from the virtualenv.
I know this is not the use of setup_virtualenv action but I guess the action should check if a sensor is running and prevent the partial deletion of the virtualenv.
Also I found that you can use packs.update_virtualenv in the first place.

ISSUE TYPE

Pick one below and delete the rest:

• Bug Report *
• Feature Idea
• Other

STACKSTORM VERSION

Paste the output of st2 --version:
st2 2.7.2, on Python 2.7

OS / ENVIRONMENT / INSTALL METHOD

Docker

STEPS TO REPRODUCE

1. Have a sensor from a pack called pack_name running using a python package from the virtualenv.
2. Try running st2 run packs.setup_virtualenv pack_name
3. Try to run an action from this pack

EXPECTED RESULTS

The action should prevent from deleting half the virtualenv.

ACTUAL RESULTS

Stack trace from setup_virtualenv:

st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "pack_name" (/opt/stackstorm/packs.dev/pack_name)
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Removing virtualenv in "/opt/stackstorm/virtualenvs/pack_name"
st2.actions.python.SetupVirtualEnvironmentAction: ERROR    Error while removing virtualenv at "/opt/stackstorm/virtualenvs/pack_name": "[Errno 39] Directory not empty: '/opt/stackstorm/virtualenvs/pack_name/bin'"Traceback (most recent call last):
  File "/opt/stackstorm/runners/python_runner/python_runner/python_action_wrapper.py", line 320, in <module>
    obj.run()
  File "/opt/stackstorm/runners/python_runner/python_runner/python_action_wrapper.py", line 179, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/setup_virtualenv.py", line 89, in run
    no_download=no_download)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/util/virtualenvs.py", line 88, in setup_pack_virtualenv
    remove_virtualenv(virtualenv_path=virtualenv_path, logger=logger)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/util/virtualenvs.py", line 215, in remove_virtualenv
    raise e
OSError: [Errno 39] Directory not empty: '/opt/stackstorm/virtualenvs/pack_name/bin'

Stack trace from running an action from the pack after the previous action:

[Errno 2] No such file or directory
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/container/base.py", line 119, in _do_run
    (status, result, context) = runner.run(action_params)
  File "/opt/stackstorm/runners/python_runner/python_runner/python_runner.py", line 263, in run
    stdin_value=stdin_params)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/util/green/shell.py", line 109, in run_command
    env=env, cwd=cwd, shell=shell, preexec_fn=preexec_func)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 58, in __init__
    subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception

[LindsayHill]

The packs.setup_virtualenv action has this parameter:

update:
      type: "boolean"
      default: false
      description: "Check this option if the virtual environment already exists and if you only want to perform an update and installation of new dependencies. If you don't check this option, the virtual environment will be destroyed then re-created. If you check this and the virtual environment doesn't exist, it will create it."

(note the default there).

I guess the action should check if a sensor is running and prevent the partial deletion of the virtualenv

I think that the desired behavior should be that it completely removes the old virtualenv. It's not about checking if a sensor is running or not. It's more about making sure that the virtualenv is completely removed, and that setup_virtualenv completes successfully.

[NikosVlagoidis]

I think that the desired behavior should be that it completely removes the old virtualenv. It's not about checking if a sensor is running or not. It's more about making sure that the virtualenv is completely removed, and that setup_virtualenv completes successfully.

The problem is that the sensor (proccess) is using files from the virtualenv and os does not allow you to completely remove the files. I tried it even manually with rm -rf. It was weird. I had to disable stackstorm with st2ctl stop. Not sure if disabling the sensor would do it.

[LindsayHill]

I just tried to reproduce this using the githubwebhooksensor pack.

I installed the pack, checked that the sensor was running, then ran st2 run packs.setup_virtualenv packs=githubwebhook.

extreme@EWC:~$ st2 run packs.setup_virtualenv packs=githubwebhook
..
id: 5b2963d999c96b2277df18e7
status: succeeded
parameters:
  packs:
  - githubwebhook
result:
  exit_code: 0
  result: 'Successfuly set up virtualenv for the following packs: githubwebhook'
  stderr: 'st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "githubwebhook" (/opt/stackstorm/packs/githubwebhook)
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Removing virtualenv in "/opt/stackstorm/virtualenvs/githubwebhook"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "githubwebhook" in "/opt/stackstorm/virtualenvs/githubwebhook"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/githubwebhook" using Python binary "/opt/stackstorm/st2/bin/python"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --no-download /opt/stackstorm/virtualenvs/githubwebhook" to create virtualenv.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base requirements
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/githubwebhook/bin/pip install six>=1.9.0,<2.0.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing pack specific requirements from "/opt/stackstorm/packs/githubwebhook/requirements.txt"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirements from file /opt/stackstorm/packs/githubwebhook/requirements.txt with command /opt/stackstorm/virtualenvs/githubwebhook/bin/pip install -U -r /opt/stackstorm/packs/githubwebhook/requirements.txt.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Virtualenv for pack "githubwebhook" successfully created in "/opt/stackstorm/virtualenvs/githubwebhook"
    '
  stdout: ''
extreme@EWC:~$

That seemed to execute with no problems. It did not have any issues with being unable to delete files. What is your sensor using?

Also, to check: when you ran st2 run packs.setup_virtualenv pack_name, it exited with an error, correct? Or did it report that it succeeded?

If it reported an error, then that's working OK. It if reported it succeeded when it didn't, then that is a problem.

[NikosVlagoidis]

I was using a sensor with a flask app running on eventlet WSGI server. The action reported an error. But after that I was unable to use any action from the pack. I am going to test it again and see if that was some super edge race condition.

[NikosVlagoidis]

I tried to reproduce the problem without success. I guess something was on a weird state locking a file in a result not letting it be deleted. @LindsayHill you can close this issue for now. If I come up with something more consistent I will let you know.

[LindsayHill]

Thanks for the update. Hopefully it either never occurs again, or we can get a reproducible example

[LindsayHill]

You’re better off posting on forum.stackstorm.com, rather than multiple closed issues. Closed issues don’t usually get any attention, because only those people in the issue ever see notification, and in my case, I don’t work on ST2 anymore.