st2 Pack Dependencies Not Resolved via st2-pack-install tool

[danielburrell]

SUMMARY

The dependency resolution system between stackstorm packs (not python dependencies) which was recently implemented in st2 v3.2 (#4769 ) does not function when trying to build via st2-pack-install (https://github.com/StackStorm/st2/pull/4256) utility mentioned in https://docs.stackstorm.com/install/k8s_ha.html#custom-st2-packs.

STACKSTORM VERSION

3.2dev

Steps to reproduce the problem

/opt/stackstorm/st2/bin/st2-pack-install -vv  https://github.com/stackstorm/stackstorm-ms

Expected Results

Expect to see the dependencies of stackstorm-ms being installed (excel, and powerpoint for example).

Actual Results

Only the stackstorm-ms pack is installed.

2020-02-13 17:44:53,902 INFO [-] Installing pack "https://github.com/StackStorm/stackstorm-ms"
2020-02-13 17:44:53,903 DEBUG [-] Popen(['git', 'clone', '-v', 'https://github.com/StackStorm/stackstorm-ms', '/root/6622c8eb7209250c62980483f40577e2'], cwd=/opt/stackstorm, universal_newlines=True, shell=None)
2020-02-13 17:44:55,509 DEBUG [-] Cmd(['git', 'clone', '-v', 'https://github.com/StackStorm/stackstorm-ms', '/root/6622c8eb7209250c62980483f40577e2'])'s unused stdout: 
2020-02-13 17:44:55,518 DEBUG [-] Popen(['git', 'cat-file', '--batch-check'], cwd=/root/6622c8eb7209250c62980483f40577e2, universal_newlines=False, shell=None)
2020-02-13 17:44:56,256 DEBUG [-] Popen(['git', 'branch', '-a', '--contains', '25089933a452dedeb5aa145d6716c0187917b854'], cwd=/root/6622c8eb7209250c62980483f40577e2, universal_newlines=False, shell=None)
2020-02-13 17:44:56,906 DEBUG [-] Popen(['git', 'checkout', '25089933a452dedeb5aa145d6716c0187917b854'], cwd=/root/6622c8eb7209250c62980483f40577e2, universal_newlines=False, shell=None)
2020-02-13 17:44:57,545 DEBUG [-] Popen(['git', 'branch', '-f', 'master', '25089933a452dedeb5aa145d6716c0187917b854'], cwd=/root/6622c8eb7209250c62980483f40577e2, universal_newlines=False, shell=None)
2020-02-13 17:44:58,292 DEBUG [-] Popen(['git', 'checkout', 'master'], cwd=/root/6622c8eb7209250c62980483f40577e2, universal_newlines=False, shell=None)
2020-02-13 17:44:58,940 DEBUG [-] Moving pack from /root/6622c8eb7209250c62980483f40577e2 to /opt/stackstorm/packs.
2020-02-13 17:44:58,942 DEBUG [-] Changing owner group of "/opt/stackstorm/packs/microsoft_test" directory to st2packs
2020-02-13 17:44:58,942 DEBUG [-] Entering st2common.util.green.run_command.
2020-02-13 17:44:58,942 DEBUG [-] env argument not provided. using process env (os.environ).
2020-02-13 17:44:58,942 DEBUG [-] Creating subprocess.
2020-02-13 17:44:58,948 DEBUG [-] Spawning timeout handler thread.
2020-02-13 17:44:58,949 DEBUG [-] Attaching to process.
2020-02-13 17:44:58,949 DEBUG [-] Using delayed stdout and stderr read mode, calling process.communicate()
2020-02-13 17:44:58,949 DEBUG [-] Starting process wait inside timeout handler.
2020-02-13 17:44:58,963 DEBUG [-] No timeout.
2020-02-13 17:44:58,963 DEBUG [-] Returning.
2020-02-13 17:44:58,966 INFO [-] Successfully installed pack "microsoft_test"
2020-02-13 17:44:58,966 INFO [-] Setting up virtualenv for pack "microsoft_test"
2020-02-13 17:44:58,968 DEBUG [-] Setting up virtualenv for pack "microsoft_test" (/opt/stackstorm/packs/microsoft_test)
2020-02-13 17:44:58,968 INFO [-] Virtualenv path "/opt/stackstorm/virtualenvs/microsoft_test" doesn't exist
2020-02-13 17:44:58,969 DEBUG [-] Creating virtualenv for pack "microsoft_test" in "/opt/stackstorm/virtualenvs/microsoft_test"
2020-02-13 17:44:58,969 DEBUG [-] Creating virtualenv in "/opt/stackstorm/virtualenvs/microsoft_test" using Python binary "/opt/stackstorm/st2/bin/python"
2020-02-13 17:44:58,969 DEBUG [-] Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --no-download /opt/stackstorm/virtualenvs/microsoft_test" to create virtualenv.
2020-02-13 17:45:01,563 DEBUG [-] Installing base requirements
2020-02-13 17:45:01,563 DEBUG [-] Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/microsoft_test/bin/pip install six>=1.9.0,<2.0.
2020-02-13 17:45:02,018 DEBUG [-] No pack specific requirements found
2020-02-13 17:45:02,018 DEBUG [-] Changing owner group of "/opt/stackstorm/virtualenvs/microsoft_test" directory to st2packs
2020-02-13 17:45:02,018 DEBUG [-] Entering st2common.util.green.run_command.
2020-02-13 17:45:02,019 DEBUG [-] env argument not provided. using process env (os.environ).
2020-02-13 17:45:02,019 DEBUG [-] Creating subprocess.
2020-02-13 17:45:02,022 DEBUG [-] Spawning timeout handler thread.
2020-02-13 17:45:02,022 DEBUG [-] Attaching to process.
2020-02-13 17:45:02,022 DEBUG [-] Using delayed stdout and stderr read mode, calling process.communicate()
2020-02-13 17:45:02,022 DEBUG [-] Starting process wait inside timeout handler.
2020-02-13 17:45:02,044 DEBUG [-] No timeout.
2020-02-13 17:45:02,044 DEBUG [-] Returning.
2020-02-13 17:45:02,044 DEBUG [-] Virtualenv for pack "microsoft_test" successfully created in "/opt/stackstorm/virtualenvs/microsoft_test"
2020-02-13 17:45:02,044 INFO [-] Successfully set up virtualenv for pack "microsoft_test"

Thanks!

[armab]

Thanks for the bug report. I've adjusted issue description a bit.

[armab]

After looking a bit closer, it turns out it's not possible to support pack dependencies via st2-pack-install tool at the moment.

st2-pack-install helper utility was designed to deploy the st2 packs, their virtualenvs without fully running st2 system and backends like DB/MQ.

While original st2 pack install in new version relies on several Orquesta workflows under the hood to gather the list, detect conflicts, download pack dependencies, meaning it requires full st2 engine up & running.

To support pack dependencies via helper st2-pack-install utility https://github.com/StackStorm/st2/pull/4769 original implementation of pack dependencies needs a rewrite: replace Orquesta workflow with the pure Python logic.