Azure-pipelines-tasks: "Permission denied" error in conda task
Environment
Server - VSTS or TFS on-premises?
VSTS
Agent - Hosted or Private:
Hosted 'Ubuntu 16.04'
Issue Description
- task: CondaEnvironment@1
inputs:
createCustomEnvironment: true
environmentName: py3.5
packageSpecs: 'python=3.5'
This task produces permission denied errors which lead to task failure.
Task logs
2018-09-12T16:22:46.9229436Z ##[section]Starting: CondaEnvironment
2018-09-12T16:22:46.9368940Z ==============================================================================
2018-09-12T16:22:46.9388229Z Task : Conda Environment
2018-09-12T16:22:46.9406025Z Description : Create and activate a Conda environment.
2018-09-12T16:22:46.9424863Z Version : 1.138.0
2018-09-12T16:22:46.9443213Z Author : Microsoft Corporation
2018-09-12T16:22:46.9462022Z Help : [More information](https://go.microsoft.com/fwlink/?linkid=873466)
2018-09-12T16:22:46.9480302Z ==============================================================================
2018-09-12T16:22:50.0126863Z [command]/usr/bin/conda update --name base conda --yes
2018-09-12T16:23:14.9062048Z Solving environment: ...working... done
2018-09-12T16:23:20.6343316Z
2018-09-12T16:23:20.7776911Z conda-4.5.11 | 1.0 MB | | 0%
2018-09-12T16:23:20.9581679Z conda-4.5.11 | 1.0 MB | ######## | 80%
2018-09-12T16:23:21.0118534Z conda-4.5.11 | 1.0 MB | #########6 | 97%
2018-09-12T16:23:21.0146714Z conda-4.5.11 | 1.0 MB | ########## | 100%
2018-09-12T16:23:21.0155199Z
2018-09-12T16:23:21.1333663Z openssl-1.0.2p | 3.5 MB | | 0%
2018-09-12T16:23:21.8155494Z openssl-1.0.2p | 3.5 MB | #######5 | 76%
2018-09-12T16:23:21.9711961Z openssl-1.0.2p | 3.5 MB | #########4 | 94%
2018-09-12T16:23:21.9736658Z openssl-1.0.2p | 3.5 MB | ########## | 100%
2018-09-12T16:23:21.9746419Z
2018-09-12T16:23:22.0127922Z certifi-2018.8.24 | 140 KB | | 0%
2018-09-12T16:23:22.0151146Z certifi-2018.8.24 | 140 KB | ########## | 100%
2018-09-12T16:23:22.0169372Z ## Package Plan ##
2018-09-12T16:23:22.0179499Z
2018-09-12T16:23:22.0198916Z environment location: /usr/share/miniconda
2018-09-12T16:23:22.0208202Z
2018-09-12T16:23:22.0226396Z added / updated specs:
2018-09-12T16:23:22.0244424Z - conda
2018-09-12T16:23:22.0254336Z
2018-09-12T16:23:22.0263495Z
2018-09-12T16:23:22.0282046Z The following packages will be downloaded:
2018-09-12T16:23:22.0291571Z
2018-09-12T16:23:22.0309753Z package | build
2018-09-12T16:23:22.0327780Z ---------------------------|-----------------
2018-09-12T16:23:22.0346469Z conda-4.5.11 | py36_0 1.0 MB
2018-09-12T16:23:22.0364764Z openssl-1.0.2p | h14c3975_0 3.5 MB
2018-09-12T16:23:22.0384794Z certifi-2018.8.24 | py36_1 140 KB
2018-09-12T16:23:22.0403797Z ------------------------------------------------------------
2018-09-12T16:23:22.0422713Z Total: 4.6 MB
2018-09-12T16:23:22.0431658Z
2018-09-12T16:23:22.0449678Z The following packages will be UPDATED:
2018-09-12T16:23:22.0459692Z
2018-09-12T16:23:22.0478395Z certifi: 2018.4.16-py36_0 --> 2018.8.24-py36_1
2018-09-12T16:23:22.0497467Z conda: 4.5.4-py36_0 --> 4.5.11-py36_0
2018-09-12T16:23:22.0516599Z openssl: 1.0.2o-h20670df_0 --> 1.0.2p-h14c3975_0
2018-09-12T16:23:22.0525578Z
2018-09-12T16:23:22.0534968Z
2018-09-12T16:23:22.0553223Z Downloading and Extracting Packages
2018-09-12T16:23:22.1194848Z Preparing transaction: ...working... done
2018-09-12T16:23:22.5713236Z Verifying transaction: ...working... done
2018-09-12T16:23:22.5736956Z Executing transaction: ...working... failed
2018-09-12T16:23:22.5757156Z Rolling back transaction: ...working... done
2018-09-12T16:23:22.5768227Z
2018-09-12T16:23:22.5788534Z ERROR conda.core.link:_execute(502): An error occurred while uninstalling package 'defaults::conda-4.5.4-py36_0'.
2018-09-12T16:23:22.5809176Z PermissionError(13, 'Permission denied')
2018-09-12T16:23:22.5829065Z Attempting to roll back.
2018-09-12T16:23:22.5838159Z
2018-09-12T16:23:22.5846988Z
2018-09-12T16:23:22.5867327Z PermissionError(13, 'Permission denied')
2018-09-12T16:23:22.5877280Z
2018-09-12T16:23:22.5886944Z
2018-09-12T16:23:22.8545233Z Prepending PATH environment variable with directory: /usr/bin
2018-09-12T16:23:22.8756428Z [command]/usr/bin/conda create --quiet --prefix /usr/envs/py3.5 --mkdir --yes python=3.5
2018-09-12T16:23:24.1875016Z Solving environment: ...working... done
2018-09-12T16:23:32.6251193Z
2018-09-12T16:23:32.6273323Z ## Package Plan ##
2018-09-12T16:23:32.6282599Z
2018-09-12T16:23:32.6301488Z environment location: /usr/envs/py3.5
2018-09-12T16:23:32.6311118Z
2018-09-12T16:23:32.6332182Z added / updated specs:
2018-09-12T16:23:32.6354775Z - python=3.5
2018-09-12T16:23:32.6364968Z
2018-09-12T16:23:32.6378026Z
2018-09-12T16:23:32.6396858Z The following packages will be downloaded:
2018-09-12T16:23:32.6406238Z
2018-09-12T16:23:32.6426046Z package | build
2018-09-12T16:23:32.6445265Z ---------------------------|-----------------
2018-09-12T16:23:32.6464958Z libstdcxx-ng-8.2.0 | hdf63c60_1 2.9 MB
2018-09-12T16:23:32.6483796Z wheel-0.31.1 | py35_0 63 KB
2018-09-12T16:23:32.6504098Z readline-7.0 | h7b6447c_5 392 KB
2018-09-12T16:23:32.6523845Z tk-8.6.8 | hbc83047_0 3.1 MB
2018-09-12T16:23:32.6545870Z sqlite-3.24.0 | h84994c4_0 1.8 MB
2018-09-12T16:23:32.6564638Z certifi-2018.8.24 | py35_1 139 KB
2018-09-12T16:23:32.6583576Z python-3.5.6 | hc3d631a_0 28.3 MB
2018-09-12T16:23:32.6602030Z setuptools-40.2.0 | py35_0 571 KB
2018-09-12T16:23:32.6622815Z pip-10.0.1 | py35_0 1.8 MB
2018-09-12T16:23:32.6642082Z libgcc-ng-8.2.0 | hdf63c60_1 7.6 MB
2018-09-12T16:23:32.6667305Z ------------------------------------------------------------
2018-09-12T16:23:32.6685232Z Total: 46.7 MB
2018-09-12T16:23:32.6694585Z
2018-09-12T16:23:32.6712739Z The following NEW packages will be INSTALLED:
2018-09-12T16:23:32.6721945Z
2018-09-12T16:23:32.6740633Z ca-certificates: 2018.03.07-0
2018-09-12T16:23:32.6759605Z certifi: 2018.8.24-py35_1
2018-09-12T16:23:32.6778112Z libedit: 3.1.20170329-h6b74fdf_2
2018-09-12T16:23:32.6795872Z libffi: 3.2.1-hd88cf55_4
2018-09-12T16:23:32.6814772Z libgcc-ng: 8.2.0-hdf63c60_1
2018-09-12T16:23:32.6834873Z libstdcxx-ng: 8.2.0-hdf63c60_1
2018-09-12T16:23:32.6853380Z ncurses: 6.1-hf484d3e_0
2018-09-12T16:23:32.6872112Z openssl: 1.0.2p-h14c3975_0
2018-09-12T16:23:32.6891673Z pip: 10.0.1-py35_0
2018-09-12T16:23:32.6910287Z python: 3.5.6-hc3d631a_0
2018-09-12T16:23:32.6929504Z readline: 7.0-h7b6447c_5
2018-09-12T16:23:32.6949241Z setuptools: 40.2.0-py35_0
2018-09-12T16:23:32.6969051Z sqlite: 3.24.0-h84994c4_0
2018-09-12T16:23:32.6989538Z tk: 8.6.8-hbc83047_0
2018-09-12T16:23:32.7008705Z wheel: 0.31.1-py35_0
2018-09-12T16:23:32.7027794Z xz: 5.2.4-h14c3975_4
2018-09-12T16:23:32.7047048Z zlib: 1.2.11-ha838bed_2
2018-09-12T16:23:32.7056267Z
2018-09-12T16:23:32.7073845Z Preparing transaction: ...working... failed
2018-09-12T16:23:32.7083022Z
2018-09-12T16:23:32.7101042Z CondaError: Unable to create prefix directory '/usr/envs/py3.5'.
2018-09-12T16:23:32.7120320Z Check that you have sufficient permissions.
2018-09-12T16:23:32.7129576Z
2018-09-12T16:23:32.8411104Z ##[error]Failed to create Conda environment /usr/envs/py3.5. Error: /usr/bin/conda failed with return code: 1
2018-09-12T16:23:32.9312236Z ##[section]Finishing: CondaEnvironment
letmaik
All 55 comments
This affects me too.
Setting the following options does not work on Linux Ubuntu 16.04, conda will install some packages into the supplied prefix but eventually try to install into the miniconda base installation directory and fail.
installOptions: '--prefix $(Agent.WorkFolder)/tmp'
createOptions: '--prefix $(Agent.WorkFolder)/tmp'
Furthermore the documentation for the CondaEnvironment@1 task is rather poor. For example, the createCustomEnvironment option is mandatory in order to make use of environmentName. But this is not documented anywhere. If createCustomEnvironment is omitted, then environmentName is silently ignored. This is very confusing because the documentation clearly states that the environment would be activated.
emmenlau
on 18 Sep 2018
Thanks for reporting this. I believe there is a permissions issue here. As a temporary workaround, can you try running on Hosted Linux Preview?
There's also a script-based workaround someone submitted here.
brcrista
on 19 Sep 2018
@emmenlau , thanks for the feedback on the docs. For the sake of transparency, let me give you the story there: in the UI-based designer, environmentName gets hidden unless createCustomEnvironment is set. Clearly, this is not the way YAML works, and the createCustomEnvironment parameter feels redundant with YAML.
Ultimately the task should be made more YAML-friendly, but in the meantime the same task code has to work for both YAML and designer. In the meantime, I'll update the docs as you suggested.
brcrista
on 19 Sep 2018
Dear @brcrista , thanks for the clarification! Things make more sense now. It was a bit challenging to get a cross platform conda build working, since the conda task works nicely on Windows but not Linux, and at the same time the Linux workaround (to create a new conda environment and activate it) does not work on Windows. I hope I did not sound too frustrated, please keep up the good work :-)
I did not report this Windows conda build issue yet, and I do not know where the correct place would be to report it. But for the sake of other people affected, I would like to document it here: calling activate <environment> on Windows stops the build. The script ends there, with a successful status. No further commands after activate are executed. I assume this has something to do with how activate works, possibly it starts a new shell or something which may confuse the build...? In any case, the following example script stops at the activate step (with status "success") and does not execute any further commands (like the last echo):
echo "Will create a new conda environment"
conda create --quiet --yes --mkdir --prefix %AGENT_WORKFOLDER%\envs\test --channel conda-forge python=$(PY_VERSION) || exit 1
echo "Will activate the conda environment"
activate %AGENT_WORKFOLDER%\envs\test || exit 1
echo "Activated the conda environment"
@emmenlau thanks for reporting. Please create a separate issue for that and I'll address it there
brcrista
on 19 Sep 2018
@brcrista any progress on this? I'm finding that not only do I get a permission denied error, but the pipeline _thinks it succeeded_ and continues with the next tasks. Here's the output of the task for me:
2018-09-26T04:32:34.2753614Z ##[section]Starting: CondaEnvironment
2018-09-26T04:32:34.2894926Z ==============================================================================
2018-09-26T04:32:34.2913850Z Task : Conda Environment
2018-09-26T04:32:34.2931900Z Description : Create and activate a Conda environment.
2018-09-26T04:32:34.2947765Z Version : 1.138.0
2018-09-26T04:32:34.2965829Z Author : Microsoft Corporation
2018-09-26T04:32:34.2982681Z Help : [More information](https://go.microsoft.com/fwlink/?linkid=873466)
2018-09-26T04:32:34.2999255Z ==============================================================================
2018-09-26T04:32:36.4810392Z Prepending PATH environment variable with directory: /usr/bin
2018-09-26T04:32:36.4986615Z [command]/usr/bin/conda install conda conda-verify conda-build ipykernel anaconda-client --quiet --yes
2018-09-26T04:33:01.3370622Z Solving environment: ...working... done
2018-09-26T04:33:10.6852986Z
2018-09-26T04:33:10.6870416Z ## Package Plan ##
2018-09-26T04:33:10.6878762Z
2018-09-26T04:33:10.6894909Z environment location: /usr/share/miniconda
2018-09-26T04:33:10.6904074Z
2018-09-26T04:33:10.6920690Z added / updated specs:
2018-09-26T04:33:10.6940005Z - anaconda-client
2018-09-26T04:33:10.6958368Z - conda
2018-09-26T04:33:10.6976890Z - conda-build
2018-09-26T04:33:10.6995397Z - conda-verify
2018-09-26T04:33:10.7015955Z - ipykernel
2018-09-26T04:33:10.7024809Z
2018-09-26T04:33:10.7032820Z
2018-09-26T04:33:10.7048352Z The following packages will be downloaded:
2018-09-26T04:33:10.7057017Z
2018-09-26T04:33:10.7075433Z package | build
2018-09-26T04:33:10.7092951Z ---------------------------|-----------------
2018-09-26T04:33:10.7111105Z libsodium-1.0.16 | h1bed415_0 302 KB
2018-09-26T04:33:10.7129493Z filelock-3.0.8 | py37_0 13 KB
2018-09-26T04:33:10.7148770Z clyent-1.2.2 | py37_1 18 KB
2018-09-26T04:33:10.7167852Z click-6.7 | py37_0 105 KB
2018-09-26T04:33:10.7194999Z jupyter_core-4.4.0 | py37_0 63 KB
2018-09-26T04:33:10.7213654Z parso-0.3.1 | py37_0 114 KB
2018-09-26T04:33:10.7234162Z ipython-6.5.0 | py37_0 1.0 MB
2018-09-26T04:33:10.7252489Z pygments-2.2.0 | py37_0 1.3 MB
2018-09-26T04:33:10.7270944Z anaconda-client-1.7.2 | py37_0 140 KB
2018-09-26T04:33:10.7289191Z future-0.16.0 | py37_0 666 KB
2018-09-26T04:33:10.7307541Z ptyprocess-0.6.0 | py37_0 23 KB
2018-09-26T04:33:10.7325764Z conda-build-3.15.1 | py37_0 459 KB
2018-09-26T04:33:10.7344181Z tqdm-4.26.0 | py37h28b3542_0 59 KB
2018-09-26T04:33:10.7362686Z python-dateutil-2.7.3 | py37_0 260 KB
2018-09-26T04:33:10.7381031Z jsonschema-2.6.0 | py37_0 62 KB
2018-09-26T04:33:10.7399592Z decorator-4.3.0 | py37_0 15 KB
2018-09-26T04:33:10.7417837Z nbformat-4.4.0 | py37_0 141 KB
2018-09-26T04:33:10.7436440Z jinja2-2.10 | py37_0 183 KB
2018-09-26T04:33:10.7454682Z beautifulsoup4-4.6.3 | py37_0 138 KB
2018-09-26T04:33:10.7473086Z zeromq-4.2.5 | hf484d3e_1 567 KB
2018-09-26T04:33:10.7491216Z pyzmq-17.1.2 | py37h14c3975_0 454 KB
2018-09-26T04:33:10.7509570Z markupsafe-1.0 | py37h14c3975_1 25 KB
2018-09-26T04:33:10.7528820Z backcall-0.1.0 | py37_0 20 KB
2018-09-26T04:33:10.7547696Z glob2-0.6 | py37_0 17 KB
2018-09-26T04:33:10.7566030Z pytz-2018.5 | py37_0 231 KB
2018-09-26T04:33:10.7601914Z traitlets-4.3.2 | py37_0 133 KB
2018-09-26T04:33:10.7620076Z ipykernel-4.9.0 | py37_1 146 KB
2018-09-26T04:33:10.7693772Z conda-verify-3.1.0 | py37_0 32 KB
2018-09-26T04:33:10.7712250Z simplegeneric-0.8.1 | py37_2 9 KB
2018-09-26T04:33:10.7730405Z psutil-5.4.7 | py37h14c3975_0 305 KB
2018-09-26T04:33:10.7749544Z pkginfo-1.4.2 | py37_1 39 KB
2018-09-26T04:33:10.7768013Z ipython_genutils-0.2.0 | py37_0 39 KB
2018-09-26T04:33:10.7786281Z pyyaml-3.13 | py37h14c3975_0 177 KB
2018-09-26T04:33:10.7804948Z tornado-5.1 | py37h14c3975_0 665 KB
2018-09-26T04:33:10.7823258Z jedi-0.12.1 | py37_0 225 KB
2018-09-26T04:33:10.7841486Z pickleshare-0.7.4 | py37_0 12 KB
2018-09-26T04:33:10.7860257Z prompt_toolkit-1.0.15 | py37_0 335 KB
2018-09-26T04:33:10.7893883Z wcwidth-0.1.7 | py37_0 23 KB
2018-09-26T04:33:10.7910250Z jupyter_client-5.2.3 | py37_0 126 KB
2018-09-26T04:33:10.7926531Z pexpect-4.6.0 | py37_0 77 KB
2018-09-26T04:33:10.7942850Z patchelf-0.9 | hf484d3e_2 68 KB
2018-09-26T04:33:10.7959133Z ------------------------------------------------------------
2018-09-26T04:33:10.7974393Z Total: 8.6 MB
2018-09-26T04:33:10.7981420Z
2018-09-26T04:33:10.7995559Z The following NEW packages will be INSTALLED:
2018-09-26T04:33:10.8002775Z
2018-09-26T04:33:10.8017513Z anaconda-client: 1.7.2-py37_0
2018-09-26T04:33:10.8033835Z backcall: 0.1.0-py37_0
2018-09-26T04:33:10.8054308Z beautifulsoup4: 4.6.3-py37_0
2018-09-26T04:33:10.8070240Z click: 6.7-py37_0
2018-09-26T04:33:10.8086351Z clyent: 1.2.2-py37_1
2018-09-26T04:33:10.8102329Z conda-build: 3.15.1-py37_0
2018-09-26T04:33:10.8118391Z conda-verify: 3.1.0-py37_0
2018-09-26T04:33:10.8134429Z decorator: 4.3.0-py37_0
2018-09-26T04:33:10.8150289Z filelock: 3.0.8-py37_0
2018-09-26T04:33:10.8166673Z future: 0.16.0-py37_0
2018-09-26T04:33:10.8191518Z glob2: 0.6-py37_0
2018-09-26T04:33:10.8207645Z ipykernel: 4.9.0-py37_1
2018-09-26T04:33:10.8223613Z ipython: 6.5.0-py37_0
2018-09-26T04:33:10.8239510Z ipython_genutils: 0.2.0-py37_0
2018-09-26T04:33:10.8255562Z jedi: 0.12.1-py37_0
2018-09-26T04:33:10.8271906Z jinja2: 2.10-py37_0
2018-09-26T04:33:10.8287724Z jsonschema: 2.6.0-py37_0
2018-09-26T04:33:10.8303668Z jupyter_client: 5.2.3-py37_0
2018-09-26T04:33:10.8319575Z jupyter_core: 4.4.0-py37_0
2018-09-26T04:33:10.8335934Z libsodium: 1.0.16-h1bed415_0
2018-09-26T04:33:10.8352906Z markupsafe: 1.0-py37h14c3975_1
2018-09-26T04:33:10.8368725Z nbformat: 4.4.0-py37_0
2018-09-26T04:33:10.8386919Z parso: 0.3.1-py37_0
2018-09-26T04:33:10.8402968Z patchelf: 0.9-hf484d3e_2
2018-09-26T04:33:10.8418709Z pexpect: 4.6.0-py37_0
2018-09-26T04:33:10.8434887Z pickleshare: 0.7.4-py37_0
2018-09-26T04:33:10.8450660Z pkginfo: 1.4.2-py37_1
2018-09-26T04:33:10.8466628Z prompt_toolkit: 1.0.15-py37_0
2018-09-26T04:33:10.8482790Z psutil: 5.4.7-py37h14c3975_0
2018-09-26T04:33:10.8498519Z ptyprocess: 0.6.0-py37_0
2018-09-26T04:33:10.8514696Z pygments: 2.2.0-py37_0
2018-09-26T04:33:10.8530763Z python-dateutil: 2.7.3-py37_0
2018-09-26T04:33:10.8546883Z pytz: 2018.5-py37_0
2018-09-26T04:33:10.8562885Z pyyaml: 3.13-py37h14c3975_0
2018-09-26T04:33:10.8578655Z pyzmq: 17.1.2-py37h14c3975_0
2018-09-26T04:33:10.8594803Z simplegeneric: 0.8.1-py37_2
2018-09-26T04:33:10.8610535Z tornado: 5.1-py37h14c3975_0
2018-09-26T04:33:10.8626568Z tqdm: 4.26.0-py37h28b3542_0
2018-09-26T04:33:10.8642569Z traitlets: 4.3.2-py37_0
2018-09-26T04:33:10.8658256Z wcwidth: 0.1.7-py37_0
2018-09-26T04:33:10.8675135Z zeromq: 4.2.5-hf484d3e_1
2018-09-26T04:33:10.8691079Z
2018-09-26T04:33:10.8705626Z Preparing transaction: ...working... failed
2018-09-26T04:33:11.7585721Z
2018-09-26T04:33:11.7604036Z # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
2018-09-26T04:33:11.7612339Z
2018-09-26T04:33:11.7629453Z Traceback (most recent call last):
2018-09-26T04:33:11.7647363Z File "/usr/share/miniconda/lib/python3.7/site-packages/conda/exceptions.py", line 819, in __call__
2018-09-26T04:33:11.7665243Z return func(*args, **kwargs)
2018-09-26T04:33:11.7688451Z File "/usr/share/miniconda/lib/python3.7/site-packages/conda/cli/main.py", line 78, in _main
2018-09-26T04:33:11.7709639Z exit_code = do_call(args, p)
2018-09-26T04:33:11.7727319Z File "/usr/share/miniconda/lib/python3.7/site-packages/conda/cli/conda_argparse.py", line 77, in do_call
2018-09-26T04:33:11.7745018Z exit_code = getattr(module, func_name)(args, parser)
2018-09-26T04:33:11.7764685Z File "/usr/share/miniconda/lib/python3.7/site-packages/conda/cli/main_install.py", line 11, in execute
2018-09-26T04:33:11.7783316Z install(args, parser, 'install')
2018-09-26T04:33:11.7802247Z File "/usr/share/miniconda/lib/python3.7/site-packages/conda/cli/install.py", line 253, in install
2018-09-26T04:33:11.7824569Z handle_txn(unlink_link_transaction, prefix, args, newenv)
2018-09-26T04:33:11.7842910Z File "/usr/share/miniconda/lib/python3.7/site-packages/conda/cli/install.py", line 282, in handle_txn
2018-09-26T04:33:11.7860304Z unlink_link_transaction.execute()
2018-09-26T04:33:11.7878139Z File "/usr/share/miniconda/lib/python3.7/site-packages/conda/core/link.py", line 223, in execute
2018-09-26T04:33:11.7895621Z self.verify()
2018-09-26T04:33:11.7913417Z File "/usr/share/miniconda/lib/python3.7/site-packages/conda/common/io.py", line 46, in decorated
2018-09-26T04:33:11.7930712Z return f(*args, **kwds)
2018-09-26T04:33:11.7948342Z File "/usr/share/miniconda/lib/python3.7/site-packages/conda/core/link.py", line 200, in verify
2018-09-26T04:33:11.7965994Z self.prepare()
2018-09-26T04:33:11.8001021Z File "/usr/share/miniconda/lib/python3.7/site-packages/conda/core/link.py", line 192, in prepare
2018-09-26T04:33:11.8018930Z stp.remove_specs, stp.update_specs)
2018-09-26T04:33:11.8036728Z File "/usr/share/miniconda/lib/python3.7/site-packages/conda/core/link.py", line 282, in _prepare
2018-09-26T04:33:11.8059444Z mkdir_p(transaction_context['temp_dir'])
2018-09-26T04:33:11.8079545Z File "/usr/share/miniconda/lib/python3.7/site-packages/conda/gateways/disk/__init__.py", line 60, in mkdir_p
2018-09-26T04:33:11.8097113Z makedirs(path)
2018-09-26T04:33:11.8113319Z File "/usr/share/miniconda/lib/python3.7/os.py", line 221, in makedirs
2018-09-26T04:33:11.8129128Z mkdir(name, mode)
2018-09-26T04:33:11.8148942Z PermissionError: [Errno 13] Permission denied: '/usr/share/miniconda/.condatmp'
2018-09-26T04:33:11.8158588Z
2018-09-26T04:33:11.8176287Z `$ /usr/bin/conda install conda conda-verify conda-build ipykernel anaconda-client --quiet --yes`
2018-09-26T04:33:11.8195992Z
2018-09-26T04:33:11.8211522Z environment variables:
2018-09-26T04:33:11.8227910Z BUILD_REPOSITORY_LOCALPATH=/home/vsts/work/1/s
2018-09-26T04:33:11.8244385Z CIO_TEST=<not set>
2018-09-26T04:33:11.8260609Z CONDA=/usr/share/miniconda
2018-09-26T04:33:11.8277143Z CONDA_ROOT=/usr/share/miniconda
2018-09-26T04:33:11.8294062Z PATH=/usr/bin:"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2018-09-26T04:33:11.8310516Z :/usr/games:/usr/local/games"
2018-09-26T04:33:11.8326746Z PYTHON_VERSION=2.7
2018-09-26T04:33:11.8342670Z REQUESTS_CA_BUNDLE=<not set>
2018-09-26T04:33:11.8358719Z SSL_CERT_FILE=<not set>
2018-09-26T04:33:11.8366744Z
2018-09-26T04:33:11.8388946Z active environment : None
2018-09-26T04:33:11.8404911Z user config file : /home/vsts/.condarc
2018-09-26T04:33:11.8421351Z populated config files :
2018-09-26T04:33:11.8437811Z conda version : 4.5.11
2018-09-26T04:33:11.8454967Z conda-build version : not installed
2018-09-26T04:33:11.8471811Z python version : 3.7.0.final.0
2018-09-26T04:33:11.8488199Z base environment : /usr/share/miniconda (read only)
2018-09-26T04:33:11.8505574Z channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
2018-09-26T04:33:11.8523020Z https://repo.anaconda.com/pkgs/main/noarch
2018-09-26T04:33:11.8541712Z https://repo.anaconda.com/pkgs/free/linux-64
2018-09-26T04:33:11.8559564Z https://repo.anaconda.com/pkgs/free/noarch
2018-09-26T04:33:11.8576869Z https://repo.anaconda.com/pkgs/r/linux-64
2018-09-26T04:33:11.8594239Z https://repo.anaconda.com/pkgs/r/noarch
2018-09-26T04:33:11.8611291Z https://repo.anaconda.com/pkgs/pro/linux-64
2018-09-26T04:33:11.8628635Z https://repo.anaconda.com/pkgs/pro/noarch
2018-09-26T04:33:11.8644776Z package cache : /usr/share/miniconda/pkgs
2018-09-26T04:33:11.8660759Z /home/vsts/.conda/pkgs
2018-09-26T04:33:11.8677049Z envs directories : /home/vsts/.conda/envs
2018-09-26T04:33:11.8723257Z /usr/share/miniconda/envs
2018-09-26T04:33:11.8738298Z platform : linux-64
2018-09-26T04:33:11.8755088Z user-agent : conda/4.5.11 requests/2.19.1 CPython/3.7.0 Linux/4.15.0-1023-azure ubuntu/16.04 glibc/2.23
2018-09-26T04:33:11.8770687Z UID:GID : 1001:999
2018-09-26T04:33:11.8784829Z netrc file : None
2018-09-26T04:33:11.8798840Z offline mode : False
2018-09-26T04:33:11.8806029Z
2018-09-26T04:33:11.8812998Z
2018-09-26T04:33:11.8827121Z An unexpected error has occurred. Conda has prepared the above report.
2018-09-26T04:33:11.8841243Z Upload successful.
2018-09-26T04:33:12.1280444Z (node:2782) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Failed to install packages. Error: /usr/bin/conda failed with return code: 1
2018-09-26T04:33:12.2262239Z ##[section]Finishing: CondaEnvironment
mcg1969
on 26 Sep 2018
I have a fix for both of these issues coming out soon -- see #8402. It is merged and pending release. I'm going to hotfix it in production so I would expect to see it in 3-4 days.
brcrista
on 26 Sep 2018
Does running on Hosted Linux Preview unblock you until then?
brcrista
on 26 Sep 2018
@brcrista, can you point me to relevant docs on how to use Hosted Linux Preview? I tried looking and found none. But it was also 2AM, so have mercy! :-)
mcg1969
on 27 Sep 2018
@mcg1969 is your pipeline defined in YAML or with the Web UI? In the Web UI, it should be an option in the agent pool dropdown. In YAML, you would just say pool: Hosted Linux Preview for your job
brcrista
on 27 Sep 2018
I defined it in YAML. I'll give that a shot. Thanks!
mcg1969
on 27 Sep 2018
I'm clearly missing important context. Azure Pipelines did not accept the modification to my YAML spec that you suggested. The closest I've come to operational is here, but this seems to be attempting a Windows build. I'll just have to wait for your fix to get pushed out to pipelines I guess.
mcg1969
on 27 Sep 2018
Not sure why that wouldn't work but you should be able to leave it undefined in YAML and it will fall back to what you choose in the dropdown
brcrista
on 27 Sep 2018
Ah! Well, I do not have that option available to me for some reason. I have all of the others...
mcg1969
on 28 Sep 2018
Oh, sorry about that. That explains why it wouldn't work in YAML at least.
Another option is you can just script it. We made the task because each build step runs in its own process. Activating a Conda environment through the normal means will only last for its current process, so the task is a bit of a shim to let you still split your job up into steps and use other tasks. But if you're just installing to the base environment, I don't think it should matter.
From your log output, I think something like this will work for you:
- script: conda install conda conda-verify conda-build ipykernel anaconda-client
displayName: Install dependencies with conda
If you had to activate an environment you would do this:
- script: |
conda create --name env conda conda-verify conda-build ipykernel anaconda-client
# other stuff, pytest, etc.
So that # other stuff, pytest, etc. runs in the environment.
brcrista
on 28 Sep 2018
If anyone was interested in the root cause, by the way, on Hosted Ubuntu we moved conda from living somewhere in the home directory to under /usr/bin. Also, Hosted Ubuntu runs as user by default (with password-less sudo) while Hosted Linux Preview ran as root. The task tries to create environments in the same directory as conda, so it doesn't have access to /usr/bin and fails.
The reason the task doesn't fail correctly on installing to the base environment is because I forgot to await a Promise in Node JS (I did foo() instead of await foo() where foo returns Promise<void>) so it fails to barf up the exception.
brcrista
on 28 Sep 2018
Thanks for all the explanation! I'm curious why the choice was made to move the conda environment out of the user directory. One of the key benefits of anaconda/miniconda is that it can be installed without root access; and indeed, is installed _by default_ in a user's home directory. And of course, any operations to the root environment will require write access. In fact, even creating new environments will require this, unless you explicitly override the pkgs_dirs and envs_dirs configurations to point to a user-writable directory.
The point being: conda use is considerably more difficult to get right with a read-only installation.
mcg1969
on 28 Sep 2018
Of course, an easy workaround is for users to install Miniconda themselves, but this seems an unfortunate waste of resources that could be avoided, given that you _have_ made the effort to provide users with conda (which I very much appreciate!)
mcg1969
on 28 Sep 2018
You're absolutely right about that, I think it was just a mix-up as we were making the new agent image. I'm going to see if I can get that fixed after pushing the fix for the task out.
brcrista
on 28 Sep 2018
Excellent. I want you to know I'm really excited about the possibility of using Azure Pipelines for conda-heavy work. Thank you for your hard work!
mcg1969
on 28 Sep 2018
I see that @dakale has rejected the request to move the conda installation into $HOME. I do understand the logic there, but I thought I'd invite David here to see if we can discuss the best path forward.
I know that we at Anaconda have been eager to exercise Azure Pipelines, and indeed we already are doing so for our Numba project to great effect. But we currently aren't using this central conda environment ourselves, instead opting to download and install Miniconda as a step in our pipelines, just like we would do in Travis or Circle. (Interestingly, Appveyor's conda installations _are_ writable, so they can be directly modified, often eliminating the need to use Miniconda.)
I personally would love to see CondaEnvironment eliminate the need to perform this separate Miniconda installation in our package building patterns. Indeed, once I make this work for my own purposes, I intend to build out a template for people to use that will simplify the process of using Azure Pipelines in other conda-centric CI jobs.
mcg1969
on 29 Sep 2018
Hi @mcg1969. Im not sure its 100% accurate for us to say we moved conda out of a home directory and into a system wide directory.
On the older Hosted Linux Preview pool, the agent process ran as root, and on Hosted Ubuntu 1604 the agent runs as "vsts" but can sudo. I ported the miniconda installation from the old pool to the new one without changing anything. In fact, you can see the installation script used here: https://github.com/Microsoft/azure-pipelines-image-generation/blob/master/images/linux/scripts/installers/miniconda.sh
The main issue here is the "vsts" user does not exist when the machine image is provisioning so I cant install anything directory to eg /home/vsts . The image generation process linked above aims to generate and generic base image, and then additional customizations like creating the vsts account are done internally as part of the hosted pool service.
Im personally not a miniconda expert by any means, so happy to work with you (or PRs welcome ;) ) on how we provision miniconda on these machines. If theres some environment variables that can be set to control where it tries to write, for example, that is something that can be changed. Or, if most of the time of an ad-hoc per-build setup is being spent downloading the same tarball, it could be pre-downloaded into a well known location to be picked up by any tasks or scripts used in your build.
dakale
on 1 Oct 2018
Ah, I agree, my characterization is not accurate; @brcrista proposed to move it there, and you declined for the reasons you explain above. I stand corrected!
So the primary use case I'm thinking of here is for package building. I think as long as it is possible modify the root environment, we _should_ be fine. That can be accomplished by the CondaEnvironment task. It will also be necessary to make sure that conda is properly detecting the fact that the root environment is not read-write for the vsts user, so that it puts additional package downloads, created environments, etc. in a different place.
As a means of testing things, it will be necessary to make sure that the CondaEnvironment task can indeed install, say, conda-build, conda-verify, and anaconda-client to the root environment. Assuming that works across Mac, Windows, and Linux, then I _think_ the rest will work, but I can certainly be a guinea pig.
mcg1969
on 1 Oct 2018
Workaround I found from someone else's azure-pipelines.yml:
- script: sudo install -d -m 0777 /usr/envs
displayName: Fix Conda permissions
asmeurer
on 3 Oct 2018
Thanks @asmeurer, there was a separate thread with a workaround that I linked to but it may have gotten buried. Here it is again: https://github.com/Microsoft/LightGBM/pull/1660#issuecomment-421428275
I would expect to see the fix in all organizations sometime tomorrow. I'll update this issue when I see it is out.
brcrista
on 3 Oct 2018
The fix has been deployed to all accounts now. When you run the task, you should see Version 1.140.2 in the banner:
==============================================================================
Task : Conda Environment
Description : Create and activate a Conda environment.
Version : 1.140.2
Author : Microsoft Corporation
Help : [More information](https://go.microsoft.com/fwlink/?linkid=873466)
==============================================================================
I still get this problem, Not 1.140, but 0.139 still.
2018-10-07T23:39:29.9030647Z ##[section]Starting: CondaEnvironment
2018-10-07T23:39:29.9033274Z ==============================================================================
2018-10-07T23:39:29.9033360Z Task : Conda Environment
2018-10-07T23:39:29.9033442Z Description : Create and activate a Conda environment.
2018-10-07T23:39:29.9033519Z Version : 0.139.0
2018-10-07T23:39:29.9033603Z Author : Microsoft Corporation
2018-10-07T23:39:29.9033671Z Help : [More information](https://go.microsoft.com/fwlink/?linkid=873466)
https://dev.azure.com/fastdotai/fastai/_build/results?buildId=14&view=logs
stas00
on 8 Oct 2018
@stas00 you're on major version 0 -- switch to version 1 of the task to get the update. Say CondaEnvironment@1 in YAML or use the dropdown in the designer.
brcrista
on 8 Oct 2018
@brcrista, that did the trick - thank you!
I was wondering what those @X meant, as I have seen some scripts with CondaEnvironment@0, others with CondaEnvironment@1. Now I know that they are version numbers.
I changed to CondaEnvironment@1 and the workaround is no longer needed.
Thank you so much!
stas00
on 8 Oct 2018
Except CondaEnvironment@1 fails with other permission problems elsewhere :(
Both on mac and linux with either python 3.6 or 3.7, the error is:
PermissionError: [Errno 13] Permission denied: '/usr/local/miniconda/.condatmp'
log in the next post
stas00
on 8 Oct 2018
2018-10-08T18:22:00.2849190Z # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
2018-10-08T18:22:00.2849470Z
2018-10-08T18:22:00.2849540Z Traceback (most recent call last):
2018-10-08T18:22:00.2850820Z File "/usr/local/miniconda/lib/python3.6/site-packages/conda/exceptions.py", line 819, in __call__
2018-10-08T18:22:00.2851050Z return func(*args, **kwargs)
2018-10-08T18:22:00.2851630Z File "/usr/local/miniconda/lib/python3.6/site-packages/conda/cli/main.py", line 78, in _main
2018-10-08T18:22:00.2851790Z exit_code = do_call(args, p)
2018-10-08T18:22:00.2852350Z File "/usr/local/miniconda/lib/python3.6/site-packages/conda/cli/conda_argparse.py", line 77, in do_call
2018-10-08T18:22:00.2852470Z exit_code = getattr(module, func_name)(args, parser)
2018-10-08T18:22:00.2853130Z File "/usr/local/miniconda/lib/python3.6/site-packages/conda/cli/main_install.py", line 11, in execute
2018-10-08T18:22:00.2853670Z install(args, parser, 'install')
2018-10-08T18:22:00.2854310Z File "/usr/local/miniconda/lib/python3.6/site-packages/conda/cli/install.py", line 253, in install
2018-10-08T18:22:00.2854830Z handle_txn(unlink_link_transaction, prefix, args, newenv)
2018-10-08T18:22:00.2855540Z File "/usr/local/miniconda/lib/python3.6/site-packages/conda/cli/install.py", line 282, in handle_txn
2018-10-08T18:22:00.2855720Z unlink_link_transaction.execute()
2018-10-08T18:22:00.2856330Z File "/usr/local/miniconda/lib/python3.6/site-packages/conda/core/link.py", line 223, in execute
2018-10-08T18:22:00.2856420Z self.verify()
2018-10-08T18:22:00.2857040Z File "/usr/local/miniconda/lib/python3.6/site-packages/conda/common/io.py", line 46, in decorated
2018-10-08T18:22:00.2857140Z return f(*args, **kwds)
2018-10-08T18:22:00.2857750Z File "/usr/local/miniconda/lib/python3.6/site-packages/conda/core/link.py", line 200, in verify
2018-10-08T18:22:00.2857840Z self.prepare()
2018-10-08T18:22:00.2858430Z File "/usr/local/miniconda/lib/python3.6/site-packages/conda/core/link.py", line 192, in prepare
2018-10-08T18:22:00.2858540Z stp.remove_specs, stp.update_specs)
2018-10-08T18:22:00.2859080Z File "/usr/local/miniconda/lib/python3.6/site-packages/conda/core/link.py", line 282, in _prepare
2018-10-08T18:22:00.2859670Z mkdir_p(transaction_context['temp_dir'])
2018-10-08T18:22:00.2860220Z File "/usr/local/miniconda/lib/python3.6/site-packages/conda/gateways/disk/__init__.py", line 60, in mkdir_p
2018-10-08T18:22:00.2860380Z makedirs(path)
2018-10-08T18:22:00.2860450Z File "/usr/local/miniconda/lib/python3.6/os.py", line 220, in makedirs
2018-10-08T18:22:00.2860600Z mkdir(name, mode)
2018-10-08T18:22:00.2861130Z PermissionError: [Errno 13] Permission denied: '/usr/local/miniconda/.condatmp'
2018-10-08T18:22:00.2861190Z
2018-10-08T18:22:00.2861760Z `$ /usr/local/miniconda/bin/conda install -y pytest pytest-runner`
2018-10-08T18:22:00.2861820Z
2018-10-08T18:22:00.2861870Z environment variables:
2018-10-08T18:22:00.2862030Z BUILD_REPOSITORY_LOCALPATH=/Users/vsts/agent/2.140.2/work/1/s
2018-10-08T18:22:00.2862090Z CIO_TEST=<not set>
2018-10-08T18:22:00.2862240Z CONDA=/usr/local/miniconda
2018-10-08T18:22:00.2862300Z CONDA_ROOT=/usr/local/miniconda
2018-10-08T18:22:00.2862430Z NUNIT3_PATH=/Library/Developer/nunit/3.6.0
2018-10-08T18:22:00.2862510Z NUNIT_BASE_PATH=/Library/Developer/nunit
2018-10-08T18:22:00.2862580Z PATH=/usr/local/miniconda/bin:/Users/vsts/hostedtoolcache/Python/3.6.5/x64/
2018-10-08T18:22:00.2862940Z bin:/Users/vsts/hostedtoolcache/Python/3.6.5/x64:/usr/local/bin:/usr/l
2018-10-08T18:22:00.2863030Z ocal/sbin:/Users/vsts/bin:/Users/vsts/.yarn/bin:/usr/local/go/bin:/Use
2018-10-08T18:22:00.2863180Z rs/vsts/Library/Android/sdk/tools:/Users/vsts/Library/Android/sdk/plat
2018-10-08T18:22:00.2863740Z form-tools:/Users/vsts/Library/Android/sdk/ndk-bundle:/Library/Framewo
2018-10-08T18:22:00.2863930Z rks/Mono.framework/Versions/Current/Commands:/usr/bin:/bin:/usr/sbin:/
2018-10-08T18:22:00.2864000Z sbin:/Users/vsts/.azcopy
2018-10-08T18:22:00.2864120Z PYTHON_VERSION=3.6
2018-10-08T18:22:00.2864190Z REQUESTS_CA_BUNDLE=<not set>
2018-10-08T18:22:00.2864250Z SSL_CERT_FILE=<not set>
2018-10-08T18:22:00.2864360Z
2018-10-08T18:22:00.2864410Z active environment : None
2018-10-08T18:22:00.2864470Z user config file : /Users/vsts/.condarc
2018-10-08T18:22:00.2864610Z populated config files :
2018-10-08T18:22:00.2864670Z conda version : 4.5.11
2018-10-08T18:22:00.2865230Z conda-build version : not installed
2018-10-08T18:22:00.2865310Z python version : 3.6.6.final.0
2018-10-08T18:22:00.2865440Z base environment : /usr/local/miniconda (read only)
2018-10-08T18:22:00.2865970Z channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
2018-10-08T18:22:00.2866230Z https://repo.anaconda.com/pkgs/main/noarch
2018-10-08T18:22:00.2866850Z https://repo.anaconda.com/pkgs/free/osx-64
2018-10-08T18:22:00.2866940Z https://repo.anaconda.com/pkgs/free/noarch
2018-10-08T18:22:00.2867520Z https://repo.anaconda.com/pkgs/r/osx-64
2018-10-08T18:22:00.2867600Z https://repo.anaconda.com/pkgs/r/noarch
2018-10-08T18:22:00.2868160Z https://repo.anaconda.com/pkgs/pro/osx-64
2018-10-08T18:22:00.2868270Z https://repo.anaconda.com/pkgs/pro/noarch
2018-10-08T18:22:00.2868330Z package cache : /usr/local/miniconda/pkgs
2018-10-08T18:22:00.2868470Z /Users/vsts/.conda/pkgs
2018-10-08T18:22:00.2868540Z envs directories : /Users/vsts/.conda/envs
2018-10-08T18:22:00.2868680Z /usr/local/miniconda/envs
2018-10-08T18:22:00.2869170Z platform : osx-64
2018-10-08T18:22:00.2869770Z user-agent : conda/4.5.11 requests/2.19.1 CPython/3.6.6 Darwin/17.7.0 OSX/10.13.6
2018-10-08T18:22:00.2869870Z UID:GID : 501:20
2018-10-08T18:22:00.2869920Z netrc file : None
2018-10-08T18:22:00.2870060Z offline mode : False
2018-10-08T18:22:00.2870100Z
2018-10-08T18:22:00.2870140Z
2018-10-08T18:22:00.2870270Z An unexpected error has occurred. Conda has prepared the above report.
2018-10-08T18:22:00.2870330Z Upload successful.
2018-10-08T18:22:00.5460210Z /Users/vsts/agent/2.140.2/work/_temp/5d01e851-2325-4802-93e7-bd74c2260bea.sh: line 2: py.test: command not found
2018-10-08T18:22:00.5591490Z ##[error]Bash exited with code '127'.
2018-10-08T18:22:00.7190470Z ##[section]Finishing: Run tests
Also, any chance a shell access can be given when things fail due to your environment issues? it's very difficult to guess what might work, it'd be much easier to get to shell on a specific image, see what works and then copy back the solution to the pipeline script - currently it's very slow and inefficient.
stas00
on 8 Oct 2018
Unfortunately there's no way to SSH into an agent when things fail; I agree that would be useful. @vtbassmatt might be counting feature requests for this
brcrista
on 8 Oct 2018
@stas00 looks like that's a script step. You just have to sudo
brcrista
on 8 Oct 2018
@brcrista @stas00 we've got SSH-into-hosted-agents on the long range radar. Thanks for the +1 on the feature.
vtbassmatt
on 8 Oct 2018
@stas00 looks like that's a script step. You just have to sudo
Why not make conda setup the environment for the user, so there is no need for sudo? It does sound like a conda setup and not a script issue. Currently I have it working with @0:
On linux:
- script: sudo install -d -m 0777 /usr/envs
displayName: Fix Conda permissions
on mac:
- script: sudo install -d -m 0777 /usr/local/miniconda/envs/
displayName: Fix Conda permissions
and then the script (identical on both OSes):
- task: CondaEnvironment@0
inputs:
environmentName: 'fastai-cpu'
packageSpecs: 'python=$(python.version)'
# Install dependencies and fastai
- script: |
conda activate fastai-cpu
conda install -y conda
conda install -y pip setuptools
# pytorch + torchvision
conda install -y -c pytorch pytorch-nightly-cpu
conda install -y -c fastai torchvision-nightly-cpu
# fastai deps
conda install -y -c fastai fastai
conda uninstall -y fastai
# fastai
pip install -e .[dev]
displayName: 'Upgrade pip/conda/setuptools. Install deps, pytorch and fastai'
continueOnError: false
You can see from the script that there should be no need for sudo, they all are an activation + installs, nothing fancy.
The errors I reported above come from trying CondaEnvironment@1 w/ or w/o the workaround at the top of this post.
stas00
on 8 Oct 2018
@stas00 I agree that it would be better not to have to sudo. See this issue for why it's this way: https://github.com/Microsoft/azure-pipelines-image-generation/issues/259
For what it's worth, it looks like you can move a couple lines from your script into the task:
updateConda: true
packageSpecs: 'python=$(python.version)' pip setuptools
It's probably best to keep the channel ones in the script.
brcrista
on 9 Oct 2018
Thank you for the follow up, @brcrista. I can see you're discussing a better solution here. I have a working workaround, so I will just wait until you sorted it out.
For what it's worth, it looks like you can move a couple lines from your script into the task:
I prefer to see exactly what the script is going doing, rather than hide it behind abstractions. This helps to improve/verify user installation instructions, since this is what they will need to run the correct installation. I hope this make sense.
stas00
on 9 Oct 2018
I'm also thinking that other than giving access to the shell to debug/solve pipeline problems what would be useful is to be able to edit a single task in place, and rerun just it. Just pop an in browser editor and let me edit the commands. Then once it's good, let me export it to add to the real yaml file. Otherwise it's a pretty slow debugging since every time I make a change in the pipeline I have to wait for all previous stages to run first before the problematic one gets run. That also mean that each sub-stage should be re-run-able, as it's possible that in order to test the current problematic stage, I also need to re-run one stage above.
Also, wrt to modifying the pipeline, there must be a way to isolate the experimentation with the pipeline code from the main pipeline, so while it's being debugged it doesn't affect the live PRs/commits. Since they get reported as failed/problematic, whereas the problem is in the pipeline script. I guess we really need 2 pipelines - one "production" and another "develop", so the "develop" pipeline won't work like the normal pipeline does. It will not get automatically activated by any event, but only run manually. Once it works well, it can overwrite the production one. I realize that I can probably make that happen in the current setup by creating a new pipeline and changing its configuration to be inactive unless run manually, I haven't tried that yet. But I thought it'd be a good concept to implement as a process/feature, since I think many others would benefit from it. I hope I communicated my intention clearly.
stas00
on 9 Oct 2018
I think you are exactly right. Now that we've shipped YAML our focus is on making the dev experience better. I think we have a few things coming you'll be happy about :)
I usually just make my YAML changes in a branch and run the builds off of that. I don't think you should need a separate pipeline.
brcrista
on 10 Oct 2018
Piling onto what Brian said: I usually set my triggers such that only master (and maybe a handful of other specific branches) fires a CI build. Then when you're working on the pipeline itself, you can put that up in a branch but _not_ open a PR yet. Then you queue builds manually until you're happy with the results. Only then do you open a PR and start that whole flow.
vtbassmatt
on 10 Oct 2018
@vtbassmatt and @brcrista, your recommendation about how to modify the pipeline yaml w/o affecting the master builds has been most helpful! Thank you.
I think this is something that's well worth documenting in your docs.
stas00
on 10 Oct 2018
Update on this: I would recommend YAML users move away from the Conda Environment task and just use script to work with environments. This grows up a lot better to more complex scenarios such as installing from multiple channels. It also lets you define your environment setup using an environment.yml, which I think meshes very well with azure-pipelines.yml 馃槉
I added some YAML snippets to our Python guide that I hope you find helpful: https://docs.microsoft.com/azure/devops/pipelines/languages/python#create-and-activate-an-anaconda-environment
brcrista
on 27 Nov 2018
I tried the new way as you suggested, @brcrista and there are still permission problems:
PermissionError: [Errno 13] Permission denied: '/usr/local/miniconda/.condatmp'
This is with macos recipe copied verbatim the the updated doc above:
- bash: echo "##vso[task.prependpath]$CONDA/bin"
displayName: Add conda to PATH
- bash: conda create --yes --quiet --name myEnvironment python=$PYTHON_VERSION # [other packages ...]
displayName: Create Anaconda environment
the full dump:
conda install -y pytest pytest-runner
2018-11-27T18:52:40.3733540Z [command]/bin/bash --noprofile --norc /Users/vsts/agent/2.142.1/work/_temp/8bfa63c8-3d1f-4a5c-9883-911554e2363e.sh
2018-11-27T18:52:46.5986850Z Solving environment: ...working... done
2018-11-27T18:52:46.8251330Z
2018-11-27T18:52:46.9259510Z pytest-4.0.0 | 335 KB | | 0%
2018-11-27T18:52:46.9497530Z pytest-4.0.0 | 335 KB | #########3 | 94%
2018-11-27T18:52:46.9500850Z pytest-4.0.0 | 335 KB | ########## | 100%
2018-11-27T18:52:46.9501000Z
2018-11-27T18:52:46.9797100Z pluggy-0.8.0 | 28 KB | | 0%
2018-11-27T18:52:46.9800810Z pluggy-0.8.0 | 28 KB | ########## | 100%
2018-11-27T18:52:46.9800960Z
2018-11-27T18:52:47.0554650Z py-1.7.0 | 138 KB | | 0%
2018-11-27T18:52:47.0558070Z py-1.7.0 | 138 KB | ########## | 100%
2018-11-27T18:52:47.0558190Z
2018-11-27T18:52:47.0842080Z pytest-runner-4.2 | 11 KB | | 0%
2018-11-27T18:52:47.0846050Z pytest-runner-4.2 | 11 KB | ########## | 100%
2018-11-27T18:52:47.0846200Z
2018-11-27T18:52:47.1097530Z atomicwrites-1.2.1 | 11 KB | | 0%
2018-11-27T18:52:47.1100660Z atomicwrites-1.2.1 | 11 KB | ########## | 100%
2018-11-27T18:52:47.1100910Z
2018-11-27T18:52:47.1619070Z attrs-18.2.0 | 50 KB | | 0%
2018-11-27T18:52:47.1623510Z attrs-18.2.0 | 50 KB | ########## | 100%
2018-11-27T18:52:47.1623810Z
2018-11-27T18:52:47.2138310Z more-itertools-4.3.0 | 82 KB | | 0%
2018-11-27T18:52:47.2139240Z more-itertools-4.3.0 | 82 KB | ########## | 100%
2018-11-27T18:52:47.2139470Z ## Package Plan ##
2018-11-27T18:52:47.2139530Z
2018-11-27T18:52:47.2139700Z environment location: /usr/local/miniconda
2018-11-27T18:52:47.2139810Z
2018-11-27T18:52:47.2139980Z added / updated specs:
2018-11-27T18:52:47.2140650Z - pytest
2018-11-27T18:52:47.2141190Z - pytest-runner
2018-11-27T18:52:47.2141380Z
2018-11-27T18:52:47.2141430Z
2018-11-27T18:52:47.2141600Z The following packages will be downloaded:
2018-11-27T18:52:47.2141690Z
2018-11-27T18:52:47.2141860Z package | build
2018-11-27T18:52:47.2142530Z ---------------------------|-----------------
2018-11-27T18:52:47.2143180Z pytest-4.0.0 | py37_0 335 KB
2018-11-27T18:52:47.2143880Z pluggy-0.8.0 | py37_0 28 KB
2018-11-27T18:52:47.2144540Z py-1.7.0 | py37_0 138 KB
2018-11-27T18:52:47.2145140Z pytest-runner-4.2 | py37_0 11 KB
2018-11-27T18:52:47.2145820Z atomicwrites-1.2.1 | py37_0 11 KB
2018-11-27T18:52:47.2146490Z attrs-18.2.0 | py37h28b3542_0 50 KB
2018-11-27T18:52:47.2147150Z more-itertools-4.3.0 | py37_0 82 KB
2018-11-27T18:52:47.2147760Z ------------------------------------------------------------
2018-11-27T18:52:47.2147980Z Total: 655 KB
2018-11-27T18:52:47.2148060Z
2018-11-27T18:52:47.2148230Z The following NEW packages will be INSTALLED:
2018-11-27T18:52:47.2148580Z
2018-11-27T18:52:47.2149240Z atomicwrites: 1.2.1-py37_0
2018-11-27T18:52:47.2149860Z attrs: 18.2.0-py37h28b3542_0
2018-11-27T18:52:47.2150430Z more-itertools: 4.3.0-py37_0
2018-11-27T18:52:47.2151060Z pluggy: 0.8.0-py37_0
2018-11-27T18:52:47.2151690Z py: 1.7.0-py37_0
2018-11-27T18:52:47.2152250Z pytest: 4.0.0-py37_0
2018-11-27T18:52:47.2152900Z pytest-runner: 4.2-py37_0
2018-11-27T18:52:47.2153070Z
2018-11-27T18:52:47.2153170Z The following packages will be UPDATED:
2018-11-27T18:52:47.2153310Z
2018-11-27T18:52:47.2153930Z certifi: 2018.8.24-py37_1 --> 2018.10.15-py37_0
2018-11-27T18:52:47.2154610Z cryptography: 2.3.1-py37hdbc3d79_0 --> 2.4.1-py37ha12b0ac_0
2018-11-27T18:52:47.2155280Z openssl: 1.0.2p-h1de35cc_0 --> 1.1.1a-h1de35cc_0
2018-11-27T18:52:47.2156100Z python: 3.7.0-hc167b69_0 --> 3.7.1-haf84260_3
2018-11-27T18:52:47.2156810Z sqlite: 3.24.0-ha441bb4_0 --> 3.25.3-ha441bb4_0
2018-11-27T18:52:47.2156890Z
2018-11-27T18:52:47.2157040Z
2018-11-27T18:52:47.2157200Z Downloading and Extracting Packages
2018-11-27T18:52:47.5474020Z Preparing transaction: ...working... failed
2018-11-27T18:52:47.7945040Z
2018-11-27T18:52:47.7945430Z
2018-11-27T18:52:47.7946420Z # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
2018-11-27T18:52:47.7946910Z
2018-11-27T18:52:47.7947500Z Traceback (most recent call last):
2018-11-27T18:52:47.7948990Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/exceptions.py", line 819, in __call__
2018-11-27T18:52:47.7949650Z return func(*args, **kwargs)
2018-11-27T18:52:47.7950690Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/main.py", line 78, in _main
2018-11-27T18:52:47.7951320Z exit_code = do_call(args, p)
2018-11-27T18:52:47.7952430Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/conda_argparse.py", line 77, in do_call
2018-11-27T18:52:47.7952790Z exit_code = getattr(module, func_name)(args, parser)
2018-11-27T18:52:47.7953490Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/main_install.py", line 11, in execute
2018-11-27T18:52:47.7954230Z install(args, parser, 'install')
2018-11-27T18:52:47.7954930Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/install.py", line 253, in install
2018-11-27T18:52:47.7955220Z handle_txn(unlink_link_transaction, prefix, args, newenv)
2018-11-27T18:52:47.7955880Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/install.py", line 282, in handle_txn
2018-11-27T18:52:47.7956140Z unlink_link_transaction.execute()
2018-11-27T18:52:47.7956870Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/link.py", line 223, in execute
2018-11-27T18:52:47.7957020Z self.verify()
2018-11-27T18:52:47.7957720Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/common/io.py", line 46, in decorated
2018-11-27T18:52:47.7957960Z return f(*args, **kwds)
2018-11-27T18:52:47.7958590Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/link.py", line 200, in verify
2018-11-27T18:52:47.7958810Z self.prepare()
2018-11-27T18:52:47.7959490Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/link.py", line 192, in prepare
2018-11-27T18:52:47.7959730Z stp.remove_specs, stp.update_specs)
2018-11-27T18:52:47.7960370Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/link.py", line 282, in _prepare
2018-11-27T18:52:47.7961040Z mkdir_p(transaction_context['temp_dir'])
2018-11-27T18:52:47.7961820Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/gateways/disk/__init__.py", line 60, in mkdir_p
2018-11-27T18:52:47.7962040Z makedirs(path)
2018-11-27T18:52:47.7962180Z File "/usr/local/miniconda/lib/python3.7/os.py", line 221, in makedirs
2018-11-27T18:52:47.7962760Z mkdir(name, mode)
2018-11-27T18:52:47.7963480Z PermissionError: [Errno 13] Permission denied: '/usr/local/miniconda/.condatmp'
@stas00 I tested it on Hosted macOS and it worked for me. I see your build is running the command line
conda install -y pytest pytest-runner
And then it's trying to install to the base directory:
2018-11-27T18:52:47.2139700Z environment location: /usr/local/miniconda
That is where the permissions error comes from, since it doesn't have access to write to /usr/local. Did you activate the environment before running the install line? Each build step runs in its own process, so you need to activate the environment for each step.
brcrista
on 27 Nov 2018
It doesn't seem to help, lots of other problems:
- job: 'MacOS_Conda'
pool:
vmImage: 'macOS-10.13'
# Run the pipeline with multiple Python versions
strategy:
matrix:
Python36:
python.version: '3.6'
# Increase the maxParallel value to simultaneously run the job for all versions in the matrix (max 10 for free open-source)
maxParallel: 4
steps:
# Set the UsePythonVersion task to reference the matrix variable for its Python version
- task: UsePythonVersion@0
inputs:
versionSpec: '$(python.version)'
architecture: 'x64'
- bash: echo "##vso[task.prependpath]$CONDA/bin"
displayName: Add conda to PATH
- bash: |
conda create --yes --quiet --name fastai-cpu python=$PYTHON_VERSION
displayName: Create Anaconda environment
# Install dependencies and fastai git HEAD or PR
- script: |
conda activate fastai-cpu
conda install -y conda
conda install -y pip setuptools
gives (the last step):
2018-11-27T23:25:58.8896800Z ##[section]Starting: Upgrade pip/conda/setuptools. Install deps, pytorch and fastai
2018-11-27T23:25:58.8901300Z ==============================================================================
2018-11-27T23:25:58.8901470Z Task : Command Line
2018-11-27T23:25:58.8901680Z Description : Run a command line script using cmd.exe on Windows and bash on macOS and Linux.
2018-11-27T23:25:58.8901870Z Version : 2.142.2
2018-11-27T23:25:58.8901990Z Author : Microsoft Corporation
2018-11-27T23:25:58.8902220Z Help : [More Information](https://go.microsoft.com/fwlink/?LinkID=613735)
2018-11-27T23:25:58.8902440Z ==============================================================================
2018-11-27T23:25:59.0617020Z Generating script.
2018-11-27T23:25:59.0677580Z [command]/bin/bash --noprofile --norc /Users/vsts/agent/2.142.1/work/_temp/4766b238-6780-41bd-b41d-36e885647124.sh
2018-11-27T23:25:59.2541790Z
2018-11-27T23:25:59.2543730Z CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
2018-11-27T23:25:59.2544100Z If your shell is Bash or a Bourne variant, enable conda for the current user with
2018-11-27T23:25:59.2544670Z
2018-11-27T23:25:59.2544900Z $ echo ". /usr/local/miniconda/etc/profile.d/conda.sh" >> ~/.bash_profile
2018-11-27T23:25:59.2545180Z
2018-11-27T23:25:59.2545400Z or, for all users, enable conda with
2018-11-27T23:25:59.2546260Z
2018-11-27T23:25:59.2547530Z $ sudo ln -s /usr/local/miniconda/etc/profile.d/conda.sh /etc/profile.d/conda.sh
2018-11-27T23:25:59.2547780Z
2018-11-27T23:25:59.2548550Z The options above will permanently enable the 'conda' command, but they do NOT
2018-11-27T23:25:59.2549270Z put conda's base (root) environment on PATH. To do so, run
2018-11-27T23:25:59.2549380Z
2018-11-27T23:25:59.2549570Z $ conda activate
2018-11-27T23:25:59.2549650Z
2018-11-27T23:25:59.2549840Z in your terminal, or to put the base environment on PATH permanently, run
2018-11-27T23:25:59.2549930Z
2018-11-27T23:25:59.2550120Z $ echo "conda activate" >> ~/.bash_profile
2018-11-27T23:25:59.2550260Z
2018-11-27T23:25:59.2550540Z Previous to conda 4.4, the recommended way to activate conda was to modify PATH in
2018-11-27T23:25:59.2550800Z your ~/.bash_profile file. You should manually remove the line that looks like
2018-11-27T23:25:59.2550890Z
2018-11-27T23:25:59.2551080Z export PATH="/usr/local/miniconda/bin:$PATH"
2018-11-27T23:25:59.2551160Z
2018-11-27T23:25:59.2551360Z ^^^ The above line should NO LONGER be in your ~/.bash_profile file! ^^^
2018-11-27T23:25:59.2551450Z
2018-11-27T23:25:59.2551590Z
2018-11-27T23:26:04.3607080Z Solving environment: ...working... done
2018-11-27T23:26:04.4951360Z
2018-11-27T23:26:04.5953990Z python-3.7.1 | 21.1 MB | | 0%
2018-11-27T23:26:04.6954580Z python-3.7.1 | 21.1 MB | ##6 | 26%
2018-11-27T23:26:04.8225380Z python-3.7.1 | 21.1 MB | #####9 | 60%
2018-11-27T23:26:06.4304100Z python-3.7.1 | 21.1 MB | #######5 | 76%
2018-11-27T23:26:07.9831890Z python-3.7.1 | 21.1 MB | ######### | 90%
2018-11-27T23:26:07.9834160Z python-3.7.1 | 21.1 MB | ########## | 100%
2018-11-27T23:26:07.9834460Z
2018-11-27T23:26:08.1413150Z cryptography-2.4.1 | 574 KB | | 0%
2018-11-27T23:26:08.1919340Z cryptography-2.4.1 | 574 KB | #########2 | 92%
2018-11-27T23:26:08.1923070Z cryptography-2.4.1 | 574 KB | ########## | 100%
2018-11-27T23:26:08.1923340Z
2018-11-27T23:26:08.2336260Z certifi-2018.10.15 | 138 KB | | 0%
2018-11-27T23:26:08.2337260Z certifi-2018.10.15 | 138 KB | ########## | 100%
2018-11-27T23:26:08.2337520Z ## Package Plan ##
2018-11-27T23:26:08.2337590Z
2018-11-27T23:26:08.2337780Z environment location: /usr/local/miniconda
2018-11-27T23:26:08.2337870Z
2018-11-27T23:26:08.2338040Z added / updated specs:
2018-11-27T23:26:08.2338600Z - conda
2018-11-27T23:26:08.2338780Z
2018-11-27T23:26:08.2338850Z
2018-11-27T23:26:08.2339020Z The following packages will be downloaded:
2018-11-27T23:26:08.2339370Z
2018-11-27T23:26:08.2339660Z package | build
2018-11-27T23:26:08.2340320Z ---------------------------|-----------------
2018-11-27T23:26:08.2341040Z python-3.7.1 | haf84260_3 21.1 MB
2018-11-27T23:26:08.2341760Z cryptography-2.4.1 | py37ha12b0ac_0 574 KB
2018-11-27T23:26:08.2342440Z certifi-2018.10.15 | py37_0 138 KB
2018-11-27T23:26:08.2343070Z ------------------------------------------------------------
2018-11-27T23:26:08.2343320Z Total: 21.8 MB
2018-11-27T23:26:08.2343410Z
2018-11-27T23:26:08.2343600Z The following packages will be UPDATED:
2018-11-27T23:26:08.2343670Z
2018-11-27T23:26:08.2344380Z certifi: 2018.8.24-py37_1 --> 2018.10.15-py37_0
2018-11-27T23:26:08.2345070Z cryptography: 2.3.1-py37hdbc3d79_0 --> 2.4.1-py37ha12b0ac_0
2018-11-27T23:26:08.2345710Z openssl: 1.0.2p-h1de35cc_0 --> 1.1.1a-h1de35cc_0
2018-11-27T23:26:08.2346420Z python: 3.7.0-hc167b69_0 --> 3.7.1-haf84260_3
2018-11-27T23:26:08.2347380Z sqlite: 3.24.0-ha441bb4_0 --> 3.25.3-ha441bb4_0
2018-11-27T23:26:08.2347480Z
2018-11-27T23:26:08.2347640Z
2018-11-27T23:26:08.2347750Z Downloading and Extracting Packages
2018-11-27T23:26:08.5695600Z Preparing transaction: ...working... failed
2018-11-27T23:26:08.8410960Z
2018-11-27T23:26:08.8411340Z
2018-11-27T23:26:08.8411980Z # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
2018-11-27T23:26:08.8412260Z
2018-11-27T23:26:08.8413570Z Traceback (most recent call last):
2018-11-27T23:26:08.8415400Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/exceptions.py", line 819, in __call__
2018-11-27T23:26:08.8415760Z return func(*args, **kwargs)
2018-11-27T23:26:08.8416500Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/main.py", line 78, in _main
2018-11-27T23:26:08.8416690Z exit_code = do_call(args, p)
2018-11-27T23:26:08.8417390Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/conda_argparse.py", line 77, in do_call
2018-11-27T23:26:08.8417670Z exit_code = getattr(module, func_name)(args, parser)
2018-11-27T23:26:08.8418380Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/main_install.py", line 11, in execute
2018-11-27T23:26:08.8418960Z install(args, parser, 'install')
2018-11-27T23:26:08.8419660Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/install.py", line 253, in install
2018-11-27T23:26:08.8419910Z handle_txn(unlink_link_transaction, prefix, args, newenv)
2018-11-27T23:26:08.8420540Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/install.py", line 282, in handle_txn
2018-11-27T23:26:08.8420790Z unlink_link_transaction.execute()
2018-11-27T23:26:08.8421480Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/link.py", line 223, in execute
2018-11-27T23:26:08.8421730Z self.verify()
2018-11-27T23:26:08.8422440Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/common/io.py", line 46, in decorated
2018-11-27T23:26:08.8422700Z return f(*args, **kwds)
2018-11-27T23:26:08.8423390Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/link.py", line 200, in verify
2018-11-27T23:26:08.8423550Z self.prepare()
2018-11-27T23:26:08.8424220Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/link.py", line 192, in prepare
2018-11-27T23:26:08.8424460Z stp.remove_specs, stp.update_specs)
2018-11-27T23:26:08.8425080Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/link.py", line 282, in _prepare
2018-11-27T23:26:08.8425750Z mkdir_p(transaction_context['temp_dir'])
2018-11-27T23:26:08.8426470Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/gateways/disk/__init__.py", line 60, in mkdir_p
2018-11-27T23:26:08.8427120Z makedirs(path)
2018-11-27T23:26:08.8427260Z File "/usr/local/miniconda/lib/python3.7/os.py", line 221, in makedirs
2018-11-27T23:26:08.8427480Z mkdir(name, mode)
2018-11-27T23:26:08.8428160Z PermissionError: [Errno 13] Permission denied: '/usr/local/miniconda/.condatmp'
2018-11-27T23:26:08.8428280Z
2018-11-27T23:26:08.8428890Z `$ /usr/local/miniconda/bin/conda install -y conda`
2018-11-27T23:26:08.8428990Z
2018-11-27T23:26:08.8429160Z environment variables:
2018-11-27T23:26:08.8429290Z BUILD_REPOSITORY_LOCALPATH=/Users/vsts/agent/2.142.1/work/1/s
2018-11-27T23:26:08.8429500Z CIO_TEST=<not set>
2018-11-27T23:26:08.8429690Z CONDA=/usr/local/miniconda
2018-11-27T23:26:08.8429820Z CONDA_ROOT=/usr/local/miniconda
2018-11-27T23:26:08.8430020Z NUNIT3_PATH=/Library/Developer/nunit/3.6.0
2018-11-27T23:26:08.8430220Z NUNIT_BASE_PATH=/Library/Developer/nunit
2018-11-27T23:26:08.8430440Z PATH=/usr/local/miniconda/bin:/Users/vsts/hostedtoolcache/Python/3.6.5/x64/
2018-11-27T23:26:08.8430610Z bin:/Users/vsts/hostedtoolcache/Python/3.6.5/x64:/usr/local/bin:/usr/l
2018-11-27T23:26:08.8431060Z ocal/sbin:/Users/vsts/bin:/Users/vsts/.yarn/bin:/usr/local/go/bin:/Use
2018-11-27T23:26:08.8431290Z rs/vsts/Library/Android/sdk/tools:/Users/vsts/Library/Android/sdk/plat
2018-11-27T23:26:08.8431940Z form-tools:/Users/vsts/Library/Android/sdk/ndk-bundle:/Library/Framewo
2018-11-27T23:26:08.8432200Z rks/Mono.framework/Versions/Current/Commands:/usr/bin:/bin:/usr/sbin:/
2018-11-27T23:26:08.8432420Z sbin:/Users/vsts/.azcopy
2018-11-27T23:26:08.8432600Z PYTHON_VERSION=3.6
2018-11-27T23:26:08.8432730Z REQUESTS_CA_BUNDLE=<not set>
2018-11-27T23:26:08.8432910Z SSL_CERT_FILE=<not set>
2018-11-27T23:26:08.8433000Z
2018-11-27T23:26:08.8433180Z active environment : None
2018-11-27T23:26:08.8433370Z user config file : /Users/vsts/.condarc
2018-11-27T23:26:08.8433500Z populated config files :
2018-11-27T23:26:08.8433680Z conda version : 4.5.11
2018-11-27T23:26:08.8434300Z conda-build version : not installed
2018-11-27T23:26:08.8434440Z python version : 3.7.0.final.0
2018-11-27T23:26:08.8434640Z base environment : /usr/local/miniconda (read only)
2018-11-27T23:26:08.8435300Z channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
2018-11-27T23:26:08.8435470Z https://repo.anaconda.com/pkgs/main/noarch
2018-11-27T23:26:08.8436130Z https://repo.anaconda.com/pkgs/free/osx-64
2018-11-27T23:26:08.8436370Z https://repo.anaconda.com/pkgs/free/noarch
2018-11-27T23:26:08.8437030Z https://repo.anaconda.com/pkgs/r/osx-64
2018-11-27T23:26:08.8437190Z https://repo.anaconda.com/pkgs/r/noarch
2018-11-27T23:26:08.8437850Z https://repo.anaconda.com/pkgs/pro/osx-64
2018-11-27T23:26:08.8438100Z https://repo.anaconda.com/pkgs/pro/noarch
2018-11-27T23:26:08.8438250Z package cache : /usr/local/miniconda/pkgs
2018-11-27T23:26:08.8438450Z /Users/vsts/.conda/pkgs
2018-11-27T23:26:08.8438730Z envs directories : /Users/vsts/.conda/envs
2018-11-27T23:26:08.8438920Z /usr/local/miniconda/envs
2018-11-27T23:26:08.8439480Z platform : osx-64
2018-11-27T23:26:08.8440180Z user-agent : conda/4.5.11 requests/2.19.1 CPython/3.7.0 Darwin/17.7.0 OSX/10.13.6
2018-11-27T23:26:08.8440410Z UID:GID : 501:20
2018-11-27T23:26:08.8440540Z netrc file : None
2018-11-27T23:26:08.8440730Z offline mode : False
2018-11-27T23:26:08.8440800Z
2018-11-27T23:26:08.8440930Z
2018-11-27T23:26:08.8441240Z An unexpected error has occurred. Conda has prepared the above report.
2018-11-27T23:26:08.8441440Z Upload successful.
2018-11-27T23:26:14.0700190Z Solving environment: ...working... done
2018-11-27T23:26:14.2095710Z
2018-11-27T23:26:14.3240930Z setuptools-40.6.2 | 621 KB | | 0%
2018-11-27T23:26:14.4261330Z setuptools-40.6.2 | 621 KB | ########5 | 85%
2018-11-27T23:26:14.4265130Z setuptools-40.6.2 | 621 KB | ########## | 100%
2018-11-27T23:26:14.4265580Z
2018-11-27T23:26:14.5727600Z pip-18.1 | 1.7 MB | | 0%
2018-11-27T23:26:14.8766760Z pip-18.1 | 1.7 MB | #######8 | 79%
2018-11-27T23:26:14.9697320Z pip-18.1 | 1.7 MB | #########5 | 95%
2018-11-27T23:26:14.9698770Z pip-18.1 | 1.7 MB | ########## | 100%
2018-11-27T23:26:14.9699350Z ## Package Plan ##
2018-11-27T23:26:14.9699470Z
2018-11-27T23:26:14.9699670Z environment location: /usr/local/miniconda
2018-11-27T23:26:14.9699810Z
2018-11-27T23:26:14.9700040Z added / updated specs:
2018-11-27T23:26:14.9700810Z - pip
2018-11-27T23:26:14.9701390Z - setuptools
2018-11-27T23:26:14.9702050Z
2018-11-27T23:26:14.9702140Z
2018-11-27T23:26:14.9702320Z The following packages will be downloaded:
2018-11-27T23:26:14.9702420Z
2018-11-27T23:26:14.9702590Z package | build
2018-11-27T23:26:14.9703240Z ---------------------------|-----------------
2018-11-27T23:26:14.9703960Z setuptools-40.6.2 | py37_0 621 KB
2018-11-27T23:26:14.9704650Z pip-18.1 | py37_0 1.7 MB
2018-11-27T23:26:14.9705280Z ------------------------------------------------------------
2018-11-27T23:26:14.9705530Z Total: 2.3 MB
2018-11-27T23:26:14.9705680Z
2018-11-27T23:26:14.9705810Z The following packages will be UPDATED:
2018-11-27T23:26:14.9705950Z
2018-11-27T23:26:14.9706560Z certifi: 2018.8.24-py37_1 --> 2018.10.15-py37_0
2018-11-27T23:26:14.9707300Z cryptography: 2.3.1-py37hdbc3d79_0 --> 2.4.1-py37ha12b0ac_0
2018-11-27T23:26:14.9707990Z openssl: 1.0.2p-h1de35cc_0 --> 1.1.1a-h1de35cc_0
2018-11-27T23:26:14.9708640Z pip: 10.0.1-py37_0 --> 18.1-py37_0
2018-11-27T23:26:14.9709350Z python: 3.7.0-hc167b69_0 --> 3.7.1-haf84260_3
2018-11-27T23:26:14.9710040Z setuptools: 40.2.0-py37_0 --> 40.6.2-py37_0
2018-11-27T23:26:14.9710680Z sqlite: 3.24.0-ha441bb4_0 --> 3.25.3-ha441bb4_0
2018-11-27T23:26:14.9710870Z
2018-11-27T23:26:14.9710930Z
2018-11-27T23:26:14.9711120Z Downloading and Extracting Packages
2018-11-27T23:26:15.2720610Z Preparing transaction: ...working... failed
2018-11-27T23:26:15.4851510Z
2018-11-27T23:26:15.4852030Z
2018-11-27T23:26:15.4852680Z # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
2018-11-27T23:26:15.4852970Z
2018-11-27T23:26:15.4853490Z Traceback (most recent call last):
2018-11-27T23:26:15.4855190Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/exceptions.py", line 819, in __call__
2018-11-27T23:26:15.4855520Z return func(*args, **kwargs)
2018-11-27T23:26:15.4856290Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/main.py", line 78, in _main
2018-11-27T23:26:15.4856630Z exit_code = do_call(args, p)
2018-11-27T23:26:15.4857390Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/conda_argparse.py", line 77, in do_call
2018-11-27T23:26:15.4857570Z exit_code = getattr(module, func_name)(args, parser)
2018-11-27T23:26:15.4858310Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/main_install.py", line 11, in execute
2018-11-27T23:26:15.4859010Z install(args, parser, 'install')
2018-11-27T23:26:15.4859740Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/install.py", line 253, in install
2018-11-27T23:26:15.4859930Z handle_txn(unlink_link_transaction, prefix, args, newenv)
2018-11-27T23:26:15.4863010Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/install.py", line 282, in handle_txn
2018-11-27T23:26:15.4863280Z unlink_link_transaction.execute()
2018-11-27T23:26:15.4863950Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/link.py", line 223, in execute
2018-11-27T23:26:15.4864210Z self.verify()
2018-11-27T23:26:15.4864940Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/common/io.py", line 46, in decorated
2018-11-27T23:26:15.4865170Z return f(*args, **kwds)
2018-11-27T23:26:15.4865830Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/link.py", line 200, in verify
2018-11-27T23:26:15.4866080Z self.prepare()
2018-11-27T23:26:15.4866800Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/link.py", line 192, in prepare
2018-11-27T23:26:15.4866970Z stp.remove_specs, stp.update_specs)
2018-11-27T23:26:15.4867750Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/link.py", line 282, in _prepare
2018-11-27T23:26:15.4868720Z mkdir_p(transaction_context['temp_dir'])
2018-11-27T23:26:15.4869650Z File "/usr/local/miniconda/lib/python3.7/site-packages/conda/gateways/disk/__init__.py", line 60, in mkdir_p
2018-11-27T23:26:15.4869920Z makedirs(path)
2018-11-27T23:26:15.4870130Z File "/usr/local/miniconda/lib/python3.7/os.py", line 221, in makedirs
2018-11-27T23:26:15.4870330Z mkdir(name, mode)
2018-11-27T23:26:15.4870970Z PermissionError: [Errno 13] Permission denied: '/usr/local/miniconda/.condatmp'
2018-11-27T23:26:15.4871170Z
2018-11-27T23:26:15.4871750Z `$ /usr/local/miniconda/bin/conda install -y pip setuptools`
2018-11-27T23:26:15.4871940Z
2018-11-27T23:26:15.4872060Z environment variables:
2018-11-27T23:26:15.4872270Z BUILD_REPOSITORY_LOCALPATH=/Users/vsts/agent/2.142.1/work/1/s
2018-11-27T23:26:15.4872480Z CIO_TEST=<not set>
2018-11-27T23:26:15.4872630Z CONDA=/usr/local/miniconda
2018-11-27T23:26:15.4872830Z CONDA_ROOT=/usr/local/miniconda
2018-11-27T23:26:15.4873050Z NUNIT3_PATH=/Library/Developer/nunit/3.6.0
2018-11-27T23:26:15.4873200Z NUNIT_BASE_PATH=/Library/Developer/nunit
2018-11-27T23:26:15.4873430Z PATH=/usr/local/miniconda/bin:/Users/vsts/hostedtoolcache/Python/3.6.5/x64/
2018-11-27T23:26:15.4873670Z bin:/Users/vsts/hostedtoolcache/Python/3.6.5/x64:/usr/local/bin:/usr/l
2018-11-27T23:26:15.4873910Z ocal/sbin:/Users/vsts/bin:/Users/vsts/.yarn/bin:/usr/local/go/bin:/Use
2018-11-27T23:26:15.4874080Z rs/vsts/Library/Android/sdk/tools:/Users/vsts/Library/Android/sdk/plat
2018-11-27T23:26:15.4874820Z form-tools:/Users/vsts/Library/Android/sdk/ndk-bundle:/Library/Framewo
2018-11-27T23:26:15.4875090Z rks/Mono.framework/Versions/Current/Commands:/usr/bin:/bin:/usr/sbin:/
2018-11-27T23:26:15.4875260Z sbin:/Users/vsts/.azcopy
2018-11-27T23:26:15.4875590Z PYTHON_VERSION=3.6
2018-11-27T23:26:15.4875720Z REQUESTS_CA_BUNDLE=<not set>
2018-11-27T23:26:15.4875930Z SSL_CERT_FILE=<not set>
2018-11-27T23:26:15.4876000Z
2018-11-27T23:26:15.4876190Z active environment : None
2018-11-27T23:26:15.4876380Z user config file : /Users/vsts/.condarc
2018-11-27T23:26:15.4876510Z populated config files :
2018-11-27T23:26:15.4876720Z conda version : 4.5.11
2018-11-27T23:26:15.4877380Z conda-build version : not installed
2018-11-27T23:26:15.4877530Z python version : 3.7.0.final.0
2018-11-27T23:26:15.4877840Z base environment : /usr/local/miniconda (read only)
2018-11-27T23:26:15.4878610Z channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
2018-11-27T23:26:15.4878780Z https://repo.anaconda.com/pkgs/main/noarch
2018-11-27T23:26:15.4880530Z https://repo.anaconda.com/pkgs/free/osx-64
2018-11-27T23:26:15.4880800Z https://repo.anaconda.com/pkgs/free/noarch
2018-11-27T23:26:15.4881510Z https://repo.anaconda.com/pkgs/r/osx-64
2018-11-27T23:26:15.4881680Z https://repo.anaconda.com/pkgs/r/noarch
2018-11-27T23:26:15.4882360Z https://repo.anaconda.com/pkgs/pro/osx-64
2018-11-27T23:26:15.4882620Z https://repo.anaconda.com/pkgs/pro/noarch
2018-11-27T23:26:15.4882770Z package cache : /usr/local/miniconda/pkgs
2018-11-27T23:26:15.4882980Z /Users/vsts/.conda/pkgs
2018-11-27T23:26:15.4883190Z envs directories : /Users/vsts/.conda/envs
2018-11-27T23:26:15.4883320Z /usr/local/miniconda/envs
2018-11-27T23:26:15.4883980Z platform : osx-64
2018-11-27T23:26:15.4884720Z user-agent : conda/4.5.11 requests/2.19.1 CPython/3.7.0 Darwin/17.7.0 OSX/10.13.6
2018-11-27T23:26:15.4884970Z UID:GID : 501:20
2018-11-27T23:26:15.4885290Z netrc file : None
2018-11-27T23:26:15.4885500Z offline mode : False
2018-11-27T23:26:15.4885580Z
2018-11-27T23:26:15.4885710Z
2018-11-27T23:26:15.4885830Z An unexpected error has occurred. Conda has prepared the above report.
2018-11-27T23:26:15.4886050Z Upload successful.
Clearly I didn't touch ~/.bash_profile that it complains about.
2018-11-27T23:39:20.8453170Z Previous to conda 4.4, the recommended way to activate conda was to modify PATH in
2018-11-27T23:39:20.8453360Z your ~/.bash_profile file. You should manually remove the line that looks like
2018-11-27T23:39:20.8456170Z
2018-11-27T23:39:20.8456340Z export PATH="/usr/local/miniconda/bin:$PATH"
2018-11-27T23:39:20.8456410Z
2018-11-27T23:39:20.8456580Z ^^^ The above line should NO LONGER be in your ~/.bash_profile file! ^^^
I even tried to add an extra line to appease to conda, as it suggested in the error.
- bash: |
conda create --yes --quiet --name fastai-cpu python=$PYTHON_VERSION
echo ". /usr/local/miniconda/etc/profile.d/conda.sh" >> ~/.bash_profile
displayName: Create Anaconda environment
but to no avail.
I had to do something similar on my own machine when conda was updated to 4.4 recently, except on CI I have no way to edit any files.
@brcrista, when you said it worked for you, did you check the logs to see that it worked, or just noting that the task was marked as successful? It reports success despite all these problems.
stas00
on 28 Nov 2018
In your script where you activate the environment you should have source activate, not conda activate.
brcrista
on 28 Nov 2018
Not since conda 4.4, where conda activate replaces source activate (while the latter still works). See https://stackoverflow.com/a/49600711/9201239. conda's online docs aren't up-to-date.
From the release docs, regarding the release of conda version 4.4.0 (released december 2017):
https://conda.io/docs/release-notes.html#id71
conda activate: The logic and mechanisms underlying environment activation have been reworked. With conda 4.4, conda activate and conda deactivate are now the preferred commands for activating and deactivating environments. You鈥檒l find they are much more snappy than the source activate and source deactivate commands from previous conda versions. The conda activate command also has advantages of (1) being universal across all OSes, shells, and platforms, and (2) not having path collisions with scripts from other packages like python virtualenv鈥檚 activate script.
stas00
on 28 Nov 2018
I see, thanks. I didn't realize that conda activate was superseding source activate. It does sound better, and it will be good for CI that it's cross-platform.
It sounds like we need to do some configuration on the agent to get conda activate to work. I'm going to be making some changes to the agents soon, so I'll make a note to look into that. (The CI environment runs in --noprofile mode, which is why adding the line to .bash_profile didn't work.)
Did you try source activate? I think that should unblock you for now.
brcrista
on 28 Nov 2018
Oh, I'm not blocked, I still successfully use the old setup with the permission fixing workaround. It's limping (conda activate fails), but it sorts itself out, installing packages into its default env, which for CI purposes is even better - why bother with custom env, when it's scraped anyway. And I don't need to conda activate in every task!
I was just trying to move to the latest setup as you suggested a few days ago and that attempt wasn't getting me anywhere. So my old setup still works.
Looking forward to trying the new setup with conda activate once you guys make it work.
I currently use:
steps:
# Set the UsePythonVersion task to reference the matrix variable for its Python version
- task: UsePythonVersion@0
inputs:
versionSpec: '$(python.version)'
architecture: 'x64'
- script: sudo install -d -m 0777 /usr/local/miniconda/envs/
displayName: Fix Conda permissions
# Conda setup environment.
# https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/package/conda-environment?view=vsts
#
- task: CondaEnvironment@0
inputs:
environmentName: 'fastai-cpu'
packageSpecs: 'python=$(python.version)'
# Install dependencies and fastai git HEAD or PR
- script: |
conda activate fastai-cpu
conda install -y conda
conda install -y pip setuptools
Still getting permission errors using the setup described in https://docs.microsoft.com/en-us/azure/devops/pipelines/languages/anaconda?view=azure-devops&tabs=ubuntu-16-04 Here is the build output even after I tried adjusting the permissions: https://dev.azure.com/pyviz/PyViz/_build/results?buildId=5
philippjfr
on 23 Feb 2019
@philippjfr this is a known issue with the Miniconda setup on Hosted Ubuntu -- discussion at https://github.com/Microsoft/azure-pipelines-image-generation/issues/259. It should only happen if you try to install packages globally.
You should be able to work around by chmod'ing Miniconda's installation directory to be writable on the hosted agent. I will add a YAML snippet for that to the doc.
brcrista
on 25 Feb 2019
That doc now has a section on how to install packages globally.
brcrista
on 26 Feb 2019
Thanks for the quick action on this @brcrista, much appreciated!
philippjfr
on 26 Feb 2019
Related issues
MikahB
路
3Comments
richard-ob
路
3Comments
jared-hexagon
路
3Comments
Mardoxx
路
3Comments
NatMarchand
路
3Comments
Most helpful comment
I think you are exactly right. Now that we've shipped YAML our focus is on making the dev experience better. I think we have a few things coming you'll be happy about :)
I usually just make my YAML changes in a branch and run the builds off of that. I don't think you should need a separate pipeline.