Azure-devops-docs: Please provide directions for Caching Anaconda Envs

[dciborow]

I think it's something like this...

• task: Cache@2
  inputs:
  key: ${{parameters.conda}}
  restoreKeys: ${{parameters.conda}}
  path: /data/anaconda/envs/${{parameters.conda}}

And I follow that with....

• task: AzureCLI@1
  displayName: 'Setup Conda'
  condition: eq('${{ parameters.fresh_install }}', false)
  inputs:
  azureSubscription: ${{parameters.azureSubscription}}
  failOnStandardError: True
  scriptLocation: inlineScript
  inlineScript: |
  conda config --set notify_outdated_conda false
  conda env update --file ${{parameters.conda_location}}/environment.yml python=3.6
  source activate ${{parameters.conda}}
  python -m ipykernel install --prefix=/data/anaconda/envs/${{parameters.conda}} --name ${{parameters.conda}}

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 77c20289-dab2-7a6e-4f63-4b30eaff2c71
• Version Independent ID: d75cc196-90c1-8291-9907-ff405bde4edf
• Content: Pipeline caching - Azure Pipelines
• Content Source: docs/pipelines/caching/index.md
• Product: devops
• Technology: devops-cicd
• GitHub Login: @chasewilson
• Microsoft Alias: phwilson

[WilliamAntonRohm]

@dciborow -- Daniel, thank you for your feedback. It looks like you're working through an issue with your scenario or implementation. You may find answers here:

• Azure DevOps Support Bot

• Azure DevOps on Stack Overflow

  @chasewilson -- Chase, please look into this potential documentation update.

[josiahlaivins]

We have examples of C++,PHP, javascript. We do not have examples of caching anything related to Python (which is not a niche language/use case). Including an example for Anaconda would at least be a good compromise for those that want to cache python dependencies.

[epassaro]

This would be really nice.

[jja262]

I would also like to have instructions for caching the dependencies of a Conda environment in Azure DevOps. I tried following the Python/Pip instructions . I am able to cache the dependencies, but my CI/CD pipeline still seems to download packages from the internet when I build the environment.

[epassaro]

Hi! Any advance on this?

[jja262]

The following code works for me. The cache key is based on a hash of requirements.txt, so if my project has new requirements, then it will download all dependencies again. In the last section, if the cache failed to restore, then it downloads all dependencies.

Some of the requirements don't seem to get included in the cache, which is why the code has a section for downloading additional dependencies listed in azure_uncached_requirements.txt.

I'm not an expert at this, and the code below probably isn't perfect, but it runs. I'm passing this along in case it's a useful starting point for others.

- job: Default
  timeoutInMinutes: 150

  pool:
    vmImage: 'vs2017-win2016'

  steps:
  - powershell: Write-Host "##vso[task.prependpath]$env:CONDA\Scripts"
    displayName: Add conda to PATH

  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.7.9' 
      addToPath: true 
      architecture: 'x64'

  - script: |
      call activate base
      conda install python==3.7.6
      conda list
    displayName:  List packages

  - task: Cache@2
    inputs:
      key: 'python | "$(Agent.OS)" | requirements.txt'
      restoreKeys: | 
        python | "$(Agent.OS)"
        python
      path: $(workingDirectory)
      cacheHitVar: CONDA_CACHE_RESTORED
    displayName: Cache dependencies
    condition: not(startsWith(variables['Build.SourceBranch'], 'refs/tags/'))

  - script: |
      call activate base
      conda install --file azure_uncached_requirements.txt -c defaults -c conda-forge -y
    workingDirectory: $(Build.SourcesDirectory)
    displayName: "Download uncached dependencies"
    condition: and(not(startsWith(variables['Build.SourceBranch'], 'refs/tags/')), not(eq(variables.CONDA_CACHE_RESTORED, false)))

  - script: |
      call activate base
      conda install --file requirements.txt -c defaults -c conda-forge -c pvlib -y
    workingDirectory: $(Build.SourcesDirectory)
    displayName: "Download all dependencies"
    condition: or(startsWith(variables['Build.SourceBranch'], 'refs/tags/'), eq(variables.CONDA_CACHE_RESTORED, false))

[makukl]

Here (https://stackoverflow.com/questions/62420695/how-to-cache-pip-packages-within-azure-pipelines) is SO thread where someone gives an example of how to make caching work with conda. This also worked for me