Cloud-custodian: Invalid c7n.output.blob

[engineertree5]

I'd like to preface my "issue" with this. I'm fairly new to cloud and how *everything works, so raising this as an issue my be the wrong thing to do. However, after troubleshooting these errors for a week I'm looking to see if I can get some assistance with the issue i'm seeing when using c7n on azure as a docker container.

Summary

I'm using c7n as a docker container which is running on AKS within azure cloud. I've been cloning down the latest cloud-custodian build and building docker images to run across my subscription. I have zero issues when running custodian against resources and pulling back data. It works lightning fast is super easy! I've even got the mailer functionality working too! The problem comes into play when I try to save the output to a container/blob....

Error

Steps to recreate

• clone latest cloud-custodian build 02/07/2020
• create policy.yml file && update Docker file to run policy.yml
• update CMD to save to blob storage
  
  
  
  • CMD ["run", "-c", "policy.yml", "-s", "azure//storageaccount.blob.core.windows.net/c7n-logs"]
  
  

When following these steps i'm getting the following error:

$ kubectl logs c7n-policy-5dddddd-hhhhh
2020-02-07 18:20:22,370: custodian.azure.session:INFO Authenticated [MSI | xxxxx-xxxxx-xxxx-xxxxxx]
2020-02-07 18:20:22,371: custodian.commands:ERROR Error while executing policy my-first-policy, continuing
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/c7n/commands.py", line 285, in run
    policy()
  File "/usr/local/lib/python3.7/site-packages/c7n/policy.py", line 1054, in __call__
    resources = mode.run()
  File "/usr/local/lib/python3.7/site-packages/c7n/policy.py", line 283, in run
    with self.policy.ctx:
  File "/usr/local/lib/python3.7/site-packages/c7n/ctx.py", line 85, in __enter__
    self.initialize()
  File "/usr/local/lib/python3.7/site-packages/c7n/ctx.py", line 59, in initialize
    self.output = blob_outputs.select(self.options.output_dir, self)
  File "/usr/local/lib/python3.7/site-packages/c7n/output.py", line 66, in select
    selector))
c7n.exceptions.InvalidOutputConfig: Invalid c7n.output.blob: azure://storageaccount.blob.core.windows.net/c7n-logs

I also removed the -s azure//storageaccount.blob.core.windows.net/c7n-logs arg and created another docker image. That image exited without issue...

2020-02-07 18:23:09,606: custodian.azure.session:INFO Authenticated [MSI | xxxx-xxxx-xxxx-xxx-]
2020-02-07 18:23:09,646: msrestazure.azure_active_directory:INFO MSI: token is found in cache.
2020-02-07 18:23:10,808: msrestazure.azure_active_directory:INFO MSI: token is found in cache.
2020-02-07 18:23:11,365: custodian.policy:INFO policy:my-first-policy resource:azure.vm region: count:1 time:1.76

I also changed the save output to -s . to save in the current directory and it worked with no error.

I've also ran the same cmd (saving output to blob) locally from my laptop (not dockerized) and i'm able to save to the same location. So I'm not sure what I could be doing wrong when attempting to save to blob storage. It cannot be access/permission related because the error would say something of the matter. I do not believe it's related to networking either. I say that because I ssh'ed into my AKS nodes and was able to save a basic .txt file to storage account under the assumed MSI.

Again, if this is not an issue please close this out and let me know the proper way to go about seeking assistance when troubleshooting something like this. If you need more information on my config or file let me know and i'd be more than happy to provide.

Config info... I dont think AKS, OS, etc are related to this error so I'm leaving that out:

---

DOCKER FILE

FROM python:3.7-slim-stretch

LABEL name="custodian" \
      description="Cloud Management Rules Engine" \
      repository="http://github.com/cloud-custodian/cloud-custodian" \
      homepage="http://github.com/cloud-custodian/cloud-custodian" \
      maintainer="Custodian Community <https://cloudcustodian.io>"

# Transfer Custodian source into container by directory
# to minimize size
ADD setup.py README.md requirements.txt /src/
ADD c7n /src/c7n/
ADD tools /src/tools/

ENV AZURE_USE_MSI=1
ENV AZURE_SUBSCRIPTION_ID="xxxxx-xxxxx-xxxx-xxxx"

WORKDIR /src

RUN adduser --disabled-login custodian
RUN apt-get --yes update && apt-get --yes upgrade \
 && cp /src/tools/policy.yml /home/custodian \
 && apt-get --yes install build-essential \
 && pip3 install -r requirements.txt  . \
 && pip3 install -r tools/c7n_gcp/requirements.txt tools/c7n_gcp \
 && pip3 install -r tools/c7n_azure/requirements.txt tools/c7n_azure \
 && apt-get --yes remove build-essential \
 && apt-get purge --yes --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
 && rm -Rf /var/cache/apt/ \
 && rm -Rf /var/lib/apt/lists/* \
 && rm -Rf /src/ \
 && rm -Rf /root/.cache/ \
 && mkdir /output \
 && chown custodian: /output 

USER custodian
WORKDIR /home/custodian
ENV LC_ALL="C.UTF-8" LANG="C.UTF-8"
VOLUME ["/home/custodian"]
ENTRYPOINT ["/usr/local/bin/custodian"]
CMD ["run", "-c", "policy.yml", "-s", "azure//storageaccount.blob.core.windows.net/logs"]

---

policy.yml

policies:
    - name: my-first-policy
      description: |
        Search for a virtual machines
      resource: azure.vm
      filters:
        - type: value
          key: name
          value: basic-vm

[kapilt]

@cloud-custodian/azure

so first I would recommend using our release tag if your pulling the custodian docker image or release. pip install c7n_azure or docker pull cloud custodian/c7n:0.8.46.1

the docker file looks a little odd in that its got "azure//storageaccount.blob.core.windows.net/logs" when it should be azure:// although the error message you have shows that, its a uri/url protocol so needs the azure:// prefix to target a storage blob. my recommend on the release, was also that trunk has seen some major changes the last few days wrt to lazy loading and while we aim to keep it always green, in general for any deploys the recommend would be to use a released version.

[kapilt]

on trunk, I think I see the issue with the lazy loading and the azure storage plugins, in that I'm not sure we're explicitly loading the provider storage plugins from azure when we see an azure resource.

incidentally there's also a separate release branch for the 0.8 releases if you are interested in pulling from git, that branch is a release branch, bug fix only on changes.

[engineertree5]

Update
I did what you recommended and pulled the 0.8 release.

git clone -b release/0.8 https://github.com/cloud-custodian/cloud-custodian.git

After that I was able to save my output to azure blob without error. Thank you for the guidance and help with this.