Loki: Add support for Azure Blob Storage

[huydinhle]

Is your feature request related to a problem? Please describe.
Add support for Azure Blob Storage.

Describe the solution you'd like
Since we have support for GCS and S3, can we add Azure blob storage as well.

[huydinhle]

would love to contribute this feature to Loki as well because we want to adopt Loki in our production system. But all of our stuff lives in Azure land :cry:

[huydinhle]

Based on the code, it looks like this is coming from Cortex library. I guess I should open an issue over there instead?

[yubozhao]

@huydinhle Yes, I think open an issue and/or contributing to the cortex library is the best way to approach this.

[tomwilkie]

Closing in favour of https://github.com/cortexproject/cortex/issues/1234

Let me know if you need any pointers.

[EraYaN]

Since https://github.com/cortexproject/cortex/pull/1913 was merged the documentation might need an update (and example).

[abelal83]

Since cortexproject/cortex#1913 was merged the documentation might need an update (and example).

documentation not updated yet, would be really helpful.

In fact azure storage is yet to be supported

level=error ts=2020-01-31T09:22:16.142661174Z caller=main.go:66 msg="error initialising loki" err="error initialising module: store: error creating object client: Unrecognized storage client azure, choose one of: aws, cassandra, inmemory, gcp, bigtable, bigtable-hashed"

This is using the latest docker image tag: v1.3.0

[abelal83]

Any updates on this please? loki --help seems to suggest there is some support

 Azure:
   -azure.account-key string:
      The Microsoft Azure account key to use.
   -azure.account-name string:
      The Microsoft Azure account name to be used
   -azure.container-name cortex:
      Name of the blob container used to store chunks. Defaults to cortex. This container must be created before running cortex. (default cortex)
   -azure.download-buffer-count int:
      Number of buffers used to used to upload a chunk. (defaults to 1) (default 1)
   -azure.download-buffer-size int:
      Preallocated buffer size for downloads (default is 512KB) (default 512000)
   -azure.max-retries int:
      Number of retries for a request which times out. (default 5)
   -azure.max-retry-delay duration:
      Maximum time to wait before retrying a request. (default 500ms)
   -azure.min-retry-delay duration:
      Minimum time to wait before retrying a request. (default 10ms)
   -azure.request-timeout duration:
      Timeout for requests made against azure blob storage. Defaults to 30 seconds. (default 30s)
   -azure.upload-buffer-size int:
      Preallocated buffer size for up;oads (default is 256KB) (default 256000)

but clearly something is wrong when adding to config.yaml

[gibbonsjohnm]

Where does this currently stand?

[sshah90]

@abelal83 yes they have support for sending chunks to Azure blob but not for indexing which is mandatory so you need to use key-value store for that (ex. boltdb supported).

you can use config like this for azure (Source).

config:
  schema_config:
    configs:
      - from: 2020-02-01
        store: boltdb #indexing yet not support in Azure so use boltdb for azure
        object_store: azure
        schema: v9
  storage_config:
    azure:
      container_name: something
      account_name: something
      account_key: something

Edit : I can confirm this config is working to send chunks to Azure blob. I used boltdb for Indexing as of now.

[abelal83]

Thanks @sshah90. I'm looking to run multi instances of loki and as indexing not supported on blob storage will run cassandra db cluster which I think should be fine.

[shapeofarchitect]

@sshah90 Just checked your config there , you also added comment that #indexing yet not support in Azure so use boltdb for azure ## I am assuming that this was before bolt-db-shipper was not merged. Now with Loki 1.5.0 with @sandeepsukhani change for botldb-shipper we can use single blob storage with this config ?

However I am still not able to use it and getting below error:

level=warn ts=2020-06-23T18:40:13.836180573Z caller=experimental.go:19 msg="experimental feature in use" feature="Azure Blob Storage"
level=error ts=2020-06-23T18:40:13.836367171Z caller=log.go:140 msg="error initialising loki" err="mkdir /loki/index: read-only file system\nerror creating index client\ngithub.com/cortexproject/cortex/pkg/chunk/storage.NewStore\n\t/src/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/storage/factory.go:153\ngithub.com/grafana/loki/pkg/storage.NewStore\n\t/src/loki/pkg/storage/store.go:53\ngithub.com/grafana/loki/pkg/loki.(*Loki).initStore\n\t/src/loki/pkg/loki/modules.go:280\ngithub.com/grafana/loki/pkg/loki.(*Loki).initModuleServices\n\t/src/loki/pkg/loki/loki.go:192\ngithub.com/grafana/loki/pkg/loki.New\n\t/src/loki/pkg/loki/loki.go:134\nmain.main\n\t/src/loki/cmd/loki/main.go:76\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nerror initialising module: store\ngithub.com/grafana/loki/pkg/loki.(*Loki).initModuleServices\n\t/src/loki/pkg/loki/loki.go:194\ngithub.com/grafana/loki/pkg/loki.New\n\t/src/loki/pkg/loki/loki.go:134\nmain.main\n\t/src/loki/cmd/loki/main.go:76\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"