Fluent-bit: Azure storage output append blob fails after 50,000 blocks

[johnjbateman]

Bug Report

Describe the bug
Followed these instructions on how to send logs to an append blob in Azure Storage.

Works great! However, when the blobs reach about 20-30MB, they are no longer being appended. Restarting the pods will create a new blob and it works until this happens again.

To Reproduce

• Example log message:

[2020/10/16 13:11:34] [error] [output:azure_blob:azure_blob.0] cannot append content to blob
<?xml version="1.0" encoding="utf-8"?><Error><Code>BlockCountExceedsLimit</Code><Message>The committed block count cannot exceed the maximum limit of 50,000 blocks.
RequestId:15cdaf1a-f01e-004f-28bd-a3bd01000000
Time:2020-10-16T13:11:34.7990633Z</Message></Error>
[2020/10/16 13:11:34] [ warn] [engine] chunk '1-1602853882.622679709.flb' cannot be retried: task_id=0, input=tail.0 > output=azure_blob.0

Expected behavior
After an append blob reaches the max number of blocks, it should create a new blob and not fail.

 /// <summary>
    /// Gets or sets a value for a condition that specifies the maximum size allowed for an append blob when a new block is committed. The append
    /// will succeed only if the size of the blob after the append operation is less than or equal to the specified size.
    /// </summary>
    /// <value>The maximum size in bytes, or <c>null</c> if no value is set.</value>
    /// <remarks>This condition only applies to append blobs.</remarks>
    public long? IfMaxSizeLessThanOrEqual

Reference: https://stackoverflow.com/questions/49627812/how-to-handle-append-blob-exception

Your Environment

• Version used: fluent/fluent-bit:1.6
• Configuration: https://gist.github.com/johnjbateman/fe802b5a6a443d3e325b4d2b1b992c6f
• Environment name and version: Azure Kubernetes Service (v1.17.4)
• Server type and version: Ubuntu 16.04.6 LTS
• Operating System and version: Kernel 4.15.0-1082-azure
• Filters and plugins: docker://3.0.10+azure

Additional context

Was going to use the ideal and official Azure storage plugin to send logs to an append blob. At this point, it will not be possible to use fluent-bit if append operations fail.

[BertelBB]

Any news on this? I am dealing with the same issue but I am not sure I understand what is happening.

As I understand it, the limitation is that a blob in Azure can be comprised of a maximum of 50,000 blocks, irregardless of the size of each block. So in order to handle this error, the azure_blob plugin should create a new blob when the error is encountered and include something like an index in the blob name?

[edsiper]

hi, I will take a look at this issue

[Kaiming-Qu]

Each block can be a different size, up to a maximum of 4 MiB. A maximum of 50,000 appends are permitted for each append blob. The maximum size of an append blob is therefore slightly more than 195 GiB (4 MiB X 50,000 blocks). If you attempt to upload a block that is larger than 4 MiB, the service returns HTTP status code 413 (Request Entity Too Large). The service also returns additional information about the error in the response, including the maximum block size permitted in bytes. If you attempt to upload more than 50,000 blocks, the service returns the BlockCountExceedsLimit error (HTTP status code 409 – Conflict).

https://docs.microsoft.com/en-us/rest/api/storageservices/append-block#remarks

[BertelBB]

@edsiper any news on this?

[edsiper]

Assigned to milestone Fluent Bit v1.7, the fix might come earlier...

[BertelBB]

I've implemented a Watcher in Kibana that monitors these logs from fluent-bit and notifies me, just as a temporary workaround until this is resolved. Just dropping it here in case it is of use for anyone else.

If anyone has a better solution, then that would be greatly appreciated. Also, identifying which blob is full is just guesswork by looking at the size of the blobs in the container. So if anyone knows of a way to correlate a warning in fluent-bit with the offending blob, that would be good to know about as well.

{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          <your-indices>
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 1,
          "query": {
            "bool": {
              "must": [
                {
                  "match": {
                    "kubernetes.labels.app_kubernetes_io/name": {
                      "query": "fluent-bit",
                      "operator": "and"
                    }
                  }
                },
                {
                  "match": {
                    "log": {
                      "query": "*<Code>BlockCountExceedsLimit</Code>*",
                      "operator": "and"
                    }
                  }
                }
              ],
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "from": "{{ctx.trigger.scheduled_time}}||-5m",
                      "to": "{{ctx.trigger.triggered_time}}"
                    }
                  }
                }
              ]
            }
          },
          "sort": [
            {
              "@timestamp": {
                "order": "desc"
              }
            }
          ]
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions": {
    "email-administrator": {
      "throttle_period_in_millis": 7200000,
      "email": {
        "profile": "standard",
        "to": [
          "<your-email>"
        ],
        "subject": "flb BlockCountExceedsLimit in Elastic",
        "body": {
          "text": "{{ctx.payload.hits.hits.0._index}}. Fluent-bit is trying to append content to a blob in Azure that cannot accept more data."
        }
      }
    }
  }
}