Fluent-bit: input forward: plugin doesn't close TCP port when it caught SIGTERM
Bug Report
Describe the bug
The input forward plugin doesn't close TCP port when it caught SIGTERM. This causes losing some logs on shutdown.
To Reproduce
- Run a fluent-bit container with Docker:
$ docker run -it --rm --name flb -p 24224:24224 fluent/fluent-bit:1.5.6 /fluent-bit/bin/fluent-bit -v -i forward -o stdout -f 1
- Run a
nccommand (port scan) container with Docker fluent log driver and forward its log to the fluent-bit container:
$ docker run -it --rm --log-driver=fluentd --log-opt fluentd-address=127.0.0.1:24224 --link flb busybox sh -c 'while true; do date; nc -vz flb 24224; sleep 1; done'
- Stop the fluent-bit container:
$ date -u; time docker stop -t 10 flb; date -u
The outputs of command are as follows:
$ docker run -it --rm --name flb -p 24224:24224 fluent/fluent-bit:1.5.6 /fluent-bit/bin/fluent-bit -v -i forward -o stdout -f 1
Fluent Bit v1.5.6
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
[2020/09/29 00:09:55] [ info] Configuration:
[2020/09/29 00:09:55] [ info] flush time | 1.000000 seconds
[2020/09/29 00:09:55] [ info] grace | 5 seconds
[2020/09/29 00:09:55] [ info] daemon | 0
[2020/09/29 00:09:55] [ info] ___________
[2020/09/29 00:09:55] [ info] inputs:
[2020/09/29 00:09:55] [ info] forward
[2020/09/29 00:09:55] [ info] ___________
[2020/09/29 00:09:55] [ info] filters:
[2020/09/29 00:09:55] [ info] ___________
[2020/09/29 00:09:55] [ info] outputs:
[2020/09/29 00:09:55] [ info] stdout.0
[2020/09/29 00:09:55] [ info] ___________
[2020/09/29 00:09:55] [ info] collectors:
[2020/09/29 00:09:55] [ info] [engine] started (pid=1)
[2020/09/29 00:09:55] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2020/09/29 00:09:55] [debug] [storage] [cio stream] new stream registered: forward.0
[2020/09/29 00:09:55] [ info] [storage] version=1.0.5, initializing...
[2020/09/29 00:09:55] [ info] [storage] in-memory
[2020/09/29 00:09:55] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/09/29 00:09:55] [debug] [in_fw] Listen='0.0.0.0' TCP_Port=24224
[2020/09/29 00:09:55] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224
[2020/09/29 00:09:55] [debug] [router] default match rule forward.0:stdout.0
[2020/09/29 00:09:55] [ info] [sp] stream processor started
[0] e9ed8c030c0a: [1601338201.000000000, {"container_id"=>"e9ed8c030c0af0df30f15bad9a5e62c49e78dfc581738c6ccf480457e0b"}]73", "container_name"=>"/laughing_feistel", "source"=>"stdout", "log"=>"Tue Sep 29 00:10:01 UTC 2020
[1] e9ed8c030c0a: [1601338201.000000000, {"container_id"=>"e9ed8c030c0af0df30f15bad9a5e62c49e78dfc581738c6ccf480457e0b"}]73", "container_name"=>"/laughing_feistel", "source"=>"stdout", "log"=>"flb (172.17.0.2:24224) open
[2020/09/29 00:10:02] [debug] [task] created task=0x7f4257a2f3a0 id=0 OK
[2020/09/29 00:10:02] [debug] [task] destroy task=0x7f4257a2f3a0 (task_id=0)
[0] e9ed8c030c0a: [1601338202.000000000, {"container_name"=>"/laughing_feistel", "source"=>"stdout", "log"=>"Tue Sep 2", "container_id"=>"e9ed8c030c0af0df30f15bad9a5e62c49e78dfc581738c6ccf480457e0b3e973"}]
[1] e9ed8c030c0a: [1601338202.000000000, {"container_id"=>"e9ed8c030c0af0df30f15bad9a5e62c49e78dfc581738c6ccf480457e0b"}]73", "container_name"=>"/laughing_feistel", "source"=>"stdout", "log"=>"flb (172.17.0.2:24224) open
[2020/09/29 00:10:03] [debug] [task] created task=0x7f4257a2f3a0 id=0 OK
[2020/09/29 00:10:03] [debug] [task] destroy task=0x7f4257a2f3a0 (task_id=0)
", "container_id"=>"e9ed8c030c0af0df30f15bad9a5e62c49e78dfc581738c6ccf480457e0b3e973", "container_name"=>"/laughing_feistel", "source"=>"stdout"}]
", "container_id"=>"e9ed8c030c0af0df30f15bad9a5e62c49e78dfc581738c6ccf480457e0b3e973", "container_name"=>"/laughing_feistel", "source"=>"stdout"}]
[2020/09/29 00:10:04] [debug] [task] created task=0x7f4257a2f3a0 id=0 OK
[2020/09/29 00:10:04] [debug] [task] destroy task=0x7f4257a2f3a0 (task_id=0)
[engine] caught signal (SIGTERM)
", "container_id"=>"e9ed8c030c0af0df30f15bad9a5e62c49e78dfc581738c6ccf480457e0b3e973", "container_name"=>"/laughing_feistel", "source"=>"stdout"}]
[1] e9ed8c030c0a: [1601338204.000000000, {"container_id"=>"e9ed8c030c0af0df30f15bad9a5e62c49e78dfc581738c6ccf480457e0b"}]73", "container_name"=>"/laughing_feistel", "source"=>"stdout", "log"=>"flb (172.17.0.2:24224) open
[2020/09/29 00:10:04] [debug] [task] created task=0x7f4257a2f3a0 id=0 OK
[2020/09/29 00:10:04] [ warn] [engine] service will stop in 5 seconds
[2020/09/29 00:10:04] [debug] [task] destroy task=0x7f4257a2f3a0 (task_id=0)
[0] e9ed8c030c0a: [1601338205.000000000, {"container_id"=>"e9ed8c030c0af0df30f15bad9a5e62c49e78dfc581738c6ccf480457e0b"}]73", "container_name"=>"/laughing_feistel", "source"=>"stdout", "log"=>"Tue Sep 29 00:10:05 UTC 2020
", "container_id"=>"e9ed8c030c0af0df30f15bad9a5e62c49e78dfc581738c6ccf480457e0b3e973", "container_name"=>"/laughing_feistel"}]
[2020/09/29 00:10:06] [debug] [task] created task=0x7f4257a2f3a0 id=0 OK
[2020/09/29 00:10:06] [debug] [task] destroy task=0x7f4257a2f3a0 (task_id=0)
[0] e9ed8c030c0a: [1601338206.000000000, {"container_id"=>"e9ed8c030c0af0df30f15bad9a5e62c49e78dfc581738c6ccf480457e0b"}]73", "container_name"=>"/laughing_feistel", "source"=>"stdout", "log"=>"Tue Sep 29 00:10:06 UTC 2020
[1] e9ed8c030c0a: [1601338206.000000000, {"container_name"=>"/laughing_feistel", "source"=>"stdout", "log"=>"flb (172.", "container_id"=>"e9ed8c030c0af0df30f15bad9a5e62c49e78dfc581738c6ccf480457e0b3e973"}]
[2020/09/29 00:10:07] [debug] [task] created task=0x7f4257a2f3a0 id=0 OK
[2020/09/29 00:10:07] [debug] [task] destroy task=0x7f4257a2f3a0 (task_id=0)
[0] e9ed8c030c0a: [1601338207.000000000, {"container_id"=>"e9ed8c030c0af0df30f15bad9a5e62c49e78dfc581738c6ccf480457e0b"}]73", "container_name"=>"/laughing_feistel", "source"=>"stdout", "log"=>"Tue Sep 29 00:10:07 UTC 2020
[1] e9ed8c030c0a: [1601338207.000000000, {"container_id"=>"e9ed8c030c0af0df30f15bad9a5e62c49e78dfc581738c6ccf480457e0b"}]73", "container_name"=>"/laughing_feistel", "source"=>"stdout", "log"=>"flb (172.17.0.2:24224) open
[2020/09/29 00:10:08] [debug] [task] created task=0x7f4257a2f3a0 id=0 OK
[2020/09/29 00:10:08] [debug] [task] destroy task=0x7f4257a2f3a0 (task_id=0)
[2020/09/29 00:10:09] [ info] [engine] service stopped
$ docker run -it --rm --log-driver=fluentd --log-opt fluentd-address=127.0.0.1:24224 --link flb busybox sh -c 'while true; do date; nc -vz flb 24224; sleep 1; done'
Tue Sep 29 00:10:01 UTC 2020
flb (172.17.0.2:24224) open
Tue Sep 29 00:10:02 UTC 2020
flb (172.17.0.2:24224) open
Tue Sep 29 00:10:03 UTC 2020
flb (172.17.0.2:24224) open
Tue Sep 29 00:10:04 UTC 2020
flb (172.17.0.2:24224) open
Tue Sep 29 00:10:05 UTC 2020
flb (172.17.0.2:24224) open
Tue Sep 29 00:10:06 UTC 2020
flb (172.17.0.2:24224) open
Tue Sep 29 00:10:07 UTC 2020
flb (172.17.0.2:24224) open
Tue Sep 29 00:10:08 UTC 2020
flb (172.17.0.2:24224) open
Tue Sep 29 00:10:09 UTC 2020
$ date -u; time docker stop -t 10 flb; date -u
Tue Sep 29 00:10:04 UTC 2020
flb
docker stop -t 10 flb 0.04s user 0.02s system 1% cpu 4.409 total
Tue Sep 29 00:10:09 UTC 2020
According to the above output,
the Docker tried to stop the fluent-bit container at 00:10:04.
The engine caught signal (SIGTERM) at 00:10:04, but some logs are still ingested and the timestamp of the last log entry is 1601338207(=2020/09/29 00:10:07) and the result of nc command shows the TCP port was still open.
The engine completed shutdown at 00:10:09 after grace period (5 sec)
I tested multiple times and I found its behavior seems to be inconsistent.
The input was sometimes paused (but TCP port was still open).
[0] cfedccd0310b: [1601337750.000000000, {"container_id"=>"cfedccd0310b0b4b747d4bd6b8362d8d009ff26b24cdd59d3b58faa3ea3"}]42", "container_name"=>"/vigilant_aryabhata", "source"=>"stdout", "log"=>"Tue Sep 29 00:02:30 UTC 2020
", "container_id"=>"cfedccd0310b0b4b747d4bd6b8362d8d009ff26b24cdd59d3b58faa3ea300b42", "container_name"=>"/vigilant_aryabhata", "source"=>"stdout"}]
[2020/09/29 00:02:30] [debug] [task] created task=0x7f8ee0e2f3a0 id=0 OK
[2020/09/29 00:02:30] [debug] [task] destroy task=0x7f8ee0e2f3a0 (task_id=0)
[engine] caught signal (SIGTERM)
[2020/09/29 00:02:30] [ warn] [engine] service will stop in 5 seconds
[2020/09/29 00:02:31] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/09/29 00:02:31] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/09/29 00:02:32] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/09/29 00:02:32] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/09/29 00:02:33] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/09/29 00:02:33] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/09/29 00:02:34] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/09/29 00:02:34] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/09/29 00:02:35] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/09/29 00:02:35] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/09/29 00:02:35] [ info] [engine] service stopped
Tue Sep 29 00:02:28 UTC 2020
flb (172.17.0.2:24224) open
Tue Sep 29 00:02:29 UTC 2020
flb (172.17.0.2:24224) open
Tue Sep 29 00:02:30 UTC 2020
flb (172.17.0.2:24224) open
Tue Sep 29 00:02:31 UTC 2020
flb (172.17.0.2:24224) open
Tue Sep 29 00:02:32 UTC 2020
flb (172.17.0.2:24224) open
Tue Sep 29 00:02:33 UTC 2020
flb (172.17.0.2:24224) open
Tue Sep 29 00:02:34 UTC 2020
flb (172.17.0.2:24224) open
Tue Sep 29 00:02:35 UTC 2020
flb (172.17.0.2:24224) open
Tue Sep 29 00:02:36 UTC 2020
nc: flb (172.17.0.2:24224): Connection timed out
Tue Sep 29 00:04:46 UTC 2020
$ date -u; time docker stop -t 10 flb; date -u
Tue Sep 29 00:02:30 UTC 2020
flb
docker stop -t 10 flb 0.04s user 0.02s system 1% cpu 5.190 total
Tue Sep 29 00:02:35 UTC 2020
Expected behavior
The input forward plugin should close TCP port whent it caught SIGTERM not to lose any logs on shutdown.
Your Environment
- Version used: v1.5.6
- Configuration: (passed with arguments described above the reproduction case)
- Environment name and version (e.g. Kubernetes? What version?): AWS ECS Fargate (platform version: 1.4.0)
- Server type and version: AWS
- Operating System and version: Docker (fluent/fluent-bit:1.5.6)
- Filters and plugins: input forward
Additional context
I'm running fluent-bit containers on AWS ECS Fargate behind AWS NLB and forwarding other container's log with the Docker fluentd log driver.
When I need to update my configuration for fluent-bit, I'm using the Blue-Green deployment strategy not to lose any logs during the deployment.
However, AWS NLB doesn't reset the existing TCP connection even if NLB's deregistration delay passed. The deregistration delay block a new connection, but it doesn't reset the existing one. If my understanding is correct, the grace period doesn't ensure to flush all buffers. This means that if fluent-bit ingest new log entries during shutdown they will be lost.
I don't think it only affects fluent-bit behind AWS NLB. I can reproduce a simple docker container setup described above.
Blocking input on shutdown consistently may be a minimum solution, but it would be great if the fluent-bit close TCP port on shutdown because it allows sender to detect quickly receiver no longer available.
Related issue
https://github.com/fluent/fluent-bit/issues/1496
minamijoyo
All 9 comments
Fixed in f303cdb5c5c9d5a7d98d824878551d916441a95a.
Binary now uses library mode so it can do graceful shutdown (in the grace period)
edsiper
on 29 Sep 2020
@edsiper Thank you for the fix!
But I cannot confirm the fix was resolved the original issue. I assumed blocking the input during shutdown consistently would be the minimum solution, however I might misunderstand the behavior during blocking.
I tested with the latest master (13735a509906b3fd089cfb5df9e04f6322089e39) and I could see the input forward was paused consistently during shutdown. However when I started a new container after shutdown the blocked records during shutdown seem to be simply discarded.
Build a docker image from the latest master (13735a509906b3fd089cfb5df9e04f6322089e39)
[fluent-bit@master|鉁擼$ git rev-parse HEAD
13735a509906b3fd089cfb5df9e04f6322089e39
[fluent-bit@master|鉁擼$ docker version
Client: Docker Engine - Community
Azure integration 0.1.7
Version: 19.03.12
API version: 1.40
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:41:33 2020
OS/Arch: darwin/amd64
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 19.03.12
API version: 1.40 (minimum version 1.12)
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:49:27 2020
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: v1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
[fluent-bit@master|鉁擼$ docker build -t fluent-bit:debug ./
Run a fluent-bit container:
[fluent-bit@master|鉁擼$ docker run -it --rm --name flb -p 24224:24224 fluent-bit:debug /fluent-bit/bin/fluent-bit -v -i forward -o stdout -f 1
Fluent Bit v1.6.0
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
[2020/10/01 07:05:28] [ info] Configuration:
[2020/10/01 07:05:28] [ info] flush time | 1.000000 seconds
[2020/10/01 07:05:28] [ info] grace | 5 seconds
[2020/10/01 07:05:28] [ info] daemon | 0
[2020/10/01 07:05:28] [ info] ___________
[2020/10/01 07:05:28] [ info] inputs:
[2020/10/01 07:05:28] [ info] forward
[2020/10/01 07:05:28] [ info] ___________
[2020/10/01 07:05:28] [ info] filters:
[2020/10/01 07:05:28] [ info] ___________
[2020/10/01 07:05:28] [ info] outputs:
[2020/10/01 07:05:28] [ info] stdout.0
[2020/10/01 07:05:28] [ info] ___________
[2020/10/01 07:05:28] [ info] collectors:
[2020/10/01 07:05:28] [ info] [engine] started (pid=1)
[2020/10/01 07:05:28] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2020/10/01 07:05:28] [debug] [storage] [cio stream] new stream registered: forward.0
[2020/10/01 07:05:28] [ info] [storage] version=1.0.6, initializing...
[2020/10/01 07:05:28] [ info] [storage] in-memory
[2020/10/01 07:05:28] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/10/01 07:05:28] [debug] [in_fw] Listen='0.0.0.0' TCP_Port=24224
[2020/10/01 07:05:28] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224
[2020/10/01 07:05:28] [debug] [router] default match rule forward.0:stdout.0
[2020/10/01 07:05:28] [ info] [sp] stream processor started
", "container_id"=>"061221f1625eb90bcee17ffb4fbb3a6b16d84da6005404799533e40d9455f183", "container_name"=>"/bold_dhawan", "source"=>"stdout"}]
[2020/10/01 07:05:38] [debug] [task] created task=0x7fc98e44a0a0 id=0 OK
[1] 061221f1625e: [1601535938.000000000, {"container_id"=>"061221f1625eb90bcee17ffb4fbb3a6b16d84da6005404799533e40d945"}]83", "container_name"=>"/bold_dhawan", "source"=>"stdout", "log"=>"flb (172.17.0.2:24224) open
[2020/10/01 07:05:38] [debug] [task] destroy task=0x7fc98e44a0a0 (task_id=0)
[0] 061221f1625e: [1601535939.000000000, {"container_name"=>"/bold_dhawan", "source"=>"stdout", "log"=>"Thu Oct 1 07:", "container_id"=>"061221f1625eb90bcee17ffb4fbb3a6b16d84da6005404799533e40d9455f183"}]
[1] 061221f1625e: [1601535939.000000000, {"container_name"=>"/bold_dhawan", "source"=>"stdout", "log"=>"flb (172.17.0.", "container_id"=>"061221f1625eb90bcee17ffb4fbb3a6b16d84da6005404799533e40d9455f183"}]
[2020/10/01 07:05:39] [debug] [task] created task=0x7fc98e44a0a0 id=0 OK
[2020/10/01 07:05:39] [debug] [task] destroy task=0x7fc98e44a0a0 (task_id=0)
", "container_id"=>"061221f1625eb90bcee17ffb4fbb3a6b16d84da6005404799533e40d9455f183", "container_name"=>"/bold_dhawan", "source"=>"stdout"}]
[1] 061221f1625e: [1601535940.000000000, {"container_id"=>"061221f1625eb90bcee17ffb4fbb3a6b16d84da6005404799533e40d945"}]83", "container_name"=>"/bold_dhawan", "source"=>"stdout", "log"=>"flb (172.17.0.2:24224) open
[2020/10/01 07:05:40] [debug] [task] created task=0x7fc98e44a0a0 id=0 OK
[2020/10/01 07:05:40] [debug] [task] destroy task=0x7fc98e44a0a0 (task_id=0)
", "container_id"=>"061221f1625eb90bcee17ffb4fbb3a6b16d84da6005404799533e40d9455f183", "container_name"=>"/bold_dhawan"}]
[1] 061221f1625e: [1601535941.000000000, {"container_id"=>"061221f1625eb90bcee17ffb4fbb3a6b16d84da6005404799533e40d945"}]83", "container_name"=>"/bold_dhawan", "source"=>"stdout", "log"=>"flb (172.17.0.2:24224) open
[2020/10/01 07:05:41] [debug] [task] created task=0x7fc98e44a0a0 id=0 OK
[2020/10/01 07:05:41] [debug] [task] destroy task=0x7fc98e44a0a0 (task_id=0)
[0] 061221f1625e: [1601535942.000000000, {"container_id"=>"061221f1625eb90bcee17ffb4fbb3a6b16d84da6005404799533e40d945"}]83", "container_name"=>"/bold_dhawan", "source"=>"stdout", "log"=>"Thu Oct 1 07:05:42 UTC 2020
[1] 061221f1625e: [1601535942.000000000, {"container_name"=>"/bold_dhawan", "source"=>"stdout", "log"=>"flb (172.17.0.", "container_id"=>"061221f1625eb90bcee17ffb4fbb3a6b16d84da6005404799533e40d9455f183"}]
[2020/10/01 07:05:42] [debug] [task] created task=0x7fc98e44a0a0 id=0 OK
[2020/10/01 07:05:42] [debug] [task] destroy task=0x7fc98e44a0a0 (task_id=0)
[0] 061221f1625e: [1601535943.000000000, {"container_name"=>"/bold_dhawan", "source"=>"stdout", "log"=>"Thu Oct 1 07:", "container_id"=>"061221f1625eb90bcee17ffb4fbb3a6b16d84da6005404799533e40d9455f183"}]
[1] 061221f1625e: [1601535943.000000000, {"container_id"=>"061221f1625eb90bcee17ffb4fbb3a6b16d84da6005404799533e40d945"}]83", "container_name"=>"/bold_dhawan", "source"=>"stdout", "log"=>"flb (172.17.0.2:24224) open
[2020/10/01 07:05:43] [debug] [task] created task=0x7fc98e44a0a0 id=0 OK
[2020/10/01 07:05:43] [debug] [task] destroy task=0x7fc98e44a0a0 (task_id=0)
[2020/10/01 07:05:43] [engine] caught signal (SIGTERM)
[2020/10/01 07:05:43] [ warn] [engine] service will stop in 5 seconds
[2020/10/01 07:05:44] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/10/01 07:05:44] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/10/01 07:05:45] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/10/01 07:05:45] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/10/01 07:05:46] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/10/01 07:05:46] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/10/01 07:05:47] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/10/01 07:05:47] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/10/01 07:05:48] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/10/01 07:05:48] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/10/01 07:05:48] [ info] [engine] service stopped
Run a port scan container:
[fluent-bit@:0580dc47|鉁擼$ docker run -it --rm --log-driver=fluentd --log-opt fluentd-address=127.0.0.1:24224 --link flb busybox sh -c 'while true; do date; nc -vz flb 24224; sleep 1; done'
Thu Oct 1 07:05:38 UTC 2020
flb (172.17.0.2:24224) open
Thu Oct 1 07:05:39 UTC 2020
flb (172.17.0.2:24224) open
Thu Oct 1 07:05:40 UTC 2020
flb (172.17.0.2:24224) open
Thu Oct 1 07:05:41 UTC 2020
flb (172.17.0.2:24224) open
Thu Oct 1 07:05:42 UTC 2020
flb (172.17.0.2:24224) open
Thu Oct 1 07:05:43 UTC 2020
flb (172.17.0.2:24224) open
Thu Oct 1 07:05:44 UTC 2020
flb (172.17.0.2:24224) open
Thu Oct 1 07:05:45 UTC 2020
flb (172.17.0.2:24224) open
Thu Oct 1 07:05:46 UTC 2020
flb (172.17.0.2:24224) open
Thu Oct 1 07:05:47 UTC 2020
flb (172.17.0.2:24224) open
Thu Oct 1 07:05:48 UTC 2020
flb (172.17.0.2:24224) open
Thu Oct 1 07:05:49 UTC 2020
flb (172.17.0.2:24224) open
Thu Oct 1 07:05:50 UTC 2020
flb (172.17.0.2:24224) open
Thu Oct 1 07:05:51 UTC 2020
flb (172.17.0.2:24224) open
Thu Oct 1 07:05:52 UTC 2020
flb (172.17.0.2:24224) open
Thu Oct 1 07:05:53 UTC 2020
flb (172.17.0.2:24224) open
Thu Oct 1 07:05:54 UTC 2020
nc: flb (172.17.0.2:24224): No route to host
Thu Oct 1 07:07:02 UTC 2020
nc: flb (172.17.0.2:24224): No route to host
Thu Oct 1 07:07:06 UTC 2020
^Cpunt!
[fluent-bit@master|鉁擼$
Stop the fluent-bit container:
[fluent-bit@:0580dc47|鉁擼$ date -u; time docker stop -t 10 flb; date -u
Thu Oct 1 07:05:43 UTC 2020
flb
docker stop -t 10 flb 0.05s user 0.03s system 0% cpu 10.298 total
Thu Oct 1 07:05:54 UTC 2020
Start a new fluent-bit container:
[fluent-bit@master|鉁擼$ docker run -it --rm --name flb -p 24224:24224 fluent-bit:debug /fluent-bit/bin/fluent-bit -v -i forward -o stdout -f 1
Fluent Bit v1.6.0
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
[2020/10/01 07:08:22] [ info] Configuration:
[2020/10/01 07:08:22] [ info] flush time | 1.000000 seconds
[2020/10/01 07:08:22] [ info] grace | 5 seconds
[2020/10/01 07:08:22] [ info] daemon | 0
[2020/10/01 07:08:22] [ info] ___________
[2020/10/01 07:08:22] [ info] inputs:
[2020/10/01 07:08:22] [ info] forward
[2020/10/01 07:08:22] [ info] ___________
[2020/10/01 07:08:22] [ info] filters:
[2020/10/01 07:08:22] [ info] ___________
[2020/10/01 07:08:22] [ info] outputs:
[2020/10/01 07:08:22] [ info] stdout.0
[2020/10/01 07:08:22] [ info] ___________
[2020/10/01 07:08:22] [ info] collectors:
[2020/10/01 07:08:22] [ info] [engine] started (pid=1)
[2020/10/01 07:08:22] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2020/10/01 07:08:22] [debug] [storage] [cio stream] new stream registered: forward.0
[2020/10/01 07:08:22] [ info] [storage] version=1.0.6, initializing...
[2020/10/01 07:08:22] [ info] [storage] in-memory
[2020/10/01 07:08:22] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/10/01 07:08:22] [debug] [in_fw] Listen='0.0.0.0' TCP_Port=24224
[2020/10/01 07:08:22] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224
[2020/10/01 07:08:22] [debug] [router] default match rule forward.0:stdout.0
[2020/10/01 07:08:22] [ info] [sp] stream processor started
[2020/10/01 07:08:52] [debug] [task] created task=0x7f2c11e4a0a0 id=0 OK
", "container_id"=>"061221f1625eb90bcee17ffb4fbb3a6b16d84da6005404799533e40d9455f183", "container_name"=>"/bold_dhawan", "source"=>"stdout"}]
", "container_id"=>"061221f1625eb90bcee17ffb4fbb3a6b16d84da6005404799533e40d9455f183", "container_name"=>"/bold_dhawan", "source"=>"stdout"}]
", "container_id"=>"061221f1625eb90bcee17ffb4fbb3a6b16d84da6005404799533e40d9455f183", "container_name"=>"/bold_dhawan", "source"=>"stdout"}]
", "container_id"=>"061221f1625eb90bcee17ffb4fbb3a6b16d84da6005404799533e40d9455f183", "container_name"=>"/bold_dhawan", "source"=>"stdout"}]
", "container_id"=>"061221f1625eb90bcee17ffb4fbb3a6b16d84da6005404799533e40d9455f183"}]stdout", "log"=>"^Cpunt!
[2020/10/01 07:08:52] [debug] [task] destroy task=0x7f2c11e4a0a0 (task_id=0)
^C[2020/10/01 07:09:46] [engine] caught signal (SIGINT)
[2020/10/01 07:09:46] [ warn] [engine] service will stop in 5 seconds
[2020/10/01 07:09:51] [ info] [engine] service stopped
[fluent-bit@master|鉁擼$
I expect the blocked records during shutdown to be retried to send by docker engine after restart, but they were actually dropped.
What do you think about it?
minamijoyo
on 1 Oct 2020
troubleshooting...
edsiper
on 1 Oct 2020
❤3
I've pushed a workaround for the issue: 4e0e6870
The only downside is that if the plugin is under high-backpressure, the server socket won't be terminated before the grade period on shutdown/SIGTERM.
Note that after 1.6 release we will refactor the server side API to address this things.
edsiper
on 2 Oct 2020
for now can you provide some feedback on 4e0e6870 ?
edsiper
on 2 Oct 2020
@edsiper thank you for the fix! I'll test it.
minamijoyo
on 5 Oct 2020
@edsiper I tested the 4e0e687098836fd0cbe99944b13ac2ccb8058678, I could see the port was closed during shutdown, but some records were still dropped.
Build a docker image from 4e0e687098836fd0cbe99944b13ac2ccb8058678 :
$ git checkout 4e0e687098836fd0cbe99944b13ac2ccb8058678
$ docker build -t fluent-bit:4e0e6870 ./
Run a fluent-bit container:
[fluent-bit@:4e0e6870|鉁擼$ docker run -it --rm --name flb -p 24224:24224 fluent-bit:4e0e6870 /fluent-bit/bin/fluent-bit -v -i forward -o stdout -f 1
Fluent Bit v1.6.0
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
[2020/10/05 07:12:16] [ info] Configuration:
[2020/10/05 07:12:16] [ info] flush time | 1.000000 seconds
[2020/10/05 07:12:16] [ info] grace | 5 seconds
[2020/10/05 07:12:16] [ info] daemon | 0
[2020/10/05 07:12:16] [ info] ___________
[2020/10/05 07:12:16] [ info] inputs:
[2020/10/05 07:12:16] [ info] forward
[2020/10/05 07:12:16] [ info] ___________
[2020/10/05 07:12:16] [ info] filters:
[2020/10/05 07:12:16] [ info] ___________
[2020/10/05 07:12:16] [ info] outputs:
[2020/10/05 07:12:16] [ info] stdout.0
[2020/10/05 07:12:16] [ info] ___________
[2020/10/05 07:12:16] [ info] collectors:
[2020/10/05 07:12:16] [ info] [engine] started (pid=1)
[2020/10/05 07:12:16] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2020/10/05 07:12:16] [debug] [storage] [cio stream] new stream registered: forward.0
[2020/10/05 07:12:16] [ info] [storage] version=1.0.6, initializing...
[2020/10/05 07:12:16] [ info] [storage] in-memory
[2020/10/05 07:12:16] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/10/05 07:12:16] [debug] [in_fw] Listen='0.0.0.0' TCP_Port=24224
[2020/10/05 07:12:16] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224
[2020/10/05 07:12:16] [debug] [router] default match rule forward.0:stdout.0
[2020/10/05 07:12:16] [ info] [sp] stream processor started
[2020/10/05 07:12:25] [debug] [task] created task=0x7ffa2ae4d0a0 id=0 OK
", "container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6ad6844", "container_name"=>"/busy_ellis", "source"=>"stdout"}]
[1] da81522ed489: [1601881945.000000000, {"container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6a"}]44", "container_name"=>"/busy_ellis", "source"=>"stdout", "log"=>"flb (172.17.0.2:24224) open
[2020/10/05 07:12:25] [debug] [task] destroy task=0x7ffa2ae4d0a0 (task_id=0)
[0] da81522ed489: [1601881946.000000000, {"container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6a"}]44", "container_name"=>"/busy_ellis", "source"=>"stdout", "log"=>"Mon Oct 5 07:12:26 UTC 2020
", "container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6ad6844", "container_name"=>"/busy_ellis"}]
[2020/10/05 07:12:26] [debug] [task] created task=0x7ffa2ae4d0a0 id=0 OK
[2020/10/05 07:12:26] [debug] [task] destroy task=0x7ffa2ae4d0a0 (task_id=0)
", "container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6ad6844", "container_name"=>"/busy_ellis", "source"=>"stdout"}]
[1] da81522ed489: [1601881947.000000000, {"container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6a"}]44", "container_name"=>"/busy_ellis", "source"=>"stdout", "log"=>"flb (172.17.0.2:24224) open
[2020/10/05 07:12:27] [debug] [task] created task=0x7ffa2ae4d0a0 id=0 OK
[2020/10/05 07:12:27] [debug] [task] destroy task=0x7ffa2ae4d0a0 (task_id=0)
[0] da81522ed489: [1601881948.000000000, {"container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6a"}]44", "container_name"=>"/busy_ellis", "source"=>"stdout", "log"=>"Mon Oct 5 07:12:28 UTC 2020
[1] da81522ed489: [1601881948.000000000, {"container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6a"}]44", "container_name"=>"/busy_ellis", "source"=>"stdout", "log"=>"flb (172.17.0.2:24224) open
[2020/10/05 07:12:28] [debug] [task] created task=0x7ffa2ae4d0a0 id=0 OK
[2020/10/05 07:12:28] [debug] [task] destroy task=0x7ffa2ae4d0a0 (task_id=0)
", "container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6ad6844", "container_name"=>"/busy_ellis"}]
", "container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6ad6844", "container_name"=>"/busy_ellis"}]
[2020/10/05 07:12:29] [debug] [task] created task=0x7ffa2ae4d0a0 id=0 OK
[2020/10/05 07:12:29] [debug] [task] destroy task=0x7ffa2ae4d0a0 (task_id=0)
[0] da81522ed489: [1601881950.000000000, {"container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6a"}]44", "container_name"=>"/busy_ellis", "source"=>"stdout", "log"=>"Mon Oct 5 07:12:30 UTC 2020
[1] da81522ed489: [1601881950.000000000, {"container_name"=>"/busy_ellis", "source"=>"stdout", "log"=>"flb (172.17.0.2", "container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6ad6844"}]
[2020/10/05 07:12:30] [debug] [task] created task=0x7ffa2ae4d0a0 id=0 OK
[2020/10/05 07:12:30] [debug] [task] destroy task=0x7ffa2ae4d0a0 (task_id=0)
", "container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6ad6844", "container_name"=>"/busy_ellis", "source"=>"stdout"}]
[1] da81522ed489: [1601881951.000000000, {"container_name"=>"/busy_ellis", "source"=>"stdout", "log"=>"flb (172.17.0.2", "container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6ad6844"}]
[2020/10/05 07:12:31] [debug] [task] created task=0x7ffa2ae4d0a0 id=0 OK
[2020/10/05 07:12:31] [debug] [task] destroy task=0x7ffa2ae4d0a0 (task_id=0)
[2020/10/05 07:12:32] [engine] caught signal (SIGTERM)
[2020/10/05 07:12:32] [ info] [input] pausing forward.0
", "container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6ad6844", "container_name"=>"/busy_ellis"}]
", "container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6ad6844", "container_name"=>"/busy_ellis", "source"=>"stdout"}]
[2020/10/05 07:12:32] [debug] [task] created task=0x7ffa2ae4d0a0 id=0 OK
[2020/10/05 07:12:32] [ warn] [engine] service will stop in 5 seconds
[2020/10/05 07:12:32] [debug] [task] destroy task=0x7ffa2ae4d0a0 (task_id=0)
[2020/10/05 07:12:33] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/10/05 07:12:34] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/10/05 07:12:35] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/10/05 07:12:36] [debug] [input chunk] forward.0 is paused, cannot append records
[2020/10/05 07:12:36] [ info] [engine] service stopped
Run a port scan container:
$ docker run -it --rm --log-driver=fluentd --log-opt fluentd-address=127.0.0.1:24224 --log-opt fluentd-async-connect="true" --link flb busybox sh -c 'while true; do date; nc -vz flb 24224; sleep 1; done'
Mon Oct 5 07:12:25 UTC 2020
flb (172.17.0.2:24224) open
Mon Oct 5 07:12:26 UTC 2020
flb (172.17.0.2:24224) open
Mon Oct 5 07:12:27 UTC 2020
flb (172.17.0.2:24224) open
Mon Oct 5 07:12:28 UTC 2020
flb (172.17.0.2:24224) open
Mon Oct 5 07:12:29 UTC 2020
flb (172.17.0.2:24224) open
Mon Oct 5 07:12:30 UTC 2020
flb (172.17.0.2:24224) open
Mon Oct 5 07:12:31 UTC 2020
flb (172.17.0.2:24224) open
Mon Oct 5 07:12:32 UTC 2020
flb (172.17.0.2:24224) open
Mon Oct 5 07:12:33 UTC 2020
Mon Oct 5 07:12:34 UTC 2020
Mon Oct 5 07:12:35 UTC 2020
Mon Oct 5 07:12:36 UTC 2020
Mon Oct 5 07:12:37 UTC 2020
Mon Oct 5 07:12:38 UTC 2020
Mon Oct 5 07:12:39 UTC 2020
Mon Oct 5 07:12:40 UTC 2020
Mon Oct 5 07:12:41 UTC 2020
Mon Oct 5 07:12:42 UTC 2020
Mon Oct 5 07:12:43 UTC 2020
flb (172.17.0.2:24224) open
Mon Oct 5 07:12:51 UTC 2020
flb (172.17.0.2:24224) open
Mon Oct 5 07:12:52 UTC 2020
flb (172.17.0.2:24224) open
Stop the fluent-bit container:
[fluent-bit@:4e0e6870|鉁擼$ date -u; time docker stop -t 10 flb; date -u
Mon Oct 5 07:12:58 UTC 2020
flb
docker stop -t 10 flb 0.04s user 0.02s system 0% cpu 10.325 total
Mon Oct 5 07:13:08 UTC 2020
Start a new fluent-bit container:
[fluent-bit@:4e0e6870|鉁擼$ docker run -it --rm --name flb -p 24224:24224 fluent-bit:4e0e6870 /fluent-bit/bin/fluent-bit -v -i forward -o stdout -f 1
Fluent Bit v1.6.0
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
[2020/10/05 07:12:49] [ info] Configuration:
[2020/10/05 07:12:49] [ info] flush time | 1.000000 seconds
[2020/10/05 07:12:49] [ info] grace | 5 seconds
[2020/10/05 07:12:49] [ info] daemon | 0
[2020/10/05 07:12:49] [ info] ___________
[2020/10/05 07:12:49] [ info] inputs:
[2020/10/05 07:12:49] [ info] forward
[2020/10/05 07:12:49] [ info] ___________
[2020/10/05 07:12:49] [ info] filters:
[2020/10/05 07:12:49] [ info] ___________
[2020/10/05 07:12:49] [ info] outputs:
[2020/10/05 07:12:49] [ info] stdout.0
[2020/10/05 07:12:49] [ info] ___________
[2020/10/05 07:12:49] [ info] collectors:
[2020/10/05 07:12:49] [ info] [engine] started (pid=1)
[2020/10/05 07:12:49] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2020/10/05 07:12:49] [debug] [storage] [cio stream] new stream registered: forward.0
[2020/10/05 07:12:49] [ info] [storage] version=1.0.6, initializing...
[2020/10/05 07:12:49] [ info] [storage] in-memory
[2020/10/05 07:12:49] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/10/05 07:12:49] [debug] [in_fw] Listen='0.0.0.0' TCP_Port=24224
[2020/10/05 07:12:49] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224
[2020/10/05 07:12:49] [debug] [router] default match rule forward.0:stdout.0
[2020/10/05 07:12:49] [ info] [sp] stream processor started
[0] da81522ed489: [1601881963.000000000, {"container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6a"}]44", "container_name"=>"/busy_ellis", "source"=>"stdout", "log"=>"Mon Oct 5 07:12:43 UTC 2020
", "container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6ad6844", "container_name"=>"/busy_ellis"}]
[2020/10/05 07:12:50] [debug] [task] created task=0x7fc661e4d0a0 id=0 OK
[2020/10/05 07:12:50] [debug] [task] destroy task=0x7fc661e4d0a0 (task_id=0)
[0] da81522ed489: [1601881971.000000000, {"container_name"=>"/busy_ellis", "source"=>"stdout", "log"=>"Mon Oct 5 07:1", "container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6ad6844"}]
[1] da81522ed489: [1601881971.000000000, {"container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6a"}]44", "container_name"=>"/busy_ellis", "source"=>"stdout", "log"=>"flb (172.17.0.2:24224) open
[2020/10/05 07:12:51] [debug] [task] created task=0x7fc661e4d0a0 id=0 OK
[2020/10/05 07:12:51] [debug] [task] destroy task=0x7fc661e4d0a0 (task_id=0)
[0] da81522ed489: [1601881972.000000000, {"container_name"=>"/busy_ellis", "source"=>"stdout", "log"=>"Mon Oct 5 07:1", "container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6ad6844"}]
[1] da81522ed489: [1601881972.000000000, {"container_name"=>"/busy_ellis", "source"=>"stdout", "log"=>"flb (172.17.0.2", "container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6ad6844"}]
[2020/10/05 07:12:52] [debug] [task] created task=0x7fc661e4d0a0 id=0 OK
[2020/10/05 07:12:52] [debug] [task] destroy task=0x7fc661e4d0a0 (task_id=0)
[0] da81522ed489: [1601881973.000000000, {"container_name"=>"/busy_ellis", "source"=>"stdout", "log"=>"Mon Oct 5 07:1", "container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6ad6844"}]
[1] da81522ed489: [1601881973.000000000, {"container_id"=>"da81522ed489d3602dc56e27b5d5dd3aa881d1961328ae25fd7d0943d6a"}]44", "container_name"=>"/busy_ellis", "source"=>"stdout", "log"=>"flb (172.17.0.2:24224) open
The engine caught SIGTERM at 07:12:32 and the result of port scan was not open during shutdown. However, the first timestamp after restart was 1601881963 (Mon Oct 5 07:12:43 UTC 2020).
I'm reading the implementation of Docker's fluentd log driver. If my understanding is correct, it seems to check an error returned by net.Conn.Write().
https://github.com/moby/moby/blob/v19.03.12/daemon/logger/fluentd/fluentd.go#L175-L177
https://github.com/fluent/fluent-logger-golang/blob/v1.4.0/fluent/fluent.go#L384
I assumed it would retry if the connection was closed. I'm not sure why this doesn't work as intended.
A good news: It will check the response if Ack mode is enabled.
A bad news: Unfortunately, the fluentd-request-ack feature is part of Docker CE 20.03, which has not been released yet for now.
https://github.com/moby/moby/pull/39086
Even though without the fluentd-request-ack feature, I think closing the connection should cause an error. What am I misunderstanding?
minamijoyo
on 5 Oct 2020
I captured packets with tcpdump to figure out what's happened here.
$ echo 'FROM alpine\nRUN apk add --no-cache tcpdump' | docker build -t debug -f - . && docker run -it --rm --net container:flb debug tcpdump -nn -X port 24224
In the below test case, SIGTERM was sent at 08:24:22:
[2020/10/05 08:24:22] [engine] caught signal (SIGTERM)
tcpdump:
08:24:20.412946 IP 172.17.0.1.59006 > 172.17.0.2.24224: Flags [P.], seq 2905:3268, ack 1, win 229, options [nop,nop,TS val 2530955343 ecr 992119111], length 363
0x0000: 4500 019f 75f8 4000 4006 6b3b ac11 0001 E...u.@[email protected];....
0x0010: ac11 0002 e67e 5ea0 bc5b 1d76 4072 6d58 .....~^..[.v@rmX
0x0020: 8018 00e5 59b7 0000 0101 080a 96db 504f ....Y.........PO
0x0030: 3b22 8947 94ac 6565 3862 3266 3439 3939 ;".G..ee8b2f4999
0x0040: 6232 d25f 7ad8 3484 ae63 6f6e 7461 696e b2._z.4..contain
0x0050: 6572 5f6e 616d 65b2 2f6d 6167 6963 616c er_name./magical
0x0060: 5f61 6c62 6174 7461 6e69 a673 6f75 7263 _albattani.sourc
0x0070: 65a6 7374 646f 7574 a36c 6f67 bd4d 6f6e e.stdout.log.Mon
0x0080: 204f 6374 2020 3520 3038 3a32 343a 3230 .Oct..5.08:24:20
0x0090: 2055 5443 2032 3032 300d ac63 6f6e 7461 .UTC.2020..conta
0x00a0: 696e 6572 5f69 64d9 4065 6538 6232 6634 iner_id.@ee8b2f4
0x00b0: 3939 3962 3239 6236 6232 3061 3034 6265 999b29b6b20a04be
0x00c0: 6462 3437 6165 6131 3330 3665 3962 3865 db47aea1306e9b8e
0x00d0: 3462 3362 6439 3333 6161 3031 3062 3732 4b3bd933aa010b72
0x00e0: 3866 3465 6133 6664 6180 94ac 6565 3862 8f4ea3fda...ee8b
0x00f0: 3266 3439 3939 6232 d25f 7ad8 3484 ac63 2f4999b2._z.4..c
0x0100: 6f6e 7461 696e 6572 5f69 64d9 4065 6538 ontainer_id.@ee8
0x0110: 6232 6634 3939 3962 3239 6236 6232 3061 b2f4999b29b6b20a
0x0120: 3034 6265 6462 3437 6165 6131 3330 3665 04bedb47aea1306e
0x0130: 3962 3865 3462 3362 6439 3333 6161 3031 9b8e4b3bd933aa01
0x0140: 3062 3732 3866 3465 6133 6664 61ae 636f 0b728f4ea3fda.co
0x0150: 6e74 6169 6e65 725f 6e61 6d65 b22f 6d61 ntainer_name./ma
0x0160: 6769 6361 6c5f 616c 6261 7474 616e 69a6 gical_albattani.
0x0170: 736f 7572 6365 a673 7464 6f75 74a3 6c6f source.stdout.lo
0x0180: 67bc 666c 6220 2831 3732 2e31 372e 302e g.flb.(172.17.0.
0x0190: 323a 3234 3232 3429 206f 7065 6e0d 80 2:24224).open..
08:24:20.412993 IP 172.17.0.2.24224 > 172.17.0.1.59006: Flags [.], ack 3268, win 361, options [nop,nop,TS val 992120121 ecr 2530955343], length 0
0x0000: 4500 0034 0ab1 4000 4006 d7ed ac11 0002 E..4..@.@.......
0x0010: ac11 0001 5ea0 e67e 4072 6d58 bc5b 1ee1 ....^..~@rmX.[..
0x0020: 8010 0169 584c 0000 0101 080a 3b22 8d39 ...iXL......;".9
0x0030: 96db 504f ..PO
08:24:21.414651 IP 172.17.0.1.59006 > 172.17.0.2.24224: Flags [P.], seq 3268:3450, ack 1, win 229, options [nop,nop,TS val 2530956344 ecr 992120121], length 182
0x0000: 4500 00ea 75f9 4000 4006 6bef ac11 0001 E...u.@[email protected].....
0x0010: ac11 0002 e67e 5ea0 bc5b 1ee1 4072 6d58 .....~^..[..@rmX
0x0020: 8018 00e5 5902 0000 0101 080a 96db 5438 ....Y.........T8
0x0030: 3b22 8d39 94ac 6565 3862 3266 3439 3939 ;".9..ee8b2f4999
0x0040: 6232 d25f 7ad8 3584 ac63 6f6e 7461 696e b2._z.5..contain
0x0050: 6572 5f69 64d9 4065 6538 6232 6634 3939 er_id.@ee8b2f499
0x0060: 3962 3239 6236 6232 3061 3034 6265 6462 9b29b6b20a04bedb
0x0070: 3437 6165 6131 3330 3665 3962 3865 3462 47aea1306e9b8e4b
0x0080: 3362 6439 3333 6161 3031 3062 3732 3866 3bd933aa010b728f
0x0090: 3465 6133 6664 61ae 636f 6e74 6169 6e65 4ea3fda.containe
0x00a0: 725f 6e61 6d65 b22f 6d61 6769 6361 6c5f r_name./magical_
0x00b0: 616c 6261 7474 616e 69a6 736f 7572 6365 albattani.source
0x00c0: a673 7464 6f75 74a3 6c6f 67bd 4d6f 6e20 .stdout.log.Mon.
0x00d0: 4f63 7420 2035 2030 383a 3234 3a32 3120 Oct..5.08:24:21.
0x00e0: 5554 4320 3230 3230 0d80 UTC.2020..
08:24:21.414697 IP 172.17.0.2.24224 > 172.17.0.1.59006: Flags [.], ack 3450, win 369, options [nop,nop,TS val 992121122 ecr 2530956344], length 0
0x0000: 4500 0034 0ab2 4000 4006 d7ec ac11 0002 E..4..@.@.......
0x0010: ac11 0001 5ea0 e67e 4072 6d58 bc5b 1f97 ....^..~@rmX.[..
0x0020: 8010 0171 584c 0000 0101 080a 3b22 9122 ...qXL......;"."
0x0030: 96db 5438 ..T8
08:24:21.414760 IP 172.17.0.3.45137 > 172.17.0.2.24224: Flags [S], seq 3055193332, win 29200, options [mss 1460,sackOK,TS val 2770144932 ecr 0,nop,wscale 7], length 0
0x0000: 4500 003c 07aa 4000 4006 daea ac11 0003 E..<..@.@.......
0x0010: ac11 0002 b051 5ea0 b61a 8cf4 0000 0000 .....Q^.........
0x0020: a002 7210 5856 0000 0204 05b4 0402 080a ..r.XV..........
0x0030: a51d 0ea4 0000 0000 0103 0307 ............
08:24:21.414795 IP 172.17.0.2.24224 > 172.17.0.3.45137: Flags [S.], seq 4150699300, ack 3055193333, win 28960, options [mss 1460,sackOK,TS val 627047421 ecr 2770144932,nop,wscale 7], length 0
0x0000: 4500 003c 0000 4000 4006 e294 ac11 0002 E..<..@.@.......
0x0010: ac11 0003 5ea0 b051 f766 a524 b61a 8cf5 ....^..Q.f.$....
0x0020: a012 7120 5856 0000 0204 05b4 0402 080a ..q.XV..........
0x0030: 255f fbfd a51d 0ea4 0103 0307 %_..........
08:24:21.414852 IP 172.17.0.3.45137 > 172.17.0.2.24224: Flags [.], ack 1, win 229, options [nop,nop,TS val 2770144932 ecr 627047421], length 0
0x0000: 4500 0034 07ab 4000 4006 daf1 ac11 0003 E..4..@.@.......
0x0010: ac11 0002 b051 5ea0 b61a 8cf5 f766 a525 .....Q^......f.%
0x0020: 8010 00e5 584e 0000 0101 080a a51d 0ea4 ....XN..........
0x0030: 255f fbfd %_..
08:24:21.415041 IP 172.17.0.3.45137 > 172.17.0.2.24224: Flags [F.], seq 1, ack 1, win 229, options [nop,nop,TS val 2770144933 ecr 627047421], length 0
0x0000: 4500 0034 07ac 4000 4006 daf0 ac11 0003 E..4..@.@.......
0x0010: ac11 0002 b051 5ea0 b61a 8cf5 f766 a525 .....Q^......f.%
0x0020: 8011 00e5 584e 0000 0101 080a a51d 0ea5 ....XN..........
0x0030: 255f fbfd %_..
08:24:21.415189 IP 172.17.0.2.24224 > 172.17.0.3.45137: Flags [F.], seq 1, ack 2, win 227, options [nop,nop,TS val 627047422 ecr 2770144933], length 0
0x0000: 4500 0034 2e7b 4000 4006 b421 ac11 0002 E..4.{@.@..!....
0x0010: ac11 0003 5ea0 b051 f766 a525 b61a 8cf6 ....^..Q.f.%....
0x0020: 8011 00e3 584e 0000 0101 080a 255f fbfe ....XN......%_..
0x0030: a51d 0ea5 ....
08:24:21.415265 IP 172.17.0.3.45137 > 172.17.0.2.24224: Flags [.], ack 2, win 229, options [nop,nop,TS val 2770144933 ecr 627047422], length 0
0x0000: 4500 0034 07ad 4000 4006 daef ac11 0003 E..4..@.@.......
0x0010: ac11 0002 b051 5ea0 b61a 8cf6 f766 a526 .....Q^......f.&
0x0020: 8010 00e5 584e 0000 0101 080a a51d 0ea5 ....XN..........
0x0030: 255f fbfe %_..
08:24:21.417224 IP 172.17.0.1.59006 > 172.17.0.2.24224: Flags [P.], seq 3450:3631, ack 1, win 229, options [nop,nop,TS val 2530956347 ecr 992121122], length 181
0x0000: 4500 00e9 75fa 4000 4006 6bef ac11 0001 E...u.@[email protected].....
0x0010: ac11 0002 e67e 5ea0 bc5b 1f97 4072 6d58 .....~^..[..@rmX
0x0020: 8018 00e5 5901 0000 0101 080a 96db 543b ....Y.........T;
0x0030: 3b22 9122 94ac 6565 3862 3266 3439 3939 ;"."..ee8b2f4999
0x0040: 6232 d25f 7ad8 3584 ae63 6f6e 7461 696e b2._z.5..contain
0x0050: 6572 5f6e 616d 65b2 2f6d 6167 6963 616c er_name./magical
0x0060: 5f61 6c62 6174 7461 6e69 a673 6f75 7263 _albattani.sourc
0x0070: 65a6 7374 646f 7574 a36c 6f67 bc66 6c62 e.stdout.log.flb
0x0080: 2028 3137 322e 3137 2e30 2e32 3a32 3432 .(172.17.0.2:242
0x0090: 3234 2920 6f70 656e 0dac 636f 6e74 6169 24).open..contai
0x00a0: 6e65 725f 6964 d940 6565 3862 3266 3439 ner_id.@ee8b2f49
0x00b0: 3939 6232 3962 3662 3230 6130 3462 6564 99b29b6b20a04bed
0x00c0: 6234 3761 6561 3133 3036 6539 6238 6534 b47aea1306e9b8e4
0x00d0: 6233 6264 3933 3361 6130 3130 6237 3238 b3bd933aa010b728
0x00e0: 6634 6561 3366 6461 80 f4ea3fda.
08:24:21.417344 IP 172.17.0.2.24224 > 172.17.0.1.59006: Flags [.], ack 3631, win 377, options [nop,nop,TS val 992121125 ecr 2530956347], length 0
0x0000: 4500 0034 0ab3 4000 4006 d7eb ac11 0002 E..4..@.@.......
0x0010: ac11 0001 5ea0 e67e 4072 6d58 bc5b 204c ....^..~@rmX.[.L
0x0020: 8010 0179 584c 0000 0101 080a 3b22 9125 ...yXL......;".%
0x0030: 96db 543b ..T;
08:24:22.418717 IP 172.17.0.1.59006 > 172.17.0.2.24224: Flags [P.], seq 3631:3813, ack 1, win 229, options [nop,nop,TS val 2530957348 ecr 992121125], length 182
0x0000: 4500 00ea 75fb 4000 4006 6bed ac11 0001 E...u.@[email protected].....
0x0010: ac11 0002 e67e 5ea0 bc5b 204c 4072 6d58 .....~^..[.L@rmX
0x0020: 8018 00e5 5902 0000 0101 080a 96db 5824 ....Y.........X$
0x0030: 3b22 9125 94ac 6565 3862 3266 3439 3939 ;".%..ee8b2f4999
0x0040: 6232 d25f 7ad8 3684 ac63 6f6e 7461 696e b2._z.6..contain
0x0050: 6572 5f69 64d9 4065 6538 6232 6634 3939 er_id.@ee8b2f499
0x0060: 3962 3239 6236 6232 3061 3034 6265 6462 9b29b6b20a04bedb
0x0070: 3437 6165 6131 3330 3665 3962 3865 3462 47aea1306e9b8e4b
0x0080: 3362 6439 3333 6161 3031 3062 3732 3866 3bd933aa010b728f
0x0090: 3465 6133 6664 61ae 636f 6e74 6169 6e65 4ea3fda.containe
0x00a0: 725f 6e61 6d65 b22f 6d61 6769 6361 6c5f r_name./magical_
0x00b0: 616c 6261 7474 616e 69a6 736f 7572 6365 albattani.source
0x00c0: a673 7464 6f75 74a3 6c6f 67bd 4d6f 6e20 .stdout.log.Mon.
0x00d0: 4f63 7420 2035 2030 383a 3234 3a32 3220 Oct..5.08:24:22.
0x00e0: 5554 4320 3230 3230 0d80 UTC.2020..
08:24:22.418794 IP 172.17.0.2.24224 > 172.17.0.1.59006: Flags [.], ack 3813, win 386, options [nop,nop,TS val 992122126 ecr 2530957348], length 0
0x0000: 4500 0034 0ab4 4000 4006 d7ea ac11 0002 E..4..@.@.......
0x0010: ac11 0001 5ea0 e67e 4072 6d58 bc5b 2102 ....^..~@rmX.[!.
0x0020: 8010 0182 584c 0000 0101 080a 3b22 950e ....XL......;"..
0x0030: 96db 5824 ..X$
08:24:22.419107 IP 172.17.0.3.42625 > 172.17.0.2.24224: Flags [S], seq 1200225671, win 29200, options [mss 1460,sackOK,TS val 2770145937 ecr 0,nop,wscale 7], length 0
0x0000: 4500 003c 72cd 4000 4006 6fc7 ac11 0003 E..<r.@[email protected].....
0x0010: ac11 0002 a681 5ea0 4789 fd87 0000 0000 ......^.G.......
0x0020: a002 7210 5856 0000 0204 05b4 0402 080a ..r.XV..........
0x0030: a51d 1291 0000 0000 0103 0307 ............
08:24:22.419168 IP 172.17.0.2.24224 > 172.17.0.3.42625: Flags [S.], seq 3907279555, ack 1200225672, win 28960, options [mss 1460,sackOK,TS val 627048426 ecr 2770145937,nop,wscale 7], length 0
0x0000: 4500 003c 0000 4000 4006 e294 ac11 0002 E..<..@.@.......
0x0010: ac11 0003 5ea0 a681 e8e4 5ac3 4789 fd88 ....^.....Z.G...
0x0020: a012 7120 5856 0000 0204 05b4 0402 080a ..q.XV..........
0x0030: 255f ffea a51d 1291 0103 0307 %_..........
08:24:22.419210 IP 172.17.0.3.42625 > 172.17.0.2.24224: Flags [.], ack 1, win 229, options [nop,nop,TS val 2770145937 ecr 627048426], length 0
0x0000: 4500 0034 72ce 4000 4006 6fce ac11 0003 E..4r.@[email protected].....
0x0010: ac11 0002 a681 5ea0 4789 fd88 e8e4 5ac4 ......^.G.....Z.
0x0020: 8010 00e5 584e 0000 0101 080a a51d 1291 ....XN..........
0x0030: 255f ffea %_..
08:24:22.419370 IP 172.17.0.3.42625 > 172.17.0.2.24224: Flags [F.], seq 1, ack 1, win 229, options [nop,nop,TS val 2770145937 ecr 627048426], length 0
0x0000: 4500 0034 72cf 4000 4006 6fcd ac11 0003 E..4r.@[email protected].....
0x0010: ac11 0002 a681 5ea0 4789 fd88 e8e4 5ac4 ......^.G.....Z.
0x0020: 8011 00e5 584e 0000 0101 080a a51d 1291 ....XN..........
0x0030: 255f ffea %_..
08:24:22.419567 IP 172.17.0.2.24224 > 172.17.0.3.42625: Flags [.], ack 2, win 227, options [nop,nop,TS val 627048426 ecr 2770145937], length 0
0x0000: 4500 0034 d58e 4000 4006 0d0e ac11 0002 E..4..@.@.......
0x0010: ac11 0003 5ea0 a681 e8e4 5ac4 4789 fd89 ....^.....Z.G...
0x0020: 8010 00e3 584e 0000 0101 080a 255f ffea ....XN......%_..
0x0030: a51d 1291 ....
08:24:22.419631 IP 172.17.0.2.24224 > 172.17.0.3.42625: Flags [F.], seq 1, ack 2, win 227, options [nop,nop,TS val 627048426 ecr 2770145937], length 0
0x0000: 4500 0034 d58f 4000 4006 0d0d ac11 0002 E..4..@.@.......
0x0010: ac11 0003 5ea0 a681 e8e4 5ac4 4789 fd89 ....^.....Z.G...
0x0020: 8011 00e3 584e 0000 0101 080a 255f ffea ....XN......%_..
0x0030: a51d 1291 ....
08:24:22.419714 IP 172.17.0.3.42625 > 172.17.0.2.24224: Flags [.], ack 2, win 229, options [nop,nop,TS val 2770145937 ecr 627048426], length 0
0x0000: 4500 0034 0000 4000 4006 e29c ac11 0003 E..4..@.@.......
0x0010: ac11 0002 a681 5ea0 4789 fd89 e8e4 5ac5 ......^.G.....Z.
0x0020: 8010 00e5 b2d8 0000 0101 080a a51d 1291 ................
0x0030: 255f ffea %_..
08:24:22.427911 IP 172.17.0.1.59006 > 172.17.0.2.24224: Flags [P.], seq 3813:3994, ack 1, win 229, options [nop,nop,TS val 2530957358 ecr 992122126], length 181
0x0000: 4500 00e9 75fc 4000 4006 6bed ac11 0001 E...u.@[email protected].....
0x0010: ac11 0002 e67e 5ea0 bc5b 2102 4072 6d58 .....~^..[!.@rmX
0x0020: 8018 00e5 5901 0000 0101 080a 96db 582e ....Y.........X.
0x0030: 3b22 950e 94ac 6565 3862 3266 3439 3939 ;"....ee8b2f4999
0x0040: 6232 d25f 7ad8 3684 ac63 6f6e 7461 696e b2._z.6..contain
0x0050: 6572 5f69 64d9 4065 6538 6232 6634 3939 er_id.@ee8b2f499
0x0060: 3962 3239 6236 6232 3061 3034 6265 6462 9b29b6b20a04bedb
0x0070: 3437 6165 6131 3330 3665 3962 3865 3462 47aea1306e9b8e4b
0x0080: 3362 6439 3333 6161 3031 3062 3732 3866 3bd933aa010b728f
0x0090: 3465 6133 6664 61ae 636f 6e74 6169 6e65 4ea3fda.containe
0x00a0: 725f 6e61 6d65 b22f 6d61 6769 6361 6c5f r_name./magical_
0x00b0: 616c 6261 7474 616e 69a6 736f 7572 6365 albattani.source
0x00c0: a673 7464 6f75 74a3 6c6f 67bc 666c 6220 .stdout.log.flb.
0x00d0: 2831 3732 2e31 372e 302e 323a 3234 3232 (172.17.0.2:2422
0x00e0: 3429 206f 7065 6e0d 80 4).open..
08:24:22.427973 IP 172.17.0.2.24224 > 172.17.0.1.59006: Flags [.], ack 3994, win 394, options [nop,nop,TS val 992122136 ecr 2530957358], length 0
0x0000: 4500 0034 0ab5 4000 4006 d7e9 ac11 0002 E..4..@.@.......
0x0010: ac11 0001 5ea0 e67e 4072 6d58 bc5b 21b7 ....^..~@rmX.[!.
0x0020: 8010 018a 584c 0000 0101 080a 3b22 9518 ....XL......;"..
0x0030: 96db 582e ..X.
08:24:23.422948 IP 172.17.0.1.59006 > 172.17.0.2.24224: Flags [P.], seq 3994:4176, ack 1, win 229, options [nop,nop,TS val 2530958353 ecr 992122136], length 182
0x0000: 4500 00ea 75fd 4000 4006 6beb ac11 0001 E...u.@[email protected].....
0x0010: ac11 0002 e67e 5ea0 bc5b 21b7 4072 6d58 .....~^..[!.@rmX
0x0020: 8018 00e5 5902 0000 0101 080a 96db 5c11 ....Y.........\.
0x0030: 3b22 9518 94ac 6565 3862 3266 3439 3939 ;"....ee8b2f4999
0x0040: 6232 d25f 7ad8 3784 a36c 6f67 bd4d 6f6e b2._z.7..log.Mon
0x0050: 204f 6374 2020 3520 3038 3a32 343a 3233 .Oct..5.08:24:23
0x0060: 2055 5443 2032 3032 300d ac63 6f6e 7461 .UTC.2020..conta
0x0070: 696e 6572 5f69 64d9 4065 6538 6232 6634 iner_id.@ee8b2f4
0x0080: 3939 3962 3239 6236 6232 3061 3034 6265 999b29b6b20a04be
0x0090: 6462 3437 6165 6131 3330 3665 3962 3865 db47aea1306e9b8e
0x00a0: 3462 3362 6439 3333 6161 3031 3062 3732 4b3bd933aa010b72
0x00b0: 3866 3465 6133 6664 61ae 636f 6e74 6169 8f4ea3fda.contai
0x00c0: 6e65 725f 6e61 6d65 b22f 6d61 6769 6361 ner_name./magica
0x00d0: 6c5f 616c 6261 7474 616e 69a6 736f 7572 l_albattani.sour
0x00e0: 6365 a673 7464 6f75 7480 ce.stdout.
08:24:23.423002 IP 172.17.0.2.24224 > 172.17.0.1.59006: Flags [.], ack 4176, win 403, options [nop,nop,TS val 992123131 ecr 2530958353], length 0
0x0000: 4500 0034 0ab6 4000 4006 d7e8 ac11 0002 E..4..@.@.......
0x0010: ac11 0001 5ea0 e67e 4072 6d58 bc5b 226d ....^..~@rmX.["m
0x0020: 8010 0193 584c 0000 0101 080a 3b22 98fb ....XL......;"..
0x0030: 96db 5c11 ..\.
08:24:23.424909 IP 172.17.0.3.41581 > 172.17.0.2.24224: Flags [S], seq 781733910, win 29200, options [mss 1460,sackOK,TS val 2770146942 ecr 0,nop,wscale 7], length 0
0x0000: 4500 003c c321 4000 4006 1f73 ac11 0003 E..<.!@[email protected]....
0x0010: ac11 0002 a26d 5ea0 2e98 5016 0000 0000 .....m^...P.....
0x0020: a002 7210 5856 0000 0204 05b4 0402 080a ..r.XV..........
0x0030: a51d 167e 0000 0000 0103 0307 ...~........
08:24:23.425216 IP 172.17.0.2.24224 > 172.17.0.3.41581: Flags [R.], seq 0, ack 781733911, win 0, length 0
0x0000: 4500 0028 0000 4000 4006 e2a8 ac11 0002 E..(..@.@.......
0x0010: ac11 0003 5ea0 a26d 0000 0000 2e98 5017 ....^..m......P.
0x0020: 5014 0000 d7eb 0000 P.......
08:24:24.430950 IP 172.17.0.1.59006 > 172.17.0.2.24224: Flags [P.], seq 4176:4358, ack 1, win 229, options [nop,nop,TS val 2530959361 ecr 992123131], length 182
0x0000: 4500 00ea 75fe 4000 4006 6bea ac11 0001 E...u.@[email protected].....
0x0010: ac11 0002 e67e 5ea0 bc5b 226d 4072 6d58 .....~^..["m@rmX
0x0020: 8018 00e5 5902 0000 0101 080a 96db 6001 ....Y.........`.
0x0030: 3b22 98fb 94ac 6565 3862 3266 3439 3939 ;"....ee8b2f4999
0x0040: 6232 d25f 7ad8 3884 ae63 6f6e 7461 696e b2._z.8..contain
0x0050: 6572 5f6e 616d 65b2 2f6d 6167 6963 616c er_name./magical
0x0060: 5f61 6c62 6174 7461 6e69 a673 6f75 7263 _albattani.sourc
0x0070: 65a6 7374 646f 7574 a36c 6f67 bd4d 6f6e e.stdout.log.Mon
0x0080: 204f 6374 2020 3520 3038 3a32 343a 3234 .Oct..5.08:24:24
0x0090: 2055 5443 2032 3032 300d ac63 6f6e 7461 .UTC.2020..conta
0x00a0: 696e 6572 5f69 64d9 4065 6538 6232 6634 iner_id.@ee8b2f4
0x00b0: 3939 3962 3239 6236 6232 3061 3034 6265 999b29b6b20a04be
0x00c0: 6462 3437 6165 6131 3330 3665 3962 3865 db47aea1306e9b8e
0x00d0: 3462 3362 6439 3333 6161 3031 3062 3732 4b3bd933aa010b72
0x00e0: 3866 3465 6133 6664 6180 8f4ea3fda.
08:24:24.431012 IP 172.17.0.2.24224 > 172.17.0.1.59006: Flags [.], ack 4358, win 411, options [nop,nop,TS val 992124139 ecr 2530959361], length 0
0x0000: 4500 0034 0ab7 4000 4006 d7e7 ac11 0002 E..4..@.@.......
0x0010: ac11 0001 5ea0 e67e 4072 6d58 bc5b 2323 ....^..~@rmX.[##
0x0020: 8010 019b 584c 0000 0101 080a 3b22 9ceb ....XL......;"..
0x0030: 96db 6001 ..`.
08:24:24.431077 IP 172.17.0.3.38779 > 172.17.0.2.24224: Flags [S], seq 1660570891, win 29200, options [mss 1460,sackOK,TS val 2770147949 ecr 0,nop,wscale 7], length 0
0x0000: 4500 003c da2d 4000 4006 0867 ac11 0003 E..<.-@[email protected]....
0x0010: ac11 0002 977b 5ea0 62fa 4d0b 0000 0000 .....{^.b.M.....
0x0020: a002 7210 5856 0000 0204 05b4 0402 080a ..r.XV..........
0x0030: a51d 1a6d 0000 0000 0103 0307 ...m........
08:24:24.431132 IP 172.17.0.2.24224 > 172.17.0.3.38779: Flags [R.], seq 0, ack 1660570892, win 0, length 0
0x0000: 4500 0028 0000 4000 4006 e2a8 ac11 0002 E..(..@.@.......
0x0010: ac11 0003 5ea0 977b 0000 0000 62fa 4d0c ....^..{....b.M.
0x0020: 5014 0000 b186 0000 P.......
08:24:25.434570 IP 172.17.0.1.59006 > 172.17.0.2.24224: Flags [P.], seq 4358:4540, ack 1, win 229, options [nop,nop,TS val 2530960364 ecr 992124139], length 182
0x0000: 4500 00ea 75ff 4000 4006 6be9 ac11 0001 E...u.@[email protected].....
0x0010: ac11 0002 e67e 5ea0 bc5b 2323 4072 6d58 .....~^..[##@rmX
0x0020: 8018 00e5 5902 0000 0101 080a 96db 63ec ....Y.........c.
0x0030: 3b22 9ceb 94ac 6565 3862 3266 3439 3939 ;"....ee8b2f4999
0x0040: 6232 d25f 7ad8 3984 a36c 6f67 bd4d 6f6e b2._z.9..log.Mon
0x0050: 204f 6374 2020 3520 3038 3a32 343a 3235 .Oct..5.08:24:25
0x0060: 2055 5443 2032 3032 300d ac63 6f6e 7461 .UTC.2020..conta
0x0070: 696e 6572 5f69 64d9 4065 6538 6232 6634 iner_id.@ee8b2f4
0x0080: 3939 3962 3239 6236 6232 3061 3034 6265 999b29b6b20a04be
0x0090: 6462 3437 6165 6131 3330 3665 3962 3865 db47aea1306e9b8e
0x00a0: 3462 3362 6439 3333 6161 3031 3062 3732 4b3bd933aa010b72
0x00b0: 3866 3465 6133 6664 61ae 636f 6e74 6169 8f4ea3fda.contai
0x00c0: 6e65 725f 6e61 6d65 b22f 6d61 6769 6361 ner_name./magica
0x00d0: 6c5f 616c 6261 7474 616e 69a6 736f 7572 l_albattani.sour
0x00e0: 6365 a673 7464 6f75 7480 ce.stdout.
08:24:25.434625 IP 172.17.0.2.24224 > 172.17.0.1.59006: Flags [.], ack 4540, win 419, options [nop,nop,TS val 992125142 ecr 2530960364], length 0
0x0000: 4500 0034 0ab8 4000 4006 d7e6 ac11 0002 E..4..@.@.......
0x0010: ac11 0001 5ea0 e67e 4072 6d58 bc5b 23d9 ....^..~@rmX.[#.
0x0020: 8010 01a3 584c 0000 0101 080a 3b22 a0d6 ....XL......;"..
0x0030: 96db 63ec ..c.
08:24:25.434679 IP 172.17.0.3.36663 > 172.17.0.2.24224: Flags [S], seq 3331640399, win 29200, options [mss 1460,sackOK,TS val 2770148952 ecr 0,nop,wscale 7], length 0
0x0000: 4500 003c 575d 4000 4006 8b37 ac11 0003 E..<W]@[email protected]....
0x0010: ac11 0002 8f37 5ea0 c694 cc4f 0000 0000 .....7^....O....
0x0020: a002 7210 5856 0000 0204 05b4 0402 080a ..r.XV..........
0x0030: a51d 1e58 0000 0000 0103 0307 ...X........
08:24:25.434714 IP 172.17.0.2.24224 > 172.17.0.3.36663: Flags [R.], seq 0, ack 3331640400, win 0, length 0
0x0000: 4500 0028 0000 4000 4006 e2a8 ac11 0002 E..(..@.@.......
0x0010: ac11 0003 5ea0 8f37 0000 0000 c694 cc50 ....^..7.......P
0x0020: 5014 0000 d6eb 0000 P.......
08:24:26.439883 IP 172.17.0.3.36427 > 172.17.0.2.24224: Flags [S], seq 3898998966, win 29200, options [mss 1460,sackOK,TS val 2770149957 ecr 0,nop,wscale 7], length 0
0x0000: 4500 003c 9e49 4000 4006 444b ac11 0003 E..<.I@[email protected]....
0x0010: ac11 0002 8e4b 5ea0 e866 00b6 0000 0000 .....K^..f......
0x0020: a002 7210 5856 0000 0204 05b4 0402 080a ..r.XV..........
0x0030: a51d 2245 0000 0000 0103 0307 .."E........
08:24:26.439969 IP 172.17.0.2.24224 > 172.17.0.3.36427: Flags [R.], seq 0, ack 3898998967, win 0, length 0
0x0000: 4500 0028 0000 4000 4006 e2a8 ac11 0002 E..(..@.@.......
0x0010: ac11 0003 5ea0 8e4b 0000 0000 e866 00b7 ....^..K.....f..
0x0020: 5014 0000 819f 0000 P.......
08:24:26.440410 IP 172.17.0.1.59006 > 172.17.0.2.24224: Flags [P.], seq 4540:4722, ack 1, win 229, options [nop,nop,TS val 2530961370 ecr 992125142], length 182
0x0000: 4500 00ea 7600 4000 4006 6be8 ac11 0001 E...v.@[email protected].....
0x0010: ac11 0002 e67e 5ea0 bc5b 23d9 4072 6d58 .....~^..[#.@rmX
0x0020: 8018 00e5 5902 0000 0101 080a 96db 67da ....Y.........g.
0x0030: 3b22 a0d6 94ac 6565 3862 3266 3439 3939 ;"....ee8b2f4999
0x0040: 6232 d25f 7ad8 3a84 ac63 6f6e 7461 696e b2._z.:..contain
0x0050: 6572 5f69 64d9 4065 6538 6232 6634 3939 er_id.@ee8b2f499
0x0060: 3962 3239 6236 6232 3061 3034 6265 6462 9b29b6b20a04bedb
0x0070: 3437 6165 6131 3330 3665 3962 3865 3462 47aea1306e9b8e4b
0x0080: 3362 6439 3333 6161 3031 3062 3732 3866 3bd933aa010b728f
0x0090: 3465 6133 6664 61ae 636f 6e74 6169 6e65 4ea3fda.containe
0x00a0: 725f 6e61 6d65 b22f 6d61 6769 6361 6c5f r_name./magical_
0x00b0: 616c 6261 7474 616e 69a6 736f 7572 6365 albattani.source
0x00c0: a673 7464 6f75 74a3 6c6f 67bd 4d6f 6e20 .stdout.log.Mon.
0x00d0: 4f63 7420 2035 2030 383a 3234 3a32 3620 Oct..5.08:24:26.
0x00e0: 5554 4320 3230 3230 0d80 UTC.2020..
08:24:26.440434 IP 172.17.0.2.24224 > 172.17.0.1.59006: Flags [.], ack 4722, win 428, options [nop,nop,TS val 992126148 ecr 2530961370], length 0
0x0000: 4500 0034 0ab9 4000 4006 d7e5 ac11 0002 E..4..@.@.......
0x0010: ac11 0001 5ea0 e67e 4072 6d58 bc5b 248f ....^..~@rmX.[$.
0x0020: 8010 01ac 584c 0000 0101 080a 3b22 a4c4 ....XL......;"..
0x0030: 96db 67da ..g.
08:24:26.749555 IP 172.17.0.2.24224 > 172.17.0.1.59006: Flags [F.], seq 1, ack 4722, win 428, options [nop,nop,TS val 992126457 ecr 2530961370], length 0
0x0000: 4500 0034 0aba 4000 4006 d7e4 ac11 0002 E..4..@.@.......
0x0010: ac11 0001 5ea0 e67e 4072 6d58 bc5b 248f ....^..~@rmX.[$.
0x0020: 8011 01ac 584c 0000 0101 080a 3b22 a5f9 ....XL......;"..
0x0030: 96db 67da ..g.
08:24:26.750562 IP 172.17.0.1.59006 > 172.17.0.2.24224: Flags [.], ack 2, win 229, options [nop,nop,TS val 2530961680 ecr 992126457], length 0
0x0000: 4500 0034 7601 4000 4006 6c9d ac11 0001 E..4v.@[email protected].....
0x0010: ac11 0002 e67e 5ea0 bc5b 248f 4072 6d59 .....~^..[$.@rmY
0x0020: 8010 00e5 584c 0000 0101 080a 96db 6910 ....XL........i.
0x0030: 3b22 a5f9 ;"..
From the above, IP address mappings are probably as follows:
172.17.0.1.59006: docker engine (fluentd log driver)172.17.0.2.24224: fluent-bit172.17.0.3.*: nc(port scanner)
As you can see, new connections from 172.17.0.3.* were rejected, but the existing connection from 172.17.0.1.59006 was still sending packets. What's actually happened for the existing connection on the socket close operation?
minamijoyo
on 5 Oct 2020
@edsiper I noticed that v1.6.0 release note said:
https://fluentbit.io/announcements/v1.6.0/
Properly handle of SIGTERM signal, graceful shutdown.
What does that mean for this issue?
minamijoyo
on 15 Oct 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
Markbnj
路
4Comments
arienchen
路
3Comments
tarokkk
路
3Comments
mhf-ir
路
4Comments
mbelchin
路
3Comments
Most helpful comment
troubleshooting...