Beats: Strange filebeat + logstash behaviour leading to truncated content

[alexef]

We have the following setup:

Log Producers
PHP scripts write to timestamp named files, using:
file_put_contents($filename, $content, FILE_APPEND | LOCK_EX);, where $content always ends with PHP_EOL, and $filename is 'filename' . $now->format('Ymd') . '.log'.

Generic logging tools like cronolog write to files, being run:
cronolog /var/log/filename.%Y%m%d.log.

Collector
Filebeat 1.2.2 (previously we had similar issues with logstash 1.4), configured with:

---
filebeat:
  prospectors:
  - paths:
    - "/var/log/*.log"
output:
  logstash:
    hosts:
    - logstash:3000

Issue
Randomly, every time the file name changes (after midnight), in the first few tens lines, we see one broken line. The broken lines truncate the content from the middle to the end, for example, instead of:

{"value": "I am a full json", "context": 3}

we see

son", "context": 3}

Let us know what other details do you need.

[alexef]

@ruflin we talked briefly about our issue at the Elasticsearch Meetup. We're currently migrating to filebeat 5.0.0 hoping this will fix the issue.

[tsg]

Thanks for trying 5.0, let us know please if it fixes the issue.

[alexef]

More than 10 days after the switch to 5.0, we don't see the broken lines - I would say the upgrade fixed it.

[tsg]

Thanks @alexef for coming back with the confirmation. I'm going to close it in this case.

[ruflin]

@alexef Glad to hear that the 5.0 release fixes this. I suggest we close this issues as there are no bug fix releases planned for the 1.2 release.