Beats: Packetbeat setup error

[liza-mae]

• Version: master latest
• Operating System: Linux
• Steps to Reproduce:
  ./packetbeat setup
  Overwriting ILM policy is disabled. Set setup.ilm.overwrite:true for enabling.

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Exiting: error generating index pattern: field is duplicated, remove it or set 'overwrite: true', {Name:version Type:keyword Description:The version of the TLS protocol used.
Format: Fields:[] MultiFields:[] Enabled: Analyzer: SearchAnalyzer: Norms:false Dynamic:{Value:} Index: DocValues: CopyTo: IgnoreAbove:0 AliasPath: MigrationAlias:false Dimension: ObjectType: ObjectTypeMappingType: ScalingFactor:0 ObjectTypeParams:[] Analyzed: Count:0 Searchable: Aggregatable: Script: Pattern: InputFormat: OutputFormat: OutputPrecision: LabelTemplate: UrlTemplate:[] OpenLinkInCurrentTab: Overwrite:false DefaultField: Path:tls.version}, {"aggregatable":true,"analyzed":false,"count":0,"doc_values":true,"indexed":true,"name":"tls.version","scripted":false,"searchable":true,"type":"string"}

I am getting the above error running packetbeat setup, not sure what it means, can someone help?
It was not doing this before, so not sure if it is a bug or change in behavior.

[liza-mae]

This is happening on oss and default packages -

packetbeat.yml.txt

./packetbeat version
packetbeat version 8.0.0 (amd64), libbeat 8.0.0 [6c75f403281068623185fb330a166a09cc9df977 built 2020-01-13 14:27:12 +0000 UTC]

Snapshot version:
8.0.0-ae74a27c

Kibana version:
BUILD 29871 COMMIT 14df4c09

Elasticsearch version:
number: "8.0.0-SNAPSHOT",
build_flavor: "oss",
build_type: "tar",
build_hash: "a0513217dba5d964bf176c8f409baccfbd25359b",
build_date: "2020-01-13T15:44:08.340307Z",

2020-01-14T00:01:30.643Z ERROR instance/beat.go:921

Exiting: error generating index pattern: field is duplicated, remove it or set 'overwrite: true', {Name:version Type:keyword Description:The version of the TLS protocol used.
Format: Fields:[] MultiFields:[] Enabled: Analyzer: SearchAnalyzer: Norms:false Dynamic:{Value:} Index: DocValues: CopyTo: IgnoreAbove:0 AliasPath: MigrationAlias:false Dimension: ObjectType: ObjectTypeMappingType: ScalingFactor:0 ObjectTypeParams:[] Analyzed: Count:0 Searchable: Aggregatable: Script: Pattern: InputFormat: OutputFormat: OutputPrecision: LabelTemplate: UrlTemplate:[] OpenLinkInCurrentTab: Overwrite:false DefaultField: Path:tls.version}, {"aggregatable":true,"analyzed":false,"count":0,"doc_values":true,"indexed":true,"name":"tls.version","scripted":false,"searchable":true,"type":"string"}

cc: @ycombinator

[ycombinator]

I can confirm that this is indeed a bug. I'm able to reproduce it on master with the following minimal steps:

1. Build packetbeat.

   cd $GOPATH/src/github.com/elastic/beats/packetbeat
   mage clean update build
   

2. Try to export the index pattern.

   ./packetbeat export index-pattern
   

[ycombinator]

Looking at the Packetbeat commit history for the past few days, I wonder if this bug was introduced by either https://github.com/elastic/beats/pull/14844 or https://github.com/elastic/beats/pull/15497. @adriansr WDYT?

[ycombinator]

Yes, doing a git bisect, it looks like the bug was introduced in https://github.com/elastic/beats/pull/14844.

[adriansr]

It looks like this was broken during the period of time between #14844 and #15497 was merged, which makes sense, because the first adds a new tls.version and the second removes the old field from packetbeat.

It shouldn't happen anymore since #15497 was merged, and indeed it doesn't fail for me. @ycombinator can you confirm with an updated master branch?

I will add a system-tests that invokes beat export index-pattern to make sure this doesn't happen again.

[ycombinator]

Yes, pulled down master again and built Packetbeat. I can no longer cause this bug. Thanks @adriansr!

[liza-mae]

Thanks @ycombinator @adriansr -- so next master snapshot should pass for me too. Also please note this fix needs to be backported to 7.x.