Logstash: input file start_position => "beginning" doesn't work for me

[yhjhoo]

As the title. I am tying logstash and I found the file input doesn't work.

Only the newly appended file records was printed out.

Here is my configuration:
input {
stdin { }
file {
path => "/apps/apache/access_log"
start_position => "beginning"
}
}

filter {
if [type] == "apache-access" { # this is where we use the type from the input section
grok {
match => [ "message", "%{COMBINEDAPACHELOG}" ]
}
}
}

output {
stdout { }
elasticsearch {
host => localhost
}
}

[jsvd]

In what way doesn't it work? I keeps reading from the end, or doesn't display nothing at all?

Logstash's input file plugin remembers how far it has read each file in a .sincedb file stored (by default) in your home directory. So once you have started logstash and processed a particular file from the beginning, the next time it runs it will resume from the last position of that file (thus ignoring "start_position"). You can try removing the "~/.sincedb*" files in your homedir and check that logstash processes the files from the start again.

[suyograo]

If you have follow up questions, we would love to help you in logstash-users ML. Thanks

[yhjhoo]

It works after I deleted .sincedb* files.

Thanks

[rmuslimov]

Same here. Can we add this to logstash file input documentation page?

[rmuslimov]

Ah, it's there, nevermind, my bad.

[joestump]

Had a similar issue and setting sincedb to /dev/null got the files read from the beginning every time. Not a great solution, but if you're testing throughput it'll get around this issue.

[guneetbh]

I am facing same issue. Even after deleting the .sincedb_*, I dont see my file getting processed from beginning.
input {
file {
path => "C:\logstash-7.9.0\data\event-data\apache_access.log"
start_position => "beginning"
}
}
output {
stdout {
}
}

Thanks,