Vector: Curly braces in the include directive, file pattern

[Tarasovych]

Vector Version

# vector --version
vector 0.10.0 (g0f0311a x86_64-unknown-linux-gnu 2020-07-22)

Vector Configuration File

[sources.nginx]
  type = "file"
  include = ["/var/log/nginx/{site1.example.com,site2.example.com,site3.example.com}.access.log"]

Debug Output

INFO source{name=nginx type=file}: vector::sources::file: Starting file server. include=["/var/log/nginx/{site1.example.com,site2.example.com,site3.example.com}.access.log"] exclude=[]

Expected Behavior

Vector is able to watch all three files.

Actual Behavior

Vector could not watch those files.

Additional Context

ls -l /var/log/nginx/{site1.example.com,site2.example.com,site3.example.com}.access.log

works just fine.

[sources.nginx]
  type = "file"
  include = ["/var/log/nginx/site1.example.com.access.log", "/var/log/nginx/site2.example.com.access.log", "/var/log/nginx/site3.example.com.access.log"]

also works fine:

INFO source{name=nginx type=file}:file_server: file_source::file_server: Found file to watch. path="/var/log/nginx/site1.example.com.access.log" file_position=0
INFO source{name=nginx type=file}:file_server: file_source::file_server: Found file to watch. path="/var/log/nginx/site2.example.com.access.log" file_position=0
INFO source{name=nginx type=file}:file_server: file_source::file_server: Found file to watch. path="/var/log/nginx/site3.example.com.access.log" file_position=0

Is it possible to use {} in the include directive or am I missing something?

[jszwedko]

It looks like this is an issue with the upstream glob crate we are using: https://github.com/rust-lang-nursery/glob/issues/2 . That crate doesn't seem particularly maintained anymore given its age, last commit, last release, and number of stale issues.

globset (https://crates.io/crates/globset) seems more active and does support the brace syntax.

[jamtur01]

@jszwedko Looks close to a drop in ish replacement too.

[lukesteensen]

I don't think it's a replacement for us, unfortunately. From the readme comparison with glob:

Doesn't provide a recursive directory iterator of matching file paths

That's the exact function that we use in the file source.

[jszwedko]

Yeah, that is an unfortunate gap. I do see some discussion around that in https://github.com/BurntSushi/ripgrep/pull/765

From there I found a link to https://github.com/gilnaa/globwalk which might be another candidate.

More discussion around glob: https://github.com/rust-lang-nursery/glob/issues/59

[lukesteensen]

globwalk looks good! I think we could go ahead and swap that in.