Vector: Vector integrations with Loki

[nadgobpx01]

Hello ,

I am trying to get logs from external device into loki with vector . So far i have not been successful . Is it posible , is there any example how the config shoul look ?
I am also facing this error :
`
root# vector --config /etc/vector/vector.toml --require-healthy

Jun 11 16:50:06.236 INFO vector: Log level "info" is enabled.
Jun 11 16:50:06.237 INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Jun 11 16:50:06.241 INFO vector: Vector is starting. version="0.9.2" git_version="v0.9.2" released="Tue, 09 Jun 2020 22:07:11 +0000" arch="x86_64"
Jun 11 16:50:06.256 INFO vector::topology: Running healthchecks.
Jun 11 16:50:06.272 INFO vector::topology::builder: Healthcheck: Passed.
Jun 11 16:50:06.272 INFO vector::topology: All healthchecks passed.
Jun 11 16:50:06.272 INFO vector::topology: Starting source "in"
Jun 11 16:50:06.272 INFO vector::topology: Starting sink "out"
Jun 11 16:50:06.272 ERROR source{name=in type=syslog}: vector::sources::util::tcp: Failed to bind to listener socket: TCP bind failed: Address in use (os error 98)
Jun 11 16:50:06.272 ERROR vector::topology: Unhandled error
Jun 11 16:50:06.272 INFO vector: Shutting down.

`
My config file looks like this :

# Note: A full config spec is located at ./vector.spec.toml and examples
#       in the ./examples folder.

data_dir = "/var/lib/vector"

# Input data. Change me to a valid input source.
#

[sources.in]
  address = "0.0.0.0:5114" # required, required when mode = "tcp" or mode = "udp"
  mode = "tcp" # required
  path = "/path/to/socket" # required, required when mode = "unix"
  type = "syslog" # required



[sinks.out]
  # General
  endpoint = "http://127.0.0.1:3100" # required
  inputs = ["in"] # required
  type = "loki" # required
#Encoding
encoding.codec = "json" # optional, default

# Labels
  labels.key = "value" # example
  labels.key = "{{ event_field }}" # example
~

[binarylogic]

@ktff would you mind looking into this? Especially the very vague Unhandled error error.

[pySilver]

also docs says:

# Encoding
  encoding.codec = "json" # optional, default

but if you omit it, loki plugin will complain and refuse to start https://github.com/timberio/vector/issues/2710

[binarylogic]

Thanks for letting us know. We’ll get it fixed.

[nadgobpx01]

Hi guys thank you for having the time to look into this . would much appreciate the confirmation that it is possible to dump the logs from a different device such as a UPS to the server.

[ktff]

# Encoding
 encoding.codec = "json" # optional, default

but if you omit it, loki plugin will complain and refuse to start #2710

The documentation is correct, that's a bug.

---

Jun 11 16:50:06.272 ERROR source{name=in type=syslog}: vector::sources::util::tcp: Failed to bind to listener socket: TCP bind failed: Address in use (os error 98)

It seams that something else is using the configured port 5114. I assume you are on linux, so you can call sudo lsof -i -P -n | grep LISTEN to see what process is occupying that port, or use a different one, if you can.

@nadgobpx01 could you check if this fixes the issue.

---

Especially the very vague Unhandled error error.

This is caused by the same error as above. It means that there was an error that vector wasn't able to fix/handle , although you would need to dive into the code to figure out that meaning, so we should change it to communicate that.

---

Other things should be ok, but there are some things to note for good measure.

path = "/path/to/socket" # required, required when mode = "unix"

This is not needed since you have mode = "tcp"

labels.key = "value" # example
labels.key = "{{ event_field }}" # example

Only one of the these will have an effect, other one will be overwritten.

[nadgobpx01]

@ktff thank you .
The OS version is Description: Ubuntu 18.04.4 LTS

I have made the suggested changes and configuration looks like this:
"`cat vector.toml

data_dir = "/var/lib/vector"

[sources.in]
address = "0.0.0.0:5114" # required, required when mode = "tcp" or mode = "udp"
mode = "tcp" # required
type = "syslog" # required

[sinks.out]

endpoint = "http://127.0.0.1:3100" # required
inputs = ["in"] # required
type = "loki" # required

encoding.codec = "json" # optional, default

labels.key = "{{ event_field }}" # example"
`
I have also runed a health check wit the following result :

vector --config /etc/vector/vector.toml --require-healthy Jun 12 21:02:33.163 INFO vector: Log level "info" is enabled. Jun 12 21:02:33.163 INFO vector: Loading configs. path=["/etc/vector/vector.toml"] Jun 12 21:02:33.169 INFO vector: Vector is starting. version="0.9.2" git_version="v0.9.2" released="Tue, 09 Jun 2020 22:07:11 +0000" arch="x86_64" Jun 12 21:02:33.184 INFO vector::topology: Running healthchecks. Jun 12 21:02:33.199 INFO vector::topology::builder: Healthcheck: Passed. Jun 12 21:02:33.200 INFO vector::topology: All healthchecks passed. Jun 12 21:02:33.200 INFO vector::topology: Starting source "in" Jun 12 21:02:33.200 INFO vector::topology: Starting sink "out" Jun 12 21:02:33.200 ERROR source{name=in type=syslog}: vector::sources::util::tcp: Failed to bind to listener socket: TCP bind failed: Address in use (os error 98) Jun 12 21:02:33.200 ERROR vector::topology: Unhandled error Jun 12 21:02:33.200 INFO vector: Shutting down.
The result of the sugested comand : sudo lsof -i -P -n | grep LISTEN is as following :

vector 542 vector 15u IPv4 73925347 0t0 TCP *:5114 (LISTEN) kapacitor 1037 kapacitor 7u IPv6 24914 0t0 TCP *:9092 (LISTEN) container 1288 root 6u IPv4 22245 0t0 TCP 127.0.0.1:43837 (LISTEN) sshd 1499 root 3u IPv4 23844 0t0 TCP *:22 (LISTEN) sshd 1499 root 4u IPv6 23848 0t0 TCP *:22 (LISTEN) loki 11516 root 7u IPv6 62373100 0t0 TCP *:3100 (LISTEN) loki 11516 root 8u IPv6 62373101 0t0 TCP *:9095 (LISTEN) sshd 11545 b 9u IPv6 73267673 0t0 TCP [::1]:6010 (LISTEN) sshd 11545 b 10u IPv4 73267674 0t0 TCP 127.0.0.1:6010 (LISTEN) promtail 11914 root 3u IPv6 62383221 0t0 TCP *:9080 (LISTEN) promtail 11914 root 5u IPv6 62383222 0t0 TCP *:46625 (LISTEN) influxd 19078 influxdb 3u IPv4 49284938 0t0 TCP 127.0.0.1:8088 (LISTEN) influxd 19078 influxdb 30u IPv6 49283036 0t0 TCP *:8086 (LISTEN) sendmail- 23295 root 4u IPv4 356361 0t0 TCP 127.0.0.1:25 (LISTEN) sendmail- 23295 root 5u IPv4 356362 0t0 TCP 127.0.0.1:587 (LISTEN) systemd-r 29531 systemd-resolve 13u IPv4 62258553 0t0 TCP 127.0.0.53:53 (LISTEN) chronogra 30215 chronograf 7u IPv6 62264669 0t0 TCP *:8888 (LISTEN) grafana-s 30555 grafana 9u IPv6 62262051 0t0 TCP *:3000 (LISTEN) plugin_st 30598 grafana 19u IPv4 62264493 0t0 TCP 127.0.0.1:40729 (LISTEN) sshd 31654 b 9u IPv6 73908197 0t0 TCP [::1]:6011 (LISTEN) sshd 31654 b 10u IPv4 73908198 0t0 TCP 127.0.0.1:6011 (LISTEN)

On service vector status i have this :

"`/etc/vector# service vector status
● vector.service - Vector
Loaded: loaded (/etc/systemd/system/vector.service; disabled; vendor preset: enabled)
Active: active (running) since Fri 2020-06-12 20:28:53 EEST; 41min ago
Docs: http://vector.dev
Main PID: 542 (vector)
Tasks: 8 (limit: 4915)
CGroup: /system.slice/vector.service
└─542 /usr/bin/vector

Jun 12 20:28:53 hostname systemd[1]: Started Vector.
Jun 12 20:28:53 hostname vector[542]: Jun 12 20:28:53.619 INFO vector: Log level "info" is enabled.
Jun 12 20:28:53 hostname vector[542]: Jun 12 20:28:53.621 INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Jun 12 20:28:53 hostname vector[542]: Jun 12 20:28:53.628 INFO vector: Vector is starting. version="0.9.2" git_version="v0.9.2" released="Tue, 09 Jun 2020 22:07:11 +0000" arch="x86_64"
Jun 12 20:28:53 hostname vector[542]: Jun 12 20:28:53.645 INFO vector::topology: Running healthchecks.
Jun 12 20:28:53 hostname vector[542]: Jun 12 20:28:53.646 INFO vector::topology: Starting source "in"
Jun 12 20:28:53 hostname vector[542]: Jun 12 20:28:53.646 INFO vector::topology: Starting sink "out"
Jun 12 20:28:53 hostname vector[542]: Jun 12 20:28:53.646 INFO source{name=in type=syslog}: vector::sources::util::tcp: listening. addr=0.0.0.0:5114
Jun 12 20:28:53 hostname vector[542]: Jun 12 20:28:53.662 INFO vector::topology::builder: Healthcheck: Passed. "
`

[ktff]

@nadgobpx01 Are you running this second vector as a service intentionally, because it's using the port 5114, so if you want to have two vector running at the same time they will need to use different ports. So one can use
address = "0.0.0.0:5114"
while the other can use
address = "0.0.0.0:5115"

[nadgobpx01]

@kff the second vector instance is not intentionally started . I figured it out that the command
"vector --config /etc/vector/vector.toml --require-healthy " runs well when i have the service stopped . When service is started running the commands for check it tries to start another instance, that's why it complains about the port usage.
So at the moment the logs show fine but still can't parse logs from a remote host to the server.

[ktff]

So at the moment the logs show fine but still can't parse logs from a remote host to the server.

@nadgobpx01 now that can be a lot of things, port mismatch, firewall, etc. https://gitter.im/timberio-vector/community is better suited to help you in that regard. But do return if you encounter some bug.

[ktff]

For now it seems that there is no unresolved issues witk loki sink that wasn't fixed with #2796, so I'll close this.