Vector: Event type in lua is userdata, not table

[mikhno-s]

Hi!

Docs says that event has lua type table, I get userdata, hence, I cannot use pairs function.

My configuration:

OS: OS X 10.14.6
Vector ver.:vector 0.6.0-nightly (the same problem on v0.4.0 and 0.5.0, probably)

config.toml:

data_dir = "${PWD}/data"

[sources.in_err_logs]
    type = "file"
    include = ["./*.log"]
    glob_minimum_cooldown = 1000
    ignore_older = 86400
    max_read_bytes = 4096


[sources.in_err_logs.fingerprinting]
    strategy = "checksum"
    fingerprint_bytes = 16
    ignored_header_bytes = 0

[transforms.test]
    type = "lua"
    inputs = ["in_err_logs"]
    source = """
       print(type(event))
    """

[sinks.print_to_console]
  type = "console"
  inputs = ["test"]
  encoding = "json"

error.log:

{"MESSAGE":{"text":"error"}}

output:

$ vector --config config.toml
Dec 02 16:55:16.012  INFO vector: Log level "info" is enabled.
Dec 02 16:55:16.013  INFO vector: Loading config. path="config.toml"
Dec 02 16:55:16.031  INFO vector: Vector is starting. version="0.6.0" git_version="v0.5.0-198-ga6668b0" released="Mon, 02 Dec 2019 11:16:59 +0000" arch="x86_64"
Dec 02 16:55:16.038  INFO vector::topology: Running healthchecks.
Dec 02 16:55:16.038  INFO vector::topology: Starting source "in_err_logs"
Dec 02 16:55:16.039  INFO vector::topology: Starting transform "test"
Dec 02 16:55:16.039  INFO vector::topology::builder: Healthcheck: Passed.
Dec 02 16:55:16.039  INFO vector::topology: Starting sink "print_to_console"
Dec 02 16:55:16.040  INFO source{name=in_err_logs type=file}: vector::sources::file: Starting file server. include=["./*.log"] exclude=[]
Dec 02 16:55:16.042  INFO source{name=in_err_logs type=file}:file_server: file_source::file_server: Found file to watch. path="error.log" file_position=0
userdata
{"timestamp":"2019-12-02T14:55:16.044518Z","file":"error.log","host":"my-mac.local","message":"{\"MESSAGE\":{\"text\":\"error\"}}"}

As you can see print(type(event)) - printed userdata, not a table as I expected due to docs.

[binarylogic]

Hi @mikhno-s, thanks for letting us know. We'll update the docs. Additionally, we're about to make a bunch of improvements to our lua integration so stay tuned!

[WillDrug]

@binarylogic hey, I need to move nested fields around, so userdata field basically means I can't use vector at all, because there is no way to manipulate a dynamic set or fields (labels, for instance). With userdata event["root.nested.key"] is a string, for instance, but event["root.nested"] is nil.
Was there an update? If not, when can we expect one?

UPD: Alternatevily, there probably should be a way to manipulate event fields without scripting. Lua should be used for something complex, while simple rearranging should be done with add_field transform (or move_field\copy\etc)

[binarylogic]

Hey @WillDrug, sorry about that and thanks for letting us know. We just started working through https://github.com/timberio/vector/milestone/23, where we expect to improve the lua (and javascript) integration considerably. I'd classify the current integration in a very MVP state, and this next pass will bring improvements like:

1. Stricter type conversion.
2. Actual nested data model representation.
3. Native lua extensions.
4. A variety of other performance and bug fixes.

@a-rodin is owning that project and I expect that we'll address this properly within 3-4 weeks. @a-rodin is there a simple workaround in the interim? Ideally, something that aligns with further changes we're going to make. If not, then we should have something within the timeframe stated.

[WillDrug]

@binarylogic well, after the update, at least pairs function works correctly, so if the matter is pressing we can use it (also, inspect). But for now we've decided to configure metadata at each source and use grokk for new fields

[a-rodin]

I wanted suggest to use pairs to get all keys and then renaming the keys values correspondingly.

However, we do plan to change the nesting approach, so that the nested fields would be represented as nested tables instead of flat key values with dots.