Ecs: Consume ECS schema

[pimjansen]

Hey,

Im trying to consume the ECS schema for PHP and convert my logs to the ECS schema. Overall it works ok but it is a bit useless to manually create all the mappings and also to keep it in sync.

Is there a solid way to consume this data instead of just maintaining it manually?

Best
Pim

[webmat]

Hi @pimjansen, we’re starting to release formatters for the popular logging libraries in various languages. You can check them out here https://github.com/elastic/ecs-logging. For PHP we currently support Monolog.

This is very much a new offering, and we’d love to have your feedback :-)

If that doesn’t solve what you’re looking for, please let us know a bit more details about what problems you’re facing.

cc @philkra

[pimjansen]

Yeah i noticed this indeed however monolog 2 is only supported where it is pretty new. I made some sort of port to monolog 1 as you can see at
https://github.com/pimjansen/monolog-ecs-formatter

What i noticed is that the php implementation doesnt do much where monolog pushes all data in a context or extra key. I made a small mapping stuff to move stuff inside these keys to the root level to ensure it atleast matches some. What actually is intended ofcourse it that it maps it correctly on the ecs schema definition in yaml. If it matches it can store it in those keys, if not it can keep it in "extra" or "context".

So i guess it would be good if we could somehow access the schema itself so we can implement so dynamic mapping on there. Since just maintaining it on multiple sides is not really going to work i guess

[webmat]

Ah I see, you're further along than I assumed :-)

Yes, currently the first step for logging formatters, is to replace a basic log formatter that properly fills 10-15 ECS fields * for a user that doesn't know much about ECS. Doing what we can for them, without having a barrier to adoption of the library.

We have also started discussing how we can help people who actively want to fill ECS fields because they see the value, as is your situation.

@philkra and @felixbarny I think it would be worth opening a public discussion on the ecs-logging repo, on how we can facilitate users populating the other ECS fields. WDYT?

* This list of 10-15 fields is here for the Java library. The PHP one doesn't fill them all, at this time, since it's newer. But I think those are the ones we can potentially fill without the developer being actively thinking about ECS.

[philkra]

@pimjansen , happy new year 😄 I just pushed a newer release to master for the Monolog formatter (ecs-logging-php), that address' your concerns about propagating the attributes from context to the root element.

At the moment, "only" user, service, tracing and error are supported, but this list can pretty easily be expanded, check out the examples here.

I also changed the namespace to open up the introduction of Monolog v1.x. You made already amazing progress in the regards of supporting v1.x, if you're open to contribute, we would be moooooore than happy about to see your name in the contributors list 😃

Please, do not hesitate to have a look at the code in ecs-logging-php, quite a lot of cheese has been moved and we're always looking for feedback.

[webmat]

Thanks @philkra :-)

Are we good to close this issue? I think further PHP & Monolog discussion should happen on the ecs-logging-php repo. WDYT?

[pimjansen]

Fine by me, however i have one question about the maintainability. Since we are adding logic in the php formatter to ensure it will map the right fields according the schema. How will we maintain it? Since we have 2 projects which will never be properly in sync.

Wouldnt it be better to auto generate some mapping files during builds of the php formatter? Meaning that somehow the schema itself needs to publish the mapping somewhere. It this case it will be much more easy (except the initial implementation).

Further, if im correct atleast, the schema is coupled to a specific elastic version which you there could also render on a given version of the formatter

[philkra]

Hi @pimjansen , sorry for the late reply!

Since we have 2 projects which will never be properly in sync.

We do have a spec to test against.

Wouldnt it be better to auto generate some mapping files during builds of the php formatter?

Yes, a generator to map the schema proper PHP objects would be nice, I added some Types manually. I wasn't sure what other types would be relevant to model as ECS is very rich in that regards, I focused primary on logging use cases. That being said, ecs-dotnet is doing exactly that.

Further, if im correct atleast, the schema is coupled to a specific elastic version which you there could also render on a given version of the formatter

We are following the major version of ECS, currently we are at 1.4.0 hence the lib will be released as 1.x. After a bump to ECS 2.x so will we comply to that as well in the lib.

[ebeahan]

Closing - feel free to open another issue if the discussion needs to continue 😄

Also, please open any PHP-specific logging ECS concerns in https://github.com/elastic/ecs-logging-php