Pyo3: Bridging respective logging facilities

[vorner]

Hello

I'm not sure this is in scope of the project, but I'd still want to present the idea here.

I'm wondering about a use case where I write an extension module in Rust and use it from python application. And I'd certainly like to be able to log from the rust part, or let my rust dependencies to log too. One way to do it is to log from the python and rust parts independently, but that doesn't sound exactly right.

So the other way would be to implement a logger on the Rust side that would delegate logging to the python side (I do see some possible issues with that approach ‒ python doesn't seem to have trace level, the performance might be a problem).

And my question is, if I give it a try and something usable comes of it:

• Would other people be interested in having this functionality?
• Would someone with more experience of using pyo3 give it a look/review?
• How much closely related to pyo3 should this be? Should it be a completely separate crate, with separate git repo and possibly mentioned as useful in some guide of pyo3 (is there a place for such „handy“ crates for various tasks like this?)? Or should it live under the same repo so it can easily follow the same release cycle? Or even be a feature-flag directly in the crate?

I guess one could also want the bridge in other direction (eg. python logging piping stuff to the rust side), but I don't currently have such use case myself and haven't thought much about how to implement it; I have some about this direction.

(I assume there isn't such thing yet, I haven't found it but maybe I haven't looked well enough)

[davidhewitt]

I'd be happy to help review it. For now I wonder if it makes sense to be a separate crate along the lines of android_logger and console_log?

Possible names to me include python_logger, or pyo3_logger if you want to take the pyo3 dependency explicit.

As to marketing this crate, I think it would be worth mentioning in the pyo3 README.

[vorner]

Thank you for the offer. I've put something together. It feels more or less ready for release (the test coverage could be better and I'll have to figure out how to test such a thing anyway). But if you could have a look if there's something obviously broken or could be done better, I'd be glad. https://github.com/vorner/pyo3-log

As to marketing this crate, I think it would be worth mentioning in the pyo3 README.

That would be nice :-). I was also thinking of adding an example somewhere into the „Advanced topics“ or adding another appendix to the guide, if that would be fine.

[davidhewitt]

Thanks - the crate looks nice! A collection of random thoughts:

The caching stuff looks very advanced. I guess because log configuration is typically global and done once at program startup, this is why you are caching the Python log hierarchy?

Minimum supported Rust version - pyo3 only supports Rust 1.39 and above, so I can't imagine pyo3-log makes sense to have MSRV lower than that?

Acquiring the Python GIL during logging - I'm planning to add Python::with_gil(|py| {...}) soon which will avoid acquiring the GIL if it's already acquired for a slight performance boost. (Watch out for that in 0.12 release in a month or two.)

Your hello world example looks good, though I think if you modified it to build with maturin then it would be easier for users to try it out.

Adding another appendix to pyo3 guide - that would be very welcome. Please feel free to open a PR!

[vorner]

Thanks. Then I think I'll just publish it and see how it goes and matures.

Should we keep this open for someone to tackle the other direction, or should we close it?

I guess because log configuration is typically global and done once at program startup, this is why you are caching the Python log hierarchy?

Yes. And because (as I describe in the docs), acquiring the GIL every time would be killing the performance of potentially multi-threaded processing in Rust.

Minimum supported Rust version

Do I mention that thing anywhere? I think I'll avoid any promises for 0.1 for now 😇.

Your hello world example looks good, though I think if you modified it to build with maturin then it would be easier for users to try it out.

I build it with maturin locally. Do you suggest to have some kind of build.sh thing around? Or am I missing something obvious?

[davidhewitt]

Do I mention that thing anywhere? I think I'll avoid any promises for 0.1 for now innocent.

You mention Rust 1.26 in the README which is surprising as I thought pyO3 didn't work below 1.39 :)

I build it with maturin locally. Do you suggest to have some kind of build.sh thing around? Or am I missing something obvious?

Ah nice, I got confused because maturin can support a few different project layouts. Maybe just add a short README to the example showing how to build & run it?

[vorner]

You mention Rust 1.26 in the README

:D Found it. That was a copy paste from another project. No, it certainly doesn't work in that ancient version, that's even pre-2018 edition thing.

Maybe just add a short README

Good idea, will do sometime soon.

[davidhewitt]

Should we keep this open for someone to tackle the other direction, or should we close it?

I think the other direction is less likely to be useful, so I'm happy for this issue to be closed.