Ignite: Add Trains logger

[vfdev-5]

🚀 Feature

Add logger to Trains similarly to Neptune, MLflow, ...

[bmartinn]

Hi @vfdev-5
That is a great idea!
I'll open an issue on Trains, for the sake of complete transparency, but we can continue the technical discussion here.

A quick question, what do you think is the most complete (feature wise) back-end logger for us to take as an example template ?
I would imagine Tensorboard, but it seems to be missing media logging (image/audio),
what do you think?

[vfdev-5]

Hi @bmartinn
yes, we can take Tensorboard as an example.
If you would like to add audio logging to Trains logger, no objections.
Concerning images, it can be tricky to provide a generic logger for every type of images (RGB 8b=OK, multi-bands M-bits=> needs rendering) and satisfy everyone with a simple interface. So, we skipped this. That being said, maybe, we can think to provide a logging for the simplest case of RGB 8bits images.
What do you think ?

[bmartinn]

Hmm @vfdev-5 I can totally see what you mean, both audio/image do not have a unified format, even channel order is a thing...

That said I think it makes a lot of sense to allow at least a simple way to upload images (I guess that's the most trivial use case).

Is pillow already in the requirements ? If so we could just support PIL.Image, if not we can just do 3-channel 8-bit numpy array with a default channel order of RGB, but optional BGR (for cv2 users). Would that work?

[vfdev-5]

Well, we do not have PIL, neither numpy in requirements. Anyway, I think what can we do is to expose your task (or experiment) instance which can log according to your API. It is done for Tensorboard in the same way: https://github.com/pytorch/ignite/blob/master/ignite/contrib/handlers/tensorboard_logger.py#L458

What do you think ?

IMO, logging just single batch of images wont make much sense and what is interesting is to log images + targets. And there you can imagine the cases if target is a single word, tags or bboxes or segmentation mask etc... So, for instance this part is put to user's side.

[bmartinn]

in the same way: https://github.com/pytorch/ignite/blob/master/ignite/contrib/handlers/tensorboard_logger.py#L458

Looks good to me, I think we will have the trains.Task and the trains.Logger as two properties, but that sounds like something that will be easy to implement and quite useful :)

IMO, logging just single batch of images wont make much sense and what is interesting is to log images + targets. And there you can imagine the cases if target is a single word, tags or bboxes or segmentation mask etc... So, for instance this part is put to user's side.

Trains allows to log multiple batches, on a per step/iteration basis, but it seems what you are suggesting is adding the ability to attach some meta-data with the image itself (like text or json), is that right?

[vfdev-5]

Trains allows to log multiple batches, on a per step/iteration basis, but it seems what you are suggesting is adding the ability to attach some meta-data with the image itself (like text or json), is that right?

Sorry, maybe I was not clear about that. I was talking about to log images and targets like that:

or as here

It is an image from tensorboard with [Image; Image+Preds; Preds; Image+GT; GT]
But as I said, this is more task specific logging.

[bmartinn]

@vfdev-5 I see, okay that's already supported , and quite easy to achieve with the trains.Logger
With that in mind, I think that actually adding metadata on top of the image might be useful as well, food for thought I guess :)

[bmartinn]

Okay, I guess next step PR :)

[vfdev-5]

@bmartinn yes, please, go ahead :)

[vfdev-5]

@bmartinn any updates on that ?

[bmartinn]

Hi @vfdev-5 , yes... with current affairs, holidays, and a new version things are bushed back.
That said, I'm hoping to have it next week.
When is the next release scheduled?

[vfdev-5]

@bmartinn sounds good ! We plan the next release in middle/end of May.

[jkhenning]

Hi @vfdev-5,

I'm working on the trains logger PR, and have a quick question - it seems BaseWeightsScalarHandler is equivalent to the Tensorboard gradients (aka histogram) but in the Tensorboard handler, BaseWeightsHistHandler is used to log histograms as well? Do you think I should just mimic this behavior even though the underlying implementation is basically the same?

Also, do you think I should add an interface for logging stuff that isn't currently supported in the base logger (such as hyper-parameters, audio, images etc.), as additional user-callable methods of the Trains logger? Or am I missing something?

Thanks!

[vfdev-5]

Hi @jkhenning, thanks for working on that ! So, @bmartinn delegated it to you if I understand correctly ?

it seems BaseWeightsScalarHandler is equivalent to the Tensorboard gradients (aka histogram) but in the Tensorboard handler, BaseWeightsHistHandler is used to log histograms as well? Do you think I should just mimic this behavior even though the underlying implementation is basically the same?

I'm sorry, I do not understand your question. All BaseWeights*Handler classes are interfaces (they do not log, just some initialization checks) and real logging classes like in TensorBoard logger inherit from them and do the job. Yes, as you correctly remarked BaseWeightsScalarHandler is the base class for logging model weights and model grads as scalars, e.g. norm. As Tensorboard UI allows to plot histograms, we can log weights and grads as histograms too. So, yes, please, if it is possible for Trains logger to inherit and do the same as for Tensorboard logger, let's do it like this.

Also, do you think I should add an interface for logging stuff that isn't currently supported in the base logger (such as hyper-parameters, audio, images etc.), as additional user-callable methods of the Trains logger? Or am I missing something?

Hyperparams loggins, yes if you can make it simple etc, please add it. For others, we discussed this point just above with @bmartinn. We can leave it for instance as is on user's side.

What do you think ?

[jkhenning]

Hi @vfdev-5 ,

So, @bmartinn delegated it to you if I understand correctly ?

Just lending a helping hand :)

As for your answers - I agree. Thanks for the quick reply!

[jkhenning]

Hi @vfdev-5,

Sorry for the delay. I have the implementation in my forked repository (branch named trains-integration) if you'd like to take a look before I open a PR.
If not - I'll just open one :)

https://github.com/jkhenning/ignite/tree/trains-integration

Thanks!

[vfdev-5]

Hi @jkhenning
I looked quickly the implementation, looks good. Please, open a PR and we can iterate on details.
Just to make recent updates, we introduced 2 methods into BaseLoggers to simplify the API etc

• https://github.com/pytorch/ignite/commit/662433798683ef72920f7ba4b248b16558df6adc
  
  
  
  • _create_output_handler
  
  
  • _create_opt_params_handler
  
  
• https://github.com/pytorch/ignite/commit/0afaaa3fc3efd1f86a979461eb265f72acafc409
  we need to add them into this code too.

Thanks !

[jkhenning]

Hi @vfdev-5

Your feedback is much appreciated :)

I've added the missing code and opened a PR (https://github.com/pytorch/ignite/pull/1020).

Please note the trains==0.14.3rc0 requirement in requirements-dev.txt - the integration required some minor adjustments to make things easier - I'll release the v0.14.3 patch version in a day or two so we can change it before merging the PR.

Thanks!