Hello.
We run all our trainings on single GPU and are therefore able to run multiple trainings at the same time. By setting a display_id all the images and plots gets messed up in Visdom. Is there anyway to set a visdom environment for each run?
Great idea! That would make things look much nicer. I'll look into this.
@davidwessman
The quickest and easiest solution i found is set different display_port for each train so each job's trainign result can be displayed on different visdom sever.
For example when you want to train 4 jobs at once. At first start 4 instances of visdom server at 4 different ports. Then for each job, set the display_port to corresponding port so the training result would appear on separated visdom server
python -m visdom.server -port 8096
python -m visdom.server -port 8097
python -m visdom.server -port 8098
python -m visdom.server -port 8099
job1: python train.py ..... --display_port 8096
job2: python train.py ..... --display_port 8097
job3: python train.py ..... --display_port 8098
job4: python train.py ..... --display_port 8099
Hope that help
Resolved in this commit: https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/commit/bcfbb81555d9518b346f9a7c28f1991257c66925
:)
Most helpful comment
@davidwessman
The quickest and easiest solution i found is set different display_port for each train so each job's trainign result can be displayed on different visdom sever.
For example when you want to train 4 jobs at once. At first start 4 instances of visdom server at 4 different ports. Then for each job, set the display_port to corresponding port so the training result would appear on separated visdom server
Hope that help