Xla: accuracy regression for resnet50 in test_train_imagenet.py

[zcain117]

I've tried 2 ways, both using the real imagenet dataset. I have not submitted https://github.com/pytorch/xla/pull/1012 with the resnet50 learning rate scheduler:

1. Brand new compute VM, using pytorch-nightly conda env
2. Brand new compute VM, pull from master on Github and run build_torch_wheels.sh

Both versions reach ~12% accuracy around epoch 2 or 3 and can't make it any higher no matter how many more epochs they train.

I have a different compute VM made on August 28 that uses pytorch-nightly conda env and on that compute VM I am able to reach 60% accuracy with no changes and 76% accuracy when I implement a learning rate schedule.

[zcain117]

Ran a diff of /usr/share/torch-xla-nightly/pytorch/vision/torchvision/models/resnet.py on the old VM that works and the new VM that does not and found no difference

[jysohn23]

Can we try bisect a little more with different Compute VM image dates? You can bisect using the Compute VM image dates using the, for example, --image=debian-9-torch-xla-v20190911 --image-project=ml-images flags when creating with gcloud CLI.

Also, could you try updating the torch_xla wheel in the August 28th VM to just verify it's not us that caused this regression?

[zcain117]

confirmed that test_train_cifar.py passes using the in-line version of resnet18 but fails to reach 80% accuracy using the torchvision version of resnet18

edit:
this doesn't seem like a useful distinction since the same things happens on my compute VM where test_train_imagenet.py has high accuracy

[taylanbil]

did we rule out the lr_scheduler PR? IIRC, high accuracy run is w/o that pr, and the low accuracy is with, is that correct?

[zcain117]

did we rule out the lr_scheduler PR? IIRC, high accuracy run is w/o that pr, and the low accuracy is with, is that correct?

No, I've been using master version of pytorch/xla and haven't checked in that PR

[dlibenzi]

So it seems like a torchvision change?

[zcain117]

So it seems like a torchvision change?

there was no difference in /usr/share/torch-xla-nightly/pytorch/vision/torchvision/models/resnet.py between the older, higher accuracy compute VM and the new compute VMs with bad accuracy

[zcain117]

I've run a lot of experiments and here are the 3 that I think are most enlightening:

1. Copied the image from the high-performing compute VM originally created August 28. Built new torch and torch_xla wheels and torchivision using update_nightly_torch_wheels.sh. Ran against a new TPU VM. This made it to ~45% accuracy on epoch 7, i.e. aligned with the trajectory that gets us to 76% final accuracy.
2. New compute VM built using build_torch_wheels.sh and a new TPU VM. Whether I run on master branch or on the branch that pulls #1012, accuracy can't make it past ~12-15%
3. New compute VM using pytorch-nightly conda env and a new TPU VM made it to ~43% by epoch 11. Looks promising

From this, it seems building from head using build_torch_wheels.sh results in test_train_imagenet.py not working for some reason.

I'll keep investigating

[ailzhang]

@zcain117 I'm a little confused. I thought we use build_torch_wheels.sh to build nightly conda env, no?
fyi https://github.com/pytorch/xla/blob/master/scripts/build_torch_wheels.sh#L98 pins version to torchvision 0.3.0, maybe check whether nightly has master or 0.3.0 to see if that makes a difference?

[zcain117]

@zcain117 I'm a little confused. I thought we use build_torch_wheels.sh to build nightly conda env, no?
fyi https://github.com/pytorch/xla/blob/master/scripts/build_torch_wheels.sh#L98 pins version to torchvision 0.3.0, maybe check whether nightly has master or 0.3.0 to see if that makes a difference?

That might be true, but @jysohn23 pointed out that we override the local source for pytorch/xla and pytorch(?). I think the pytorch/xla override is here but not sure about pytorch.

I just checked and I see 0.3.0 in both the newer conda env that has bad accuracy and the older conda env that has good accuracy.

I think next it's best to see differences in python source code between the high-accuracy and low-accuracy compute VMs

[dlibenzi]

Did we rule out this is not a pytorch or pytorch/xla commit breakage?
IOW, does it exist a valid VM configuration where building from HEAD works?

[zcain117]

Did we rule out this is not a pytorch or pytorch/xla commit breakage?
IOW, does it exist a valid VM configuration where building from HEAD works?

I have not been able to get >20% accuracy when building from head

[zcain117]

I'm preparing 3 compute VMs using different pytorch/xla commits to hopefully see whether or not this problem came from a pytorch/xla change

[dlibenzi]

I think I have building a bisection infrastructure among our team's tasks ... for a reason 😄

[zcain117]

I tried with several different day's versions of pytorch/xla commits and none of them seemed to work. Next I wanted to try several different commits of pytorch but was running into various issues trying to build and run with different commits of pytorch.

@jysohn23 recommended revisiting the dated image approach, e.g. --image=debian-9-torch-xla-v20190911, so I'm approach again. Last time, I built on those dated images using build_torch_wheels.sh and still had the accuracy issue, but Daniel recommends not using build_torch_wheels.sh and just using the pre-installed pytorch in the pytorch-nightly conda env

[jysohn23]

To rule out it's not a LR scheduler missing issue, I'm trying latest nightly image + @zcain117 provided https://github.com/pytorch/xla/commit/5e50b17c88e491e3b8b86afcc6d2754bcefdbcfd that worked and got to 76%. Although the accuracies were measured before the first /5 of the LR (epoch 20) there still is the difference of learning rate ramp up, so I'll test that out.

[jysohn23]

Based on the above experiment I mention, I don't think we have any regression here. With hacked-up version (non-pr) version of the LR scheduling I am able to attain the following accuracy curve:

Here are the logs for the above run: float32_batch128_lr0.5_divideBy5Every20Epochs_warmup1EpochLinear.txt

Will let run overnight all the way to 90 epochs and report final accuracy.

[dlibenzi]

Great! :tada:
But we still have a mystery to solve.
Zach is getting bad accuracy. Could it be the TPU VMs he is using?

[jysohn23]

I think its in the optimizer PR.

On Fri, Sep 13, 2019, 8:29 AM Davide Libenzi [e-mail]
wrote:

Great! 🎉
But we still have a mystery to solve.
Zach is getting bad accuracy. Could it be the TPU VMs he is using?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pytorch/xla/issues/1025?email_source=notifications&email_token=AEUXZQUTBUBCQUAN46PDCVDQJOWUVA5CNFSM4IVWQFBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6VL4WI#issuecomment-531283545,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEUXZQSYC6JH2NRWMK3WQQDQJOWUVANCNFSM4IVWQFBA
.

[dlibenzi]

I think its in the optimizer PR.

But Zach said he tested vanilla test_train_imagenet.py ...

[jysohn23]

Yes and I think there was no problem with vanilla test_train_imagenet.py.
Vanilla just doesnt get that high of accuracy w/o LR scheduling by epoch

1. With the rampup and stepping we are now at 73%+ top 1 at 77th epoch.
   Will update later today at 90th epoch.

On Fri, Sep 13, 2019, 8:41 AM Davide Libenzi [e-mail]
wrote:

I think its in the optimizer PR.

But Zach said he tested vanilla test_train_imagenet.py ...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pytorch/xla/issues/1025?email_source=notifications&email_token=AEUXZQWGTQR66DTRFSED6CTQJOYCXA5CNFSM4IVWQFBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6VM7OA#issuecomment-531287992,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEUXZQTYHUYV3R6GOKAYVIDQJOYCXANCNFSM4IVWQFBA
.

[zcain117]

I think the potential difference is that unmodified test_train_imagenet.py no longer hits 50-60% accuracy, or at least not on the schedule it did before (high 40's in by epoch 20 and then slowly creeps up to 50-60%).

From a recent unmodified run:

[xla:1] Accuracy=13.80%
[xla:4] Accuracy=15.82%
[xla:7] Accuracy=13.97%
[xla:5] Accuracy=16.61%
[xla:6] Accuracy=12.18%
[xla:8] Accuracy=10.70%
[xla:2] Accuracy=13.90%
[xla:3] Accuracy=19.30%
Epoch: 33, Mean Accuracy: 14.53%

However, with a learning rate scheduler (in this case, we were using this hacky version) there seems to be no issues with accuracy.

I don't think the regression of unmodified resnet50 is worth spending more time on - maybe the 50-60% I was getting before was just lucky. The convergence is a lot more reliable with a learning rate scheduler

[zcain117]

Previously, I was pulling in the PR for the LR scheduling, running build_torch_wheels.sh, and then running test_train_imagenet.py but I was getting low accuracy.

With Daniel's finding, I think this narrows down our current problem to 1 of 2 things:

1. There is a mistake in that PR
2. Something strange happens with build_torch_wheels.sh

I'm going to try again now using 2 new compute VMs, once where I pull in the PR and build from scratch with build_torch_wheels.sh and once where I patch the changes in directly to the conda environment source code

[jysohn23]

OK great! We're good and hit 76.3% accuracy at 90th epoch.

Full logs:
float32_batch128_lr0.5_divideBy5Every20Epochs_warmup1EpochLinear (2).txt

[zcain117]

There were 5 tests I wanted to run:

1. Run using latest pytorch-nightly with #1012 patched in
2. Build from head using build_torch_wheels.sh after pulling #1012
3. Run with no learning rate scheduler using latest pytorch-nightly
4. Build from head and run with no learning rate scheduler
5. Run with no learning rate scheduler using August 28 pytorch-nightly

Results:

1. 76.2% accuracy latest_pytorch_nightly_including_new_scheduler.txt
2. Failed: grabbed source during a torch breakage caused by this. Will try again since they reverted that change
3. 15.51% accuracy
   float32_batch128_lr0.5_divideBy5Every20Epochs_warmup1EpochLinear_builtFromHead.txt
4. 15.51% accuracy
   float32_batch128_lr0.5_divideBy5Every20Epochs_warmup1EpochLinear_unmodifiedConda.txt
5. TODO

It seems we are good to go when using a learning rate scheduler but lower accuracy than before if not using a learning rate scheduler. I'll run test 5 to see exactly what accuracy we used to get, but from my memory it was ~40-50%.

I'll also repeat test 2 with latest source to make sure the new scheduler PR is safe to submit

[zcain117]

I was able to get the no-scheduler version up to ~45% accuracy by playing with learning rate and it looks the same whether on Sept 14 pytorch-nightly or Aug 28, so I think there is no regression.

My only remaining question is whether we reliably get the same results with build from head vs. from the installed conda version, but I'll follow up on that outside of this issue and make a new one if I see a difference