Apex: RuntimeError: Unexpected optimization level 01. Option are '00' '01' '02' '03'

[wangyingyu]

When I made the following changes to the original code and reported this error, I don't know where the problem occurred. Can you give me some advice? Thank you.
model, optimizer = amp.initialize(model, optimizer, opt_level="O1") with amp.scale_loss(loss, optimizer) as scaled_loss: scaled_loss.backward()

[ptrblck]

Hi @wangyingyu,
could you check if you are trying to pass the opt_level as two integers (i.e. "zero one" "01") instead of the expected uppercase O letter and 1 ("O1")?

[vijaytida]

I have some issue after the change made made as you mentioned. Can you please give me some advice Traceback (most recent call last):
File "CIFAR.py", line 71, in
model, optimizer = amp.initialize(model, optimizer, opt_level="O1")
File "/home/gputest/.local/lib/python3.6/site-packages/apex/amp/frontend.py", line 356, in initialize
return _initialize(models, optimizers, _amp_state.opt_properties, num_losses, cast_model_outputs)
File "/home/gputest/.local/lib/python3.6/site-packages/apex/amp/_initialize.py", line 230, in _initialize
optimizers[i] = _process_optimizer(optimizer, properties)
File "/home/gputest/.local/lib/python3.6/site-packages/apex/amp/_process_optimizer.py", line 258, in _process_optimizer
raise RuntimeError("A given optimizer should only be passed through amp.initialize once.")
RuntimeError: A given optimizer should only be passed through amp.initialize once.

[ptrblck]

It looks like you are trying to pass the same optimizer instance multiple times to amp.initialize.
Could this be the case? If so, could you pass it only once or do you have a special use case where you think this might be needed?

[mcarilli]

@vijaytida Are you calling amp.initialize multiple times, and passing the same optimizer through it multiple times?

[vijaytida]

It looks like you are trying to pass the same optimizer instance multiple times to amp.initialize.
Could this be the case? If so, could you pass it only once or do you have a special use case where you think this might be needed?

Thank you I figured it out.

[thammegowda]

(Glad to find I am not the only one)
the names couldn't get any better than O1, O2, O3 😆!
Oh, by the way, that's not zero😲, it's the O as in 🍊🤦🏻‍♂️

Suggestion (before moving to pytorch core):

1. consider aliasing to lowercase o at least that would make it obvious when written like o1, o2, o3
2. better yet: please alias them level_1, level_2, level_3 or opt_1, opt_2, opt_3

🙏🏼

[mcarilli]

I named them O0-O3 in loose analogy to gcc (the analogy is loose because with Amp, O1 is strongly recommended as the best option for speed and safety). I don't want to clutter the existing API with more aliases, but I changed the RuntimeError to explain this exact case.

[thammegowda]

great! thanks