Transformers: get_linear_schedule_with_warmup Scheduler

[h56cho]

Hello,

When I try to execute the line of code below, Python gives me an import error:

from pytorch_transformers import (GPT2Config, GPT2LMHeadModel, GPT2DoubleHeadsModel,
                                  AdamW, get_linear_schedule_with_warmup)

ImportError: cannot import name 'get_linear_schedule_with_warmup' from 'pytorch_transformers' (/Users/hyunjindominiquecho/opt/anaconda3/lib/python3.7/site-packages/pytorch_transformers/__init__.py)

What should I then import to use the linear scheduler with warm up?

Thank you,

[bkkaggle]

You might be using an old version of the library, try updating it to v2.2.0

[TheEdoardo93]

You're trying to import a method only available in more recent Transformers versions from a (very old) Transformers version called _pytorch-transformers_.

With Transformers 2.1.1 (the second recent one) and the new version 2.2.0, you can import correctly the get_linear_schedule_with_warmup. In fact, Transformers modifies its source code for what concern the optimization process (e.g. learning rate). You can see the changes here.

Hello,

When I try to execute the line of code below, Python gives me an import error:

from pytorch_transformers import (GPT2Config, GPT2LMHeadModel, GPT2DoubleHeadsModel,
                                  AdamW, get_linear_schedule_with_warmup)

ImportError: cannot import name 'get_linear_schedule_with_warmup' from 'pytorch_transformers' (/Users/hyunjindominiquecho/opt/anaconda3/lib/python3.7/site-packages/pytorch_transformers/__init__.py)

What should I then import to use the linear scheduler with warm up?

Thank you,

[rlouf]

You should use the transformers library instead of the pytorch_transformers. The get_linear_schedule_with_warmup is only defined in the former, in its latest version.

[h56cho]

Thank you all,

[h56cho]

Hello,

So I installed transformers 2.2.0,

pip install transformers

and tried to import the same things:

from transformers import (GPT2Config, GPT2LMHeadModel, GPT2DoubleHeadsModel,
                                  AdamW, get_linear_schedule_with_warmup) 

and it's still giving me the same error:

from transformers import (GPT2Config, GPT2LMHeadModel, GPT2DoubleHeadsModel,
                                  AdamW, get_linear_schedule_with_warmup)
2019-11-27 08:40:15.940560: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-27 08:40:15.954925: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fe20d2f5a50 executing computations on platform Host. Devices:
2019-11-27 08:40:15.954938: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
Traceback (most recent call last):

  File "<ipython-input-1-99af21631e15>", line 1, in <module>
    from transformers import (GPT2Config, GPT2LMHeadModel, GPT2DoubleHeadsModel,

ImportError: cannot import name 'get_linear_schedule_with_warmup' from 'transformers' (/Users/hyunjindominiquecho/opt/anaconda3/lib/python3.7/site-packages/transformers/__init__.py)

What should I do to be able to use the linear warm up scheduler?

Thank you,

[TheEdoardo93]

It's very strange.. In my environment works as expected the import statement.

> import transformers
> transformers.__version__
>>> '2.2.0'
> from transformers import get_linear_schedule_with_warmup
> ...

Please, share with us your OS and your Python version.

Hello,

So I installed transformers 2.2.0,

pip install transformers

and tried to import the same things:

from transformers import (GPT2Config, GPT2LMHeadModel, GPT2DoubleHeadsModel,
                                  AdamW, get_linear_schedule_with_warmup) 

and it's still giving me the same error:

from transformers import (GPT2Config, GPT2LMHeadModel, GPT2DoubleHeadsModel,
                                  AdamW, get_linear_schedule_with_warmup)
2019-11-27 08:40:15.940560: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-27 08:40:15.954925: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fe20d2f5a50 executing computations on platform Host. Devices:
2019-11-27 08:40:15.954938: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
Traceback (most recent call last):

  File "<ipython-input-1-99af21631e15>", line 1, in <module>
    from transformers import (GPT2Config, GPT2LMHeadModel, GPT2DoubleHeadsModel,

ImportError: cannot import name 'get_linear_schedule_with_warmup' from 'transformers' (/Users/hyunjindominiquecho/opt/anaconda3/lib/python3.7/site-packages/transformers/__init__.py)

What should I do to be able to use the linear warm up scheduler?

Thank you,

[h56cho]

Hello,

I tried uninstalling transformers and install the module again and it works now.

Thank you for all your help,