Bert: module 'tokenization' has no attribute 'FullTokenizer'

[1337-Pete]

I'm importing tokenization, have installed via pip, and cannot instantiate the tokenizer. I'm using the following code below and continue to get an error message of "module 'tokenization' has no attribute 'FullTokenizer'".

Anyone have a sense as to why?

tokenizer = tokenization.FullTokenizer(vocab_file=vocab_file, do_lower_case=do_lower_case)

[nicsuzor]

Looks the same as my code; what’s your import statement?

On Sun, 12 May 2019 at 9:52 am, phum126 [e-mail] wrote:

I'm importing tokenization, have installed via pip, and cannot instantiate
the tokenizer. I'm using the following code below and continue to get an
error message of "module 'tokenization' has no attribute 'FullTokenizer'".

Anyone have a sense as to why?

tokenizer = tokenization.FullTokenizer(vocab_file=vocab_file,
do_lower_case=do_lower_case)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/google-research/bert/issues/638, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABCJ4URV5LIW6NULDFKAFQDPU5L5PANCNFSM4HMJGSPA
.

>

Sent from a mobile device.

[1337-Pete]

import tokenization

[nicsuzor]

If you installed from pip (I'm presuming pip install bert-tensorflow), try:
from bert import tokenization

[1337-Pete]

If you installed from pip (I'm presuming pip install bert-tensorflow), try:
from bert import tokenization

Yes! Working now :) Thanks a million.

[Ceceu]

@nicsuzor and @phum126, I getting the same error using bert-for-tf2:

from bert import tokenization

...

tokenizer = tokenization.FullTokenizer(
        vocab_file="resources/model/bert/bert_vocab.txt", 
        do_lower_case=True
    )

...

output:
AttributeError: module 'bert.tokenization' has no attribute 'FullTokenizer'

Could you give me some directions?

[Ceceu]

Maybe it can help someone using Tensorflow 2 and bert-for-tf2. There was a little change to create an instance of FullTokenizer:

from bert import bert_tokenization

...

tokenizer = bert_tokenization.FullTokenizer(
        vocab_file="resources/model/bert/bert_vocab.txt", 
        do_lower_case=True
    )

[gittinjiggywithit]

I've been struggling with this problem too for the last couple of hours.
I'm running tensorflow 2.0, python 3.7.4 and installed bert-for-tf2 using pip3 install.
What I found out was, that for some reason I was missing a lot of files in the folder where bert was installed, when compared to my friend who also installed it using pip3. For instance the "__init__.py" file was missing. Then I:

• Uninstalled bert using "pip3 uninstall bert-for-tf2"

• Deleted the bert folder, which apparently wasn't removed during the uninstall. In my case, the folder was located at "/usr/local/lib/python3.7/site-packages/bert"

• Reinstalled using "pip3 install bert-for-tf2"

After reinstalling, I now have the following files in my bert directory: