Farm: can't make inference on conll03-en

[renaud]

I am trying to perform NER inference on conll03-en.

This is what I tried:

1. Train conll03-en from config, then perform inference with saved model

from farm.experiment import run_experiment, load_experiments
experiments = load_experiments('experiments/ner/conll2003_en_config.json')
run_experiment(experiments[0])

model = Inferencer.load('saved_models/CONLL2003')
basic_texts = [
    {"text": "Japan began the defence of their Asian Cup title with a lucky 2 win against Syria in a Group C championship match on Friday."},
]
result = model.inference_from_dicts(dicts=basic_texts)
print(result)

This does not return any predictions: [{'task': 'ner', 'predictions': []}]

1. I also try to rewrite the conll2003_en_config.json as pure python code, by adapting examples/ner.py, but no luck either

2. Also note that the NER evaluations work on both examples above, just not the inference.

3. Note also that examples/ner.py (German CONLL) worked fine for inference.

What am I missing?

Thanks

[renaud]

For the record, here is the rewrite of the conll2003_en_config.json as pure python code

import logging
from pathlib import Path

from farm.data_handler.data_silo import DataSilo
from farm.data_handler.processor import NERProcessor
from farm.modeling.optimization import initialize_optimizer
from farm.infer import Inferencer
from farm.modeling.adaptive_model import AdaptiveModel
from farm.modeling.language_model import LanguageModel
from farm.modeling.prediction_head import TokenClassificationHead
from farm.modeling.tokenization import Tokenizer
from farm.train import Trainer
from farm.utils import set_all_seeds, MLFlowLogger, initialize_device_settings

logging.basicConfig(
    format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
    datefmt="%m/%d/%Y %H:%M:%S",
    level=logging.INFO,
)

ml_logger = MLFlowLogger(tracking_uri="https://public-mlflow.deepset.ai/")
ml_logger.init_experiment(experiment_name="Public_FARM", run_name="Ren_en-ner-colb")

##########################
########## Settings
##########################
set_all_seeds(seed=42)
device, n_gpu = initialize_device_settings(use_cuda=True)#, local_rank=-1, use_amp=None)
n_epochs = 2
# TODO gradient_accumulation_steps?
# TODO layer_dims?
warmup_proportion = 0.4
batch_size = 64
evaluate_every = 60
lang_model =  "bert-base-cased"
do_lower_case = False

# 1.Create a tokenizer
tokenizer = Tokenizer.load(
    pretrained_model_name_or_path=lang_model, do_lower_case=do_lower_case
)

# 2. Create a DataProcessor that handles all the conversion from raw text into a pytorch Dataset
ner_labels = ["[PAD]", "X", "O", "B-MISC", "I-MISC", "B-PER", "I-PER", "B-ORG", "I-ORG", "B-LOC", "I-LOC", "B-OTH", "I-OTH"]

processor = NERProcessor(
    tokenizer=tokenizer, 
    max_seq_len=128, 
    data_dir=Path("../data/conll03-en"), 
    delimiter=" ", 
    metric="seq_f1", 
    dev_split=0.0,
    label_list=ner_labels
)

# 3. Create a DataSilo that loads several datasets (train/dev/test), provides DataLoaders for them and calculates a few descriptive statistics of our datasets
data_silo = DataSilo(processor=processor, batch_size=batch_size)

# 4. Create an AdaptiveModel
# a) which consists of a pretrained language model as a basis
language_model = LanguageModel.load(lang_model)
# b) and a prediction head on top that is suited for our task => NER
prediction_head = TokenClassificationHead(num_labels=len(ner_labels))

model = AdaptiveModel(
    language_model=language_model,
    prediction_heads=[prediction_head],
    embeds_dropout_prob=0.1,
    lm_output_types=["per_token"],
    device=device,
)

# 5. Create an optimizer
model, optimizer, lr_schedule = initialize_optimizer(
    model=model,
    learning_rate=5e-5,
    schedule_opts={"name": "LinearWarmup", "warmup_proportion": warmup_proportion},
    n_batches=len(data_silo.loaders["train"]),
    n_epochs=n_epochs,
    device=device,
)

# 6. Feed everything to the Trainer, which keeps care of growing our model into powerful plant and evaluates it from time to time
trainer = Trainer(
    model=model,
    optimizer=optimizer,
    data_silo=data_silo,
    epochs=n_epochs,
    n_gpu=n_gpu,
    lr_schedule=lr_schedule,
    evaluate_every=evaluate_every,
    device=device,
)

# 7. Let it grow
trainer.train()

# 8. Hooray! You have a model. Store it:
save_dir = "saved_models_bert-en-ner"
model.save(save_dir)
processor.save(save_dir)


# 9. Load it & harvest your fruits (Inference)
basic_texts = [
    {"text": "Japan began the defence of their Asian Cup title with a lucky 2 win against Syria in a Group C championship match on Friday."},
]
model = Inferencer.load(save_dir)
result = model.inference_from_dicts(dicts=basic_texts)
print(result)

[brandenchan]

Hi Renaud thanks very much for reporting this. I have fixed it with the above PR. Can you train a new model using the script you provided and let us know if it works?

[brandenchan]

I should just add that the issue was that the conll03-en wasn't being converted from IOB1 to IOB2 in our Processor class. We now also have checks now to ensure that this won't fail so silently in future

[renaud]

Thanks a lot @brandenchan, working fine now.

Here's a gist for the record: https://gist.github.com/renaud/2317b0c8e6d4ced7abd8088f5594c547