Allennlp: How to run my own decoding method in IDE and debug it?

[leo-liuzy]

I found it hard to debug my own MMI decoding method by running through allennlp in command line. I wonder if there is any way I can run things in debugger mode in PyCharm. Here are my decoding method. I simply change the register name of simple seq2seq and here are some

@Model.register("WebQABase")
class WebQABase(Model):

    def __init__(self,
                 vocab: Vocabulary,
                 source_embedder: TextFieldEmbedder,
                 encoder: Seq2SeqEncoder,
                 mmi: dict,
                 beam_width: int,
                 max_decoding_steps: int,
                 target_namespace: str = "tokens",
                 target_embedding_dim: int = None,
                 attention_function: SimilarityFunction = None,
                 scheduled_sampling_ratio: float = 0.0) -> None:
        # simple seq2seq settings with my modification
        .........
        self.mmi = mmi
        self.beam_width = beam_width

    @overrides
    def forward(self,  # type: ignore
                source_tokens: Dict[str, torch.LongTensor],
                target_tokens: Dict[str, torch.LongTensor] = None) -> Dict[str, torch.Tensor]:
        # running encoder
        ..........
        # self._decoder_cell = self._decoder_cell.cuda() if use_cuda else self._decoder_cell

        if target_tokens:
            targets = target_tokens["tokens"]
            target_sequence_length = targets.size(1)
            # The last input from the target is either padding or the end symbol. Either way, we
            # don't have to process it.
            num_decoding_steps = target_sequence_length - 1
        else:
            num_decoding_steps = self._max_decoding_steps
        beam_width = self.beam_width
        # in this decoding, we only use "folding" beam search

        if self.mmi["mode"] == "anti":
            beam = MMIBeam(batch_size=batch_size,
                           beam_width=beam_width,
                           end_index=self._end_index,
                           decoder_output_dim=self._decoder_output_dim,
                           lambd=self.mmi["lambda"])

            output_dict = self.mmi_antiLM(encoder_outputs=encoder_outputs,
                                          final_encoder_output=final_encoder_output,
                                          batch_size=batch_size,
                                          beam_width=beam_width,
                                          beam=beam,
                                          source_mask=get_text_field_mask(source_tokens),
                                          gamma=self.mmi["gamma"],
                                          max_decoding_step=num_decoding_steps)
        elif self.mmi["mode"] == "bidi":
            pass  # self.mmi_bidi(source_tokens, )
        else:
            assert False
        return output_dict
        .......

   @classmethod
    def from_params(cls, vocab, params: Params) -> 'SimpleSeq2Seq':
        ......seq2seq setting ...... 
        mmi = params.pop("mmi")
        beam_width = params.pop("beam_width")

        return cls(vocab,
                   source_embedder=source_embedder,
                   encoder=encoder,
                   mmi=mmi,
                   beam_width=beam_width,
                   max_decoding_steps=max_decoding_steps,
                   target_namespace=target_namespace,
                   attention_function=attention_function,
                   scheduled_sampling_ratio=scheduled_sampling_ratio)

Here are some changes I made to my config.json:

{
   ......normal setting ......
    "model": {
        "type": "WebQABase",
        "source_embedder": {
            "tokens": {
                "type": "embedding",
                "embedding_dim": 300,
                "vocab_namespace": "source_tokens",
                "trainable": true
            }
        },
        "encoder": {
            "type": "lstm",
            "input_size": 300,
            "hidden_size": 100,
            "num_layers": 3
        },
        "decoder": {
            "mode": "mmi"
        },
        "scheduled_sampling_ratio": 1.0,
        "max_decoding_steps": 200,
        "target_namespace": "target_tokens",
        "attention_function": {
            "type": "dot_product"
        },
        "mmi":{
            "mode": "anti",
            "gamma": 200,
            "lambda": 0.7
        },
        "beam_width": 200
    },
   ......normal setting ......
}

Then, I try to run my decoding method through a run.py:

import allennlp.commands.predict as predict
from allennlp.commands.predict import Predict
from WebQA.prediction.webQApredictor import WebQABaselinesPredictor

if __name__ == "__main__":
    # predict = WebQABaselinesPredictor()
    predict._run(WebQABaselinesPredictor, open('./prediction/experiment.jsonl', 'r'), open('./baseline+mmi.txt', 'w'),
                 batch_size=1,
                 print_to_console=True, cuda_device=-1)

However, I can't get things working due to my limited knowledge about allennlp.
Can you offer me some guidance?

[matt-gardner]

We're going to need a stack trace and a better description of the problem you're running into in order to help you.

[leo-liuzy]

Oh, I am just generally asking how to use a run.py to run my code in an IDE, since I can't use debugger when running with command line.

[matt-gardner]

I don't use pycharm, so I can't help you there unless you give me some more info, like a description of the actual problem that you're running into.

[nelson-liu]

since I can't use debugger when running with command line.

Does it not work to use something like pdb?

[leo-liuzy]

@matt-gardner I guess what I expect from pycharm is that I can see whether my algorithm is doing the right thing, and check if I misunderstand the usage of PyTorch functions and tensors when realizing the decoding method. For example, it may be caused by how I train my model that the decoded sequence contains lots of tokens like "it, yes, .[period]", but I can't be sure without assuring my implementation is correct.

@nelson-liu Thanks! I will look into it!

[joelgrus]

if you look in allennlp/commands/train.py there is a train_model_from_file function that is basically what the command line tool calls.

https://github.com/allenai/allennlp/blob/master/allennlp/commands/train.py#L102

if you write a script that calls that function, you can run it in your IDE. I think that's what you're looking for?

[leo-liuzy]

Yes! And I can run just my decoding method with that method right? It does seems that function won't do stuffs like loading and indexing dataset. Just double check, as the naming is a little confusing. I only need to rewrite the decoding.

[joelgrus]

that's the method for training a model. if you have a model that's already trained, you can make predictions programmatically as well, see

https://github.com/allenai/allennlp/blob/master/tutorials/getting_started/using_pretrained_models.md#making-predictions-programatically

[leo-liuzy]

But, is there any way to run prediction through an IDE, so that I can use debugger to assure my code is correct?

[kearnsw]

@LeoLiu1998 , the link Joel posted worked for me by placing a breakpoint at the predict method.

[joelgrus]

ok, here is a simpler way that I have tested to work in pycharm.

basically the command line command allennlp ... does the exact same thing as python -m allennlp.run ...

so if you replace your command

allennlp train ...

with

python -m allennlp.run train ...

then in pycharm you can do Run -> Attach to Local Process and find that process and that works.

And that should work with any of the commands.