Parlai: Failed to Reproduce Human Evaluation of Controllable Dialogue with Pretrained Model

Created on 19 Jul 2020  路  8Comments  路  Source: facebookresearch/ParlAI

Bug description
I failed to reproduce the human evaluation of the controllable dialogue with pre-trained model. The dialogue log in evaluation_logs/ folder.

Reproduction steps
For the first dialogue in evaluation_logs/baseline_model.jsonl
MetaData

  • model_persona: "i am terrified of clowns due to childhood trauma.ni enjoy working with my hands and love gardening.ni just got a job working in a gas station.ni was adopted at a young age to an older couple.nmy father keeps trying to set me up on blind dates."
  • dialog:[{"speaker":"model","text":"Hello, how are you today?"},{"speaker":"human_evaluator","text":"hello! im doing well. how are you"},{"speaker":"model","text":"I am well, thank you. What do you do for a living?"},{"speaker":"human_evaluator","text":"i work part time at burger king making minimum wage. what about you"},{"speaker":"model","text":"I work in a gas station."},{"speaker":"human_evaluator","text":"what do you do for fun"},{"speaker":"model","text":"I work in a gas station. What about you?"},{"speaker":"human_evaluator","text":"i work at burger king"},{"speaker":"model","text":"What do you do for a living?"},{"speaker":"human_evaluator","text":"i live with my parents but i work part time in fast food"},{"speaker":"model","text":"What city are you from?"},{"speaker":"human_evaluator","text":"i am from boston. where are you"}]

Step 1. Open interactive chat with baseline agent.

python projects/controllable_dialogue/interactive.py \
-mf models:controllable_dialogue/convai2_finetuned_baseline

Step 2.
Enter your Message: your persona: i am terrified of clowns due to childhood trauma.\nyour persona: i enjoy working with my hands and love gardening.\nyour persona: i just got a job working in a gas station.\nyour persona: i was adopted at a young age to an older couple.\nyour persona: my father keeps trying to set me up on blind dates.\n__SILENCE__
Step 3.
Get agent response: Hi how are you doing?. Which does not match the evaluation log Hello, how are you today?

For all the 111 dialogues in evaluation_logs/baseline_model.jsonl, I can only reproduce 44 of them.

Expected behavior
When the pre-trained model is given the same model persona and human response, I expect to reproduce the dialogue in evaluation_logs.

Logs
Please paste the command line output:

The command line output is slightly different from the original script, where I compare the model output with the evaluation_logs/baseline_model.jsonl.

The first dialogue in evaluation_logs/baseline_model.jsonl:

{'text': 'your persona: i am terrified of clowns due to childhood trauma.\nyour persona: i enjoy workingwith my hands and love gardening.\nyour persona: i just got a job working in a gas station.\nyour persona: i was adopted at a young age to an older couple.\nyour persona: my father keeps trying to set me up onblind dates.\n__SILENCE__', 'episode_done': False}
[PERSON_2]: Hi how are you doing?
[REFDATA]: Hello, how are you today?  
Warning! Mismatched
[Human]: hello! im doing well. how are you  
[PERSON_2]: I am good thanks for asking
[REFDATA]: I am well, thank you. What do you do for a living?  
Warning! Mismatched
[Human]: i work part time at burger king making minimum wage. what about you  
[PERSON_2]: I work in a gas station
[REFDATA]: I work in a gas station. 
Warning! Mismatched
[Human]: what do you do for fun
[PERSON_2]: I work at a gas station
[REFDATA]: I work in a gas station. What about you? 
Warning! Mismatched
[Human]: i work at burger king  
[PERSON_2]: What do you do for a living
[REFDATA]: What do you do for a living?  
Warning! Mismatched
[Human]: i live with my parents but i work part time in fast food 
[PERSON_2]: What city are you from?
[REFDATA]: What city are you from?

I can reproduce the second dialogue.

The third dialogue in evaluation_logs/baseline_model.jsonl:

{'text': 'your persona: i am a professional wrestler.\nyour persona: i enjoy being around people.\nyour persona: i like to workout a a times a week.\nyour persona: i volunteer in a homeless shelter.\nyour persona: in my spare time i do volunteer work.\n__SILENCE__', 'episode_done': False}
[PERSON_2]: Hello, how are you today?
[REFDATA]: Hi how are you today  
Warning! Mismatched
[Human]: im good and how are you  
[PERSON_2]: I am doing well. Just got back from a long workout.
[REFDATA]: I am good do you work  
Warning! Mismatched
[Human]: yes i do work and what about you  
[PERSON_2]: I am a professional wrestler.
[REFDATA]: I am a professional wrestler  
Warning! Mismatched
[Human]: thats cool i work online  
[PERSON_2]: What do you do online?
[REFDATA]: What do you do online  
Warning! Mismatched
[Human]: i work for a company to answer phones   
[PERSON_2]: That sounds like a lot of fun.
[REFDATA]: What do you do online  
Warning! Mismatched
[Human]: i  just told you  
[PERSON_2]: What do you do online?
[REFDATA]: What do you do for a living  
Warning! Mismatched

Additional context
I tried to roll back to different code-base versions and got no luck.

picture HMJiangGatech

All 8 comments

Hmm so I think it must be some sort of mismatch in something, because we have a test that verifies these pretrained models don't deviate over time. That hasn't fired, so I'm reasonably confident that it isn't a new bug, but some mismatch between the settings at the time we collected those longs, and these experiments.

One thing that stands out to me is that in MTurk we had some normalization of punctuation and casing on both humans and models to remove surface level discrimination.

Normalizing the model output:
https://github.com/facebookresearch/ParlAI/blob/c5bd36d065532695d31b3f4a76cff21347a100ca/projects/controllable_dialogue/mturk/worlds.py#L405
https://github.com/facebookresearch/ParlAI/blob/c5bd36d065532695d31b3f4a76cff21347a100ca/projects/controllable_dialogue/mturk/worlds.py#L474

Normalizing the human:
https://github.com/facebookresearch/ParlAI/blob/c5bd36d065532695d31b3f4a76cff21347a100ca/projects/controllable_dialogue/mturk/worlds.py#L422
https://github.com/facebookresearch/ParlAI/blob/c5bd36d065532695d31b3f4a76cff21347a100ca/projects/controllable_dialogue/mturk/worlds.py#L466
https://github.com/facebookresearch/ParlAI/blob/c5bd36d065532695d31b3f4a76cff21347a100ca/projects/controllable_dialogue/mturk/worlds.py#L491

However, the model still observes its own responses non-normalized.

One problem with this is that the model still seems to have mismatches on the very first turn. I'm taking at look at this.

stephenroller picture stephenroller on 19 Jul 2020

Okay I found it. We have a postprocessing script that took the raw dumps from the MTurk runs and put them in a readable and standardized format. In that script, it had roughly personas = sorted(personas). This was useful because it helped us normalize the personas in some of the ACUTE-Eval experiments. However, it made the logs not reproducible (since the order of the personas matters to the model).

I've reprocessed them without sorting them, and will upload those new logs shortly, as well as the verification script I wrote.

stephenroller picture stephenroller on 19 Jul 2020

@stephenroller Thanks for the prompt reply. Looking forward to the new logs!

IMHO, reproducibility of these logs is very beneficial to the whole research community. As far as I know, this is the only open-source dialog project that did comprehensive human evaluation and also released both the evaluation logs and pre-trained models. I really appreciate it!

HMJiangGatech picture HMJiangGatech on 19 Jul 2020
👍1

I really appreciate you pointing out the bug and posting a thorough bug report.

stephenroller picture stephenroller on 19 Jul 2020

Fix in #2856

verify_control.py:

#!/usr/bin/env python3

import json

from parlai.core.agents import create_agent
from parlai.scripts.script import ParlaiScript
from parlai.core.params import ParlaiParser
from parlai.utils.strings import normalize_reply
import parlai.utils.logging as logging

SILENCE = '__SILENCE__'
PERSONA_PREFIX = "your persona: "


def setup_args():
    parser = ParlaiParser(True, True)
    parser.add_argument('--eval-file', required=True)
    return parser


def _run_conversation(conversation_id, conversation, agent):
    agent.reset()
    model_persona = conversation['model_persona']
    model_persona = PERSONA_PREFIX + model_persona.replace("\n", "\n" + PERSONA_PREFIX)

    for turn_id, turn in enumerate(conversation['dialog']):
        speaker = turn['speaker']
        reference_text = turn['text']

        if turn_id == 0 and speaker == 'model':
            silenced_text = model_persona + '\n' + SILENCE
            observed = agent.observe(
                {'id': 'SPEAKER_2', 'text': silenced_text, 'episode_done': False}
            )
        elif turn_id == 0 and speaker == 'human_evaluator':
            reference_text = model_persona + '\n' + reference_text

        if speaker == 'human_evaluator':
            observed = agent.observe(
                {'id': 'SPEAKER_2', 'text': reference_text, 'episode_done': False}
            )
        if speaker == 'model':
            response = agent.act()
            response_normalized = normalize_reply(response['text'])
            if response_normalized != reference_text:
                logging.error(
                    f'{conversation_id}:{turn_id}: ref {repr(reference_text)} '
                    f'!= resp {repr(response_normalized)}. Context:\n{repr(observed)}'
                )
            else:
                logging.info(f'{conversation_id}:{turn_id} OK')


def main():
    parser = setup_args()
    opt = parser.parse_args()
    agent = create_agent(opt, True)
    parser.opt = agent.opt
    parser.print_args()

    with open(opt['eval_file']) as eval_file:
        for i, line in enumerate(eval_file):
            if not line.strip():
                continue
            conversation = json.loads(line)
            _run_conversation(i, conversation, agent)


if __name__ == '__main__':
    main()

Example usage:
python verify_control.py -mf zoo:controllable_dialogue/convai2_finetuned_baseline --beam-size 1 --batchsize 1 --eval-file data/controllable_dialogue/evaluation_logs_reproducible/greedy_model.jsonl --beam-size 1 --beam-min-n-best 10

Output from example usage:

[ optional arguments: ] 
[  eval_file: data/controllable_dialogue/evaluation_logs_reproducible/greedy_model.jsonl ]
[ Main ParlAI Arguments: ] 
[  batchsize: 1 ]
[  datapath: /private/home/roller/working/parlai/data ]
[  datatype: train ]
[  download_path: /private/home/roller/working/parlai/downloads ]
[  dynamic_batching: None ]
[  hide_labels: False ]
[  image_mode: raw ]
[  init_opt: None ]
[  loglevel: info ]
[  multitask_weights: [1] ]
[  numthreads: 1 ]
[  show_advanced_args: False ]
[  task: None ]
[ ParlAI Model Arguments: ] 
[  dict_class: parlai.core.dict:DictionaryAgent ]
[  init_model: None ]
[  model: None ]
[  model_file: /private/home/roller/working/parlai/data/models/controllable_dialogue/convai2_finetuned_baseline ]
[ ParlAI Image Preprocessing Arguments: ] 
[  image_cropsize: 224 ]
[  image_size: 256 ]
[ ControllableSeq2seqAgent Arguments: ] 
[  add_control: False ]
[  attention: none ]
[  attention_length: 48 ]
[  attention_time: post ]
[  beam_block_ngram: 0 ]
[  beam_dot_log: False ]
[  beam_min_length: 3 ]
[  beam_min_n_best: 10 ]
[  beam_reorder: none ]
[  beam_size: 1 ]
[  bidirectional: False ]
[  control_embeddingsize:  ]
[  control_num_buckets:  ]
[  control_vars:  ]
[  decoder: same ]
[  dropout: 0.1 ]
[  embeddingsize: 128 ]
[  hiddensize: 128 ]
[  init_model: None ]
[  input_dropout: 0.0 ]
[  lookuptable: unique ]
[  numlayers: 2 ]
[  numsoftmax: 1 ]
[  rnn_class: lstm ]
[  set_controls:  ]
[  verbose: False ]
[  weighted_decoding:  ]
[ TorchAgent Arguments: ] 
[  add_p1_after_newln: False ]
[  delimiter: 
 ]
[  embedding_projection: random ]
[  embedding_type: random ]
[  force_fp16_tokens: False ]
[  fp16: False ]
[  fp16_impl: apex ]
[  gpu: -1 ]
[  history_add_global_end_token: None ]
[  history_size: -1 ]
[  interactive_mode: False ]
[  label_truncate: None ]
[  no_cuda: False ]
[  person_tokens: False ]
[  rank_candidates: False ]
[  special_tok_lst: None ]
[  split_lines: False ]
[  text_truncate: None ]
[  truncate: -1 ]
[  use_reply: label ]
[ Optimizer Arguments: ] 
[  adafactor_eps: (1e-30, 0.001) ]
[  adam_eps: 1e-08 ]
[  betas: (0.9, 0.999) ]
[  gradient_clip: 0.1 ]
[  learningrate: 1 ]
[  momentum: 0 ]
[  nesterov: True ]
[  nus: (0.7,) ]
[  optimizer: sgd ]
[  weight_decay: None ]
[ Learning Rate Scheduler: ] 
[  invsqrt_lr_decay_gamma: -1 ]
[  lr_scheduler: reduceonplateau ]
[  lr_scheduler_decay: 0.5 ]
[  lr_scheduler_patience: 3 ]
[  max_lr_steps: -1 ]
[  update_freq: 1 ]
[  warmup_rate: 0.0001 ]
[  warmup_updates: -1 ]
[ Dictionary Arguments: ] 
[  bpe_debug: False ]
[  dict_endtoken: __end__ ]
[  dict_file: None ]
[  dict_initpath: None ]
[  dict_language: english ]
[  dict_lower: False ]
[  dict_max_ngram_size: -1 ]
[  dict_maxtokens: -1 ]
[  dict_minfreq: 0 ]
[  dict_nulltoken: __null__ ]
[  dict_starttoken: __start__ ]
[  dict_textfields: text,labels ]
[  dict_tokenizer: re ]
[  dict_unktoken: __unk__ ]
[ BPEHelper Arguments: ] 
[  bpe_add_prefix_space: None ]
[  bpe_merge: None ]
[  bpe_vocab: None ]
[  hf_skip_special_tokens: True ]
2020-07-19 22:02:07,309 INFO     | Current ParlAI commit: 11860b84be651b9e9335686e99cf24d3f31775ab
2020-07-19 22:02:07,319 INFO     | Current internal commit: fbc7ff58d751608dcd4247e398421a464e282075
2020-07-19 22:02:07,321 WARNING  | Overriding opt["model_file"] to /private/home/roller/working/parlai/data/models/controllable_dialogue/convai2_finetuned_baseline (previously: /private/home/abisee/models/baseline)
2020-07-19 22:02:07,321 WARNING  | Overriding opt["batchsize"] to 1 (previously: 64)
Loading up controllable features...
Loading word count stats from /private/home/roller/working/parlai/data/controllable_dialogue/word2count.pkl...
num_sents:  5328706
Done loading word2nidf dictionary.
Loading Arora embedding info from /private/home/roller/working/parlai/data/controllable_dialogue/arora.pkl...
Done loading arora info.
2020-07-19 22:02:08,521 INFO     | Using CUDA
2020-07-19 22:02:08,522 INFO     | loading dictionary from /private/home/roller/working/parlai/data/models/controllable_dialogue/convai2_finetuned_baseline.dict
2020-07-19 22:02:08,616 INFO     | num words = 35213
[ Loading existing model params from /private/home/roller/working/parlai/data/models/controllable_dialogue/convai2_finetuned_baseline ]
[ optional arguments: ] 
[  eval_file: data/controllable_dialogue/evaluation_logs_reproducible/greedy_model.jsonl ]
[ Main ParlAI Arguments: ] 
[  batchsize: 1 ]
[  datapath: /private/home/roller/working/parlai/data ]
[  datatype: train ]
[  download_path: /private/home/roller/working/parlai/downloads ]
[  dynamic_batching: None ]
[  hide_labels: False ]
[  image_mode: raw ]
[  init_opt: None ]
[  loglevel: info ]
[  multitask_weights: [1] ]
[  numthreads: 1 ]
[  show_advanced_args: False ]
[  task: fromfile:parlaiformat ]
[ ParlAI Model Arguments: ] 
[  dict_class: parlai.core.dict:DictionaryAgent ]
[  init_model: /private/home/abisee/models/seq2seq_twitterpretrained_baseline ]
[  model: projects.controllable_dialogue.controllable_seq2seq.controllable_seq2seq:ControllableSeq2seqAgent ]
[  model_file: /private/home/roller/working/parlai/data/models/controllable_dialogue/convai2_finetuned_baseline ]
[ ParlAI Image Preprocessing Arguments: ] 
[  image_cropsize: 224 ]
[  image_size: 256 ]
[ ControllableSeq2seqAgent Arguments: ] 
[  add_control: False ]
[  attention: general ]
[  attention_length: 48 ]
[  attention_time: post ]
[  beam_block_ngram: 0 ]
[  beam_dot_log: False ]
[  beam_min_length: 3 ]
[  beam_min_n_best: 10 ]
[  beam_reorder: False ]
[  beam_size: 1 ]
[  bidirectional: False ]
[  control_embeddingsize:  ]
[  control_num_buckets:  ]
[  control_vars:  ]
[  decoder: same ]
[  dropout: 0.1 ]
[  embeddingsize: 300 ]
[  hiddensize: 1024 ]
[  init_model: /private/home/abisee/models/seq2seq_twitterpretrained_baseline ]
[  input_dropout: 0.0 ]
[  lookuptable: enc_dec ]
[  numlayers: 2 ]
[  numsoftmax: 1 ]
[  rnn_class: lstm ]
[  set_controls:  ]
[  verbose: False ]
[  weighted_decoding:  ]
[ TorchAgent Arguments: ] 
[  add_p1_after_newln: True ]
[  delimiter: 
 ]
[  embedding_projection: random ]
[  embedding_type: glove ]
[  force_fp16_tokens: False ]
[  fp16: False ]
[  fp16_impl: apex ]
[  gpu: -1 ]
[  history_add_global_end_token: None ]
[  history_size: -1 ]
[  interactive_mode: False ]
[  label_truncate: None ]
[  no_cuda: False ]
[  person_tokens: True ]
[  rank_candidates: False ]
[  special_tok_lst: None ]
[  split_lines: False ]
[  text_truncate: None ]
[  truncate: -1 ]
[  use_reply: label ]
[ Optimizer Arguments: ] 
[  adafactor_eps: (1e-30, 0.001) ]
[  adam_eps: 1e-08 ]
[  betas: (0.9, 0.999) ]
[  gradient_clip: 0.1 ]
[  learningrate: 3 ]
[  momentum: 0.9 ]
[  nesterov: True ]
[  nus: (0.7,) ]
[  optimizer: sgd ]
[  weight_decay: None ]
[ Learning Rate Scheduler: ] 
[  invsqrt_lr_decay_gamma: -1 ]
[  lr_scheduler: reduceonplateau ]
[  lr_scheduler_decay: 0.5 ]
[  lr_scheduler_patience: 3 ]
[  max_lr_steps: -1 ]
[  update_freq: 1 ]
[  warmup_rate: 0.0001 ]
[  warmup_updates: -1 ]
[ Dictionary Arguments: ] 
[  bpe_debug: False ]
[  dict_endtoken: __end__ ]
[  dict_file: /private/home/roller/working/parlai/data/models/controllable_dialogue/convai2_finetuned_baseline.dict ]
[  dict_initpath: None ]
[  dict_language: english ]
[  dict_lower: True ]
[  dict_max_ngram_size: -1 ]
[  dict_maxtokens: -1 ]
[  dict_minfreq: 0 ]
[  dict_nulltoken: __null__ ]
[  dict_starttoken: __start__ ]
[  dict_textfields: text,labels ]
[  dict_tokenizer: split ]
[  dict_unktoken: __unk__ ]
[ BPEHelper Arguments: ] 
[  bpe_add_prefix_space: None ]
[  bpe_merge: None ]
[  bpe_vocab: None ]
[  hf_skip_special_tokens: True ]
2020-07-19 22:02:12,649 INFO     | 0:0 OK
2020-07-19 22:02:12,823 INFO     | 0:2 OK
2020-07-19 22:02:12,988 INFO     | 0:4 OK
2020-07-19 22:02:13,155 INFO     | 0:6 OK
2020-07-19 22:02:13,326 INFO     | 0:8 OK
2020-07-19 22:02:13,498 INFO     | 0:10 OK
2020-07-19 22:02:13,658 INFO     | 1:0 OK
2020-07-19 22:02:13,819 INFO     | 1:2 OK
2020-07-19 22:02:13,981 INFO     | 1:4 OK
2020-07-19 22:02:14,147 INFO     | 1:6 OK
2020-07-19 22:02:14,314 INFO     | 1:8 OK
2020-07-19 22:02:14,483 INFO     | 1:10 OK
2020-07-19 22:02:14,642 INFO     | 2:0 OK
2020-07-19 22:02:14,803 INFO     | 2:2 OK
2020-07-19 22:02:14,965 INFO     | 2:4 OK
2020-07-19 22:02:15,129 INFO     | 2:6 OK
2020-07-19 22:02:15,295 INFO     | 2:8 OK
2020-07-19 22:02:15,462 INFO     | 2:10 OK
2020-07-19 22:02:15,621 INFO     | 3:1 OK
2020-07-19 22:02:15,781 INFO     | 3:3 OK
2020-07-19 22:02:15,944 INFO     | 3:5 OK
2020-07-19 22:02:16,110 INFO     | 3:7 OK
2020-07-19 22:02:16,278 INFO     | 3:9 OK
2020-07-19 22:02:16,448 INFO     | 3:11 OK
2020-07-19 22:02:16,607 INFO     | 4:1 OK
2020-07-19 22:02:16,768 INFO     | 4:3 OK
2020-07-19 22:02:16,931 INFO     | 4:5 OK
2020-07-19 22:02:17,097 INFO     | 4:7 OK
2020-07-19 22:02:17,265 INFO     | 4:9 OK
2020-07-19 22:02:17,435 INFO     | 4:11 OK
2020-07-19 22:02:17,594 INFO     | 5:0 OK
2020-07-19 22:02:17,756 INFO     | 5:2 OK
2020-07-19 22:02:17,919 INFO     | 5:4 OK
2020-07-19 22:02:18,085 INFO     | 5:6 OK
2020-07-19 22:02:18,252 INFO     | 5:8 OK
2020-07-19 22:02:18,424 INFO     | 5:10 OK
2020-07-19 22:02:18,583 INFO     | 6:1 OK
2020-07-19 22:02:18,744 INFO     | 6:3 OK
2020-07-19 22:02:18,909 INFO     | 6:5 OK
2020-07-19 22:02:19,078 INFO     | 6:7 OK
2020-07-19 22:02:19,249 INFO     | 6:9 OK
2020-07-19 22:02:19,423 INFO     | 6:11 OK
2020-07-19 22:02:19,582 INFO     | 7:0 OK
2020-07-19 22:02:19,743 INFO     | 7:2 OK
2020-07-19 22:02:19,908 INFO     | 7:4 OK
2020-07-19 22:02:20,072 INFO     | 7:6 OK
2020-07-19 22:02:20,238 INFO     | 7:8 OK
2020-07-19 22:02:20,405 INFO     | 7:10 OK
2020-07-19 22:02:20,566 INFO     | 8:1 OK
2020-07-19 22:02:20,728 INFO     | 8:3 OK
2020-07-19 22:02:20,894 INFO     | 8:5 OK
2020-07-19 22:02:21,061 INFO     | 8:7 OK
2020-07-19 22:02:21,230 INFO     | 8:9 OK
2020-07-19 22:02:21,403 INFO     | 8:11 OK
2020-07-19 22:02:21,561 INFO     | 9:1 OK
2020-07-19 22:02:21,720 INFO     | 9:3 OK
2020-07-19 22:02:21,882 INFO     | 9:5 OK
2020-07-19 22:02:22,047 INFO     | 9:7 OK
2020-07-19 22:02:22,214 INFO     | 9:9 OK
2020-07-19 22:02:22,382 INFO     | 9:11 OK
2020-07-19 22:02:22,543 INFO     | 10:1 OK
2020-07-19 22:02:22,705 INFO     | 10:3 OK
2020-07-19 22:02:22,868 INFO     | 10:5 OK
2020-07-19 22:02:23,035 INFO     | 10:7 OK
2020-07-19 22:02:23,205 INFO     | 10:9 OK
2020-07-19 22:02:23,376 INFO     | 10:11 OK
2020-07-19 22:02:23,534 INFO     | 11:0 OK
2020-07-19 22:02:23,695 INFO     | 11:2 OK
2020-07-19 22:02:23,856 INFO     | 11:4 OK
2020-07-19 22:02:24,019 INFO     | 11:6 OK
2020-07-19 22:02:24,184 INFO     | 11:8 OK
2020-07-19 22:02:24,350 INFO     | 11:10 OK
2020-07-19 22:02:24,507 INFO     | 12:0 OK
2020-07-19 22:02:24,666 INFO     | 12:2 OK
2020-07-19 22:02:24,826 INFO     | 12:4 OK
2020-07-19 22:02:24,987 INFO     | 12:6 OK
2020-07-19 22:02:25,152 INFO     | 12:8 OK
2020-07-19 22:02:25,318 INFO     | 12:10 OK
2020-07-19 22:02:25,476 INFO     | 13:0 OK
2020-07-19 22:02:25,636 INFO     | 13:2 OK
2020-07-19 22:02:25,798 INFO     | 13:4 OK
2020-07-19 22:02:25,962 INFO     | 13:6 OK
2020-07-19 22:02:26,129 INFO     | 13:8 OK
2020-07-19 22:02:26,297 INFO     | 13:10 OK
2020-07-19 22:02:26,455 INFO     | 14:0 OK
2020-07-19 22:02:26,614 INFO     | 14:2 OK
2020-07-19 22:02:26,775 INFO     | 14:4 OK
2020-07-19 22:02:26,938 INFO     | 14:6 OK
2020-07-19 22:02:27,104 INFO     | 14:8 OK
2020-07-19 22:02:27,271 INFO     | 14:10 OK
2020-07-19 22:02:27,429 INFO     | 15:1 OK
2020-07-19 22:02:27,592 INFO     | 15:3 OK
2020-07-19 22:02:27,758 INFO     | 15:5 OK
2020-07-19 22:02:27,926 INFO     | 15:7 OK
2020-07-19 22:02:28,098 INFO     | 15:9 OK
2020-07-19 22:02:28,272 INFO     | 15:11 OK
2020-07-19 22:02:28,430 INFO     | 16:1 OK
2020-07-19 22:02:28,591 INFO     | 16:3 OK
2020-07-19 22:02:28,755 INFO     | 16:5 OK
2020-07-19 22:02:28,923 INFO     | 16:7 OK
2020-07-19 22:02:29,092 INFO     | 16:9 OK
2020-07-19 22:02:29,263 INFO     | 16:11 OK
2020-07-19 22:02:29,424 INFO     | 17:0 OK
2020-07-19 22:02:29,586 INFO     | 17:2 OK
2020-07-19 22:02:29,750 INFO     | 17:4 OK
2020-07-19 22:02:29,916 INFO     | 17:6 OK
2020-07-19 22:02:30,084 INFO     | 17:8 OK
2020-07-19 22:02:30,255 INFO     | 17:10 OK
2020-07-19 22:02:30,414 INFO     | 18:1 OK
2020-07-19 22:02:30,574 INFO     | 18:3 OK
2020-07-19 22:02:30,737 INFO     | 18:5 OK
2020-07-19 22:02:30,902 INFO     | 18:7 OK
2020-07-19 22:02:31,070 INFO     | 18:9 OK
2020-07-19 22:02:31,239 INFO     | 18:11 OK
2020-07-19 22:02:31,399 INFO     | 19:1 OK
2020-07-19 22:02:31,561 INFO     | 19:3 OK
2020-07-19 22:02:31,726 INFO     | 19:5 OK
2020-07-19 22:02:31,893 INFO     | 19:7 OK
2020-07-19 22:02:32,062 INFO     | 19:9 OK
2020-07-19 22:02:32,234 INFO     | 19:11 OK
2020-07-19 22:02:32,394 INFO     | 20:1 OK
2020-07-19 22:02:32,556 INFO     | 20:3 OK
2020-07-19 22:02:32,721 INFO     | 20:5 OK
2020-07-19 22:02:32,889 INFO     | 20:7 OK
2020-07-19 22:02:33,058 INFO     | 20:9 OK
2020-07-19 22:02:33,231 INFO     | 20:11 OK
2020-07-19 22:02:33,390 INFO     | 21:1 OK
2020-07-19 22:02:33,551 INFO     | 21:3 OK
2020-07-19 22:02:33,714 INFO     | 21:5 OK
2020-07-19 22:02:33,879 INFO     | 21:7 OK
2020-07-19 22:02:34,045 INFO     | 21:9 OK
2020-07-19 22:02:34,213 INFO     | 21:11 OK
2020-07-19 22:02:34,372 INFO     | 22:1 OK
2020-07-19 22:02:34,533 INFO     | 22:3 OK
2020-07-19 22:02:34,696 INFO     | 22:5 OK
2020-07-19 22:02:34,863 INFO     | 22:7 OK
2020-07-19 22:02:35,034 INFO     | 22:9 OK
2020-07-19 22:02:35,206 INFO     | 22:11 OK
2020-07-19 22:02:35,365 INFO     | 23:1 OK
2020-07-19 22:02:35,527 INFO     | 23:3 OK
2020-07-19 22:02:35,693 INFO     | 23:5 OK
2020-07-19 22:02:35,861 INFO     | 23:7 OK
2020-07-19 22:02:36,031 INFO     | 23:9 OK
2020-07-19 22:02:36,203 INFO     | 23:11 OK
2020-07-19 22:02:36,362 INFO     | 24:0 OK
2020-07-19 22:02:36,523 INFO     | 24:2 OK
2020-07-19 22:02:36,686 INFO     | 24:4 OK
2020-07-19 22:02:36,851 INFO     | 24:6 OK
2020-07-19 22:02:37,018 INFO     | 24:8 OK
2020-07-19 22:02:37,187 INFO     | 24:10 OK
2020-07-19 22:02:37,346 INFO     | 25:0 OK
2020-07-19 22:02:37,506 INFO     | 25:2 OK
2020-07-19 22:02:37,668 INFO     | 25:4 OK
2020-07-19 22:02:37,833 INFO     | 25:6 OK
2020-07-19 22:02:38,000 INFO     | 25:8 OK
2020-07-19 22:02:38,170 INFO     | 25:10 OK
2020-07-19 22:02:38,329 INFO     | 26:1 OK
2020-07-19 22:02:38,489 INFO     | 26:3 OK
2020-07-19 22:02:38,651 INFO     | 26:5 OK
2020-07-19 22:02:38,815 INFO     | 26:7 OK
2020-07-19 22:02:38,981 INFO     | 26:9 OK
2020-07-19 22:02:39,149 INFO     | 26:11 OK
2020-07-19 22:02:39,307 INFO     | 27:1 OK
2020-07-19 22:02:39,470 INFO     | 27:3 OK
2020-07-19 22:02:39,634 INFO     | 27:5 OK
2020-07-19 22:02:39,800 INFO     | 27:7 OK
2020-07-19 22:02:39,968 INFO     | 27:9 OK
2020-07-19 22:02:40,144 INFO     | 27:11 OK
2020-07-19 22:02:40,310 INFO     | 28:1 OK
2020-07-19 22:02:40,479 INFO     | 28:3 OK
2020-07-19 22:02:40,648 INFO     | 28:5 OK
2020-07-19 22:02:40,825 INFO     | 28:7 OK
2020-07-19 22:02:40,993 INFO     | 28:9 OK
2020-07-19 22:02:41,164 INFO     | 28:11 OK
2020-07-19 22:02:41,323 INFO     | 29:0 OK
2020-07-19 22:02:41,482 INFO     | 29:2 OK
2020-07-19 22:02:41,643 INFO     | 29:4 OK
2020-07-19 22:02:41,805 INFO     | 29:6 OK
2020-07-19 22:02:41,968 INFO     | 29:8 OK
2020-07-19 22:02:42,132 INFO     | 29:10 OK
2020-07-19 22:02:42,292 INFO     | 30:1 OK
2020-07-19 22:02:42,453 INFO     | 30:3 OK
2020-07-19 22:02:42,617 INFO     | 30:5 OK
2020-07-19 22:02:42,783 INFO     | 30:7 OK
2020-07-19 22:02:42,952 INFO     | 30:9 OK
2020-07-19 22:02:43,123 INFO     | 30:11 OK
2020-07-19 22:02:43,282 INFO     | 31:0 OK
2020-07-19 22:02:43,442 INFO     | 31:2 OK
2020-07-19 22:02:43,604 INFO     | 31:4 OK
2020-07-19 22:02:43,769 INFO     | 31:6 OK
2020-07-19 22:02:43,937 INFO     | 31:8 OK
2020-07-19 22:02:44,106 INFO     | 31:10 OK
2020-07-19 22:02:44,265 INFO     | 32:0 OK
2020-07-19 22:02:44,426 INFO     | 32:2 OK
2020-07-19 22:02:44,591 INFO     | 32:4 OK
2020-07-19 22:02:44,757 INFO     | 32:6 OK
2020-07-19 22:02:44,926 INFO     | 32:8 OK
2020-07-19 22:02:45,095 INFO     | 32:10 OK
2020-07-19 22:02:45,254 INFO     | 33:1 OK
2020-07-19 22:02:45,415 INFO     | 33:3 OK
2020-07-19 22:02:45,580 INFO     | 33:5 OK
2020-07-19 22:02:45,748 INFO     | 33:7 OK
2020-07-19 22:02:45,918 INFO     | 33:9 OK
2020-07-19 22:02:46,089 INFO     | 33:11 OK
2020-07-19 22:02:46,248 INFO     | 34:0 OK
2020-07-19 22:02:46,408 INFO     | 34:2 OK
2020-07-19 22:02:46,569 INFO     | 34:4 OK
2020-07-19 22:02:46,731 INFO     | 34:6 OK
2020-07-19 22:02:46,895 INFO     | 34:8 OK
2020-07-19 22:02:47,059 INFO     | 34:10 OK
2020-07-19 22:02:47,218 INFO     | 35:0 OK
2020-07-19 22:02:47,380 INFO     | 35:2 OK
2020-07-19 22:02:47,543 INFO     | 35:4 OK
2020-07-19 22:02:47,708 INFO     | 35:6 OK
2020-07-19 22:02:47,874 INFO     | 35:8 OK
2020-07-19 22:02:48,044 INFO     | 35:10 OK
2020-07-19 22:02:48,204 INFO     | 36:0 OK
2020-07-19 22:02:48,366 INFO     | 36:2 OK
2020-07-19 22:02:48,531 INFO     | 36:4 OK
2020-07-19 22:02:48,698 INFO     | 36:6 OK
2020-07-19 22:02:48,868 INFO     | 36:8 OK
2020-07-19 22:02:49,039 INFO     | 36:10 OK
2020-07-19 22:02:49,198 INFO     | 37:1 OK
2020-07-19 22:02:49,359 INFO     | 37:3 OK
2020-07-19 22:02:49,522 INFO     | 37:5 OK
2020-07-19 22:02:49,688 INFO     | 37:7 OK
2020-07-19 22:02:49,858 INFO     | 37:9 OK
2020-07-19 22:02:50,029 INFO     | 37:11 OK
2020-07-19 22:02:50,188 INFO     | 38:1 OK
2020-07-19 22:02:50,350 INFO     | 38:3 OK
2020-07-19 22:02:50,516 INFO     | 38:5 OK
2020-07-19 22:02:50,685 INFO     | 38:7 OK
2020-07-19 22:02:50,856 INFO     | 38:9 OK
2020-07-19 22:02:51,031 INFO     | 38:11 OK
2020-07-19 22:02:51,190 INFO     | 39:0 OK
2020-07-19 22:02:51,350 INFO     | 39:2 OK
2020-07-19 22:02:51,512 INFO     | 39:4 OK
2020-07-19 22:02:51,676 INFO     | 39:6 OK
2020-07-19 22:02:51,842 INFO     | 39:8 OK
2020-07-19 22:02:52,008 INFO     | 39:10 OK
2020-07-19 22:02:52,167 INFO     | 40:0 OK
2020-07-19 22:02:52,326 INFO     | 40:2 OK
2020-07-19 22:02:52,486 INFO     | 40:4 OK
2020-07-19 22:02:52,648 INFO     | 40:6 OK
2020-07-19 22:02:52,811 INFO     | 40:8 OK
2020-07-19 22:02:52,976 INFO     | 40:10 OK
2020-07-19 22:02:53,136 INFO     | 41:1 OK
2020-07-19 22:02:53,298 INFO     | 41:3 OK
2020-07-19 22:02:53,464 INFO     | 41:5 OK
2020-07-19 22:02:53,632 INFO     | 41:7 OK
2020-07-19 22:02:53,802 INFO     | 41:9 OK
2020-07-19 22:02:53,974 INFO     | 41:11 OK
2020-07-19 22:02:54,133 INFO     | 42:0 OK
2020-07-19 22:02:54,293 INFO     | 42:2 OK
2020-07-19 22:02:54,456 INFO     | 42:4 OK
2020-07-19 22:02:54,621 INFO     | 42:6 OK
2020-07-19 22:02:54,788 INFO     | 42:8 OK
2020-07-19 22:02:54,956 INFO     | 42:10 OK
2020-07-19 22:02:55,116 INFO     | 43:0 OK
2020-07-19 22:02:55,278 INFO     | 43:2 OK
2020-07-19 22:02:55,443 INFO     | 43:4 OK
2020-07-19 22:02:55,611 INFO     | 43:6 OK
2020-07-19 22:02:55,781 INFO     | 43:8 OK
2020-07-19 22:02:55,954 INFO     | 43:10 OK
2020-07-19 22:02:56,113 INFO     | 44:0 OK
2020-07-19 22:02:56,273 INFO     | 44:2 OK
2020-07-19 22:02:56,435 INFO     | 44:4 OK
2020-07-19 22:02:56,598 INFO     | 44:6 OK
2020-07-19 22:02:56,763 INFO     | 44:8 OK
2020-07-19 22:02:56,929 INFO     | 44:10 OK
2020-07-19 22:02:57,088 INFO     | 45:0 OK
2020-07-19 22:02:57,248 INFO     | 45:2 OK
2020-07-19 22:02:57,411 INFO     | 45:4 OK
2020-07-19 22:02:57,575 INFO     | 45:6 OK
2020-07-19 22:02:57,741 INFO     | 45:8 OK
2020-07-19 22:02:57,910 INFO     | 45:10 OK
2020-07-19 22:02:58,069 INFO     | 46:0 OK
2020-07-19 22:02:58,230 INFO     | 46:2 OK
2020-07-19 22:02:58,392 INFO     | 46:4 OK
2020-07-19 22:02:58,558 INFO     | 46:6 OK
2020-07-19 22:02:58,726 INFO     | 46:8 OK
2020-07-19 22:02:58,895 INFO     | 46:10 OK
2020-07-19 22:02:59,054 INFO     | 47:0 OK
2020-07-19 22:02:59,213 INFO     | 47:2 OK
2020-07-19 22:02:59,376 INFO     | 47:4 OK
2020-07-19 22:02:59,542 INFO     | 47:6 OK
2020-07-19 22:02:59,707 INFO     | 47:8 OK
2020-07-19 22:02:59,875 INFO     | 47:10 OK
2020-07-19 22:03:00,034 INFO     | 48:1 OK
2020-07-19 22:03:00,195 INFO     | 48:3 OK
2020-07-19 22:03:00,358 INFO     | 48:5 OK
2020-07-19 22:03:00,523 INFO     | 48:7 OK
2020-07-19 22:03:00,689 INFO     | 48:9 OK
2020-07-19 22:03:00,858 INFO     | 48:11 OK
2020-07-19 22:03:01,019 INFO     | 49:0 OK
2020-07-19 22:03:01,182 INFO     | 49:2 OK
2020-07-19 22:03:01,347 INFO     | 49:4 OK
2020-07-19 22:03:01,514 INFO     | 49:6 OK
2020-07-19 22:03:01,685 INFO     | 49:8 OK
2020-07-19 22:03:01,857 INFO     | 49:10 OK
2020-07-19 22:03:02,018 INFO     | 50:1 OK
2020-07-19 22:03:02,181 INFO     | 50:3 OK
2020-07-19 22:03:02,347 INFO     | 50:5 OK
2020-07-19 22:03:02,516 INFO     | 50:7 OK
2020-07-19 22:03:02,686 INFO     | 50:9 OK
2020-07-19 22:03:02,859 INFO     | 50:11 OK
2020-07-19 22:03:03,017 INFO     | 51:1 OK
2020-07-19 22:03:03,176 INFO     | 51:3 OK
2020-07-19 22:03:03,339 INFO     | 51:5 OK
2020-07-19 22:03:03,504 INFO     | 51:7 OK
2020-07-19 22:03:03,671 INFO     | 51:9 OK
2020-07-19 22:03:03,840 INFO     | 51:11 OK
2020-07-19 22:03:03,999 INFO     | 52:0 OK
2020-07-19 22:03:04,159 INFO     | 52:2 OK
2020-07-19 22:03:04,321 INFO     | 52:4 OK
2020-07-19 22:03:04,486 INFO     | 52:6 OK
2020-07-19 22:03:04,652 INFO     | 52:8 OK
2020-07-19 22:03:04,820 INFO     | 52:10 OK
2020-07-19 22:03:04,978 INFO     | 53:0 OK
2020-07-19 22:03:05,137 INFO     | 53:2 OK
2020-07-19 22:03:05,297 INFO     | 53:4 OK
2020-07-19 22:03:05,458 INFO     | 53:6 OK
2020-07-19 22:03:05,621 INFO     | 53:8 OK
2020-07-19 22:03:05,786 INFO     | 53:10 OK
2020-07-19 22:03:05,947 INFO     | 54:1 OK
2020-07-19 22:03:06,110 INFO     | 54:3 OK
2020-07-19 22:03:06,277 INFO     | 54:5 OK
2020-07-19 22:03:06,446 INFO     | 54:7 OK
2020-07-19 22:03:06,617 INFO     | 54:9 OK
2020-07-19 22:03:06,790 INFO     | 54:11 OK
2020-07-19 22:03:06,949 INFO     | 55:0 OK
2020-07-19 22:03:07,111 INFO     | 55:2 OK
2020-07-19 22:03:07,275 INFO     | 55:4 OK
2020-07-19 22:03:07,442 INFO     | 55:6 OK
2020-07-19 22:03:07,612 INFO     | 55:8 OK
2020-07-19 22:03:07,784 INFO     | 55:10 OK
2020-07-19 22:03:07,942 INFO     | 56:0 OK
2020-07-19 22:03:08,102 INFO     | 56:2 OK
2020-07-19 22:03:08,263 INFO     | 56:4 OK
2020-07-19 22:03:08,426 INFO     | 56:6 OK
2020-07-19 22:03:08,591 INFO     | 56:8 OK
2020-07-19 22:03:08,758 INFO     | 56:10 OK
2020-07-19 22:03:08,917 INFO     | 57:1 OK
2020-07-19 22:03:09,079 INFO     | 57:3 OK
2020-07-19 22:03:09,243 INFO     | 57:5 OK
2020-07-19 22:03:09,410 INFO     | 57:7 OK
2020-07-19 22:03:09,578 INFO     | 57:9 OK
2020-07-19 22:03:09,748 INFO     | 57:11 OK
2020-07-19 22:03:09,907 INFO     | 58:0 OK
2020-07-19 22:03:10,068 INFO     | 58:2 OK
2020-07-19 22:03:10,231 INFO     | 58:4 OK
2020-07-19 22:03:10,395 INFO     | 58:6 OK
2020-07-19 22:03:10,561 INFO     | 58:8 OK
2020-07-19 22:03:10,728 INFO     | 58:10 OK
2020-07-19 22:03:10,887 INFO     | 59:0 OK
2020-07-19 22:03:11,048 INFO     | 59:2 OK
2020-07-19 22:03:11,211 INFO     | 59:4 OK
2020-07-19 22:03:11,377 INFO     | 59:6 OK
2020-07-19 22:03:11,545 INFO     | 59:8 OK
2020-07-19 22:03:11,715 INFO     | 59:10 OK
2020-07-19 22:03:11,874 INFO     | 60:1 OK
2020-07-19 22:03:12,036 INFO     | 60:3 OK
2020-07-19 22:03:12,201 INFO     | 60:5 OK
2020-07-19 22:03:12,368 INFO     | 60:7 OK
2020-07-19 22:03:12,537 INFO     | 60:9 OK
2020-07-19 22:03:12,708 INFO     | 60:11 OK
2020-07-19 22:03:12,867 INFO     | 61:0 OK
2020-07-19 22:03:13,027 INFO     | 61:2 OK
2020-07-19 22:03:13,190 INFO     | 61:4 OK
2020-07-19 22:03:13,354 INFO     | 61:6 OK
2020-07-19 22:03:13,520 INFO     | 61:8 OK
2020-07-19 22:03:13,686 INFO     | 61:10 OK
2020-07-19 22:03:13,846 INFO     | 62:1 OK
2020-07-19 22:03:14,007 INFO     | 62:3 OK
2020-07-19 22:03:14,172 INFO     | 62:5 OK
2020-07-19 22:03:14,339 INFO     | 62:7 OK
2020-07-19 22:03:14,508 INFO     | 62:9 OK
2020-07-19 22:03:14,680 INFO     | 62:11 OK
2020-07-19 22:03:14,841 INFO     | 63:1 OK
2020-07-19 22:03:15,003 INFO     | 63:3 OK
2020-07-19 22:03:15,169 INFO     | 63:5 OK
2020-07-19 22:03:15,337 INFO     | 63:7 OK
2020-07-19 22:03:15,508 INFO     | 63:9 OK
2020-07-19 22:03:15,681 INFO     | 63:11 OK
2020-07-19 22:03:15,840 INFO     | 64:0 OK
2020-07-19 22:03:16,002 INFO     | 64:2 OK
2020-07-19 22:03:16,164 INFO     | 64:4 OK
2020-07-19 22:03:16,329 INFO     | 64:6 OK
2020-07-19 22:03:16,493 INFO     | 64:8 OK
2020-07-19 22:03:16,660 INFO     | 64:10 OK
2020-07-19 22:03:16,819 INFO     | 65:1 OK
2020-07-19 22:03:16,982 INFO     | 65:3 OK
2020-07-19 22:03:17,148 INFO     | 65:5 OK
2020-07-19 22:03:17,318 INFO     | 65:7 OK
2020-07-19 22:03:17,489 INFO     | 65:9 OK
2020-07-19 22:03:17,663 INFO     | 65:11 OK
2020-07-19 22:03:17,824 INFO     | 66:0 OK
2020-07-19 22:03:17,987 INFO     | 66:2 OK
2020-07-19 22:03:18,153 INFO     | 66:4 OK
2020-07-19 22:03:18,323 INFO     | 66:6 OK
2020-07-19 22:03:18,496 INFO     | 66:8 OK
2020-07-19 22:03:18,671 INFO     | 66:10 OK
2020-07-19 22:03:18,832 INFO     | 67:1 OK
2020-07-19 22:03:18,994 INFO     | 67:3 OK
2020-07-19 22:03:19,159 INFO     | 67:5 OK
2020-07-19 22:03:19,326 INFO     | 67:7 OK
2020-07-19 22:03:19,494 INFO     | 67:9 OK
2020-07-19 22:03:19,664 INFO     | 67:11 OK
2020-07-19 22:03:19,823 INFO     | 68:1 OK
2020-07-19 22:03:19,984 INFO     | 68:3 OK
2020-07-19 22:03:20,148 INFO     | 68:5 OK
2020-07-19 22:03:20,314 INFO     | 68:7 OK
2020-07-19 22:03:20,483 INFO     | 68:9 OK
2020-07-19 22:03:20,653 INFO     | 68:11 OK
2020-07-19 22:03:20,814 INFO     | 69:1 OK
2020-07-19 22:03:20,978 INFO     | 69:3 OK
2020-07-19 22:03:21,146 INFO     | 69:5 OK
2020-07-19 22:03:21,317 INFO     | 69:7 OK
2020-07-19 22:03:21,491 INFO     | 69:9 OK
2020-07-19 22:03:21,668 INFO     | 69:11 OK
2020-07-19 22:03:21,830 INFO     | 70:1 OK
2020-07-19 22:03:21,995 INFO     | 70:3 OK
2020-07-19 22:03:22,162 INFO     | 70:5 OK
2020-07-19 22:03:22,332 INFO     | 70:7 OK
2020-07-19 22:03:22,505 INFO     | 70:9 OK
2020-07-19 22:03:22,681 INFO     | 70:11 OK
2020-07-19 22:03:22,839 INFO     | 71:0 OK
2020-07-19 22:03:22,998 INFO     | 71:2 OK
2020-07-19 22:03:23,159 INFO     | 71:4 OK
2020-07-19 22:03:23,322 INFO     | 71:6 OK
2020-07-19 22:03:23,486 INFO     | 71:8 OK
2020-07-19 22:03:23,651 INFO     | 71:10 OK
2020-07-19 22:03:23,811 INFO     | 72:1 OK
2020-07-19 22:03:23,974 INFO     | 72:3 OK
2020-07-19 22:03:24,140 INFO     | 72:5 OK
2020-07-19 22:03:24,308 INFO     | 72:7 OK
2020-07-19 22:03:24,479 INFO     | 72:9 OK
2020-07-19 22:03:24,652 INFO     | 72:11 OK
2020-07-19 22:03:24,810 INFO     | 73:0 OK
2020-07-19 22:03:24,969 INFO     | 73:2 OK
2020-07-19 22:03:25,130 INFO     | 73:4 OK
2020-07-19 22:03:25,292 INFO     | 73:6 OK
2020-07-19 22:03:25,455 INFO     | 73:8 OK
2020-07-19 22:03:25,620 INFO     | 73:10 OK
2020-07-19 22:03:25,779 INFO     | 74:1 OK
2020-07-19 22:03:25,939 INFO     | 74:3 OK
2020-07-19 22:03:26,103 INFO     | 74:5 OK
2020-07-19 22:03:26,271 INFO     | 74:7 OK
2020-07-19 22:03:26,443 INFO     | 74:9 OK
2020-07-19 22:03:26,616 INFO     | 74:11 OK
2020-07-19 22:03:26,774 INFO     | 75:0 OK
2020-07-19 22:03:26,935 INFO     | 75:2 OK
2020-07-19 22:03:27,097 INFO     | 75:4 OK
2020-07-19 22:03:27,262 INFO     | 75:6 OK
2020-07-19 22:03:27,429 INFO     | 75:8 OK
2020-07-19 22:03:27,599 INFO     | 75:10 OK
2020-07-19 22:03:27,760 INFO     | 76:1 OK
2020-07-19 22:03:27,924 INFO     | 76:3 OK
2020-07-19 22:03:28,091 INFO     | 76:5 OK
2020-07-19 22:03:28,261 INFO     | 76:7 OK
2020-07-19 22:03:28,434 INFO     | 76:9 OK
2020-07-19 22:03:28,609 INFO     | 76:11 OK
2020-07-19 22:03:28,770 INFO     | 77:1 OK
2020-07-19 22:03:28,935 INFO     | 77:3 OK
2020-07-19 22:03:29,103 INFO     | 77:5 OK
2020-07-19 22:03:29,275 INFO     | 77:7 OK
2020-07-19 22:03:29,448 INFO     | 77:9 OK
2020-07-19 22:03:29,624 INFO     | 77:11 OK
2020-07-19 22:03:29,783 INFO     | 78:1 OK
2020-07-19 22:03:29,944 INFO     | 78:3 OK
2020-07-19 22:03:30,107 INFO     | 78:5 OK
2020-07-19 22:03:30,272 INFO     | 78:7 OK
2020-07-19 22:03:30,439 INFO     | 78:9 OK
2020-07-19 22:03:30,608 INFO     | 78:11 OK
2020-07-19 22:03:30,769 INFO     | 79:1 OK
2020-07-19 22:03:30,934 INFO     | 79:3 OK
2020-07-19 22:03:31,101 INFO     | 79:5 OK
2020-07-19 22:03:31,270 INFO     | 79:7 OK
2020-07-19 22:03:31,442 INFO     | 79:9 OK
2020-07-19 22:03:31,615 INFO     | 79:11 OK
2020-07-19 22:03:31,774 INFO     | 80:0 OK
2020-07-19 22:03:31,933 INFO     | 80:2 OK
2020-07-19 22:03:32,096 INFO     | 80:4 OK
2020-07-19 22:03:32,260 INFO     | 80:6 OK
2020-07-19 22:03:32,425 INFO     | 80:8 OK
2020-07-19 22:03:32,592 INFO     | 80:10 OK
2020-07-19 22:03:32,751 INFO     | 81:1 OK
2020-07-19 22:03:32,912 INFO     | 81:3 OK
2020-07-19 22:03:33,075 INFO     | 81:5 OK
2020-07-19 22:03:33,241 INFO     | 81:7 OK
2020-07-19 22:03:33,408 INFO     | 81:9 OK
2020-07-19 22:03:33,577 INFO     | 81:11 OK
2020-07-19 22:03:33,736 INFO     | 82:0 OK
2020-07-19 22:03:33,895 INFO     | 82:2 OK
2020-07-19 22:03:34,058 INFO     | 82:4 OK
2020-07-19 22:03:34,221 INFO     | 82:6 OK
2020-07-19 22:03:34,387 INFO     | 82:8 OK
2020-07-19 22:03:34,555 INFO     | 82:10 OK
2020-07-19 22:03:34,714 INFO     | 83:0 OK
2020-07-19 22:03:34,875 INFO     | 83:2 OK
2020-07-19 22:03:35,038 INFO     | 83:4 OK
2020-07-19 22:03:35,203 INFO     | 83:6 OK
2020-07-19 22:03:35,369 INFO     | 83:8 OK
2020-07-19 22:03:35,537 INFO     | 83:10 OK
2020-07-19 22:03:35,697 INFO     | 84:1 OK
2020-07-19 22:03:35,857 INFO     | 84:3 OK
2020-07-19 22:03:36,020 INFO     | 84:5 OK
2020-07-19 22:03:36,187 INFO     | 84:7 OK
2020-07-19 22:03:36,356 INFO     | 84:9 OK
2020-07-19 22:03:36,528 INFO     | 84:11 OK
2020-07-19 22:03:36,687 INFO     | 85:1 OK
2020-07-19 22:03:36,849 INFO     | 85:3 OK
2020-07-19 22:03:37,013 INFO     | 85:5 OK
2020-07-19 22:03:37,179 INFO     | 85:7 OK
2020-07-19 22:03:37,348 INFO     | 85:9 OK
2020-07-19 22:03:37,518 INFO     | 85:11 OK
2020-07-19 22:03:37,679 INFO     | 86:1 OK
2020-07-19 22:03:37,842 INFO     | 86:3 OK
2020-07-19 22:03:38,007 INFO     | 86:5 OK
2020-07-19 22:03:38,175 INFO     | 86:7 OK
2020-07-19 22:03:38,344 INFO     | 86:9 OK
2020-07-19 22:03:38,515 INFO     | 86:11 OK
2020-07-19 22:03:38,675 INFO     | 87:1 OK
2020-07-19 22:03:38,838 INFO     | 87:3 OK
2020-07-19 22:03:39,003 INFO     | 87:5 OK
2020-07-19 22:03:39,171 INFO     | 87:7 OK
2020-07-19 22:03:39,343 INFO     | 87:9 OK
2020-07-19 22:03:39,518 INFO     | 87:11 OK
2020-07-19 22:03:39,677 INFO     | 88:0 OK
2020-07-19 22:03:39,838 INFO     | 88:2 OK
2020-07-19 22:03:40,001 INFO     | 88:4 OK
2020-07-19 22:03:40,167 INFO     | 88:6 OK
2020-07-19 22:03:40,335 INFO     | 88:8 OK
2020-07-19 22:03:40,504 INFO     | 88:10 OK
2020-07-19 22:03:40,662 INFO     | 89:0 OK
2020-07-19 22:03:40,821 INFO     | 89:2 OK
2020-07-19 22:03:40,983 INFO     | 89:4 OK
2020-07-19 22:03:41,146 INFO     | 89:6 OK
2020-07-19 22:03:41,311 INFO     | 89:8 OK
2020-07-19 22:03:41,477 INFO     | 89:10 OK
2020-07-19 22:03:41,638 INFO     | 90:1 OK
2020-07-19 22:03:41,800 INFO     | 90:3 OK
2020-07-19 22:03:41,965 INFO     | 90:5 OK
2020-07-19 22:03:42,131 INFO     | 90:7 OK
2020-07-19 22:03:42,300 INFO     | 90:9 OK
2020-07-19 22:03:42,472 INFO     | 90:11 OK
2020-07-19 22:03:42,631 INFO     | 91:0 OK
2020-07-19 22:03:42,791 INFO     | 91:2 OK
2020-07-19 22:03:42,955 INFO     | 91:4 OK
2020-07-19 22:03:43,122 INFO     | 91:6 OK
2020-07-19 22:03:43,290 INFO     | 91:8 OK
2020-07-19 22:03:43,459 INFO     | 91:10 OK
2020-07-19 22:03:43,619 INFO     | 92:1 OK
2020-07-19 22:03:43,780 INFO     | 92:3 OK
2020-07-19 22:03:43,943 INFO     | 92:5 OK
2020-07-19 22:03:44,109 INFO     | 92:7 OK
2020-07-19 22:03:44,275 INFO     | 92:9 OK
2020-07-19 22:03:44,444 INFO     | 92:11 OK
stephenroller picture stephenroller on 20 Jul 2020
👍1

@stephenroller
Thanks for the new logs and the script. I just verified all logs in the evaluation_logs_reproducible. It works perfectly except for
responsiveness_model_bfw_setting_00: line 93
repetition_model_setting35_settinginf: line 52
interesting_nidf_model_bfw_setting_08: line 88
interesting_nidf_model_bfw_setting_06: line 93
inquisitive_model_ct_setting07: line 12,70

It's not a big deal though. Or I may got things wrong here, I will check again.

HMJiangGatech picture HMJiangGatech on 20 Jul 2020
👍1

Hmm, if that's like roughly 5 out of several hundred or thousand examples, I'm willing to call that just bit rot (pytorch 1.1 vs 1.5; parlai changing over time), or hardware differences (we ran on CPU during collection; perhaps you ran on GPU?).

stephenroller picture stephenroller on 20 Jul 2020

Yeah, I think that's fine. I ran on V100 GPUs with Pytorch v1.4 .

HMJiangGatech picture HMJiangGatech on 20 Jul 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

wanghaoran-ucas picture wanghaoran-ucas  路  4Comments
zhaoguangxiang picture zhaoguangxiang  路  5Comments
reppolice picture reppolice  路  4Comments
mvh57 picture mvh57  路  3Comments
Ufukdogann picture Ufukdogann  路  6Comments