Parlai: "parlai train_model" CLI fails in python 3.8

[lazyrenee]

Bug description
parlai train_model for parlai 0.9.2 seems not working.

Reproduction steps
I tried to run parlai train_model -t twitter -mf /tmp/tr_twitter -m transformer/ranker as the first example to test after I just rebased the repository, it went wrong. I checked with the the current parlai version is parlai 0.9.2 But this doesn't happen to my previous one parlai 0.9, not sure if that's something related to version update.

Logs
Please paste the command line output:

(base) xxx$ parlai train_model -t twitter -mf /tmp/tr_twitter -m transformer/ranker
usage: parlai build_candidates [-h] [--helpall] [-o INIT_OPT] [-t TASK] [-dt DATATYPE] [-bs BATCHSIZE]
                               [-dynb {full,batchsort,None}] [-dp DATAPATH] [-n NUM_EXAMPLES] [-of OUTFILE]
                               [-ltim LOG_EVERY_N_SECS]

Build the candidate responses for a retrieval model

optional arguments:
  -h, --help
        show this help message and exit
  --helpall
        Show usage, including advanced arguments.
  -o, --init-opt INIT_OPT
        Path to json file of options. Note: Further Command-line arguments override file-based options. (default: None)
  -t, --task TASK
        ParlAI task(s), e.g. "babi:Task1" or "babi,cbt" (default: None)
  -dt, --datatype DATATYPE
        choose from: train, train:ordered, valid, test. to stream data add ":stream" to any option (e.g.,
        train:stream). by default train is random with replacement, valid is ordered, test is ordered. (default:
        train:evalmode)
  -bs, --batchsize BATCHSIZE
        batch size for minibatch training schemes (default: 1)
  -dynb, --dynamic-batching {full,batchsort,None}
        Use dynamic batching (default: None)
  -dp, --datapath DATAPATH
        path to datasets, defaults to {parlai_dir}/data (default: None)
  -n, --num-examples NUM_EXAMPLES
        Total number of exs to convert, -1 to convert all examples (default: -1)
  -of, --outfile OUTFILE
        Output file where to save, by default will be created in /tmp (default: None)
  -ltim, --log-every-n-secs LOG_EVERY_N_SECS

optional arguments:
  -h, --help
        show this help message and exit
  --helpall
        Show usage, including advanced arguments.
  -n, --num-examples NUM_EXAMPLES
        Total number of exs to convert, -1 to convert all examples (default: -1)
  -of, --outfile OUTFILE
        Output file where to save, by default will be created in /tmp (default: None)
  -ltim, --log-every-n-secs LOG_EVERY_N_SECS

Main ParlAI Arguments:
  -o, --init-opt INIT_OPT
        Path to json file of options. Note: Further Command-line arguments override file-based options. (default: None)
  -t, --task TASK
        ParlAI task(s), e.g. "babi:Task1" or "babi,cbt" (default: None)
  -dt, --datatype DATATYPE
        choose from: train, train:ordered, valid, test. to stream data add ":stream" to any option (e.g.,
        train:stream). by default train is random with replacement, valid is ordered, test is ordered. (default:
        train:evalmode)
  -bs, --batchsize BATCHSIZE
        batch size for minibatch training schemes (default: 1)
  -dynb, --dynamic-batching {full,batchsort,None}
        Use dynamic batching (default: None)
  -dp, --datapath DATAPATH
        path to datasets, defaults to {parlai_dir}/data (default: None)

Parse Error: argument -mtw/--multitask-weights: invalid 'multitask_weights' value: 'transformer/ranker'

Additional context
Add any other context about the problem here. (like proxy settings, network setup, overall goals, etc.)

[stephenroller]

Ugh yes, this is a bug with python 3.8 and parlai I haven't figured out yet.

Use --model instead of -m and --task instead of -t as a temporary mitigation.

[stephenroller]

Alternatively, if python 3.7 is viable, you could use it.

[lazyrenee]

Thanks!!! so much appreciated! will try :)

[stephenroller]

Reopening to track the root cause

[lazyrenee]

uptate: I changed my current python version to python 3.7.9 and the command works for me :)

But as I changed the arguments of the CLI using command parlai train_model --task twitter -mf /tmp/tr_twitter -model transformer/ranker for python 3.8, seems it's not working.

The issue occus after it was successfully download the data and created the dictionary, the logs are:

04:53:10 | Current ParlAI commit: 65a28fd07218d56da4ff08d6523010885097704e
04:53:10 | creating task(s): twitter
04:53:10 | Loading ParlAI text data: xxx/ParlAI/data/Twitter/train.txt
04:53:30 | training...
04:53:30 | [ Executing train mode with provided inline set of candidates ]
Traceback (most recent call last):
  File "xxx/bin/parlai", line 33, in <module>
    sys.exit(load_entry_point('parlai', 'console_scripts', 'parlai')())
  File "xxx/ParlAI/parlai/core/script.py", line 272, in superscript_main
    SCRIPT_REGISTRY[cmd].klass._run_from_parser_and_opt(opt, parser)
  File "xxx/ParlAI/parlai/core/script.py", line 88, in _run_from_parser_and_opt
    return script.run()
  File "xxx/ParlAI/parlai/scripts/train_model.py", line 768, in run
    return self.train_loop.train()
  File "xxx/ParlAI/parlai/scripts/train_model.py", line 658, in train
    world.parley()
  File "xxx/ParlAI/parlai/core/worlds.py", line 345, in parley
    acts[1] = agents[1].act()
  File "/home/eilab/workspace/rj-questing/ParlAI/parlai/core/torch_agent.py", line 1913, in act
    response = self.batch_act([self.observation])[0]
  File "/home/eilab/workspace/rj-questing/ParlAI/parlai/core/torch_agent.py", line 1969, in batch_act
    output = self.train_step(batch)
  File "xxx/ParlAI/parlai/core/torch_ranker_agent.py", line 448, in train_step
    cands, cand_vecs, label_inds = self._build_candidates(
  File "xxx/ParlAI/parlai/core/torch_ranker_agent.py", line 719, in _build_candidates
    raise ValueError(
ValueError: If using candidate source 'inline', then batch.candidate_vecs cannot be None. If your task does not have inline candidates, consider using one of --candidates={'batch','fixed','vocab'}.

[stephenroller]

For context, https://github.com/python/cpython/commit/b1e4d1b6032d4c82b549233fa08a2c7cfe7e818b is where the short options were broken in python 3.8.

[stephenroller]

@lazyrenee for that task, you need to use --candidates batch --eval-candidates batch.

[lazyrenee]

I see, thanks for getting back so quickly :) For now guess I'll just continue to use python 3.7 for the parlai CLIs I'm more familiar with.

[stephenroller]

I'm thinking about how to solve this problem. It looks to me like python has straight banned the behavior we were using.

The only remedy I can think of is switching to --mtw to replace -mtw.

[lazyrenee]

Hi @stephenroller ,

Thanks for getting back to me so quick :)

I checked the code, it seems the issue with argparse and python 3.8. I tried to explore a bit and this seems the issue with the argparse & add_help parameter, so I wrote a tiny script to test it out:

import argparse
parser = argparse.ArgumentParser(description = 'tests.', add_help = True)

parser.add_argument('-s', '--short', type = str)

args = parser.parse_args()
print("test for -s with args.short: {}".format(args.short))
print("test for -short with args.short: {}".format(args.short))

The test result is as follows:

(python3.8) xxxx:~$ conda activate python3.8
(python3.8) xxxx:~$ python --version
Python 3.8.3
(python3.8) xxxx:~$ python exp_args.py --short Hello_world
test for -s with args.short: Hello_world
test for -short with args.short: Hello_world
(python3.8) xxxx:~$ python exp_args.py -s Hello_world
test for -s with args.short: Hello_world
test for -short with args.short: Hello_world

may could enable to : https://github.com/facebookresearch/ParlAI/blob/master/parlai/core/params.py#L283

(not fully sure if that could solve the problem)

Best,
Renee