Nmt: Can't inference big file

[lyy1994]

When I use configuration --out_dir=./nmt_attention_model --inference_input_file=./nmt_data/train.vi --inference_output_file=./nmt_model/output_infer --inference_ref_file=./nmt_data/train.en in Pycharm, I got following error:

Traceback (most recent call last):
  File "/home/liyanyang/projects/baseline/nmt.py", line 481, in <module>
    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/home/liyanyang/projects/baseline/nmt.py", line 460, in main
    trans_file, hparams, num_workers, jobid)
  File "/home/liyanyang/projects/baseline/inference.py", line 164, in inference
    scope=scope)
  File "/home/liyanyang/projects/baseline/inference.py", line 222, in _single_worker_inference
    tgt_eos=hparams.eos)
  File "/home/liyanyang/projects/baseline/utils/nmt_utils.py", line 51, in decode_and_evaluate
    nmt_outputs, _ = model.decode(sess)
  File "/home/liyanyang/projects/baseline/model.py", line 448, in decode
    _, infer_summary, _, sample_words = self.infer(sess)
  File "/home/liyanyang/projects/baseline/model.py", line 435, in infer
    self.infer_logits, self.infer_summary, self.sample_id, self.sample_words
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 789, in run
    run_metadata_ptr)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 997, in _run
    feed_dict_string, options, run_metadata)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1132, in _do_run
    target_list, options, run_metadata)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1152, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [All values in memory_sequence_length must greater than zero.] [Condition x > 0 did not hold element-wise:] [x (IteratorGetNext:1) = ] [11 39 41...]
     [[Node: dynamic_seq2seq/decoder/decoder/while/BasicDecoderStep/decoder/attention/assert_positive/assert_less/Assert/Assert = Assert[T=[DT_STRING, DT_STRING, DT_STRING, DT_INT32], summarize=3, _device="/job:localhost/replica:0/task:0/cpu:0"](dynamic_seq2seq/decoder/decoder/while/BasicDecoderStep/decoder/attention/assert_positive/assert_less/All/_191, dynamic_seq2seq/decoder/decoder/while/BasicDecoderStep/decoder/attention/assert_positive/assert_less/Assert/Assert/data_0, dynamic_seq2seq/decoder/decoder/while/BasicDecoderStep/decoder/attention/assert_positive/assert_less/Assert/Assert/data_1, dynamic_seq2seq/decoder/decoder/while/BasicDecoderStep/decoder/attention/assert_positive/assert_less/Assert/Assert/data_2, dynamic_seq2seq/decoder/decoder/while/BasicDecoderStep/decoder/attention/assert_positive/assert_less/Less/Enter/_193)]]

Caused by op u'dynamic_seq2seq/decoder/decoder/while/BasicDecoderStep/decoder/attention/assert_positive/assert_less/Assert/Assert', defined at:
  File "/home/liyanyang/projects/baseline/nmt.py", line 481, in <module>
    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/home/liyanyang/projects/baseline/nmt.py", line 460, in main
    trans_file, hparams, num_workers, jobid)
  File "/home/liyanyang/projects/baseline/inference.py", line 164, in inference
    scope=scope)
  File "/home/liyanyang/projects/baseline/inference.py", line 189, in _single_worker_inference
    infer_model = create_infer_model(model_creator, hparams, scope)
  File "/home/liyanyang/projects/baseline/inference.py", line 80, in create_infer_model
    scope=scope)
  File "/home/liyanyang/projects/baseline/attention_model.py", line 53, in __init__
    scope=scope)
  File "/home/liyanyang/projects/baseline/model.py", line 88, in __init__
    res = self.build_graph(hparams, scope=scope)
  File "/home/liyanyang/projects/baseline/model.py", line 226, in build_graph
    encoder_outputs, encoder_state, hparams)
  File "/home/liyanyang/projects/baseline/model.py", line 380, in _build_decoder
    scope=decoder_scope)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/contrib/seq2seq/python/ops/decoder.py", line 286, in dynamic_decode
    swap_memory=swap_memory)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2770, in while_loop
    result = context.BuildLoop(cond, body, loop_vars, shape_invariants)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2599, in BuildLoop
    pred, body, original_loop_vars, loop_vars, shape_invariants)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2549, in _BuildLoop
    body_result = body(*packed_vars_for_body)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/contrib/seq2seq/python/ops/decoder.py", line 234, in body
    decoder_finished) = decoder.step(time, inputs, state)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/contrib/seq2seq/python/ops/basic_decoder.py", line 139, in step
    cell_outputs, cell_state = self._cell(inputs, state)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/rnn_cell_impl.py", line 844, in __call__
    return self._cell(inputs, state, scope=scope)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/rnn_cell_impl.py", line 180, in __call__
    return super(RNNCell, self).__call__(inputs, state)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/layers/base.py", line 441, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/contrib/seq2seq/python/ops/attention_wrapper.py", line 727, in call
    cell_output, previous_alignments=state.alignments)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/contrib/seq2seq/python/ops/attention_wrapper.py", line 357, in __call__
    alignments = self._probability_fn(score, previous_alignments)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/contrib/seq2seq/python/ops/attention_wrapper.py", line 185, in <lambda>
    _maybe_mask_score(score, memory_sequence_length, score_mask_value),
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/contrib/seq2seq/python/ops/attention_wrapper.py", line 120, in _maybe_mask_score
    [check_ops.assert_positive(memory_sequence_length, message=message)]):
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/check_ops.py", line 198, in assert_positive
    return assert_less(zero, x, data=data, summarize=summarize)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/check_ops.py", line 401, in assert_less
    return control_flow_ops.Assert(condition, data, summarize=summarize)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/util/tf_should_use.py", line 170, in wrapped
    return _add_should_use_warning(fn(*args, **kwargs))
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 124, in Assert
    condition, data, summarize, name="Assert")
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/gen_logging_ops.py", line 37, in _assert
    summarize=summarize, name=name)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/liyanyang/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
    self._traceback = _extract_stack()

But it works fine with --out_dir=./nmt_attention_model --inference_input_file=./nmt_data/tst2013.vi --inference_output_file=./nmt_attention_model/output_infer --inference_ref_file=./nmt_data/tst2013.en.

[hanwei2008]

I meet the same problem, hoping for answer

[oahziur]

@lyy1994 @hanwei2008 I think this error happens when you have zero length sequence (empty lines) in your inference file.

[kmario23]

Yes, @oahziur is correct!! I faced similar issue when running the evaluation (which interrupted the training), and I found that it is caused due to empty lines in my dev files. After removing those zero length sequences, this issue is resolved.

[ngovinhtn]

I run training with "python -m nmt.nmt --attention=scaled_luong --src=ja --tgt=vi --vocab_prefix=/home/ngovinh/nmt/nmt/corpus/vocab --train_prefix=/home/ngovinh/nmt/nmt/corpus/train --dev_prefix=/home/ngovinh/nmt/nmt/corpus/dev --test_prefix=/home/ngovinh/nmt/nmt/corpus/test --out_dir=/data/ngovinh/nmt-attention --src_max_len=80 --tgt_max_len=80 --num_train_steps=12000 --steps_per_stats=100 --num_layers=2 --num_units=128 --dropout=0.2 --metrics=bleu 1"
and then I get error after 1000 step as follow:
InvalidArgumentError (see above for traceback): assertion failed: [All values in memory_sequence_length must greater than zero.] [Condition x > 0 did not hold element-wise:] [x (IteratorGetNext:1) = ] [25 54 28...]

I have checked all file of data, but they don't have any empty line. I do not know why? can anyone help me?
thanks,

[kmario23]

@ngovinhtn I presume this error is thrown when an internal evaluation is run on the dev set. There has to be something wrong with your dev set data. Make sure that they don't have any empty lines in both ja & vi files.

[ngovinhtn]

@kmario23 thank your reply!
My data set surely does not have any empty lines!

[manojpamk]

I got the same error as above. Indeed my dev file contains empty lines.
However, this error is thrown only when I try to use an attention model.