Xla: machine translation validation fails with multi-process

Created on 31 Oct 2019 · 11Comments · Source: pytorch/xla

❓ Questions and Help

To Reproduce

Steps to reproduce the behavior:

create an instance using the latest torch-xla

export PROJECT_NAME=xxx
gcloud config set project ${PROJECT_NAME}
gcloud compute --project=${PROJECT_NAME} instances create instance-1 \
--zone=europe-west4-a  \
--machine-type=n1-standard-8  \
--image=debian-9-torch-xla-v20191026 \
--image-project=ml-images  \
--boot-disk-size=200GB

conda activate torch-xla-nightly
run machine translation scirpt following https://cloud.google.com/tpu/docs/tutorials/transformer-pytorch in tpu branch of fairseq-tpu (https://github.com/pytorch-tpu/fairseq/tree/tpu) as

gcloud compute tpus create transformer-pytorch-tutorial \
--zone=europe-west4-a \
--network=default \
--range=10.2.3.0 \
--version=pytorch-nightly \
--accelerator-type=v3-8

export TPU_IP_ADDRESS=ip-address; \
export XRT_TPU_CONFIG="tpu_worker;0;$TPU_IP_ADDRESS:8470";

python train.py \
  $HOME/pytorch-tutorial-data/wmt18_en_de_bpej32k \
  --save-interval=1 \
  --arch=transformer_vaswani_wmt_en_de_big \
  --max-target-positions=64 \
  --attention-dropout=0.1 \
  --no-progress-bar \
  --criterion=label_smoothed_cross_entropy \
  --source-lang=en \
  --lr-scheduler=inverse_sqrt \
  --min-lr 1e-09 \
  --skip-invalid-size-inputs-valid-test \
  --target-lang=de \
  --label-smoothing=0.1 \
  --update-freq=1 \
  --optimizer adam \
  --adam-betas '(0.9, 0.98)' \
  --warmup-init-lr 1e-07 \
  --lr 0.0005 \
  --warmup-updates 4000 \
  --share-all-embeddings \
  --dropout 0.3 \
  --weight-decay 0.0 \
  --valid-subset=valid \
  --max-epoch=25 \
  --input_shapes 128x64 \
  --num_cores=8 \
  --metrics_debug \
  --log_steps=100

After the first epoch during validation, it reports
/anaconda3/envs/torch-xla-nightly/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown len(cache)) and then crushes. There is no checkpoint saved, too.

Expected behavior

It crushes with the SIGKILL from multiprocessing:

Traceback (most recent call last):
  File "train.py", line 632, in <module>
    cli_main()
  File "train.py", line 623, in cli_main
    xmp.spawn(_mp_fn, args=(args,), nprocs=args.num_cores)
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 154, in spawn
    _start_fn, args=(fn, args), nprocs=nprocs, join=join, daemon=daemon)
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
    while not spawn_context.join():
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 107, in join
    (error_index, name)
Exception: process 0 terminated with signal SIGKILL

Environment

reproducible on XLA backend [CPU/TPU]: TPU
torch_xla version: torch-xla-nightly (v1026)
Any other relevant information:

question

Source

sIncerass

All 11 comments

Hello,

This is not a fatal error, right? The process should be going on after you see this message in the stderr, can you confirm?

This was discussed here. As far as I can tell, this issue is not really related to TPUs and it is benign.

taylanbil on 31 Oct 2019

sIncerass on 31 Oct 2019

I see. Is there another error message you see? Something legitimately errors in the code, but this is independent of the semaphore_tracker message above. I'm going to be going through the steps now to see if I can reproduce.

taylanbil on 31 Oct 2019

Oh I just noticed that the commands above create a VM in Europe, and TPUs are in US. Can you retry w/ same region?

taylanbil on 31 Oct 2019

Sorry, that's a typo, the TPU and VM instance are all in Europe.
Thanks for helping. That's the only error message I have seen.

sIncerass on 31 Oct 2019

I am trying to repro currently. I'll report back if epoch 1 validation errors or completes.

taylanbil on 31 Oct 2019

👍1

Many thanks for helping! I am also restarting a new run to see if it reports the same issue.
confirmed the same issue after the first epoch.

sIncerass on 31 Oct 2019

👍1

You can also reproduce this error by just adding an

if i == 10:
    return tracker

inside train_loop_fn so you don't have to wait for epoch 1 training to finish.

Eric-Wallace on 1 Nov 2019

So I created a new VM + tpu, and ran through the tutorial. The process indeed died as described in the issue, around validation step ~300. It received a SIGKILL. Looking at sudo dmesg -T, it became obvious that this is an OOM error.

The reason for this is, I believe, the following:

the tutorial is created assuming the environment torch-xla-0.5. Whereas you are using torch-xla-nightly. But there has been big changes since the 0.5 release, including switching to use multiprocessing instead of multithreading.
Multiprocessing loads the input data to all the processes, whereas multithreading loads once, so the memory usage is significantly higher in MP.
Since the tutorial uses n1-standard-8, the process OOMs.

I have verified that the combo n1-standard-64 and torch-xla-nightly works. I will now verify that it works on torch-xla-0.5 and n1-standard-8.

Does that make sense?

taylanbil on 1 Nov 2019

Yes, it makes sense.
@Eric-Wallace and we found that it might be better to merge the https://github.com/pytorch/fairseq/commit/a1c997bd9a4e626b0b75aa2dd8d08d8d2beb0c71 into the pytorch-tpu/fairseq repo, which offers more efficient data loader and maybe resolves this problem easily. ("mmap" makes the script doesn't copy the memory across all the different processes).

sIncerass on 1 Nov 2019

👍1

Thanks for the suggestion, that seems like a useful commit indeed. It is in our plans to rebase our tpu branch on top of fairseq master, which will include this change too. Feel free to submit a PR if you have cherry picked that commit and resolved conflicts etc already.

I verified that both combinations below work.