Xla: RuntimeError: tensorflow/compiler/xla/xla_client/xrt_computation_client.cc:1223

[ogulcanogul]

Issue description

While I try to run example script (torch-xla-nightly)$ python -m torch_xla.distributed.xla_dist --tpu=$TPU_POD_NAME --conda-env=torch-xla-nightly --vm=$VM_NAME --env=XLA_USE_BF16=1 -- python /usr/share/torch-xla-0.5/pytorch/xla/test/test_train_imagenet.py --fake_data, I get an error message as below:

   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
100    19  100    19    0     0   3247      0 --:--:-- --:--:-- --:--:--  3800
   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    19  100    19    0     0   3262      0 --:--:-- --:--:-- --:--:--  3800
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100    19  100    19    0     0   3241      0 --:--:-- --:--:-- --:--:--  3800
 2020-05-04 19:20:43.062835: E    6735 tensorflow/compiler/xla/xla_client/tf_logging.cc:11] Check failed: session.Run({tensorflow::Output(result, 0)}, &outputs) == ::tensorflow::Status::OK() (Invalid argument: No matching devices found for '/job:c_tpu_worker/replica:0/task:0/device:TPU_SYSTEM:0' vs. OK)
 *** Begin stack trace ***
    tensorflow::CurrentStackTrace[abi:cxx11]()
    xla::XrtComputationClient::InitializeAndFetchTopology(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tensorflow::ConfigProto const&)
    xla::XrtComputationClient::InitializeDevices(std::unique_ptr<tensorflow::tpu::TopologyProto, std::default_delete<tensorflow::tpu::TopologyProto> >)
    xla::XrtComputationClient::XrtComputationClient(xla::XrtComputationClient::Options, std::unique_ptr<tensorflow::tpu::TopologyProto, std::default_delete<tensorflow::tpu::TopologyProto> >)
    xla::ComputationClient::Create()
    xla::ComputationClient::Get()

    _PyCFunction_FastCallDict

    _PyEval_EvalFrameDefault



    _PyEval_EvalFrameDefault


    _PyEval_EvalFrameDefault


    _PyEval_EvalFrameDefault
        _PyFunction_FastCallDict
    _PyObject_FastCallDict
    _PyObject_Call_Prepend
    PyObject_Call
    _PyEval_EvalFrameDefault

    _PyFunction_FastCallDict
    _PyObject_FastCallDict
    _PyObject_Call_Prepend
    PyObject_Call

    _PyObject_FastCallDict

    _PyEval_EvalFrameDefault

    _PyFunction_FastCallDict
    _PyObject_FastCallDict
    _PyObject_Call_Prepend
    PyObject_Call
    _PyEval_EvalFrameDefault

    _PyFunction_FastCallDict
    _PyObject_FastCallDict
    _PyObject_Call_Prepend
    PyObject_Call

    _PyObject_FastCallDict

    _PyEval_EvalFrameDefault

    _PyFunction_FastCallDict
    _PyObject_FastCallDict
    _PyObject_Call_Prepend
    PyObject_Call
    _PyEval_EvalFrameDefault

    _PyFunction_FastCallDict
    _PyObject_FastCallDict
    _PyObject_Call_Prepend
    PyObject_Call

    _PyObject_FastCallDict

        _PyEval_EvalFrameDefault

    _PyEval_EvalFrameDefault

    _PyEval_EvalFrameDefault

    _PyFunction_FastCallDict
    _PyObject_FastCallDict
    _PyObject_Call_Prepend
    PyObject_Call
    _PyObject_FastCallDict
    _PyObject_FastCallKeywords

    _PyEval_EvalFrameDefault

    _PyEval_EvalFrameDefault
    PyEval_EvalCodeEx
    PyEval_EvalCode

    PyRun_FileExFlags
    PyRun_SimpleFileExFlags
    Py_Main
    main
    __libc_start_main

 *** End stack trace ***

Actually, everything seems okay while the code is initialized:

Cluster configuration: {client_workers: [{$VM_IP, n1-highmem-4, europe-west4-a, $VM_NAME}], service_workers: [{$TPU_IP, 8470, v3-8, europe-west4-a, pytorch-nightly, $TPU_NAME}]}

How can I solve this problem?

Thanks for your help in advance.

[jysohn23]

Hi @ogulcanogul, are you training on a v3-8 device? That's what it looks like based on your cluster config you posted. xla_dist is mainly meant for TPU Pod training (v2-32/v3-32 and larger). For single device training refer to this tutorial instead.

[taylanbil]

Also, the conda environment (torch-xla-nightly) and the code (torch-xla-0.5) should match for best results.

[ogulcanogul]

Hi @ogulcanogul, are you training on a v3-8 device? That's what it looks like based on your cluster config you posted. xla_dist is mainly meant for TPU Pod training (v2-32/v3-32 and larger). For single device training refer to this tutorial instead.

I don't understand. Can I use only one of 8 TPUs?

[jysohn23]

No, by running something like:

python /usr/share/torch-xla-${VERSION?}/pytorch/xla/test/test_train_mp_imagenet.py --fake_data --model=resnet50 --num_epochs=2 --batch_size=128 --log_steps=20 --num_cores=8

You are running on all 8 cores (1 v3-8).

[ogulcanogul]

No, by running something like:

python /usr/share/torch-xla-${VERSION?}/pytorch/xla/test/test_train_mp_imagenet.py --fake_data --model=resnet50 --num_epochs=2 --batch_size=128 --log_steps=20 --num_cores=8

You are running on all 8 cores (1 v3-8).

Yes, it works. Thank you very much. I was struggling for 3 days. I should have asked before.