Autokeras: An issue on multiprocessing with CUDA

[kmbmjn]

The example code on MNIST showed an issue on multiprocessing with CUDA.
I guess CUDA doesn't work well with the multiprocessing https://pytorch.org/docs/stable/notes/windows.html
I use Windows10, Python 3.6.5, CUDA 9.0, and tensorflow-gpu==1.8.0. I followed requirements.txt to install others.
Using commands "torch.version.cuda" and "torch.cuda.is_available()", I also checked CUDA 9.0 works well on my environment.

...........................................
Epoch 1: loss 1.8169708251953125, metric_value 0.987
...........................................
Epoch 2: loss 1.694014310836792, metric_value 0.9866
...........................................
Epoch 3: loss 1.5495505332946777, metric_value 0.9884
...........................................
Epoch 4: loss 1.7369670867919922, metric_value 0.9882
...........................................
Epoch 5: loss 2.2431726455688477, metric_value 0.9842
...........................................
Epoch 6: loss 1.821488618850708, metric_value 0.9864
...........................................
Epoch 7: loss 1.3172129392623901, metric_value 0.9908
...........................................
Epoch 8: loss 1.3580961227416992, metric_value 0.9906
...........................................
Epoch 9: loss 1.2147481441497803, metric_value 0.992
...........................................
Epoch 10: loss 1.3565152883529663, metric_value 0.9916
...........................................
Epoch 11: loss 1.201681137084961, metric_value 0.9916
...........................................
Epoch 12: loss 1.9307321310043335, metric_value 0.987
...........................................
Epoch 13: loss 1.2660953998565674, metric_value 0.993
...........................................
Epoch 14: loss 1.2389497756958008, metric_value 0.9924
...........................................
Epoch 15: loss 1.3451954126358032, metric_value 0.992
...........................................
Epoch 16: loss 2.506535291671753, metric_value 0.9844
...........................................
Epoch 17: loss 2.018087387084961, metric_value 0.989
...........................................
Epoch 18: loss 1.1533660888671875, metric_value 0.9934
...........................................
Epoch 19: loss 1.493323802947998, metric_value 0.9908
No loss decrease after 5 epochs
THCudaCheck FAIL file=c:\programdata\miniconda3\conda-bld\pytorch_1533090623466\work\torch\csrc\generic\StorageSharing.cpp line=231 error=63 : OS call failed or operation not supported on this OS
Traceback (most recent call last):
File "ak_mnist.py", line 10, in
clf.fit(x_train, y_train, time_limit=12 * 60 * 60)
File "C:\Users\kmbmjn\Anaconda3\lib\site-packages\autokeras\image_supervised.py", line 238, in fit
run_searcher_once(train_data, test_data, self.path, int(time_remain))
File "C:\Users\kmbmjn\Anaconda3\lib\site-packages\autokeras\image_supervised.py", line 41, in run_searcher_once
searcher.search(train_data, test_data, timeout)
File "C:\Users\kmbmjn\Anaconda3\lib\site-packages\autokeras\search.py", line 189, in search
metric_value, loss, graph = train_results.get(timeout=remaining_time)[0]
File "C:\Users\kmbmjn\Anaconda3\lib\multiprocessing\pool.py", line 644, in get
raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '[(0.9899199999999999, tensor(1.7033, device='cuda:0'), )]'. Reason: 'RuntimeError('cuda runtime error (63) : OS call failed or operation not supported on this OS at c:\programdata\miniconda3\conda-bld\pytorch_1533090623466\work\torch\csrc\generic\StorageSharing.cpp:231',)'

[eagleoflqj]

Same error.
My solution:
remove

pool = multiprocessing.Pool(1)

and

finally:
        # terminate and join the subprocess to prevent any resource leak
        pool.terminate()
        pool.join()

then change

train_results = pool.map_async(train, [(graph, train_data, test_data, self.trainer_args,
                                                 os.path.join(self.path, str(model_id) + '.png'),
                                                 self.metric, self.loss, self.verbose)])

to

train_results = list(map(train, [(graph, train_data, test_data, self.trainer_args,
                                                os.path.join(self.path, str(model_id) + '.png'),
                                                self.metric, self.loss, self.verbose)]))

and

metric_value, loss, graph = train_results.get(timeout=remaining_time)[0]

to

metric_value, loss, graph = train_results[0]

[eagleoflqj]

Well I just saw issues #76 , aa18514's method seems better, since there is only one item in the list.

[kmbmjn]

I could conduct some experiments after applying eagleoflqj's comment. I think it solved my problem. Thanks!

[xialulu826]

@eagleoflqj Thanks for your solution,it works.

[haifeng-jin]

Thank you @eagleoflqj !
We have changed the multiprocessing to torch.multiprocessing.
@kmbmjn @xialulu826 Would you like to try again with our latest release to see if it works?

[eagleoflqj]

It works! @jhfjhfj1