Autokeras: Autokeras does not work on Windows due to fork

[csteel45]

I have windows machine with CUDA 10.0, Cudnn 7.5, and Python 3.7. I have successfully installed Tensorflow 1.11 and confirmed it works with non-trivial examples.

I installed Autokeras 0.2.19 and tried to run the example below:
import autokeras as ak
from keras.datasets import mnist

if __name__ == '__main__':
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train.reshape(x_train.shape + (1,))
x_test = x_test.reshape(x_test.shape + (1,))

clf = ak.ImageClassifier(verbose=True)
clf.fit(x_train, y_train, time_limit=12 * 60 * 60)
clf.final_fit(x_train, y_train, x_test, y_test, retrain=True)
y = clf.evaluate(x_test, y_test)
print(y)

The following is the output:
Using TensorFlow backend.

Initializing search.
Initialization finished.

+----------------------------------------------+
| Training model 0 |
+----------------------------------------------+
Traceback (most recent call last):
File "akerastest.py", line 10, in
clf.fit(x_train, y_train, time_limit=12 * 60 * 60)
File "C:\Users\Chris\Anaconda3\envs\tensorflow\lib\site-packages\autokeras\image\image_supervised.py", line 161, in fit
self.cnn.fit(self.get_n_output_node(), x_train.shape, train_data, test_data, time_limit)
File "C:\Users\Chris\Anaconda3\envs\tensorflow\lib\site-packages\autokeras\cnn_module.py", line 50, in fit
searcher.search(train_data, test_data, int(time_remain))
File "C:\Users\Chris\Anaconda3\envs\tensorflow\lib\site-packages\autokeras\search.py", line 175, in search
ctx = mp.get_context('fork')
File "C:\Users\Chris\Anaconda3\envs\tensorflow\lib\multiprocessing\context.py", line 238, in get_context
return super().get_context(method)
File "C:\Users\Chris\Anaconda3\envs\tensorflow\lib\multiprocessing\context.py", line 192, in get_context
raise ValueError('cannot find context for %r' % method) from None
ValueError: cannot find context for 'fork'

It looks like Autokeras is using fork, which I believe from Googling is not supported on Windows. If this is the case, is there a work-around or is this a known limitation?

[haifeng-jin]

@tl-yang Would you please look into this issue?
Thanks.

[tl-yang]

@csteel45 Yes, you're right, Windows doesn't support fork. The recent why we use fork is just to be able to run it on Google Colab. So I think this issue can be easily addressed by detecting the current structure. I'll take care of it ASAP!

[csteel45]

Great. Please also ensure that you address the use of virtual environments. I changed my local copy to use spawn instead of fork but I am having issues that I believe may be related to my virtualenv for tensorflow.

[csteel45]

Finally got it working after I copied some code from a post showing how to copy the environment info over. Also, when I first ran it, I kept getting an access denied error. I found a different post that suggested that I change to the directory of the process (my virtualenv). That fixed the issue. Lastly, you need to add the NVDIA-SMI directory to your path or you get:
'nvidia-smi' is not recognized as an internal or external command

[tl-yang]

Hi @csteel45 , will the error cause Autokeras to crash? Or it just appear on the terminal?

[csteel45]

It will crash

From: tl-yang [e-mail]
Reply-To: jhfjhfj1/autokeras [e-mail]
Date: Saturday, November 10, 2018 at 4:54 PM
To: jhfjhfj1/autokeras [e-mail]
Cc: Christopher Steel [e-mail], Mention [e-mail]
Subject: Re: [jhfjhfj1/autokeras] Autokeras does not work on Windows due to fork (#300)

Hi @csteel45 , will the error cause Autokeras to crash? Or it just appear on the terminal?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[Timmmeyyy]

So I got i right, under Ubuntu it will work and i didnt get the fork error?

[csteel45]

Correct. It doesn’t work under Windows which doesn’t support fork. Instead, you need to change the fork to a spawn so that it is cross-platform. The alternative is to check the OS and call fork for Unix-based systems and Spawn for Windows-based systems.

-Chris

From: Timmmeyyy [e-mail]
Reply-To: jhfjhfj1/autokeras [e-mail]
Date: Monday, November 12, 2018 at 11:12 AM
To: jhfjhfj1/autokeras [e-mail]
Cc: Christopher Steel [e-mail], Mention [e-mail]
Subject: Re: [jhfjhfj1/autokeras] Autokeras does not work on Windows due to fork (#300)

So I got i right, under Ubuntu it will work and i didnt get the fork error?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[Timmmeyyy]

Thx for your answer chris, can you give me the changes i have to do, to get it working under windows?

Greetz Tim

[Bjoux2]

@csteel45 I got the same error, could you please explain how to fix this error?

[csteel45]

@csteel45 I got the same error, could you please explain how to fix this error?

@Bjoux2 Sorry, I was on travel this week.

In search.py change the line: ctx = mp.get_context('fork')
to
ctx = mp.get_context('spawn')

I added some extra code to address running in a virtualenv so I don't have the exact line number.