Autokeras: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown len(cache))

[chuanmingliu]

Bug Description

Hi, everyone! When I try to use autokeras fit some data and label, I encounter such warnings as:
"semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown len(cache))"

After some iterations, for example, after prompt "Model 9", the program is hanging due to insufficient memory.

I don't think it has connection with my code, here is my code:

from autokeras.image_supervised import ImageClassifier
import numpy as np

if __name__ == '__main__':
    x_train = np.load('x_train.npy')
    y_train = np.load('y_train.npy')
    x_test = np.load('x_test.npy')
    y_test = np.load('y_test.npy')
    x_train = x_train.reshape((-1, 32, 32, 3))
    y_train.reshape((-1, 1))
    x_test = x_test.reshape((-1, 32, 32, 3))
    y_test = y_test.reshape((-1, 1))
    clf = ImageClassifier(verbose=True)
    clf.fit(x_train, y_train, time_limit=12*60*60)
    clf.final_fit(x_train, y_train, x_test, y_test, retrain=True)
    pred = clf.evaluate(x_test, y_test)
    print(pred)

The magnitude of my dataset is similar to CIFAR10.

I want to know, how to fix this warning and following memory issue? Does anyone has experience in fixing such things? Thanks in advance.

Reproducing Steps

Steps to reproduce the behavior:

• Step 1: ...
• Step 2: ...

Expected Behavior

Setup Details

Include the details about the versions of:

• OS type and version: Ubuntu 16.04
• Python: 3.6
• autokeras:
• scikit-learn:
• numpy:
• keras: 2.2.2
• scipy:
• tensorflow: 1.10.0
• pytorch:
  All requirements are installed by "pip install autokeras" and I run my code in a Docker instance,
  Some info on my Docker environment:
  chuanming@Galadriel:~$ nvidia-docker version
  NVIDIA Docker: 2.0.3
  Client:
  Version: 18.03.1-ce
  API version: 1.37
  Go version: go1.9.5
  Git commit: 9ee9f40
  Built: Thu Apr 26 07:17:20 2018
  OS/Arch: linux/amd64
  Experimental: false
  Orchestrator: swarm

Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:15:30 2018
OS/Arch: linux/amd64
Experimental: false

Additional context

[monikatomar92]

I experienced the same warning.
UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
len(cache))

[VictorZhang2014]

Same issue here. But how long will autokeras terminate on MNIST training? It's training on my mac in 8 hours. As screenshot below.

How to get the resulted model? Which path does these models save?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.