Ludwig: Assertion Error while running hyperopt notebook

[Juggernaut1997]

Hi, From last couple of weeks I am exploring Ludwig. While exploring I came across the Hyper-parameter optimization visualization done in Ludwig. So, I forked the repo and start looking into it. While doing that I got an error related to multiprocessing pool which I am not sure what it is. So I am mentioning the error below, please look into it and guide me what's causing this error and how I can resolve it. The notebook I am using is hyperopt notebook.

---

RemoteTraceback Traceback (most recent call last)
RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(args, kwds))
File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
return list(map(args))
File "/usr/local/lib/python3.6/dist-packages/ludwig/hyperopt/execution.py", line 202, in _run_experiment
train_stats, eval_stats = run_experiment(hyperopt_dict)
File "/usr/local/lib/python3.6/dist-packages/ludwig/hyperopt/execution.py", line 703, in run_experiment
debug=debug,
File "/usr/local/lib/python3.6/dist-packages/ludwig/api.py", line 1035, in experiment
debug=debug,
File "/usr/local/lib/python3.6/dist-packages/ludwig/api.py", line 397, in train
random_seed=random_seed
File "/usr/local/lib/python3.6/dist-packages/ludwig/data/preprocessing.py", line 1208, in preprocess_for_training
random_seed=random_seed
File "/usr/local/lib/python3.6/dist-packages/ludwig/data/preprocessing.py", line 169, in preprocess_for_training
random_seed=random_seed
File "/usr/local/lib/python3.6/dist-packages/ludwig/data/preprocessing.py", line 1423, in _preprocess_df_for_training
random_seed=random_seed
File "/usr/local/lib/python3.6/dist-packages/ludwig/data/preprocessing.py", line 925, in build_dataset
metadata
File "/usr/local/lib/python3.6/dist-packages/ludwig/data/preprocessing.py", line 1014, in build_data
preprocessing_parameters
File "/usr/local/lib/python3.6/dist-packages/ludwig/features/image_feature.py", line 298, in add_feature_data
with Pool(num_processes) as pool:
File "/usr/lib/python3.6/multiprocessing/context.py", line 119, in Pool
context=self.get_context())
File "/usr/lib/python3.6/multiprocessing/pool.py", line 174, in __init__
self._repopulate_pool()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 239, in _repopulate_pool
w.start()
File "/usr/lib/python3.6/multiprocessing/process.py", line 103, in start
'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children
"""

The above exception was the direct cause of the following exception:

AssertionError Traceback (most recent call last)
in ()
10 config,
11 dataset=train_df.sample(4000, random_state=42),
---> 12 output_directory='results_random_parallel'
13 )

3 frames
/usr/local/lib/python3.6/dist-packages/ludwig/hyperopt/run.py in hyperopt(config, dataset, training_set, validation_set, test_set, training_set_metadata, data_format, experiment_name, model_name, skip_save_training_description, skip_save_training_statistics, skip_save_model, skip_save_progress, skip_save_log, skip_save_processed_input, skip_save_unprocessed_output, skip_save_predictions, skip_save_eval_stats, skip_save_hyperopt_statistics, output_directory, gpus, gpu_memory_limit, allow_parallel_threads, use_horovod, random_seed, debug, *kwargs)
289 random_seed=random_seed,
290 debug=debug,
--> 291 *kwargs
292 )
293

/usr/local/lib/python3.6/dist-packages/ludwig/hyperopt/execution.py in execute(self, config, dataset, training_set, validation_set, test_set, training_set_metadata, data_format, experiment_name, model_name, skip_save_training_description, skip_save_training_statistics, skip_save_model, skip_save_progress, skip_save_log, skip_save_processed_input, skip_save_unprocessed_output, skip_save_predictions, skip_save_eval_stats, output_directory, gpus, gpu_memory_limit, allow_parallel_threads, use_horovod, random_seed, debug, **kwargs)
425 else:
426 batch_results = pool.map(self._run_experiment,
--> 427 hyperopt_parameters)
428
429 self.hyperopt_sampler.update_batch(

/usr/lib/python3.6/multiprocessing/pool.py in map(self, func, iterable, chunksize)
264 in a list that is returned.
265 '''
--> 266 return self._map_async(func, iterable, mapstar, chunksize).get()
267
268 def starmap(self, func, iterable, chunksize=None):

/usr/lib/python3.6/multiprocessing/pool.py in get(self, timeout)
642 return self._value
643 else:
--> 644 raise self._value
645
646 def _set(self, i, obj):

AssertionError: daemonic processes are not allowed to have children

[jimthompson5802]

@Juggernaut1997 I'm trying to understand your set up where the error occurs.

You referenced hyperopt notebook from the Ludwig examples. I ran this notebook, including the parallel grid search example, and was not able to reproduce your error.

A few questions:

• Did you modify the notebook in anyway?
• Does the error occur in your modified code? If yes, please post the modified code.
• What OS are you using to run the notebook?
• What version of Python are you using?

Another piece of useful information is the output of pip list so I can understand what packages and their versions you have installed.

[jimthompson5802]

@Juggernaut1997 I've been looking at your issue and have some ideas on a solution. However to know if these ideas will work, I need answers to the questions I posted earlier. The most important one is see the code you have written and trying to execute. For image data, you may need to use another Ludwig api ludwig.data.preprocessing.preprocess_for_training().

Looking forward to your responses and helping you to resolve the issue.

[Juggernaut1997]

Hi @jimthompson5802 , Following are the replies to your information you needed to understand the error. Please take a look and let me know if you need any other information too.

1. Did you modify the notebook in anyway?
   -> Yes

2. Does the error occur in your modified code? If yes, please post the modified code.
   -> I am trying to implement the Hyperopt on the MNIST handwritten digits dataset. The only thing I changed is the dataset in
   the original hyperopt notebook.

3. What OS are you using to run the notebook?
   -> I am using Windows 10, but running Ludwig on Google Colab.

4. What version of Python are you using?
   -> Python version -- 3.6

Thank you for the quick response. Waiting for your response. 👍

[Juggernaut1997]

@jimthompson5802 You mentioned ludwig.data.preprocessing.preprocess_for_training(), can you suggest where I can find the details on it. Like what inputs needs to be given and how can I use this API?

Thank you once again. :)

[jimthompson5802]

@Juggernaut1997

re: response to "Does the error occur in your modified code? If yes, please post the modified code." I was hoping to see the code you are executing vs a verbal description. Without seeing the actual code, I don't know if the suggestion shown below will work for your context. Unless I can recreate the problem with your code, the following may or may not work for you.

Give the following a try and let me know if it solves the problem. If it does not solve the problem, then I need to see the actual code you are trying to run.

#%%

preprocess_for_training(config, dataset='./data/mnist_dataset_training_subset.csv')

#%%

%%time
print("starting:", datetime.datetime.now())
config['hyperopt']['executor'] = {'type': 'parallel', 'num_workers': 4}
config['hyperopt']['sampler'] = {'type': 'random', 'num_samples': 10}
random_parallel_results = hyperopt(
    config,
    dataset='./data/mnist_dataset_training_subset.csv',
    output_directory='results_random_parallel'  # location to place results
)

[Juggernaut1997]

@jimthompson5802

I am not sure if this is working or not for me, because it keep on loading. So I am putting the notebook below, please take a look.

import os
import logging
import shutil
import yaml
import pandas as pd
import numpy as np
from ludwig.api import LudwigModel
from ludwig.utils.data_utils import load_json
from ludwig.visualize import learning_curves
from ludwig.visualize import roc_curves
from ludwig.hyperopt.run import hyperopt
from ludwig.hyperopt.utils import update_hyperopt_params_with_defaults
from ludwig.visualize import learning_curves, compare_performance, confusion_matrix, compare_classifiers_performance_from_pred, roc_curves_from_test_statistics
from ludwig.visualize import hyperopt_results_to_dataframe, hyperopt_hiplot_cli, hyperopt_report_cli
from ludwig.data.preprocessing import preprocess_for_training

```python
for name in ['training', 'testing']:
with open('/content/drive/My Drive/ColabNotebooks/mnist_ludwig/mnist_dataset_{}.csv'.format(name), 'w') as output_file:
print('=== creating {} dataset ==='.format(name))
output_file.write('image_path,label\n')
for i in range(10):
path = '/content/drive/My Drive/ColabNotebooks/mnist_ludwig/mnist_png/{}/{}'.format(name, i)
for file in os.listdir(path):
if file.endswith(".png"):
output_file.write('{},{}\n'.format(os.path.join(path, file), str(i)))

```python
train_df = pd.read_csv('/content/drive/My Drive/ColabNotebooks/mnist_ludwig/mnist_dataset_training.csv')
train_df.shape

```python
with open('/content/drive/My Drive/ColabNotebooks/mnist_ludwig/model_definition.yaml','r') as f:
config = yaml.safe_load(f.read())

train_df.dtypes

```python
def training():
    shutil.rmtree('/content/drive/My Drive/ColabNotebooks/mnist_ludwig/results', ignore_errors=True)

    with open('/content/drive/My Drive/ColabNotebooks/mnist_ludwig/model_definition.yaml','r') as f:
        model_definition = yaml.safe_load(f.read())

    model = LudwigModel(model_definition, logging_level=logging.INFO)

    train_stats, _, output_directory = model.train(
        training_set='/content/drive/My Drive/ColabNotebooks/mnist_ludwig/mnist_dataset_training.csv',
        test_set='/content/drive/My Drive/ColabNotebooks/mnist_ludwig/mnist_dataset_testing.csv',
        experiment_name='simple_image_experiment',
        model_name='single_model',
        output_directory='/content/drive/My Drive/ColabNotebooks/mnist_ludwig/results',
        skip_save_processed_input=True
    )

    visualize()

```python
def visualize():
test_data = {'image_path': [], 'label': []}
current_dir = os.getcwd()
test_dir = os.path.join(current_dir, 'drive', 'My Drive', 'ColabNotebooks', 'mnist_ludwig', 'mnist_png', 'testing')
print(test_dir)
for label in os.listdir(test_dir):
files = os.listdir(os.path.join(test_dir, label))
test_data['image_path'] += [os.path.join(test_dir, label, f) for f in files]
test_data['label'] += len(files) * [label]

# collect data into a data frame
test_df = pd.DataFrame(test_data)

test_stats_list = []
preds_list = []

experiment_model_dir = '/content/drive/My Drive/ColabNotebooks/mnist_ludwig/results/simple_image_experiment_single_model/model' 

# print(train_stats['training'])
model = LudwigModel.load(experiment_model_dir)
test_stats, pred_df, _ = model.evaluate(dataset=test_df, collect_predictions=True, collect_overall_stats=True)
print(pred_df)
preds_list.append(pred_df['label_predictions'].astype('int'))
test_stats_list.append(test_stats)

train_metadata_json = load_json('/content/drive/My Drive/ColabNotebooks/mnist_ludwig/results/simple_image_experiment_single_model/model/training_set_metadata.json')

confusion_matrix(
  test_stats_list,
  train_metadata_json,
  'label',
  [10,10,10],
  False,
  output_directory='./visualizations',
  file_format='png'
)

compare_classifiers_performance_from_pred(
  preds_list,
  test_df['label'].to_numpy().astype('int'),
  train_metadata_json,
  'label',
  10,
  output_directory='./visualizations',
  file_format='png'
)

```python
training()

```python
import yaml

with open('/content/drive/My Drive/ColabNotebooks/mnist_ludwig/model_definition.yaml','r') as f:
config = yaml.safe_load(f.read())
f.close()
config

SEED=13

hyperopt_configs = {
"parameters": {
"training.learning_rate": {
"type": "float",
"low": 0.0001,
"high": 0.01,
"space": "log",
"steps": 3,
},
"training.batch_size": {
"type": "int",
"low": 32,
"high": 256,
"space": "log",
"steps": 5,
"base" : 2
},
"label.fc_size": {
"type": "int",
"low": 32,
"high": 256,
"steps": 5
},
"label.num_fc_layers": {
'type': 'int',
'low': 1,
'high': 5,
'space': 'linear',
'steps': 4
}
},
"goal": "minimize",
'output_feature': "label",
'validation_metrics': 'loss'
}

add hyperopt parameter space to the config

config['hyperopt'] = hyperopt_configs
config

```python
# clean out old results
shutil.rmtree('/content/drive/My Drive/ColabNotebooks/mnist_ludwig/results_random_parallel', ignore_errors=True)
shutil.rmtree('/content/drive/My Drive/ColabNotebooks/mnist_ludwig/results_random_serial', ignore_errors=True)
shutil.rmtree('/content/drive/My Drive/ColabNotebooks/mnist_ludwig/results_grid_parallel', ignore_errors=True)

preprocess_for_training(config, dataset='/content/drive/My Drive/ColabNotebooks/mnist_ludwig/mnist_dataset_training.csv')

config['hyperopt']['executor'] = {'type': 'parallel', 'num_workers': 4}
config['hyperopt']['sampler'] = {'type': 'random', 'num_samples': 10}
random_parallel_results = hyperopt(
    config,
    dataset='/content/drive/My Drive/ColabNotebooks/mnist_ludwig/mnist_dataset_training.csv',
    output_directory='/content/drive/My Drive/ColabNotebooks/mnist_ludwig/results_random_parallel'
)

[Juggernaut1997]

Hi @jimthompson5802, Can you tell, how much time will it take to run this section of code?

config['hyperopt']['executor'] = {'type': 'parallel', 'num_workers': 4}
config['hyperopt']['sampler'] = {'type': 'random', 'num_samples': 10}
random_parallel_results = hyperopt(
    config,
    dataset='/content/drive/My Drive/ColabNotebooks/mnist_ludwig/mnist_dataset_training.csv',
    output_directory='/content/drive/My Drive/ColabNotebooks/mnist_ludwig/results_random_parallel'
)

[jimthompson5802]

Without knowing the config settings and your setup, it is hard to say. The primary driver for run-time will be the amount of data and the number of epochs. Do you have sufficient cpus to support 4 parallel tasks?

To help you, I've been using a 20% subset of training data. 10 random samples, 4 parallel workers with 5 epochs takes about 5 minutes on my MacBook with 4 cores (8 threads of execution). If you'r doing the full size, I'd multiple the time by 5. That should put you in the ball park.

[w4nderlust]

It also depends on your gpu availability.

[Juggernaut1997]

Hi @w4nderlust @jimthompson5802 ,

This solution worked. It just took an hour to process, which is fine because we have an image dataset. It will take time to process.
So thank you for your time and solution you guys provided.

Thank you once again and be safe!
😉 👍

[w4nderlust]

That's great to hear! We'll keep the issue open as we still need to fully solve it, but glad the workaround helped!