Sagemaker-python-sdk: DeepAR cardinality with empty string is not honored

[ChandraLingam]

Please fill out the form below.

System Information

• Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): SageMaker/DeepAR
• Framework Version: sagemaker notebook instance
• Python Version: conda_python3
• CPU or GPU: CPU
• Python SDK Version: conda_python3
• Are you using a custom image: No

Describe the problem

I have a DeepAR training set in JSON format. It has "cat" field specified as part of time series.
I wanted to train two models: one using categories and another without categories.

According to sagemaker DeepAR hyperparameters document, I can control training process to use or ignore categories using cardinality hyperparameter.

I specified these hyperparameters as part of the job
{'time_freq': 'H', 'epochs': '400', 'early_stopping_patience': '40', 'mini_batch_size': '64', 'learning_rate': '5E-4', 'context_length': '288', 'prediction_length': '288', 'cardinality': ''}

However, the training job is automatically setting cardinality to "auto" and using categories (even though I asked it to ignore it)

[12/16/2018 18:25:21 INFO 140678005200704] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/default-input.json: {u'num_dynamic_feat': u'auto', u'dropout_rate': u'0.10', u'mini_batch_size': u'128', u'test_quantiles': u'[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]', u'_tuning_objective_metric': u'', u'_num_gpus': u'auto', u'num_eval_samples': u'100', u'learning_rate': u'0.001', u'num_cells': u'40', u'num_layers': u'2', u'embedding_dimension': u'10', u'_kvstore': u'auto', u'_num_kv_servers': u'auto', u'cardinality': u'auto', u'likelihood': u'student-t', u'early_stopping_patience': u''}
[12/16/2018 18:25:21 INFO 140678005200704] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'learning_rate': u'5E-4', u'prediction_length': u'288', u'epochs': u'400', u'time_freq': u'H', u'context_length': u'288', u'mini_batch_size': u'64', u'early_stopping_patience': u'40'}
[12/16/2018 18:25:21 INFO 140678005200704] Final configuration: {u'dropout_rate': u'0.10', u'test_quantiles': u'[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]', u'_tuning_objective_metric': u'', u'num_eval_samples': u'100', u'learning_rate': u'5E-4', u'num_layers': u'2', u'epochs': u'400', u'embedding_dimension': u'10', u'num_cells': u'40', u'_num_kv_servers': u'auto', u'mini_batch_size': u'64', u'likelihood': u'student-t', u'num_dynamic_feat': u'auto', u'cardinality': u'auto', u'_num_gpus': u'auto', u'prediction_length': u'288', u'time_freq': u'H', u'context_length': u'288', u'_kvstore': u'auto', u'early_stopping_patience': u'40'}
[12/16/2018 18:25:21 INFO 140678005200704] [cardinality=auto] `cat` field was found in the file `/opt/ml/input/data/train/train.json` and will be used for training.
[12/16/2018 18:25:21 INFO 140678005200704] [num_dynamic_feat=auto] `dynamic_feat` field was NOT found in the file `/opt/ml/input/data/train/train.json` and will NOT be used for training.
[12/16/2018 18:25:21 INFO 140678005200704] [cardinality=auto] Inferred value of cardinality=[3] from dataset.

[12/16/2018 18:25:21 INFO 140678005200704] [cardinality=auto] Inferred value of cardinality=[3] from dataset.

Minimal repro / logs

Please provide any logs and a bare minimum reproducible test case, as this will be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

• Exact command to reproduce:

[ChoiByungWook]

Hello @ChandraLingam,

Can you provide the exact code you're using to call your DeepAR algorithm in SageMaker?

I am assuming that you're using SageMaker DeepAR by specifying this value: https://github.com/aws/sagemaker-python-sdk/blob/db1876d0191e831fc397a00a633f7080e94d44ca/src/sagemaker/amazon/amazon_estimator.py#L321

If that is the case, unfortunately it seems that because the value is "" it ends up being interpreted as None https://github.com/aws/sagemaker-python-sdk/blob/db1876d0191e831fc397a00a633f7080e94d44ca/src/sagemaker/amazon/hyperparameter.py#L70 and ends up not being passed to SageMaker properly. I am not too sure how the DeepAR image handles a hyperparameter that wasn't provided by default, however I imagine it defaults off.

[ChoiByungWook]

I understand that based on the docs. Making cardinality = "" should be honored. Can you please try passing None instead and see if you get the desired result?

[ChandraLingam]

default mode in deepar is "auto" (log file content pasted above shows this) and it automatically tries to infer from the data.
I can check passing None and see if that helps

[ChoiByungWook]

@ChandraLingam,

Awesome! Thank you so much! Please let us know.

[ChandraLingam]

@ChoiByungWook It does not work. Passing None as value for cardinality causes an error. Output attached below. I suspect same issue exists for num_dynamic_feat
https://docs.aws.amazon.com/sagemaker/latest/dg/deepar_hyperparameters.html

INFO:sagemaker:Creating training-job with name: deepar-biketrain-no-categories-experime-2018-12-18-13-57-14-225
2018-12-18 13:57:14 Starting - Starting the training job...
2018-12-18 13:57:27 Starting - Launching requested ML instances......
2018-12-18 13:58:27 Starting - Preparing the instances for training......
2018-12-18 13:59:25 Downloading - Downloading input data..
Arguments: train
[12/18/2018 13:59:59 INFO 139749802891072] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/default-input.json: {u'num_dynamic_feat': u'auto', u'dropout_rate': u'0.10', u'mini_batch_size': u'128', u'test_quantiles': u'[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]', u'_tuning_objective_metric': u'', u'_num_gpus': u'auto', u'num_eval_samples': u'100', u'learning_rate': u'0.001', u'num_cells': u'40', u'num_layers': u'2', u'embedding_dimension': u'10', u'_kvstore': u'auto', u'_num_kv_servers': u'auto', u'cardinality': u'auto', u'likelihood': u'student-t', u'early_stopping_patience': u''}
[12/18/2018 13:59:59 INFO 139749802891072] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'mini_batch_size': u'64', u'learning_rate': u'5E-4', u'prediction_length': u'288', u'epochs': u'400', u'time_freq': u'H', u'context_length': u'288', u'cardinality': u'None', u'early_stopping_patience': u'40'}
[12/18/2018 13:59:59 INFO 139749802891072] Final configuration: {u'dropout_rate': u'0.10', u'test_quantiles': u'[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]', u'_tuning_objective_metric': u'', u'num_eval_samples': u'100', u'learning_rate': u'5E-4', u'num_layers': u'2', u'epochs': u'400', u'embedding_dimension': u'10', u'num_cells': u'40', u'_num_kv_servers': u'auto', u'mini_batch_size': u'64', u'likelihood': u'student-t', u'num_dynamic_feat': u'auto', u'cardinality': u'None', u'_num_gpus': u'auto', u'prediction_length': u'288', u'time_freq': u'H', u'context_length': u'288', u'_kvstore': u'auto', u'early_stopping_patience': u'40'}
Process 1 is a worker.
[12/18/2018 13:59:59 INFO 139749802891072] Detected entry point for worker worker
[12/18/2018 13:59:59 ERROR 139749802891072] Customer Error: Hyperparameter cardinality has to be an int or array of int. Found: None

2018-12-18 14:00:04 Training - Training image download completed. Training in progress.
2018-12-18 14:00:04 Uploading - Uploading generated training model
2018-12-18 14:00:04 Failed - Training job failed
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-19-65df836a5860> in <module>()
      1 # This step takes around 35 minutes to train the model with m4.xlarge instance
----> 2 estimator.fit(inputs=data_channels)

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name)
    232         self.latest_training_job = _TrainingJob.start_new(self, inputs)
    233         if wait:
--> 234             self.latest_training_job.wait(logs=logs)
    235 
    236     def _compilation_job_name(self):

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in wait(self, logs)
    571     def wait(self, logs=True):
    572         if logs:
--> 573             self.sagemaker_session.logs_for_job(self.job_name, wait=True)
    574         else:
    575             self.sagemaker_session.wait_for_job(self.job_name)

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in logs_for_job(self, job_name, wait, poll)
   1126 
   1127         if wait:
-> 1128             self._check_job_status(job_name, description, 'TrainingJobStatus')
   1129             if dot:
   1130                 print()

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in _check_job_status(self, job, desc, status_key_name)
    826             reason = desc.get('FailureReason', '(No reason provided)')
    827             job_type = status_key_name.replace('JobStatus', ' job')
--> 828             raise ValueError('Error for {} {}: {} Reason: {}'.format(job_type, job, status, reason))
    829 
    830     def wait_for_endpoint(self, endpoint, poll=5):

ValueError: Error for Training job deepar-biketrain-no-categories-experime-2018-12-18-13-57-14-225: Failed Reason: ClientError: Hyperparameter cardinality has to be an int or array of int. Found: None

[ChoiByungWook]

Hello @ChandraLingam,

Just for clarification, I assumed that "" would be interpreted as None, however it seems that "" is not None. You stated "" still ends up defaulting?

Most likely that code line isn't the problem.

[ChandraLingam]

yes, "" is ignored and default configuration is used. If you look at this line pasted in my original message, cardinality with "" is not even logged here. Looks like some sdk/sagemaker code in-the-middle is removing the hyperparameters with value set to "":

[12/16/2018 18:25:21 INFO 140678005200704] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'learning_rate': u'5E-4', u'prediction_length': u'288', u'epochs': u'400', u'time_freq': u'H', u'context_length': u'288', u'mini_batch_size': u'64', u'early_stopping_patience': u'40'}

[nadiaya]

Could you post the hyperparameters and their values from SageMaker console if you got to deepar-biketrain-no-categories-experime-2018-12-18-13-57-14-225 training job?

[laurenyu]

I was able to reproduce this using the DeepAR notebook and adding an empty string for cardinality. The logs show it defaulting to auto and the console showed no value whatsoever for cardinality. I then tried to clone the training job in the SageMaker Console (bypassing the Python SDK) and add "" or '', both of which were rejected by the regex-based validation. Leaving it empty and creating a new job led to cardinality being set to auto once again.

I also took a look through the code path, and the only function that's being called on any hyperparameter is str(), but str('') still returns '', so it doesn't seem like there's anything with the Python SDK that's causing the empty string to tell SageMaker to use the default auto instead.

I'll pass this along to the team that owns SageMaker's DeepAR support and see if they have any insight.

[jaheba]

Hello @ChandraLingam,

I'm sorry to hear that you are having troubles to force DeepAR to ignore the categorical features that you are providing in the data.

As a first workaround I would suggest to use a non-empty string (i.e. a space) as the value for the cardinality parameter like this:

    ...
    "cardinality": " ",
    ...

We've discussed the issue in the team and are planning to add a more explicit way to ignore provided features in the dataset.

Why didn't it work as expected?

The boto3-API for sagemaker ignores fields that are just empty-strings. Thus setting a hyper-parameter to an empty-value result in it being ignored and not send to the training job:

boto3.client("sagemaker").create_training_job(
    ...
    HyperParameters={
        # this will be ignored
        "cardinality": ""
    },
    ...
)

Since the sagemaker-python-sdk uses boto3 internally, the cardinality field was not sent to the DeepAR training-job.

[ChandraLingam]

@jaheba

We've discussed the issue in the team and are planning to add a more explicit way to ignore provided features in the dataset.

yes, that is the right approach.

For now, I created separated datasets one with categories and one without categories. Thank you for looking into the issue and getting to the root cause.