Sagemaker-python-sdk: Early stop for training

[PedroCardoso]

System Information

• **Framework
  Tensorflow
• Framework Version:
  1.8.0
• Python Version:
  2.7.14
• CPU or GPU:
  GPU and CPU
• Python SDK Version:
  1.7.0
• Are you using a custom image:
  No

Describe the problem

When running a training on Sagemaker, the train_and_evaluate is stoping before reaching the total training_steps . In reality it is stoping at the end of the training Dataset (defined with a specific number of epochs)

When running locally, there is an evaluation at the end of each training Dataset and after the evaluation it runs again up to the defined training_steps.

• Exact command to reproduce:

Nothing particular,

dnnModel = TensorFlow( entry_point=model_def_file,
                      requirements_file='requirements.txt',
                               role=role,
                               training_steps= training_steps,                                  
                               evaluation_steps= None,
                               hyperparameters=params,
                               train_instance_count=1,
                               train_instance_type='ml.p2.xlarge',
                               source_dir = libPath,
                             base_job_name='train-image-pet',
                     )
dnnModel.fit(s3_train_data, run_tensorboard_locally=False)

[yangaws]

Hi @PedroCardoso ，

Do you have logs for this?

[PedroCardoso]

Hi @yangaws

I have the logs, but they say nothing special.
No error, just a normal exit at the end of the Dataset for training for 5 epochs, even if the training_steps are defined for 30 epochs.

I can give you a Sagemaker Job ID if you have the means of checking. It is with a simple newsgroup20 training.

[ChoiByungWook]

Hmmm, this is going to be difficult to diagnose without seeing your entry_point script. Was this a problem in both CPU and GPU?

[zmjjmz]

This is likely not possible without upgrading your Tensorflow version to 1.10 due to the fact that train_and_evaluate (which is used by the Sagemaker TensorFlow estimator) didn't support arbitrary stopping conditions prior to that. See https://github.com/tensorflow/tensorflow/issues/18394

[laurenyu]

hi @PedroCardoso, can you provide us your entry script?

[khu834]

Are you using file mode or pipe mode?
In pipe mode, if you do not set PipeModeDataset.repeat(None) in the input function, the training will stop after 1 epoch.

[PedroCardoso]

I was checking on this .

I believe the reason for this was a change in Tensorflow's train_and_evaluate(). It was not respecting max_steps if the training dataset had an "early stop". This was the case if you specify a number to .repeat(). Before it would stop, evaluate the checkpoint and restart training. Now it just stops.
I was using it to evaluate every N epochs.

With the new version 1.10 I am allows to do it again, and so this is not longer a problem. I just need to have a repeat(None), set save_checkpoints_steps to the epoch I want and set a low throttle_secs.

So this is not a bug on Sagemaker, but on Tensorflow. And maybe not a bug, but a unexpected behaviour as max_steps does not respect the definition.

[PedroCardoso]

That said, I do think there is a little bug on Sagemaker.

If we define save_checkpoints_steps hyperparameter, we are forced to define also save_checkpoints_secs to None as it seems to have a default value of 300.

[zmjjmz]

@PedroCardoso -- I've noted the above issue here if you want something to follow. Also I realize that my earlier comment might not have been helpful because you were having issues with training stopping before you wanted it to, whereas I thought you were talking about early stopping which has a specific meaning in ML, so sorry for any confusion that may have caused.

[mvsusp]

Hello @zmjjmz and @PedroCardoso,

We are currently working in the next version of the TF container. It will cover the painpoints pointed here by you.

Thanks for the patience and let me know if you have any questions.