Sagemaker-python-sdk: Cannot modify SM_MODULE_DIR so cannot control output location of source.tar.gz on s3

[drewthayer]

Reference: MLFW-2737

This is an issue with a SageMaker python SDK design element, not a particular process.

Problem: The way Sagemaker currently works, I get folders written in my output s3 bucket at different levels, creating a cluttered bucket. I would like to:

• High level: control the output location for source.tar.gz so I can write this file to the same output location (e.g. on s3) as the model artifacts (everything in /opt/ml/model on the training instance)
• lower level: a way to set the environment variable SM_MODULE_DIR, which sets the output location for source.tar.gz. AFAIK this variable cannot be changed.

Specifics

• I can set MODEL_DIR with an output_path argument passed to Estimator(), and everything in opt/ml/model will be conveniently zipped and uploaded to MODEL_DIR. That's great.
• specifically, SM_HP_MODEL_DIR = s3://bucket/project/model_name/model/
  
  
  
  • (where model_name = estimator.latest_training_job.name)
  
  
• I want the source.tar.gz and tensor board files to save to s3://bucket/project/model_name/source/
• instead, source.tar.gz is being saved to SM_MODULE_DIR, and I we cannot set a subfolder for this path; it seems like it only writes to the root bucket: s3://bucket/model_name/source/

The result is a cluttered bucket:

bucket/
-project/
    - model_1/ (with model files from /opt/ml/model/)
    - model_2/
    - model_3/
    - etc...
- model_1/ (with source/)
- model_2/
- model_3/
- etc...

I can't save the directories with source/ inside project/. Any chance we could get a --module-dir cmd line arg, or another way to modify the SM_MODULE_DIR environment variable?

[ajaykarpur]

Hi @drewthayer, thanks for the suggestion. I'll add a backlog item to allow for more control over the output location to avoid creating a cluttered bucket.

Reference: MLFW-2737