Machinelearningnotebooks: AML large file support doc link

[pospanet]

Your total project size exceeds the limit of 300.0 MB.
Please see http://aka.ms/aml-largefiles on how to work with large files.

link above pointing to this page, but nothing related to large file is mentioned here

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: c2464cc9-aef4-8929-142f-a1ce7ae69d78
• Version Independent ID: 7b46b2e5-965f-1a0e-7c35-686900196f6e
• Content: azureml.core.Run class - Azure Machine Learning Python
• Content Source: AzureML/docs-ref-autogen/azureml-core/azureml.core.Run(class).yml
• Service: machine-learning
• Sub-service: core
• GitHub Login: @j-martens
• Microsoft Alias: jmartens

[chengyin38]

Having the same problem as well. Came across this: https://github.com/MicrosoftDocs/azure-docs/issues/26076. But not sure where to add snapshot_directory=None.

[Jess0-0]

Having the same problem as well and also not sure where to add snapshot_directory=None. @j-martens wondering if you can expand on that a little bit. My code that triggers this issue looks like this

pipeline = Pipeline(workspace=ws, steps=[batch_score_step])
pipeline_run = Experiment(ws, experiment_name).submit(pipeline, pipeline_params={"param_batch_size": 8}) 

Thanks!

[j-martens]

Thanks for your posts. Adding others to help get answers.

@rastala @sanpil

[nibaccam]

@nibaccam

[sanpil]

Adding @heatherbshapiro for the AKA link and the message.
Please make sure you use a sub directory for all the associated files for a given step so we are under 300 MB. If there are bigger data files, then should be in the data store.

[Jess0-0]

After removing all the files in the subdirectory, the code is able to run further. Thank you very much for your help!

[bootsinthecity]

Hi there - I'm experiencing the same issue. I can create the workspace and the experiment just fine. I also followed the guidance here:
https://github.com/MicrosoftDocs/azure-docs/issues/26076

To no avail. I thought perhaps my environment was causing issues because I used conda to create most of it but then needed to use pip to install azureml-sdk. The new environment did not fix it.

I could see the kernels "running" but they were empty. I created new workspaces and experiments several times.

[sanpil]

Are you creating an Estimator object? Then can you put the script and the dependent files in a subdirectory and specify that directory as the source_directory? I assume that currently, you local directory which is more than 300MB is used as the source_directory by default.

[bootsinthecity]

Hi Sanpil,
I am trying to follow the instructions listed here:
https://docs.microsoft.com/en-us/learn/modules/intro-to-azure-machine-learning-service/6-train-model

If I need to change the source_directory, I'm happy to do that, but is there documentation with details on how to? I'm still learning azure ml service.

[sanpil]

What is the directory from which you are launching Jupyter notebook? Before starting Jupyter Notebook (step 6), create a new sub directory and the cd to that newly created directory. Then do step 6.

[bootsinthecity]

Ahh. Ok. I can do that - I have a ton of other stuff in this directory. Who should I reach out to as a suggestion that they include this detail in the instructions for the online learning module?

[bootsinthecity]

I created a new folder and ran it in an empty folder and it worked like a charm. Thanks so much for your help. I really appreciate it! 😃

[sanpil]

@j-martens

[j-martens]

Thanks to all who have posted here. I've added a defect to our documentation backlog to provide the following guidance if the 300MB error is encountered for snapshots:

1. Where to specify snapshot_directory=None and include an example.

1. Explain that users should put their scripts and the dependency files in a subdirectory and specify that directory as the source_directory. If there are bigger data files, then should be in the data store.

   • If you are using Jupyter notebooks, move your notebook into a new, empty, subdirectory and try again.

   • If you are using pipelines, you can use a different sub-directory for each step.

2. Fix the aka.ms link to point to better content about large files.

For experiments, we make a snapshot of your code (based on the directory you suggest when you configure the run). This has a total limit of 300mb.

This maximum can be overridden by setting the max size to anything you want like

import azureml._restclient.snapshots_client
azureml._restclient.snapshots_client. SNAPSHOT_MAX_SIZE_BYTES =

Also, you can make an ignore file to ignore files like local data files that are not really a part of the source code, as follows:

To prevent files from being included in the snapshot, create a .gitignore or .amlignore file in the directory and add the files to it. The .amlignore file uses the same syntax and patterns as the .gitignore file. If both files exist, the .amlignore file takes precedence.

If you are satified, #please-close.

[nibaccam]

The above referenced PR addresses this issue and the link http://aka.ms/aml-largefiles has been updated
@pospanet
If satisfied #please-close

[spacewaterbear]

Having the same problem as well. Came across this: MicrosoftDocs/azure-docs#26076. But not sure where to add snapshot_directory=None.

run = exp.start_logging(snapshot_directory=None)