Azure-sdk-for-python: Persist logs/outputs to linked blob storage

[little-eyes]

Hi,

I'm running batch with docker container workload, but I want to upload the job and task's output/errors/logs to a linked blob storage account. Is this possible using python sdk? Can you give an example?

[annatisch]

Hi @little-eyes - yes this should be possible using the storage SDK for Python.
However Batch already supports automatically uploading the output/errors/logs of the workload to storage via declaration in the API - could this not be used instead?

Reference docs:
https://docs.microsoft.com/en-us/rest/api/batchservice/task/add#outputfile
Article:
https://docs.microsoft.com/en-us/azure/batch/batch-task-output-files
Python docs (look for parameter output_files):
https://docs.microsoft.com/en-us/python/api/azure.batch.models.taskaddparameter?view=azure-python

If this feature looks like it will meet you requirements I can put together a snippet demonstrating it's use.

[little-eyes]

@annatisch thanks a lot. do you know what's the working directory it runs on Ubuntu? I tried to copy the std*.txt files, but it seems to succeed with nothing copied.

    new_task = batchmodels.TaskAddParameter(
        id=task_id,
        command_line=wrap_commands_in_shell("linux", job_tasks),
        output_files=[
            batchmodels.OutputFile(
                "std*.txt", 
                batchmodels.OutputFileDestination(container=batchmodels.OutputFileBlobContainerDestination(log_container_url, path=task_id)),
                batchmodels.OutputFileUploadOptions(batchmodels.OutputFileUploadCondition.task_completion))],
        user_identity=batchmodels.UserIdentity(
            auto_user=batchmodels.AutoUserSpecification(
                scope=batchmodels.AutoUserScope.task, 
                elevation_level=batchmodels.ElevationLevel.admin)))

[annatisch]

No problem @little-eyes - I believe the stdout and stderr files are a directory above the working directory (for both Windows and Linux).
I have used this in the past which seems to work:
"filePattern": "../stdout.txt",

[annatisch]

Hi @little-eyes - did this work out for you? If so I'll close the issue :)

[little-eyes]

@annatisch, yes, it works. Thanks a lot!