Azure-docs: Get Meta Data Activity - Azure File Storage - Child Items

[vishalaj1]

Hi Team,

I am using Azure File Storage as source. I have both Folders and Files in the Azure File Storage Folder.

In childitems i only get the File List. I do not get the folder items.

Is this exepcted behaviour.

Regards,
Vishal

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 8e4997d4-36fd-4f1c-464b-ea3190531e02
• Version Independent ID: ebf216ca-d096-549f-3dcc-9442d7c4a8dc
• Content: Get metadata activity in Azure Data Factory
• Content Source: articles/data-factory/control-flow-get-metadata-activity.md
• Service: data-factory
• GitHub Login: @linda33wj
• Microsoft Alias: jingwang

[ChiragMishra-MSFT]

@vishalaj1 Get Metadata activity with ChildItems in the field list is meant to give as an output all the child items (files and subfolders) within a particular folder. If you have a folder(that you set in the linked service) which contains files and subfolders, you would get a list of all of the files and subfolders.
Eg- My dataset is an Azure file storage account with folders - ".ssh", "data", "LogFiles", "site" and a file named "eventlog.xml". The output of the Get Metadata activity with Child Items is :
{ "childItems": [ { "name": ".ssh", "type": "Folder" }, { "name": "data", "type": "Folder" }, { "name": "eventlog.xml", "type": "File" }, { "name": "LogFiles", "type": "Folder" }, { "name": "site", "type": "Folder" } ], "effectiveIntegrationRuntime": "DefaultIntegrationRuntime (East US)", "executionDuration": 93 }

Please note that you won't get the list of contents of subfolders (as it gives a list of the immediate child items only).

Hope this helps.

[ChiragMishra-MSFT]

@vishalaj1 I will now proceed to close the issue. Please feel free still to comment and tag me if you want any further clarification.

[vishalaj1]

Hi,

Thanks for your feedback.

I think i realised the issue here:

Case when it works:

When the Data Set is not provided any Modified Start Date and End Date.

There are two things seems to be incorrect(unless it is expected behaviour):

1. We have main folder A modified date is 20190121
   It has following :

".ssh" - Modified - 20190117
"data" - Modified - 20190118
"LogFiles", - Modified - 20190118
"site" - Modified - 20190121
eventlog.xml - Modified - 20190121

When we set data set and give the modified start and end date to it
20190118 to 20190121

Returns BLANK child items.

1. Issue is more related to Storage Sync Service. Folder's modified date is not synchronized to what is on premise when we are using Azure File Sync.

Regards,
Vishal

[vishalaj1]

Hi,

I did not understand, how this is closed?

Regards,
Vishal

[vishalaj1]

@ChiragMishra-MSFT

I did not understand how this is closed?

Can you confirm if this does not work with Modified Start and End Date?

If yes, it is much better if the document clearly mentions that.

Regards,
Vishal

[ChiragMishra-MSFT]

@vishalaj1 I had closed the issue expecting that the issue was resolved since there has been no response from you for 5 days.

The way it works is that in the dataset, you give the folder name (A in your case). There's an option to filter the contents (files and folders) based on the last modified date using a start date and end date. Hence, taking an example of your case :

If you have a folder named "A",

A's modified date is 20190121
It has following :
".ssh" - Modified - 20190117
"data" - Modified - 20190118
"LogFiles", - Modified - 20190118
"site" - Modified - 20190121
eventlog.xml - Modified - 20190121

If you set filters for start and end times as 20190118 and 20190121 respectively, it should ideally output eventlog.xml only. This is so because the filters based on modifiedTime are meant to get new and changed files only. To read more, please refer this doc.

Hence with filters, this is expected behavior. To be double sure, I will now assign the issue to the content owner and work internally with the team and let you know if it could be a bug.

[vishalaj1]

Hi,

I tried again:

When we give dates for dataset results are blank for childitems.

Regards,
Vishal

[ChiragMishra-MSFT]

@vishalaj1 I have assigned the issue to the content owner and work internally with the team and let you know if it could be a bug.

[vishalaj1]

@ChiragMishra-MSFT

Hi,

Any update on this?

Regards,
Vishal

[ChiragMishra-MSFT]

@vishalaj1 It looks like its expected behavior to see files only(and not subfolders) when you filter using modifiedTime.

@linda33wj To provide more information.

[ghost]

I had the same issue with SFTP dataset. Subfolders are not listed if dataset has a modified date filter.

@ChiragMishra-MSFT, if this behaviour is expected, it seems that documentation is out of date then. I could not find any references to it behaving like that.

[linda33wj]

Thanks all for the inquiries and feedback. You are right that when modified date filter is used, only files will be returned in child items but no sub-folders. We have clarify the behavior in the doc - see note at https://docs.microsoft.com/en-us/azure/data-factory/control-flow-get-metadata-activity#metadata-options.

please-close

[artsheiko]

Same here.
Get Metadata with "Child Items" argument does not list subfolders.
Eg : My dataset is an Azure file Storage account with folders - "data", and a file named "eventlog.xml". The output of the Get Metadata activity with Child Items is :
"childItems": [{ "name": "eventlog.xml", "type": "File" }]

[artsheiko]

Strange, but it seems that now everything worked - I can’t reproduce the case

[vishalaj1]

Strange, but it seems that now everything worked - I can’t reproduce the case

"Are you giving LastModifiedStartDate values?"