Azure-docs: DataLake Gen2 Az Module API

[DigitalNightmare]

Hello,

I am just wondering if there is going to be an API using the Az Module for powershell to interact with the datalake gen2 product. Currently right now. I think the only two ways to interact are manually through the explorer, and through Azure CLI.

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 8ab1d75b-3cbf-58f2-6d6f-8f7445fc27a7
• Version Independent ID: 9d4e2945-27ec-b88e-7d20-0693df6982ea
• Content: Azure Data Lake Storage Gen2 Introduction
• Content Source: articles/storage/blobs/data-lake-storage-introduction.md
• Service: storage
• GitHub Login: @jamesbak
• Microsoft Alias: jamesbak

[AdamS-MSFT]

@ThirstyAdmin Thank you for the valuable feedback,we are investigating the issue.

[CHEEKATLAPRADEEP-MSFT]

@ThirstyAdmin To interact with Data Lake Storage Gen2 by using PowerShell, you'll need to install module Az.Storage version 0.7 or later.

• You can create an account using the Azure portal, Azure PowerShell, or via the Azure CLI.
• To view or manage Data Lake Storage Gen2 accounts by using Azure Storage Explorer.

For more details, refer "ADLS Gen2 - Quickstart" and “Known issues with ADLS Gen2”.

Hope this helps.

[DigitalNightmare]

Hello,

That doesn't really help. What i am trying to do is create an azure devops release pipeline that will let me create the Storage account Gen 2. Once created I am looking to have a step that creates my file system, and then folders in that file system. Once i have the folders created i am looking to set permissions on the folders based on azure ad groups. I was able to do this with Gen1.

I am looking to do all of this with Powershell steps.

what i see if i search for any commands that have data lake in the name

what i see if i search for any commands with new-azstorage in the name

i've loaded the az.storage module but i don't see a way to create a new filesystem via the cmdlets.

If i try to create a container under the storage account i have i get the following message. So all i am trying to ask in this post is when can we expect this functionality :)

[CHEEKATLAPRADEEP-MSFT]

@ThirstyAdmin Note: Blob storage APIs aren't yet available to Azure Data Lake Storage Gen2 accounts.

These APIs are disabled to prevent inadvertent data access issues that could arise because Blob Storage APIs aren't yet interoperable with Azure Data Lake Gen2 APIs.

For more details, refer “Azure Data Lake Gen2: Blob storage APIs”.

Hope this helps.

[DigitalNightmare]

@CHEEKATLAPRADEEP-MSFT

That's ok :) i am just confirming that there is no API that i can currently use with Datalake Gen2. And am really just asking when / if they will be available soon or not, as setting up the security even if i'm using AAD groups per folder is very time consuming and tedious. I just want to speed up is all.

i read the line: Blob Storage APIs and Azure Data Lake Gen2 APIs aren't interoperable with each other.

for the life of me i cannot find a Data Lake Gen2 API. All i see is Blob storage APIs which i can't use due to having hierarchical namespace enabled. Are these "Azure Data Lake Gen2" APIs available in the new Az module by chance?

Thanks,

[CHEEKATLAPRADEEP-MSFT]

@jamesbak Could you please look into this issue?

[Jyoti492]

@CHEEKATLAPRADEEP-MSFT and @ThirstyAdmin :: I am trying to create the same. Did you get any method to do so ??

[Jyoti492]

Through the rest API

[DigitalNightmare]

Hey Just following up,

I have hierarchical namespace enabled and going through the REST API listed in this document https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/filesystem/create

will not work as i receive the following error message:
Invoke-RestMethod : BlobApiNotYetSupportedForHierarchicalNamespaceAccountsBlob API is not yet supported for hierarchical namespace accounts.

Is there a way to create the "FileSystem" through any means other than manually, while using hierarchical namespace?

Thanks,

[MisiekBest]

Is there a way to create the "FileSystem" through any means other than manually, while using hierarchical namespace?

Thanks,

@ThirstyAdmin , look at my blog post:

http://sql.pawlikowski.pro/2019/03/10/connecting-to-azure-data-lake-storage-gen2-from-powershell-using-rest-api-a-step-by-step-guide/

I created an example on how to connect to ADLS Gen2 API from PowerShell using REST and how to create a filesystem.

Hope it will help you.
Regards,
m.

[CHEEKATLAPRADEEP-MSFT]

@MisiekBest Thanks for adding your insights for the issue, which might be beneficial to other community members reading this thread.

[DigitalNightmare]

Thank you very much @MisiekBest :)

[CHEEKATLAPRADEEP-MSFT]

We will now proceed to close this thread. If there are further questions regarding this matter, please comment and we will gladly continue the discussion.

[awaemmanuel]

It there support to create the filesystem via ARM now please? My scenario, using ARM, is to deploy the AML Workspace with a StorageV2, create a filesystem and then use that filesystem as ADLS v2 filesystem.

[jamesbak]

Hi @awaemmanuel,

Yes, with the release of Multi-Protocol Access the ARM container-level operations are now available for accounts with the Hierarchical Namespace enabled.

HTH,
James