Sql-docs: File referenced by --file needs to reside in HDFS

[dgshaver]

Hi, the documentation for the --file argument for azdata bdc spark batch create needs to be updated to reflect the requirement the file reside in HDFS. If one references a file from the local drive of the workstation where the command is executed, the command fails with a 400, Bad Request.
This command works:
C:\Users\sql2019css>azdata bdc spark batch create -f hdfs://nmnode-0/scripts/pi.py

These commands that reference the pi.py file on the local disk of the workstation where the command was submitted return a 400 (Bad Request):
C:\Users\sql2019css>azdata bdc spark batch create -f file:///c:/users/sql2019css/pi.py
C:\Users\sql2019css>azdata bdc spark batch create -f pi.py

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 6600107a-20b8-6ee3-1616-4c6ca1c06159
• Version Independent ID: 46981b99-72b9-16d4-bfe3-cc383708f26d
• Content: azdata bdc spark batch reference - SQL Server big data clusters
• Content Source: docs/big-data-cluster/reference-azdata-bdc-spark-batch.md
• Product: sql
• Technology: big-data-cluster
• GitHub Login: @MikeRayMSFT
• Microsoft Alias: mikeray

[WilliamAntonRohm]

@dgshaver -- thank you for your feedback.

@MikeRayMSFT -- please look into this issue 3606.

[MikeRayMSFT]

Thanks! @dgshaver -- investigating.

[ev2900]

Was this issue resolved?

I am looking at the documentation for azdata bdc spark batch create. It does not include any information that that file must be located on the HDFS of the BDC cluster and not on the local host.

[dgshaver]

It doesn’t appear so. I think Mike discussed it with the developer that owns the feature, but I don’t recall what the outcome was.

Thanks,
Dan Shaver
Microsoft Big Data and Analytics Support
980 776 7817
Work schedule: M – F, 08:00 to 17:00 US Eastern Standard Time Zone

From: Christopher Sharkey [e-mail]
Sent: Monday, June 1, 2020 1:31 PM
To: MicrosoftDocs/sql-docs [e-mail]
Cc: Dan Shaver [e-mail]; Mention [e-mail]
Subject: Re: [MicrosoftDocs/sql-docs] File referenced by --file needs to reside in HDFS (#3606)

Was this issue resolved?

I am looking at the documentation for azdata bdc spark batch createhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fsql%2Fbig-data-cluster%2Freference-azdata-bdc-spark-batch%3Fview%3Dsql-server-ver15%23azdata-bdc-spark-batch-create&data=02%7C01%7Cdansha%40microsoft.com%7C6ada40fadb8340508b6308d806517cc6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637266294340982551&sdata=2UOLEKDcpVir%2BCzHAXiZq%2FNNegLEOTeT5mkZBbvPzeU%3D&reserved=0. It does not include any information that that file must be located on the HDFS of the BDC cluster and not on the local host.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fsql-docs%2Fissues%2F3606%23issuecomment-637002367&data=02%7C01%7Cdansha%40microsoft.com%7C6ada40fadb8340508b6308d806517cc6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637266294340992553&sdata=tbvSrSfkasaPstyAoGrvGan9z9NJQUiBhFITJzib66Q%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAA4YAX3WXAWILTY5D25PO3LRUPQTRANCNFSM4JN7LJ3A&data=02%7C01%7Cdansha%40microsoft.com%7C6ada40fadb8340508b6308d806517cc6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637266294340992553&sdata=M1laDorJq3kRaYZmBg8BeYsKxz5Me2TjmR7gpOVFlfI%3D&reserved=0.

[ev2900]

Ok should I reopen the issue then or do you feel that it isn't appropriate to include this information in the documentation?

[dgshaver]

I think it should be documented.

Thanks,
Dan Shaver
Microsoft Big Data and Analytics Support
980 776 7817
Work schedule: M – F, 08:00 to 17:00 US Eastern Standard Time Zone

From: Christopher Sharkey [e-mail]
Sent: Tuesday, June 2, 2020 10:50 AM
To: MicrosoftDocs/sql-docs [e-mail]
Cc: Dan Shaver [e-mail]; Mention [e-mail]
Subject: Re: [MicrosoftDocs/sql-docs] File referenced by --file needs to reside in HDFS (#3606)

Ok should I reopen the issue then or do you feel that it isn't appropriate to include this information in the documentation?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fsql-docs%2Fissues%2F3606%23issuecomment-637592727&data=02%7C01%7Cdansha%40microsoft.com%7C3ca3c17bd605441dce0508d80704365d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637267061966362152&sdata=mavntA3EK2%2FfIWNuZviCqOYMGhP07MvCNfW%2BCd8Q7ag%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAA4YAX62LFE2W3IZVCOFRDLRUUGRFANCNFSM4JN7LJ3A&data=02%7C01%7Cdansha%40microsoft.com%7C3ca3c17bd605441dce0508d80704365d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637267061966362152&sdata=OOZw2ltCsMza6XwCUWIKV9zDuJzU0p%2FeCGtKh5yh6sE%3D&reserved=0.