Azure-docs: Please could the example be made a bit more realistic? (Same data is written to warehouse/Capture directly to SQL database?)

[patrickjlee]

Thanks, this was an interesting tutorial. I would have preferred it if the example was a bit more realistic: at the moment, the data that is transferred to the transferred to the data warehouse (effectively an Azure SQL database) is identical to the data that is written (via the Event Hub's Capture setting) to Azure storage blob files. There seems little point to this, and indeed it raises the question as to whether Capture should include the option to writing directly to an Azure SQL database?

---

Document details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 9ec5477a-c4b4-a37c-f853-3d517a095e93
• Version Independent ID: 51b5ddf3-109d-4f9e-5d2d-afe7d46dab42
• Content: Azure Event Grid and Event Hubs integration
• Content Source: articles/event-grid/event-grid-event-hubs-integration.md
• Service: event-grid
• GitHub Login: @tfitzmac
• Microsoft Alias: tomfitz

[asergaz]

@patrickjlee Thanks for the feedback! I have assigned the issue to the content author to evaluate and update as appropriate.

[xurui203]

@patrickjlee Thank you for the feedback! As you correctly pointed out, Capture today only supports writing directly to Azure Storage or Data Lake, and not Azure SQL database (hence the use of an Azure Function).

SQL works great for structured data with relations and indexing. Azure Storage on the other hand stores large volumes of non-relational data at a more affordable price. As a big data streaming platform, we find that it makes more sense for Event Hubs to support Capturing the large volumes of streamed data directly to Storage simply because it is the most cost effective option for our users.

That being said, we understand that some Event Hubs customers would like to migrate their Event Hubs data to a SQL server, and this tutorial demonstrates how to do that using Capture and an Azure Function. Mapping the raw Avro files to a SQL data table requires users to format the data e.g. defining column names, which is how the Azure Function in this tutorial is transforming the data. There are definitely more creative ways you can modify the data, but we leave that to our users to experiment within the sample Function. If you have some good ideas on how to tweak this function, please feel free to create a PR, we rely on contributors like you to grow our community :)

We are always looking to improve and prioritize our features to best serve customer needs, if you use Capture for a scenario that we do not support today or have any suggestions on how we can improve we would love to hear more from you!

[tfitzmac]

@xurui203 - thanks for providing this insight.

@patrickjlee - we have some changes planned for this article, but we won't be changing how the Function works. However, like @xurui203 mentioned, if you want to provide changes, that is always appreciated.

[tfitzmac]

please-close