Azure-docs: Warm Store Retention Clarification

[xtellurian]

When creating a new TSI, we can select a Warm Store option and choose a retention time from 7 - 31 days.

This documentation does not clarify how the retention time is calculated. Is the retention period calculated with respect to the ingestion time or event time-stamp?

Some specific questions for clarification:

• if an event is ingested with a timestamp from last year, will that event be in warm storage?
• if an event is ingested with a timestamp in the future, will that event be in warm storage?

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 19657c0f-adbb-bf24-207d-60e792403464
• Version Independent ID: 8f33278b-ea3d-870f-336d-9e9a47e995fe
• Content: Plan your Preview environment - Azure Time Series Insights
• Content Source: articles/time-series-insights/time-series-insights-update-plan.md
• Service: time-series-insights
• GitHub Login: @deepakpalled
• Microsoft Alias: dpalled

[xtellurian]

cc @darsney

[AshokPeddakotla-MSFT]

@xtellurian Thanks for the feedback! We are currently investigating and will update you shortly.

[AshokPeddakotla-MSFT]

@deepakpalled I haven't found more details regarding this question. Please check and provide your suggestions. It would be great if we can can add more details to the documentation for more clarity.

[KingdomOfEnds]

Hi @xtellurian , @AshokPeddakotla-MSFT . Thanks for the great question and Happy Holidays! I wanted to supply a few quick answers to help address the original question!

1. By default, the $ts (timestamp property) will be set to the _event enqueued time_. When using this default setting, events will be charted according to the time they arrive. Their _ingestion time_ can be retained as a column (and you can sort by that in a UI chart or filter by that properties using TSX as needed).
2. When you configure a TSI environment you may also choose a relevant property to be the $ts property. However, when I test this scenario in Preview - the older and future timestamps are overridden and replaced by the _event enqueued time_. I believe that's an intended feature in Preview at this time - I've reached out to verify that as well.

Thanks!

[xtellurian]

Hi @KingdomOfEnds

I'm using a custom timestamp property in a TSI preview environment. Future and Past dates work OK (although I haven't tried as far away as 2049). My custom timestamp property is called 't' and it doesn't show up in the events table in the explorer view

the older and future timestamps are overridden and replaced by the event enqueued time. I believe that's an intended feature

I can't believe that's the intended or actual behavior. Not intended, because it would make the custom timestamp defunct (always overridden), and not actual because I'm using a custom timestamp with past and figure values and it works OK.

My question relates to how and when values in the the warm store are added/ removed. For example, is there an expiry time for each row, after which the row is removed from warm store?

Thanks

[lyrana]

Hi @xtellurian, great question, and thank you for pointing out that the documentation is lacking an explanation for how retention is calculated.
Today warm store retention is evaluated based on ingested time--the time that TSI receives the event--irrespective of the timestamp value. Thus as you correctly guessed above after ingested time + configured retention period has passed the entry will be removed from WS. We are gathering additional feedback from customers who have different use-cases such as future/predictive data that they'd like to be able to query via warm store. But that is the current implementation in place today. Please let me know if you have additional questions.
Thanks and keep growing the potatoes,
Lyrana

[KingdomOfEnds]

@xtellurian - a quick clarificatory comment on my end:

I can't believe that's the intended or actual behavior. Not intended, because it would make the custom timestamp defunct (always overridden), and not actual because I'm using a custom timestamp with past and figure values and it works OK.

Correct! I meant that even though the instance had the custom timestamp configured as $ts it will be overridden by the time the event hits the resource. _If the times are also submitted as an independent column, they will appear nevertheless (per the images)._ Sorry for my lack of clarity there! Please let me know if there's anything else I can help with!

[AshokPeddakotla-MSFT]

@xtellurian Does the above suggestions answered your question? Do let us know if you have any other questions.

We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.