Skywalking: 6.2.0 DataTTLKeeperTimer do not delete the expired records in the table segment

[shinelon]

Please answer these questions before submitting your issue.

• Why do you submit this issue?
• [ ] Question or discussion
• [x] Bug
• [ ] Requirement
• [ ] Feature or performance improvement

---

Bug

• Which version of SkyWalking, OS and JRE?
  SkyWalking 6.2.0
  storage Mysql 5.7.16
  jdk 1.8.0_77

• What happen?
  DataTTLKeeperTimer do not delete the expired records in the table segment.

the column time_bucket of table segment is second unit
but GeneralStorageTTL.java do not has SecondTTLCalculator

debug the 6.2.0 code :

sql param unit is minute
table column unit is seconds

---

### Requirement or improvement

• 
  add switch case Second to the GeneralStorageTTL.java can fix this bug??
  Will it cause another bug?

case Second:
return new SecondTTLCalculator();

[wu-sheng]

We are deleting index rather than delete record today.

[wu-sheng]

Are you talking about bug in Mysql storage?

[wu-sheng]

Could you test locally and submit a pull request?

[wu-sheng]

Cc @kezhenxu94 ttl e2e may be needed too. But a little complex, I think

@peng-yongsheng Your refator of TTL looks like breaking mysql

[yangvipguang]

类似问题：
skywalking: 6.2 GA
ES 6.8.1

我这边部署完成后，一开始页面数据都正常。过一会再打开页面就没数据了。查看日志时间点正好是如下日志：2019-07-12 15:26:33,020 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer - 61 [pool-17-thread-1] INFO [] - Beginning to remove expired metrics from the storage.
猜测原因： 删除了当前的metrics
疑问：
是否因为配置问题还是本身bug

暂时解决方案就是：关闭配置文件中的自动删除后，没有出现这种问题，系统时间是正常的。

[shook2012]

Suppressed: org.elasticsearch.client.ResponseException: method [DELETE], host [http://192.168.20.160:9200], URI [/skywalkingapm_skywalkingapm_alarm_record-20190720?master_timeout=30s&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true&timeout=30s], status line [HTTP/1.1 404 Not Found]

{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":"skywalkingapm_skywalkingapm_alarm_record-20190720","index":"skywalkingapm_skywalkingapm_alarm_record-20190720"}],"type":"index_not_found_exception","reason":"no such index","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":"skywalkingapm_skywalkingapm_alarm_record-20190720","index":"skywalkingapm_skywalkingapm_alarm_record-20190720"},"status":404}
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:705) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:522) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.IndicesClient.delete(IndicesClient.java:77) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.deleteIndex(ElasticSearchClient.java:152) ~[library-client-6.2.0.jar:6.2.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.HistoryDeleteEsDAO.deleteHistory(HistoryDeleteEsDAO.java:72) ~[storage-elasticsearch-plugin-6.2.0.jar:6.2.0]
at org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer.execute(DataTTLKeeperTimer.java:73) ~[server-core-6.2.0.jar:6.2.0]
at org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer.lambda$delete$1(DataTTLKeeperTimer.java:66) ~[server-core-6.2.0.jar:6.2.0]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_131]
at org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer.delete(DataTTLKeeperTimer.java:64) ~[server-core-6.2.0.jar:6.2.0]

It 'likes the timer delete the expired index caused a bug。

[wu-sheng]

@shook2012 Your issue doesn't belong to this topic. This is a namespace bug, which has been fixed too.

[shook2012]

Is this issue: 3017 that has been fixed?

[wu-sheng]

Yes. #3017 is the one.

[shook2012]

ok, Thanks!