Skywalking: Skywalking oap NullPointerException

Created on 14 Oct 2020  路  16Comments  路  Source: apache/skywalking

Please answer these questions before submitting your issue.

  • Why do you submit this issue?
  • [ ] Question or discussion
  • [ x] Bug
  • [ ] Requirement
  • [ ] Feature or performance improvement

Bug

  • Which version of SkyWalking, OS and JRE?
    K8s 1.18.6
    Skywalking 8.1

  • What happened?
    I've installed skywalking on K8s following below steps:

git clone https://github.com/apache/skywalking-kubernetes.git
kubectl create namespace demo
cd chart
helm repo add elastic https://helm.elastic.co
helm dep up skywalking
helm install skywalking skywalking -n demo

It works fine for a few hours, and then the skywalking oap pod keeps crashing(maybe after reboot the k8s VM?):

skywalking-oap-59dc695b5f-qfgwp    0/1     CrashLoopBackOff   9          18m

log shows

2020-10-14 04:13:07,072 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -4030 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentDispatcher is added into DefaultScopeDefine 12.
2020-10-14 04:13:07,072 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -4030 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.service.ServiceTrafficDispatcher is added into DefaultScopeDefine 1.
2020-10-14 04:13:07,417 - org.apache.skywalking.oap.server.library.module.BootstrapFlow -4375 [main] INFO  [] - start the provider default in agent-analyzer module.
2020-10-14 04:13:07,790 - org.apache.skywalking.oap.server.core.remote.client.RemoteClientManager -4748 [pool-3-thread-1] ERROR [] - null
java.lang.NullPointerException: null
    at org.apache.skywalking.oap.server.cluster.plugin.kubernetes.NamespacedPodListInformer.listPods(NamespacedPodListInformer.java:96) ~[cluster-kubernetes-plugin-8.1.0.jar:8.1.0]
    at org.apache.skywalking.oap.server.cluster.plugin.kubernetes.KubernetesCoordinator.queryRemoteNodes(KubernetesCoordinator.java:59) ~[cluster-kubernetes-plugin-8.1.0.jar:8.1.0]
    at org.apache.skywalking.oap.server.core.remote.client.RemoteClientManager.refresh(RemoteClientManager.java:128) ~[server-core-8.1.0.jar:8.1.0]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_212]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_212]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_212]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
2020-10-14 04:13:08,413 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -5371 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.EndpointRelationDispatcher is added into DefaultScopeDefine 6.
2020-10-14 04:13:08,413 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -5371 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.ServiceRelationDispatcher is added into DefaultScopeDefine 4
question
Source
picture benjaminhuo

Most helpful comment

@kezhenxu94 I think this us only a master branch issue as you talked about. This issue is talking about 8.1.

Could you confirm?

The NPE exists since v8.1.0, and I also encountered this in the ALS test case, it occurs in a relatively low possibility and can be recovered by Kubernetes' restarting, the other issue that I mentioned by DM is "broke pipe", which is only reproducible in the master branch, they're different problems

picture kezhenxu94 on 14 Oct 2020
👍2

All 16 comments

@innerpeacez Any idea why?

wu-sheng picture wu-sheng on 14 Oct 2020

I also notice this in the ALS E2E test, will submit the fix in the test case altogether

kezhenxu94 picture kezhenxu94 on 14 Oct 2020

Seems that a fix release is necessary for such a critical bug?

benjaminhuo picture benjaminhuo on 14 Oct 2020

@kezhenxu94 I think this us only a master branch issue as you talked about. This issue is talking about 8.1.

Could you confirm?

wu-sheng picture wu-sheng on 14 Oct 2020

I installed the helm chart by checking out the master branch of https://github.com/apache/skywalking-kubernetes/tree/master

Should I try checking out branch https://github.com/apache/skywalking-kubernetes/tree/v3.1.0 ?

The image is 8.1.0

image

benjaminhuo picture benjaminhuo on 14 Oct 2020

@kezhenxu94 I think this us only a master branch issue as you talked about. This issue is talking about 8.1.

Could you confirm?

The NPE exists since v8.1.0, and I also encountered this in the ALS test case, it occurs in a relatively low possibility and can be recovered by Kubernetes' restarting, the other issue that I mentioned by DM is "broke pipe", which is only reproducible in the master branch, they're different problems

kezhenxu94 picture kezhenxu94 on 14 Oct 2020
👍2

@kezhenxu94 's issue is caused by the base image of the main repo only. Not relate to the 8.1.0 or the codes.

If you like, check this PR about deploying SkyWalking of the latest codes on the k8s with Istio/Envoy, and analyze the service mesh.

wu-sheng picture wu-sheng on 15 Oct 2020

@wu-sheng @kezhenxu94 If this is a base image issue, will a new image for skywalking-oap-server:8.1.0-es7 be released?

Thanks
Ben

benjaminhuo picture benjaminhuo on 16 Oct 2020

@wu-sheng @kezhenxu94 If this is a base image issue, will a new image for skywalking-oap-server:8.1.0-es7 be released?

Thanks
Ben

You misunderstood, NPE is not caused by base image, NPE is caused by codes, base image is another problem that is not related to this issue, you only need to build from the master branch and build an image to use, nothing to do with the base image

kezhenxu94 picture kezhenxu94 on 16 Oct 2020

@kezhenxu94 Got it, so for now this issue could be fixed by building a skywalking-oap-server image by myself.
And will there be a fix release for this issue?

benjaminhuo picture benjaminhuo on 16 Oct 2020

@kezhenxu94 Got it, so for now this issue could be fixed by building a skywalking-oap-server image by myself.
And will there be a fix release for this issue?

Will be included in the next version (may be within 1 month)

kezhenxu94 picture kezhenxu94 on 16 Oct 2020

Looking forward to the new release

benjaminhuo picture benjaminhuo on 16 Oct 2020

I meet the same problem with the version 8.1.0, Could you tell me how to solved it ?

beAnger picture beAnger on 10 Mar 2021
👀2

@kezhenxu94 Any update?

xuanyuanaosheng picture xuanyuanaosheng on 19 Mar 2021

@kezhenxu94 Any update?

@xuanyuanaosheng what update? The issues is marked as resolved and tagged in 8.2.0

kezhenxu94 picture kezhenxu94 on 19 Mar 2021

@kezhenxu94 Thanks.

Now I use the new version: 8.4.0-es7, the es version: 7.5.1, use the skywalking-helm to install. But I still can not start the skywalking-oap.

The logs has 

2021-03-22 03:48:58,124 - com.coxautodev.graphql.tools.SchemaClassScanner -2412 [main] WARN  [] - Schema type was defined but can never be accessed, and can be safely deleted: NodeType
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.esotericsoftware.reflectasm.AccessClassLoader (file:/skywalking/oap-libs/reflectasm-1.11.7.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.reflectasm.AccessClassLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2021-03-22 03:48:58,455 - org.apache.skywalking.oap.server.library.module.ModuleDefine -2743 [main] INFO  [] - Prepare the default provider in alarm module.
2021-03-22 03:48:58,472 - org.apache.skywalking.oap.server.core.alarm.provider.AlarmRulesWatcher -2760 [main] INFO  [] - Update alarm rules to Rules(rules=[AlarmRule(alarmRuleName=service_resp_time_rule, metricsName=service_resp_time, includeNames=[], includeNamesRegex=, excludeNames=[], excludeNamesRegex=, includeLabels=[], includeLabelsRegex=, excludeLabels=[], excludeLabelsRegex=, threshold=1000, op=>, period=10, count=3, silencePeriod=5, message=Response time of service {name} is more than 1000ms in 3 minutes of last 10 minutes., onlyAsCondition=false), AlarmRule(alarmRuleName=service_sla_rule, metricsName=service_sla, includeNames=[], includeNamesRegex=, excludeNames=[], excludeNamesRegex=, includeLabels=[], includeLabelsRegex=, excludeLabels=[], excludeLabelsRegex=, threshold=8000, op=<, period=10, count=2, silencePeriod=3, message=Successful rate of service {name} is lower than 80% in 2 minutes of last 10 minutes, onlyAsCondition=false), AlarmRule(alarmRuleName=service_resp_time_percentile_rule, metricsName=service_percentile, includeNames=[], includeNamesRegex=, excludeNames=[], excludeNamesRegex=, includeLabels=[], includeLabelsRegex=, excludeLabels=[], excludeLabelsRegex=, threshold=1000,1000,1000,1000,1000, op=>, period=10, count=3, silencePeriod=5, message=Percentile response time of service {name} alarm in 3 minutes of last 10 minutes, due to more than one condition of p50 > 1000, p75 > 1000, p90 > 1000, p95 > 1000, p99 > 1000, onlyAsCondition=false), AlarmRule(alarmRuleName=service_instance_resp_time_rule, metricsName=service_instance_resp_time, includeNames=[], includeNamesRegex=, excludeNames=[], excludeNamesRegex=, includeLabels=[], includeLabelsRegex=, excludeLabels=[], excludeLabelsRegex=, threshold=1000, op=>, period=10, count=2, silencePeriod=5, message=Response time of service instance {name} is more than 1000ms in 2 minutes of last 10 minutes, onlyAsCondition=false), AlarmRule(alarmRuleName=database_access_resp_time_rule, metricsName=database_access_resp_time, includeNames=[], includeNamesR
egex=, excludeNames=[], excludeNamesRegex=, includeLabels=[], includeLabelsRegex=, excludeLabels=[], excludeLabelsRegex=, threshold=1000, op=>, period=10, count=2, silencePeriod=10, message=Response time of database access {name} is more than 1000ms in 2 minutes of last 10 minutes, onlyAsCondition=false), AlarmRule(alarmRuleName=endpoint_relation_resp_time_rule, metricsName=endpoint_relation_resp_time, includeNames=[], includeNamesRegex=, excludeNames=[], excludeNamesRegex=, includeLabels=[], includeLabelsRegex=, excludeLabels=[], excludeLabelsRegex=, threshold=1000, op=>, period=10, count=2, silencePeriod=10, message=Response time of endpoint relation {name} is more than 1000ms in 2 minutes of last 10 minutes, onlyAsCondition=false)], webhooks=[], grpchookSetting=null, slacks=null, wecchats=null, compositeRules=[], dingtalks=null, feishus=null)
2021-03-22 03:48:58,538 - org.apache.skywalking.oap.server.library.module.ModuleDefine -2826 [main] INFO  [] - Prepare the default provider in log-analyzer module.
2021-03-22 03:48:58,539 - org.apache.skywalking.oap.server.library.module.ModuleDefine -2827 [main] INFO  [] - Prepare the default provider in envoy-metric module.
2021-03-22 03:48:58,544 - org.apache.skywalking.oap.server.library.module.ModuleDefine -2832 [main] INFO  [] - Prepare the default provider in configuration-discovery module.
2021-03-22 03:48:58,554 - org.apache.skywalking.oap.server.library.module.ModuleDefine -2842 [main] INFO  [] - Prepare the none provider in configuration module.
2021-03-22 03:48:58,555 - org.apache.skywalking.oap.server.library.module.ModuleDefine -2843 [main] INFO  [] - Prepare the default provider in agent-analyzer module.
2021-03-22 03:48:58,557 - org.apache.skywalking.oap.server.analyzer.provider.trace.UninstrumentedGatewaysConfig -2845 [main] INFO  [] - Default configured gateways: null
2021-03-22 03:48:58,557 - org.apache.skywalking.oap.server.analyzer.provider.trace.UninstrumentedGatewaysConfig -2845 [main] INFO  [] - Updating uninstrumented gateways with: null
2021-03-22 03:48:58,578 - org.apache.skywalking.oap.server.library.module.BootstrapFlow -2866 [main] INFO  [] - start the provider none in configuration module.
2021-03-22 03:48:58,578 - org.apache.skywalking.oap.server.library.module.BootstrapFlow -2866 [main] INFO  [] - start the provider graphql in query module.
2021-03-22 03:48:58,585 - org.apache.skywalking.oap.server.library.server.jetty.JettyServer -2873 [main] INFO  [] - Bind handler GraphQLQueryHandler into jetty server 0.0.0.0:12800
2021-03-22 03:48:58,590 - org.apache.skywalking.oap.server.library.module.BootstrapFlow -2878 [main] INFO  [] - start the provider none in telemetry module.
2021-03-22 03:48:58,590 - org.apache.skywalking.oap.server.library.module.BootstrapFlow -2878 [main] INFO  [] - start the provider default in core module.
2021-03-22 03:48:58,592 - org.apache.skywalking.oap.server.library.server.grpc.GRPCServer -2880 [main] INFO  [] - Bind handler RemoteServiceHandler into gRPC server 0.0.0.0:11800
2021-03-22 03:48:58,609 - org.apache.skywalking.oap.server.library.server.grpc.GRPCServer -2897 [main] INFO  [] - Bind handler HealthCheckServiceHandler into gRPC server 0.0.0.0:11800
2021-03-22 03:48:58,722 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -3010 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecordDispatcher is added into DefaultScopeDefine 34.
2021-03-22 03:48:58,722 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -3010 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.browser.manual.endpoint.BrowserAppPageTrafficDispatcher is added into DefaultScopeDefine 40.
2021-03-22 03:48:58,723 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -3011 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.relation.endpoint.EndpointCallRelationDispatcher is added into DefaultScopeDefine 6.
2021-03-22 03:48:58,723 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -3011 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceCallRelationDispatcher is added into DefaultScopeDefine 4.
2021-03-22 03:48:58,723 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -3011 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecordDispatcher is added into DefaultScopeDefine 41.
2021-03-22 03:48:58,723 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -3011 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.service.ServiceTrafficDispatcher is added into DefaultScopeDefine 1.
2021-03-22 03:48:58,723 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -3011 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.service.ServiceMetaDispatcher is added into DefaultScopeDefine 29.
2021-03-22 03:48:58,724 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -3012 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.instance.InstanceUpdateDispatcher is added into DefaultScopeDefine 30.
2021-03-22 03:48:58,724 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -3012 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.networkalias.NetworkAddressAliasSetupDispatcher is added into DefaultScopeDefine 31.
2021-03-22 03:48:58,724 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -3012 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.database.DatabaseStatementDispatcher is added into DefaultScopeDefine 18.
2021-03-22 03:48:58,725 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -3013 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.browser.manual.service.BrowserAppTrafficDispatcher is added into DefaultScopeDefine 38.
2021-03-22 03:48:58,725 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -3013 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.browser.manual.instance.BrowserAppSingleVersionTrafficDispatcher is added into DefaultScopeDefine 39.
2021-03-22 03:48:58,725 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -3013 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.relation.instance.ServiceInstanceCallRelationDispatcher is added into DefaultScopeDefine 5.
2021-03-22 03:48:58,725 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -3013 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentDispatcher is added into DefaultScopeDefine 12.
2021-03-22 03:48:58,726 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -3014 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.endpoint.EndpointMetaDispatcher is added into DefaultScopeDefine 42.
2021-03-22 03:48:58,726 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -3014 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.endpoint.EndpointTrafficDispatcher is added into DefaultScopeDefine 3.
2021-03-22 03:48:58,727 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -3015 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.instance.InstanceTrafficDispatcher is added into DefaultScopeDefine 2.
2021-03-22 03:48:58,858 - org.apache.skywalking.oap.server.library.module.BootstrapFlow -3146 [main] INFO  [] - start the provider default in agent-analyzer module.
2021-03-22 03:48:59,840 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -4128 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.EndpointRelationDispatcher is added into DefaultScopeDefine 6.
2021-03-22 03:48:59,840 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -4128 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.ServiceRelationDispatcher is added into DefaultScopeDefine 4.
2021-03-22 03:48:59,841 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -4129 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.AllDispatcher is added into DefaultScopeDefine 0.
2021-03-22 03:48:59,841 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -4129 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.EndpointDispatcher is added into DefaultScopeDefine 3.
2021-03-22 03:48:59,841 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -4129 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.DatabaseAccessDispatcher is added into DefaultScopeDefine 17.
2021-03-22 03:48:59,841 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -4129 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.ServiceInstanceDispatcher is added into DefaultScopeDefine 2.
2021-03-22 03:48:59,841 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -4129 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.ServiceDispatcher is added into DefaultScopeDefine 1.
2021-03-22 03:48:59,842 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -4130 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.ServiceInstanceRelationDispatcher is added into DefaultScopeDefine 5.
2021-03-22 03:49:01,359 - org.apache.skywalking.oap.server.library.module.BootstrapFlow -5647 [main] INFO  [] - start the provider default in alarm module.
2021-03-22 03:49:01,359 - org.apache.skywalking.oap.server.library.module.BootstrapFlow -5647 [main] INFO  [] - start the provider kubernetes in cluster module.
2021-03-22 03:49:01,662 - org.apache.skywalking.oap.server.library.module.BootstrapFlow -5950 [main] INFO  [] - start the provider elasticsearch7 in storage module.
2021-03-22 03:49:01,663 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -5951 [main] INFO  [] - elasticsearch cluster nodes: es-cluster-sre-es-http.elastic-system:9200
2021-03-22 03:49:01,666 - io.kubernetes.client.informer.cache.Controller -5954 [informer-controller-V1Pod] INFO  [] - informer#Controller: ready to run resync & reflector runnable
2021-03-22 03:49:01,666 - io.kubernetes.client.informer.cache.Controller -5954 [informer-controller-V1Pod] INFO  [] - informer#Controller: resync skipped due to 0 full resync period
2021-03-22 03:49:01,669 - io.kubernetes.client.informer.cache.ReflectorRunnable -5957 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-0] INFO  [] - class io.kubernetes.client.openapi.models.V1Pod#Start listing and watching...
2021-03-22 03:49:02,021 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -6309 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:05,025 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -9313 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:08,030 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -12318 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:11,035 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -15323 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:14,038 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -18326 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:17,044 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -21332 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:20,050 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -24338 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:23,055 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -27343 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:26,060 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -30348 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:29,064 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -33352 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:32,068 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -36356 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:35,072 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -39360 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:38,076 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -42364 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:41,080 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -45368 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:44,097 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -48385 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:47,101 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -51389 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:50,105 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -54393 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:53,108 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -57396 [main] INFO  [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2021-03-22 03:49:55,134 - io.kubernetes.client.informer.cache.ReflectorRunnable -59422 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-0] INFO  [] - class io.kubernetes.client.openapi.models.V1Pod#Watch connection get exception IO Exception during hasNext method.
2021-03-22 03:49:55,134 - io.kubernetes.client.informer.cache.ReflectorRunnable -59422 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-0] INFO  [] - class io.kubernetes.client.openapi.models.V1Pod#Read timeout retry list and watch
2021-03-22 03:49:55,133 - io.kubernetes.client.informer.cache.Controller -59421 [informer-controller-V1Pod] ERROR [] - DefaultController#processLoop get interrupted null
java.lang.InterruptedException: null
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2090) ~[?:?]
        at io.kubernetes.client.informer.cache.DeltaFIFO.pop(DeltaFIFO.java:307) ~[client-java-8.0.0.jar:?]
        at io.kubernetes.client.informer.cache.Controller.processLoop(Controller.java:146) [client-java-8.0.0.jar:?]
        at io.kubernetes.client.informer.cache.Controller.run(Controller.java:115) [client-java-8.0.0.jar:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]

Could you please help me ?

xuanyuanaosheng picture xuanyuanaosheng on 22 Mar 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

gaoweijie picture gaoweijie  路  3Comments
vanlin picture vanlin  路  6Comments
itziklavon picture itziklavon  路  4Comments
dingsongjie picture dingsongjie  路  4Comments
valyala picture valyala  路  3Comments