Spring-cloud-config: Config Server cannot pull from remote repo

[magiccrafter]

Hi,

After almost 3 months without any issues the cloud config server suddenly failed pulling from the repo.
This happened silently on Saturday and on Monday we found out after checking the logs that the new config changes were not being picked from the config-server.

Exception thrown: Cannot pull from remote https://repo.git, the working tree is not clean.

No changes were pushed to the repo or new version of cloud server deployed anytime close to the silent crash. The good thing is that the issue can be resolved just by restarting the cloud config server(s). After that everything goes back to normal but now the question is how to monitor and get notified for such issues. It would be nice if there is a HealthIndicator reporting the health of the config server taking into consideration such state or/and the logging level changed to ERROR.

Detailed exception:

2016-12-10 03:47:09,118 [http-nio-8888-exec-5] INFO  o.s.c.a.AnnotationConfigApplicationContext Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@498613ae: startup date [Sat Dec 10 03:47:09 CET 2016]; root of context hierarchy
2016-12-10 03:47:09,123 [http-nio-8888-exec-5] INFO  o.s.c.c.s.e.NativeEnvironmentRepository Adding property source: file:/tmp/config-repo-7239745589329148613/application.properties
2016-12-10 03:47:09,123 [http-nio-8888-exec-5] INFO  o.s.c.a.AnnotationConfigApplicationContext Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@498613ae: startup date [Sat Dec 10 03:47:09 CET 2016]; root of context hierarchy
2016-12-10 03:48:08,949 [http-nio-8888-exec-10] INFO  o.s.c.c.s.e.MultipleJGitEnvironmentRepository Cannot pull from remote https://repo.git, the working tree is not clean.
2016-12-10 03:48:08,983 [http-nio-8888-exec-10] INFO  o.s.c.a.AnnotationConfigApplicationContext Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@11af7f6b: startup date [Sat Dec 10 03:48:08 CET 2016]; root of context hierarchy
2016-12-10 03:48:08,988 [http-nio-8888-exec-10] INFO  o.s.c.c.s.e.NativeEnvironmentRepository Adding property source: file:/tmp/config-repo-7239745589329148613/application.properties
2016-12-10 03:48:08,988 [http-nio-8888-exec-10] INFO  o.s.c.a.AnnotationConfigApplicationContext Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@11af7f6b: startup date [Sat Dec 10 03:48:08 CET 2016]; root of context hierarchy
2016-12-10 03:49:08,954 [http-nio-8888-exec-8] INFO  o.s.c.c.s.e.MultipleJGitEnvironmentRepository Cannot pull from remote https://repo.git, the working tree is not clean.
2016-12-10 03:49:08,981 [http-nio-8888-exec-8] INFO  o.s.c.a.AnnotationConfigApplicationContext Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@7f1733bd: startup date [Sat Dec 10 03:49:08 CET 2016]; root of context hierarchy

Thanks

[marcosbarbero]

Have you tried the force-pull property? https://github.com/spring-cloud/spring-cloud-config/pull/473

[magiccrafter]

Thanks for the hint. It will do the job.
I still believe that the server has to report its health state as OUT_OF_SERVICE or at least log a WARN message in such situation where the config server is malfunctioning and is not able to auto-recover.

[marcosbarbero]

@magiccrafter it's a good catch. I'll try something out

[marcosbarbero]

@magiccrafter actually there's a log when such situation happens, you can see it in this method https://github.com/spring-cloud/spring-cloud-config/blob/master/spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/environment/JGitEnvironmentRepository.java#L241

But in fact it does log warn only when you've the force-pull flag activated otherwise you gonna have an info log level.

[magiccrafter]

@marcosbarbero yes, I saw that but it is quite the opposite from expected, isn't it? When you have set 'force-pull' to true ideally you are aware of the behaviour and most likely such messages are expected and/or have been suffering from at least once. While if you haven't set that property and the config-server got stuck in this state you won't be notified by any monitoring system until you try to deploy/restart an application.

[marcosbarbero]

You're right. I'll work on this log level and I'll try to also change the health state to OUT_OF_SERVICE.

[magiccrafter]

Just let me know if you don't have time for that and I will be more than happy to contribute.

[marbon87]

I am working on a pull request to update the health status to out of service, if the repository cannot be updated.

I saw that the Environment class has as a variable "state", that actually is not used. My idea is to set the state of the environment to something like "latest" and "outdated" and check that in the health check.
If this variable is free we maybe can change the type to something that offers a selection.

What do you think about this idea?

These are my current changes:
Changes

[spencergibb]

The state variable is used. It's just not obvious.

[marbon87]

Ok, then i create a new variable. Anything else that is not ideal in my current changes?

[marbon87]

I updated my fork to use a new variable called cached, to signal if the environment is created from cached information or not

[marbon87]

@spencergibb Any update on this? Are you working on this problem?

It happened a few times in the last two months that our two instances of the config server were not able to pull the latest version from the git repository and we have to restart the servers. The Health-Indicator actually does not help to monitor this problem, which is a huge problem.

[spencergibb]

@marbon87 not currently.

[cforce]

We have the same issue now.
@marbon87 You are absolutely right ;(

[marcosbarbero]

@cforce what if you just use the foce-pull? I know it's not ideal but should solve your problem for now.

I agree that a health status as OUT_OF_SERVICE (or whatever) would be a good fit for monitoring, unfortunately right now it's not like that.

[marcingrzejszczak]

Is it still a problem with Finchley.RELEASE?

[cforce]

Force-pull works around.
Sadly not upgraded to finchley

[uesleilima]

Have you tried the force-pull property? #473

this was very helpfull

[SeongcheolHong]

I tried with 'force-pull' property. But problem still being occured.
"org.springframework.cloud.config.server.environment.EnvironmentController.defaultLabel(java.lang.String,java.lang.String)]: org.springframework.cloud.config.server.environment.NoSuchRepositoryException: Cannot clone or checkout repository:"
Isn't there any solution?

[Mistborn94]

We also encountered this problem: Every once in a while our config server would fail to load new config and there was a log line Cannot pull from remote https://repo.git, the working tree is not clean.

It turns out the cause was that our config repo was being cloned into the tmp folder (the default setting) and the operating system cleared it out periodically. From the docs:

Some operating systems routinely clean out temporary directories. This can lead to unexpected behavior, such as missing properties.

You can change the clone location by setting the spring.cloud.config.server.git.basedir property.