Spring-cloud-config: Best practices on handling GIT repository inavailability

[pavlo]

Hey guys!

Yesterday Bitbucket SSH was down for quite a bit and it caused us issues with the config server and a bunch of staging apps. I'd like to see if there's recommended way to handle such situations.

Prerequisites

1. There's a GIT repo with configuration property files
2. There's a Config Server deployment that clones the GIT repo and pulls from there up on client's request
3. There's an app, a client, that, during its startup, obtains configuration properties from the config server
   

   Case 1

Say the GIT repository is not accessible for a reason. We launch config server for the first time and it fails to clone the repo.

This is fine and expectable, no issue here.

Case 2

We have successfully launched the Config Server in the past so it has the GIT repository cloned. It however fails to start if we restart it because it can not connect to GIT repo.

Would it be good to let it boot and get the stuff from the repository it had locally cloned in the past?

Case 3

The Config Server is up and running but at some point Bitbucket SSH goes down. Config Server continues to operate normally. It however fails when a client request comes in because the server can not pull fresh stuff from GIT repository.

Would it be good to let it serve the client with data it already has in the local repository in this case?

Can you please elaborate a bit on the the second and third cases? Am I missing something obvious or even misuse the config server?

Thanks!

[ghost]

Thumbs up as I, too, am trying to solve these issues. Looking through open/closed issues it looks like various folks have had these issues as well. This issue #390 has references to other issues that has addressed Case 2 and Case 3 but not specifically. This issue fixed other issues that happens to fix the issue that that person was having but not the cases that you described above.

Issue #411 seems to have solved this problem for SVN, and similar changes just need to be applied to the Git environment repo.

Hopefully someone can chime in and help out if I am understanding things correctly.

[ryanjbaxter]

I think your suggestions seem reasonable. Feel free to provide some PRs.

[martinspielmann]

I faced the same problem and just wanted to start implementing a fix. But as I was scrolling through the code, I noticed that the desired features are already in there:

Case 1:

like @pavlo said: fine and expectable behavior

Case 2:

This case can easily be achieved by setting the "basedir" property. As soon as you set this property, configserver will take this directory instead of a temporary one which is created during startup. This way the locally cloned repositories will survive configserver reboots.

Case 3:

was already added some commits ago. no more error (just a warn in the logs if remote is not reachable)

[ghost]

@pingunaut could you possibly link the commit, or where in the codebase you see this?

[martinspielmann]

I'm sorry i'm not able to find exactly the commits, but there are two classes involved:

1. JGitEnvironmentRepository

Its fetch() method catches all kinds of exceptions and just warns if the fetch fails. If the git repo was cloned before, that's no problem. Configserver will provide the latest locally available configs.

private FetchResult fetch(Git git, String label) {
    //...
    catch (Exception ex) {
        this.logger.warn("Could not fetch remote for " + label + " remote: " + git
                .getRepository().getConfig().getString("remote", "origin", "url"));
        return null;
    }
}

2. AbstractScmAccessor

It has a basedir property which is set during construction with createBaseDir(). Like you can see below, this is a temp directory by default.

/**
 * Base directory for local working copy of repository.
 */
private File basedir;

protected File createBaseDir() {
    try {
        final File basedir = Files.createTempDirectory("config-repo-").toFile();
        Runtime.getRuntime().addShutdownHook(new Thread() {
            @Override
            public void run() {
                try {
                    FileUtils.delete(basedir, FileUtils.RECURSIVE);
                }
                catch (IOException e) {
                    AbstractScmAccessor.this.logger.warn(
                            "Failed to delete temporary directory on exit: " + e);
                }
            }
        });
        return basedir;
    }
    catch (IOException e) {
        throw new IllegalStateException("Cannot create temp dir", e);
    }
}

However, MultipleJGitEnvironmentRepository makes use of @ConfigurationProperties:

@ConfigurationProperties("spring.cloud.config.server.git")
public class MultipleJGitEnvironmentRepository extends JGitEnvironmentRepository {
//...
}

So if you set cloud.config.server.git.basedir (e.g. cloud.config.server.git.basedir: ./foo), this will override the default behavior because these props are set after construction.

Watch out: this will not work if "cloneOnStart" property is set. It causes the directory to be deleted during startup.

A small optimization could be implemented here:
createBaseDir() could be called after properties are set by spring. One could then check if the basedir is aready set by the user and create a temp folder only if nothing is set.
I would be glad to provide a pull request if wanted.

What are your thoughts?

Regards
Martin

[ghost]

Thank you. Taking a look around the new changes it looks like they might have fixed the cases in this issue.

[ghost]

Update: I updated my Spring Config Server to Spring Cloud Camden.SR3 which pulled in Spring Cloud Config 1.2.2.RELEASE and I was able to verify that by setting the base directory to something other than the default "/tmp/" location, and using this newer version I can see that warning logs are written to the logs, and I still get a response of the apps properties.

I think it would be beneficial that if this happens that a field on the response specifies that you are possibly running older versions of property due to connectivity issues.

[Juliet-Unsilo]

I am unable to get Case 2 to work with Camden.SR6, when I set clone-on-start and basedir. In fact when the clone fails the basedir is deleted.

[spencergibb]

@Juliet-Unsilo if you think there is a bug, please open a new issue explaining how to reproduce the problem.