Schema-registry: Unknown Host exception starting schema registry

[AndrewSwerlick]

In trying to start the schema registry I'm getting the following error.

 Exception in thread "main" java.lang.ExceptionInInitializerError
    at io.confluent.kafka.schemaregistry.rest.Main.main(Main.java:40)
    Caused by: io.confluent.common.config.ConfigException: Invalid value java.net.UnknownHostException: axon-dev: axon-dev: unknown error for configuration Unknown local hostname
    at io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig.getDefaultHost(SchemaRegistryConfig.java:195)
    at io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig.<clinit>(SchemaRegistryConfig.java:126)

This is on a Centos 6.6 box where zookeeper and kafka are both installed, and I'm using the default configuration for both after having installed the confluent platform via yum. The box is hosted at digital ocean.

[ewencp]

@AndrewSwerlick It seems like it's InetAddress.getLocalHost() is returning axon-dev and then getCanonicalHost() is failing to resolve the FQDN. We could potentially make the getCanonicalHost() step optional (i.e. if it fails, just use whatever we got from InetAddress.getLocalHost()), but I'm not sure that's actually a good idea since only the FQDN is going to be generally safe to use when running multiple schema registry instances.

[AndrewSwerlick]

So should I be able to avoid this by explicitly setting something for host.name in schema-registry-properties? I've tried setting a couple of values there (localhost, 127.0.0.1) but it doesn't seem change anything.

[ewencp]

I think it's not changing anything because the default host is looked up as part of the setup of the configuration options -- it is computed in order to fill in the default value. So this exception is happening even before you've hit the configuration parsing code.

I think that shouldn't be a problem to fix since we could, for example, leave the host.name default as blank and make it look up the default hostname if it is not overridden. That's just a bit error prone because we'd need to do that check everywhere we use that config variable (or replace the standard config lookup with a method call and remember not to use that option directly). Though I'm still curious how this node is configured that looking up the FQDN is failing...

[AndrewSwerlick]

It's basically a stock Digital Ocean server running Centos 6.6. After digging around a while I was able to find some documentation on setting the FQDN

https://github.com/DigitalOcean-User-Projects/Articles-and-Tutorials/blob/master/set_hostname_fqdn_on_ubuntu_centos.md

by following those instructions and adding the proper /etc/hosts entry I was able to get it working properly.

Of course now I'm fighting an unrelated issue where kafka will only start when run as a root user, and will throw out of memory exceptions as non root, so in general it seems like there's some wonkiness with the base digital ocean centos images.

[miguno]

@AndrewSwerlick Any news? Have you been able to sort out the issue? If so, we can close this ticket.

[ewencp]

@miguno I was thinking there might be a better solution to this. The problem is this method. What if we separate the getLocalHost() call and the subsequent getCanonicalHostname() call? We can save the value the first one returns. If the second call throws an exception, we can fall back on the first value instead.

It'd be worth testing this patch on this Digital Ocean Centos 6.6. setup before committing a change like this since I'm not sure whether it helps in that case (i.e. if ignoring the canonicalization step leaves you with a usable hostname), but if it at least works in the localhost case (i.e. at least works for another process running on the same host as SR), it would be better behavior than we're currently providing.

[apg]

I'd like to propose an alternative solution, which is to allow the fallback to the value in an environment variable. Will send a PR, as we're running into this exact issue as well.

[duytruong]

@AndrewSwerlick Could you share your /etc/hosts file?

[AndrewSwerlick]

I don't have the box anymore because we upgraded to Centos 7.0 and we aren't experimenting with kafka anymore. Sorry.

[leelakrishna]

This error is sporadic and seen when multiple Kafka services are added to the same docker container (one of the services fails to with an Unknown Host exception. One way, I got to resolve it is by adding a "hostname: localhost" tag to every service. Sample is attached.
docker-compose-yml.txt

[gmocquet]

Hello, I got the same issue running on latest Arch Linux x64 with latest Docker and Docker compose version. I don't have any problem with my host hostname configuration. Both hostname and hostname -f give me the right result.

[mageshn]

Since there are known workarounds for the issue, I'm closing this issue for now. If we think, this needs to be addressed in the service, please feel free to reopen.