Onpremise: Upgrade fails with KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}

Created on 25 May 2020 · 3Comments · Source: getsentry/onpremise

Updating via ./install.sh terminates with an error. Some weeks ago the same mechanism was working to update an onpremise installation.
I had updated to latest master (83160e8bdbebfea2c0f02849b545b9cf1b30c324) before running this command.

This shouldn't be a duplicate of https://github.com/getsentry/onpremise/issues/472, as I've already pulled the changes. Please let me know if you need anything else from me.

Logs:
sentry-install.txt
zookeeper.txt
kafka.txt

Question

Source

jansorg

👍1

Most helpful comment

Here's the source of the issue:

org.apache.kafka.common.KafkaException: Failed to acquire lock on file .lock in /var/lib/kafka/data. A Kafka instance in another process or thread is using this directory.

Probably you had an unclean shutdown of kafka earlier. You should be able to do

docker-compose run --rm kafka bash -c "rm /var/lib/kafka/data/.lock"

and proceed at your own risk.

BYK on 25 May 2020

👍2

All 3 comments

Here's the source of the issue:

org.apache.kafka.common.KafkaException: Failed to acquire lock on file .lock in /var/lib/kafka/data. A Kafka instance in another process or thread is using this directory.

Probably you had an unclean shutdown of kafka earlier. You should be able to do

docker-compose run --rm kafka bash -c "rm /var/lib/kafka/data/.lock"

and proceed at your own risk.

BYK on 25 May 2020

👍2

Thank you for your response. This helped me to solve this.
Even after a docker-compose down there were running sentry containers. This was probably caused by failed attempts with install.sh. So only a few containers were shutdown and setup.sh proceeded with some left running. This seems to have caused the kafka errors because the lock file was still in use.

After stopping all sentry cotainers with a manual docker stop command the install.sh finished successfully.

jansorg on 25 May 2020

🎉1

I had this issue on CentOS 8, it turned out it was a docker-firewalld config issue on CentOS 8 (in case anyone else encounters this issue)
https://github.com/docker/for-linux/issues/957#issuecomment-627166787