[Kafka] 카프카 서버 실행 오류

카프카 브로커를 추가하는 과정에서 기존 카프카 브로커도 재기동을 해야했는데 실패했다.  

$ sudo systemctl list-units
  UNIT                                                              LOAD   ACTIVE SUB       DESCRIPTION
...
● confluent-kafka.service                                           loaded failed failed    Apache Kafka - broker
...

흑흑 왜 내가 건들기만 하면 실패일까ㅠㅠ

 

일단 해결을 봐야하니까 로그를 뒤져봐야 한다.

어플리케이션이 제대로 안떠서인지 systemctl 로그에서도, log 디렉토리에서도 별다른 로그를 확인할 수 없었다 ‼️ ‼️ ‼️

$ sudo systemctl status confluent-kafka.service
● confluent-kafka.service - Apache Kafka - broker
   Loaded: loaded (/usr/lib/systemd/system/confluent-kafka.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2022-08-01 14:15:14 JST; 41min ago
     Docs: http://docs.confluent.io/
  Process: 18770 ExecStart=/usr/bin/kafka-server-start /etc/kafka/server.properties (code=exited, status=1/FAILURE)
 Main PID: 18770 (code=exited, status=1/FAILURE)

Aug 01 14:15:14  kafka-server-start[18770]: [2022-08-01 14:15:14,332] INFO Session: 0x2000046df81009f closed (org.apache.zookeeper.ZooKeeper)
Aug 01 14:15:14  kafka-server-start[18770]: [2022-08-01 14:15:14,332] INFO EventThread shut down for session: 0x2000046df81009f (org.apache.zookeeper.ClientCnxn)
Aug 01 14:15:14  kafka-server-start[18770]: [2022-08-01 14:15:14,333] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
Aug 01 14:15:14  kafka-server-start[18770]: [2022-08-01 14:15:14,337] INFO shut down completed (kafka.server.KafkaServer)
Aug 01 14:15:14  kafka-server-start[18770]: [2022-08-01 14:15:14,337] INFO Shutting down SupportedServerStartable (io.confluent.support.metrics.SupportedServerStartable)
Aug 01 14:15:14  kafka-server-start[18770]: [2022-08-01 14:15:14,338] INFO Shutting down KafkaServer (io.confluent.support.metrics.SupportedServerStartable)
Aug 01 14:15:14  kafka-server-start[18770]: [2022-08-01 14:15:14,338] INFO shutting down (kafka.server.KafkaServer)
Aug 01 14:15:14  systemd[1]: confluent-kafka.service: main process exited, code=exited, status=1/FAILURE
Aug 01 14:15:14  systemd[1]: Unit confluent-kafka.service entered failed state.
Aug 01 14:15:14  systemd[1]: confluent-kafka.service failed.

 

뭐라도 남아있을 것 같은 시스템 로그를 뒤져보자.

$ sudo tail -n 100 /var/log/messages
...
Aug  1 14:15:13 kafka-n-zk004 kafka-server-start[18770]: [2022-08-01 14:15:13,936] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
Aug  1 14:15:14 kafka-n-zk004 kafka-server-start[18770]: [2022-08-01 14:15:14,211] INFO Cluster ID = VzmDpBaoQzujg-IR751OLg (kafka.server.KafkaServer)
// 여기가 중요!!!
Aug  1 14:15:14 kafka-n-zk004 kafka-server-start[18770]: [2022-08-01 14:15:14,222] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
Aug  1 14:15:14 kafka-n-zk004 kafka-server-start[18770]: kafka.common.InconsistentClusterIdException: The Cluster ID VzmDpBaoQzujg-IR751OLg doesnt match stored clusterId Some(wzYRAL0ARymSpG3SY2kV7A) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
Aug  1 14:15:14 kafka-n-zk004 kafka-server-start[18770]: at kafka.server.KafkaServer.startup(KafkaServer.scala:220)
Aug  1 14:15:14 kafka-n-zk004 kafka-server-start[18770]: at io.confluent.support.metrics.SupportedServerStartable.startup(SupportedServerStartable.java:114)
Aug  1 14:15:14 kafka-n-zk004 kafka-server-start[18770]: at io.confluent.support.metrics.SupportedKafka.main(SupportedKafka.java:66)
...
Aug  1 14:15:14 kafka-n-zk004 systemd[1]: confluent-kafka.service: main process exited, code=exited, status=1/FAILURE
Aug  1 14:15:14 kafka-n-zk004 systemd[1]: Unit confluent-kafka.service entered failed state.
Aug  1 14:15:14 kafka-n-zk004 systemd[1]: confluent-kafka.service failed.

 

문제 발견

ERROR 로 직힌 로그 확인해보면 `meta.properties`에 Cluster ID 가 잘못되었다고 나온다.

Cluster ID 값으로 VzmDpBaoQzujg-IR751OLg 를 사용하려고 하는데 meta.properties 의 값인 wzYRAL0ARymSpG3SY2kV7A 와 일치하지 않아서 실행할 수 없다는 것 같다.

 

해결 방법

1. meta.properties 파일을 삭제한다.
2. meta.properties 파일의 cluster.id 값을 수정한다. (wzYRAL0ARymSpG3SY2kV7A -> VzmDpBaoQzujg-IR751OLg)  

$ cat meta.properties
cluster.id=wzYRAL0ARymSpG3SY2kV7A
version=0
broker.id=0

위의 두가지 방법 중 하나를 선택해서 진행한 후 다시 카프카 서버를 실행시킨다.  
meta.properties 파일은 kafka 설정파일(server.properties)의 log.dirs 값인 path 하위에 존재한다.