zookeeper启动之后又异常死掉的处理

通过in-place的方式把一台CentOS 6.5的Linux机器升级到Centos 7.2之后,发现该机器上之前运行的zookeeper启动异常,记录如下:

 
[root@zkserver1 bin]# sh zkServer.sh start
JMX enabled by default
Using config: /usr/local/zookeeper-3.4.5/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@zkserver1 bin]# ps -ef|grep zook
root       891   843  0 14:10 pts/0    00:00:00 grep --color=auto zook
[root@zkserver1 bin]# cat zookeeper.out 
2017-06-22 14:10:04,154 [myid:] - INFO  [main:QuorumPeerConfig@101] - Reading configuration from: /usr/local/zookeeper-3.4.5/bin/../conf/zoo.cfg
2017-06-22 14:10:04,166 [myid:] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-06-22 14:10:04,167 [myid:] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2017-06-22 14:10:04,168 [myid:] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2017-06-22 14:10:04,168 [myid:] - WARN  [main:QuorumPeerMain@113] - Either no config or no quorum defined in config, running  in standalone mode
2017-06-22 14:10:04,190 [myid:] - INFO  [main:QuorumPeerConfig@101] - Reading configuration from: /usr/local/zookeeper-3.4.5/bin/../conf/zoo.cfg
2017-06-22 14:10:04,191 [myid:] - INFO  [main:ZooKeeperServerMain@95] - Starting server
2017-06-22 14:10:04,206 [myid:] - INFO  [main:Environment@100] - Server environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
2017-06-22 14:10:04,207 [myid:] - INFO  [main:Environment@100] - Server environment:host.name=zkserver1
2017-06-22 14:10:04,207 [myid:] - INFO  [main:Environment@100] - Server environment:java.version=1.7.0_21
2017-06-22 14:10:04,208 [myid:] - INFO  [main:Environment@100] - Server environment:java.vendor=Oracle Corporation
2017-06-22 14:10:04,209 [myid:] - INFO  [main:Environment@100] - Server environment:java.home=/usr/java/jdk1.7.0_21/jre
2017-06-22 14:10:04,209 [myid:] - INFO  [main:Environment@100] - Server environment:java.class.path=/usr/local/zookeeper-3.4.5/bin/../build/classes:/usr/local/zookeeper-3.4.5/bin/../build/lib/*.jar:/usr/local/zookeeper-3.4.5/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/local/zookeeper-3.4.5/bin/../lib/slf4j-api-1.6.1.jar:/usr/local/zookeeper-3.4.5/bin/../lib/netty-3.2.2.Final.jar:/usr/local/zookeeper-3.4.5/bin/../lib/log4j-1.2.15.jar:/usr/local/zookeeper-3.4.5/bin/../lib/jline-0.9.94.jar:/usr/local/zookeeper-3.4.5/bin/../zookeeper-3.4.5.jar:/usr/local/zookeeper-3.4.5/bin/../src/java/lib/*.jar:/usr/local/zookeeper-3.4.5/bin/../conf:
2017-06-22 14:10:04,210 [myid:] - INFO  [main:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-06-22 14:10:04,211 [myid:] - INFO  [main:Environment@100] - Server environment:java.io.tmpdir=/tmp
2017-06-22 14:10:04,212 [myid:] - INFO  [main:Environment@100] - Server environment:java.compiler=
2017-06-22 14:10:04,212 [myid:] - INFO  [main:Environment@100] - Server environment:os.name=Linux
2017-06-22 14:10:04,213 [myid:] - INFO  [main:Environment@100] - Server environment:os.arch=amd64
2017-06-22 14:10:04,214 [myid:] - INFO  [main:Environment@100] - Server environment:os.version=3.10.0-327.el7.x86_64
2017-06-22 14:10:04,215 [myid:] - INFO  [main:Environment@100] - Server environment:user.name=root
2017-06-22 14:10:04,215 [myid:] - INFO  [main:Environment@100] - Server environment:user.home=/root
2017-06-22 14:10:04,216 [myid:] - INFO  [main:Environment@100] - Server environment:user.dir=/usr/local/zookeeper-3.4.5/bin
2017-06-22 14:10:04,227 [myid:] - INFO  [main:ZooKeeperServer@726] - tickTime set to 2000
2017-06-22 14:10:04,228 [myid:] - INFO  [main:ZooKeeperServer@735] - minSessionTimeout set to -1
2017-06-22 14:10:04,229 [myid:] - INFO  [main:ZooKeeperServer@744] - maxSessionTimeout set to -1
2017-06-22 14:10:04,274 [myid:] - INFO  [main:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:2181
2017-06-22 14:10:04,343 [myid:] - ERROR [main:FileTxnSnapLog@210] - Parent /dubbo/com.onlyou.es.bck.service.IUserStoreService/consumers missing for /dubbo/com.onlyou.es.bck.service.IUserStoreService/consumers/consumer%3A%2F%2F172.16.11.73%2Fcom.onlyou.es.bck.service.IUserStoreService%3Fapplication%3Des-client-web%26category%3Dconsumers%26check%3Dfalse%26default.check%3Dfalse%26default.reference.filter%3DdubboConsumerFilter%26default.retries%3D0%26default.timeout%3D1000000%26dubbo%3D2.5.3%26interface%3Dcom.onlyou.es.bck.service.IUserStoreService%26methods%3DgetUserStoreByCorpIdAndStoreId%2CdeleteStoreUserById%2ClistStoreCode%2CgetUserStoreByUserId%2CsearchStoreIdsByUserId%2CsearchUserStaffList%2CsaveStoreUser%2CgetUserStoreByCorpId%2CsearchStoreInfosByUserId%26pid%3D488%26revision%3Ddev-SNAPSHOT%26side%3Dconsumer%26timestamp%3D1496999541671
2017-06-22 14:10:04,345 [myid:] - ERROR [main:ZooKeeperServerMain@63] - Unexpected exception, exiting abnormally
java.io.IOException: Failed to process transaction type: 1 error: KeeperErrorCode = NoNode for /dubbo/com.onlyou.es.bck.service.IUserStoreService/consumers
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:153)
        at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
        at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:250)
        at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:377)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:122)
        at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:112)
        at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
        at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /dubbo/com.onlyou.es.bck.service.IUserStoreService/consumers
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:211)
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
        ... 9 more
[root@zkserver1 bin]#

原因:可能是服务器升级的过程中,导致zookeeper的数据文件被”破坏”了,导致重启zookeeper时异常。

解决:删除zookeeper的数据文件/tmp/zookeeper,然后重启zookeeper。

发表评论

邮箱地址不会被公开。 必填项已用*标注