[prev in list] [next in list] [prev in thread] [next in thread]
List: hadoop-user
Subject: =?GB2312?Q?Re=3A_=B4=F0=B8=B4=3A_Hadoop_HA_error_=22JOURNAL_is_not_supporte?= =?GB2312?Q?d_in_state_
From: orahad bigdata <oraclehad () gmail ! com>
Date: 2013-08-30 15:52:40
Message-ID: CAK5PjpLkHhnjFLeow6AghDpguu59Tp-ZwkpAfqwXstdY-YFyfA () mail ! gmail ! com
[Download RAW message or body]
Hi,
Many thanks to everyone.
Now issue got resolve after changing client version.
Regards
On Fri, Aug 30, 2013 at 1:12 PM, Francis.Hu <francis.hu@reachjunction.com>w=
rote:
> Did you start up your ZKFC service on both of your name nodes ?
>
> Thanks,
> Francis.Hu
>
> -----=D3=CA=BC=FE=D4=AD=BC=FE-----
> =B7=A2=BC=FE=C8=CB: orahad bigdata [mailto:oraclehad@gmail.com]
> =B7=A2=CB=CD=CA=B1=BC=E4: Friday, August 30, 2013 4:09
> =CA=D5=BC=FE=C8=CB: user
> =D6=F7=CC=E2: Hadoop HA error "JOURNAL is not supported in state standby"
>
> Hi,
>
> I'm facing an error while starting Hadoop in HA(2.0.5) cluster , both
> the NameNode started in standby mode and not changing the state.
>
> When I tried to do health check through "hdfs haadmin -checkhealth
> <service id> " it's giving me below error.
>
> Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required fields: callId, status; Host Details : local host is:
> "clone2/XX.XX.XX.XX"; destination host is: "clone1":8020;
>
> I checked the logs at NN side.
>
> 2013-08-30 00:49:16,074 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:hadoop (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.StandbyException: Operation category
> JOURNAL is not supported in state standby
> 2013-08-30 00:49:16,074 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 8020, call
> org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog
> from 192.168.126.31:48266: error:
> org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is
> not supported in state standby
> 2013-08-30 00:49:32,391 INFO
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering
> log roll on remote NameNode clone2:8020
> 2013-08-30 00:49:32,403 WARN
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to
> trigger a roll of the active NN
>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyExcept=
ion
> ):
> Operation category JOURNAL is not supported in state standby
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(Sta=
ndb
> yState.java:87)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOp=
era
> tion(NameNode.java:1411)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSName=
sys
> tem.java:859)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesys=
tem
> .java:4445)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(Name=
Nod
> eRpcServer.java:766)
> at
>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.=
rol
> lEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
> at
>
> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProt=
oco
> lService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)
> at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(Pr=
oto
> bufRpcEngine.java:454)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation=
.ja
> va:1478)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.=
jav
> a:202)
> at $Proxy11.rollEditLog(Unknown Source)
> at
>
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLo=
g(N
> amenodeProtocolTranslatorPB.java:139)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogR=
oll
> (EditLogTailer.java:268)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLo=
gTa
> iler.java:61)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThre=
ad.
> doWork(EditLogTailer.java:310)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThre=
ad.
> access$200(EditLogTailer.java:279)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThre=
ad$
> 1.run(EditLogTailer.java:296)
> at
>
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil=
.ja
> va:456)
> at
>
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThre=
ad.
> run(EditLogTailer.java:292)
>
> Did I missed something?
>
> Thanks
>
>
[Attachment #3 (text/html)]
<div>Hi,</div>
<div> </div>
<div>Many thanks to everyone.</div>
<div> </div>
<div>Now issue got resolve after changing client version.</div>
<div> </div>
<div>Regards</div>
<div>On Fri, Aug 30, 2013 at 1:12 PM, Francis.Hu <span dir="ltr"><<a \
href="mailto:francis.hu@reachjunction.com" \
target="_blank">francis.hu@reachjunction.com</a>></span> wrote:<br></div> <div \
class="gmail_quote"> <blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px \
0.8ex;PADDING-LEFT:1ex" class="gmail_quote">Did you start up your ZKFC service on \
both of your name nodes ?<br><br>Thanks,<br>Francis.Hu<br><br>-----邮件原件-----<br> \
发件人: orahad bigdata [mailto:<a \
href="mailto:oraclehad@gmail.com">oraclehad@gmail.com</a>]<br>发送时间: Friday, \
August 30, 2013 4:09<br>收件人: user<br>主题: Hadoop HA error "JOURNAL is not \
supported in state standby"<br>
<div class="HOEnZb">
<div class="h5"><br>Hi,<br><br>I'm facing an error while starting Hadoop in \
HA(2.0.5) cluster , both<br>the NameNode started in standby mode and not changing the \
state.<br><br>When I tried to do health check through "hdfs haadmin \
-checkhealth<br> <service id> " it's giving me below \
error.<br><br>Failed on local \
exception:<br>com.google.protobuf.InvalidProtocolBufferException: Message \
missing<br>required fields: callId, status; Host Details : local host is:<br> \
"clone2/XX.XX.XX.XX"; destination host is: \
"clone1":8020;<br><br> I checked the logs at NN \
side.<br><br>2013-08-30 00:49:16,074 \
ERROR<br>org.apache.hadoop.security.UserGroupInformation:<br>PriviledgedActionException \
as:hadoop (auth:SIMPLE)<br> cause:org.apache.hadoop.ipc.StandbyException: Operation \
category<br>JOURNAL is not supported in state standby<br>2013-08-30 00:49:16,074 INFO \
org.apache.hadoop.ipc.Server: IPC Server<br>handler 1 on 8020, \
call<br>org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog<br> from \
<a href="http://192.168.126.31:48266/" target="_blank">192.168.126.31:48266</a>: \
error:<br>org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL \
is<br>not supported in state standby<br>2013-08-30 00:49:32,391 \
INFO<br>
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering<br>log roll on \
remote NameNode clone2:8020<br>2013-08-30 00:49:32,403 \
WARN<br>org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to<br>trigger \
a roll of the active NN<br> \
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException<br>):<br>Operation \
category JOURNAL is not supported in state standby<br> \
at<br>org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(Standb<br>
yState.java:87)<br> \
at<br>org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOpera<br>tion(NameNode.java:1411)<br> \
\
at<br>org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesys<br>
tem.java:859)<br> \
at<br>org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem<br>.java:4445)<br> \
\
at<br>org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNod<br>eRpcServer.java:766)<br>
\
at<br>org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rol<br>lEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)<br> \
\
at<br>org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtoco<br>
lService$2.callBlockingMethod(NamenodeProtocolProtos.java:8758)<br> \
at<br>org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(Proto<br>bufRpcEngine.java:454)<br> \
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)<br> \
at \
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)<br> \
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)<br> \
at java.security.AccessController.doPrivileged(Native \
Method)<br> at \
javax.security.auth.Subject.doAs(Subject.java:396)<br> \
at<br>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja<br>va:1478)<br> \
at \
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)<br> <br> \
at org.apache.hadoop.ipc.Client.call(Client.java:1235)<br> \
at<br>org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.jav<br>a:202)<br> \
at $Proxy11.rollEditLog(Unknown Source)<br> \
at<br>org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(N<br>amenodeProtocolTranslatorPB.java:139)<br> \
\
at<br>org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll<br>
(EditLogTailer.java:268)<br> \
at<br>org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTa<br>iler.java:61)<br> \
\
at<br>org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.<br>
doWork(EditLogTailer.java:310)<br> \
at<br>org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.<br>access$200(EditLogTailer.java:279)<br> \
\
at<br>org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$<br>
1.run(EditLogTailer.java:296)<br> \
at<br>org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.ja<br>va:456)<br> \
\
at<br>org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.<br>
run(EditLogTailer.java:292)<br><br>Did I missed \
something?<br><br>Thanks<br><br></div></div></blockquote></div><br>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic