[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    gobblin on yarn - "Invalid AMRMToken" error
From:       Piotr_Szełemej <szamasz1112 () gmail ! com>
Date:       2019-01-23 14:45:02
Message-ID: CAP1g_LHacgknw7aAYGGJZ==nxWb-nN5aR+b49VnU3DL9k3GpGQ () mail ! gmail ! com
[Download RAW message or body]

Hi,

we have an issue with gobblin application failing every 2 days with
following error:

CEST INFO  [AMRM Heartbeater thread]
org.apache.hadoop.io.retry.RetryInvocationHandler  - Exception while
invoking allocate of class ApplicationMasterProtocolPBClientImpl over
rm365. Trying to fail over immediately.
org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
AMRMToken from appattempt_1535039367371_143153_000001
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
    at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
    at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
  at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
  at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
  at com.sun.proxy.$Proxy20.allocate(Unknown Source)
    at
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:277)
  at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
 Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
 Invalid AMRMToken from appattempt_1535039367371_143153_000001
    at org.apache.hadoop.ipc.Client.call(Client.java:1502)
    at org.apache.hadoop.ipc.Client.call(Client.java:1439)
    at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
    at com.sun.proxy.$Proxy19.allocate(Unknown Source)
    at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)


We are running cloudera hadoop 5.11.2 which is hadoop 2.6 with backported
fixes and enhancements. Do you know why would AM fail to refresh the token
before its 48h expiration period passes?
Any help is much appreciated.

Best regards,
Piotr


[Attachment #3 (text/html)]

<div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi,</div><div><br></div><div>we \
have an issue with gobblin application failing every 2 days with following \
error:</div><div><br></div><div>CEST INFO   [AMRM Heartbeater thread] \
org.apache.hadoop.io.retry.RetryInvocationHandler   - Exception while invoking \
allocate of class ApplicationMasterProtocolPBClientImpl over rm365. Trying to fail \
over immediately.<br>org.apache.hadoop.security.token.SecretManager$InvalidToken: \
Invalid AMRMToken from appattempt_1535039367371_143153_000001<br>       at \
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)<br>       at \
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)<br> \
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)<br> \
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)<br>       at \
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)<br>       at \
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)<br>      \
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)<br> \
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)<br>       at \
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<br> \
at java.lang.reflect.Method.invoke(Method.java:483)<br>       at \
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)<br> \
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)<br> \
at com.sun.proxy.$Proxy20.allocate(Unknown Source)<br>       at \
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:277)<br> \
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:224)<br>Caused \
by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): \
Invalid AMRMToken from appattempt_1535039367371_143153_000001<br>       at \
org.apache.hadoop.ipc.Client.call(Client.java:1502)<br>       at \
org.apache.hadoop.ipc.Client.call(Client.java:1439)<br>       at \
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)<br> \
at com.sun.proxy.$Proxy19.allocate(Unknown Source)<br>       at \
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)<br><br></div><div>We \
are running cloudera hadoop 5.11.2 which is hadoop 2.6 with backported fixes and \
enhancements. Do you know why would AM fail to refresh the token before its 48h \
expiration period passes?<br></div></div><div>Any help is much \
appreciated.<br></div><div><br></div><div>Best \
regards,</div><div>Piotr<br></div></div></div>



[prev in list] [next in list] [prev in thread] [next in thread]