[prev in list] [next in list] [prev in thread] [next in thread]
List: hadoop-user
Subject: Task failing due to failed on connection exception
From: Joey Andres <yeojserdna () gmail ! com>
Date: 2017-10-29 6:15:03
Message-ID: 76217591-a415-bc47-5317-333131dc20d3 () gmail ! com
[Download RAW message or body]
I'm trying to setup hadoop cluster for version 2.8.2 with two slaves.
Whenever I run:
hadoop jar
$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar
teragen -Dmapreduce.job.maps=1000 10t random-data
I get the following error for one of the slave:
17/10/29 00:02:51 INFO mapreduce.Job: Task Id :
attempt_1509256119340_0001_m_000445_1, Status : FAILED
Container launch failed for container_1509256119340_0001_01_000682 :
java.net.ConnectException: Call From linux-01/127.0.0.1 to
localhost:37411 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
Going to that slave's yarn-hadoop-nodemanager-linux-02.log, I get:
2017-10-28 23:48:42,719 INFO \
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Node ID assigned is \
: localhost:37411 2017-10-28 23:48:42,725 INFO org.apache.hadoop.yarn.client.RMProxy: \
Connecting to ResourceManager at linux-01.local/192.168.1.1:8031 2017-10-28 \
23:48:42,762 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: \
Sending out 0 NM container statuses: [] 2017-10-28 23:48:42,767 INFO \
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registering with RM \
using containers :[] 2017-10-28 23:48:43,181 INFO \
org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager: \
Rolling master-key for container-tokens, got key with id -410082181 2017-10-28 \
23:48:43,187 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM: \
Rolling master-key for container-tokens, got key with id -1296212863 2017-10-28 \
23:48:43,188 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: \
Registered with ResourceManager as localhost:37411 with total resource of \
<memory:8192, vCores:8> 2017-10-28 23:48:43,188 INFO \
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying \
ContainerManager to unblock new container-requests 2017-10-28 23:53:46,156 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:53:46,156 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:53:47,156 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:53:48,159 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:53:49,162 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:53:50,164 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:53:51,169 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:53:53,174 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:56:48,588 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:56:50,592 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:56:51,595 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:56:52,598 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event
In it, you can see "Node ID assigned is : localhost:37411", which
matches the complain from the mapreduce job.
I don't know why the resourcemanager, in my linux-01 won't call slave
linux-02, but instead call th enon-existent localhost:37411... I'm very
confused.
I can clarify and/or provide more info if needed.
Cheers,
Joey Andres
[Attachment #3 (text/html)]
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>I'm trying to setup hadoop cluster for version 2.8.2 with two
slaves. Whenever I run:</p>
<p>hadoop jar
$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar
teragen -Dmapreduce.job.maps=1000 10t random-data</p>
<p>I get the following error for one of the slave:<br>
</p>
<p>17/10/29 00:02:51 INFO mapreduce.Job: Task Id :
attempt_1509256119340_0001_m_000445_1, Status : FAILED<br>
Container launch failed for container_1509256119340_0001_01_000682
: java.net.ConnectException: Call From linux-01/127.0.0.1 to
localhost:37411 failed on connection exception:
java.net.ConnectException: Connection refused; For more details
see: <a class="moz-txt-link-freetext" \
href="http://wiki.apache.org/hadoop/ConnectionRefused">http://wiki.apache.org/hadoop/ConnectionRefused</a><br>
</p>
<p>Going to that slave's yarn-hadoop-nodemanager-linux-02.log, I
get:</p>
<pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: \
normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: \
2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; \
word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; \
text-decoration-color: initial; word-wrap: break-word; white-space: \
pre-wrap;">2017-10-28 23:48:42,719 INFO \
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Node ID assigned is \
: localhost:37411 2017-10-28 23:48:42,725 INFO org.apache.hadoop.yarn.client.RMProxy: \
Connecting to ResourceManager at linux-01.local/192.168.1.1:8031 2017-10-28 \
23:48:42,762 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: \
Sending out 0 NM container statuses: [] 2017-10-28 23:48:42,767 INFO \
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registering with RM \
using containers :[] 2017-10-28 23:48:43,181 INFO \
org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager: \
Rolling master-key for container-tokens, got key with id -410082181 2017-10-28 \
23:48:43,187 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM: \
Rolling master-key for container-tokens, got key with id -1296212863 2017-10-28 \
23:48:43,188 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: \
Registered with ResourceManager as localhost:37411 with total resource of \
<memory:8192, vCores:8> 2017-10-28 23:48:43,188 INFO \
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying \
ContainerManager to unblock new container-requests 2017-10-28 23:53:46,156 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:53:46,156 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:53:47,156 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:53:48,159 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:53:49,162 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:53:50,164 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:53:51,169 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:53:53,174 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:56:48,588 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:56:50,592 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:56:51,595 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event 2017-10-28 23:56:52,598 WARN \
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: \
couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS \
event
</pre>
In it, you can see "Node ID assigned is : localhost:37411", which
matches the complain from the mapreduce job.<br>
<br>
I don't know why the resourcemanager, in my linux-01 won't call
slave linux-02, but instead call th enon-existent localhost:37411...
I'm very confused.<br>
<br>
I can clarify and/or provide more info if needed.<br>
<br>
Cheers,<br>
Joey Andres<br>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic