'Re: common-user Digest 26 Sep 2010 04:20:50 -0000 Issue 1548'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    Re: common-user Digest 26 Sep 2010 04:20:50 -0000 Issue 1548
From:       Sudhir Vallamkondu <Sudhir.Vallamkondu () icrossing ! com>
Date:       2010-09-26 6:55:31
Message-ID: C8C43C73.4EAB5%Sudhir.Vallamkondu () icrossing ! com
[Download RAW message or body]

The exceptions below before shutdown are because of inability to access the
jobtracker and namenode. I am guessing jobtracker and namenode are getting
shutdown before these tasktracker and datanodes get shutdown. The errors
below are with connecting to port 9000 and 9001 which are default ports for
namenode and jobtracker (one of the default options)

http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/

- Sudhir


On Sep/25/ 9:20 PM, "common-user-digest-help@hadoop.apache.org"
<common-user-digest-help@hadoop.apache.org> wrote:

> From: shangan <shangan@corp.kaixin001.com>
> Date: Sun, 26 Sep 2010 11:08:22 +0800
> To: hadoop-user <common-user@hadoop.apache.org>
> Subject: datanode and tasktracker shutdown error
> 
> previously I have a cluster containing 8 nodes and it woks well. I add 24 new
> datanodes to the cluster, tasktracker and datanode deamons can start but when
> I shutdown the cluster I find those errors on these new added datanodes. Can
> anyone explain it?
> 
> log from tasktracker
> 
> 2010-09-26 09:52:21,672 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG: 
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG:   host = localhost/127.0.0.1
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 0.20.2
> STARTUP_MSG:   build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707;
> compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-09-26 09:52:21,876 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2010-09-26 09:52:22,006 INFO org.apache.hadoop.http.HttpServer: Port returned
> by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening
> the listener on 50060
> 2010-09-26 09:52:22,014 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50060
> webServer.getConnectors()[0].getLocalPort() returned 50060
> 2010-09-26 09:52:22,014 INFO org.apache.hadoop.http.HttpServer: Jetty bound to
> port 50060
> 2010-09-26 09:52:22,014 INFO org.mortbay.log: jetty-6.1.14
> 2010-09-26 09:52:42,715 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50060
> 2010-09-26 09:52:42,722 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=TaskTracker, sessionId=
> 2010-09-26 09:52:42,737 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=TaskTracker, port=28404
> 2010-09-26 09:52:42,793 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2010-09-26 09:52:42,793 INFO org.apache.hadoop.ipc.Server: IPC Server listener
> on 28404: starting
> 2010-09-26 09:52:42,794 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 0 on 28404: starting
> 2010-09-26 09:52:42,795 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 1 on 28404: starting
> 2010-09-26 09:52:42,795 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 3 on 28404: starting
> 2010-09-26 09:52:42,795 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 2 on 28404: starting
> 2010-09-26 09:52:42,795 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 4 on 28404: starting
> 2010-09-26 09:52:42,795 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 5 on 28404: starting
> 2010-09-26 09:52:42,796 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 6 on 28404: starting
> 2010-09-26 09:52:42,796 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 7 on 28404: starting
> 2010-09-26 09:52:42,796 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 8 on 28404: starting
> 2010-09-26 09:52:42,796 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 9 on 28404: starting
> 2010-09-26 09:52:42,796 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 10 on 28404: starting
> 2010-09-26 09:52:42,796 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 11 on 28404: starting
> 2010-09-26 09:52:42,796 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 12 on 28404: starting
> 2010-09-26 09:52:42,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 13 on 28404: starting
> 2010-09-26 09:52:42,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 14 on 28404: starting
> 2010-09-26 09:52:42,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 15 on 28404: starting
> 2010-09-26 09:52:42,797 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker
> up at: localhost/127.0.0.1:28404
> 2010-09-26 09:52:42,797 INFO org.apache.hadoop.mapred.TaskTracker: Starting
> tracker tracker_localhost:localhost/127.0.0.1:28404
> 2010-09-26 09:52:55,025 INFO org.apache.hadoop.mapred.TaskTracker: Starting
> thread: Map-events fetcher for all reduce tasks on
> tracker_localhost:localhost/127.0.0.1:28404
> 2010-09-26 09:52:55,027 INFO org.apache.hadoop.mapred.TaskTracker:  Using
> MemoryCalculatorPlugin :
> org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@2de12f6d
> 2010-09-26 09:52:55,031 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
> disabled.
> 2010-09-26 09:52:55,032 INFO org.apache.hadoop.mapred.IndexCache: IndexCache
> created with max memory = 10485760
> 2010-09-26 09:54:13,298 ERROR org.apache.hadoop.mapred.TaskTracker: Caught
> exception: java.io.IOException: Call to vm221/10.11.2.221:9001 failed on local
> exception: java.io.EOFException
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:743)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at org.apache.hadoop.mapred.$Proxy4.heartbeat(Unknown Source)
> at 
> org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1215)
> at 
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1037)
> at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1720)
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2833)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at 
> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
> 2010-09-26 09:54:13,298 INFO org.apache.hadoop.mapred.TaskTracker: Resending
> 'status' to 'vm221' with reponseId '25
> 2010-09-26 09:54:14,303 INFO org.apache.hadoop.ipc.Client: Retrying connect to
> server: vm221/10.11.2.221:9001. Already tried 0 time(s).
> 2010-09-26 09:54:14,379 INFO org.apache.hadoop.mapred.TaskTracker:
> SHUTDOWN_MSG: 
> /************************************************************
> SHUTDOWN_MSG: Shutting down TaskTracker at localhost/127.0.0.1
> ************************************************************/
> 
> datanode log:
> 
> 2010-09-26 09:54:16,234 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> java.io.IOException: Call to vm221/10.11.2.221:9000 failed on local exception:
> java.io.EOFException
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:743)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy4.sendHeartbeat(Unknown Source)
> at 
> 
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:702>
)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1186)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at 
> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
> 2010-09-26 09:54:17,660 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> SHUTDOWN_MSG: 
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at localhost/127.0.0.1
> ************************************************************/
> 2010-09-26 09:55:43,072 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> STARTUP_MSG: 
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG:   host = localhost/127.0.0.1
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 0.20.2
> STARTUP_MSG:   build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707;
> compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-09-26 09:56:39,903 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Registered FSDatasetStatusMBean
> 2010-09-26 09:56:39,905 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Opened info server at 50010
> 2010-09-26 09:56:39,908 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Balancing bandwith is 10485760 bytes/s
> 2010-09-26 09:56:39,969 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2010-09-26 09:56:40,040 INFO org.apache.hadoop.http.HttpServer: Port returned
> by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening
> the listener on 50075
> 2010-09-26 09:56:40,040 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50075
> webServer.getConnectors()[0].getLocalPort() returned 50075
> 2010-09-26 09:56:40,040 INFO org.apache.hadoop.http.HttpServer: Jetty bound to
> port 50075
> 2010-09-26 09:56:40,040 INFO org.mortbay.log: jetty-6.1.14
> 2010-09-26 09:56:55,697 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50075
> 2010-09-26 09:56:55,705 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=DataNode, sessionId=null
> 2010-09-26 09:56:55,724 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=DataNode, port=50020
> 2010-09-26 09:56:55,727 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2010-09-26 09:56:55,728 INFO org.apache.hadoop.ipc.Server: IPC Server listener
> on 50020: starting
> 2010-09-26 09:56:55,729 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 0 on 50020: starting
> 2010-09-26 09:56:55,730 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 1 on 50020: starting
> 2010-09-26 09:56:55,730 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 2 on 50020: starting
> 2010-09-26 09:56:55,730 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> dnRegistration = DatanodeRegistration(localhost:50010,
> storageID=DS-626517105-127.0.0.1-50010-1285405012557, infoPort=50075,
> ipcPort=50020)
> 2010-09-26 09:56:55,737 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(10.11.3.207:50010,
> storageID=DS-626517105-127.0.0.1-50010-1285405012557, infoPort=50075,
> ipcPort=50020)In DataNode.run, data =
> FSDataset{dirpath='/data0/hadoop-root/dfs/data/current'}
> 2010-09-26 09:56:55,738 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> using BLOCKREPORT_INTERVAL of 3600000msec Initial delay: 0msec
> 
> 
> 
> 
> 
> 
> 2010-09-26 
> 
> 
> 
> shangan 


iCrossing Privileged and Confidential Information
This email message is for the sole use of the intended recipient(s) and may contain \
confidential and privileged information of iCrossing. Any unauthorized review, use, \
disclosure or distribution is prohibited. If you are not the intended recipient, \
please contact the sender by reply email and destroy all copies of the original \
message.


[prev in list] [next in list] [prev in thread] [next in thread]
Configure | About | News | Add a list | Sponsored by KoreLogic