[prev in list] [next in list] [prev in thread] [next in thread]
List: hadoop-user
Subject: randomtextwriter can not write 100GB text file on a 500GB cluster
From: "Richard Zhang" <richardtechzh () gmail ! com>
Date: 2008-06-29 21:29:28
Message-ID: f127732c0806291429p132bc151r5d41a53c857b5241 () mail ! gmail ! com
[Download RAW message or body]
Hi folks:
I am trying to write a 100GB text file on a cluster with 500GB free storage
space. For smaller scale writing such as 100MB, 1G, it works fine. But it
shows the DFS client can not complete writing errors when I ran with 100GB
writing. Does anyone has any ideas on these types of errors or met it
before?
Below is the command I used:
bin/hadoop jar hadoop-0.17.0-examples.jar randomtextwriter -D
test.randomtextwrite.min_words_key=5 -D
test.randomtextwrite.max_words_key -D
test.randomtextwrite.min_words_value
test.randomtextwrite.max_words_value0 -D
test.randomtextwrite.total_bytes7374182000 rand-text.
And the below are the errors:
08/06/29 14:18:18 INFO mapred.JobClient: map 1% reduce 0%
08/06/29 14:18:20 INFO mapred.JobClient: Task Id :
task_200806291409_0001_m_000012_0, Status : FAILED
org.apache.hadoop.ipc.RemoteException: java.io.IOException: Could not
complete write to file
/user/hadoop/rand-text/_temporary/_task_200806291409_0001_m_000012_0/part-00012
by DFSClient_task_200806291409_0001_m_000012_0
at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:332)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
at org.apache.hadoop.ipc.Client.call(Client.java:557)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
at org.apache.hadoop.dfs.$Proxy1.complete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at org.apache.hadoop.dfs.$Proxy1.complete(Unknown Source)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:2655)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:2576)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:59)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:79)
at
org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:969)
at
org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:79)
at
org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:257)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:224)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic