[prev in list] [next in list] [prev in thread] [next in thread]
List: hadoop-user
Subject: Why "java.util.zip.ZipOutputStream" need to use /tmp?
From: Steve Gao <steve.gao () yahoo ! com>
Date: 2009-08-27 19:42:13
Message-ID: 6395.68829.qm () web112107 ! mail ! gq1 ! yahoo ! com
[Download RAW message or body]
The hadoop version is 0.18.3 . Recently we got "out of space" issue. It's from \
"java.util.zip.ZipOutputStream". We found that /tmp is full and after cleaning /tmp \
the problem is solved.
However why hadoop needs to use /tmp? We had already configured hadoop tmp to a local \
disk in: hadoop-site.xml
<property>
<name>hadoop.tmp.dir</name>
<value> ... some large local disk ... </value>
</property>
Could it because java.util.zip.ZipOutputStream uses /tmp even if we configured \
hadoop.tmp.dir to a large local disk?
The error log is here FYI:
java.io.IOException: No space left on device
at java.io.FileOutputStream.write(Native Method)
at java.util.zip.ZipOutputStream.writeInt(ZipOutputStream.java:445)
at java.util.zip.ZipOutputStream.writeEXT(ZipOutputStream.java:362)
at java.util.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:220)
at java.util.zip.ZipOutputStream.finish(ZipOutputStream.java:301)
at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:146)
at java.util.zip.ZipOutputStream.close(ZipOutputStream.java:321)
at org.apache.hadoop.streaming.JarBuilder.merge(JarBuilder.java:79)
at org.apache.hadoop.streaming.StreamJob.packageJobJar(StreamJob.java:628)
at org.apache.hadoop.streaming.StreamJob.setJobConf(StreamJob.java:843)
at org.apache.hadoop.streaming.StreamJob.go(StreamJob.java:110)
at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) \
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) \
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)
Executing Hadoop job failure
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic