[prev in list] [next in list] [prev in thread] [next in thread]
List: flume-user
Subject: Re: Skippin those gost darn 0 byte diles
From: Bertrand Dechoux <dechouxb () gmail ! com>
Date: 2014-07-23 6:00:35
Message-ID: CAO6W-2d_Dqr4uPBHPrvkyfso5h5d9pWE4-t0MH1OM9k-S4Rggg () mail ! gmail ! com
[Download RAW message or body]
The best would be to get a hold on a Flume developer. I am not strictly
sure of all the differences between sync/flush/hsync/hflush and the
different hadoop versions. It might be the case that you are only flushing
on the client side. Even if it was a clean strategy, creation+flush is
unlikely to be an atomic operation.
It is worth testing the read of an empty sequence file (real empty and with
only header). It should be quite easy with a unit test. A solution would
indeed to validate the behaviour of SequenceFileReader / InputFormat on
edge cases. But nothing guarantee you that you won't have a record split
between two HDFS blocks. This implies that during the writing only the
first block is visible and only a part of the record. It would be normal
for the reader to fail on that case. You could tweak mapreduce bad records
skipping but that feels like hacking a system where the design is wrong
from the beginning.
Anyway, a solution (seen in Flume if I remember correctly) is having a good
file name strategy. For exemple, all new files should end in ".open" and
only when they are finished the suffix is removed. Then for processing, you
only target the latter.
For Hive, you might need to adapt the strategy a bit because Hive may not
be able to target only files with a specific name (you are the expert). A
simple move of the file from a temporary directory to the table directory
would have the same effect (because from the point of view of HDFS, it's
the same operation : metadata change only).
Bertrand Dechoux
On Wed, Jul 23, 2014 at 12:16 AM, Edward Capriolo <edlinuxguru@gmail.com>
wrote:
> Here is the stack trace...
>
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:267)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329)
> at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:2072)
> at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:2139)
> at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2214)
> at org.apache.hadoop.mapred.SequenceFileRecordReader.getCurrentValue(SequenceFileRecordReader.java:109)
> at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:84)
> at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:274)
>
> ... 15 more
>
>
>
>
> On Tue, Jul 22, 2014 at 6:14 PM, Edward Capriolo <edlinuxguru@gmail.com>
> wrote:
>
> > Currently using:
> >
> > <dependency>
> > <groupId>org.apache.hadoop</groupId>
> > <artifactId>hadoop-hdfs</artifactId>
> > <version>2.3.0</version>
> > </dependency>
> >
> >
> > I have this piece of code that does.
> >
> > writer = SequenceFile.createWriter(fs, conf, p, Text.class, Text.class,
> > CompressionType.BLOCK, codec);
> >
> > Then I have a piece of code like this...
> >
> > public static final long SYNC_EVERY_LINES = 1000;
> > if (meta.getLinesWritten() % SYNC_EVERY_LINES == 0){
> > meta.getWriter().sync();
> > }
> >
> >
> > And I commonly see:
> >
> > [ecapriolo@staging-hadoop-cdh-67-14 ~]$ hadoop dfs -ls /user/beacon/
> > 2014072117
> > DEPRECATED: Use of this script to execute hdfs command is deprecated.
> > Instead use the hdfs command for it.
> >
> > Found 12 items
> > -rw-r--r-- 3 service-igor supergroup 1065682 2014-07-21 17:50
> > /user/beacon/2014072117/0bb6cd71-70ac-405a-a8b7-b8caf9af8da1
> > -rw-r--r-- 3 service-igor supergroup 1029041 2014-07-21 17:40
> > /user/beacon/2014072117/1b0ef6b3-bd51-4100-9d4b-1cecdd565f93
> > -rw-r--r-- 3 service-igor supergroup 1002096 2014-07-21 17:10
> > /user/beacon/2014072117/34e2acb4-2054-44df-bbf7-a4ce7f1e5d1b
> > -rw-r--r-- 3 service-igor supergroup 1028450 2014-07-21 17:30
> > /user/beacon/2014072117/41c7aa62-d27f-4d53-bed8-df2fb5803c92
> > -rw-r--r-- 3 service-igor supergroup 0 2014-07-21 17:50
> > /user/beacon/2014072117/5450f246-7623-4bbd-8c97-8176a0c30351
> > -rw-r--r-- 3 service-igor supergroup 1084873 2014-07-21 17:30
> > /user/beacon/2014072117/8b36fbca-6f5b-48a3-be3c-6df6254c3db2
> > -rw-r--r-- 3 service-igor supergroup 1043108 2014-07-21 17:20
> > /user/beacon/2014072117/949da11a-247b-4992-b13a-5e6ce7e51e9b
> > -rw-r--r-- 3 service-igor supergroup 986866 2014-07-21 17:10
> > /user/beacon/2014072117/979bba76-4d2e-423f-92f6-031bc41f6fbd
> > -rw-r--r-- 3 service-igor supergroup 0 2014-07-21 17:50
> > /user/beacon/2014072117/b76db189-054f-4dac-84a4-a65f39a6c1a9
> > -rw-r--r-- 3 service-igor supergroup 1040931 2014-07-21 17:50
> > /user/beacon/2014072117/bba6a677-226c-4982-8fb2-4b136108baf1
> > -rw-r--r-- 3 service-igor supergroup 1012137 2014-07-21 17:40
> > /user/beacon/2014072117/be940202-f085-45bb-ac84-51ece2e1ba47
> > -rw-r--r-- 3 service-igor supergroup 1028467 2014-07-21 17:20
> > /user/beacon/2014072117/c336e0c8-76e7-40e7-98e2-9f529f25577b
> >
> > Sometimes even though they show as 0 bytes you can read data from them.
> > Sometimes it blows up with a stack trace I have lost.
> >
> >
> > On Tue, Jul 22, 2014 at 5:45 PM, Bertrand Dechoux <dechouxb@gmail.com>
> > wrote:
> >
> > > I looked at the source by curiosity, for the latest version (2.4), the
> > > header is flushed during the writer creation. Of course, key/value classes
> > > are provided. By 0-bytes, you really mean even without the header? Or 0
> > > bytes of payload?
> > >
> > >
> > > On Tue, Jul 22, 2014 at 11:05 PM, Bertrand Dechoux <dechouxb@gmail.com>
> > > wrote:
> > >
> > > > The header is expected to have the full name of the key class and value
> > > > class so if it is only detected with the first record (?) indeed the file
> > > > can not respect its own format.
> > > >
> > > > I haven't tried it but LazyOutputFormat should solve your problem.
> > > >
> > > > https://hadoop.apache.org/docs/current/api/index.html?org/apache/hadoop/mapred/lib/LazyOutputFormat.html
> > > >
> > > > Regards
> > > >
> > > > Bertrand Dechoux
> > > >
> > > >
> > > > Bertrand Dechoux
> > > >
> > > >
> > > > On Tue, Jul 22, 2014 at 10:39 PM, Edward Capriolo <
> > > > edlinuxguru@gmail.com> wrote:
> > > >
> > > > > I have two processes. One that writes sequence files directly to hdfs,
> > > > > the other that is a hive table that reads these files.
> > > > >
> > > > > All works well with the exception that I am only flushing the files
> > > > > periodically. SequenceFile input format gets angry when it encounters
> > > > > 0-bytes seq files.
> > > > >
> > > > > I was considering flush and sync on first record write. Also was
> > > > > thinking should just be able to hack sequence file input format to skip 0
> > > > > byte files and not throw exception on readFully() which it sometimes does.
> > > > >
> > > > > Anyone ever tackled this?
> > > > >
> > > >
> > > >
> > >
> >
>
[Attachment #3 (text/html)]
<div dir="ltr">The best would be to get a hold on a Flume developer. I am not \
strictly sure of all the differences between sync/flush/hsync/hflush and the \
different hadoop versions. It might be the case that you are only flushing on the \
client side. Even if it was a clean strategy, creation+flush is unlikely to be an \
atomic operation.<div> <br></div><div>It is worth testing the read of an empty \
sequence file (real empty and with only header). It should be quite easy with a unit \
test. A solution would indeed to validate the behaviour of SequenceFileReader / \
InputFormat on edge cases. But nothing guarantee you that you won't have a record \
split between two HDFS blocks. This implies that during the writing only the first \
block is visible and only a part of the record. It would be normal for the reader to \
fail on that case. You could tweak mapreduce bad records skipping but that feels like \
hacking a system where the design is wrong from the beginning.<br> \
<div><br></div><div>Anyway, a solution (seen in Flume if I remember correctly) is \
having a good file name strategy. For exemple, all new files should end in \
".open" and only when they are finished the suffix is removed. Then for \
processing, you only target the latter.</div> </div><div><br></div><div>For Hive, you \
might need to adapt the strategy a bit because Hive may not be able to target only \
files with a specific name (you are the expert). A simple move of the file from a \
temporary directory to the table directory would have the same effect (because from \
the point of view of HDFS, it's the same operation : metadata change only).</div> \
<div class="gmail_extra"><br clear="all"><div>Bertrand Dechoux</div> <br><br><div \
class="gmail_quote">On Wed, Jul 23, 2014 at 12:16 AM, Edward Capriolo <span \
dir="ltr"><<a href="mailto:edlinuxguru@gmail.com" \
target="_blank">edlinuxguru@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Here is the stack trace...<br><br><pre \
style="overflow:auto;min-height:149px"> Caused by: java.io.EOFException at \
java.io.DataInputStream.readByte(DataInputStream.java:267) at \
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308) at \
org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329) at \
org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:2072) at \
org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:2139) \
at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2214) \
at org.apache.hadoop.mapred.SequenceFileRecordReader.getCurrentValue(SequenceFileRecordReader.java:109)
at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:84)
at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:274)
... 15 more</pre><br></div><div class=""><div class="h5"><div \
class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jul 22, 2014 at 6:14 PM, \
Edward Capriolo <span dir="ltr"><<a href="mailto:edlinuxguru@gmail.com" \
target="_blank">edlinuxguru@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div \
dir="ltr"><div><div><div>Currently using:<br><br> <dependency><br> \
<groupId>org.apache.hadoop</groupId><br>
<artifactId>hadoop-hdfs</artifactId><br> \
<version>2.3.0</version><br> </dependency><br></div><div><br><br>I \
have this piece of code that does.<br><br>writer = SequenceFile.createWriter(fs, \
conf, p, Text.class, Text.class, CompressionType.BLOCK, codec);<br><br></div>Then I \
have a piece of code like this...<br>
<br> public static final long SYNC_EVERY_LINES = 1000;<br> if \
(meta.getLinesWritten() % SYNC_EVERY_LINES == 0){<br> \
meta.getWriter().sync();<br> }<br><br><br></div>And I commonly \
see:<br><br>[ecapriolo@staging-hadoop-cdh-67-14 ~]$ hadoop dfs -ls /user/beacon/<a \
href="tel:2014072117" value="+12014072117" target="_blank">2014072117</a><br>
DEPRECATED: Use of this script to execute hdfs command is deprecated.<br>Instead use \
the hdfs command for it.<br><br>Found 12 items<br>-rw-r--r-- 3 service-igor \
supergroup 1065682 2014-07-21 17:50 /user/beacon/<a href="tel:2014072117" \
value="+12014072117" \
target="_blank">2014072117</a>/0bb6cd71-70ac-405a-a8b7-b8caf9af8da1<br>
-rw-r--r-- 3 service-igor supergroup 1029041 2014-07-21 17:40 \
/user/beacon/<a href="tel:2014072117" value="+12014072117" \
target="_blank">2014072117</a>/1b0ef6b3-bd51-4100-9d4b-1cecdd565f93<br>-rw-r--r-- \
3 service-igor supergroup 1002096 2014-07-21 17:10 /user/beacon/<a \
href="tel:2014072117" value="+12014072117" \
target="_blank">2014072117</a>/34e2acb4-2054-44df-bbf7-a4ce7f1e5d1b<br>
-rw-r--r-- 3 service-igor supergroup 1028450 2014-07-21 17:30 \
/user/beacon/<a href="tel:2014072117" value="+12014072117" \
target="_blank">2014072117</a>/41c7aa62-d27f-4d53-bed8-df2fb5803c92<br>-rw-r--r-- \
3 service-igor supergroup 0 2014-07-21 17:50 /user/beacon/<a \
href="tel:2014072117" value="+12014072117" \
target="_blank">2014072117</a>/5450f246-7623-4bbd-8c97-8176a0c30351<br>
-rw-r--r-- 3 service-igor supergroup 1084873 2014-07-21 17:30 \
/user/beacon/<a href="tel:2014072117" value="+12014072117" \
target="_blank">2014072117</a>/8b36fbca-6f5b-48a3-be3c-6df6254c3db2<br>-rw-r--r-- \
3 service-igor supergroup 1043108 2014-07-21 17:20 /user/beacon/<a \
href="tel:2014072117" value="+12014072117" \
target="_blank">2014072117</a>/949da11a-247b-4992-b13a-5e6ce7e51e9b<br>
-rw-r--r-- 3 service-igor supergroup 986866 2014-07-21 17:10 \
/user/beacon/<a href="tel:2014072117" value="+12014072117" \
target="_blank">2014072117</a>/979bba76-4d2e-423f-92f6-031bc41f6fbd<br>-rw-r--r-- \
3 service-igor supergroup 0 2014-07-21 17:50 /user/beacon/<a \
href="tel:2014072117" value="+12014072117" \
target="_blank">2014072117</a>/b76db189-054f-4dac-84a4-a65f39a6c1a9<br>
-rw-r--r-- 3 service-igor supergroup 1040931 2014-07-21 17:50 \
/user/beacon/<a href="tel:2014072117" value="+12014072117" \
target="_blank">2014072117</a>/bba6a677-226c-4982-8fb2-4b136108baf1<br>-rw-r--r-- \
3 service-igor supergroup 1012137 2014-07-21 17:40 /user/beacon/<a \
href="tel:2014072117" value="+12014072117" \
target="_blank">2014072117</a>/be940202-f085-45bb-ac84-51ece2e1ba47<br>
-rw-r--r-- 3 service-igor supergroup 1028467 2014-07-21 17:20 \
/user/beacon/<a href="tel:2014072117" value="+12014072117" \
target="_blank">2014072117</a>/c336e0c8-76e7-40e7-98e2-9f529f25577b<br><br></div>Sometimes \
even though they show as 0 bytes you can read data from them. Sometimes it blows up \
with a stack trace I have lost.<br>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jul 22, 2014 \
at 5:45 PM, Bertrand Dechoux <span dir="ltr"><<a href="mailto:dechouxb@gmail.com" \
target="_blank">dechouxb@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div \
dir="ltr">I looked at the source by curiosity, for the latest version (2.4), the \
header is flushed during the writer creation. Of course, key/value classes are \
provided. By 0-bytes, you really mean even without the header? Or 0 bytes of \
payload?<div class="gmail_extra">
<br><br><div class="gmail_quote">On Tue, Jul 22, 2014 at 11:05 PM, Bertrand Dechoux \
<span dir="ltr"><<a href="mailto:dechouxb@gmail.com" \
target="_blank">dechouxb@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">The header is expected to have the full name of the key class and \
value class so if it is only detected with the first record (?) indeed the file can \
not respect its own format.<div><br></div><div>I haven't tried it but \
LazyOutputFormat should solve your problem.</div>
<div><a href="https://hadoop.apache.org/docs/current/api/index.html?org/apache/hadoop/mapred/lib/LazyOutputFormat.html" \
target="_blank">https://hadoop.apache.org/docs/current/api/index.html?org/apache/hadoop/mapred/lib/LazyOutputFormat.html</a><br>
</div><div><br></div><div>Regards</div><div><br></div><div>Bertrand \
Dechoux</div><span><font color="#888888"><div><br></div></font></span></div><div \
class="gmail_extra"><span><font color="#888888"><br clear="all"> <div>Bertrand \
Dechoux</div></font></span><div><div> <br><br><div class="gmail_quote">On Tue, Jul \
22, 2014 at 10:39 PM, Edward Capriolo <span dir="ltr"><<a \
href="mailto:edlinuxguru@gmail.com" \
target="_blank">edlinuxguru@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div><div><div>I have two processes. One that writes sequence files \
directly to hdfs, the other that is a hive table that reads these \
files.<br><br></div>All works well with the exception that I am only flushing the \
files periodically. SequenceFile input format gets angry when it encounters 0-bytes \
seq files.<br>
<br></div>I was considering flush and sync on first record write. Also was thinking \
should just be able to hack sequence file input format to skip 0 byte files and not \
throw exception on readFully() which it sometimes does.<br>
<br></div>Anyone ever tackled this?<br></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div></div>
</blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic