[prev in list] [next in list] [prev in thread] [next in thread]
List: flume-user
Subject: decompress snappy file if it was left open in hdfs
From: <Indrek.Maestu () nortal ! com>
Date: 2017-06-22 10:06:23
Message-ID: 3566b2f3be594915bdcfe06eabd38756 () talexN2 ! webmedia ! int
[Download RAW message or body]
We using flume ingestig data into HDFS.
Flume sink is configured fileType=CompressedStream and codeC=snappy
If on any reason flume agent dies or namenode (HA) restarts, flume current sink file \
will left open - .tmp extension. testfile.snappy.tmp for example.
Is there any way to decompress such file and get data back human readable form?
Or is there any tool to fix such files?
We can use any other compression too, if there is a way to fix such files.
Indrek
[Attachment #3 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=windows-1257">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
mso-fareast-language:EN-US;}
@page WordSection1
{size:8.5in 11.0in;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ET" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">We using flume ingestig data into HDFS.<o:p></o:p></p>
<p class="MsoNormal">Flume sink is configured fileType=CompressedStream and \
codeC=snappy<o:p></o:p></p> <p class="MsoNormal">If on any reason flume agent dies or \
namenode (HA) restarts, flume current sink file will left open - .tmp \
extension.<o:p></o:p></p> <p class="MsoNormal">testfile.snappy.tmp for \
example.<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Is there any way to decompress such file and get data back human \
readable form?<o:p></o:p></p> <p class="MsoNormal">Or is there any tool to fix such \
files?<o:p></o:p></p> <p class="MsoNormal">We can use any other compression too, if \
there is a way to fix such files.<o:p></o:p></p> <p \
class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">Indrek<o:p></o:p></p>
</div>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic