[prev in list] [next in list] [prev in thread] [next in thread]
List: flume-user
Subject: AWS S3 Flume Sink issue
From: Obaid Salikeen <Obaid.Salikeen () iacpublishinglabs ! com>
Date: 2016-01-06 22:00:55
Message-ID: E40C1D160258E64495F5BC69BD7FEDF56C709E80 () IAC-EXMAIL ! iac ! corp
[Download RAW message or body]
Hi All,
I am trying to use Flume (1.6.0) to gather events from Kafka and publish them on HDFS \
and S3. I am experiencing errors whenever I try to use S3 Sink. It seems like HDFS \
Sink does not recognize S3 filesystem, or I may be missing a configuration (Link I \
used for help: http://stackoverflow.com/questions/26028096/use-flume-to-stream-data-to-s3).
1. Configuration File:
# Sinks
# 2. S3
etl.sinks.s3-sink.type = hdfs
etl.sinks.s3-sink.hdfs.path = s3n://XXXX:YYYY@myBucket/dev/
etl.sinks.s3-sink.hdfs.fileType = DataStream
2. Following is the error I get:
06 Jan 2016 13:00:40,941 WARN [hdfs-hdfs-sink-call-runner-0] \
(org.apache.hadoop.util.NativeCodeLoader.<clinit>:62) - Unable to load native-hadoop \
library for your platform... using builtin-java classes where applicable 06 Jan 2016 \
13:00:41,093 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] \
(org.apache.flume.sink.hdfs.BucketWriter.open:234) - Creating \
s3://XXX:YYY@dp-unified-etl//FlumeData.1452114040235.tmp 06 Jan 2016 13:00:41,121 \
WARN [SinkRunner-PollingRunner-DefaultSinkProcessor] \
(org.apache.flume.sink.hdfs.HDFSEventSink.process:455) - HDFS IO \
error
java.io.IOException: No FileSystem for scheme: s3
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243)
at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235)
at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Any help to resolve this issue will be highly appreciated !
Thanks
Obaid
[Attachment #3 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:356926139;
mso-list-type:hybrid;
mso-list-template-ids:1291096498 67698703 67698713 67698715 67698703 67698713 \
67698715 67698703 67698713 67698715;} @list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi All,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am trying to use Flume (1.6.0) to gather events from Kafka and \
publish them on HDFS and S3. I am experiencing errors whenever I try to use S3 Sink. \
It seems like HDFS Sink does not recognize S3 filesystem, or I may be missing a \
configuration (Link I used for help: <a \
href="http://stackoverflow.com/questions/26028096/use-flume-to-stream-data-to-s3"> \
http://stackoverflow.com/questions/26028096/use-flume-to-stream-data-to-s3</a>). \
<o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if \
!supportLists]><b><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times \
New Roman""> \
</span></span></b><![endif]><b>Configuration File:<o:p></o:p></b></p> <p \
class="MsoNormal" style="margin-left:.25in;text-autospace:none"><span \
style="font-size:10.0pt;font-family:Consolas;color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"># \
Sinks<o:p></o:p></span></p> <p class="MsoNormal" \
style="margin-left:.25in;text-autospace:none"><span \
style="font-size:10.0pt;font-family:Consolas;color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"># \
2. S3<o:p></o:p></span></p> <p class="MsoNormal" \
style="margin-left:.25in;text-autospace:none"><span \
style="font-size:10.0pt;font-family:Consolas;color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%">etl.sinks.<b>s3-sink</b>.type \
= hdfs<o:p></o:p></span></p> <p class="MsoNormal" \
style="margin-left:.25in;text-autospace:none"><span \
style="font-size:10.0pt;font-family:Consolas;color:#1F4E79;mso-style-textfill-fill-col \
or:#1F4E79;mso-style-textfill-fill-alpha:100.0%">etl.sinks.<b>s3-sink</b>.hdfs.path = \
s3n://XXXX:YYYY@myBucket/dev/<o:p></o:p></span></p> <p class="MsoNormal" \
style="margin-left:.25in;text-autospace:none"><span \
style="font-size:10.0pt;font-family:Consolas;color:#1F4E79;mso-style-textfill-fill-col \
or:#1F4E79;mso-style-textfill-fill-alpha:100.0%">etl.sinks.<b>s3-sink</b>.hdfs.fileType \
= DataStream<o:p></o:p></span></p> <p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if \
!supportLists]><b><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times \
New Roman""> \
</span></span></b><![endif]><b>Following is the error I get:<o:p></o:p></b></p> <p \
class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%">06 \
Jan 2016 13:00:40,941 WARN [hdfs-hdfs-sink-call-runner-0] \
(org.apache.hadoop.util.NativeCodeLoader.<clinit>:62)
- Unable to load native-hadoop library for your platform... using builtin-java \
classes where applicable<o:p></o:p></span></p> <p class="MsoNormal" \
style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%">06 \
Jan 2016 13:00:41,093 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] \
(org.apache.flume.sink.hdfs.BucketWriter.open:234)
- Creating s3://XXX:YYY@dp-unified-etl//FlumeData.1452114040235.tmp<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%">06 \
Jan 2016 13:00:41,121 WARN [SinkRunner-PollingRunner-DefaultSinkProcessor] \
(org.apache.flume.sink.hdfs.HDFSEventSink.process:455)
- HDFS IO error<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%">java.io.IOException:
<b><span style="background:yellow;mso-highlight:yellow">No FileSystem for scheme: \
s3</span></b><o:p></o:p></span></p> <p class="MsoNormal" \
style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"> \
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"> \
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"> \
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"> \
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"> \
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"> \
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)<o:p></o:p></span></p> <p \
class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"> \
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)<o:p></o:p></span></p> <p \
class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"> \
at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"> \
at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"> \
at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"> \
at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"> \
at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"> \
at java.util.concurrent.FutureTask.run(FutureTask.java:262)<o:p></o:p></span></p> <p \
class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"> \
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"> \
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span \
style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"> \
at java.lang.Thread.run(Thread.java:745)<o:p></o:p></span></p> <p \
class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Any help to resolve this issue will be highly appreciated \
!<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks<o:p></o:p></p>
<p class="MsoNormal">Obaid<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic