[prev in list] [next in list] [prev in thread] [next in thread]
List: flume-user
Subject: Re: HDFS escape sequence and use of timestamp header
From: Mohit Anchlia <mohitanchlia () gmail ! com>
Date: 2012-08-23 18:33:45
Message-ID: CAOT3TWq9cgLDD2wFjXcSJ7CTL1E+=jXNUQK2oHTXQ7EiYRnbMQ () mail ! gmail ! com
[Download RAW message or body]
thanks that works
On Wed, Aug 22, 2012 at 5:52 PM, Kathleen Ting <kathleen@apache.org> wrote:
> Mohit,
>
> As it happens, both the example and the table are correct. You can
> specify either the fully qualified class name or the alias 'TIMESTAMP'
>
> Regards, Kathleen
>
> On Wed, Aug 22, 2012 at 5:05 PM, Mohit Anchlia <mohitanchlia@gmail.com>
> wrote:
> >
> >
> > On Wed, Aug 22, 2012 at 2:16 PM, Kathleen Ting <kathleen@apache.org>
> wrote:
> >>
> >> Hi Mohit,
> >>
> >> You can configure a timestamp interceptor onto your source as follows:
> >>
> >> agent.sources.src-0.interceptors.ts.type = TIMESTAMP
> >>
> >> Once the timestamp interceptor is in place, you can use, in sinks, the
> >> timestamp it writes. Here is an example of using the timestamp in an
> >> HDFS sink:
> >>
> >> agent.sinks.sink-0.hdfs.filePrefix = FlumeData.%Y-%m-%d
> >>
> >> Here is more info about the timestamp interceptor:
> >> http://flume.apache.org/FlumeUserGuide.html#timestamp-interceptor
> >
> > Thanks this helps. The type specified in the example is a fully qualified
> > class name, however in the table it says use type as TIMESTAMP. I guess
> > example is not correct and I should just use TIMESTAMP.
> >>
> >> Regards, Kathleen
> >>
> >> On Wed, Aug 22, 2012 at 11:50 AM, Mohit Anchlia <mohitanchlia@gmail.com
> >
> >> wrote:
> >> > I see this JIRA https://issues.apache.org/jira/browse/FLUME-1215 but
> how
> >> > do
> >> > I take advantage of it? I am using 1.2.0 but %Y %m doesn't work. I
> just
> >> > get
> >> > number format exceptions.
> >> >
> >> >
> >> > On Tue, Aug 21, 2012 at 5:52 PM, Mohit Anchlia <
> mohitanchlia@gmail.com>
> >> > wrote:
> >> >>
> >> >> I am using flume-ng 1.2.0 and I need to use %Y%M%D escape sequence.
> Do
> >> >> I
> >> >> need to write some custom interceptor? Could you please point me to
> an
> >> >> example? Currently my AvroClient looks like this:
> >> >>
> >> >>
> >> >> public void sendDataToFlume(String data) {
> >> >>
> >> >> // Create flume event object
> >> >>
> >> >> Event event = EventBuilder.withBody(data, Charset.forName(
> >> >>
> >> >> "UTF-8"));
> >> >>
> >> >> Map<String,String> headers =
> >> >>
> >> >> new HashMap<String,String>();
> >> >>
> >> >> headers.put(
> >> >>
> >> >> "host", hostName);
> >> >>
> >> >> event.setHeaders(headers);
> >> >>
> >> >> try {
> >> >>
> >> >> rpcClient.append(event);
> >> >>
> >> >> }
> >> >>
> >> >> catch (EventDeliveryException e) {
> >> >>
> >> >> connect();
> >> >>
> >> >> }
> >> >>
> >> >> }
> >> >
> >> >
> >
> >
>
[Attachment #3 (text/html)]
thanks that works<br><br>
<div class="gmail_quote">On Wed, Aug 22, 2012 at 5:52 PM, Kathleen Ting <span \
dir="ltr"><<a href="mailto:kathleen@apache.org" \
target="_blank">kathleen@apache.org</a>></span> wrote:<br> <blockquote \
style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" \
class="gmail_quote">Mohit,<br><br>As it happens, both the example and the table are \
correct. You can<br>specify either the fully qualified class name or the alias \
'TIMESTAMP'<br> <br>Regards, Kathleen<br>
<div class="HOEnZb">
<div class="h5"><br>On Wed, Aug 22, 2012 at 5:05 PM, Mohit Anchlia <<a \
href="mailto:mohitanchlia@gmail.com">mohitanchlia@gmail.com</a>> \
wrote:<br>><br>><br>> On Wed, Aug 22, 2012 at 2:16 PM, Kathleen Ting <<a \
href="mailto:kathleen@apache.org">kathleen@apache.org</a>> wrote:<br> \
>><br>>> Hi Mohit,<br>>><br>>> You can configure a timestamp \
interceptor onto your source as follows:<br>>><br>>> \
agent.sources.src-0.interceptors.ts.type = TIMESTAMP<br>>><br>>> Once the \
timestamp interceptor is in place, you can use, in sinks, the<br> >> timestamp \
it writes. Here is an example of using the timestamp in an<br>>> HDFS \
sink:<br>>><br>>> agent.sinks.sink-0.hdfs.filePrefix = \
FlumeData.%Y-%m-%d<br>>><br>>> Here is more info about the timestamp \
interceptor:<br> >> <a \
href="http://flume.apache.org/FlumeUserGuide.html#timestamp-interceptor" \
target="_blank">http://flume.apache.org/FlumeUserGuide.html#timestamp-interceptor</a><br>><br>> \
Thanks this helps. The type specified in the example is a fully qualified<br> > \
class name, however in the table it says use type as TIMESTAMP. I guess<br>> \
example is not correct and I should just use TIMESTAMP.<br>>><br>>> \
Regards, Kathleen<br>>><br>>> On Wed, Aug 22, 2012 at 11:50 AM, Mohit \
Anchlia <<a href="mailto:mohitanchlia@gmail.com">mohitanchlia@gmail.com</a>><br>
>> wrote:<br>>> > I see this JIRA <a \
href="https://issues.apache.org/jira/browse/FLUME-1215" \
target="_blank">https://issues.apache.org/jira/browse/FLUME-1215</a> but \
how<br>>> > do<br>>> > I take advantage of it? I am using 1.2.0 but \
%Y %m doesn't work. I just<br> >> > get<br>>> > number format \
exceptions.<br>>> ><br>>> ><br>>> > On Tue, Aug 21, 2012 \
at 5:52 PM, Mohit Anchlia <<a \
href="mailto:mohitanchlia@gmail.com">mohitanchlia@gmail.com</a>><br> >> > \
wrote:<br>>> >><br>>> >> I am using flume-ng 1.2.0 and I need \
to use %Y%M%D escape sequence. Do<br>>> >> I<br>>> >> need to \
write some custom interceptor? Could you please point me to an<br> >> >> \
example? Currently my AvroClient looks like this:<br>>> >><br>>> \
>><br>>> >> public void sendDataToFlume(String data) {<br>>> \
>><br>>> >> // Create flume event object<br> >> \
>><br>>> >> Event event = EventBuilder.withBody(data, \
Charset.forName(<br>>> >><br>>> >> \
"UTF-8"));<br>>> >><br>>> >> \
Map<String,String> headers =<br> >> >><br>>> >> new \
HashMap<String,String>();<br>>> >><br>>> >> \
headers.put(<br>>> >><br>>> >> "host", \
hostName);<br>>> >><br> >> >> \
event.setHeaders(headers);<br>>> >><br>>> >> try \
{<br>>> >><br>>> >> rpcClient.append(event);<br>>> \
>><br>>> >> }<br>>> >><br> >> >> catch \
(EventDeliveryException e) {<br>>> >><br>>> >> \
connect();<br>>> >><br>>> >> }<br>>> \
>><br>>> >> }<br>>> ><br>>> ><br> \
><br>><br></div></div></blockquote></div><br>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic