[prev in list] [next in list] [prev in thread] [next in thread] 

List:       flume-user
Subject:    Re: Query regarding customizing baseNameHeader value
From:       Jyotsna G <jyotsna.g () gmail ! com>
Date:       2018-06-29 10:01:25
Message-ID: CA+p0n+metjXWXF-gX0Au3fhrMgnSYU2NWzvYxZyA+QqQswt_sw () mail ! gmail ! com
[Download RAW message or body]

Thank you Peter.
Followed your advice and implemented a custom interceptor to remove the ":"
character.

Regards,
Jyotsna

On Thu, Jun 28, 2018 at 2:43 PM, Peter Turcsanyi <turcsanyi@cloudera.com>
wrote:

> Hi Jyotsna,
>
> I'm afraid you need to write a custom interceptor to replace characters in
> the event header fields (similar to SearchAndReplaceInterceptor, but this
> works only with the event body).
>
> Regards,
> Peter Turcsanyi
>
>
> On Wed, Jun 27, 2018 at 2:00 PM, Jyotsna G <jyotsna.g@gmail.com> wrote:
>
>> Hello,
>>
>> I have a spoolDir flume agent configured and the source path contains ":"
>> colon character in it which results in the below exception while writing in
>> to HDFS.
>>
>> org.apache.flume.EventDeliveryException: java.lang.IllegalArgumentException:
>> Pathname /TPA_PAAS/20180626.01:21:14/IDCS-CertificateAnalysis.csv.1530084044272.tmp
>> from hdfs://slc15zcc:8020/TPA_PAAS/*20180626.01:21:14*/IDCS-Certifi
>> cateAnalysis.csv.1530084044272.tmp* is not a valid DFS filename.*
>>
>> https://issues.apache.org/jira/browse/HADOOP-14217
>>
>> What's the best way to work around this issue?
>>
>> 1) Is there a flume interceptor which I can configure to edit the
>> baseNameHeader value in the header?
>>
>> 2) Should I build a custom interceptor to remove/replace the colon
>> character in the basenameHeader?
>>
>> Thanks & Regards,
>> Jyotsna
>>
>
>

[Attachment #3 (text/html)]

<div dir="ltr">Thank you Peter.  <div>Followed your advice and implemented a custom \
interceptor to remove the &quot;:&quot; character.  \
</div><div><br></div><div>Regards,</div><div>Jyotsna</div></div><div \
class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 28, 2018 at 2:43 PM, \
Peter Turcsanyi <span dir="ltr">&lt;<a href="mailto:turcsanyi@cloudera.com" \
target="_blank">turcsanyi@cloudera.com</a>&gt;</span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr">Hi  <span \
style="font-size:12.800000190734863px">Jyotsna,</span><div><span \
style="font-size:12.800000190734863px"><br></span></div><div><span \
style="font-size:12.800000190734863px">I&#39;m afraid you need to write a custom \
interceptor to replace characters in the event header fields (similar to  \
</span><span style="color:rgb(0,0,0);font-family:Menlo;font-size:9pt">SearchAndReplaceInterceptor</span><span \
style="font-size:12.800000190734863px"><wbr>, but this works only with the event \
body).</span></div><div><span \
style="font-size:12.800000190734863px"><br></span></div><div><span \
style="font-size:12.800000190734863px">Regards,</span></div><div><span \
style="font-size:12.800000190734863px">Peter Turcsanyi</span></div><div><span \
style="font-size:12.800000190734863px"><br></span></div></div><div \
class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div \
class="gmail_quote">On Wed, Jun 27, 2018 at 2:00 PM, Jyotsna G <span dir="ltr">&lt;<a \
href="mailto:jyotsna.g@gmail.com" target="_blank">jyotsna.g@gmail.com</a>&gt;</span> \
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div>I have a \
spoolDir flume agent configured and the source path contains &quot;:&quot; colon \
character in it which results in the below exception while writing in to HDFS.  \
</div><div><br></div><div>org.apache.flume.EventDelivery<wbr>Exception: \
java.lang.IllegalArgumentExcep<wbr>tion: Pathname \
/TPA_PAAS/20180626.01:21:14/ID<wbr>CS-CertificateAnalysis.csv.153<wbr>0084044272.tmp \
from hdfs://slc15zcc:8020/TPA_PAAS/<b><wbr>20180626.01:21:14</b>/IDCS-Certifi<wbr>cateAnalysis.csv.1530084044272<wbr>.tmp<b> \
is not a valid DFS filename.</b><br></div><div><br></div><div><a \
href="https://issues.apache.org/jira/browse/HADOOP-14217" \
target="_blank">https://issues.apache.org/jira<wbr>/browse/HADOOP-14217</a><br></div><div><br></div><div>What&#39;s \
the best way to work around this issue?  </div><div><br></div><div>1) Is there a \
flume interceptor which I can configure to edit the baseNameHeader value in the \
header?    </div><div><br></div><div>2) Should I build a custom interceptor to \
remove/replace the colon character in the \
basenameHeader?</div><div><br></div><div>Thanks &amp; \
Regards,</div><div>Jyotsna</div></div> </blockquote></div><br></div>
</div></div></blockquote></div><br></div>



[prev in list] [next in list] [prev in thread] [next in thread]