[prev in list] [next in list] [prev in thread] [next in thread] 

List:       flume-dev
Subject:    [jira] [Created] (FLUME-2887) Source fan-out and event duplication
From:       "Adam Alloul (JIRA)" <jira () apache ! org>
Date:       2016-02-23 9:13:18
Message-ID: JIRA.12941335.1456218784000.119551.1456218798147 () Atlassian ! JIRA
[Download RAW message or body]

Adam Alloul created FLUME-2887:
----------------------------------

             Summary: Source fan-out and event duplication
                 Key: FLUME-2887
                 URL: https://issues.apache.org/jira/browse/FLUME-2887
             Project: Flume
          Issue Type: Bug
          Components: Channel, Sinks+Sources
    Affects Versions: v1.5.0
         Environment: VMs using RedHat
            Reporter: Adam Alloul


Hi there, 

in my current work I have come to discover a behaviour of Flume that doesn't seem to \
be expected. Explanation:

1- Architecture (flume 1.5, not tested on 1.6)
  a- flume server = virtual machine running one (and one only) flume server
  b- First Flume server (receives 1 flow of data and transfers it using AVRO to \
another Flume Server.  c- the 2nd Flume replicates the source 4 times to 4 different \
sinks (the final sinks)  d- All flume servers use the file channel

2- Problem description:
  a- one of the final sinks fails
  b- the channel feeding it buffers and fills up
  c- the first flume then starts buffering until channel is full even though the 3 \
other final sinks are still receiving and working correctly  d- the channel is full \
on flume 1 and starts complaining.  e- when the faulty sink is back again, flume 1 \
flushes its events  f- this causes the events to be written multiple times into the 3 \
other non faulty final sinks in addition to the faulty one => very very high number \
of duplication (up to 24)


I believe that this is a bug and I did not see it reported by someone else. Please \
let me know if I am mistaken.

thank you,

Adam.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[prev in list] [next in list] [prev in thread] [next in thread]