[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-dev
Subject:    [jira] [Created] (HADOOP-18321) Fix when to read an additional record from a BZip2 text file split
From:       "Ashutosh Gupta (Jira)" <jira () apache ! org>
Date:       2022-06-30 15:42:00
Message-ID: JIRA.13469492.1656603673000.2981.1656603720004 () Atlassian ! JIRA
[Download RAW message or body]

Ashutosh Gupta created HADOOP-18321:
---------------------------------------

             Summary: Fix when to read an additional record from a BZip2 text file \
split  Key: HADOOP-18321
                 URL: https://issues.apache.org/jira/browse/HADOOP-18321
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 3.3.3
            Reporter: Ashutosh Gupta
            Assignee: Ashutosh Gupta


Fix data correctness issue with TextInputFormat that can occur when reading BZip2 \
compressed text files. When triggered this bug would cause a split to return the \
first record of the succeeding split that reads the next BZip2 block, thereby \
duplicating that record.

*When the bug is triggered?*

The condition for the bug to occur requires that the flag "needAdditionalRecord" in \
CompressedSplitLineReader to be set to true by #fillBuffer at an inappropriate time: \
when we haven't read the remaining bytes of split. This can happen when the \
inDelimiter parameter is true while #fillBuffer is invoked while reading the next \
line. The inDelimiter parameter is true when either 1) the last byte of the buffer is \
a CR character ('\r') if using the default delimiters, or 2) the last bytes of the \
buffer are a common prefix of the delimiter if using a custom delimiter.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org


[prev in list] [next in list] [prev in thread] [next in thread]