[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-dev
Subject:    [jira] [Resolved] (HADOOP-17015) ABFS: Make PUT and POST operations idempotent
From:       "Thomas Marqardt (Jira)" <jira () apache ! org>
Date:       2020-06-24 16:17:00
Message-ID: JIRA.13301153.1587959687000.327064.1593015420491 () Atlassian ! JIRA
[Download RAW message or body]


     [ https://issues.apache.org/jira/browse/HADOOP-17015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel \
]

Thomas Marqardt resolved HADOOP-17015.
--------------------------------------
    Resolution: Fixed

Sneha and I discussed this.   The common Hadoop scenario is a case where you have one \
or more tasks, each operating on different source files, all attempting to rename to \
a common destination. In this scenario, the fix in PR 2021 is correct.   There are \
scenarios where PR 2021 will lead to incorrect results, but they seem to be very \
contrived and unlikely in Hadoop.   A work item will be opened to investigate the \
need to improve this on the server-side, for example by allowing an operation-id to \
be passed to the rename operation and persisted in the destination metadata, but for \
now we have this fix to the driver on the client-side.  

> ABFS: Make PUT and POST operations idempotent
> ---------------------------------------------
> 
> Key: HADOOP-17015
> URL: https://issues.apache.org/jira/browse/HADOOP-17015
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.2.1
> Reporter: Sneha Vijayarajan
> Assignee: Sneha Vijayarajan
> Priority: Major
> Fix For: 3.4.0
> 
> 
> Currently when a PUT or POST operation timeouts and the server has already \
> successfully executed the operation, there is no check in driver to see if the \
> operation did succeed or not and just retries the same operation again. This can \
> cause driver to through invalid user errors. 
> Sample scenario:
> # Rename request times out. Though server has successfully executed the operation.
> # Driver retries rename and get source not found error.
> In the scenario, driver needs to check if rename is being retried and success if \
> source if not found, but destination is present. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org


[prev in list] [next in list] [prev in thread] [next in thread]