[prev in list] [next in list] [prev in thread] [next in thread] 

List:       linux-aio
Subject:    Re: The order of reaped io events
From:       Daniel Ehrenberg <dehrenberg () google ! com>
Date:       2014-07-22 17:08:52
Message-ID: CAAK6Zt3kP+=aCAxvtc0mxRB4DD3ZdsPnGKh_5OtGEY91pYEHdg () mail ! gmail ! com
[Download RAW message or body]

On Mon, Jul 21, 2014 at 5:37 PM, Weiming Shi <weimingshi@gatech.edu> wrote:
> Thank you Daniel. Good to know that you are the author of AIOUserGuide. Your
> guide really helped a lot for me to get started with libaio.

I'm happy to hear that that document helped you! By the way, all the
code there has been released into the public domain; I was a bit slow
in updating the doc to reflect that fact there, but feel free to copy
and paste it wherever you want. Any suggestions for the doc would be
appreciated.
>
> Just for understanding purpose, a follow-up question would be whether there
> is an efficient way to check which io requests have completed earlier than
> others.
>
> Right now, I see that there is a field called 'struct iocb *obj' in 'struct
> io_event' and a field called 'key' in 'structure iocb'. I think it is
> possible to maintain some data structure to keep track of which iocbs have
> been completed based on the 'key'. I suspect that linux kernel maintains
> such kind of data structure internally but it is not exposed to the users.

I'm not sure what the key is for--looks like the kernel sets it to 0
while the operation is in flight, and checks that it is 0 when
cancelling, so maybe this guards against some kinds of
cancel-after-free issues. About keeping track of completions that have
come: The pointer to the iocb helps you find the originally submitted
operation. You can use the data field to associate commands and
completions--I'd say typically the data field is a better way to
structure your code to find out about the submitted operation. You can
make it a pointer to something describing the operation, or you can
make it an index of an array containing these things if you want to be
super-paranoid.

You might be surprised by how ignorant the Linux kernel is in this
regard. An ioctx is a really thin thing--it's just a ringbuffer, not
associated with a file and definitely not associated with a backing
device. The backing devices (or block layer above them) are the ones
which have this logic which thinks about reordering. There's a lot of
code in the middle which just doesn't think about ordering, and the
ringbuffer code certainly doesn't. I like this design: it's a lot
easier to manage as a kernel programmer if everything isn't keeping
track of everything else 5 times, and things are kept simple and
modular. But that means you'll have to keep track of your completions
in userspace.
>
>
> On Mon, Jul 21, 2014 at 6:09 PM, Daniel Ehrenberg <dehrenberg@google.com>
> wrote:
>>
>> On Mon, Jul 21, 2014 at 3:24 PM, Weiming Shi
>> <weimingshi.gatech@gmail.com> wrote:
>> > Hi,
>> >
>> > I just started using libaio and I have a question about the order of the
>> > io
>> > completion events and the io requests.
>> >
>> > Is the order of the reaped io events from io_getevents the same as the
>> > order
>> > of iocbs passed to the io_submit given the same io context?
>>
>> No, the events will come back in whatever order they are completed by
>> the underlying device and kernel. This may be affected by many factors
>> such as scheduling in the kernel and on the device, as well as some
>> commands just taking longer than others, e.g., due to their size (if
>> able to proceed in parallel).
>>
>> Dan
>> >
>> > I haven't digged into the source code of linux aio implementation. I
>> > will
>> > appreciate it if someone can help me understand the behavior of libaio
>> > better.
>> >
>> > Thanks
>> >
>> > --
>> > To unsubscribe, send a message with 'unsubscribe linux-aio' in
>> > the body to majordomo@kvack.org.  For more info on Linux AIO,
>> > see: http://www.kvack.org/aio/
>> > Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
>
>
>
>
> --
> Weiming Shi
> PhD Student
> School of Electrical and Computer Engineering
> Georgia Institute of Technology
> Email: weimingshi@gatech.edu

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
[prev in list] [next in list] [prev in thread] [next in thread]