[prev in list] [next in list] [prev in thread] [next in thread]
List: libusb-devel
Subject: Re: [libusb] LIBUSB Read Speeds Differing from windows IWiaItemExtra->Escape()
From: Tom Brady <brady.tom315 () gmail ! com>
Date: 2016-11-21 17:15:50
Message-ID: CACf3k2vvijpyOiG3FJvt3PhsaPspO+BkfPJhY3GywKDc2ggmgA () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
As others have said, the other reason for having multiple outstanding
transfers is for streaming, so that you don't get a gap between the
completion of one transfer and the initiation of the next transfer. But
if you are seeing a single 20MB transfer takes 1/2 a second on WIndows
and 2 seconds on Linux, then the issue is not asynchronous transfers.
Are you QUITE sure that Windows isn't requesting a compressed format?
Hi Tim, thanks for your time on this. Yes, I'm certain that the data
coming over the wire is identical for the Linux & Windows scenario. It's
one raw image.
Best,
Tom
On Mon, Nov 21, 2016 at 11:31 AM Tom Brady <brady.tom315@gmail.com> wrote:
> Got it - thank you very much for your explanation.
>
> Based on your feedback, that leads me to believe there would be no benefi=
t
> to break up my large read into smaller reads, and therefore my
> implementation of the asynchronous API is correct.
>
> I suppose that leaves me at a loss then as to why the windows
> implementation is substantially faster. Any other ideas as to why this
> might be the case?
>
>
> On Mon, Nov 21, 2016 at 11:14 AM =D0=9B=D0=B0=D1=80=D0=B8=D0=BE=D0=BD=D0=
=BE=D0=B2 =D0=94=D0=B0=D0=BD=D0=B8=D0=B8=D0=BB <scumcoder@yandex.ru>
> wrote:
>
> > I will get more USB throughput by breaking this into several smaller
> transfers?
>
> No. If you queue a single huge LibUSB "read" request, LibUSB will keep
> submitting "read" URBs to the kernel until it either fills your 20 MiB
> buffer, or the camera sends a "short" packet. Until then, the USB
> controller will continue to spam the camera with "read" requests, and the
> camera will answer each such request with a packet. There is no way to
> speed up this process.
>
> You can lose bandwidth only if you *stream* packets. Then, if you don't
> have several pending LibUSB transfers, you will lose those precious
> milliseconds between the moment your request is fulfilled and the moment
> you re-submit it again.
>
> --
> =D0=94=D0=B0=D0=BD=D0=B8=D0=B8=D0=BB =D0=9B=D0=B0=D1=80=D0=B8=D0=BE=D0=BD=
=D0=BE=D0=B2
>
>
>
> 21.11.2016, 19:04, "Tom Brady" <brady.tom315@gmail.com>:
> > Thank you for the clarification.
> >
> > So just to confirm, if I want to do a single 20MB file transfer from th=
e
> camera to my computer, I will get more USB throughput by breaking this in=
to
> several smaller transfers?
> >
> > Best,
> > Tom
> >
> > On Mon, Nov 21, 2016 at 10:30 AM =D0=9B=D0=B0=D1=80=D0=B8=D0=BE=D0=BD=
=D0=BE=D0=B2 =D0=94=D0=B0=D0=BD=D0=B8=D0=B8=D0=BB <scumcoder@yandex.ru>
> wrote:
> >>> Why not handle chunking up larger packets in the libusb internals?
> >>
> >> LibUSB does just that while *sending* data. If you queue a huge "write=
"
> transfer, it is automatically serialized as a stream of packets, with the
> size of each packet being equal to the maximum size of the corresponding
> endpoint.
> >>
> >> With "read" inquires, there is nothing for LibUSB to "chunk up" becaus=
e
> it doesn't have any data. It's the device that has, and (of course) it
> *does* send the data as a stream of aforementioned small packets. LibUSB
> then concatenates these packets, puts this huge lump of data into the
> buffer that you supplied, and calls the callback that you specified. As
> soon as this happens, the kernel part of LibUSB stops submitting URBs to
> the kernel, and thus the USB controller stops spamming the device with
> "read" requests, *unless* there is another pending "read" transfer
> submitted to the LibUSB.
> >>
> >> Also you are advised to read this in order to choose right size of the
> "read" buffer that you submit to LibUSB:
> >> http://libusb.sourceforge.net/api-1.0/packetoverflow.html
> >>
> >> --
> >> =D0=94=D0=B0=D0=BD=D0=B8=D0=B8=D0=BB =D0=9B=D0=B0=D1=80=D0=B8=D0=BE=D0=
=BD=D0=BE=D0=B2
> >>
> >> 21.11.2016, 18:17, "Tom Brady" <brady.tom315@gmail.com>:
> >>> Thank you for your speedy reply. That's very interesting. So libusb
> does *not* create multiple asynchronous transfers from large ones? Why no=
t
> handle chunking up larger packets in the libusb internals?
> >>>
> >>> Thank you again for your help.
> >>>
> >>> Best,
> >>> Tom
> >>>
> >>> On Mon, Nov 21, 2016 at 10:00 AM =D0=9B=D0=B0=D1=80=D0=B8=D0=BE=D0=BD=
=D0=BE=D0=B2 =D0=94=D0=B0=D0=BD=D0=B8=D0=B8=D0=BB <scumcoder@yandex.ru>
> wrote:
> >>>>> In my asynchronous implementation, I set up and submitted a single
> 20MB transfer
> >>>>
> >>>> To squeeze the maximum available speed from USB, you absolutely
> _need_ to queue several asynchronous transfers, so that the USB controlle=
r
> doesn't stop spamming the camera with "read" inquiries while you deal wit=
h
> the latest received packet. As soon as you've parsed the packet,
> immediately resubmit the transfer, so that there is at least one pending
> "read" inquiry.
> >>>>
> >>>> --
> >>>> =D0=94=D0=B0=D0=BD=D0=B8=D0=B8=D0=BB =D0=9B=D0=B0=D1=80=D0=B8=D0=BE=
=D0=BD=D0=BE=D0=B2
> >>>>
> >>>> 21.11.2016, 17:47, "Tom Brady" <brady.tom315@gmail.com>:
> >>>>> Hi all,
> >>>>>
> >>>>> New to the list - many thanks for all the work you guys have done i=
n
> making this library!
> >>>>>
> >>>>> I am working with a Sony camera which implements a PTP over USB
> exchange protocol. Sony has provided a sample application which uses the
> windows usb api. For some reason, the sample application is able to trigg=
er
> and receive pictures about 2.5 times faster than my linux application whi=
ch
> uses libusb. (My application requires that it be installed on a linux
> machine, so using the windows API is out of the question). I've narrowed =
it
> down to the actual device read. I'm able to get about 10MB/s read speeds
> out of LibUSB, whereas the windows implementation manages nearly 40MB/s.
> The files I'm retrieving from the camera are approximately 20MB in size.
> >>>>>
> >>>>> I've used libusb's synchronous and asynchronous API.In my
> asynchronous implementation, I set up and submitted a single 20MB transfe=
r.
> Not sure if this is the right way to do it, as I haven't found definitive
> documentation anywhere telling me I should split it up versus sending one
> large transfer. My current understanding is that libusb handles the
> splitting of the transfer internally.
> >>>>>
> >>>>> Here are the links to two wireshark logs for both the windows &
> libusb implementation.
> >>>>>
> >>>>> LibUSB
> >>>>> Windows
> >>>>>
> >>>>> Would appreciate any direction you guys can offer!
> >>>>>
> >>>>> Best,
> >>>>> Tom
> >>>>>
> >>>>> ,
> >>>>>
> >>>>>
> -------------------------------------------------------------------------=
-----
> >>>>> ,
> >>>>>
> >>>>> _______________________________________________
> >>>>> libusb-devel mailing list
> >>>>> libusb-devel@lists.sourceforge.net
> >>>>> https://lists.sourceforge.net/lists/listinfo/libusb-devel
> >>> ,
> >>>
> >>>
> -------------------------------------------------------------------------=
-----
> >>> ,
> >>>
> >>> _______________________________________________
> >>> libusb-devel mailing list
> >>> libusb-devel@lists.sourceforge.net
> >>> https://lists.sourceforge.net/lists/listinfo/libusb-devel
> > ,
> >
> >
> -------------------------------------------------------------------------=
-----
> > ,
> >
> > _______________________________________________
> > libusb-devel mailing list
> > libusb-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/libusb-devel
>
>
[Attachment #5 (text/html)]
<div dir="ltr"><div class="inbox-inbox-uyb8Gf" style="color:rgb(33,33,33)"><div><div \
class="inbox-inbox-pv" style="border-left:1px solid \
rgb(224,224,224);color:rgb(117,117,117);padding:10px"><div dir="ltr" \
class="gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote \
gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex">As others have said, the other reason for having \
multiple outstanding<br class="gmail_msg">transfers is for streaming, so that you \
don't get a gap between the<br class="gmail_msg">completion of one transfer and \
the initiation of the next transfer. But<br class="gmail_msg">if you are seeing a \
single 20MB transfer takes 1/2 a second on WIndows<br class="gmail_msg">and 2 seconds \
on Linux, then the issue is not asynchronous transfers.<br class="gmail_msg">Are you \
QUITE sure that Windows isn't requesting a compressed format?<br \
class="gmail_msg"></blockquote><div class="gmail_msg"><br \
class="gmail_msg"></div></div></div></div></div></div><div class="inbox-inbox-uyb8Gf" \
style="color:rgb(33,33,33)"><div><div class="inbox-inbox-F3hlO"><div dir="ltr" \
class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> Hi Tim, \
thanks for your time on this. Yes, I'm certain that the data coming over the \
wire is identical for the Linux & Windows scenario. It's one raw image. \
</div><div class="gmail_msg"><br class="gmail_msg"></div><div \
class="gmail_msg">Best,</div><div \
class="gmail_msg">Tom</div></div></div></div></div></div></div><br><div \
class="gmail_quote"><div dir="ltr">On Mon, Nov 21, 2016 at 11:31 AM Tom Brady <<a \
href="mailto:brady.tom315@gmail.com">brady.tom315@gmail.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" \
class="gmail_msg">Got it - thank you very much for your explanation.<div \
class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Based on your \
feedback, that leads me to believe there would be no benefit to break up my large \
read into smaller reads, and therefore my implementation of the asynchronous API is \
correct. </div><div class="gmail_msg"><div class="gmail_msg"><br \
class="gmail_msg"></div><div class="gmail_msg">I suppose that leaves me at a loss \
then as to why the windows implementation is substantially faster. Any other ideas \
as to why this might be the case?<br class="gmail_msg"><br \
class="gmail_msg"></div></div></div><br class="gmail_msg"><div class="gmail_quote \
gmail_msg"><div dir="ltr" class="gmail_msg">On Mon, Nov 21, 2016 at 11:14 AM \
Ларионов Даниил <<a href="mailto:scumcoder@yandex.ru" \
class="gmail_msg" target="_blank">scumcoder@yandex.ru</a>> wrote:<br \
class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex">> I will get more USB throughput \
by breaking this into several smaller transfers?<br class="gmail_msg"> <br \
class="gmail_msg"> No. If you queue a single huge LibUSB "read" request, \
LibUSB will keep submitting "read" URBs to the kernel until it either fills \
your 20 MiB buffer, or the camera sends a "short" packet. Until then, the \
USB controller will continue to spam the camera with "read" requests, and \
the camera will answer each such request with a packet. There is no way to speed up \
this process.<br class="gmail_msg"> <br class="gmail_msg">
You can lose bandwidth only if you *stream* packets. Then, if you don't have \
several pending LibUSB transfers, you will lose those precious milliseconds between \
the moment your request is fulfilled and the moment you re-submit it again.<br \
class="gmail_msg"> <br class="gmail_msg">
-- <br class="gmail_msg">
Даниил Ларионов<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
21.11.2016, 19:04, "Tom Brady" <<a href="mailto:brady.tom315@gmail.com" \
class="gmail_msg" target="_blank">brady.tom315@gmail.com</a>>:<br \
class="gmail_msg"> > Thank you for the clarification.<br class="gmail_msg">
><br class="gmail_msg">
> So just to confirm, if I want to do a single 20MB file transfer from the camera \
to my computer, I will get more USB throughput by breaking this into several smaller \
transfers?<br class="gmail_msg"> ><br class="gmail_msg">
> Best,<br class="gmail_msg">
> Tom<br class="gmail_msg">
><br class="gmail_msg">
> On Mon, Nov 21, 2016 at 10:30 AM Ларионов Даниил <<a \
href="mailto:scumcoder@yandex.ru" class="gmail_msg" \
target="_blank">scumcoder@yandex.ru</a>> wrote:<br class="gmail_msg"> >>> \
Why not handle chunking up larger packets in the libusb internals?<br \
class="gmail_msg"> >><br class="gmail_msg">
>> LibUSB does just that while *sending* data. If you queue a huge \
"write" transfer, it is automatically serialized as a stream of packets, \
with the size of each packet being equal to the maximum size of the corresponding \
endpoint.<br class="gmail_msg"> >><br class="gmail_msg">
>> With "read" inquires, there is nothing for LibUSB to "chunk \
up" because it doesn't have any data. It's the device that has, and (of \
course) it *does* send the data as a stream of aforementioned small packets. LibUSB \
then concatenates these packets, puts this huge lump of data into the buffer that you \
supplied, and calls the callback that you specified. As soon as this happens, the \
kernel part of LibUSB stops submitting URBs to the kernel, and thus the USB \
controller stops spamming the device with "read" requests, *unless* there \
is another pending "read" transfer submitted to the LibUSB.<br \
class="gmail_msg"> >><br class="gmail_msg">
>> Also you are advised to read this in order to choose right size of the \
"read" buffer that you submit to LibUSB:<br class="gmail_msg"> >> <a \
href="http://libusb.sourceforge.net/api-1.0/packetoverflow.html" rel="noreferrer" \
class="gmail_msg" target="_blank">http://libusb.sourceforge.net/api-1.0/packetoverflow.html</a><br \
class="gmail_msg"> >><br class="gmail_msg">
>> --<br class="gmail_msg">
>> Даниил Ларионов<br class="gmail_msg">
>><br class="gmail_msg">
>> 21.11.2016, 18:17, "Tom Brady" <<a \
href="mailto:brady.tom315@gmail.com" class="gmail_msg" \
target="_blank">brady.tom315@gmail.com</a>>:<br class="gmail_msg"> >>> \
Thank you for your speedy reply. That's very interesting. So libusb does \
*not* create multiple asynchronous transfers from large ones? Why not handle chunking \
up larger packets in the libusb internals?<br class="gmail_msg"> >>><br \
class="gmail_msg"> >>> Thank you again for your help.<br class="gmail_msg">
>>><br class="gmail_msg">
>>> Best,<br class="gmail_msg">
>>> Tom<br class="gmail_msg">
>>><br class="gmail_msg">
>>> On Mon, Nov 21, 2016 at 10:00 AM Ларионов Даниил <<a \
href="mailto:scumcoder@yandex.ru" class="gmail_msg" \
target="_blank">scumcoder@yandex.ru</a>> wrote:<br class="gmail_msg"> \
>>>>> In my asynchronous implementation, I set up and submitted a \
single 20MB transfer<br class="gmail_msg"> >>>><br class="gmail_msg">
>>>> To squeeze the maximum available speed from USB, you absolutely \
_need_ to queue several asynchronous transfers, so that the USB controller \
doesn't stop spamming the camera with "read" inquiries while you deal \
with the latest received packet. As soon as you've parsed the packet, immediately \
resubmit the transfer, so that there is at least one pending "read" \
inquiry.<br class="gmail_msg"> >>>><br class="gmail_msg">
>>>> --<br class="gmail_msg">
>>>> Даниил Ларионов<br class="gmail_msg">
>>>><br class="gmail_msg">
>>>> 21.11.2016, 17:47, "Tom Brady" <<a \
href="mailto:brady.tom315@gmail.com" class="gmail_msg" \
target="_blank">brady.tom315@gmail.com</a>>:<br class="gmail_msg"> \
>>>>> Hi all,<br class="gmail_msg"> >>>>><br \
class="gmail_msg"> >>>>> New to the list - many thanks for all the \
work you guys have done in making this library!<br class="gmail_msg"> \
>>>>><br class="gmail_msg"> >>>>> I am working with a \
Sony camera which implements a PTP over USB exchange protocol. Sony has provided a \
sample application which uses the windows usb api. For some reason, the sample \
application is able to trigger and receive pictures about 2.5 times faster than my \
linux application which uses libusb. (My application requires that it be installed on \
a linux machine, so using the windows API is out of the question). I've narrowed \
it down to the actual device read. I'm able to get about 10MB/s read speeds out \
of LibUSB, whereas the windows implementation manages nearly 40MB/s. The files \
I'm retrieving from the camera are approximately 20MB in size.<br \
class="gmail_msg"> >>>>><br class="gmail_msg">
>>>>> I've used libusb's synchronous and asynchronous API.In \
my asynchronous implementation, I set up and submitted a single 20MB transfer. Not \
sure if this is the right way to do it, as I haven't found definitive \
documentation anywhere telling me I should split it up versus sending one large \
transfer. My current understanding is that libusb handles the splitting of the \
transfer internally.<br class="gmail_msg"> >>>>><br class="gmail_msg">
>>>>> Here are the links to two wireshark logs for both the windows \
& libusb implementation.<br class="gmail_msg"> >>>>><br \
class="gmail_msg"> >>>>> LibUSB<br class="gmail_msg">
>>>>> Windows<br class="gmail_msg">
>>>>><br class="gmail_msg">
>>>>> Would appreciate any direction you guys can offer!<br \
class="gmail_msg"> >>>>><br class="gmail_msg">
>>>>> Best,<br class="gmail_msg">
>>>>> Tom<br class="gmail_msg">
>>>>><br class="gmail_msg">
>>>>> ,<br class="gmail_msg">
>>>>><br class="gmail_msg">
>>>>> \
------------------------------------------------------------------------------<br \
class="gmail_msg"> >>>>> ,<br class="gmail_msg">
>>>>><br class="gmail_msg">
>>>>> _______________________________________________<br \
class="gmail_msg"> >>>>> libusb-devel mailing list<br \
class="gmail_msg"> >>>>> <a \
href="mailto:libusb-devel@lists.sourceforge.net" class="gmail_msg" \
target="_blank">libusb-devel@lists.sourceforge.net</a><br class="gmail_msg"> \
>>>>> <a \
href="https://lists.sourceforge.net/lists/listinfo/libusb-devel" rel="noreferrer" \
class="gmail_msg" target="_blank">https://lists.sourceforge.net/lists/listinfo/libusb-devel</a><br \
class="gmail_msg"> >>> ,<br class="gmail_msg">
>>><br class="gmail_msg">
>>> ------------------------------------------------------------------------------<br \
class="gmail_msg"> >>> ,<br class="gmail_msg">
>>><br class="gmail_msg">
>>> _______________________________________________<br class="gmail_msg">
>>> libusb-devel mailing list<br class="gmail_msg">
>>> <a href="mailto:libusb-devel@lists.sourceforge.net" class="gmail_msg" \
target="_blank">libusb-devel@lists.sourceforge.net</a><br class="gmail_msg"> \
>>> <a href="https://lists.sourceforge.net/lists/listinfo/libusb-devel" \
rel="noreferrer" class="gmail_msg" \
target="_blank">https://lists.sourceforge.net/lists/listinfo/libusb-devel</a><br \
class="gmail_msg"> > ,<br class="gmail_msg">
><br class="gmail_msg">
> ------------------------------------------------------------------------------<br \
class="gmail_msg"> > ,<br class="gmail_msg">
><br class="gmail_msg">
> _______________________________________________<br class="gmail_msg">
> libusb-devel mailing list<br class="gmail_msg">
> <a href="mailto:libusb-devel@lists.sourceforge.net" class="gmail_msg" \
target="_blank">libusb-devel@lists.sourceforge.net</a><br class="gmail_msg"> > <a \
href="https://lists.sourceforge.net/lists/listinfo/libusb-devel" rel="noreferrer" \
class="gmail_msg" target="_blank">https://lists.sourceforge.net/lists/listinfo/libusb-devel</a><br \
class="gmail_msg"> </blockquote></div></blockquote></div>
------------------------------------------------------------------------------
_______________________________________________
libusb-devel mailing list
libusb-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libusb-devel
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic