[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kde-core-devel
Subject:    Re: KIO: Mass Copy of Files from Different Sources to Different
From:       David Faure <faure () kde ! org>
Date:       2009-08-20 15:04:22
Message-ID: 200908201704.22504.faure () kde ! org
[Download RAW message or body]

On Thursday 20 August 2009, Dawit A. wrote:
> "Direct scheduling" guarantees that if I make 4 requests one
> after the other, all those requests will immediately spawn a kioslave. 

Which is a dangerous guarantee, if you end up with "FTP server: too many
connections". So we need this to be configurable somehow (and I think
the somehow is "by the user" rather than "through the kio api", since apps
shouldn't have to care whether they're talking to often-limited-FTP or to
fish/sftp/smb/file/etc.).

> In the scheduled case, no matter what everything is queued and 
> processed serially one after the other.

Are you sure? That would be bad, we don't want that, we want "process
things at the pace of N slaves per protocol or host". But this might be
what maelcum started working on and that isn't finished, though.
In that case, we have to postpone this change indeed.

For kio_file I certainly want the behavior where N is quite big, so that
things happen in parallel -- but not too much, cf the initial post in this 
thread. N should be maybe 50, but not infinite.

> Well the "not waiting much" is a very important point here. And as an
> example I will give you a download manager that provides the user the
> option to set the number of connections to make per site. How would it be
> able to do that if the default is changed to "Scheduled" and "Direct" is
> completely gone ? 

By having that option being honoured by the scheduler ;-)

But this looks like the heart of our disagreement here; I'm thinking of a 
situation where the kio scheduler would implement that, and therefore -all- 
apps benefit from "N connections per site" limitations (which would fix a very 
old kde bug), rather than the user _having_ to use a download manager in order 
to get that feature.

> In addition to the fact that KIO::Scheduler::scheduleJob does not allow
> anything but a SimpleJob (dunno why that restriction) to be scheduled,

Because only a SimpleJob is talking to an actual kioslave.
A CopyJob is a high-level job which takes care of subjobs: one for doing a 
stat (file or directory?), one for doing a recursive listing of that 
directory, one job for copying each file, one mkdir job per directory to 
create, etc. That "high-level" job is totally unknown to the scheduler,
since the scheduler is about giving slaves to jobs ;) But the highlevel 
CopyJob doesn't need any slave for itself, it's just a qobject with timers and 
slots and subjobs.

> the function name itself is the source of confusion. It would have been 
> better if the function was named queueJobs or something along that line and
> allowed you to provide the maximum number of ioslaves that can be spawned
> to service any request per site/location.

As I said, I don't think that decision belongs to the application. It belongs 
to the KDE configuration (i.e. the user, with sensible defaults).
The fact that ftp.foo.org only accepts 1 connection per IP is not something 
that all apps should have to care about. Or the fact that the KIO scheduler 
shouldn't see 150 incoming jobs at the same time. We can take care of such 
problems inside KIO, without duplicating logic in the apps.

An app which -wants- to control this, like a download manager, should of 
course be able to tune settings in the kio scheduler, I'm not rejecting that 
possibilty of course, but this is only for special kinds of apps, not 
something all apps should have to do.

Oh and with my patch all kio jobs automatically get scheduled so scheduleJob() 
becomes completely internal, no need to call it from anywhere, and therefore 
its name doesn't really matter :-))

-- 
David Faure, faure@kde.org, sponsored by Qt Software @ Nokia to work on KDE,
Konqueror (http://www.konqueror.org), and KOffice (http://www.koffice.org).
[prev in list] [next in list] [prev in thread] [next in thread]