[prev in list] [next in list] [prev in thread] [next in thread] 

List:       proftpd-committers
Subject:    [ProFTPD-committers] [Bug 2821] New: issues with LIST command when
From:       bugzilla-daemon () horde ! net
Date:       2006-07-29 1:17:10
Message-ID: bug-2821-5 () http ! bugs ! proftpd ! org/
[Download RAW message or body]

http://bugs.proftpd.org/show_bug.cgi?id=2821

           Summary: issues with LIST command when there are a  large number
                    of files and with using globing
           Product: ProFTPD
           Version: 1.2.10
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: mod_ls
        AssignedTo: proftp-committers@lists.sourceforge.net
        ReportedBy: z_gerg@yahoo.com


I've run into the following issues when sending a LIST request to
proftpd  

1) As the file list gets larger there is a noticeable long delay
   between when the last data block that has the last file name is
   sent and when the response 226 is sent.  

2) When using globing in the list command,  lists greater then 99999
   are not fully sent. Max number of files names in a list seems to be
   99998 (+/-). Under a rare case (that I can't seem to figure out how
   to reproduce reliably) the total number sent was 99998, but it had
   skipped over some files. Normally I see the list being sent in the
   order the files were created: completed_000000, completed_000001,
   completed_000003 ... completed_99997. But in this rare case I see
   completed_000000, completed_000001, completed_000002,
   completed_000005, completed_000006, ... completed_109999 where
   other files are skipped through out the list.  

My main testing was done using an ftp client that was written using
libCurl and is run on windows 2000 connecting to protftpd 1.2.10
running on suse enterprise 9 Linux 2.6.5-7.97 box. Since my main setup
still uses (and all my timing collection was done with) profptd
1.2.10, I'm reporting against that version. However I did try some
testing with protfd 1.3.0 and experienced similar issues. I also did
minimal testing with proftpd running on Red Hat 4 which also had
similar issues. 

To confirm it was not the ftp client I was using, I also tried using the
standard ftp client on windows 2000, standar ftp client on my suse box (running
locally), and standard ftp client running on remote linux box. All had the
same issues.  I also ran WireShark on the proftpd machine and looked
at the packets being sent to confirm that the issue was with
proftpd. Finally I repeated all the tests using all the mentioned
clients against vsftpd and saw no issues with vsftpd. 

What I'm doing is sending the following ftp request: 

 LIST <path_to_files>/completed_*

The file names in the dir I'm trying to get a listing for have the
following naming convention  

   completed_nnnnnn

Where nnnnnn is from 000000 to 109999 (just so I can easily tell which
files are being sent and the order they are being sent).

Here are timings I collected. Basically I generated large number of
files (using the script listed at the end of this report) and then
sent the list command (as described above). I'm running on a 100Mbit
LAN. I used WireShark to collect the timings. I started at 30,000
files, as the delay for 20,000 or less files was under 1 second. The
"total time" is from the time that the list command was received until
the final response 226 was sent. As you can, see most of the total
time is waiting for the response 226 to be sent.   

file    | full list | total    | delay from last data sent until
count   | sent      | time     | Response 226 transfer complete
--------+-----------+----------+---------------------------------
 30,000 | yes       | 0:05.843 | 0:02.375
--------+-----------+----------+---------------------------------
 50,000 | yes       | 0:29.790 | 0:19.720
--------+-----------+----------+---------------------------------
 70,000 | yes       | 1:21.122 | 1:02.525
--------+-----------+----------+---------------------------------
 90,000 | yes       | 2:22.048 | 1:57.444
--------+-----------+----------+---------------------------------
110,000 | *no       | 2:55.192 | 2:24.055
--------+-----------+----------+---------------------------------

* if I change the request to

 LIST <path_to_files>/

I.e. no globing, then the entire list is sent but still has the
delays. If NLST is used then no delays. But NLST still has the problem
when globing is used; not all the files are sent when list is > 99999.    


script to generate the file names (note: the max count and idx should
be 999999, but there is no check against larger numbers).  

#!/bin/sh
idx=0
zeros="000000"
if [ "$1" = "" ] ; then
  echo "usage: $0 count [idx]"
  echo "  count: number of file names to generate"
  echo "  idx (optional): index to start at"
  exit
fi
cnt=$1
if [ "$2" != "" ] ; then
  idx=$2
  cnt=$[$cnt + $idx]
fi
dt=`date +%s`
timeStamp=`date +%s%N`
while test $idx -lt $cnt ; do
  zCnt=$[${#zeros} - ${#idx}]
  touch "completed_${zeros:0:$zCnt}${idx}"
  idx=$[$idx + 1]
done


-- 
Configure bugmail: http://bugs.proftpd.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
ProFTPD Committers Mailing List
proftpd-committers@proftpd.org
https://lists.sourceforge.net/lists/listinfo/proftp-committers
[prev in list] [next in list] [prev in thread] [next in thread]