Slow FTP Dir Listing

I am having problems getting the directory listing if there are a lot of files in a folder.

By accessing files located in a completely different folder, I can verify that access is working properly (the user name and password are correct and there are no firewall issues, etc). So, the software does work as needed.

Back to the problem folder... the folder in question contains a lot of files (several thousand), however, all I get is the "refresh" indicators. I have upped the timeout to over 15 minutes and I still do not get the listing (remains in refresh mode). On the same site, on a different computer, using the same credentials, I get the dir listing in about 60 seconds.

I actually bought your software because of its ability to work with a large number of files. I am hopeful that you can provide some direction for me.

Thanks!

RT

If it works on one computer but not on the other it might be a network problem.

Two reasons why I don't think that is the case:
1.) I can access other folders (with less files) on the same FTP site just fine from the problematic PC.
2.) The test PC that worked ok is on the same network segment, behind the same firewall.

I also did a quick speed test on the PC with the problems... it coped a 750k file in a second or so from the same site, but a folder with less files in it.

RT

Any thought on the cause of the issue?

Ronald Thies

Please post the log from the remote browser of both computers here. Also include the system information which you can find in the menu: Help->About "System Information" in SmartFTP.

Logs from Good PC

[22:19:50] SmartFTP v4.0.1140.0
[22:19:57] Resolving host name "ftp.wininspectionreport.com"
[22:19:57] Connecting to 67.199.121.130 Port: 21
[22:19:57] Connected to ftp.wininspectionreport.com.
[22:19:57] 220-FileZilla Server version 0.9.30 beta
[22:19:57] 220 AbsoluteESolutions
[22:19:57] USER win
[22:19:57] 331 Password required for win
[22:19:57] PASS (hidden)
[22:19:57] 230 Logged on
[22:19:57] SYST
[22:19:57] 215 UNIX emulated by FileZilla
[22:19:57] Detected Server Type: UNIX
[22:19:57] RTT: 64.332 ms
[22:19:57] FEAT
[22:19:57] 211-Features:
[22:19:57] MDTM
[22:19:57] REST STREAM
[22:19:57] SIZE
[22:19:57] MLST type*;size*;modify*;
[22:19:57] MLSD
[22:19:57] UTF8
[22:19:57] CLNT
[22:19:57] MFMT
[22:19:57] 211 End
[22:19:57] CLNT SmartFTP 4.0.1140.0
[22:19:57] 200 Don't care
[22:19:57] OPTS UTF8 ON
[22:19:57] 200 UTF8 mode enabled
[22:19:57] Detected Server Software: FileZilla Server
[22:19:57] PWD
[22:19:57] 257 "/" is current directory.
[22:19:57] CWD /docs/12-2010
[22:19:57] 250 CWD successful. "/docs/12-2010" is current directory.
[22:19:57] PWD
[22:19:57] 257 "/docs/12-2010" is current directory.
[22:19:57] TYPE A
[22:19:57] 200 Type set to A
[22:19:57] PASV
[22:19:57] 227 Entering Passive Mode (67,199,120,130,19,138)
[22:19:57] Passive ip address returned from server different from server ip.
[22:19:57] Opening data connection to 67.199.120.130 Port: 5002
[22:19:57] MLSD
[22:19:58] 150 Connection accepted
[22:20:10] 226 Transfer OK
[22:21:13] Transfer Timeout (60s). Closing data connection.
[22:21:13] 186880 bytes transferred. (2.40 KB/s) (00:01:15)
[22:21:44] NOOP
[22:21:44] Server closed connection

Logs from Bad PC

[22:32:32] SmartFTP v4.0.1145.0
[22:32:34] Resolving host name "ftp.wininspectionreport.com"
[22:32:34] Connecting to 67.199.121.130 Port: 21
[22:32:34] Connected to ftp.wininspectionreport.com.
[22:32:35] 220-FileZilla Server version 0.9.30 beta
[22:32:35] 220 AbsoluteESolutions
[22:32:35] USER win
[22:32:35] 331 Password required for win
[22:32:35] PASS (hidden)
[22:32:35] 230 Logged on
[22:32:35] SYST
[22:32:35] 215 UNIX emulated by FileZilla
[22:32:35] Detected Server Type: UNIX
[22:32:35] RTT: 98.656 ms
[22:32:35] FEAT
[22:32:35] 211-Features:
[22:32:36] MDTM
[22:32:36] REST STREAM
[22:32:36] SIZE
[22:32:36] MLST type*;size*;modify*;
[22:32:36] MLSD
[22:32:36] UTF8
[22:32:36] CLNT
[22:32:36] MFMT
[22:32:36] 211 End
[22:32:36] CLNT SmartFTP 4.0.1145.0
[22:32:36] 200 Don't care
[22:32:36] OPTS UTF8 ON
[22:32:36] 200 UTF8 mode enabled
[22:32:36] Detected Server Software: FileZilla Server
[22:32:36] PWD
[22:32:36] 257 "/" is current directory.
[22:32:36] CWD /docs/11-2010
[22:32:36] 250 CWD successful. "/docs/11-2010" is current directory.
[22:32:36] PWD
[22:32:36] 257 "/docs/11-2010" is current directory.
[22:32:36] TYPE A
[22:32:36] 200 Type set to A
[22:32:36] PASV
[22:32:36] 227 Entering Passive Mode (67,199,120,130,19,138)
[22:32:36] Passive ip address returned from server different from server ip.
[22:32:36] Opening data connection to 67.199.120.130 Port: 5002
[22:32:36] MLSD
[22:32:37] 150 Connection accepted


System Info (bad computer)

+- System -----------------------------
Microsoft Windows 7
(Build 7600)

CPU Speed : 3103 MHz
Total Memory : 6142 MB
Free Memory : 2758 MB

+- SmartFTP ---------------------------
Version : 4.0.1145.0
Time Stamp : 2010-10-28 00:28:15
Platform : x64
Id : 400129464
Maintenance : 2011-08-31
Days in use : 284

+- Language ---------------------------
en-US

+- Internet Explorer ------------------
Version : 8.0.7600.16385

+- Winsock ----------------------------
Winsock : 2.2

Install the latest version of SmartFTP on both computers. Then try it again and post the complete logs again (the 2nd log was not incomplete).

I updated to SmartFTP 4.0.1156.0 (64-bit) on the bad PC. 30 minutes later I still do not have a dir listing.

I updated to SmartFTP 4.0.1156.0 (64-bit) on the bad PC. 30 minutes later I still do not have a dir listing.

The log was incomplete because I didn't wait 2 hours for the software to throw the error. I figured an hour was enough time to get a dir listing

[22:08:28] SmartFTP v4.0.1156.0
[22:08:28] Resolving host name "ftp.wininspectionreport.com"
[22:08:29] Connecting to 67.199.121.130 Port: 21
[22:08:29] Connected to ftp.wininspectionreport.com.
[22:08:29] 220-FileZilla Server version 0.9.30 beta
[22:08:29] 220 AbsoluteESolutions
[22:08:29] USER win
[22:08:29] 331 Password required for win
[22:08:29] PASS (hidden)
[22:08:30] 230 Logged on
[22:08:30] SYST
[22:08:30] 215 UNIX emulated by FileZilla
[22:08:30] Detected Server Type: UNIX
[22:08:30] RTT: 59.849 ms
[22:08:30] FEAT
[22:08:30] 211-Features:
[22:08:31] MDTM
[22:08:31] REST STREAM
[22:08:31] SIZE
[22:08:31] MLST type*;size*;modify*;
[22:08:31] MLSD
[22:08:31] UTF8
[22:08:31] CLNT
[22:08:31] MFMT
[22:08:31] 211 End
[22:08:31] CLNT SmartFTP 4.0.1156.0
[22:08:31] 200 Don't care
[22:08:31] OPTS UTF8 ON
[22:08:31] 200 UTF8 mode enabled
[22:08:31] Detected Server Software: FileZilla Server
[22:08:31] PWD
[22:08:31] 257 "/" is current directory.
[22:08:31] CWD /docs/11-2010
[22:08:31] 250 CWD successful. "/docs/11-2010" is current directory.
[22:08:31] PWD
[22:08:31] 257 "/docs/11-2010" is current directory.
[22:08:31] TYPE A
[22:08:31] 200 Type set to A
[22:08:31] PASV
[22:08:31] 227 Entering Passive Mode (67,199,120,130,19,138)
[22:08:31] Passive IP address returned from server different from server IP.
[22:08:31] Opening data connection to 67.199.120.130 Port: 5002
[22:08:31] MLSD
[22:08:31] 150 Connection accepted
[23:08:34] Transfer Timeout (3600s). Closing data connection.
[23:08:34] 16060 bytes transferred. (4 bytes/s) (01:00:02)
[00:08:34] Timeout (3600s).
[00:08:34] Active Help: https://www.smartftp.com/support/kb/74
[00:08:34] Client closed the connection.

Your timeout is way too high. Set it to 120 seconds.

The problem is that SmartFTP never receives the close event. This is most likely caused by a buggy router, software firewall, antivirus product etc. My recommendation is to reinstall the operating system (Windows 7). Then try it before you install any 3rd party products.

It was on 60 seconds and it didn't work either. The PC is less than 2 months old. It sux if I need to have a dedicated PC for your software and another for all other software. Keep in mind, the software works great if there are less files on the server side. How could other software on my PC cause the software not to work if certain requirements are met on the server side?

Your timeout is way too high. Set it to 120 seconds.

The problem is that SmartFTP never receives the close event. This is most likely caused by a buggy router, software firewall, antivirus product etc. My recommendation is to reinstall the operating system (Windows 7). Then try it before you install any 3rd party products.

What would cause SmartFTP not to receive the close event? Is a close event needed to get a dir listing?

Let's review your other comments:
* The router being buggy makes no sense. It consistently works on one PC and not the other. On the PC I am having problems with it is fine if there are only a few files in the directory. A router doesn't care about about that and operates at a different level in the OSI reference model - right?
* Software firewall doesn't make sense to me, either. I tested it by disabling and the problem still exists
* Anti virus has been disabled to rule that out as well. Anyway, would AV software care how many files are in a dir listing and would be ok with fewer files on a remote site? My PC resources are FAR from being taxed getting the dir

The PC is pretty new, less than a month and SmartFTP never worked on this PC. I can't reinstall just to test the theory. Half of the items you mentioned as possible root causes make no sense so I feel like this is just a way to pass the buck to me and close the ticket.

Do other WIN 7 64 bit machines work fine with thousands of files?

I also opened this issue in the public forums to see if others can help. It was closed as a duplicate wit this. Can I not get public support since you had closed this ticket?

I also have access to the serve it is connecting to. I am using x. Is there anything in the server logs that could help in troubleshooting?

Ok, I think I may have found something. Don't know what to do about it though...

In the logs, if the directory listing is long, it bombs. If it is short, it doesn't bomb. Here are snippets from the log files for failures:

[08:55:20] 16060 bytes transferred. (87 bytes/s) (00:03:02)
[09:00:05] 16060 bytes transferred. (87 bytes/s) (00:03:02)
[09:07:08] 16060 bytes transferred. (88 bytes/s) (00:03:02)

They all fail at exactly 16060 bytes. If less, it is ok. What in the software would restrict it to 16060? Is that a hard coded limitation?

Ron T

Please install the latest version of SmartFTP. But if it works on one system and not on the other, it's a strong indication that the problem might be outside of SmartFTP's control.

Please install the latest version of SmartFTP. But if it works on one system and not on the other, it's a strong indication that the problem might be outside of SmartFTP's control.

I installed the latest version this morning. Let me make this clear - SmartFTP works on the computer giving me problems. It connects to the FTP site, it downloads files, it does everything. However, If the dir listing is greater than 16k, the SmartFTP software stops. My computer, all other applications, my firewall, everything, allows apps to download more than 16k in every other application I own. Moreover, SmartFTP works fine on documents greater than 16k. The problem is ONLY with SmartFTP, where dirs are great than 16k. It works in all other cases. I don't understand how that would be my PC and why my PC doesn't like your software in only certain cases. Please explain how come this is a "strong indication that the problem might be outside of SmartFTP's control". It is the only application having problems, performing one function, meeting a very specific criteria.

16k in file size or in number of files in the list (Where dir is much larger than 16k in size...which is not that much data to send)? I often have the same thing with transferring large files where they are several MB like 30-100MB in size. I have another issue in the bug section here that I believe is the cause of it too. My FTP Server disconnects me after 15min of no activity (NOOP is ignored as activity too).

In FTP you have a command channel/port and a data channel/port. Internally it uses the command channel to tell the server to send/receive, then the data channel for the transfer, and when finished sends the status back on the command channel. If my transfers take over the 15min to complete I think my command channel is getting disconnected, so there is no way to get the status back. The data channel still sends the data though, so in my queue I see the file transfer go all the way up to 100% then sit there for a while and restart because it thinks the transfer failed. A Dir/Folder listing is transferred the same way as a file, so if your host does the same thing and your listing takes too long it may be happening to you.

Unfortunately, even if/when SmartFTP is fixed to reconnect better for my other error it would not fix this problem since reconnecting would be a new connection and not the one sending the transfer status reply. In the mean time I have modified my Keep Alive settings for my connection to send a PWD command (There isn't much documentation in that tab, but from a RAW FTP command list that won't mess up SmartFTP- PWD, LIST(NLST should be an option too but returns no data connection), STAT, and NOOP are pretty much your only choices...and since your folders are large LIST would not be a good choice). PWD just prints the current folder and seems to be seen as activity better although my connections still disconnect after a while...most likely because the queue connections don't appear to send Keep Alive. Since the listings are usually the remote browser's connection this may work for you though.

Roger: Last warning, please stop taking over other people's thread that are unrelated to your case.

Ronald Thies: Please send a test account to access the exactly same directory on your server to support at smartftp.com and we will try to reproduce the problem.

Can we ease up on the defensive posts about taking over threads. I offered sound advice. And, unlike you who only posts to delete settings out of the registry or blame firewalls I actually looked at his logs and saw things like:

[22:21:13] Transfer Timeout (60s). Closing data connection.
[22:21:13] 186880 bytes transferred. (2.40 KB/s) (00:01:15)
[22:21:44] NOOP
[22:21:44] Server closed connection

Even after NOOPs his server is closing his connection, so something is idle too long. He can try out the Keep Alive options I mentioned. There is also the option of reusing Queue connections that can be turned off so new connections are made that won't time out at the expense of reconnecting each time taking a while. Like I said, even fixing the issue client side won't fix the problem. It is server side and he has to do something to make the server not disconnect him.

You also suggested to send a test account to you for this issue which you never did on mine. You've been on my case from the beginning and I only try to help.

Ronald Thies: Please send a test account to access the exactly same directory on your server to support at smartftp.com and we will try to reproduce the problem.

Where can I send the account info? Do you have an email address I can send it to?

email to: support at smartftp.com

email to: support at smartftp.com

I just sent the info. Let me know if you have any questions.