This is taking forevvvvvvver....

SamOldman

2014-06-28 21:49:17

First up - I like SmartFTP. I really do... but it does some things that really really drive me crazy.

I'm trying to upload somewhere's in the neighborhood of 15,000 .jpg files, all of which are around 40KB in size.

Unfortunately SmartFTP insists on using the MLST feature to list the specs of each file - ostensibly for File Exist checking. The problem is that I cannot disable the MLST announcement, and SmartFTP seems incapable of ignoring it. All of this means that I have to sit here for a good 30 minutes or so watching this idiotic system send 15,000 MLST commands to my server before uploading any files.

So for refresher :
• I cannot disable the MLST broadcast from the feature list.
• I cannot stop SmartFTP from sending an MLST for each file before uploading.

I'd REALLY appreciate it if someone could tell me how to upload WITHOUT SmartFTP trying to send the MLST command.

SamOldman

2014-07-01 21:30:47

Just had to sit here for 1 hour and 21 minutes while SmartFTP sent MLST requests, trying to upload files.

2014-07-02 00:20:37

We are looking into the problem.

2014-07-02 12:21:47

Can you please post the log from 1-2 file transfers?

SamOldman

2014-07-02 15:30:08

Sure thing. I had a small upload (only around 6,000 files) this morning so I might as well post that log. This took around 8 minutes *just to process* before beginning the upload. Some days I have 100,000 files to upload. Those days suck.

Anyway - here's the log, which I have censored to hide personal information :

[11:12:38] SmartFTP v6.0.2023.0
[11:12:38] 1>Resolving host name "XXX.XXX.XXX.XXX"
[11:12:38] 1>Connecting to XXX.XXX.XXX.XXX Port: 21
[11:12:38] 1>Connected to XXX.XXX.XXX.XXX.
[11:12:38] 1>220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
[11:12:38] 1>220-You are user number 21 of 500 allowed.
[11:12:38] 1>220-Local time is now 08:14. Server port: 21.
[11:12:38] 1>220-This is a private system - No anonymous login
[11:12:38] 1>220 You will be disconnected after 3 minutes of inactivity.
[11:12:38] 1>USER XXXXXXXXXXXX
[11:12:38] 1>331 User XXXXXXXXXXXX OK. Password required
[11:12:38] 1>PASS (hidden)
[11:12:38] 1>230 OK. Current restricted directory is /
[11:12:38] 1>SYST
[11:12:38] 1>215 UNIX Type: L8
[11:12:38] 1>Detected Server Type: UNIX
[11:12:38] 1>RTT: 78.559 ms
[11:12:38] 1>FEAT
[11:12:39] 1>211-Extensions supported:
[11:12:39] 1> EPRT
[11:12:39] 1> IDLE
[11:12:39] 1> MDTM
[11:12:39] 1> SIZE
[11:12:39] 1> MFMT
[11:12:39] 1> REST STREAM
[11:12:39] 1> MLST type*;size*;sizd*;modify*;UNIX.mode*;UNIX.uid*;UNIX.gid*;unique*;
[11:12:39] 1> MLSD
[11:12:39] 1> AUTH TLS
[11:12:39] 1> PBSZ
[11:12:39] 1> PROT
[11:12:39] 1> UTF8
[11:12:39] 1> TVFS
[11:12:39] 1> ESTA
[11:12:39] 1> PASV
[11:12:39] 1> EPSV
[11:12:39] 1> SPSV
[11:12:39] 1> ESTP
[11:12:39] 1>211 End.
[11:12:39] 1>OPTS UTF8 ON
[11:12:39] 1>200 OK, UTF-8 enabled
[11:12:39] 1>PWD
[11:12:39] 1>257 "/" is your current location
[11:12:39] 1>CWD /XXXXXXXX/XXXXXXXX/test
[11:12:39] 1>250 OK. Current directory is /XXXXXXXX/XXXXXXXX/test
[11:12:39] 1>PWD
[11:12:39] 1>257 "/XXXXXXXX/XXXXXXXX/test" is your current location
[11:12:39] 1>TYPE A
[11:12:39] 1>200 TYPE is now ASCII
[11:12:39] 1>PASV
[11:12:39] 1>227 Entering Passive Mode (XXX,XXX,XXX,XXX,195,190)
[11:12:39] 1>Opening data connection to XXX.XXX.XXX.XXX Port: 50110
[11:12:39] 1>LIST -aL
[11:12:39] 1>150 Accepted data connection
[11:12:39] 1>226-Options: -a -l
[11:12:39] 1>226 2 matches total
[11:12:39] 1>131 bytes transferred. (1.45 KB/s) (88 ms)
[11:13:08] 1>MLST wmdin1b-1k.jpg
[11:13:08] 1>550 Can't check for file existence
[11:13:08] 1>MLST wmdin1b-3r.jpg
[11:13:08] 1>550 Can't check for file existence
[11:13:08] 1>MLST wmdin1b-4rb.jpg
[11:13:08] 1>550 Can't check for file existence
[11:13:08] 1>MLST wmdin1b-5f.jpg
[11:13:08] 1>550 Can't check for file existence
[11:13:08] 1>MLST wmdin1b-14n.jpg
[11:13:08] 1>550 Can't check for file existence
[11:13:08] 1>MLST wmdin1h-1k.jpg
[11:13:09] 1>550 Can't check for file existence
[11:13:09] 1>MLST wmdin1h-3r.jpg
[11:13:09] 1>550 Can't check for file existence
[11:13:09] 1>MLST wmdin1h-4rb.jpg
[11:13:09] 1>550 Can't check for file existence
[11:13:09] 1>MLST wmdin1h-5f.jpg
[11:13:09] 1>550 Can't check for file existence
[11:13:09] 1>MLST wmdin1h-14n.jpg
[11:13:09] 1>550 Can't check for file existence
[11:13:09] 1>MLST wmdin1l-1k.jpg
[11:13:09] 1>550 Can't check for file existence
[11:13:09] 1>MLST wmdin1l-3r.jpg
[11:13:09] 1>550 Can't check for file existence
[11:13:09] 1>MLST wmdin1l-4rb.jpg
[11:13:09] 1>550 Can't check for file existence

.
. This is me skipping ahead since you don't really want to see 12,000 lines of MLST commands which all result in the same reply.
.

[11:22:00] 1>MLST wmdin228s-5f.jpg
[11:22:00] 1>550 Can't check for file existence
[11:22:00] 1>MLST wmdin228s-14n.jpg
[11:22:00] 1>550 Can't check for file existence
[11:22:00] 1>MLST wmdin228t-1k.jpg
[11:22:00] 1>550 Can't check for file existence
[11:22:00] 1>MLST wmdin228t-3r.jpg
[11:22:00] 1>550 Can't check for file existence
[11:22:00] 1>MLST wmdin228t-4rb.jpg
[11:22:00] 1>550 Can't check for file existence
[11:22:00] 1>MLST wmdin228t-5f.jpg
[11:22:00] 1>550 Can't check for file existence
[11:22:00] 1>MLST wmdin228t-14n.jpg
[11:22:00] 1>550 Can't check for file existence
[11:22:00] 1>MLST wmdin228ta-1k.jpg
[11:22:01] 1>550 Can't check for file existence
[11:22:01] 1>MLST wmdin228ta-3r.jpg
[11:22:01] 1>550 Can't check for file existence
[11:22:01] 1>MLST wmdin228ta-4rb.jpg
[11:22:01] 1>550 Can't check for file existence
[11:22:01] The operation has been added to the transfer queue. Start the transfer queue to process it.

2014-07-02 16:47:46

The new behavior in version 6 will be reverted back in the next build. This should solve the problem you are seeing.
The new build with the changes is available now:
https://www.smartftp.com/download

SamOldman

2014-07-03 19:16:11

Okay, so I closed the remote browser during the batch upload, but I only noticed an increase in speed of around 10%. It looks like all that did was hide the output from the server. It's an improvement, but not a significant one.

Any other ideas?

2014-07-03 20:00:18

No your assumption is wrong.
You can increase the number of workers in the transfer queue.

SamOldman

2014-07-04 13:46:38

Come on... Just because I closed the browser window doesn't mean magically nothing happens in the background. The server still maintains the log, even if I can't see it. I checked the log on the server after the upload actually began and it showed that SmartFTP still continued with its MLST requests. The increase in speed I saw could just as easily be attributed to decreased server load.

Also, it doesn't matter how many workers I have in the queue because the MLST requests take place BEFORE the transfer begins. I still have to wait for all of this to complete before the files begin to upload.

Belazor

2014-07-08 18:39:15

I have the same problem, and closing the remote browser does not work.

mb wrote: No your assumption is wrong.

You can increase the number of workers in the transfer queue.

This is a pretty pathetic answer. Increasing the amount of workers doesn't magically make the server accept more connections per IP, and it doesn't change the fact that SmartFTP v5 did not have this bug. This bug was introduced in v6, and needs to be resolved ASAP.

2014-07-13 20:23:01

A new build with the changes is available now:
https://www.smartftp.com/download

SamOldman

2014-07-15 19:53:20

I upgraded to 6.0.2048.0, but it is still exhibiting the exact same behavior. Is there a setting I need to adjust, now?

2014-07-15 22:39:38

If you upload a folder and a folder with the same already exists on the server, you will see a folder replace dialog. In this dialog, select "Use automatic rules".

SamOldman

2014-07-16 14:00:35

mb wrote: If you upload a folder and a folder with the same already exists on the server, you will see a folder replace dialog. In this dialog, select "Use automatic rules".

I am not overwriting or replacing folders/files. Every file that I am uploading is a new file, and every folder that I create is a new folder.

2014-07-16 14:18:28

If the folder does not exist on the server and you upload a folder then there is only a single MLST command before the item is added to the transfer queue. If you upload a big selection of files then there is a MLST command for each file. This kind of makes sense. So instead of uploading the big selection, just upload a folder containing the files.

SamOldman

2014-07-16 15:16:22

That did the trick, but it is in no way an optimal solution. It pretty much makes sure that if I *do* need to upload to an existing folder then I will still have to deal with the thousands and thousands of MLST commands.

Oh well. Maybe this "feature" will be fixed in future versions.