overwrite in loop

Hi,
 
I'm currently testing the enterprise edition.
I want to synchronize files from a FTP server down to a local drive.
Since it's a one way sync with delete it needs to check if the file is already existing locally.
that works well....but
it keeps resyncing the files again and again. Minutes, hours and datestamp are correct.
Simple ftp transfer in the internal network.
The favorite settings are set to "enable" in "keep modify time" settings.
Remote FTP server is an IIS 8.5 with the following supported commands:
 ABOR
 ACCT
 ADAT *
 ALLO
 APPE
 AUTH
 CCC
 CDUP
 CWD
 DELE
 ENC *
 EPRT
 EPSV
 FEAT
 HELP
 HOST
 LANG
 LIST
 MDTM
 MIC *
 MKD
 MODE
 NLST
 NOOP
 OPTS
 PASS
 PASV
 PBSZ
 PORT
 PROT
 PWD
 QUIT
 REIN
 REST
 RETR
 RMD
 RNFR
 RNTO
 SITE
 SIZE
 SMNT
 STAT
 STOR
 STOU
 STRU
 SYST
 TYPE
 USER
 XCUP
 XCWD
 XMKD
 XPWD
 XRMD
 
 
This is the first sync before the file exists locally.
2016-11-08T09:49:51Z Operation begin
2016-11-08T09:49:51Z Copying file. Source="::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log", Destination="E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Getting properties (Size, Date modified) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Item: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:51Z Transfer restarting at position 0.
2016-11-08T09:49:51Z 9506 bytes transferred. (97,7 KB/s) (95 ms)
2016-11-08T09:49:51Z Getting properties (Date modified, Date created) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Item: Date modified=2016-09-21T21:14:00
2016-11-08T09:49:51Z Setting properties (Date modified) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Getting properties (Size) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Item: Size=9506
2016-11-08T09:49:51Z Using cached properties
2016-11-08T09:49:51Z Getting properties (Size) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Item: Size=9506
2016-11-08T09:49:51Z Source File Size=9506, Destination file size=9506
2016-11-08T09:49:51Z Operation end

And this is the second that should skip the file (default ruleset of smartftp is in use with checked setting of hash values too):
2016-11-08T09:49:57Z Operation begin
2016-11-08T09:49:57Z Copying file. Source="::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log", Destination="E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Getting properties (Size, Date modified) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z Getting properties (Size, Date modified) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z Source file: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z Destination file: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z No rule matched. Default action="Overwrite".
2016-11-08T09:49:57Z Transfer restarting at position 0.
2016-11-08T09:49:57Z 9506 bytes transferred. (165 KB/s) (56 ms)
2016-11-08T09:49:57Z Getting properties (Date modified, Date created) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z Setting properties (Date modified) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Getting properties (Size) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Size=9506
2016-11-08T09:49:57Z Using cached properties
2016-11-08T09:49:57Z Getting properties (Size) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Size=9506
2016-11-08T09:49:57Z Source File Size=9506, Destination file size=9506
2016-11-08T09:49:57Z Operation end
 
 
Thanks a lot

Another thing I'm experiencing is a timeshift of the remote files wiht one hour ahead. It seems to have to do with DST.
Tested both (Automatci and correct timezone) settings in favorite. No change...thanks.
Just have seen this after a smartftp restart. this has been running for days and was started before time changes due to DST.

We are unable to reproduce the issue you have reported. We have tested it with the .txt file on our ftp server (IIS): ftp.smartftp.com
 
The log looks like this in our case:
 
<blockquote class="ipsBlockquote">
2016-11-08T11:58:18Z Item: Size=109, Date modified=2012-08-12T03:25:55
2016-11-08T11:58:18Z Source file: Size=109, Date modified=2012-08-12T03:25:55
2016-11-08T11:58:18Z Destination file: Size=109, Date modified=2012-08-12T03:25:55
2016-11-08T11:58:18Z Rule "IF Destination Time=Equal AND Size=Equal AND Transfer=No matter THEN Skip" matched. Action="Skip".
2016-11-08T11:58:18Z Skipping file "/This is a DEMO server.txt". Reason: File exist rules.
 
</blockquote>
 
 
Did you change the file exist rules?

thanks. no the rules are the default ones.
It is really strange. I'm deleting some files now and will resync.
How to deal with DST?

>sync
You can also try it with the same .txt file on our server (ftp.smartftp.com)
 
>dst
Do you see the time discrepancy in the remote browser (file view)? One problem is that the IIS FTP server still does not support MLSD and therefore returns the times in the directory listing in the local system time. This time is then converted to UTC. For the keep file time/get file time function, MDTM is used which returns the time in UTC and you should not see any issues because in this case the time is never converted.

thanks. I have set it back to Automatic timezone. I have seen the discrepancy in thr remote browser. somehow it is now showing the files correctly. but it still overwrites. the directory has 10K+ of files. maybe thats the problem. I have excluded this directory completely to go on with testing.
the other dirs work well. its really strange. for testing i have the cache en- or disabled...doesnt change anything.
 
Another question: If during syncing a file that was in queue to be synced is deleted in source and therefore cannot and also should not be synced anymore...how to cover this case the item NOT to stuck in queue with "the system cannot find the file specified"?
 
Sidenote: importing a schedule over smartftp (with or without admin privileges) on WinServer2012R2x64 lets smartftp crash.
Tested by creating a schedule the intended way over smartftp and then export it, import it.

Thanks

>overwrite
Test it first with the .txt file on our server: ftp.smartftp.com
Then create a new connection (new favorite) and test it with a single file on your server.
 
>stuck in queue with "the system cannot find the file specified"?
You need to rethink your process if you have concurrent writers/readers.
 
>schedule import
Fixed. Bug fix will be in the next build.

thanks...but like I wrote it is working for certain directories on our server. so it will work with your txt file and your server too. anyway.
 
regarding your server. I dont want to open the huge passive port range from LAN to public you might have setup in your IIS.
So I have downloaded your file and tested.. it works. Single file with new favorite and old favorite works. Like I already wrote. But it doesnt for this "special" directory. Which is only special by the meaning of amount of files (10K+) with ~16KB each in average.
Name pattern is 20160923 160000 01 {DC07C408-378E-410E-8062-1709E705FBE6}.log
 
Have you maybe tested with 10K+ files?
Anything regarding my questions? Thanks
 

>overwrite
the file name and the number of files is irrelevant. Does it happen for all files in this "special folder" or randomly for some files (or is it always the same file)?
 
Your other questions have already been addressed in the last reply.

thanks.
 
>overwrite
the file name and the number of files is irrelevant. Does it happen for all files in this "special folder" or randomly for some files (or is it always the same file)?
All files. not random.
 
>stuck in queue with "the system cannot find the file specified"?
You need to rethink your process if you have concurrent writers/readers.
thanks. its just the case if the sync of previous files take so long that another process has deleted some old files that do not have to be synced anymore.
is there an option to kick an item out if queue after X failed retries?

I have tested one random file from the "faulty" directory in the directory were the other files work fine.
And the file is not working there as well while all others do. How can I send you the file in private as ZIP? Thanks

I made further test. I took one random file from the "faulty" directory and cpoied it three time in the directory were the other files work fine.
one I left unchanged, second was just renamed to something simple like test and in the third one deleted the content.
The files with the name (empty or not) fail. the third one not.
Have you tested a file with Name pattern?
20160923 160000 01 {DC07C408-378E-410E-8062-1709E705FBE6}.log
Thanks

The name is irrelevant.
 
Are you testing everything with 1 worker? If not, test it with 1 worker.

yes, I'm testing with 1 worker (in favorite settings and transfer queue).
And the file name is somehow relevant. It is reproducable.
It happens whenever there is a space in the filename. But only with the mentioned files.
creating a new files with spaces are working fine.
 
can I track it down further? are there different types of blank spaces in filenames?

The file name is not relevant for SmartFTP, maybe for your file system.
Is the destination E: a local drive with NTFS?
 
We will log the times with milliseconds in the next build.

The new build 2263 is available now:
https://www.smartftp.com/download
 
Please test it and post the log.

tested. still the same. attached the logs. I have also send an email with the zipped file.

Please extract the empty logfile, copy it to your server and do a short test. Thanks

1st transfer creates file:

2016-11-09T09:58:38Z Operation begin
2016-11-09T09:58:38Z Copying file. Source="::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20161109 105301 01 {0B004120-59B8-4770-A035-D1D2DBB770BB}.log", Destination="E:\LOGS\20161109 105301 01 {0B004120-59B8-4770-A035-D1D2DBB770BB}.log"
2016-11-09T09:58:38Z Getting properties (Size, Date modified) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20161109 105301 01 {0B004120-59B8-4770-A035-D1D2DBB770BB}.log"
2016-11-09T09:58:38Z Item: Size=4742, Date modified=2016-11-09T09:53:00.000
2016-11-09T09:58:38Z Transfer restarting at position 0.
2016-11-09T09:58:38Z 4742 bytes transferred. (73,5 KB/s) (63 ms)
2016-11-09T09:58:38Z Getting properties (Date modified, Date created) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20161109 105301 01 {0B004120-59B8-4770-A035-D1D2DBB770BB}.log"
2016-11-09T09:58:38Z Item: Date modified=2016-11-09T09:53:00.000
2016-11-09T09:58:38Z Setting properties (Date modified) of "E:\LOGS\20161109 105301 01 {0B004120-59B8-4770-A035-D1D2DBB770BB}.log"
2016-11-09T09:58:38Z Getting properties (Size) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20161109 105301 01 {0B004120-59B8-4770-A035-D1D2DBB770BB}.log"
2016-11-09T09:58:38Z Item: Size=4742
2016-11-09T09:58:38Z Using cached properties
2016-11-09T09:58:38Z Getting properties (Size) of "E:\LOGS\20161109 105301 01 {0B004120-59B8-4770-A035-D1D2DBB770BB}.log"
2016-11-09T09:58:38Z Item: Size=4742
2016-11-09T09:58:38Z Source File Size=4742, Destination file size=4742
2016-11-09T09:58:38Z Operation end

2nd transfer overwrites it:

2016-11-09T09:58:42Z Operation begin
2016-11-09T09:58:42Z Copying file. Source="::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20161109 105301 01 {0B004120-59B8-4770-A035-D1D2DBB770BB}.log", Destination="E:\LOGS\20161109 105301 01 {0B004120-59B8-4770-A035-D1D2DBB770BB}.log"
2016-11-09T09:58:42Z Getting properties (Size, Date modified) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20161109 105301 01 {0B004120-59B8-4770-A035-D1D2DBB770BB}.log"
2016-11-09T09:58:42Z Item: Size=4742, Date modified=2016-11-09T09:53:00.000
2016-11-09T09:58:42Z Getting properties (Size, Date modified) of "E:\LOGS\20161109 105301 01 {0B004120-59B8-4770-A035-D1D2DBB770BB}.log"
2016-11-09T09:58:42Z Item: Size=4742, Date modified=2016-11-09T09:53:00.000
2016-11-09T09:58:42Z Source file: Size=4742, Date modified=2016-11-09T09:53:00.000
2016-11-09T09:58:42Z Destination file: Size=4742, Date modified=2016-11-09T09:53:00.000
2016-11-09T09:58:42Z No rule matched. Default action="Overwrite".
2016-11-09T09:58:42Z Transfer restarting at position 0.
2016-11-09T09:58:42Z 4742 bytes transferred. (94,5 KB/s) (49 ms)
2016-11-09T09:58:42Z Getting properties (Date modified, Date created) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20161109 105301 01 {0B004120-59B8-4770-A035-D1D2DBB770BB}.log"
2016-11-09T09:58:42Z Item: Date modified=2016-11-09T09:53:00.000
2016-11-09T09:58:42Z Setting properties (Date modified) of "E:\LOGS\20161109 105301 01 {0B004120-59B8-4770-A035-D1D2DBB770BB}.log"
2016-11-09T09:58:42Z Getting properties (Size) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20161109 105301 01 {0B004120-59B8-4770-A035-D1D2DBB770BB}.log"
2016-11-09T09:58:42Z Item: Size=4742
2016-11-09T09:58:42Z Using cached properties
2016-11-09T09:58:42Z Getting properties (Size) of "E:\LOGS\20161109 105301 01 {0B004120-59B8-4770-A035-D1D2DBB770BB}.log"
2016-11-09T09:58:42Z Item: Size=4742
2016-11-09T09:58:42Z Source File Size=4742, Destination file size=4742
2016-11-09T09:58:42Z Operation end

 

It is a bug (not RFC 3659 compliant) with IIS's FTP server. It returns access denied if it the filename starts with a date string, followed by a space:
MDTM 20161109 094300.log
550 Access is denied.
 
It means that SmartFTP is not able to retrieve the exact date/time (STAT, LIST only returns minutes and no seconds) for this file and as a consequence, none of the file exist rules match (time != equal).

thanks a lot. "thanks" IIS.
The glory file name
So my options are renaming files, changing server or fixing creation of logs to some other pattern.
any chance reagrding this topic:
>stuck in queue with "the system cannot find the file specified"?
You need to rethink your process if you have concurrent writers/readers.
thanks. its just the case if the sync of previous files take so long that another process has deleted some old files that do not have to be synced anymore.
is there an option to kick an item out if queue after X failed retries?
or something else?
thanks a lot

>is there an option to kick an item out if queue after X failed retries?
No. But if they reach the max retry count, their state changes to "failed" and they will be ignored by the transfer queue.

thank you. that works and the job keeps running.
I have set the application to close after finishing queue and start queue without remembering.
Can I set it somehow to start with a clean queue?
thanks

You can delete the %appdata%\SmartFTP\Client 2.0\Transfer Queue.xml file

is this file just updated during closing?
I would prefer setting it to read-only. Will this cause any problems? My first test was ok.
Thanks

The file is automatically saved (see Auto Save transfer queue plugin) after some time and when the application is closed. It probably silently fails in your case.
 
I would like to emphasis that neither deleting the file, nor setting it to read-only is something we are recommending.

Thanks. I know that this is not the best solution, but it works.
Unfortunately the sync process in smartftp does not cover this regular sync case.
I would expect the sync process to do a rescan in case of a failure (at least after a program restart) to check if failed items are really failed due to connection or simply for being missing in source.
thats what sync tools do. depending on rules and sync mode like "A>B with delete" this would be the normal way.
Maybe you can add it in a future version. Thanks.