SFTP Uploads file before it's complete

bhilgenkamp

2008-12-11 23:14:33

I encode video files live at events to get them up on our website as soon as possible. I am trying to use SFTP to monitor the folder I capture the clips to to automatically upload the clips once they are complete. The problem is that SFTP constantly sees the files being changed so it is constantly uploading the changes in the file to my FTP server. This places considerable load on my FTP server since it is getting thousands of hits per minute from SFTP uploading the changes to the file. I thought I could get around this by setting a schedule for every 3 minutes, but as soon as the upload starts after the 3 minutes the same problem occurs. So my question is, can I get around this somehow, or is this a problem with my encoding software? I used this method about 9 months ago with an older version of SFTP and did not have this problem. I need a solution that is as simple as possible because the guys that are doing the encoding at the events aren't very computer literate and I don't want them to have to do more then click start and stop in the encoder software. Thanks

+- System -----------------------------
Microsoft Windows XP Professional
Service Pack 2 (Build 2600)

CPU Speed : 1994 MHz
Total Memory : 2046 MB
Free Memory : 1412 MB

+- SmartFTP ---------------------------
Version : 3.0.1024.32
Time Stamp : 2008-11-23 16:59:11
Platform : x86
Id : 400057274
Days in use : 310

+- Application DLL --------------------
sfFTPLib.dll : 1.5.15.6
sfFavorites.dll : 1.0.20.32
sfFavoritesShellExtension.dll : 1.0.1.22
sfTransferQueue.dll : 1.0.17.32
sfFTPShellExtension.dll : 1.0.15.32
SmartFTPPS.dll : 3.0.1024.32

+- Language ---------------------------
SmartFTP.exe : 3.0.1024.32

+- Internet Explorer ------------------
Version : 7.0.5730.13

+- Winsock ----------------------------
Winsock : 2.2

2008-12-11 23:55:30

Hello ..

SmartFTP should resume the existing file server.
Can you post the log from the file in the transfer queue? To get it double click on the file SmartFTP is monitoring/uploading.

Regards,
Mat

bhilgenkamp

2008-12-12 00:01:11

Hello ..

SmartFTP should resume the existing file server.
Can you post the log from the file in the transfer queue? To get it double click on the file SmartFTP is monitoring/uploading.

Regards,
Mat

This repeats over and over so I've only copied a portion of the log

[17:57:53] 257 "/event_upload/FLV/" is current directory.
[17:57:53] SIZE 516346_549_10_Fowler_Mustangs.f4v
[17:57:53] 213 32274852 bytes
[17:57:53] MDTM 516346_549_10_Fowler_Mustangs.f4v
[17:57:53] 213 20081211235753
[17:57:53] CWD /event_upload/FLV
[17:57:53] 250 CWD command successful
[17:57:53] PWD
[17:57:53] 257 "/event_upload/FLV/" is current directory.
[17:57:53] CWD /event_upload/FLV
[17:57:53] 250 CWD command successful
[17:57:53] PWD
[17:57:53] 257 "/event_upload/FLV/" is current directory.
[17:57:53] CWD /event_upload/FLV
[17:57:53] 250 CWD command successful
[17:57:53] PWD
[17:57:53] 257 "/event_upload/FLV/" is current directory.
[17:57:53] CWD /event_upload/FLV
[17:57:53] 250 CWD command successful
[17:57:53] PWD
[17:57:53] 257 "/event_upload/FLV/" is current directory.
[17:57:53] Source File: Size=32322737, SizeUnit=Byte, Time=2008-12-11T23:55:25, TimeFormat=Exact
[17:57:53] Destination File: Size=32274852, SizeUnit=Byte, Time=2008-12-11T23:57:53, TimeFormat=Exact
[17:57:53] RecentTime=2008-12-11T23:57:53
[17:57:53] Rule "IF Destination Time=Recent AND Size=Smaller AND Transfer=No Matter THEN Resume" matched. Action="Resume".
[17:57:53] CWD /event_upload/FLV
[17:57:53] 250 CWD command successful
[17:57:53] PWD
[17:57:53] 257 "/event_upload/FLV/" is current directory.
[17:57:53] PASV
[17:57:54] 227 Entering Passive Mode (192,168,1,200,11,185)
[17:57:54] Passive ip address returned from server different from server ip.
[17:57:54] Opening data connection to 192.168.1.200 Port: 3001
[17:57:54] REST 32274852
[17:57:54] 350 Operation pending more information
[17:57:54] STOR 516346_549_10_Fowler_Mustangs.f4v
[17:57:54] 150 Opening connection
[17:57:54] 90470 bytes transferred. (N/A/s) (0 ms)
[17:57:54] 226 Transfer complete
[17:57:54] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] SIZE 516346_549_10_Fowler_Mustangs.f4v
[17:57:55] 213 32365322 bytes
[17:57:55] Source File Size = 32608778, Destination File Size = 32365322
[17:57:55] File size mismatch.
[17:57:55] Transfer failed.
[17:57:55] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] Remote file exist check: "516346_549_10_Fowler_Mustangs.f4v".
[17:57:55] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] SIZE 516346_549_10_Fowler_Mustangs.f4v
[17:57:55] 213 32365322 bytes
[17:57:55] MDTM 516346_549_10_Fowler_Mustangs.f4v
[17:57:55] 213 20081211235753
[17:57:55] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] Source File: Size=32608778, SizeUnit=Byte, Time=2008-12-11T23:55:25, TimeFormat=Exact
[17:57:55] Destination File: Size=32365322, SizeUnit=Byte, Time=2008-12-11T23:57:53, TimeFormat=Exact
[17:57:55] RecentTime=2008-12-11T23:57:54
[17:57:55] Rule "IF Destination Time=Recent AND Size=Smaller AND Transfer=No Matter THEN Resume" matched. Action="Resume".
[17:57:55] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] PASV
[17:57:55] 227 Entering Passive Mode (192,168,1,200,11,185)
[17:57:55] Passive ip address returned from server different from server ip.
[17:57:55] Opening data connection to 192.168.1.200 Port: 3001
[17:57:55] REST 32365322
[17:57:55] 350 Operation pending more information
[17:57:55] STOR 516346_549_10_Fowler_Mustangs.f4v
[17:57:55] 150 Opening connection
[17:57:55] 305679 bytes transferred. (2.33 MB/s) (125 ms)
[17:57:55] 226 Transfer complete
[17:57:55] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] SIZE 516346_549_10_Fowler_Mustangs.f4v
[17:57:55] 213 32671001 bytes
[17:57:55] Source File Size = 32703850, Destination File Size = 32671001
[17:57:55] File size mismatch.
[17:57:55] Transfer failed.
[17:57:55] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] Remote file exist check: "516346_549_10_Fowler_Mustangs.f4v".
[17:57:55] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] SIZE 516346_549_10_Fowler_Mustangs.f4v
[17:57:55] 213 32671001 bytes
[17:57:55] MDTM 516346_549_10_Fowler_Mustangs.f4v
[17:57:55] 213 20081211235755
[17:57:55] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] Source File: Size=32703850, SizeUnit=Byte, Time=2008-12-11T23:55:25, TimeFormat=Exact
[17:57:55] Destination File: Size=32671001, SizeUnit=Byte, Time=2008-12-11T23:57:55, TimeFormat=Exact
[17:57:55] RecentTime=2008-12-11T23:57:55
[17:57:55] Rule "IF Destination Time=Recent AND Size=Smaller AND Transfer=No Matter THEN Resume" matched. Action="Resume".
[17:57:55] CWD /event_upload/FLV
[17:57:55] 250 CWD command successful
[17:57:55] PWD
[17:57:55] 257 "/event_upload/FLV/" is current directory.
[17:57:55] PASV
[17:57:55] 227 Entering Passive Mode (192,168,1,200,11,185)
[17:57:55] Passive ip address returned from server different from server ip.
[17:57:55] Opening data connection to 192.168.1.200 Port: 3001
[17:57:55] REST 32671001
[17:57:55] 350 Operation pending more information
[17:57:55] STOR 516346_549_10_Fowler_Mustangs.f4v
[17:57:55] 150 Opening connection
[17:57:55] 69154 bytes transferred. (4.39 MB/s) (15 ms)
[17:57:56] 226 Transfer complete
[17:57:56] CWD /event_upload/FLV
[17:57:56] 250 CWD command successful
[17:57:56] PWD
[17:57:56] 257 "/event_upload/FLV/" is current directory.
[17:57:56] CWD /event_upload/FLV
[17:57:56] 250 CWD command successful
[17:57:56] PWD
[17:57:56] 257 "/event_upload/FLV/" is current directory.
[17:57:56] SIZE 516346_549_10_Fowler_Mustangs.f4v
[17:57:56] 213 32740155 bytes
[17:57:56] Source File Size = 32786194, Destination File Size = 32740155
[17:57:56] File size mismatch.
[17:57:56] Transfer failed.
[17:57:56] CWD /event_upload/FLV
[17:57:56] 250 CWD command successful
[17:57:56] PWD
[17:57:56] 257 "/event_upload/FLV/" is current directory.
[17:57:56] Remote file exist check: "516346_549_10_Fowler_Mustangs.f4v".
[17:57:56] CWD /event_upload/FLV
[17:57:56] 250 CWD command successful
[17:57:56] PWD
[17:57:56] 257 "/event_upload/FLV/" is current directory.
[17:57:56] SIZE 516346_549_10_Fowler_Mustangs.f4v
[17:57:56] 213 32740155 bytes
[17:57:56] MDTM 516346_549_10_Fowler_Mustangs.f4v
[17:57:56] 213 20081211235755
[17:57:56] CWD /event_upload/FLV
[17:57:56] 250 CWD command successful
[17:57:56] PWD
[17:57:56] 257 "/event_upload/FLV/" is current directory.
[17:57:56] CWD /event_upload/FLV
[17:57:56] 250 CWD command successful
[17:57:56] PWD
[17:57:56] 257 "/event_upload/FLV/" is current directory.
[17:57:56] CWD /event_upload/FLV
[17:57:56] 250 CWD command successful
[17:57:56] PWD
[17:57:56] 257 "/event_upload/FLV/" is current directory.
[17:57:56] CWD /event_upload/FLV
[17:57:56] 250 CWD command successful
[17:57:56] PWD
[17:57:56] 257 "/event_upload/FLV/" is current directory.
[17:57:56] Source File: Size=32792422, SizeUnit=Byte, Time=2008-12-11T23:55:25, TimeFormat=Exact
[17:57:56] Destination File: Size=32740155, SizeUnit=Byte, Time=2008-12-11T23:57:55, TimeFormat=Exact
[17:57:56] RecentTime=2008-12-11T23:57:56
[17:57:56] Rule "IF Destination Time=Recent AND Size=Smaller AND Transfer=No Matter THEN Resume" matched. Action="Resume".
[17:57:56] CWD /event_upload/FLV
[17:57:56] 250 CWD command successful
[17:57:56] PWD
[17:57:56] 257 "/event_upload/FLV/" is current directory.
[17:57:56] PASV
[17:57:56] 227 Entering Passive Mode (192,168,1,200,11,185)
[17:57:56] Passive ip address returned from server different from server ip.
[17:57:56] Opening data connection to 192.168.1.200 Port: 3001
[17:57:56] REST 32740155
[17:57:56] 350 Operation pending more information
[17:57:56] STOR 516346_549_10_Fowler_Mustangs.f4v
[17:57:56] 150 Opening connection
[17:57:56] 99648 bytes transferred. (6.33 MB/s) (15 ms)
[17:57:56] 226 Transfer complete
[17:57:56] CWD /event_upload/FLV
[17:57:57] 250 CWD command successful
[17:57:57] PWD
[17:57:57] 257 "/event_upload/FLV/" is current directory.
[17:57:57] CWD /event_upload/FLV
[17:57:57] 250 CWD command successful
[17:57:57] PWD
[17:57:57] 257 "/event_upload/FLV/" is current directory.
[17:57:57] SIZE 516346_549_10_Fowler_Mustangs.f4v
[17:57:57] 213 32839803 bytes
[17:57:57] Source File Size = 33097407, Destination File Size = 32839803
[17:57:57] File size mismatch.
[17:57:57] Transfer failed.
[17:57:57] CWD /event_upload/FLV
[17:57:57] 250 CWD command successful
[17:57:57] PWD
[17:57:57] 257 "/event_upload/FLV/" is current directory.
[17:57:57] Remote file exist check: "516346_549_10_Fowler_Mustangs.f4v".
[17:57:57] CWD /event_upload/FLV
[17:57:57] 250 CWD command successful
[17:57:57] PWD
[17:57:57] 257 "/event_upload/FLV/" is current directory.
[17:57:57] SIZE 516346_549_10_Fowler_Mustangs.f4v
[17:57:57] 213 32839803 bytes
[17:57:57] MDTM 516346_549_10_Fowler_Mustangs.f4v
[17:57:57] 213 20081211235756
[17:57:57] CWD /event_upload/FLV
[17:57:57] 250 CWD command successful
[17:57:57] PWD
[17:57:57] 257 "/event_upload/FLV/" is current directory.
[17:57:57] CWD /event_upload/FLV
[17:57:57] 250 CWD command successful
[17:57:57] PWD
[17:57:57] 257 "/event_upload/FLV/" is current directory.
[17:57:57] CWD /event_upload/FLV
[17:57:57] 250 CWD command successful
[17:57:57] PWD
[17:57:57] 257 "/event_upload/FLV/" is current directory.
[17:57:57] CWD /event_upload/FLV
[17:57:57] 250 CWD command successful
[17:57:57] PWD
[17:57:57] 257 "/event_upload/FLV/" is current directory.
[17:57:57] Source File: Size=33097407, SizeUnit=Byte, Time=2008-12-11T23:55:25, TimeFormat=Exact
[17:57:57] Destination File: Size=32839803, SizeUnit=Byte, Time=2008-12-11T23:57:56, TimeFormat=Exact
[17:57:57] RecentTime=2008-12-11T23:57:56
[17:57:57] Rule "IF Destination Time=Recent AND Size=Smaller AND Transfer=No Matter THEN Resume" matched. Action="Resume".
[17:57:57] CWD /event_upload/FLV
[17:57:57] 250 CWD command successful
[17:57:57] PWD
[17:57:57] 257 "/event_upload/FLV/" is current directory.
[17:57:57] PASV
[17:57:57] 227 Entering Passive Mode (192,168,1,200,11,185)
[17:57:57] Passive ip address returned from server different from server ip.
[17:57:57] Opening data connection to 192.168.1.200 Port: 3001
[17:57:57] REST 32839803
[17:57:57] 350 Operation pending more information
[17:57:57] STOR 516346_549_10_Fowler_Mustangs.f4v
[17:57:57] 150 Opening connection
[17:57:57] 341195 bytes transferred. (2.60 MB/s) (125 ms)
[17:57:58] 226 Transfer complete
[17:57:58] CWD /event_upload/FLV
[17:57:58] 250 CWD command successful
[17:57:58] PWD
[17:57:58] 257 "/event_upload/FLV/" is current directory.
[17:57:58] CWD /event_upload/FLV
[17:57:58] 250 CWD command successful
[17:57:58] PWD
[17:57:58] 257 "/event_upload/FLV/" is current directory.
[17:57:58] SIZE 516346_549_10_Fowler_Mustangs.f4v
[17:57:58] 213 33180998 bytes
[17:57:58] Source File Size = 33199216, Destination File Size = 33180998
[17:57:58] File size mismatch.
[17:57:58] Transfer failed.
[17:57:58] CWD /event_upload/FLV
[17:57:58] 250 CWD command successful
[17:57:58] PWD
[17:57:58] 257 "/event_upload/FLV/" is current directory.
[17:57:58] Remote file exist check: "516346_549_10_Fowler_Mustangs.f4v".
[17:57:58] CWD /event_upload/FLV
[17:57:58] 250 CWD command successful
[17:57:58] PWD
[17:57:58] 257 "/event_upload/FLV/" is current directory.
[17:57:58] SIZE 516346_549_10_Fowler_Mustangs.f4v
[17:57:58] 213 33180998 bytes
[17:57:58] MDTM 516346_549_10_Fowler_Mustangs.f4v
[17:57:58] 213 20081211235757
[17:57:58] CWD /event_upload/FLV
[17:57:58] 250 CWD command successful
[17:57:58] PWD
[17:57:58] 257 "/event_upload/FLV/" is current directory.
[17:57:58] CWD /event_upload/FLV
[17:57:58] 250 CWD command successful
[17:57:58] PWD
[17:57:58] 257 "/event_upload/FLV/" is current directory.
[17:57:58] CWD /event_upload/FLV
[17:57:58] 250 CWD command successful
[17:57:58] PWD
[17:57:58] 257 "/event_upload/FLV/" is current directory.
[17:57:58] CWD /event_upload/FLV
[17:57:58] 250 CWD command successful
[17:57:58] PWD
[17:57:58] 257 "/event_upload/FLV/" is current directory.
[17:57:58] Source File: Size=33199216, SizeUnit=Byte, Time=2008-12-11T23:55:25, TimeFormat=Exact
[17:57:58] Destination File: Size=33180998, SizeUnit=Byte, Time=2008-12-11T23:57:57, TimeFormat=Exact
[17:57:58] RecentTime=2008-12-11T23:57:58
[17:57:58] Rule "IF Destination Time=Recent AND Size=Smaller AND Transfer=No Matter THEN Resume" matched. Action="Resume".
[17:57:58] CWD /event_upload/FLV
[17:57:58] 250 CWD command successful
[17:57:58] PWD
[17:57:58] 257 "/event_upload/FLV/" is current directory.
[17:57:58] PASV
[17:57:58] 227 Entering Passive Mode (192,168,1,200,11,185)
[17:57:58] Passive ip address returned from server different from server ip.
[17:57:58] Opening data connection to 192.168.1.200 Port: 3001
[17:57:58] REST 33180998
[17:57:58] 350 Operation pending more information
[17:57:58] STOR 516346_549_10_Fowler_Mustangs.f4v
[17:57:58] 150 Opening connection
[17:57:58] 56127 bytes transferred. (3.56 MB/s) (15 ms)
[17:57:58] 226 Transfer complete
[17:57:58] CWD /event_upload/FLV
[17:57:58] 250 CWD command successful
[17:57:58] PWD
[17:57:58] 257 "/event_upload/FLV/" is current directory.
[17:57:58] CWD /event_upload/FLV
[17:57:58] 250 CWD command successful

2008-12-12 02:43:13

Hello ...

From the log it looks that SmartFTP is correctly resuming the file. If you can tell your encoder to write the file to a folder you don't monitor and as soon as the file is completely encoded move it to a folder (you should be able to automate this as well) to a folder which SmartFTP monitors.

Regards,
Mat

bhilgenkamp

2008-12-12 03:22:45

Hello ...

From the log it looks that SmartFTP is correctly resuming the file. If you can tell your encoder to write the file to a folder you don't monitor and as soon as the file is completely encoded move it to a folder (you should be able to automate this as well) to a folder which SmartFTP monitors.

Regards,
Mat

That's what I was thinking of for a backup plan if there were no other options. Just trying to keep it as simple as possible. Thanks Mat