Memory Leak

I have a folder on our network \\server\d$\http and I have it monitor it for changes, I processed the folder to get the remote FTP site up to date. Then next day smartFTP was using 3 GB of RAM and about the same Virtual memory. I stopped and saved the queue and shut down smartFTP. The queue file was 138 MB and smartFTP said it had 139,000+ items found. When I restarted smartFTP it automatically loaded the queue and was using 900MB. A little while later it hit a folder (about 440 files) and said it was adding items and the memory really started climbing fast.

What tools can I use to help find where a memory leak may be coming from?

Here's help info after I stopped the operation and deleted the queue.

+- System -----------------------------
Microsoft Windows XP Professional x64 Edition
Service Pack 2 (Build 3790)

CPU Speed : 2667 MHz
Total Memory : 4094 MB
Free Memory : 2618 MB

+- SmartFTP ---------------------------
Version : 3.0.1021.14
Time Stamp : 2008-08-07 21:42:10
Platform : x64

Days in use : 1

+- Application DLL --------------------
sfFTPLib.dll : 1.5.13.42
sfFavorites.dll : 1.0.17.14
sfFavoritesShellExtension.dll : 1.0.1.13
sfTransferQueue.dll : 1.0.14.14
sfFTPShellExtension.dll : 1.0.12.14
SmartFTPPS.dll : 3.0.1021.14

+- Language ---------------------------
SmartFTP.exe : 3.0.1021.14

+- Internet Explorer ------------------
Version : 7.0.5730.13

+- Winsock ----------------------------
Winsock : 2.2

Hello ...

There are several tools to check for memory leaks but most of them target the experienced developers. Just by looking at the task manager it is impossible to find memory leaks. We can relatively easy spot memory leaks with the debug version of SmartFTP but to do this we need a reproducible case. The fact that 139k items use 900MB of memory is no reason to believe that there is a memory leak.

Regards,
Mat

Hello ...

There are several tools to check for memory leaks but most of them target the experienced developers. Just by looking at the task manager it is impossible to find memory leaks. We can relatively easy spot memory leaks with the debug version of SmartFTP but to do this we need a reproducible case. The fact that 139k items use 900MB of memory is no reason to believe that there is a memory leak.

Regards,
Mat

OK, what do you suggeset we do to make this program not eat up 3GB or RAM, and max out my CPU?

Let's say it takes 900mb to load a 138MB file containg 139K items. How can the memory usage grow 300% more (over 3GB) when the total files in the folder are only 182k (or 32% more).

Also, is it necessary to show the user that many files in the listview? When I created a program with a listview that looked like SmartFTP, it used added about 14 MB for every 10k items in the list (around 200MB for 140k files) of RAM utilization. If there were a setting to limit the amount of items to display in the listview it could save a decent amount of memory. When I watch the memory util of smartFTP, I see the memory jump when a large number of items are added, but when the item count drops considerably, the memory usage drops about 10% of the increase. So if the queue jumps from 1k to 8k items and it uses 60MB more RAM and then the queue drops back down to 1k, i'm only seeing like 6MB freed. My assumption is that the times I do see the memory usage go down it's because momory is being freed by the listview being smaller, however whatever arrays/collections/objects used by the program are not decreasing in size or being destroyed, or maybe it's building a string or XML file in memory for a log and not writing it to disk to free memory.

EDIT: I just now saw memory drop about 200MB so maybe it does actually flush some stuff out very seldomly... This was around the time that quite a few differences in the FTP site and the local site were found and it actually started uploading stuff.... Maybe there are some checks to free up some memory that only fires when a file is uploaded as apposed to checking hashes and not uploading since they are the same? There still has to be a way to work with a very large queue w/o using 3GB or RAM.

I think your 182'000 items is a very extreme case. The number of items displayed in the transfer queue do not significantly increase the memory usage. But all items loaded - regardless if they are displayed or not - use a certain amount of memory. Also the log for each item uses memory as well. This probably explains the reason why the memory increases when the transfer queue is running. The memory usage has been optimized for normal use and not for extreme cases like yours.
I recommend you optimize your process to reduce the amount of total transfer queue items (do you really need 182k items in the transfer queue?). If your business process requires you to have 182K items in the transfer queue you should probably have no problems to justify the increase of the total memory in your computer to 32GB. Minimizing the application window to reduce the GUI updates should also decrease the CPU usage.

Regards,
Mat

I think your 182'000 items is a very extreme case. The number of items displayed in the transfer queue do not significantly increase the memory usage. But all items loaded - regardless if they are displayed or not - use a certain amount of memory. Also the log for each item uses memory as well. The memory usage has been optimized for normal use and not for extreme cases like yours.
I recommend you optimize your process to reduce the amount of total transfer queue items (do you really need 182k items in the transfer queue?). If your business process requires you to have 182K items in the transfer queue you should probably have no problems to justify the increase of the total memory in your computer to 32GB. Minimizing the application window to reduce the GUI updates should also decrease the CPU usage.

Regards,
Mat

All I did was add a folder to monitor and told it to process, the items did not get added right away, I came in the next day and they were there, not sure why. If I try to process the folder again it seems to do ok for now, it peeked at 330MB RAM. My FTP server did auto-ban me for too many connections in a time frame overnight so maybe SmartFTP kept adding to a que? I tried shutting down the FTP service while it was running and SmartFTP stopped adding items to the list... I also tried banning myself so that it could connect to the FTP server but get booted like it would have during the auto-ban but SmartFTP still didn't add more items to the list, so I'm not sure if the auto-ban caused an issue. I'm not sure how the items got so high overnight, maybe it's a glitch that won't happen again, I'll have to see how it goes tonight.

This has nothing to do with the garbage collector (C++ does not use a GC per se). Basically when you minimize a window, the Windows memory management is doing some magic but it does not free the memory. The values you see in the task manager are not exactly valid so do not trust them.

If you would like to know more about memory management, Windows development I recommend to read a couple of papers and books and do some active development.

Regards,
Mat

OK, so the working set is not so useful... It still looks like there is a problem. The following is what smartFTP was using after everything was finished and the list was empty and there were no more connections to the FTP server:

846 MB (Commited - Private Bytes)
1,563 MB (Reserved - Virtual Size)

I closed the local/remote/transfer queue windows as well but didn't help.

The private bytes looks excessive.

Please try the latest version:
https://www.smartftp.com/download

The memory leaks have been fixed.

Regards,
Mat