One of the file servers running Windows Server 2008 R2 encountered a problem of high RAM load resulting in the issues with the server and applications performance. It turned out that the memory was swamped by the system file cache containing file system metadata. The problem potentially affects all file servers with large numbers of files that are accessed by users. It is the most critical for x64 Windows versions, in which the size of the metadata file in the memory can enlarge almost to the whole size of RAM.
High RAM Load on Windows File Server
The problem manifests itself as follows: in the Task Manager we see that physical memory is busy by 95-99%.
In the Processes tab, there is no any leaked process with abnormally high memory consumption. Moreover, if you sum the approximate values of memory used by all processes you won’t even get 50% of physical memory you have on your server. What is eating the memory then?
The actual data on RAM usage can be obtained with the help of small utility – RAMMap (by Mark Russinovich). Download the archive containing the tool and run RAMMap.exe with the administrator privileges. In the Use Counts tab, we see that Metafile is using the largest amount of RAM. (In our case, it is using 11 from 25 GB of the server RAM).
What is a Metafile in Windows?
A metafile is a part of the system cache containing NTFS metadata and used to increase the performance of the file system when accessing files. NTFS metadata include the data of MFT (Master File Table). For each file or folder, accessed by the users, a corresponding block of at least 1 KB (the record of an attribute of each file is 1 KB, and each file has at least one attribute) is created in the metafile. Thus, on file servers with a large number of files, the metafile size (NTFS cache) may exceed several tens of gigabytes.
It is impossible to disable this cache or manage it using built-in Windows tools. As a solution, you can increase the size of the memory on the server, but it is not always possible.
To free up memory, you can restart the server, but in some time the size of the metafile in the memory starts growing incredibly.
For example, you can estimate the size of the MFT using another tool by Russinovich – ntfsinfo. In our case, the size of the MFT on a 2 TB disk is 13 GB.
How to Quickly Clean Up metafile
RAMMap allows to quickly clear the used memory from MFT garbage without server restart. To do it, select Empty -> Empty System Working Set in the menu.
After that, the size of the metafile in the memory reduced dozens of times, and the percentage of RAM use by CPU dropped from 95% to 26%.
The main disadvantage of this method is that the clearing is manual and cannot be done automatically.
Dynamic Cache Service to Manage the File Cache
Another, more cardinal, solution of high memory load by the file system metafile is the installation of Dynamic Cache Service (http://www.microsoft.com/en-us/download/details.aspx?id=9258). This service allows to manage the parameters of the dedicated MFT cache using system APIs.
It is rather easy to install DynCache (there are detailed instructions in the archive).
- Copy the file DynCache.exe to %SystemRoot%\System32 folder
- Create DynCache service using this command:
sc create DynCache binpath= %SystemRoot%\System32\DynCache.exe start= auto type= own DisplayName= "Dynamic Cache Service"
- Import DynCache.reg to the registry (it contains default values)
- Change the values of the following register keys: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\DynCache\Parameters
- MaxSystemCacheMBytes: 4096 (dec) – the maximum cache size(Mb)
- MinSystemCacheMBytes: 100 (dec) – the minimum cache size (MB)
Note. These and other DynCache service settings have to be edited according to the RAM size, the server load, required performance, etc. As a rule, it is not recommended to set the cache size more than half of the physical RAM installed on the server. After the changes are made, you don’t need to restart DynCache, since all changes are applied dynamically. - Run the service using this command:
sc start DynCache
In our case, after DynCache service had been installed, the use of memory by the metafile stopped getting over 4 GB we had set. The users have not reported any performance issues on the file server.
1 comment
Followed the instructions (there are multiple DynCache.exe in the extracted files so this could be the issue) but I get a 193 error when trying to start the service. Doesn’t work.