Memory Leak – Microsoft.Reporting.WinForms.LocalReport

Several weeks ago I was asked to assist with a sluggish ASP.NET C# MVC application. This application is an OLTP system and processes several thousand orders a day. It was hosted on a Windows 2012 R2 64bit 16 Core machine. After discussing with the developer, I was told that the application has acceptable levels of performance for a while and then becomes sluggish (this behavior suggests some kind of  a leak). At times, it becomes extremely slow and rebooting the machine solves the problem. The problem exhibits itself more frequently during days when there were above average volumes.

Downloading Process Explorer from SysInternals, I found the w3wp.exe (there were several, but 2 of them stood out). Process Explorer does not display handle information by default. I will probably blog about this separately. There are many articles out there that explain how to look for Handles using Process Explorer. This should get you started. Here is another excellent blog explaining the theoretical limit of handles. It’s a theoretical limit because there are many types of thresholds that an application can reach which causes severe performance degradation.

I knew that we had a potential handle leak. There were also other issues within the application, that were not related to the handle leak. The challenge was to isolate the handle leak versus defects in the program logic, 3rd party hardware behaviors (like non genuine printer cartridges) and various other user errors.  Debugging a live system with several 100 users connected wasn’t happening. We were able to get a 4.5Gb memory dump.

When I moved the memory dump to a machine where I could analyze it, I found that the Production server was running an older version of the .NET Framework. The MS Symbol server wasn’t much help. I ended up copying the complete framework64 folder to the local machine (a different folder, not the Microsoft.NET\Framework folder) and loaded SOS using the .load command.

01-load sos
Using the time command, the overall CPU utilization by the Process compared to the System uptime showed that it wasn’t CPU contention.

02-time

The dumpheap command displays the managed heap. The output contains the Method Table, Object Instance Count, Memory Allocated and the Type from left to right. !dumpheap -stat reported about 164k lines of output and 9.7M object references. Many of these objects pointed to dynamically created Report Expressions.

01-dumpheap -stat

Typically the next logical step is to use !dumpheap -mt <<method table address>> to display all the individual object instances and then reviewing the individual objects to understand them better.

02-dumpheap -stat

Looking for loaded modules using the !eeheap -loader command showed that the worker process had 2622 App Domains (with a heapsize of 537mb). Dumping one of the modules using !DumpModule command pointed to a Dynamically loaded Expression Library associated with SQL Server Reporting Services.

05-eeheap -loader06-eeheap -loader

At this point, I knew that the issue may be related to the Dynamically Loaded Expression Assembly and its associated AppDomain that failed to unload. What was keeping the AppDomain from unloading? The dynamic expression library is used by the LocalReport object, which is part of the Microsoft.Reporting.Winforms namespace.

Further research showed that creating a new AppDomain is per design. The developers @ Microsoft had identified an issue with the dynamic expression library being loaded into the Default AppDomain and using up more memory each time the LocalReport object was created. An assembly once loaded into an AppDomain can never be unloaded. Hence the dynamic assembly was loaded in a new AppDomain and if all goes well, the AppDomain can be safely unloaded. As per the documentation calling LocalReport.ReleaseSandBoxAppDomain should unload the AppDomain. Unfortunately, the unload never happens because there is a problem with the internal implementation of the Dispose Method.

Using .NET Memory Profiler (JetBrains dotPeek), I was able to identify the Event Delegate

Screen Shot 2017-04-11 at 11.13.42 AM

Looking at the implementation, the handlers are not removed in the Dispose method. This in turn keeps the AppDomain from unloading. Hope this gets resolved in a future release. For now, we are recycling the Application Pool more frequently to work around the memory leak and the developers are looking at an alternate solution to print the labels.

08

One of the questions that I was asked while presenting my findings with the project manager, is why did this suddenly become a problem, since the application had been in Production for over 2 years. The issue had been there from the very beginning. The OOTB Application Pool Recycle was  causing the host process to shutdown after a period of inactivity (this app was used only between 8am-5pm). So it had enough time to shutdown after hours and the problem would never surface until the transaction volume increased, at which point the application started hitting thresholds.

It always helps to set baselines. Keep tabs on System configuration, Transaction Volume and major system change events. This will greatly shorten the time it takes to troubleshoot performance issues.

 

 

Advertisements

One Response to Memory Leak – Microsoft.Reporting.WinForms.LocalReport

  1. Stoitcho Goutsev says:

    Hi Trevor,

    I was recently looking into a code that was supposedly having a memory leak caused by a “leaky” LocalReport object. I was also directed to your article that is providing in-depth analysis of the problem.

    As long as everything in your article is technically correct IMHO what you suggest can’t be a cause for a leak.
    Initially I wrote a rather lengthy explanation of my findings but it seems to be a too long for a comment. So I will give here just the gist of it:
    1. No references can prevent an app domain from unloading. If AppDomain.Unload methods is called the domain is as good as gone. There just couple if things that can get in the way one of them being a running thread. But in this case you must get a CannotUnloadAppDomainException exception which a debugger tool should diagnose easily.
    2. LocalReport.Dispose method does eventually call AppDomain.Unload, thus the domain almost certainly will go away.
    3. If nothing special is done LocalReport object will linger in memory alive up to 7 min after the last reference to the object is removed. This is caused by the .NET Remoting platform. This could be changed but it is most likely unnecessary. Eventually the object will become garbage. This, I believe, is what causes confusion while using a memory profiler.

    So the question is “Does the LocalReport actually leak memory?”.

    My answer is NO, but the class is definitely not foolproof. However following few simple steps one can prevent memory leaks:
    1. Call LocalReport.Dispose() as soon the report object is no longer needed.
    2. When adding data sources to LocalReport.DataSources the code force you to wrap the actual data source in a ReportDataSource object. Never keep long term references to these wrapper objects. This will keep the whole report object alive. The event hooking code that you found is related to this.
    3. If one can’t wait those 7-ish min, one can call LoclaReport.DataSources.Clear() right before or after calling Dispose. This will at least release any DataSet or whatever data source objects. Those objects could carry some weight.

    Microsoft could’ve done better job and saved some internet bandwidth by simply adding a finalizer method that calls the Dispose.
    Also IMHO those data source wrapper objects are kind of bad design on their part.

    Just as a little disclaimer – I am not an expert in ASP.NET and know almost nothing about web page’s life cycle and such. There might be some intricacies that may cause an actual memory leak. Also there might be other “bad” scenarios that I didn’t come across while working on our particular problem.

    In your case however if you your code is app domains I think this is most likely caused by your code failing to call Dispose.

    Regards,
    Stoitcho Goutsev

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: