New Posts New Posts RSS Feed: Memory leak
  FAQ FAQ  Forum Search   Calendar   Register Register  Login Login

Memory leak

 Post Reply Post Reply
Author
Niels Verkaart View Drop Down
Newbie
Newbie
Avatar

Joined: 05-Aug-2009
Location: Netherlands
Posts: 9
Post Options Post Options   Quote Niels Verkaart Quote  Post ReplyReply Direct Link To This Post Topic: Memory leak
    Posted: 06-Aug-2010 at 6:30am
Hello

This last year I've been trying to get DevForce 2010 and Silverlight 4 to work for me, which I bought a year ago (Devforce Universal). But everytime I stumble upon things that don't work for me. Learning moments! Beside some bugs in Silverlight 4 for which work arounds have been presented, I'm now stuck al together with this:

When ever I bind a datagrid or listbox via a ViewModel to an ObservableCollection which has been build up by the TypeHelper.CreateObservableCollection(results) in a Callback method from a simple query (for example query=Manager.Artikel) and I refresh the records with a new call to the query, I end up consuming memory which will never be released, even not after disposing of the EntityManger, View, Viewmodel, etc. This behavior can be reproduced in your Prism Explorer when querying Customers, Orders, and back a few times. You can see this in Performance Monitor (perfmon) under process->private bytes  where the count keeps climbing up.

At first I (about a month ago) thought that had to do with 'the inline DataTemplate bug', but I have removed al these instances and it still exists.

Could someone please help me?
thanks
Niels


Niels Verkaart
Back to Top
WardBell View Drop Down
IdeaBlade
IdeaBlade
Avatar

Joined: 31-Mar-2009
Location: Emeryville, CA,
Posts: 338
Post Options Post Options   Quote WardBell Quote  Post ReplyReply Direct Link To This Post Posted: 06-Aug-2010 at 3:54pm
You are (mostly) correct and oh ... how awful!
 
As I demonstrate below, memory is recovered when you clear the EntityManager. It would be recovered if you disposed of the EntityManager itself.  I'm not sure why you didn't see these effects. 

However, you are absolutely right that something horrible is happening. I spent much of the day researching it.
 
The only good part of the story is that it has nothing to do with DevForce. The cause is something in Silverlight, probably in the Silverlight DataGrid.
 
I will explain but it will be a bit of a journey leading ... ultimately ... to a sad-but-necessary workaround.
 
If you want to skip the journey and go right to the destination, the lesson for your code is:
 
Don't reassign the ObservableCollection<T> bound to an ItemsSource; repopulate it!
 
And now ... my sad tale ...
 
The main "ModelExplorer" view binds a Silverlight DataGrid.ItemsSource to the QueryResults property of the supporting ModelExplorerViewModel (the VM).
 
"QueryResults" is defined as IList but is always an ObservableCollection<T> in practice (henceforth written as "OC<T>").
 
The problem starts with the repeated resetting of the QueryResults in the ModelExplorerViewModel after every query.
 
A peculiarity of PrismExplorer (PE) is that the main "ModelExplorer" view could be asked to display objects of any type in the DataGrid. That's its purpose as a demo screen. Your grids typically only show one kind of object. But PE must work with arbitrary query result types.
 
Because those types change, the OC<T> must change to match type.
 
The obvious thing to do was craft a new OC<T> after every query ... one designed for the query results type, assign that new OC to the QueryResults, and raise PropertyChanged on "QueryResults". The DataGrid dutifully hears the call and refreshes its display.
 
Unfortunately, it consumes an extra 300K every single time. "300K" is not a misprint.
 
Here's the code with the garbage collection call and logging (aside: we now inject ILoggerFacade into the VM now so that we can write to the Visual Studio Output window).
    public IList QueryResults {
      get { return _queryResults; }
      set {
        _queryResults = value;

        Log("=== Total Memory = " + GC.GetTotalMemory(true));

        RaisePropertyChanged("QueryResults""QueryResultsCount");
        SelectFirstQueryResultsItem();
      }
    }
If you add this instrumentation, query for "All Customers", and click the "Query" button repeatedly, the Output window will show that your memory consumption is climbing ~300K each time.  Here are actual measurement.
 
  Prism Explorer: "=== Total Memory = 3406212";
  Prism Explorer: "=== Total Memory = 3914916";
  Prism Explorer: "=== Total Memory = 4336668";
  Prism Explorer: "=== Total Memory = 4756884";
  Prism Explorer: "=== Total Memory = 5177356";
  Prism Explorer: "=== Total Memory = 5782900";
  Prism Explorer: "=== Total Memory = 6202860";
  Prism Explorer: "=== Total Memory = 6616220";
  // Cleared the EntityManager
  Prism Explorer: "=== Total Memory = 3655320";
  Prism Explorer: "=== Total Memory = 4076292";
  Prism Explorer: "=== Total Memory = 4498300";
  Prism Explorer: "=== Total Memory = 4918260";
  Prism Explorer: "=== Total Memory = 5358912";
  // Cleared the EntityManager
  Prism Explorer: "=== Total Memory = 3655384";
 
Note that DevForce isn't going anywhere. You are issuing the same query repeatedly and DevForce satisfies it from the EntityManager cache. However, the PE ViewModel is building a new OC<Customer> each time and reassigning QueryResults. The DataGrid is rebinding the ItemsSource to the new OC<Customer> each time ... and chewing up memory each time.
 
If we comment out "_queryResult = value", DevForce and PE will continue to do their things ...  DevForce may query the database or satisfy from cache if it can. PE will always build a new OC<T>, assign QueryResults with that OC<T>, raise PropertyChanged, stimulate the DataGrid to rebind to the QueryResults. Exactly the same code paths.
 
The only programmatic difference is that QueryResults always returns the exact same (empty) object.
 
The memory consumption goes flat ... as it should.
 
Of course it climbs everytime you issue a query that fetches new data; that is to be expected. Press "Clear" to clear the EntityManager cache and memory is recovered.
 
This is proof enough for me that the memory leak is in Silverlight ... perhaps in the Silverlight DataGrid. Our entities are involved in some way ... that's why clearing the EntityManager cache allows the GC to recover memory. But the DataGrid is clearly chewing up memory at a fast clip without our adding any new entities to memory.
 
What could possibly take 300k? The entity data involved are miniscule (it's Northwind!  7 Employees ~90 customers). All entities combined do not weigh 300K.
 
The memory consumption grows by 300K when there are no new entities. Evidently Silverlight or OC is in a deadly embrace with the entities that prevents garbage collection (there are events involved). But even if so ... the numbers are staggering. We aren't adding 300K of new data each time we click the button. You can't write enough event handlers to consume 300K. The cost of an OC isn't 300K. 
 
In my research, I tripped over an old WPF blog post (can't find it now) that said something about WPF creating completely new data and control templates for each OC. Now THAT is a fast way to burn memory. I'm betting that Silverlight is creating and caching a complete visual representation of the DataGrid and the associated entities each time I reset the ItemsSource with a new OC<T>!
 
I tried a number of things to shake up the binding ... anything to make it let go of whatever it was holding:
  • clear the _queryResults ( _queryResults.Clear() ) before re-assigning it with the new OC
  • clear the _queryResults and set it null before re-assigning it with the new OC
  • clear the _queryResults, set it null, raise PropertyChanged ... then assign with new OC
None of these attempts worked.  Only pressing "Clear" worked.
 
What to do? The obvious answer is reuse the OC ... and that is the lesson for your application: Don't reassign the ItemsSource; repopulate it!
 
Of course OC<T> requires strongly typed contents. I tried OC<object> with the intention of simply clearing and refilling it with new results after each query. That executes without error. But it doesn't show anything on screen either ... pretty useless :-).
 
It actually worked if I use List<T> instead of OC<T>. But then I wouldn't have the benefits of ICollectionPropertyChanged and I'd have to manage adds, removals, clears by hand. Yuck.
 
The WorkAround
 
As a last resort, on each pass I check to see if the type of the fresh results matches the element type of the OC. If it does, I can re-populate the OC. If it does not match, I have to replace the ItemsSource with the new OC ... and pay the 300K tax.
 
If you always query for the same type, you'll be fine. You can change the query itself - there are numerous Customer and Employee queries in PE - without paying the tax. It's changing the result type that incurs the tax.
 
I tried to be clever and keep a dictionary of previously used OC types. If I was querying customers, I'd pull out the OC<Customer>; if I was querying employees, I'd pull out the OC<Employee>.  That didn't help. The DataGrid doesn't remember OCs that it has seen before. Re-using old OCs didn't change its penchant for devouring another 300K 
 
So all I can do is slow it down. If PE starts to blow, you can "Clear" the EntityManager and it will recover.
 
Here's the revised code (which will appear in the next PE release, minus the GC & Log calls)
    public IList QueryResults {
      get { return _queryResults; }
      set {

        ResetQueryResults(value);

        Log("=== Total Memory = " + GC.GetTotalMemory(true));

        RaisePropertyChanged("QueryResults""QueryResultsCount");
        SelectFirstQueryResultsItem();
      }
    }


    private void ResetQueryResults(IList results) {

      if (null != _queryResults) _queryResults.Clear();
      if (null == results || 0 == results.Count) return;

      var resultsType = TypeHelper.GetItemType(results);

      if (_queryResultsElementType == resultsType) {
        // Same results collection type; can refill it
        foreach (var item in results) {
          _queryResults.Add(item);
        }
      } else {
        _queryResults = results;
        _queryResultsElementType = resultsType;
      }
    }

    private Type _queryResultsElementType;
 Here are memory reports:
 
 // Query for All Customers
 Prism Explorer: "=== Total Memory = 3411896"
 Prism Explorer: "=== Total Memory = 3425356"
 Prism Explorer: "=== Total Memory = 3416064"
 Prism Explorer: "=== Total Memory = 3416076"
 Prism Explorer: "=== Total Memory = 3416076"
 Prism Explorer: "=== Total Memory = 3429460"
 Prism Explorer: "=== Total Memory = 3416076"
 Prism Explorer: "=== Total Memory = 3415020"
 // All Employees
 Prism Explorer: "=== Total Memory = 4203344"
 Prism Explorer: "=== Total Memory = 4212964"
 Prism Explorer: "=== Total Memory = 4213000"
 Prism Explorer: "=== Total Memory = 4213000"
 Prism Explorer: "=== Total Memory = 4213000"
 // Clear Entity Cache ... followed by All Employees
 Prism Explorer: "=== Total Memory = 3706416"
 Prism Explorer: "=== Total Memory = 3588660"
 Prism Explorer: "=== Total Memory = 3593996"
 // First Employee
 Prism Explorer: "=== Total Memory = 3590928"
 Prism Explorer: "=== Total Memory = 3604372"
 Prism Explorer: "=== Total Memory = 3589676"
 // Employees named Smith (there are none)
 Prism Explorer: "=== Total Memory = 3605584"
 // All Employees (again)
 Prism Explorer: "=== Total Memory = 3757472"
 Prism Explorer: "=== Total Memory = 3762120"
 Prism Explorer: "=== Total Memory = 3770336"
 Prism Explorer: "=== Total Memory = 3762144"


Edited by WardBell - 06-Aug-2010 at 4:08pm
Back to Top
WardBell View Drop Down
IdeaBlade
IdeaBlade
Avatar

Joined: 31-Mar-2009
Location: Emeryville, CA,
Posts: 338
Post Options Post Options   Quote WardBell Quote  Post ReplyReply Direct Link To This Post Posted: 09-Aug-2010 at 12:31pm
I posted this story on my blog. One of my readers found a link to a Silverlight Forum post that sheds light on the situation and affords hope for a MS fix in the near future. Stay tuned:
 
---
Thanks, Anonymous, ... nice find!

That link in the SilverlightForum (http://forums.silverlight.net/forums/p/171739/387384.aspx#387384 ) aligns well with the scenario presented here. The autogenerating DataGrid is likely creating DataTemplates on the fly ("inline") and triggering this unwanted defect.

Looks like the root-cause will affect all controls, including 3rd party controls.

Tim Heuer writes in his July 30 comments "Right now we have a plan [for releasing a service pack that includes a fix to the DataTemplate leak problem] and are driving toward that plan which I've mentioned here as [expected] before end of Summer."

We'll just have to be patient.


Edited by WardBell - 10-Aug-2010 at 2:26am
Back to Top
Niels Verkaart View Drop Down
Newbie
Newbie
Avatar

Joined: 05-Aug-2009
Location: Netherlands
Posts: 9
Post Options Post Options   Quote Niels Verkaart Quote  Post ReplyReply Direct Link To This Post Posted: 10-Aug-2010 at 2:03am
Thank you for your outstanding reply Ward.
 
It kept me busy for weeks and exhausted me quite a bit, for I'm new to most of the technologies. However it still is very promising to me especially with the (more than) starter kit 'Prism Explorer' is. For now I'll try this temporary work around (also for learning excitement) and maybe I'll focus on getting PRISM 4 RTC to work for me.
 
The forum the anonymous poster mentioned was my source for the DataTemplate issue, if I had know this was the source for the problem I presented I would have given you the link, sorry. (your link 'link in the silverlightForum' in this post misses this at the end: .aspx#387384)
 
I'll keep a close I on Tim Heuer's twitter.
 
thanks and best regards,
Niels
 
Niels Verkaart
Back to Top
WardBell View Drop Down
IdeaBlade
IdeaBlade
Avatar

Joined: 31-Mar-2009
Location: Emeryville, CA,
Posts: 338
Post Options Post Options   Quote WardBell Quote  Post ReplyReply Direct Link To This Post Posted: 10-Aug-2010 at 2:27am
Thanks, Niels. I repaired the link ... as you suggested.
Back to Top
Niels Verkaart View Drop Down
Newbie
Newbie
Avatar

Joined: 05-Aug-2009
Location: Netherlands
Posts: 9
Post Options Post Options   Quote Niels Verkaart Quote  Post ReplyReply Direct Link To This Post Posted: 10-Aug-2010 at 6:37am
Hi Ward,

I've got things working now in my viewmodel. I just newed up an OC<T> up front because indeed I dont really need variable queries at this moment and then add/clear items on the way. 'All' my viewmodels are fixed in a Unit of Work or just used in a general screen like an Article screen or Relationship/party screen.

I can now create MDI view/model in a tabbed way almost (!) without memory loss (there must always something to figure out :) )

Learned from experience (most obvious for most I guess) and might not at all be accurate:
- clean up after use, implement IDisposable on ViewModel, ScreenFactory, Repository, Gateway
- unsubscribe events from EventAggregator
- unregister/remove from eventhandlers (like DisplayError, BusyChanged from the PE example)
- unregister/remove from eventhandler instantiated in XAML of the View (i.e. in XAML:

... Loaded="PageLoaded" ...

then in code behind I immediately do: 

        private void PageLoaded(object sender, RoutedEventArgs e)
        {
            Loaded -= PageLoaded; 
            ...
        }

otherwise these objects would not unload or can leak memory. 

I don't know if these things could help anyone, but I'd like to suggest creating some room on your new Wiki with this kind of 'knowledge'/experience as some sort of 'things to read when working with DevForce, Silverlight (and MVVM)'? And create a subsection in your Forum with other users suggestions that might be added in a wiki way. Although on your forum much is discussed, they still are discussions, some with an open end, and a bit unstructured (not the sections themselves of course).

ah well, babbling too much, so much work to do.

Niels Verkaart
Back to Top
WardBell View Drop Down
IdeaBlade
IdeaBlade
Avatar

Joined: 31-Mar-2009
Location: Emeryville, CA,
Posts: 338
Post Options Post Options   Quote WardBell Quote  Post ReplyReply Direct Link To This Post Posted: 10-Aug-2010 at 11:10am
Thanks, Niels. There is much good advice here. Some of it is what we call "belt-and-suspenders" ... you take extra steps to cover yourself (to hold your pants up) even though such steps should not be necessary in theory.
 
Unhooking from EventAggregator, for example, shouldn't be necessary unless you explicitly defeated its default weakly referenced handlers.
 
Rarely hurts to be careful.
 
We are eager to open a "Good Practices" section of the Wiki ... and your findings would be welcome there when we do.
 
Happy coding!


Edited by WardBell - 10-Aug-2010 at 11:12am
Back to Top
Niels Verkaart View Drop Down
Newbie
Newbie
Avatar

Joined: 05-Aug-2009
Location: Netherlands
Posts: 9
Post Options Post Options   Quote Niels Verkaart Quote  Post ReplyReply Direct Link To This Post Posted: 07-Sep-2010 at 5:41am
Hi all,

The DataTemplate bug has been fixed by the Silverlight Team in the september 2010 service release:


And I can confirm :-)

Best wishes,
Niels
Back to Top
 Post Reply Post Reply

Forum Jump Forum Permissions View Drop Down