DevForce Community Forum : Performance issues with ImportEntities

Performance issues with ImportEntities : Thanks, Stephen. Thanks too...

Wed, 06 Mar 2013 07:50:10 -700

Author: kimj
Subject: 3978
Posted: 06-Mar-2013 at 7:50am

Thanks, Stephen. Thanks too for adding this issue to User Voice.

Performance issues with ImportEntities : You can find the code I use here....

Tue, 05 Mar 2013 13:42:43 -700

Author: stephenmcd1
Subject: 3978
Posted: 05-Mar-2013 at 1:42pm

You can find the code I use here. As I say on that page, it's certainly not the most robust form of serialization. But it's ended up working well for us.

Performance issues with ImportEntities : That's interesting about...

Thu, 07 Feb 2013 11:27:03 -700

Author: kimj
Subject: 3978
Posted: 07-Feb-2013 at 11:27am

That's interesting about the ECS deserialization issue with large graphs. Would you mind sharing more information on your custom serialization/deserialization logic, if not here in a PM or email?

Performance issues with ImportEntities : Thanks for the thorough response....

Thu, 07 Feb 2013 10:39:02 -700

Author: stephenmcd1
Subject: 3978
Posted: 07-Feb-2013 at 10:39am

Thanks for the thorough response. Well, it's probably clear but I vote for the deep cloning logic to be revisited :-). But I'm sure it's not exactly the hottest new feature that most people would care about.

I think we'll switch over some of our logic to use the fast way. Hopefully will find any problems that might come up. :-). Luckily, I don't think tempids, validation errors, navigation properties, etc. will come into play in the cases we'll be using this.

If we do run into problems with our method, we might go with your suggestion about saving a pre-serialized EntityCacheState. That could be a good middle ground.

On a related note, we've tried saving EntityCacheStates to isolated storage as a way of caching large amounts of data across restarts of the app. Unfortunately, we found the performance of deserializing the data was not that great and so ended up writing our own logic to serialize and deserialize entities to isolated storage. We were trying to save a very deep/complex graph of entities so perhaps that is a case where the standard EntityCacheState stuff suffers and some custom logic can help.

Performance issues with ImportEntities : Deep cloning using the DataContractSerializer...

Wed, 06 Feb 2013 18:15:02 -700

Author: kimj
Subject: 3978
Posted: 06-Feb-2013 at 6:15pm

Deep cloning using the DataContractSerializer has not done us any favors, as it's the cause of several other performance issues, notably a 2-tier save problem.

Aside from the deep clone, ImportEntities is also responsible for temp ID fixup, merging with entities already in cache, fixing up navigation properties, re-validating entities with existing errors, and generally ensuring that the state after the import is correct and consistent. Whether these activities are needed for all entities all the time, particularly if their state is Unchanged, is hard to say. I'm hesitant to say that the fast cloning solution will always be safe, but if your testing shows it's reliable and you haven't introduced new and obscure errors, then it probably is safe for your needs.

We need to look at when and why we do a deep clone to determine when it's really necessary and worth the cost. We also need to look at using something other than the DCS to perform the deep clone. These items our on our defect list, but we won't have an answer for them soon.

One alternative you might consider, which would be more appropriate for unchanging data and may not suit your requirements, is to store an EntityCacheState in either a MemoryStream or isolated storage, and use the em.CacheStateManager.RestoreCacheState overload which accepts a stream to load a destination EntityManager as needed. This would about halve the DCS time of the deep clone, but would still take care of the other merge logic done by ImportEntities.

Performance issues with ImportEntities : We cache a lot of data in our...

Tue, 05 Feb 2013 14:13:32 -700

Author: stephenmcd1
Subject: 3978
Posted: 05-Feb-2013 at 2:13pm

We cache a lot of data in our Silverlight client. One of the main ways we accomplish this is by having an Entity Manager that has a bunch of commonly used entities. Then when somebody asks for data, we check that Entity Manager and use those entities if possible. However, in our case it is not safe to just return the same copy of the cached entities to all the callers - we need to give them copies. We do this by importing the cached entities into the EntityManager of the code asking for the data (we have one Entity Manager for each window and sometimes even for sections of the window that need isolation). Also, most of our code assumes that entities will be owned by an entity manager so it's not really safe to give back free-floating entities.

We've been doing some performance testing of our app and noticed that the ImportEntities logic is actually starting to be a bottleneck. I did a bunch of timings and it seems like a simple ImportEntities from a nearly empty entity manager into an empty entity manager will always take a certain, non-trivial amount of time. On my machine, I can import from one to twenty entities at a time and it will always take about 60 milliseconds. It's not until I start importing like 40 entities at the same time that the time starts to increase (40 takes like 66ms). If you're doing big imports, that extra 60ms is probably not a big deal....but in our case, we are sometimes doing dozens of tiny imports of just one to four entities at a time. And those 60ms start to add up.

I looked at how ImportEntities works and it seems like it makes a copy of the EntityCacheState using the DataContractSerializer to basically do a deep clone. In all my profiling, that is the logic that seems to be taking the longest time in various operations in our app. I'm not sure if there is anything that can be done. Perhaps many small imports is not really the intended use of that method? But if there could be some way to speed that up, I would very much appreciate it....

In trying to find alternate ways for quickly copying entities from one Entity Manager to another, I've come across the following solution and wanted to get your opinion. Basically, I check whether any of the entities I need to copy have changes or exist in the destination Entity Manager. In that case, it seems complicated so I just fall back to Import Entities. Luckily, in our code that rarely happens. Most of the time, we are moving unchanged entities into an empty (or mostly empty) Entity Manager. In that case, I just clone the entities and attach them. My code looks like this:

private static void QuickImport(IList toImport, EntityManager destinationManager, MergeStrategy mergeStrategy){
    //If any of the passed in entities have pending changes or exist in the destination entity manager, use the 
    //   standard (slow) Import Entities logic.  Otherwise, we can just clone each entity and attach it to the destination
    //   for better performance.

    if (toImport.Any(e =>
                        e.EntityAspect.EntityState != EntityState.Unchanged ||
                        destinationManager.FindEntity(e.EntityAspect.EntityKey, includeDeleted: true) != null))
    {
        destinationManager.ImportEntities(toImport, mergeStrategy);
    }
    else
    {
        destinationManager.AttachEntities(toImport.Select(e => ((ICloneable)e).Clone()));
    }
}

Does that seem like a good and safe solution? I tried to restrict it to the cases that I assume don't require any complicated import/merging logic.

For reference, here are some numbers on my timings comparing how long imports take for various numbers of entities and also the average time per entity for doing the merge. You'll notice that the ImportEntities logic takes a long time even with just a handful of entities.

Thanks!