New Posts New Posts RSS Feed: CompositePredicateDescription times out and gets progressively slower when combining more than 15 or so PredicateDescriptions
  FAQ FAQ  Forum Search   Calendar   Register Register  Login Login

CompositePredicateDescription times out and gets progressively slower when combining more than 15 or so PredicateDescriptions

 Post Reply Post Reply
Author
pk55 View Drop Down
Senior Member
Senior Member


Joined: 22-Jul-2009
Location: CA
Posts: 105
Post Options Post Options   Quote pk55 Quote  Post ReplyReply Direct Link To This Post Topic: CompositePredicateDescription times out and gets progressively slower when combining more than 15 or so PredicateDescriptions
    Posted: 27-Jul-2011 at 10:07pm

Tyring to OR more than 10-12 single predicate descriptions for the same property one a time into a composite predicate description causes timeouts on the entity server and each time you OR a new predicate, it gets progressively slower, getting really bad as you get over 15 ORs.  The last call we see (before hitting the timeout) is to IdeaBlade.Linq.CompositePredicateDescription.InstanceType.get() line 85.

I've been able to use LinqPad against our generic EF datamodels with the standard EF PredicateBuilder to do the same thing and it can handle over 400 predicates ORed together (LinqPad blows up if I try 500) so this can't be an EF limitation.

The property in this case was an INT field (but this probably happens for any type) and each predicate was just using the filter for equality.  The code that creates these is very dynamic but essentially, after first creating a single predicate description, if it finds another predicate for the column, it builds a new predicate description and combines the first 2 single predicates into a composite.  Subsequent predicates on that same property uses the composite to combine it with an OR on each new predicate. 

In this particular case, I can't really use an InList filter operator without massive code restructuring to try and gather up all the related filter requests for a particular column as our filtering logic allows for some wildly dynamic conditions.  Besides, it seems that either DevForce or EF (can't tell who) is smart enough to convert multiple ORs into a single WHERE IN clause when it generates the SQL so there isn't any performance issue at the SQL server level.
 
I've created an example you can use to duplicate this against NorthwindIB. Try just setting a breakpoint on "c1 = c1.Or(p);" and watch how each loop gets slower until it times out:


                        // test for > 20 predicates ORed together
                        PredicateDescription p1 = null;
                        CompositePredicateDescription c1 = null;

                        for (int i = 0; i <= 20; i++)
                        {
                            var p = new PredicateDescription(typeof (Product), "SupplierID", FilterOperator.IsEqualTo, i);

                            switch (i)
                            {
                                case 0:
                                    p1 = p;
                                    break;

                                case 1:
                                    c1 = p1.Or(p);
                                    p1 = null;
                                    break;

                                default:
                                    c1 = c1.Or(p);
                                    break;
                            }
                        }


 
Back to Top
robertg View Drop Down
IdeaBlade
IdeaBlade
Avatar

Joined: 15-Mar-2011
Location: California
Posts: 87
Post Options Post Options   Quote robertg Quote  Post ReplyReply Direct Link To This Post Posted: 28-Jul-2011 at 4:25pm

I've tested this, and it does seem that adding additional predicates to the CompositePredicateDescription takes an increasingly long time once you've gone past a certain number of predicates. I'm not sure if this is a limitation of DevForce or EntityFramework, so I'll open a defect ticket for us to investigate it.

You might want to try using PredicateBuilder instead. I did a little bit of initial testing with it, and it looks like creating a chain of 20 predicates linked with OR does not take significantly longer than just 5 or so. Information on this is available in the DRC at http://drc.ideablade.com/xwiki/bin/view/Documentation/predicatebuilder-methods
Back to Top
pk55 View Drop Down
Senior Member
Senior Member


Joined: 22-Jul-2009
Location: CA
Posts: 105
Post Options Post Options   Quote pk55 Quote  Post ReplyReply Direct Link To This Post Posted: 28-Jul-2011 at 9:38pm
I doubt it's an EF thing since I can build 400 of them using the native EF predicate builder in under one second in LinqPad (this is against the linqpad demo db model):
 

var predicate = PredicateBuilder.False<Product>();

for (int i = 0; i <= 400; i++)

{

var item = i;

predicate = predicate.Or (p => p.ID == item);

}

Products.Where (predicate).Dump ();

Back to Top
robertg View Drop Down
IdeaBlade
IdeaBlade
Avatar

Joined: 15-Mar-2011
Location: California
Posts: 87
Post Options Post Options   Quote robertg Quote  Post ReplyReply Direct Link To This Post Posted: 04-Aug-2011 at 4:41pm
PK55,
 
In testing this, I found that there was a bug after all. One of our developers was able to rush a fix in to the new release, which came out yesterday. When you can, I recommend upgrading to the new version of DF (which requires you to regenerate your model, as usual) and take a look. The problem should be resolved.
 
Yours,
Robert
Back to Top
pk55 View Drop Down
Senior Member
Senior Member


Joined: 22-Jul-2009
Location: CA
Posts: 105
Post Options Post Options   Quote pk55 Quote  Post ReplyReply Direct Link To This Post Posted: 09-Aug-2011 at 5:10pm
We've upgraded and retested and this is fixed.  Thanks for getting that in.
Back to Top
 Post Reply Post Reply

Forum Jump Forum Permissions View Drop Down