Print Page | Close Window

Many Queries running at the same time

Printed From: IdeaBlade
Category: DevForce
Forum Name: DevForce 2009
Forum Discription: For .NET 3.5
URL: http://www.ideablade.com/forum/forum_posts.asp?TID=1953
Printed Date: 28-Mar-2024 at 5:37am


Topic: Many Queries running at the same time
Posted By: tj62
Subject: Many Queries running at the same time
Date Posted: 08-Jul-2010 at 5:38am
Hi, I have been from time to time creating a framwork using PRISM and DevForce.Silverlight. As you may know modules is one of the base consepts in PRISM.
When I load my PRISM modules each screenfactory has to retrieve some data from the database and sometimes some updates are needed in the module loading by each module.
 
My framework is based on the PRISM Explorer sample ( http://www.ideablade.com/DevforceSilverlight/DevForceSilverlight_PrismExplorer.aspx - http://www.ideablade.com/DevforceSilverlight/DevForceSilverlight_PrismExplorer.aspx )
If I use the PersistenceGateway class unchanged the module load code will hit exceptoin in the
GuardIsServiceBusy(string operationName) function as this happens.
 
  1. Module A sends an async query A.
  2. Before response is received from query A, Module B sends async query B and IsBusy returns true as the Persitence Gateway is busy with query A.

So my first try was to ignore the IsBusy property. First this seemed to work fine Modual A, B, C, D, E where running their queris simulatneously against the back-end asynchornously. But from time to time I get a "Repository persistence operation failed; The remote server returned an error: NotFound." exception.

Is it such that the front-end may only run a single async. query against the back-end at any time?
Or is the source for the "Not Found" exceptoin somethign else?
 
The things got worse when I added some updates via the PersistenceGateway class ignoring the IsBusy property. Then I get key-lock conflicts etc.
If it is realy such that the EntityManager is supposed to handle only one query/update at a time, how should I synchronose that in the async. world of Silvrelight/DevForce?



Replies:
Posted By: WardBell
Date Posted: 08-Jul-2010 at 11:59pm

The IsBusy flag is purely advisory and does not reflect what the EntityManager (EM) can do. IsBusy exists to illustrate how a UI could block user input until the data-to-display have "settled down".

Meanwhile, the EM can handle simultaneously running async queries and updates. How many? I don't know. Maybe you can issue "too many" at once and flood the number of EM's background threads beyond some limit. I haven't encountered such a limit personally but I have wondered about it. 
 
It also doesn't matter whether the EM is handling requests from one module or one hundred.
 
That's the way it's supposed to work. That's the way it works for me when I fire off a battery of such async requests.
 
I'm not able to reproduce the experience you are describing.
 
Question: are you calling the EM on different threads? Are the modules calling the EM on their own threads?
 
If so, there's your answer. The EM is not thread-safe. By that I mean that a consumer of the EM API must be on the same thread as the EM; of course the EM itself is busy managing its own worker threads on which it issues its asynchronous requests. The point is that all callers upon the EM must be on the same thread with the EM.
 
Btw, this is perfectly normal. Similar"context" objects from other technologies aren't thread-safe either.
 
If all your EM callers are on the same thread, the mystery remains. Let us know and we'll happily look at your failing case. We'll ask you to trim it down to the minimum code, database, and model necessary to reproduce the problem (we can't investigate an entire application). I realize that reproducing it may be difficult ... but we need something to work with.
 
In sum:
- EM's can handle many simultaneously async requests
- Prism shouldn't matter
- IsBusy shouldn't matter
- Multiple calling threads would matter
 
I hope my report is encouraging and that, together, we can isolate and resolve your troubles.
 
 


Posted By: tj62
Date Posted: 09-Jul-2010 at 2:29pm
Thank you for your reply. I will investigate this better after the weekend and come back to you.


Posted By: WardBell
Date Posted: 09-Jul-2010 at 10:05pm

I added a “StressTest” module to my Silverlight PrismExplorer (PE) demo app today.  I'm using a pre-release version of DevForce 2010 6.0.4; would expect it to work the same with current 6.0.3.

 
   I suspect we'll release this version next week.
 
One can experience the effects of firing arbitrary numbers of query operations (of various kinds) while trying to do other work. All UI screens in PE, including StressTest, share the same EM.

 

Works surprisingly well. I did find ONE problem: uncatchable exception thrown by us when you pause SQL Server in the middle of an async query flurry. I registered a Defect report about that.

 

Key points:

 

While executing in debugger I ran over 6000 operations simultaneously in two browser sessions and one Out-of-Browser session. Remember that each session has it's own EM.

 
Periodically I manually switched to the other screens, ran queries, made changes, inserted, deleted, saved, cleared, restored from local file cache … and it didn’t fail once.
 
You do have to dodge the "IsBusy" when manipulating the UI screens while stress testing the session; tiny window when you can sneak in a request. Quite doable.

 

I was firing batches ("runs") of 100 “DataSourceOnly” queries with 3 second pause between runs. Probably should toss in automation of some inserts, updates, and deletes. I'll get around to it at some point.

 

Of course it got a little pokey when I queried manually … but not bad at all. I half expected some SQL Server deadlocks on tables but didn’t get any; will have to try harder.

 

I also watched memory consumption over the course of these tests. Stayed flat.

 

All the action pretty much pegged the dual CPUs on my little notebook. Curious about how it behaves in a "proper environment" such as IIS on our public web server.

 
Seems pretty solid to me right now.
 
P.S: Let it run solo OOB while writing this. After 14 minutes and 14,000 operations (mixed with some manual insert, delete, clears, restores) still no failures. I realize 1000 ops/min is not much but remember that
  •  I pause 3 seconds between each run of 100 calls
  • Some of the calls fetch all of the orders
  • Database, VS debugger, Casinni web server, the application, and everything else I happen to be doing are all competing for resources on a notebook computer. This hardly qualifies as a production environment.

I'll be keeping my eye on it.



Posted By: tj62
Date Posted: 12-Jul-2010 at 4:03am
Thank you for this Ward,
I checked my code over the weekend.
First of all I'm running only one Entity Manager (single instance) that all modules use for querying, updating, inserting and deleting.
I checked the threading. I did that by naming the thread that created the EM instance such:

System.Threading.Thread.CurrentThread.Name = "Persistence Gateway Thread";

Then in the debugger I checked the tread name whenever a query, insert, update or a delete happened. It was allways the same as the EM instance creator (the name was the same). So threading is not an issue in my case.
 
The only differences between my environment and yours are as follow:
 
  •  I'm using version 5.2.4.2 of Ideablade Devforce for SL. Should I upgrade?....then to which  version?
  •  The underlying database is IBM Informix Dynamic Server 11.5.
  • Regarding the database, I had a small problem that I ran into few months ago using DevForce that I did a workaround for. This problem might lead us to the source. Let me describe that one:

    I had a little complex procedure to implement that needed quite a server work. Instead of calling many times from the client and doing the calculations there, I created a Async Method (by calling InvokeServerMethodAsync() ) and did all the work on server side.
    On the server side function I had an SQL-instert command (using ExecuteNonQuery() from a normal SQL Command Object).
    It all worked fine if calling and fininishing the server function compleetly before calling again. However I my SL client code is such that the async method is called simultaneously, many time asynchronously from different modules. Then I got Index Locking on the server side. The work around was to use a named-semaphore on the server side that guranteed that simultaneous calls waited on the semaphore for the current to finish before the next was fired.
     
    This might explain the insert problems I get with EM, but not the simultaneous query-problem returning "Not Found" exception.


    Posted By: tj62
    Date Posted: 12-Jul-2010 at 4:08am
    a follow up to my last post (something I forgot)....or could it have something to do with this that you describe in your last post:

    Works surprisingly well. I did find ONE problem: uncatchable exception thrown by us when you pause SQL Server in the middle of an async query flurry. I registered a Defect report about that.



    Posted By: WardBell
    Date Posted: 12-Jul-2010 at 5:07pm
    @tj62 -
     
    After over 1 Million runs of the Stress Tester in 3 sessions I too encountered the dread "NotFound" exception. One of the sessions threw it 14 times in the space of 30 seconds.
     
    Discouraging.
     
    A web search of "NotFound" indicates WCF involvement and tons of frustration but not much in the way of answers. For example http://stackoverflow.com/questions/2316201/silverlight-and-wcf-notfound-error - http://stackoverflow.com/questions/2316201/silverlight-and-wcf-notfound-error  and http://www.silverlightshow.net/news/The-remote-server-returned-an-error-NotFound.-Silverlight-Exception-.aspx - http://www.silverlightshow.net/news/The-remote-server-returned-an-error-NotFound.-Silverlight-Exception-.aspx .
     
    One hypothesis: server timeout. I'm running in Cassinni (the Web Server in Visual Studio) which is throttled. Maybe it got overwhelmed and barfed.
     
    We'll keep looking. We'll try on IIS and something like a real server environment. Meanwhile, I suggest you catch-and-retry your queries after a small delay.
     
    You wrote "from time to time I get a "... The remote server returned an error: NotFound." exception". What does "from time to time" mean? How often? Were you running in Cassinni? With a debug build? On what kind of hardware? These facts are potentially pertinent.
     
    ---
     
    I traced the "uncatchable exception" problem to a particular DevForce method that is new in the not-yet-released DevForce 2010 v.6.0.4. That can't be your problem. We'll fix it before release I'm sure.
     
    ---
     
    I conducted all my investigations with DevForce 2010. You are on DevForce 2009 ... which is feature frozen. I would definitely upgrade if I had the choice.
     
     


    Posted By: WardBell
    Date Posted: 12-Jul-2010 at 5:17pm
    @tj62 - I agree that your InvokeServerMethod intervention is unlikely to influence queries.
     
    The fact that you are using Informix complicates our ability to help you diagnose the problem. Maybe the Informix provider is dying ... although I don't know why that would result in the mysterious "NotFound' which (we think) comes from WCF.
     
    Our support responsibilities pretty much stop where Entity Framework takes over. If it comes to the point where you need to send us some code, you'll first have to recreate your test case with a SQL Server Northwind example. Sorry about that.


    Posted By: tj62
    Date Posted: 13-Jul-2010 at 4:07am
    Ward,
    I'm aware that if this is an INformix issue or IBM data provider issue, this is not your deal. However first of all I'm going to find a workaround such that this issue does not stop our development while we are solving the case, that can take months in particular if we need to work with IBM for a solution :-(

    My work around was to create a query/insert/update/delete FIFO-queue for the EM, such that queries etc. stack up in the queue while a query is not replied from the server. In the Manager.Fetched and Manager.Saved handlers I check the queue for waiting items and send the next query/save command.  I need a small help to do this.
     
    I keep the queue in PersistenceGateway class and do a little ugly addition to IEntityManagerProvider class to enable the Repository class to access the PersistencGateway (I know, I know this should not be done...but needed for this):
     
    using IdeaBlade.EntityModel;
    namespace RN.Modules.Infrastructure {
       public interface IEntityManagerProvider {
          EntityManager EntityManager { get; }
          bool IsBusy { get; }
          void QueueIt(object obj);
       }
    }

    Now I change the Untyped query Query metod in the Repository class (se red addition):
     
       //A a small data-class to store the query on the queue:
        public class UntypedQuery
        {
            public IEntityQuery PerformQuery;
            public Action<IEnumerable> PerformCallback;
            public delegate void FuncToCall(IEntityQuery query, Action<IEnumerable> callback);
            public FuncToCall FuncToCallDelegate;
        }

    protected void Query(IEntityQuery query, Action<IEnumerable> callback) {
          string testThreadName = System.Threading.Thread.CurrentThread.Name;
          if (null == query) {
            QueryCallbackImpl(null, new List<object>() , callback );
            return;
          }
          if(p_entityManagerProvider.IsBusy)
          {
              UntypedQuery q = new UntypedQuery();
              q.PerformQuery = query;
              q.PerformCallback = callback;
              q.FuncToCallDelegate = this.Query;
              p_entityManagerProvider.QueueIt(q);
              return;
          }
          p_entityManager.DefaultQueryStrategy = QueryStrategy.Normal; // Ensure going to data source
          try {
            p_entityManager.ExecuteQueryAsync(
              query,
              args => QueryCallback(args, callback),
              null);
          } catch (Exception e) {
            QueryExceptionHandler(e, callback);
          }
        } 
     
    All fine and everything works. Now I want to do the same for the Strongly Typed Query method, but then I run into the problem. The strongly typed query method is defined such:

    protected void Query<T>(IEntityQuery<T> query, Action<IEnumerable<T>> callback)

    My problem is how in heck can I create the the data-class to store the parameters on the queue like I did for the untyped query:
        public class TypedQuery
        {
            public IEntityQuery PerformQuery; //Not sure if this works to store the IEntityQuery<T> query parmaeter
            public ????? PerformCallback; // to store Action<IEnumerable<T>> callback parameter
            public delegate void FuncToCall<T>(IEntityQuery<T> query, Action<IEnumerable<T>> callback);
            public FuncToCall<?????> FuncToCallDelegate;  // To store  a "function pointer" to the strongly typed Query<T>()
        }
    I'm not good at this typed stuff.  Is it in general possible to create such a data class to stor the parametes of the strongly typed Query method?

    Then my last question: If I upgrade to DevForce 2010. I will have to upgrade to Visual Studio 2010 and .NET4, isn't it?
    As a DevForce customer whered do I download the DevForce upgrade from?
    Do I simply download the free Express Edition and apply our licence to it...or is it something more complex?


    Posted By: WardBell
    Date Posted: 13-Jul-2010 at 4:50pm
    Hi Tj62 - Last questions first.
     
    Upgrading to DF2010 does mean upgrading to VS2010 and EF4 and SL4. It's "4" all the way.
    Yes, you can download the Express edition and your license should work. I may have missed a memo so, if it does not, please contact us ( mailto:sales@devforce.com - sales@devforce.com ) and we'll get you a new one immediately.
     
    That's a lot of code you've typed in. I want to step back a bit to see if I understand your intent. It seems you want to issue your operations in sequence, initiating the next operation after the prior action completes.
     
    You are doing this because you can not issue them in parallel because you think that DevForce is misbehaving. You would issue them in parallel if you could.

    Do I understand correctly?
     
    I do not recommend "working around" a perceived DF failure until we have demonstrated that there is a failure. As I said, I cannot reproduced the problem you have with any frequency. If I were going to work-around it, I would intercept the occasional failure and retry ... rather than change parallel tasks into sequential tasks.
     
    For some workflows a sequential approach is required. Some times you need one result before you can even compose the next query. We offer the "AsyncSerialTask" class for this purpose and I recommend that you look at that.
     
    The AsyncParallelTask class is another option to consider when you have a bunch of operations that you want to execute in parallel.
     
    You can mix and match these.
     
    --
     
    As for architecting your way ... which is a variation on the Command Pattern ..
     
    You write a strongly typed QueuedQuery<T> (akin to your "UntypedQuery) that preserves the strongly typed definition of a query. The trick you're looking for is to make it adhere to a non-generice interface; call it IQueuedQuery. Your UntypeQuery(which I would rename simply "QueuedQuery") should also implement this interface. The IQueuedQuery interface members are all untyped (e.g., "Perform").
     
    "QueueIt" shouldn't take object; it should take IQueuedQuery instances.
     
    That's it in rough.
     
    --
     
    But I can't end without coming back to originating issue, namely, your experience of DF not behaving. I really wish you could prove that with a reproducable test case that is independent of the larger context of your applications. It's never good to sweep problems under the rug, IMHO.
     


    Posted By: tj62
    Date Posted: 15-Jul-2010 at 5:27am
    Yes I agree with you. I have to find the source for the problem and have it corrected. But real life is not that easy. From experience I know that this can take a long time. While that is happening the clock is ticking and deadline for first prototype of our system geting closer. So it is better for us have a "handycaped" prototype than no prototype at the deadline.
    So first thing is to secure that I can finish the prototype independent if this problem is solved or not. However after that I will for sure return to the problem to get it solved.
     
    Regarding your suggestion imnplementing IQueuedQuery, etc. I ran again into a problem. The problem arise as you can not convert IEnumerable<T> to IEnumerable. So if I try to implement such:
     
        public delegate void QueryFuncToCall(IEntityQuery query, Action<IEnumerable> callback);
        public delegate void QueryFuncToCall<T>(IEntityQuery<T> query, Action<IEnumerable<T>> callback);

        // None generic interface
        public interface IQueuedQuery {
            IEntityQuery PerformQuery { get; set;}
            Action<IEnumerable> PerformCallback { get; set;}       
            QueryFuncToCall FuncToCallDelegate { get; set;}
        }
      // Strigtly typed version of QueuedQuery
       public class QueuedQuery<T> : IQueuedQuery
       {
           private IEntityQuery _performQuery;
           private Action<IEnumerable> _performCallback;
           private QueryFuncToCall _funcToCallDelegate;
           public IEntityQuery PerformQuery
           {
               get { return _performQuery; }
               set { _performQuery = value; }
           }
           public Action<IEnumerable> PerformCallback
           {
               get {return _performCallback }
               set { _performCallback = value; }
           }
           public QueryFuncToCall FuncToCallDelegate
           {
               get { return _funcToCallDelegate; }
               set { _funcToCallDelegate = value; }
           }

           QueuedQuery(IEntityQuery<T> query, Action<IEnumerable<T>> callback, QueryFuncToCall queryDelegate)
           {
               PerformQuery = query; // conversion from strongly typed to not typep allowed.
               PerformCallback = callback;  // does not compile, conversion error
               FuncToCallDelegate = queryDelegate;  // Use the untyped delegate allways ??!!??
           }
       }
    Notice the red line above. The conversion from IEnumerable<T> to IEnumerable is not possible...or is it?
    The above constructor (or similar) has to exist as Query method defination is such:

    protected void Query<T>(IEntityQuery<T> query, Action<IEnumerable<T>> callback)



    Posted By: tj62
    Date Posted: 07-Sep-2010 at 4:12am
    Hi Ward,
    Strangely enough this problem seems to have disappeard after I upgraded DevForce  2009 to version 5.2.7.0. So there seems to have been a "bug" or something in previous versions of DevForce that caused this problem with INformix?!?!

    Regards
      TJ



    Print Page | Close Window