New Posts New Posts RSS Feed: Get only changed entities from the database
  FAQ FAQ  Forum Search   Calendar   Register Register  Login Login

Get only changed entities from the database
 Post Reply Post Reply
Author
  Share Topic Share Topic  Topic Search Topic Search  Topic Options Topic Options
sieepvvalk View Drop Down
Newbie
Newbie
Avatar

Joined: 13-Aug-2007
Location: United States
Posts: 7 		Post Options Post Options   Quote sieepvvalk Quote  Post ReplyReply Direct Link To This Post Topic: Get only changed entities from the database
    Posted: 09-Dec-2007 at 7:31pm
Thankd David... I will take this into consideration.
Back to Top
davidklitzke View Drop Down
IdeaBlade
IdeaBlade
Avatar

Joined: 14-Jun-2007
Posts: 715 		Post Options Post Options   Quote davidklitzke Quote  Post ReplyReply Direct Link To This Post Posted: 09-Dec-2007 at 5:38pm
 Option 2 is better than Option 3 because:
 
(1) Option 2 takes less time for the database to execute (one time comparison versus one time comparison plus one NOT IN comparison.
 
(2) Option 2 takes less processing time on the client to formulate.
 
As the number of items in the entity cache gets larger and larger, Option 2 becomes faster and faster relative to Option 3.
Back to Top
sieepvvalk View Drop Down
Newbie
Newbie
Avatar

Joined: 13-Aug-2007
Location: United States
Posts: 7 		Post Options Post Options   Quote sieepvvalk Quote  Post ReplyReply Direct Link To This Post Posted: 09-Dec-2007 at 12:16pm
Thank you for the suggestion.  I want to clarify my understanding of your solution and see if there is any better way to do this.  I will present 3 options and hopefully you can help me to figure out which of these is the best way to do things with DevForce.
 
OPTION 1: This is the most basic way we can query for new orders:
 
(1) I run a query that returns 30 orders meeting a certain criteria
(2) 10 new orders are added to the system over the next 10 minutes
(3) I run the same query again, and this time i should get 40 orders.  If I use DevForce, I will have to use a DataSourceOnly or DataSourceThenCache QueryStrategy because I need to check the database to get the new orders.  This strategy will send all 40 orders back over the network, clogging network resources.
 
OPTION 2: Your solution is to follow these steps:
 
(1) I run a query that returns 30 orders meeting a certain criteria
(2) I store the date that the query was last run as our (LastRunDate)
(3) 10 new orders are added to the system over the next 10 minutes.
(4) I run a query to get all orders which match my criteria that were added since our (LastRunDate).
(5) I merge these new orders with my old orders, thus minimizing network traffic.
 
OPTION 3: Is options 2 a more optimized solution than the following?
 
(1) I run a query that returns 30 orders meeting a certain criteria
(3) 10 new orders are added to the system over the next 10 minutes.
(4) I run a query to get all orders which match my criteria and I add a "NOT" clause which excludes order which match the primary keys of the orders already in the list.
(5) I merge these new orders with my old orders, thus minimizing network traffic.
 
I am wondering what the most optimized way to do this would be.  I suppose that since our primary key is automatically indexed, the best processing speed (on the server side) would be to eliminate certain records based on their primary key.  This however poses the problem that we are adding a good amount of additional outgoing network traffic to send the list of primary keys to the server.  A big concern with our application is speed, so I would like to know which way would be more likely to minimize the overall query time for new orders.
 
I appreciate your help with this issue.
Back to Top
davidklitzke View Drop Down
IdeaBlade
IdeaBlade
Avatar

Joined: 14-Jun-2007
Posts: 715 		Post Options Post Options   Quote davidklitzke Quote  Post ReplyReply Direct Link To This Post Posted: 09-Dec-2007 at 10:31am
I think  you are asking for all the entities that have been added or changed using DevForce (or perhaps all entities that have been added or changed recently).
 
My recommendation (if this is possible) is to add CreationTimeStamp (CrtnTs) and ModificationTimeStamp (ModTs) columns (as in Employee table in IdeaBlade Tutorial database).  Then you just have to query for objects which have creation or modification time stamps newer than a particular time.
 
With respect to "CacheThenDataSource", I agree that earlier definitions were definitely misleading.  Fortunately, we recently cleaned up this documentation.  Here is the new description.
 
CacheThenDataSource
 
Check the query cache to see if the code has previously submitted the current query or an identifiable superset of it.[1]  If so, satisfy the query from the entity cache, and skip the trip to the datasource.

If the query cache contains no query matching or encompassing the current query, DevForce will go to the datasource for the requested data.  Having retrieved data from the datasource into the entity cache, it will then obtain from the entity cache the final set of entity references it returns to the caller.  By so doing, it will return references to new entities that exist only in the cache, as well as those that duplicate rows from the datasource.

Some overloads of GetEntity() and GetEntities() permit search for entities by primary key. These queries work differently from those that specify arbitrary criteria. For primary key queries, there is no preliminary search of the query cache for a matching query: rather, DevForce searches the entity cache directly for the provided key value or values. If matching entities are not found in the entity cache, the datasource is also searched.

[1] For example, if you’re now asking for all Customers from the Eastern region, and you previously retrieved all Customers (without qualification), the all-Customers-without-qualification query will be recognized as a superset of the all-Customers-from-the-Eastern-region query, and the new query will be satisfied from the entity cache.



Edited by davidklitzke - 09-Dec-2007 at 10:32am
Back to Top
sieepvvalk View Drop Down
Newbie
Newbie
Avatar

Joined: 13-Aug-2007
Location: United States
Posts: 7 		Post Options Post Options   Quote sieepvvalk Quote  Post ReplyReply Direct Link To This Post Posted: 08-Dec-2007 at 1:44pm
Is there any way to get ONLY entities which have been added or changed using DevForce?
 
An application we created gets a list of entities at startup.  Every 60 seconds, it re-downloads the list from the server to make sure the user is always up to date with the latest changes.  In order to do this, we are using a QueryStrategy of "DataSourceThanCache", but this causes ALL the data to be refreshed from the server.  This causes excess network traffic and that is something we would like to avoid when only a few records change every 60 seconds.
 
It seems that the "CacheThanDataSource" FetchStrategy is a bit misleading, because it doesn't check the cache THEN get the additional records from the server... it actually should be "CacheORDataSource".  Is there something that really does perform a "CacheThanDataSource" query to minimize network traffic?
Back to Top
 Post Reply Post Reply

Forum Jump Forum Permissions View Drop Down