Batch Processing Best Practices with DevForce EF

 If I think I’m going to want to navigate object graphs and apply business logic (e.g., validations or in-object workflow), I’ll strongly favor the object approach and live with the performance consequences. On the other hand, if I’m reading a short record, adding something to it, and shoving the revised record out to the database again … well I’m not getting much value from converting data into an entity.

 As with all things performance related I would stick to my usual practices until I had a compelling reason to change them. It confuses everyone to change data access modalities within an application. So, even in the second example, I’d stay with the entity approach until I was convinced that the performance was unacceptable. Even then I’d consider scaling with hardware before I changed the programming model. Why? Because humans (read “developers”) are your most expensive resources.

 On the other hand, if I sensed impending danger, I’d make sure that I had a reasonable alternative option … some place to move to if necessary.

 And you do. If you had to do so, you could author and invoke a server-side batch process that has direct access to the database and uses traditional ADO.NET techniques. According to a recent study, straight ADO could yield a 9x query improvement even with a 3-part object graph. Of course you won’t be able to re-use all of your entity business logic and you’ll spend a lot of time doing the plumbing – writing and testing – that could otherwise go into your application. I wouldn’t go here unless the logic were exceptionally simple and the volume / response time requirements were especially severe.

 Example: there are millions of ACH records in the database. ACH records are the basis for electronic funds transfers. I must generate a file of selected and reformatted ACH records for an external  ACH process that reads and processes flat files. My job is simple: query, format, write. DataReader and Writer sounds like the right call here.

 So much for general principles. What about your case?

 You describe a complex object graph (10 objects in a graph = complex to me) with business logic that is applicable to your high-volume task. You say you use a “home-grown ORM” today to handle the “batch process”. You don’t seem to be complaining about the performance from your own ORM. This suggests that, whatever the volume, the ORM-approach is working for you. Will it work with DevForce EF? Well I don’t know what “built-in some ways of handling” high-volume data you’ve got … but we can talk about some facilities available in DevForce EF

 ·         You can page your queries

·         Adding “includes” to a query enables fetching of just those objects that are related to the query results. If your paged query returns 10 Order objects, with  appropriate include clauses, it could also return just the Customer, LineItem, and Product objects that are related to those 10 Orders … and will do so in the same trip to the server as the root Order query itself.

I’m betting you’ll do something like this with your member query:
   manager.TransferLines.Where(t => t.Member.Id = theCurrentMemberId).Include(“Payments”);

·         You can subdivide your batches into reasonable chunks (say 1000 members at a time). Let a single EntityManager hold the data for a single chunk. Create multiple EMs for each chunk as follows:

o   break the process into a load phase, a calc phase, and a save phase.

o   Your calc phase is the main thread and will organize the workflow as well as perform calculations on a chunk.

o   In the “Load phase” you fetch a chunk’s worth of data asynchronously into one (or more) EMs; when the EM is primed, put it back on the main thread’s “work queue”; the async approach lets your main calculation phase thread iterate over the pre-loaded EMs without waiting for IO.

o   use our async Save method to persist EMs after you’ve run your calculations and produced output going back to the database; again, the async approach frees your main thread from the slower IO task of saving data.

o   You don’t have to create your own background threads to do any of this; we do it for you. You do have to manage the queues and the multiple EMs.

 You might recognize this as the ACTIVE OBJECT pattern. It is well described in Robert Martin’s great book Agile Principles, Patterns, and Practices in C#, pp.305-310