New Posts New Posts RSS Feed: Random Crashes caused by a compilation error in the EntityServer.svc
  FAQ FAQ  Forum Search   Calendar   Register Register  Login Login

Random Crashes caused by a compilation error in the EntityServer.svc

 Post Reply Post Reply
Author
JohnBloom View Drop Down
Groupie
Groupie
Avatar

Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
Post Options Post Options   Quote JohnBloom Quote  Post ReplyReply Direct Link To This Post Topic: Random Crashes caused by a compilation error in the EntityServer.svc
    Posted: 23-Feb-2012 at 1:38pm
We are getting a random crash on our customer's site about every other day. What I know from what they have told me is that they are doing normal tasks and all of a sudden the program crashes. Here is the info from event viewer:
 
WebHost failed to process a request.
 Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/12671853
 Exception: System.ServiceModel.ServiceActivationException: The service '/EntityServer.svc' cannot be activated due to an exception during compilation.  The exception message is: Object reference not set to an instance of an object.. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Web.Compilation.DiskBuildResultCache.CacheBuildResult(String cacheKey, BuildResult result, Int64 hashCode, DateTime utcStart)
   at System.Web.Compilation.BuildManager.CacheBuildResultInternal(String cacheKey, BuildResult result, Int64 hashCode, DateTime utcStart)
   at System.Web.Compilation.BuildManager.CompileWebFile(VirtualPath virtualPath)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate)
   at System.Web.Compilation.BuildManager.GetVPathBuildResult(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean ensureIsUpToDate)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.GetCompiledCustomString(String normalizedVirtualPath)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.CreateService(String normalizedVirtualPath)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.ActivateService(String normalizedVirtualPath)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath)
   --- End of inner exception stack trace ---
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath)
   at System.ServiceModel.ServiceHostingEnvironment.EnsureServiceAvailableFast(String relativeVirtualPath)
 Process Name: w3wp
 Process ID: 4832
 
Specs:
 
Devforce version 6.1.2.0
Silverlight 4.
IIS 7.5
Really old version of DAF
 
I havent been able to get very far using google so a little nudging in the right direction would be nice.
 
Thanks,
John


Edited by JohnBloom - 24-Feb-2012 at 9:05am
-John Bloom
Back to Top
DenisK View Drop Down
IdeaBlade
IdeaBlade


Joined: 25-Aug-2010
Posts: 715
Post Options Post Options   Quote DenisK Quote  Post ReplyReply Direct Link To This Post Posted: 23-Feb-2012 at 3:10pm
Hi John,

Could you post the DevForce DebugLog.xml generated on the server after the program crashes? The log may contain more clues.
Back to Top
JohnBloom View Drop Down
Groupie
Groupie
Avatar

Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
Post Options Post Options   Quote JohnBloom Quote  Post ReplyReply Direct Link To This Post Posted: 24-Feb-2012 at 6:20am
I looked in the log and there are no logs around the time of the crash. It looks like they have been overwritten. Is there somewhere else I can get more info or do I need to wait for it to happen again?
-John Bloom
Back to Top
DenisK View Drop Down
IdeaBlade
IdeaBlade


Joined: 25-Aug-2010
Posts: 715
Post Options Post Options   Quote DenisK Quote  Post ReplyReply Direct Link To This Post Posted: 24-Feb-2012 at 2:57pm
You may have to wait until it happens again for the log to have meaningful clues.

For the time being, I have some questions/suggestions.

1. Can you find out if the crash occurs after IIS recycle?

2. Do you have load balancing on? The debug log will indicate this.

3. Does the client application start and stay open for very long periods? I found a similar posting of this issue if the client app has a long period of inactivity. http://www.ideablade.com/forum/forum_posts.asp?TID=2597
Back to Top
JohnBloom View Drop Down
Groupie
Groupie
Avatar

Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
Post Options Post Options   Quote JohnBloom Quote  Post ReplyReply Direct Link To This Post Posted: 05-Mar-2012 at 12:52pm

Ok we just had the problem again. Unfortunately we didn’t know what happened and it wasn’t until later that I looked at the event log and saw the error, so no devforce log.

Here are the answers to your questions:

 

1) I really don’t know if it is after an IIS recycle. Is there an easy way to tell? This time it happened a minute or so after we updated our website so I doubt it was already recycling.

 

2) I am pretty sure we don’t use load balancing since our web service is on a single machine.

 

3) The client app stays on solid for 8 hrs a day. However, the errors seem to happen during a time when the user is active. This most recent time we had just updated the system so there hadn’t been any down time

 

4) We do use Coroutine a lot. We are on a very early release of the DAF. We are trying to move to the latest version but there are some major changes between us and the latest.

 

5) We are also using ASP.Net security. I regularly see these types of things in the Event Viewer:

       Event code: 4005

       Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired.

       I am not sure if those are related.

 


Edited by JohnBloom - 05-Mar-2012 at 1:01pm
-John Bloom
Back to Top
DenisK View Drop Down
IdeaBlade
IdeaBlade


Joined: 25-Aug-2010
Posts: 715
Post Options Post Options   Quote DenisK Quote  Post ReplyReply Direct Link To This Post Posted: 06-Mar-2012 at 3:21pm
Okay. Looks like we still need more clues. Here are further suggestions:

1. Turn on archiving for the debug log so it doesn't keep getting overwritten and hopefully it'll provide better clues the next time the crash happens. See http://drc.ideablade.com/xwiki/bin/view/Documentation/deploy-configuration#HLoggingElement.

2. If it's possible, try manually recycling the app pool for your application. This is so we can prove quickly whether or not IIS recycle takes part here.

3. You said that the errors seem to happen when the user is active. Is there any pattern as to which part of the application the user's activity was at during the time? This might enable us to narrow down the possible causes.
Back to Top
JohnBloom View Drop Down
Groupie
Groupie
Avatar

Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
Post Options Post Options   Quote JohnBloom Quote  Post ReplyReply Direct Link To This Post Posted: 08-Mar-2012 at 6:31am
Ok Ill let you know when I have some more information.
-John Bloom
Back to Top
JohnBloom View Drop Down
Groupie
Groupie
Avatar

Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
Post Options Post Options   Quote JohnBloom Quote  Post ReplyReply Direct Link To This Post Posted: 13-Mar-2012 at 6:57am
Ok I got the client error but it was not in the devforce logs. It appears that whatever causes this problem crashes the entityserver before it can log the error.
The client side error is this: An error occurred while communicating with the server. A common cause for this error is a return type from the invoked method which is either not serializable or not marked as a known type.
 
We are getting it to repoduce on our client's machine pretty regularly but I cannot get it to reproduce in test. The only thing that I was really able to find in common between the crashes was the fact that they are all during [AllowRpc] calles. Not only that, but it appears that it happens only when two Rpc calles are made at the same time.
 
We are trying to move to Cocktail to see if it has been resolved in a later version but we are kinda in a catch 22. We want to get our customers on a solid build before we do a major change but it might appear that the major change is required for the solid build.
-John Bloom
Back to Top
kimj View Drop Down
IdeaBlade
IdeaBlade
Avatar

Joined: 09-May-2007
Posts: 1391
Post Options Post Options   Quote kimj Quote  Post ReplyReply Direct Link To This Post Posted: 13-Mar-2012 at 9:32am
Hi, Denis is out sick today, so I'll try to help with this until he's back on his feet.
 
Any of the types passed into or out of an RPC method has to be serializable and a "known" type.   Generally DevForce won't find these errors until you call the method, and the communication error you see is usually the result.  I wouldn't expect to see the compilation error you reported in your first post, but the problem with the types should be easier to resolve and could be the root cause, so let's tackle that first.
 
If you know the RPC methods in question, then check all parameters and all return values, and make sure every type is defined on both the client and server.  Here's some info on serialization and known types - http://drc.ideablade.com/xwiki/bin/view/Documentation/knowntypes.  If you find this article inscrutable, just post your RPC signatures and the type definitions here and we can take a look. 
 
If the type definitions all look good then we can look at the multi-threading aspect. 
Back to Top
JohnBloom View Drop Down
Groupie
Groupie
Avatar

Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
Post Options Post Options   Quote JohnBloom Quote  Post ReplyReply Direct Link To This Post Posted: 13-Mar-2012 at 9:54am

The type definitions are good. We have been using these same Rpcs for months with this problem happening only occasionally.

This is a very sporadic bug that we have not been able to reproduce in test at all. Up until now we haven’t had a reproducible set of steps at our customers site. Now we can repo the steps on our latest build but it is too unstable for them to use.

The multi-threading angle sounds promising. What does that entail?

-John Bloom
Back to Top
DenisK View Drop Down
IdeaBlade
IdeaBlade


Joined: 25-Aug-2010
Posts: 715
Post Options Post Options   Quote DenisK Quote  Post ReplyReply Direct Link To This Post Posted: 13-Mar-2012 at 12:20pm
Hi John,

I'm back. Could you post your RPC methods in question and all the types definitions involved? We have a theory and would like to do some tests.
Back to Top
JohnBloom View Drop Down
Groupie
Groupie
Avatar

Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
Post Options Post Options   Quote JohnBloom Quote  Post ReplyReply Direct Link To This Post Posted: 14-Mar-2012 at 10:14am
Ok here is my service and the return object. Again, this runs in test but fails in production on one specific site.
namespace MyNamespace
{     public class MyService     {         [AllowRpc]         public static ResponseObject ServiceMethod1(IPrincipal principal, EntityManager manager, params object[] parameters)         {             var guid = ((UserToken)principal.Identity).UserID.ToString();             return //code
        }         [AllowRpc]         public static ResponseObject ServiceMethod2(IPrincipal principal, EntityManager manager, params object[] parameters)         {             return //code
}         [AllowRpc]         public static ResponseObject ServiceMethod2(IPrincipal principal, EntityManager manager, params object[] parameters)         {             return //code
        }     } }
namespace MyOtherNamespace
{     [DataContract]     [DiscoverableType(DiscoverableTypeMode.KnownType)]     public class ResponseObject     {         [DataMember]         public long RequestID { getset; }         [DataMember]         public long RecordID { getset; }         [DataMember]         public bool HasBeenProcessed { getset; }     } }
-John Bloom
Back to Top
kimj View Drop Down
IdeaBlade
IdeaBlade
Avatar

Joined: 09-May-2007
Posts: 1391
Post Options Post Options   Quote kimj Quote  Post ReplyReply Direct Link To This Post Posted: 14-Mar-2012 at 1:33pm
Hi John,
 
If you're not passing in parameters, and your return type of ResponseObject doesn't contain any additional properties, then our initial theory concerning multi-threading problems with these types doesn't hold up.
 
But I'm now more concerned that there's a thread safety problem within the code in these methods.  These methods are all defined in the same class, which is OK, but you need to be careful accessing any instance and static properties.  Do you share any objects among these methods?  If so, you can't without locking.  Do you hold onto anything across method calls?  If so, don't.  :)
 
At this point it would probably be best if you try to reproduce the problem, since you have all the code.  Set up a parallel coroutine which calls these RPC methods, and put the whole thing in a loop for good measure.  You want to set up a situation where the methods are executing in parallel and the thread safety problem arises.   If this test doesn't show anything we can show you how to set up a console application which will synchronize multiple threads and helps in flushing out multi-threading problems. 
 
If you'd like, we can also take a look at the code in these methods to see if anything stands out as a red flag.
 
 
Back to Top
JohnBloom View Drop Down
Groupie
Groupie
Avatar

Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
Post Options Post Options   Quote JohnBloom Quote  Post ReplyReply Direct Link To This Post Posted: 25-Apr-2012 at 7:50am
Ok we just updated to Cocktail and we are now on the latest version of Devforce. That means whatever this bug is it is not because we are using an old version. The bug doesn't lock up the server anymore which is good. Here is our latest error report: 

 System.ServiceModel.ServiceHostingEnvironment+HostingManager/22171959 
   System.ServiceModel.ServiceActivationException: The service '/EntityServer.svc' cannot be activated due to an exception during compilation. The exception message is: Object reference not set to an instance of an object.. ---> System.NullReferenceException: Object reference not set to an instance of an object. at System.Web.Compilation.DiskBuildResultCache.CacheBuildResult(String cacheKey, BuildResult result, Int64 hashCode, DateTime utcStart) at System.Web.Compilation.BuildManager.CacheBuildResultInternal(String cacheKey, BuildResult result, Int64 hashCode, DateTime utcStart) at System.Web.Compilation.BuildManager.CompileWebFile(VirtualPath virtualPath) at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate) at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate) at System.Web.Compilation.BuildManager.GetVPathBuildResult(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean ensureIsUpToDate) at System.ServiceModel.ServiceHostingEnvironment.HostingManager.GetCompiledCustomString(String normalizedVirtualPath) at System.ServiceModel.ServiceHostingEnvironment.HostingManager.CreateService(String normalizedVirtualPath) at System.ServiceModel.ServiceHostingEnvironment.HostingManager.ActivateService(String normalizedVirtualPath) at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath) --- End of inner exception stack trace --- at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath) at System.ServiceModel.ServiceHostingEnvironment.EnsureServiceAvailableFast(String relativeVirtualPath) 
   w3wp 
   9288 

I will try to work with the customer to get this to reproduce in a controlled environment. To answer your previous question: We are not accessing variables on the server. We keep our server stateless.

I will report back when I find something.
-John Bloom
Back to Top
JohnBloom View Drop Down
Groupie
Groupie
Avatar

Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
Post Options Post Options   Quote JohnBloom Quote  Post ReplyReply Direct Link To This Post Posted: 25-Apr-2012 at 9:28am
Finally solved this one I think. We had a COM object registered in the webservice. We had moved the process that uses the COM out to a windows service to make sure that process was not interfering and causing the problem. However, we did not unregister the COM object on our customers sites. 

This means all of the customers who we had from the beginning were having the problem because they still had the registered COM object in the bin. I am guessing that when the process accessed the COM object from the windows service it caused the webservice to freak out and die (in not so technical terms). 

I unregistered it and registered it in an isolated place and it worked. This is also why we were not able to repo in test since we were always testing from a fresh install.

Thanks to all who chipped in and helped try to debug this issue. 
-John Bloom
Back to Top
 Post Reply Post Reply

Forum Jump Forum Permissions View Drop Down