Author |
Share Topic Topic Search Topic Options
|
JohnBloom
Groupie
Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
|
Post Options
Quote Reply
Topic: Random Crashes caused by a compilation error in the EntityServer.svc Posted: 23-Feb-2012 at 1:38pm |
We are getting a random crash on our customer's site about every other day. What I know from what they have told me is that they are doing normal tasks and all of a sudden the program crashes. Here is the info from event viewer: WebHost failed to process a request. Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/12671853 Exception: System.ServiceModel.ServiceActivationException: The service '/EntityServer.svc' cannot be activated due to an exception during compilation. The exception message is: Object reference not set to an instance of an object.. ---> System.NullReferenceException: Object reference not set to an instance of an object. at System.Web.Compilation.DiskBuildResultCache.CacheBuildResult(String cacheKey, BuildResult result, Int64 hashCode, DateTime utcStart) at System.Web.Compilation.BuildManager.CacheBuildResultInternal(String cacheKey, BuildResult result, Int64 hashCode, DateTime utcStart) at System.Web.Compilation.BuildManager.CompileWebFile(VirtualPath virtualPath) at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate) at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate) at System.Web.Compilation.BuildManager.GetVPathBuildResult(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean ensureIsUpToDate) at System.ServiceModel.ServiceHostingEnvironment.HostingManager.GetCompiledCustomString(String normalizedVirtualPath) at System.ServiceModel.ServiceHostingEnvironment.HostingManager.CreateService(String normalizedVirtualPath) at System.ServiceModel.ServiceHostingEnvironment.HostingManager.ActivateService(String normalizedVirtualPath) at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath) --- End of inner exception stack trace --- at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath) at System.ServiceModel.ServiceHostingEnvironment.EnsureServiceAvailableFast(String relativeVirtualPath) Process Name: w3wp Process ID: 4832 Specs: Devforce version 6.1.2.0 Silverlight 4. IIS 7.5 Really old version of DAF I havent been able to get very far using google so a little nudging in the right direction would be nice. Thanks, John
Edited by JohnBloom - 24-Feb-2012 at 9:05am
|
-John Bloom
|
|
DenisK
IdeaBlade
Joined: 25-Aug-2010
Posts: 715
|
Post Options
Quote Reply
Posted: 23-Feb-2012 at 3:10pm |
Hi John,
Could you post the DevForce DebugLog.xml generated on the server after the program crashes? The log may contain more clues.
|
|
JohnBloom
Groupie
Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
|
Post Options
Quote Reply
Posted: 24-Feb-2012 at 6:20am |
I looked in the log and there are no logs around the time of the crash. It looks like they have been overwritten. Is there somewhere else I can get more info or do I need to wait for it to happen again?
|
-John Bloom
|
|
DenisK
IdeaBlade
Joined: 25-Aug-2010
Posts: 715
|
Post Options
Quote Reply
Posted: 24-Feb-2012 at 2:57pm |
You may have to wait until it happens again for the log to have meaningful clues.
For the time being, I have some questions/suggestions.
1. Can you find out if the crash occurs after IIS recycle?
2. Do you have load balancing on? The debug log will indicate this.
|
|
JohnBloom
Groupie
Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
|
Post Options
Quote Reply
Posted: 05-Mar-2012 at 12:52pm |
Ok we
just had the problem again. Unfortunately we didn’t know what happened and it wasn’t
until later that I looked at the event log and saw the error, so no devforce
log.
Here
are the answers to your questions: 1) I
really don’t know if it is after an IIS recycle. Is there an easy way to tell?
This time it happened a minute or so after we updated our website so I doubt it
was already recycling. 2) I
am pretty sure we don’t use load balancing since our web service is on a single
machine. 3) The
client app stays on solid for 8 hrs a day. However, the errors seem to happen
during a time when the user is active. This most recent time we had just
updated the system so there hadn’t been any down time
4) We
do use Coroutine a lot. We are on a very early release of the DAF. We are
trying to move to the latest version but there are some major changes between
us and the latest. 5) We
are also using ASP.Net security. I regularly see these types of things in the
Event Viewer:
Event code: 4005
Event message: Forms authentication failed for the
request. Reason: The ticket supplied has expired.
I am not sure if those are related.
Edited by JohnBloom - 05-Mar-2012 at 1:01pm
|
-John Bloom
|
|
DenisK
IdeaBlade
Joined: 25-Aug-2010
Posts: 715
|
Post Options
Quote Reply
Posted: 06-Mar-2012 at 3:21pm |
Okay. Looks like we still need more clues. Here are further suggestions:
2. If it's possible, try manually recycling the app pool for your application. This is so we can prove quickly whether or not IIS recycle takes part here.
3. You said that the errors seem to happen when the user is active. Is there any pattern as to which part of the application the user's activity was at during the time? This might enable us to narrow down the possible causes.
|
|
JohnBloom
Groupie
Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
|
Post Options
Quote Reply
Posted: 08-Mar-2012 at 6:31am |
Ok Ill let you know when I have some more information.
|
-John Bloom
|
|
JohnBloom
Groupie
Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
|
Post Options
Quote Reply
Posted: 13-Mar-2012 at 6:57am |
Ok I got the client error but it was not in the devforce logs. It appears that whatever causes this problem crashes the entityserver before it can log the error. The client side error is this: An error occurred while communicating with the server. A common cause for this error is a return type from the invoked method which is either not serializable or not marked as a known type. We are getting it to repoduce on our client's machine pretty regularly but I cannot get it to reproduce in test. The only thing that I was really able to find in common between the crashes was the fact that they are all during [AllowRpc] calles. Not only that, but it appears that it happens only when two Rpc calles are made at the same time. We are trying to move to Cocktail to see if it has been resolved in a later version but we are kinda in a catch 22. We want to get our customers on a solid build before we do a major change but it might appear that the major change is required for the solid build.
|
-John Bloom
|
|
kimj
IdeaBlade
Joined: 09-May-2007
Posts: 1391
|
Post Options
Quote Reply
Posted: 13-Mar-2012 at 9:32am |
Hi, Denis is out sick today, so I'll try to help with this until he's back on his feet. Any of the types passed into or out of an RPC method has to be serializable and a "known" type. Generally DevForce won't find these errors until you call the method, and the communication error you see is usually the result. I wouldn't expect to see the compilation error you reported in your first post, but the problem with the types should be easier to resolve and could be the root cause, so let's tackle that first. If you know the RPC methods in question, then check all parameters and all return values, and make sure every type is defined on both the client and server. Here's some info on serialization and known types - http://drc.ideablade.com/xwiki/bin/view/Documentation/knowntypes. If you find this article inscrutable, just post your RPC signatures and the type definitions here and we can take a look. If the type definitions all look good then we can look at the multi-threading aspect.
|
|
JohnBloom
Groupie
Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
|
Post Options
Quote Reply
Posted: 13-Mar-2012 at 9:54am |
The type definitions are good. We have been using these
same Rpcs for months with this problem happening only occasionally.
This is a very sporadic bug that we have not been able to
reproduce in test at all. Up until now we haven’t had a reproducible set of steps at
our customers site. Now we can repo the steps on our latest build but it is too unstable for them to use.
The
multi-threading angle sounds promising. What does that entail?
|
-John Bloom
|
|
DenisK
IdeaBlade
Joined: 25-Aug-2010
Posts: 715
|
Post Options
Quote Reply
Posted: 13-Mar-2012 at 12:20pm |
Hi John,
I'm back. Could you post your RPC methods in question and all the types definitions involved? We have a theory and would like to do some tests.
|
|
JohnBloom
Groupie
Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
|
Post Options
Quote Reply
Posted: 14-Mar-2012 at 10:14am |
Ok here is my service and the return object. Again, this runs in test but fails in production on one specific site. namespace MyNamespace {
public class MyService
{
[AllowRpc]
public static ResponseObject ServiceMethod1(IPrincipal principal, EntityManager manager, params object[] parameters)
{
var guid = ((UserToken)principal.Identity).UserID.ToString();
return //code }
[AllowRpc]
public static ResponseObject ServiceMethod2(IPrincipal principal, EntityManager manager, params object[] parameters)
{
return //code }
[AllowRpc]
public static ResponseObject ServiceMethod2(IPrincipal principal, EntityManager manager, params object[] parameters)
{
return //code }
}
} namespace MyOtherNamespace {
[DataContract]
[DiscoverableType(DiscoverableTypeMode.KnownType)]
public class ResponseObject
{
[DataMember]
public long RequestID { get; set; }
[DataMember]
public long RecordID { get; set; }
[DataMember]
public bool HasBeenProcessed { get; set; }
}
}
|
-John Bloom
|
|
kimj
IdeaBlade
Joined: 09-May-2007
Posts: 1391
|
Post Options
Quote Reply
Posted: 14-Mar-2012 at 1:33pm |
Hi John, If you're not passing in parameters, and your return type of ResponseObject doesn't contain any additional properties, then our initial theory concerning multi-threading problems with these types doesn't hold up. But I'm now more concerned that there's a thread safety problem within the code in these methods. These methods are all defined in the same class, which is OK, but you need to be careful accessing any instance and static properties. Do you share any objects among these methods? If so, you can't without locking. Do you hold onto anything across method calls? If so, don't. :) At this point it would probably be best if you try to reproduce the problem, since you have all the code. Set up a parallel coroutine which calls these RPC methods, and put the whole thing in a loop for good measure. You want to set up a situation where the methods are executing in parallel and the thread safety problem arises. If this test doesn't show anything we can show you how to set up a console application which will synchronize multiple threads and helps in flushing out multi-threading problems. If you'd like, we can also take a look at the code in these methods to see if anything stands out as a red flag.
|
|
JohnBloom
Groupie
Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
|
Post Options
Quote Reply
Posted: 25-Apr-2012 at 7:50am |
Ok we just updated to Cocktail and we are now on the latest version of Devforce. That means whatever this bug is it is not because we are using an old version. The bug doesn't lock up the server anymore which is good. Here is our latest error report:
System.ServiceModel.ServiceHostingEnvironment+HostingManager/22171959 System.ServiceModel.ServiceActivationException: The service '/EntityServer.svc' cannot be activated due to an exception during compilation. The exception message is: Object reference not set to an instance of an object.. ---> System.NullReferenceException: Object reference not set to an instance of an object. at System.Web.Compilation.DiskBuildResultCache.CacheBuildResult(String cacheKey, BuildResult result, Int64 hashCode, DateTime utcStart) at System.Web.Compilation.BuildManager.CacheBuildResultInternal(String cacheKey, BuildResult result, Int64 hashCode, DateTime utcStart) at System.Web.Compilation.BuildManager.CompileWebFile(VirtualPath virtualPath) at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate) at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean throwIfNotFound, Boolean ensureIsUpToDate) at System.Web.Compilation.BuildManager.GetVPathBuildResult(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile, Boolean ensureIsUpToDate) at System.ServiceModel.ServiceHostingEnvironment.HostingManager.GetCompiledCustomString(String normalizedVirtualPath) at System.ServiceModel.ServiceHostingEnvironment.HostingManager.CreateService(String normalizedVirtualPath) at System.ServiceModel.ServiceHostingEnvironment.HostingManager.ActivateService(String normalizedVirtualPath) at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath) --- End of inner exception stack trace --- at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath) at System.ServiceModel.ServiceHostingEnvironment.EnsureServiceAvailableFast(String relativeVirtualPath) w3wp 9288
I will try to work with the customer to get this to reproduce in a controlled environment. To answer your previous question: We are not accessing variables on the server. We keep our server stateless.
I will report back when I find something.
|
-John Bloom
|
|
JohnBloom
Groupie
Joined: 30-Nov-2010
Location: Topeka, KS
Posts: 95
|
Post Options
Quote Reply
Posted: 25-Apr-2012 at 9:28am |
Finally solved this one I think. We had a COM object registered in the webservice. We had moved the process that uses the COM out to a windows service to make sure that process was not interfering and causing the problem. However, we did not unregister the COM object on our customers sites.
This means all of the customers who we had from the beginning were having the problem because they still had the registered COM object in the bin. I am guessing that when the process accessed the COM object from the windows service it caused the webservice to freak out and die (in not so technical terms).
I unregistered it and registered it in an isolated place and it worked. This is also why we were not able to repo in test since we were always testing from a fresh install.
Thanks to all who chipped in and helped try to debug this issue.
|
-John Bloom
|
|