Service Unavailable

[hafa]

Server Application Unavailable

[RANT]

This is happening about 70% of the time on a very important, busy domain. It’s all due to the .NET upgrade.

Traffic has dropped off on this site, and nothing has changed, so I have difficulty accepting the argument that the site has too much traffic for a shared environment, as it’s been fine for the past 3 years at its current level.

70% downtime is simply not acceptable by any standards. I’ve been repeatedly assured that Jodohost are monitoring the site, but it was down for 5 HOURS today X(

I’ve been a big fan of Jodohost for many years, but my enthusiasm is waning. What do we need to do to fix this? Why do we need a VPS to handle this now, when it’s been fine for years at the current level of traffic? Why are all of these problems, which are limited entirely to .NET (aspx) pages just now cropping up after the upgrade? Why are classic pages working just fine?

I need answers, not excuses; action, not reassurance, and a reasonable amount of uptime.

If I’m forced to move to a VPS, so be it, but it seems to me that this move is being forced not due to actual need, but rather an inability or unwillingness to actually fix the main issue.

[/RANT]

http://www.angildesign.com/images/comp/dead.jpg

And now to add insult to injury I just got booted off of live chat…

[Pratik]

We have now moved your application to a different pool and there should not be issues.

The chat got booted as our office network went offline.

[hafa]

Hi Pratik

This is the fourth time that the application has been moved to a different pool; transcripts available upon request. This seems to be a temporary solution at best. It seems to me that the pool hanging is an issue with .net, and moving it is a work around.

Since the site goes down every time the pool hangs and nothing gets done until I post a ticket, we’re having huge swaths of downtime on a site that gets constant traffic. This is not acceptable.

Please indicate precisely what steps are needed to resolve this issue permanently. If it’s due to abuse of a script, tell me and I’ll change it. If it’s due to hacking, DDOS or other issues, let’s block some IPs. If it’s due to .net upgrades, let’s move to an older machine, as the scripts are all circa 2004 anyway…

[hafa]

Hey Pratik

Regarding your reply in this thread: Is this the first time it’s been moved to a dedicated pool? What matter of monitoring is being done, in terms of frequency? The reason I ask is because the last time it was down for over 5 hours so the monitoring was obviously inadequate.

Thanks for the replies.

Hafa

[Pratik]

Hello hafa,

What i have done is watching the back end process working under your application pool and what is going on there. I will have to check it for some time to be sure what is causing this.

I will update you on the ticket.

[Stephen]

Hafa,

Your site is in a dedicated pool only to itself, there is NO other site to conflict, in addition your site is not on a server that had an asp.net upgrade, win27 has had 3.5 from the start.

[hafa]

Just a couple of clarifications and an update, Stephen:

[ol]
[li]AFAIK, the site was only moved to a dedicated pool subsequent to my posting this thread.
[/li]
[li]The site was moved to win23 last week after the problem began.
[/li][/ol]

Pratik has done the following and things seem stable for the time being:

[ol]
[li]…have made some manual adjustment for the pool so that it do(es) not hangup, also i have put up a auto recycle so if it has some overload it will recycle it self.
[/li]
[li]I have also placed a tracer for your website and that will help me find if any of your URL is doing or causing the pool to crash, I am just keeping this on my observation continuously so that i can see if any of the URL is doing that.
[/li][/ol]

Pratik subsequently found this error:

Server Error in ‘/’ Application.
Exception of type System.OutOfMemoryException was thrown.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.OutOfMemoryException: Exception of type System.OutOfMemoryException was thrown.

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:

[OutOfMemoryException: Exception of type System.OutOfMemoryException was thrown.]

Version Information: Microsoft .NET Framework Version:1.1.4322.2407; ASP.NET Version:1.1.4322.2407

After this was found, he made further changes to the pool settings.

My point remains that this domain has enjoyed nearly 100% uptime for the past 4 years at JH, traffic has remained consistent for the past year or so, and there have been virtually no changes to the code.

As I mentioned, the .NET update was coincidental and while Correlation does not imply causation, something has changed very recently to bring about this problem.

[Stephen]

@hafa

Just a couple of clarifications and an update, Stephen:

[ol]
[li]AFAIK, the site was only moved to a dedicated pool subsequent to my posting this thread.
[/li]
[li]The site was moved to win23 last week after the problem began.
[/li][/ol]

Pratik has done the following and things seem stable for the time being:

[ol]
[li]…have made some manual adjustment for the pool so that it do(es) not hangup, also i have put up a auto recycle so if it has some overload it will recycle it self.
[/li]
[li]I have also placed a tracer for your website and that will help me find if any of your URL is doing or causing the pool to crash, I am just keeping this on my observation continuously so that i can see if any of the URL is doing that.
[/li][/ol]

Pratik subsequently found this error:

After this was found, he made further changes to the pool settings.

My point remains that this domain has enjoyed nearly 100% uptime for the past 4 years at JH, traffic has remained consistent for the past year or so, and there have been virtually no changes to the code.

As I mentioned, the .NET update was coincidental and while Correlation does not imply causation, something has changed very recently to bring about this problem.

The change I am almost certain was hsphere update, it has had some asp.net errors we have been battling, you can see a post in the general non cluster specific network status forum for some time it was not swithcing asp.net versions at all.

[hafa]

@Stephen

The change I am almost certain was hsphere update, it has had some asp.net errors we have been battling, you can see a post in the general non cluster specific network status forum for some time it was not swithcing asp.net versions at all.

Right. I saw that post and then looked at when our troubles started appearing and drew a correlation. Given the other factors involved (consistency of traffic, lack of changes, time up, only .net pages affected) it seemed a strong case that the switching issue or some corollary thereof was to blame.

I applaud Pratik’s heroic efforts in finding a workable solution to this issue but remain concerned that the root cause may have not been addressed due to the fact that it may remain unknown. As we know all too well, unknowns in this area have a bad habit of turning around and biting you in the @$$.

[Stephen]

@hafa

Right. I saw that post and then looked at when our troubles started appearing and drew a correlation. Given the other factors involved (consistency of traffic, lack of changes, time up, only .net pages affected) it seemed a strong case that the switching issue or some corollary thereof was to blame.

I applaud Pratik’s heroic efforts in finding a workable solution to this issue but remain concerned that the root cause may have not been addressed due to the fact that it may remain unknown. As we know all too well, unknowns in this area have a bad habit of turning around and biting you in the @$$.

the biggest issue is the amount of RAM it is taking even in its own pool, will discuss with you elsewhere on that

[hafa]

@Stephen

the biggest issue is the amount of RAM it is taking even in its own pool, will discuss with you elsewhere on that

Feel free to drop me a PM or contact me via email anytime; I obviously have a vested interest in dealing quickly with any outstanding issues

[jpveen]

Hi Hafa,
If I may ask: what application do you run?
I also have a client which gets randomly ‘errors in application’, my fix is to edit web.config, and save it, which causes IIS to restart the app.
This customer is running DotNetNuke 4.8 on ASP.net 2.0 at win22.

Best regards, Jan-Pieter

[hafa]

@jpveen

Hi Hafa,
If I may ask: what application do you run?
I also have a client which gets randomly ‘errors in application’, my fix is to edit web.config, and save it, which causes IIS to restart the app.
This customer is running DotNetNuke 4.8 on ASP.net 2.0 at win22.

Best regards, Jan-Pieter

Hey, Jan-Pieter

The application is a custom .NET 1.1 app I wrote myself. Since Pratik moved it to its own pool and got it set up right it’s been just fine. Thanks for the input!

Stephen hasn’t contacted me yet on this one either, so until that happens, I’ll proceed as if all is well…

[Stephen]

@hafa

Hey, Jan-Pieter

The application is a custom .NET 1.1 app I wrote myself. Since Pratik moved it to its own pool and got it set up right it’s been just fine. Thanks for the input!

Stephen hasn’t contacted me yet on this one either, so until that happens, I’ll proceed as if all is well…

Well I have been busy on other tasks and it is not anything critical right now.