April 5 Migration Problems

A

ArlingtonMall

Guppy
Win32 web page/ASP and FTP access are both dog-slow right now. :(

I have been meaning to contact your firm about this issue for quite a while - we have noticed on-going performance problems with Win32 on-and-off for quite some time now, and rather than fix the problem, this migration seems to have made it much worse!

It should not take SEVERAL MINUTES to get a server reply for a simple web page!

More importantly, it appears something is now wrong with the files permissions for the existing files which were transferred by the migration? Attempting to either update or even manually delete existing files via ftp is now returning a error! -

550 filename.asp: Access is denied.

Attempting to upload, update or delete files that did not exist prior to the migration appears to be working fine, the problem is with pre-existing files that were migrated over to the new server.

This incredibly slow server response combined with an inability to even alter files on the server has brought development on a major project to a standstill. X(

Please get these file permissions fixed ASAP, and figure-out where the process went wrong, so that this problem is not repeated when you migrate the other servers hosting our other client's web sites.

Finally, we have also noticed when the server appears to be down or very slow, and we have gone to JodoPulse.Com to check this server's status, that win32 is not even listed on the Windows Web Server page!

One of our programmers once joked that the reason was you were too embarrassed with the response times on win32 to post that data publicly:p, but regardless of the reason, it is another problem that needs to be fixed.

Thank you.
 
File permissions we can reset with a ticket.

Yes we are aware speeds are slower, that is due to remote SQL making added load on connections and cpu. We are working to migrate SQL servers (both mysql which is causing php to cpu load) and MSSQL (causing high connections).

monitoring is due to be filly revamped for a long time now but we haven't done it due to the migrations planning, delays in that, then it is happening now and will be revamped. It is not having all services there publicly due to licenses limit reached on the software, and the fact that the master monitoring server starts to timeout itself due to load of additional monitors.

MSSQL12 is being prepped for migration now, along with some additional windows servers that mostly use MSSQL12, this will improve access greatly, along with the MySQL moves which are already prepped and just not quite completed.

We've also been open about win32 and win20 issues, and why they were chosen to move first, they are having for about a month RAID controller issues preventing them from rebuilding RAID properly, so we've kept good backups and moved as soon as we could. This will have a few teething issues, but long term will be much better.

In the worst case here with win32 I will throw some more CPU at it to help counter the php taking away a lot of MySQL remote connections. It will require a reboot to add the extra CPUs (but just a reboot, that is the beauty of the cloud platforms), and that is the main reason I've not done it yet. I made some maintenance changes and sites were loading a lot better after that.
 
The reason I reported the problem with last night's migration here rather than as a ticket was that it seemed highly unlikely that a error with migrating the entire server would be limited to only our files.

I also figured that you would want to correct the problem with last night's migration for the entire win32 server. Are you suggesting that rather than fix this migration problem globally, you are going to instead require all the users of win32 to each file individual trouble tickets once they discover the faulty server migration messed-up their files???

Moreover, if you do not find the cause of the migration problem now, it seems likely that this same mistake may be repeated when you migrate the other servers that hold our other client's web sites??? ?(

they are having for about a month RAID controller issues preventing them from rebuilding RAID properly, so we've kept good backups and moved as soon as we could.
Click to expand...

The performance problems we have been experiencing on-and-off with win32 have been going on a lot longer than a month. :(

But even if you discovered this a month ago...Look, if we found a bad RAID controller in one of our mission-critical PC's, it would get replaced THAT DAY. We would NOT let a bad hard disk controller fester for an entire MONTH, making backups and crossing our fingers that it didn't crap-out on us in the meantime!

That our hosting company would do such a thing, let alone with a PC that is handling client's web sites, distresses me more than I can put into words. :nono:

With regards to the slow performance, the web page that we were trying to view earlier did not contain ANY database code - MSSQL, MySQL or otherwise - it was virtually straight HTML. So it doesn't seem to be limited to just database accesses.

If what you are saying is not that SQL access is slow right now, but that you migrated servers containing heavy-use SQL sites while leaving all their data in Florida, and the new servers are not beefy enough to handle the processing load caused by this migration choice, thus resulting in ridiculously slow performance for ALL users of win32???

If this is the cause, then your idea of adding extra CPU horsepower is a good one. If that doesn't work, then I would suggest trying one of the following -

A) Try to limit the CPU load on any heavy-traffic database site on win32, so that they do not drag down everybody else's web site; or

B) Move the heavy-traffic database site(s) to a separate server until you can rectify this; or

C) Disable the highest-traffic database sites on win32 until this problem can be resolved, as it is better to have a few clients mad at you, than every client using this server!

But you need to do SOMETHING to fix this, as the performance is so ridiculously bad as to constitute a loss of service. My God, performance of win32 was so bad as to make free hosting sites look like a superior alternative!!! 8o We have a meeting with our largest client late today to show-off our latest progress, and I shudder to think how they will react if win32 is anything like it was this morning!

UPDATE ON MIGRATION PROBLEM - Testing has revealed that attempts to programmically update or write to migrated win32 local database files by ASP web pages are ALSO broken post-migration. When an ASP program attempts to update a local database, the web page blows-up with an error, stating that the database is read-only.

Hmm...I'm wondering if a tech changed all the server user files and/or web directories to Read-Only (for the same reason they also disabled ftp access - to prevent file changes during final file sync), and then forgot to change them back after file sync was completed???

That might be a good starting-point for diagnosing the cause of this server migration error - both to correct it, and also to prevent it from happening again when you migrate the other servers.
 
ArlingtonMall said:
Hmm...I'm wondering if a tech changed all the server user files and/or web directories to Read-Only (for the same reason they also disabled ftp access - to prevent file changes during final file sync), and then forgot to change them back after file sync was completed???

That might be a good starting-point for diagnosing the cause of this server migration error - both to correct it, and also to prevent it from happening again when you migrate the other servers.
Click to expand...
Calm down, it is just what rsync does when copying the files. I know it is a pain and we know it happens and the resolution, it doesn't happen on every file, but does happen on many. the read only flag can get done before the final sync of data as well and hold over sticky.

no the RAID was not totally broken, it was rebuilding and failing several days later. your data was secure via multiple means, including synced to remote servers, and a local server as well, and then backups.

We are going through and fixing the read only on all files that have it, but it takes time for that to process and it is going, in fact it could already be solved now.

Setting the permissions also helps the speed, so if you get a ticket in it can be resolved sooner, in fact the read only issue corrections are helping the server as a whole on non SQL queries.
 
Just to clarify one more thing, we've done this now on over 15 windows servers and we know many of the stumbling blocks that occur, the server is in no way underpowered, in fact it is more than 8x powerful than the last server. I don't mind you questioning things, and we are very open about it.
These servers yes had a RAID issue but were in fact more protected than any others, we will continue to migrate servers in this manner, this was the first to have read only issues this widespread, but it won't be the last to have this problem I am sure.
The same issue can occur on the user level moves as they also use rsync, that is what you proposed by the 'those with heavy sql' and in general we have no way to profile such without really prying into user data which we have no need, reason, or care to do. (not to mention sheer time such would take)
 
You must log in or register to reply here.
Back
Top