We're seeing a very random issue with some http requests failing

Stephen

Stephen

US Operations
Staff member
Across all servers we are seeing a very random issue with some HTTP requests failing to load. Not a long timeout or anything but a random fast fail, and then a refresh it works properly every time, sometimes even the browser auto refreshes before we have a chance to do so.

We're getting some alerts based off this occasionally and checking into the matter now. Traffic flows all look within normal levels and not to be out of the ordinary.
 
This is still happening some. It's very random, not a server side issue, and can't see any issue on the routing side in our control either, but we're continuing to check. We've gotten a few clients reporting it, but a refresh fixes for them as well.
 
We've been working to resolve some issues mostly seen in monitoring systems showing HTTP connection refused errors at times. We're not seeing any errors on routing or switching after extended checks, and don't believe it's a routing issue but going to restart the router just to make sure. There will be a 8-12 minute emergency outage during the next 30 minutes.
 
And, that did not help just as we had expected because we are certain issue is not on our side.
 
The issue is still happening some, we're still checking it and working with upstream network engineers to find out why the connections are bounding around between a couple routers way too much and not even making it in for some routes. Overall traffic levels are normal level, so the impact is not great except on monitoring side where it shows a lot of false outages, when you actually load it up it will load in most cases. There are a few that seem to be impacted more than others.
 
The issue is completely resolved now with some upstream changes in how the BGP is working and seeing all routes working in all places like it should. We will have a scheduled maintenance period in the future to make some corresponding changes on our side to increase resiliency and prevent such from happening again in the future.
 
You must log in or register to reply here.
Back
Top