Network problem - Craft Packet Attack - one subnet still down.

Stephen

Stephen

US Operations
Staff member
We are seeing about 40-50% traffic drop off at the moment, not complete so a lot is working, but there is for sure some issue in routing as a lot is fallen off in traffic. We are checking it now.
 
Working on it now, from outside it looks like 10-20% packetloss, and some sites loading slow, but loading.

We're trying to get to the bottom of it nothing happened on this side.
 
traffic is now going up on the graph but some of outside showing higher packetloss rates, interesting, we are still working the matter.
 
It went up in traffic, then back lower than before, we are still working to find the cause.
 
This isn't a good day, seems it is one of the core distributon switches acting up, we are going to get a replacement in for core aspects and get main links switched over ASAP. We made some changes and it was better for a short time, then back to bad packetloss.
 
Well, we've moved all uplinks, and it looks the same......this is a really great day. The switch when we rebooted it showed activity normals, so we thought it had to be the switch since it then faded off to 50-60% of norms. All debug data looked good however.
 
We have not given up, and won't give up, even though I am very frustrated here I am working to stay calm and collected and work thorugh every possible scenerio.

This happened when no one was in the space, so we know nothing changed physically.
 
  • Deleted by Stephen
Show…
We are going to remove every switch from the network for a few moments while we do an isolated network segment test.
 
We are working to bypass the platform that has the most probable cause of todays problems.
 
Some days nothing goes right, that would be, TODAY for us.

Running into problems at every stage, we do know what is going on, internal routing is messed up, we've tried 2 different workarounds so far without success, as we designed this to work right and not fail. It seems the software platform used for the routing has a glitch making this failure. It is auto rebooting very often, so it works, fails, fails over to other machine, works a few, fails, other one comes up and it works, failed, reboots, in a continual loop which you can see just by pinging.

This is a crazy system of fails that simply should not happen, but it is and we're having to defeat it with some creative solutions now.
 
Going to take another route, I am going to purchase some new supplies for it and be back ASAP. Network is 'up' right now but between 25-45% packetloss depending on route.
 
We are going to do one test now to make sure what we thought was the problem, was the case, and then resolve it, this will help us prevent from happening again.
 
Win6 Cluster1 shared IP is down at the moment as it is taking a rather large attack of crafted packets, not that large, but making havoc on the routes due to the errors mentioned above causing the reboots.
 
The 199.127.218.xxx subnet is down right now, as the attacks are still coming in very heavy. Working to find a filter that works properly for it.
 
All subnets up. Issue resolved. We sincerely apologize for the inconvenience caused.
 
You must log in or register to reply here.
Back
Top