Hi there,
Can you tell us what happened yesterday and if it was finally fixed in its entirely?
Do you plan on sending any type of updates/comments by e-mail when stuff like this happens?
Thank you very much,
Ezequiel
We will do a post problem summary here on the forums and link it on twitter like we’ve done in the past. We know what happened, what was targetted, and how to work with the upstreams nowt o filter it before it gets to us should such happen again. It will take a couple days for me to get that together as I have a few business appointments arranged.
We are just an hour and a half north of Austin, TX where HostingCon is, and have a number of meetings setup with vendors there.
i too would really like to hear what happened ASAP- even an initial summary, i have a major client breathing down my back, who chances are i will lose if i dont give them plausible reassurance very soon.
thanks
snooper,
issue is there on the forums in status.
Craft Packet attack. “internal IP” (aka our IP range) IP coming in from internet side as “source” of attack. Packet spoofing. These spoofed packet were coming in very heavily, not large in megabit, but in amount of packets.
We worked on filters for along time and none fully stopped it from coming in until we got the upstreams to filter out internal IPs from coming in via internet as source IP, not destination.
that is a quick sumary, it is rather complex and even the NOC techs (not the IP network engineers) didn’t understand it, and they deal with IP routing and such all day long. Tanmaya and I got a grasp of it from the logs and realized this was causing the routers to go wild, restart, and not route traffic properly in general.
got it. thanks for working so hard on this