Network inaccessible on some routes - Resolved

[tanmaya]

A few of our clients have reported parts of our network being inaccessible. This is just to let you all know our network is all up and fine, and you can use external websites like (alertra .com & anoymouse .org) to verify accessibility of your website(s). Meanwhile we are looking further into the problem here.

 

[tanmaya]

Re: Network inaccessible on some routes

Issue was isolated to one of our exit routes, and we have been working to get it resolved as soon as possible with the provider responsible.

 

[tanmaya]

Re: Network inaccessible on some routes

The issue appears fixed. We are still awaiting details from Engineers of the respective providers.

 

[tanmaya]

Explaination of this routing issue

SUMMARY:

One of the peers of our upstream made a change last night to their routing config which filtered JodoHost IPs from the advertisements they receive from their upstream carriers which meant any traffic on their network had no path to get back to the JodoHost IP’s.


DETAIL:

Any IPs advertised to this peer are filtered by them from the routes they receive from their upstreams to prevent circular routes. However this peer provider had two different versions of BGP Filter lists for some subnets which were supposed to be the same, but were out of sync due to a legacy copy, so they were filtering some routes from their carriers that they were not receiving or even allowing to be advertised to them.

On Wednesday night an engineer there was adding a new carrier when he noticed that he was not applying this filter list against one of his existing carriers, so he “fixed” the problem by applying it to this carrier (which happens our return path), like his existing carriers were. Doing this filtered the last routes for those IPs from his network and made those IPs unreachable.

RESOLUTION:
The problem was corrected by removing the IPs from their carrier filter list.