Service Status

27/06/07
11:17

Auto Draft

Our main transit provider (AAISP) now has a new primary core router installed, to replace the troublesome Juniper. After some thorough testing it looks as though both the primary and backup routers are working well, and this should be an end to the problems we had over the last couple of days. Thank you for your patience.

Posted by Paul Malkowski
26/06/07
11:08

Auto Draft

In view of the routing problems experienced over the last couple of days, we are working with our main transit provider to urgently replace the core routers over the next few days. Naturally we will do our utmost to prevent any disruption, though we may be running on backup routers at times. As ever we are striving to provide the best possible service.

Posted by Paul Malkowski
26/06/07
10:57

Auto Draft

There was a burst of severe packet loss around 10:40 this morning. This appears to have been another BGP related problem for our main transit provider, AAISP. We are currently working with them to replace the core routers. In the meantime we apologise for any inconvenience this has caused.

Posted by Paul Malkowski
26/06/07
10:40

Auto Draft

We are currently seeing serious packet loss on Level 3 transit, investigating

Posted by Paul Malkowski
25/06/07
14:59

Auto Draft

An update on today’s two outages.

The first outage was due to the failure of a core router. Our main transit provider (AAISP) runs dual redundant core routers, using VRRP to switch between them. The primary router failed in such a way that it was still announcing VRRP (so the backup router didn’t take over), but was not routing traffic. This is a disastrous failure mode for a router, and they are investigating how it could happen. The result was a routing outage that would have affected most if not all of our customers.

The second outage was due to a BGP failure between the L2TP box our DSL customers are logged on to, and the upstream router. This resulted in a 7 minute outage for all our DSL customers, but did not affect other services (hosted servers etc.). It was unrelated to the first outage, and we are investigating what caused it.

We do apologise for any problems this may have caused, and we are planning changes to our network to improve resilience to these sorts of failures.

Posted by Paul Malkowski
25/06/07
13:56

Auto Draft

Our main transit provider has suffered another routing problem between 13:42 and 13:49, which as far as we can tell was unrelated to the earlier problem. We will post more information here as it becomes available. Apologies for any problems this has caused.

Posted by Paul Malkowski
25/06/07
13:12

Auto Draft

We have suffered a routing outage from 12:49 to 13:08 today. We are investigating the cause and will post more information shortly.

Posted by Paul Malkowski
13/06/07
09:57

Auto Draft

We have been seeing some low levels of packet loss on broadband lines. We are investigating although the situation has returned to normal, as this is not the quality of service we aim for.

Posted by Paul Malkowski
12/06/07
09:16

Auto Draft

There was a DoS attack at 9:04 this morning which affected our L2TP box, thus affecting all our ADSL customers. The outage lasted about 5 minutes, and all services are now back to normal. Apologies for any inconvenience this may have caused you.

Posted by Paul Malkowski