29
Jan
2010
15:41

There are currently faults at CHORLTON, DIDSBURY, RUSHOLME, STOCKPORT and ALTRINCHAM. exchanges BT are being chased.

Posted by admin

28
Jan
2010
09:20

An upgrade carried out by one of our transit providers last night produced a series of brief outages as a result of new firmware causing issues. This resulted in several reboots of the switches affected. We apologise for the inconvenience this may have caused.

Posted by admin

20
Jan
2010
11:13

A brief post-mortem of this morning’s events.

We have layer 2 links between our data centres, for core routing, with backup tunnels over transit. In the early hours, the link to Amsterdam, provided by Packet Exchange, suffered a brief outage. Unfortunately, due to an error in the backup tunnel configuration, the backup tunnel started to flap. This tripped a bug in our core routers which killed OSPF (which is how our core routers negotiate routing between themselves). This resulted in our routers in London and Maidenhead announcing routes to Amsterdam (and vice-versa) that weren’t actually working. The result was that if you tried to access machines in our Amsterdam data centre from source IPs that would normally route via London or Maidenhead, then you would fall into a black hole. And because OSPF had now died, routing didn’t recover when the Packet Exchange link came back up.

The solution was to reboot the routers, which restored OSPF and everything returned to normal. We have now identified and rectified the config error, so it shouldn’t happen again. To be absolutely sure, we intend running some intrusive tests on the live system, to make sure it now fails over correctly. Sadly this will cause some brief routing outages (which we will announce in planned maintenance). Because of the complex sequence of events that led to this failure, we believe that testing the live system is the only way to be sure we have fixed it.

If you have any further queries please contact support.

Posted by admin

20
Jan
2010
08:22

Services are now back to normal. The earlier Packet Exchange outage triggered a problem with our core routers that was causing them to create network “black holes”, whereby our routers in London and Maidenhead were announcing routes to Amsterdam that they couldn’t provide. This meant that our Amsterdam data centre was accessible from certain IP addresses, and not from others. This has now been rectified, and we are investigating the cause of the problem. Once again apologies for any problems this has caused.

Posted by admin

20
Jan
2010
08:10

It seems an earlier outage of our Packet Exchange transit links triggered a problem with our core routers which means that our Amsterdam data centre lost connectivity. We are working on this, and hope to restore connectivity shortly. Apologies for this

Posted by admin

20
Jan
2010
07:38

Our Amsterdam data centre appears to be suffering an outage, we are investigating…

Posted by admin

08
Jan
2010
15:14

Our services are running normally, however we are still having problems shipping equipment.

Posted by admin

06
Jan
2010
09:01

The severe weather conditions are having no impact on our services other than our ability to ship equipment.

Posted by admin

Current Status

    GoodNo current issues

Status Archive