29
Jun
2008
17:19

Just before 5:00am this morning we suffered a power failure in our Maidenhead datacentre affecting all equipment. The outage lasted approximately 30 minutes. Apologies to all customers with services provided from Maidenhead. London and Amsterdam were unaffected.

The datacentre have issued the following statement:-

Following on from the issues this morning, we are continuing to work with suppliers to find the cause of the outage.

Current investigations appear to have identified a phase detection relay which may have failed in the generator auto-changeover panel. This could have caused the panel to think there was a mains power failure when in fact there had not been.

The UPS systems worked as expected without fault, however due to the mains being forced off by the potentially failed relay, and power from the generators also locked out, the UPS batteries fully discharged after 10 minutes resulting in a critical power failure.

We have engineers collecting a new relay, along with other spares that maybe needed from the manufactures in North Yorkshire. We expect these to arrive in the early hours of tomorrow morning.

During this time we have manually disabled the mains auto changeover system in order to aviod any potential further mains cuts. In the event of a real power outage on site we will have staff perminatly onsite to operate the generator changeover procedure manually.

This should be treated as an ‘at risk’ period for customers in BlueSquare 2 & 3 until further notice.

Once we have further information we will update this ticket.

We apologise for any inconvenience caused.

Posted by admin

27
Jun
2008
11:46

An update on yesterday’s 10 minute outage – this was due to a massive DoS attack that took down our L2TP connections. Over the past few years we have been improving our resilience to attacks, and we have been riding most attacks without any noticeable affects. However the magnitude of this latest attack was a first for us. The good news is that we are developing new L2TP boxes that should ride even this magnitude of attack.

Posted by admin

26
Jun
2008
15:01

Some ADSL routers have failed to log back in after the outage. If you are still having problems, please try power cycling the ADSL router. If that doesn’t help, please contact us and we will investigate further

Posted by admin

26
Jun
2008
14:57

Our ADSL services suffered an outage at 14:15 today, of around 10 minutes duration. It appears that this was due to an attack that took down our L2TP connections with BT. We are investigating exactly what happened, so we can take steps to avoid similar problems in the future. Apologies to all those affected.

Posted by admin

26
Jun
2008
14:32

Services are now returning to normal. I will post more details shortly. Apologies for the outage

Posted by admin

26
Jun
2008
14:27

We are currently suffering a major outage More news shortly

Posted by admin

19
Jun
2008
09:28

There was a brief problem with routing around 9am this morning, that may have affected some customers. It was due to a problem with BGP that has now been resolved. Apologies for any inconvenience this may have caused.

Posted by admin

11
Jun
2008
12:00

We are currently seeing low levels of packet loss on some ADSL lines. We are working on this, please bear with us.

Posted by admin

Current Status

    GoodNo current issues

Status Archive