You may have seen packet loss early this morning, between 01:40 and 07:15. This was due to a faulty switch in our main transit provider’s rack, which has now been reset. Apologies for any inconvenience.

Here’s a more detailed post mortem of this morning’s events. First the power went off at Redbus for 15 minutes from 00:20 – this really should not happen, and we are trying to get answers from Redbus. This resulted in loss of connectivity until 01:25. Then at around 06:54 a switch in our main transit provider’s rack started flooding our network with broadcast packets, causing packet loss until it was reset at around 08:30 – we are not sure whether this was related to the power outage. Then finally BGP routing died between 09:04 and 09:41, as a result of side effects of the earlier power outage. Everything is now back to normal, and we are planning changes to reduce the possibility of this happening again.

We suffered a loss of power in our data centre this morning, which will have affected all customers. All services are now back to normal, and we will be asking Redbus some serious questions. Apologies for any inconvenience.

You may have seen massive packet loss this morning up to around 09:15. This was due to a faulty switch in our main transit provider’s rack, which has now been reset. Apologies for any inconvenience.

We suffered an outage of our main transit provider this afternoon, between 16:42 and 17:08. This was due to a power outage in one of their racks. It will have affected most of our customers. Apologies for any inconvenience.

ADSL customers may have noticed a brief outage of a few minutes at around 15:30 today. We believe it was within BT’s network. All is now back to normal and we apologise for any inconvenience.

The backup transit feed is now up and running again. Sorry again for any inconvenience or poor performance.

Our backup transit provider is currently suffering a DOS attack which spilled over into our network, causing packet loss. The backup feed is temporarily suspended until this is resolved. Apologies for the poor performance between 17:30 and 18:30 – more later.

There was a major outage at our data centre this afternoon, between 16:40 and 17:20, which would have affected all ADSL and colocation customers. We apologise for any inconvenience this will have caused.

Our primary authoritative dns server (ns0.watchfront.net and local caching server 217.169.24.209) was down for about an hour this morning. During the outage the secondary dns server (ns1.watchfront.net and local caching server 217.169.31.244) provided backup dns as expected, so if you experienced problems please check that your machines are configured to use both servers. Apologies if anyone was affected.