Update: The issue with the router has now been resolved. Service is back to normal.

One of our routers is experiencing problems. We have despatched an engineer to Maidenhead.

We apologies for any inconvenience caused.

We will update as soon as possible.

We have finally received an all-clear from Pulsant that the earlier problems have been resolved. They have promised an investigation will be started on Monday – we will publish a summary with as much detail as possible later next week.
Apologies once again for the disruption and thank you for your patience.

Pulsant have announced that they will be replacing the faulty equipment they believe to be at the root of this evening’s problems imminently. There *may* be a further outage whilst this occurs, but after that normal service should be restored.

Apologies to all customers for the outage this evening at Maidenhead. Both our Layer2 (backhaul) and Layer3 (transit) connections through Pulsant had failed. These services are supposedly supplied through separate equipment, so we will be making enquiries next week as to the circumstances which caused both to fail.

We are currently experiencing problems with our transit provider in Maidenhead, Pulsant. We have manually de-peered with them so all traffic is currently flowing via our Amsterdam datacentre. Apologies for the loss of routing – we will re-connect with Pulsant as soon as we are assured that it will be safe to do so.

Update: Normal operation has now been restored. Should customers still have any problems then please contact us immediately. A technical note update will be posted explaining today’s service issue.

There is currently an issue with one of the switches not responding in the cluster housed at bluesquare 2. An Engineer is being despatched to bluesquare to replace the faulty switch.

We apologise to any customer who may be affected during this time. We will update when more information becomes available.

There is currently an issue with some BT connections. The problem is outside of our control hower as soon as we have an indication of a time to repair we will let you know. Sorry for any inconvenience this is causing.

I am pleased to say that the core router upgrades went to plan today – we believe no packets were injured during this period.

The exploit that has been occurring on and off all weekend was diagnosed to be a series of malformed TCP port 80 packets – i.e. probably trying to exploit weaknesses in webservers. The FireBrick team have issued new firmware to protect against this and we have accordingly updated all or FireBricks except our core routers.

Customers using FB2500 or FB2700 models with default settings will see their bricks have updated firmware automatically. Anyone who has disabled auto-updates may have had ongoing problems all weekend – the solution is to update to the current factory firmware.

We are planning an update to our core network routers, which should not be disruptive but must be considered an ‘at-risk’ period. Until this has been completed, the network status will remain on Amber. Please see the planned maintenance page for details for the core upgrade.

There currently seems to be an exploit attack which is causing FireBricks to spontaneously reboot. Fortunately they do so very quickly, but this has resulted in many brief outages as parts of our core network are resetting.
The exploit has been understood and the FireBrick team have already issued a new release of firmware to avoid it. We will be deploying this across our core as soon as feasible.