Core network problems
Since approximately 22:00 on 05/06/24 our core network has been exhibiting severe packet loss. Investigations are ongoing, but we have no ETF yet. We apologise for the poor performance.
Update:-
We believe the problem last night was caused by a flood of TCP SYN packets. They don’t use much bandwidth, but do use a lot of CPU resource, leading to the packet loss. We are working with the FireBrick development team to see what improvements can be made to mitigate the effect should it occur again.
Update on today’s connectivity issues
Following the second outage this afternoon we undertook an emergency investigation at our data centre to determine the root cause of the outages today.
We apologies for the poor/non-existent connectivity customers experienced.
The cause was found to be a rogue machine that was sending a large amount of traffic where it shouldn’t have, and in the process tripped various router and switch limits. It has now been fixed, the trips have been cleared and customers would have seen connectivity restored around 3pm this afternoon.
We will as always continue to monitor.
Thanks for your patience.
Update – Connectivity issue 28/09/23
Update: Customers should now be seeing their services restored.
We restarted a core router which has brought services back.
We will continue to keep a close watch on things.
We’d like to thank customers for their patience whilst the issue was resolved.
We are aware that some customers may be experiencing problems with connectivity this morning.
We are investigating the issue and are working to resolve the problem as quickly as possible.
We apologise for any inconvenience caused.
Amsterdam Switch Issue
We had an issue with a switch in Amsterdam this evening for around 2 hours.
This started after 5pm where a switch started to constantly reboot itself until remote hands were able to stop this.
We apologise for any inconvenience this may have caused.
We have scheduled maintenance to replace the switch and restore full resilience.
If you have any question please email support@watchfront.co.uk .
Link to Amsterdam
Hello,
We have had an issue with a link from London to Amsterdam overnight and this morning.
This was causing issues with tunnelled connections which we have re-routed.
We are working with our supplier to repair the link and will update you once done.
There may be some issue with some hosting in Amsterdam until this is resolved, we apologise for the inconvenience this is causing.
UPDATE 1pm 11/02/2023
We have managed to restore the network and everything is running as it should.
We apologise for this issue and are in talks with the supplier to avoid this issue being repeated.
Have a nice weekend.
Hello,
After the rack migration to Pulsant 2 we now are in a position to consolidate the network and remove the now extra routing node.
On Wednesday (12/10/2022) between 6pm and 7pm we plan to complete some switching and routing configuration changes.
This will have a minimal outage for some colo customer and some tunnelled connections for around 5 minutes.
If you have any questions please let us know via email – support@watchfront.co.uk.
Hello,
On Sunday 25 September 2022 we will be moving racks in one of our data centres.
Customers with servers who are directly affected by this have been contacted.
Furthermore people who utilise our tunnelling services will experience a short outage as the routers are moved.
We will be starting work on this at 10am and hope to be finished in a couple of hours to minimise downtime.
We have the maintenance period until 6pm as a contingency although we do not expect there to be any issues.
If you have any questions please email support@watchfront.co.uk.
Hello,
One of our suppliers of Leased Lines will be conducting maintenance on their connection with us between midnight and 8am on 05/10/2021.
There may be outages to your leased lines during this time.
We apologise for any inconvenience that this might cause.
If you have any questions please email support@watchfront.co.uk.
Hello,
We will be doing some testing on a transit link at midday on Sunday the 21/03/21.
This will be for 1 hour.
Although we do not foresee any issue with this work we are giving a courteous “At Risk” notification for this.
Please contact us if you have any questions.
Hello,
Pulsant will be carrying out maintenance on power feed Z to our racks in Pulsant 2 between midday and 4pm on Saturday 13th of February.
There will be disruption to the power feed during this time.
We will be in contact with every customer who has single feed equipment that will lose power during this maintenance window.
As this is the Data Centre owners who are doing this we can not change the time of this maintenance.
If you have any questions about this please get in touch by emailing support@watchfront.co.uk or calling 020 7517 4999.