Core network problems
Update:-
We believe the problem last night was caused by a flood of TCP SYN packets. They don’t use much bandwidth, but do use a lot of CPU resource, leading to the packet loss. We are working with the FireBrick development team to see what improvements can be made to mitigate the effect should it occur again.
Update on today’s connectivity issues
Following the second outage this afternoon we undertook an emergency investigation at our data centre to determine the root cause of the outages today.
We apologies for the poor/non-existent connectivity customers experienced.
The cause was found to be a rogue machine that was sending a large amount of traffic where it shouldn’t have, and in the process tripped various router and switch limits. It has now been fixed, the trips have been cleared and customers would have seen connectivity restored around 3pm this afternoon.
We will as always continue to monitor.
Thanks for your patience.
Update – Connectivity issue 28/09/23
Update: Customers should now be seeing their services restored.
We restarted a core router which has brought services back.
We will continue to keep a close watch on things.
We’d like to thank customers for their patience whilst the issue was resolved.
We are aware that some customers may be experiencing problems with connectivity this morning.
We are investigating the issue and are working to resolve the problem as quickly as possible.
We apologise for any inconvenience caused.
Amsterdam Switch Issue
We had an issue with a switch in Amsterdam this evening for around 2 hours.
This started after 5pm where a switch started to constantly reboot itself until remote hands were able to stop this.
We apologise for any inconvenience this may have caused.
We have scheduled maintenance to replace the switch and restore full resilience.
If you have any question please email support@watchfront.co.uk .
Link to Amsterdam
Hello,
We have had an issue with a link from London to Amsterdam overnight and this morning.
This was causing issues with tunnelled connections which we have re-routed.
We are working with our supplier to repair the link and will update you once done.
There may be some issue with some hosting in Amsterdam until this is resolved, we apologise for the inconvenience this is causing.
UPDATE 1pm 11/02/2023
We have managed to restore the network and everything is running as it should.
We apologise for this issue and are in talks with the supplier to avoid this issue being repeated.
Have a nice weekend.
Internet lines issue
We are aware of an issue with customers accessing the internet this morning.
There is a problem with a switch in one of our data centres and we have staff on route to fix the issue.
Congestion and Latency
Hello,
There was some congestion and high latency on some of our DSL lines this afternoon.
This coincided with the football.
Our supplier is investigating the cause of this problem and have assured us there wiil be fix in place soon.
Apologies for any inconvenience that may have occurred during this.
DSL Interruption
Hello,
We experienced an interruption to DSL service from one of our suppliers where some lines disconnected for around 5 minutes.
We apologise for this and have sought an explanation from our supplier as to the cause of this.
All lines are back up and running normally now.
Firewalls
Hello,
We had a brief outage as two of our firewalls rebooted at around 1:15pm this afternoon.
The down time was around 3 minutes and we apologise for any inconvenience this may have caused to colocation and/or leased lines.
There was no outage with DSL services.
We are investigating this incident to help prevent it from happening again.
DSL Supplier Outage
One of our suppliers had a DSL outage this morning and some of our customer were affected. All services have recovered before 9:00 AM.
We regret any inconvenience this may cause.
If you have any questions, then please contact support@watchfront.co.uk or call us on 020 7517 4900
Issues with some DSL lines
Hello,
One of our providers is having issues with their network which is causing outages to some but not all ADSL, FTTC and FTTP lines from them.
They are currently work on resolving these outages.
We apologise for the inconvenience this is causing.
We will update you with progress of this soon.
Update 14:34
Some of the outages are now back in service, we expect more to return very soon.
Thank you for your patience.
Update 15:26
All lines are now working.
Network Schism
Around 4pm yesterday we suffered a network schism where some of our leased lines and some servers in Pulsant 2 suffered packet loss.
The leased lines issue was fixed quickly while the colocated sever issues took a few hours to fix.
We apologise for the inconvenience this caused and we have now implemented a new support procedure for this happening again in the future.
Low Level Packet Loss
We have been experiencing some low level packet loss on some fibre and DSL lines this morning between 7am and 9.40am.
We have identified that one of our cross connects between London and Maidenhead was flapping and has now been taken out of service.
We have spoken to our provider and plan to get this fixed asap.
There is no packet loss after removing the flapping link from service.
We apologise for this inconvenience.
UPDATE 27/11/2020
As of 10am this morning we reinstated the cross connect and everything is fine.
The issue was part of a problem with an upstream provider connection going down which was fixed late last night.
We offer our apologies for any inconvenience caused by this.
Power Outage in HEX 8/9
There has been a power outage in Harbour Exchange 8/9 in London.
This is ongoing with a eta of 1 hour to be fixed.
All leased lines and any connection between Amsterdam and Maidenhead are effected by this outage.
All FTTC and ADSL lines switched to the secondary route and are still working.
We can only apologise for this outage.
We will update you here once we have more information.
Update 12:00 18/08/2020
We have been informed that power is being restored to customers via a new UPS infrastructure in HEX 8/9.
This is a gradual process and is taking more time than was anticipated.
Once we have power back we will have service again.
Apologies on the length of time it is taking for this to be fixed.
Update 08:00 19/08/2020
Services started to return at 19:30 last night with Amsterdam and the majority of leased lines being operational again then.
Other leased lines returned before 2am this morning.
We apologise again for this interruption.
Brief connectivity outage
We apologise to customers for the brief outage that affected some TalkTalk services today just after noon.
Services resumed for those affected shortly afterwards.
Watchfront’s Covid-19 Protocol – Update 20/03/20
Further to our previous post about the current situation with the Covid-19 virus we are giving another update in relations to any data centre works that may need done during this time.
Yesterdays post can be found here:
https://panator.watchfront.net.uk/watchfronts-covid-19-protocol/
As this is a fluid situation where the situation is changing daily we will be updating you when there are changes to what is possible Access to the data centres, in both Maidenhead and Amsterdam, is allowed but advised to be only for essential maintenance. There are no firm restrictions on access at this time.
We will be working as normal for anything in Maidenhead where we will visit in person. Currently Amsterdam is reachable but maybe not advised unless in absolute emergency.
We can however employ remote hands where such things as disks changes can be done with out a visit from us. Both of the data centre owners are stating like us that their stance is based on the latest advice from the government and is subject to review and change.
If you have any queries about this please email support@waganator.watchfront.net.uk
Power Outage at Pulsant 2
Hello,
We have experienced a power outage in our Maidenhead Pulsant 2 data centre. Power has returned and we are working to get all services back online.
Updates to follow.
Brief Connectivity Outage
We apologise to customers for the brief outage caused by one of our suppliers between London and Maidenhead.
Everything is back to normal now.
Connectivity Outage
At approximately 23:30 on 06/03/19 we lost connectivity between London and Maidenhead, affecting many services. The link is comprised of 2 connections from different suppliers. Unfortunately both of these connections were unavailable for a period, for reasons that are not currently fully understood.
We are working with both link suppliers and will post a fuller update when further information is available.
All services are now restored – if you should see anything unexpected please contact support on 020 7517 4999 or support@waganator.watchfront.net.uk
Aplogies for the inconvenience caused.
Interruption to internet lines
Apologies to all our customer who have lines with us and experienced packet loss. There was an issue with one of our connections in London which has now been fixed.
Interruption to Service
Apologies to all Leased Line and DSL customers who suffered a brief outage around 13:45 today.
This was caused by an interruption between London and Maidenhead by our suppliers having a network glitch.
Interruption to Service
Apologies to our customers who suffered a brief outage this morning.
Unfortunately, whilst Kev as conducting some housekeeping work in Blue Square 1 he accidentality rebooted the Peregrine switch. As this was a totally accidental occurrence we do not anticipate there to be a repeat.
We apologise for any inconvenience this may have caused.
Switch problem in Maidenhead
Update 07/09/18: The faulty unit was replaced yesterday and has been functioning normally sine.
One half of switch cluster ‘willem’ in Maidenhead is currently unresponsive. Any customers with single-homed CoLo connected to the faulty unit will currently be suffering loss of connectivity.
We will investigate and repair as soon as possible and will publish an update in the morning.
In the meantime, apologies for the inconvenience.
Interruption to Service
Hello,
We suffered a DDoS attack on our network in the early hours of Monday the 28th of May.
One of our upstream providers noticed this and null routed the attack.
All services were back to normal in under two hours.
We apologise for any inconvenience this may have caused.
:2018-04-10 Glitch on TalkTalk services this morning
Update: All affected lines have now resumed normal service.
Customers who have a TalkTalk connected service may have experienced a glitch in service for a minute or so at around 11:00 this morning, services are slowly returning to normal to all affected lines.
The cause of the glitch is unknown at this time but is being investigated, as soon as we have more information from TalkTalk we will post an update.
We apologise for any inconvenience caused.
:2018-01-25 Glitch on TalkTalk services this morning
Customers who have a TalkTalk connected service may have experienced a glitch in service for a minute or so at around 11:00 this morning, normal service is now running.
The cause of the glitch is unknown at this time, as soon as we have more information from TalkTalk we will post an update.
We apologise for any inconvenience caused.
Issue with TalkTalk lines
This morning TalkTalk are having a big outage which is in turn is making any EoFTTC or Leased Line from them unavailable.
We apologise for this problem and are assured TalkTalk are working to fix the issue as soon as possible,
We will update you here once we know more.
Update 09:13 27/11/2017
All lines have come back now and the problem is gone.
Apologies once more for this issue.
:2017-11-17 Problems with BT DSL connections
We apologise to our customers with BT lines who have been affected by the outage this afternoon. Our wholesale supplier for BT connected lines is investigating the cause of the issue.
Apologies to those customers affected.
:2017-08-29
We are observing packet loss affecting cartain data paths in our AMS datacentre, which is being investigated.
:2017-08-11 Glitch on TalkTalk services this morning
Customers who have a TalkTalk connected service may have experienced a glitch in service for a minute or so at around 09:00 this morning, normal service is now running.
The cause of the glitch is unknown at this time, as soon as we have more information from TalkTalk we will post an update.
We apologise for any inconvenience caused.
:2017-07-20 Problems with BT DSL connections
We apologise to our customers with BT lines who have been affected by the outage this afternoon. Our wholesale supplier for BT connected lines carried out investigative testing following the issues last week and unfortunately made an error with the configuration on one of their switches. The configuration change has been rolled back and the service to customers restored.
Apologies to those customers affected.
:2017-07-14 Problems with BT DSL connections
Update 12:17 – All lines have now reconnected and been stable for approximately one hour.
Our wholesale supplier for BT connected lines is experiencing severe problems this morning. Once they have rectified the situation we expect affected lines to reconnect automatically. We will monitor to ensure this is so.
Apologies to those customers affected.
Problems with BT DSL connections
Update 21:50 – All lines have now reconnected and been stable for approximately one hour.
Our wholesale supplier for BT connected lines is experiencing severe problems this evening. Once they have rectified the situation we expect affected lines to reconnect automatically. We will monitor to ensure this is so.
Apologies to those customers affected.
Network Outages
We are experiencing interruptions to our connectivity with one of our upstream providers, Pulsant, this afternoon.
As we use diverse suppliers for transit we switched the Pulsant transit off until they had fixed their issue.
We apologise for any inconvenience caused.
Network Outages
Update: Pulsant have notified us that they will be performing emergency maintenance between 21:00 and 23:00 this evening related to the earlier outage.
This maintenance should be considered as an ‘at risk’ time, rather than a definite outage.
Earlier message:
We unfortunately experienced network outages outwith our network earlier today.
Our infrastructure was working normally but the issues were with one transit provider.
The issue was with Pulsant; more information can be found at http://status.pulsant.com/
We apologise for the three blips these issues caused.
We have been notified by our Leased line/EoFTTC provider that TalkTalk have Planned Engineering Works scheduled to carry out an upgrade to some of their core links. They do not anticipate any impact to customers Leased line/EoFTTC services but it is advised that during the maintenance window it is an ‘at risk’ period.
The maintenance will take place during the following time window:
Duration:
Start: 19/10/2016 00:05 BST
End: 19/10/2016 06:00 BST
We have been notified by our BT wholesale provider that they will be carrying out planned maintenance to upgrade the software on their core switches. Our BT customers are advised to expect a disruption to their broadband service whilst the maintenance is carried out.
The details of this maintenance are as follows:
Start Date: Thursday 20/10/16
Start Time: 01:00
End time: 04:00
We apologise for any inconvenience this may cause.
We are planning to upgrade the firmware of our switch cluster Beatrix in Pulsant 2 on Tuesday 10th of January 2017 after 6pm.
This will involve the switches rebooting.
The time window for this work is 30 minutes.
We do not expect the work to take longer than 10 minutes but we are building in a buffer in case of unforeseen events.
All services will be affected during this time.
2017-01-17 18:17 – This maintenance has been completed successfully
We will be doing further firmware upgrades to our core switches on Tuesday 17th of January 2017 after 6pm.
This will involve switches the following switches:
Korenwolf
Peregrine
Jasper
Zwaluw
The time window for this work is 1 hour 30 minutes.
We do not expect the work to take longer than 30 minutes but we are building in a buffer in case of unforeseen events.
DSL and Leased Line services will be affected.
Connectivity to Pulsant 1, Amsterdam and rack 1 in Pulsant 2 will also be affected.
UPDATE– 02/04/2017 This work will be going ahead at 12pm today.
We have had to move this work back a week due to technical reasons.
We now plan to do a major change to our switching infrastructure in Pulsant 2 on Sunday the 2nd of April 2017.
The work will start at midday and is expected to last a maximum of 3 hours.
This will mean there will be outages to all services as the full network will be down during this time.
We do not expect the work to take 3 hours but we are building in a buffer for any unforeseen circumstances.
Our Planned Maintenance today was completed successfully.
Supplier Planned Maintenance
We have been notified that one of our inter-connectivity suppliers will be conducting some essential core network maintenance, on Tuesday 16th May 2017 between the hours of 20:00 and 06:00.
There may be a slight blip during this period for Talk Talk DSL connections until they reconnect automatically through a different interconnect.
This maintenance task is required so that Talk Talk Business can perform essential vendor recommended maintenance on their DSL infrastructure at Telehouse North.
On Thursday 25th of May 2017 we are planning to move our interconnect for BT DSL and FTTC lines at 18:30.
This is a low risk change although there may be a slight outage of 5 minutes in a worse case scenario.
Update 15:16 13/08/2017
The transfer is now complete and all access to mail boxes has returned to normal.
We can only apologise for the length of time this took.
Update 11.15 13/08/2017
Unfortunately this process is taking longer than expected.
We are monitoring the situation and expected the transfer to the new host to be complete in a couple of hours.
Thank you for your patience during this time.
We will be moving one of our mail servers to a new host on Saturday 12th of August 2017.
We are starting work on this on at 8pm and have scheduled a maintenance window of 12 hours for this to be completed.
There will be a delay in receiving mails through out this window.
There will be no delay in sending mails.
There will be updates sent before and after the work is done.
–Update
For clarity whilst the works are ongoing people will not be able to access their mailboxes.
Update 00:20 20/08/2017
This work has been completed successfully.
We will be moving one of our shared services servers to a new host this weekend.
Some customer websites will not be available during this time.
This work will begin at 8pm on the 19th of August 2017 and is expected to last 12 hours.
We will post updates as the work progresses.
We will be moving our email services to new servers on Sunday 29th of April 2018.
We expect that this work will take no more than 3 hours to complete.
During this time all connections to send and receive emails will be unreachable however no emails will be lost.
This work will bring upgrades to our services which we will highlight in a news article once the work is complete.
Update 22:32:
The updates have been applied successfully.
Hello,
We will be conducting essential firmware updates to our LNS and Tunnel Head End routers on the evening of 2nd July 2018.
The hours of this maintenance are between 10pm and midnight.
There will be slight interruption to all DSLs and tunnels during this period.
We apologise for any inconvenience this may cause.
Due to the recent issues with our connections to London we are doing some maintenance tonight after 6.30pm.
This is to make sure that we have redundancy between London and Maidenhead.
There may be a small outage at this time of around 5 minutes.
This maintenance has been completed successfully, with all resilient links now functioning correctly. 14/03/19 21:10 KRH.
We will be conducting more maintenance on our interconnects between London and Maidenhead at 9pm on Thursday 14th of March 2019.
We are confident that this maintenance will work immediately but there may be a small outage of up to 15 minutes during this time.
This would effect leased line customers and any connection between Maidenhead and Amsterdam.
We will be conducting some maintenance tonight at 18:30 on our edge router grappa in Pulsant 2.
This should take no more than 5 minutes.
We do not expect there to be any outages due to this but it is to be considered an at risk period.
We will be making configuration changes to our main LNS, sweep, this weekend. This will result in all ADSL and FTTC connections briefly dropping but immediately reconnecting.
Update 15:00 2020-04-05
This maintenance has been completed successfully. We will continue to monitor the network.
We will be conducting some changes to our switching between Maidenhead and London this weekend.
This will improve our resilience to an outage between the two Data Centers.
There will be a couple of minor glitches to connectivity as this is being done which is why it is the work is being conducted out with business hours.
There should be no major outages whilst this is being conducted.
If you have any queries please email support@waganator.watchfront.net.uk.
We will post again once the work is complete.
Hello,
We have been instructed by our Amsterdam data centre provider that they will be conducting preventative maintenance on the power breakers that feed our racks overnight on Tuesday the 22/09/2020.
This is to start 9pm UK time and can last until 5am the next morning.
During this time there will be a power outage to the rack.
We will be contacting customers directly about this.
We apologise for the interruption.
Hello,
We are replacing a power feed in our core network on Sunday 13th of December 2020 (13/12/2020) at 11am.
The vast majority of services will unaffected due to the resilience built into our network.
However there are a few customers who will be effected and we will contact you directly about this.
The maintenance is scheduled for half an hour but it is likely to take much less time than this.
Please get in contact if you have any queries.
14/02/2020
This work was completed with no issues.
Hello,
We are doing updates to our Firebrick networking equipment from midday to 4pm on Saturday 30th of January 2021.
There is no expected downtime during this maintenance and this is a courtesy “At Risk” notification.
If you have any questions about this please get in touch by emailing support@watchfront.co.uk or calling 020 7517 4999.
Hello,
Pulsant will be carrying out maintenance on power feed X to our racks in Pulsant 2 between midday and 4pm on Saturday 5th of February.
There will be disruption to the power feed during this time.
We will be in contact with every customer who has single feed equipment that will lose power during this maintenance window.
As this is the Data Centre owners who are doing this we can not change the time of this maintenance.
If you have any questions about this please get in touch by emailing support@watchfront.co.uk or calling 020 7517 4999.
Hello,
Pulsant will be carrying out maintenance on power feed Z to our racks in Pulsant 2 between midday and 4pm on Saturday 13th of February.
There will be disruption to the power feed during this time.
We will be in contact with every customer who has single feed equipment that will lose power during this maintenance window.
As this is the Data Centre owners who are doing this we can not change the time of this maintenance.
If you have any questions about this please get in touch by emailing support@watchfront.co.uk or calling 020 7517 4999.
Hello,
We will be doing some testing on a transit link at midday on Sunday the 21/03/21.
This will be for 1 hour.
Although we do not foresee any issue with this work we are giving a courteous “At Risk” notification for this.
Please contact us if you have any questions.
Hello,
One of our suppliers of Leased Lines will be conducting maintenance on their connection with us between midnight and 8am on 05/10/2021.
There may be outages to your leased lines during this time.
We apologise for any inconvenience that this might cause.
If you have any questions please email support@watchfront.co.uk.
Hello,
On Sunday 25 September 2022 we will be moving racks in one of our data centres.
Customers with servers who are directly affected by this have been contacted.
Furthermore people who utilise our tunnelling services will experience a short outage as the routers are moved.
We will be starting work on this at 10am and hope to be finished in a couple of hours to minimise downtime.
We have the maintenance period until 6pm as a contingency although we do not expect there to be any issues.
If you have any questions please email support@watchfront.co.uk.
Hello,
After the rack migration to Pulsant 2 we now are in a position to consolidate the network and remove the now extra routing node.
On Wednesday (12/10/2022) between 6pm and 7pm we plan to complete some switching and routing configuration changes.
This will have a minimal outage for some colo customer and some tunnelled connections for around 5 minutes.
If you have any questions please let us know via email – support@watchfront.co.uk.
Apologies to customers with virtual machines running on our host ‘impersonator’, which suffered partial RAM failure and rebooted this afternoon. It is currently up and running OK, albeit with less RAM than normal. Maintenance will be scheduled to restore the full quota of RAM.
Following yesterday’s successful router changes, later this evening (not before 20:00) we will be testing router fail-over configurations. This should not affect service, but must be considered an ‘at risk’ period.
Kev, 01/09/2011: This maintenance has now been completed
This evening after 18:00 will be updating firmware on our Be ADSL LNS routers. This will result in some lines being logged out, but immediately reconnecting. We will publish a notice when this has been completed.
Kev, 06/08/2011: This maintenance has now been completed
On Saturday 06/08 we will be updating firmware on our Be ADSL LNS routers. This will result in some lines being logged out, but immediately reconnecting. This may happen a few times as various updates are applied. We will publish a notice when this has been completed.
Kev, 14/07/2011: This maintenance has now been completed
We will be doing some disk maintenance to our primary mail server this morning. No mail will be lost, but mail collection will be unavailable for a few minutes during this time.
Kev, 24/06/2012: This maintenance has now been completed
We are planning to conduct a maintenance upgrade on our Beatrix switches. The provisional planned date for this is Sunday 24th June 2012, to minimise disruption to customers’ who have services provided by these switches.
Customers affected will be contacted prior to the planned date but may experience an outage for a few minutes while the essential maintenance is conducted. We apologise for any disruption to services in advance.
Kev, 30/11/2011: This maintenance has now been completed
Apologies that the work took longer than expected, due to forced disk-checks on reboot.
We will be performing routine updates on our mail and web servers this evening starting at 18:00. Service will be interrupted for a few minutes whilst this is carried out.
KRH 25/09/16: This work has been completed successfully.
We are planning to reboot our switches in Amsterdam on Sunday 25th of September.
This is for routine maintenance.
This work will be carried out in the morning.
Our transit provider in Amsterdam will be carrying out planned maintenance from 23:00 on 15th January 2013 to 01:00 on 16th January 2013.
Transit will automatically switch over via the UK but there may be minor glitches in routing during this time.
We apologise if any inconvenience is caused.