19
Jan
2009
11:41

Regarding the ADSL problems of a fortnight ago (see Status History), the new ATM box has been running well for a week, so we are now considering it stable. The standby ATM box is being thoroughly tested, to ensure we have a much shorter outage should it fail again in the future.

Posted by admin

12
Jan
2009
08:40

Yesterday’s maintenance went to plan, with just a single short outage around 13:40. ADSL routers took around 10 minutes to log back in, though I notice some are still off line – if yours is, please power cycle it. There have been some improvements to the config of the Cisco ATM box, which should help with the packet loss we were seeing last week.

We decided not to replace the Cisco box at this stage, though if we see further problems this may still be an option.

Incidentally we are seeing unusually high levels of traffic, possibly due to the release of the Windows 7 Beta. This is not going to help.

As mentioned before, this only affects ADSL still on the old 20CN BT network – lines that have moved over to the new 21CN are unaffected.

We will continue to monitor the situation

Posted by admin

09
Jan
2009
10:50

Regarding the slow downlink and packet loss problem seen yesterday, various configuration tweaks have been made to the new Cisco ATM box, and this does seem to have helped.

We are also considering fitting a more powerful variant, as part of the planned maintenance on Sunday (see planned maintenance).

Posted by admin

08
Jan
2009
13:46

We are getting reports of slow downlink and low level packet loss on some ADSL lines. We believe this may relate to the replacement ATM box following Tuesday’s outage. We are investigating, and will post more information as it becomes available.

Posted by admin

06
Jan
2009
19:12

A summary of today’s events:-

All our ADSL connections from the 20CN (ATM) BT network come into a single rack. Although they come in on two BT Centrals, all traffic goes through a single ATM box that translates between BT’s ATM and our IP network.

Today we suffered a very unfortunate sequence of events. Firstly power to the entire rack was tripped. When that was restored, we diagnosed a faulty power supply. A replacement was sourced and fitted, but we then found the ATM box had failed. A replacement for that was sourced and fitted, but the engineers had trouble configuring it. Simultaneously an alternative ATM box was sourced and fitted (plan B). The plan B box finally started working about 18:00. Incidentally the plan A box also started working at 18:20.

As you may be aware, for many years we have been improving our infrastructure to use multiple redundant systems and interconnects, so as to avoid a single failure taking down the network. Sadly this ATM box was one of the last remaining single points of failure. We are in the process of moving all ADSL over to BT’s new 21st Century Network, which is delivered by IP over multiple redundant links. Sadly delays by BT mean this hasn’t happened yet, so we still need the ATM box.

Just to clarify, today’s outage only affected 20CN ADSL connections. ADSL connections on 21CN were unaffected, as were all our other services (server colo etc.)

Not surprisingly, we have had several clients ask us how they can avoid such outages in the future. The simple answer is to have multiple redundant internet connections, with no single point of failure. For example, you could have a backup connection using 3G. If you are interested, please do talk to us about your requirements, as we are continually developing new solutions (and as always can offer bespoke solutions if needed).

Posted by admin

06
Jan
2009
19:09

The ATM box has now been stable for about an hour. If you still have problems, please try power-cycling your ADSL router. If that doesn’t help, please contact us.

Posted by admin

06
Jan
2009
18:11

The ATM box is now up and running, and ADSL lines are reconnecting. Due to the number of lines trying to connect, you may find your ADSL is not yet stable. I will post another message when things are stable

Posted by admin

06
Jan
2009
17:25

Another brief update. “Plan A” ATM box is still being worked on, the engineers sound hopeful that they are close to sorting it. “Plan B” ATM box is currently being configured, again the engineers are sounding optimistic. The race is on to see which plan works first.

We will of course carry on working this evening until it is fixed. We will also be answering support phone calls (020 7517 4902) and emails support@watchfront.net , so please don’t hesitate to contact us if you need to.

Once again thank you for your patience

Posted by admin

06
Jan
2009
15:52

A brief update – plan A (replacement ATM box, problems with config) is still being worked on, and plan B (alternative replacement ATM box) hardware should be arriving at the data centre very shortly.

Posted by admin

06
Jan
2009
14:48

As a Plan B we are also sourcing an alternative replacement ATM interface, in case we continue to have trouble getting the current replacement working.

Posted by admin

06
Jan
2009
14:29

Unfortunately the engineers are having trouble configuring the new hardware platform. Difficult to give an accurate time estimate at the moment, it really depends how quickly the engineers can find the cause of their troubles.

Thank you for your patience, and please rest assured we are doing all we can to rectify the problem.

Posted by admin

06
Jan
2009
12:00

Replacing the power supply has not helped, so now replacing the entire ATM interface. Hope to be back up in a couple of hours. More news shortly.

Posted by admin

Current Status

    GoodNo current issues

Status Archive