Incident Details

Resolved [31/08/2020 22:47]

The root cause was TalkTalk maintained hardware failure affecting 1 Ethernet NNI of ours and 6000 other B2B clients. TalkTalk fault incident resolution states:

<-- snip -->
NOC monitoring identified an FPC10 (Flexible PIC Concentrator) failure at NGE001.LOH. This caused a total loss of service to approx. 6k B2B circuits from approx. 12:47 (31/08). The Core Network Ops team were engaged and their investigations found that the FPC10 had failed and could not be restored remotely. To restore service as of approx. 17:23 a field engineer attended site and replaced the faulty FTP10 with support from the core network ops team. This incident will now be closed with any further root cause analysis being completed via the problem management process.
<-- snip -->

Apologies for the disruption caused this afternoon.

Update [31/08/2020 17:26]

We are seeing services restore. We will share further details from the carrier once received.

Update [31/08/2020 17:19]

The latest update reads:

<-- snip -->
The field engineer has now arrived at fault location and will commence replacement of the faulty FPC10 with assistance from the Network Support team. A further update will be provided by 18:00.
<-- snip -->

Update [31/08/2020 15:32]

TalkTalk are currently trying to source a replacement line card. Their latest update reads:

<-- snip -->

The field engineer has attend stores however, there are no replacement cards of the required type available. The Field Engineer will now attend another location to pull a card from a newly commissioned NGE which is not yet carrying traffic. Equinix data centre access has been arranged by NOC and ETA for the FE at the fault location in LOH has been confirmed as 16:30.

<-- snip -->

Apologies for the prolonged outage. We hope for a swift resolution once the replacement card is in place.

Update [31/08/2020 14:03]

TalkTalk have issued an incident report which can be followed via their status feed or via our control panel:

https://managed.mytalktalkbusiness.co.uk/network-status-report.php?reportid=30384
https://control.interdns.co.uk/notification.aspx?id=13404650

Identified [31/08/2020 13:19]

TalkTalk have confirmed that they are aware of an issue affecting Harbour Exchange / LD8. We have been given the incident reference of 13404650 and await further details. Currently this is only affecting 1 of our NNI at this location.

Investigating [31/08/2020 13:04]

We are investigating an outage affecting one of our NNI at LD8. This will be impacting leased line connectivity for some circuits. Further details will be shared once the fault has been understood.