rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_red
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_red
icon_blue
icon_red
icon_blue
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_green
icon_orange
 

FS#6632 — FS#10528 — bhs < > Nwk

Attached to Project— Network
Incident
Beauharnois
CLOSED
100%
The Newark < > Beauharnois fibre is down.

We will investigate.
Date:  Wednesday, 02 April 2014, 09:49AM
Reason for closing:  Done
Comment by OVH - Tuesday, 01 April 2014, 09:37AM

The traffic is flowing via Montreal. We will contact our provider.


Comment by OVH - Tuesday, 01 April 2014, 09:38AM

The provider has reported a cut in the fibre at New York.


Comment by OVH - Tuesday, 01 April 2014, 09:38AM

Technicians are on site.


Comment by OVH - Tuesday, 01 April 2014, 09:43AM

The cut in the fibre is at the New York exit towards Albanie
and it's affecting multiple providers and services
- Level3 NWK / BHS (10x10G)
- Level3 NWK / MTL (2x10G)
- Hibernia NWK / MTL (2x10G)
- Telia NWK / CHI (6x10G)
- Telia NWK / PAL (2x10G)
- Level3 Paris / New York / BHS (4x10G)

This is normal for the NWK / MTL-BHS link, being the
Newark/Albanie/Montréal route. However the Newark/Chicago by Albanie route
will be changed to a more direct route in order to avoid
access issues to the DC when there's a problem with NWK/MTL.


Comment by OVH - Tuesday, 01 April 2014, 09:43AM

We have redirected the traffic from:
- Miami,FL
- Atlanta,GA
- Ashburn,VA
- Newark,NJ
- New York,NYC
on the West Coast of North America.
We have activated the PNI with Comcast.


Comment by OVH - Tuesday, 01 April 2014, 09:44AM

Level3 GFS field technician shot the fibre from the site,
and a fault has been isolated approximately 60 kilometers out.
Outside Plant Engineers have been contacted, and investigations
are ongoing to determine whether the fault is on a fibre
provider's portion of the span.


Comment by OVH - Tuesday, 01 April 2014, 09:44AM

The Transport NOC has advised that the issue is believed
to have been isolated to a fibre provider's network.
Efforts to contact the fibre provider are currently ongoing.


Comment by OVH - Tuesday, 01 April 2014, 09:47AM

The fibre provider has confirmed a fibre cut on their portion of the span between Suffern and Woodbury, NY. At this time, the fibre provider as well as Level 3 are dispatching field technicians to assist with isolating the exact location of the break. The Level 3 field technician has provided an estimated arrival time of approximately two hours, and an estimated arrival time for the fibre provider's field technician will be provided once it has been made available.


Comment by OVH - Tuesday, 01 April 2014, 09:47AM

The fibre provider has reported that their field technicians are currently on site at two adjacent locations, and are working to isolate the exact break point.


Comment by OVH - Tuesday, 01 April 2014, 09:50AM

The provider's field technicians continue working to locate the fault.


Comment by OVH - Tuesday, 01 April 2014, 09:50AM

The provider has advised that the fault has been isolated to a fibre section along a bridge. Splice crews have been engaged to assist with locating the physical damage, and are currently en route to the area. An estimated arrival time is not yet available.


Comment by OVH - Tuesday, 01 April 2014, 09:50AM

The Transport NOC has confirmed that the Level 3 field technician has arrived to the area. Efforts to isolate the location of the damage continue.


Comment by OVH - Tuesday, 01 April 2014, 09:51AM

The provider has reported that the fault has been located, and was determined to have been caused by road construction. It appears that the 120-count fibre was damaged by a new post being installed during the repairs to a highway guardrail. Splice crews remain en route to the area; an estimated arrival time has not yet been provided.


Comment by OVH - Tuesday, 01 April 2014, 09:51AM

The provider has confirmed that their splice crew has arrived to the fault location, and they are currently assessing the situation. Once the extent of the damage is known, an appropriate repair strategy will be developed for implementation, and an estimated restoral time can be provided.


Comment by OVH - Tuesday, 01 April 2014, 09:51AM

The provider's splice crews continue assessing the damage.


Comment by OVH - Tuesday, 01 April 2014, 09:51AM

The Transport NOC advised that the provider has engaged an excavation crew, but an estimated time of arrival is not available at this time. Upon completion of the excavation, the splice crew will complete the necessary fibre repairs.


Comment by OVH - Tuesday, 01 April 2014, 09:57AM

The Transport NOC advised that the fibre provider continues to await the arrival of the excavation crew. An estimated time of arrival is not available at this time.


Comment by OVH - Tuesday, 01 April 2014, 10:45AM

The Transport NOC advised that the excavation crew has arrived to the site and efforts to excavate the damaged fibres are underway. An estimated time of restoral is not available at this time.


Comment by OVH - Tuesday, 01 April 2014, 12:08PM

The Transport NOC advised that the splicing of the additional damaged cable continues. In an attempt to complete repairs quickly, four splice crews are on site working on the damaged fibres. An estimated restoration time is not currently available.


Comment by OVH - Tuesday, 01 April 2014, 14:18PM

The splice crews have begun prepping the 120-count fibre for splicing.


Comment by OVH - Tuesday, 01 April 2014, 17:15PM

The Transport NOC has advised that the additional damaged cable has been fully repaired and the restoration cable for the 120-count fiber has been set in the conduit. The Transport NOC is currently attempting to obtain an estimated time of restoral.


Comment by OVH - Tuesday, 01 April 2014, 18:01PM

The Transport NOC reports that splicing is now underway on both ends of the cable. No estimated restoration time has been provided as of yet. The Level 3 field technician on site is working to obtain an estimated restoration time.


Comment by OVH - Wednesday, 02 April 2014, 09:37AM

The field technician reports that the provider has completed the splicing of three buffer tubes on both sides and is beginning splicing on the fourth tube at this time. The fibre provider has advised they are not able to provide an estimated restoration time, but may as work progresses further. The Level 3 field technician remains on site to obtain updates and observe the progress of the repair efforts.


Comment by OVH - Wednesday, 02 April 2014, 09:38AM

The cable has been repaired and the service back up. We are redeploying the peering and transit on New York/Newark/Ashburn and Miami.


Comment by OVH - Wednesday, 02 April 2014, 09:38AM

The fibre provider has completed splicing eight of the ten buffer tubes on one side and has completed splicing of five buffer tubes on the other end. Once all the splicing has been completed on one side, those resources will assist with splicing on the other end for a faster restoration. No estimated restoration time has been provided.


Comment by OVH - Wednesday, 02 April 2014, 09:39AM

The fibre provider continues with restoration efforts at this time. The Transport NOC is monitoring alarms and confirms some systems have been restored.


Comment by OVH - Wednesday, 02 April 2014, 09:40AM

The fibre provider is completing splicing on one end and has two buffer tubes remaining to splice on the other end. The Transport NOC verified that additional alarms have cleared and services are being restored.


Comment by OVH - Wednesday, 02 April 2014, 09:43AM

The Transport NOC sees the majority of alarms clear with all affected systems up at this time. The Technical Service Center has begun to confirm service restoration with customers.


Comment by OVH - Wednesday, 02 April 2014, 09:44AM

The Transport NOC confirms that all alarms are clear at this time and the final system restored at approximately 17:17 GMT. The Technical Service Center continues to confirm restoration with customers. The Transport NOC will continue to monitor alarms while awaiting confirmation that the fibre provider is hands-off.


Comment by OVH - Wednesday, 02 April 2014, 09:46AM

The fibre provider has completed all splicing. The cables have been buried and their crews are hands-off at this time. The Transport NOC confirms that alarms remain clear and the Technical Service Center has confirmed restoration with the affected customers. The fibre cut was caused by road construction. The fibre was damaged during the installation of a new post during repairs to a highway guard rail along a bridge. The provider confirmed that this was a permanent restoration.