rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_blue
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_red
icon_blue
icon_red
icon_orange
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_orange
icon_green
 

FS#15031 — fra-1-n7

Attached to Project— Network
Incident
Whole Network
CLOSED
100%
Due to a reboot (Because of this issue: http://status.ovh.net/?do=details&id=15030), one card is down. A ticket has been raised to the constructor.
The fr version (updated in real time): https://travaux.ovh.net/?do=details&id=27166
Date:  Thursday, 14 September 2017, 10:56AM
Reason for closing:  Done
Comment by OVH - Wednesday, 13 September 2017, 09:06AM

We are changing the linecard


Comment by OVH - Thursday, 14 September 2017, 10:34AM

Done at 07:16 UTC (13/09/2017).


Comment by OVH - Thursday, 14 September 2017, 10:35AM

The card crashed rather quickly after being inserted (10:20 UTC).
We sent technicians on site to move the crad.


Comment by OVH - Thursday, 14 September 2017, 10:37AM

At 10:20 UTC:
The removal / insertion in the same module again crashed the card.
We move it to another module of the n7.


Comment by OVH - Thursday, 14 September 2017, 10:38AM

At 10:30 UTC:
The card boots well on another slot.
We are currently analyzing with our supplier.
For the moment, we have not reinserted the optics.


Comment by OVH - Thursday, 14 September 2017, 10:40AM

At 13:04 UTC:
After investigation with our supplier, we will leave the card on the other module for now.
By reconfiguring the ports of the new module, we had a problem in configuring the port-channel to fra-1-a9.
We will leave these ports isolated for the moment (no saturation problem, there are 11 other 100G links that take charge).
We are putting back into production the optical links between DC and peer.


Comment by OVH - Thursday, 14 September 2017, 10:43AM

At 13:20 UTC:
All peers and links between DCs have been restored.
We will plan a maintenance to restore the 4 links to fra-1-a9.
We are still in analysis with our supplier to understand everything that has happened.


Comment by OVH - Thursday, 14 September 2017, 10:49AM

At 04:14 UTC (14/09/2017):
The new card in Module 3 has been reload twice in a row between 4:00 am and 4:10 am UTC.
We analyze.


Comment by OVH - Thursday, 14 September 2017, 10:50AM

At 04:23 UTC:
The card just crashed again.
We cut the module 3 at this time.
We analyze.


Comment by OVH - Thursday, 14 September 2017, 10:51AM

At 04:39 UTC:
We decide to downgrade the version.


Comment by OVH - Thursday, 14 September 2017, 10:52AM

04:46 UTC:
We isolate all the peer present on fra-1-n7.


Comment by OVH - Thursday, 14 September 2017, 10:53AM

At 05:10 UTC:
We downgraded the version.
The n7 seems stable now.
The two cards (Modules 3 and 7) booted well.
We isolate the peers while monitoring the nexus.


Comment by OVH - Thursday, 14 September 2017, 10:54AM

At 06:07 UTC:
Everything seems stable since the downgrade an hour ago.