rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_red
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_red
icon_blue
icon_red
icon_blue
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_green
icon_orange
 

FS#4099 — FS#8035 — 176.31.226.0/24

Attached to Project— Network
Incident
the whole network
CLOSED
100%
We are updating the switches:
http://status.ovh.net/?do=details&id=4091

on 176.31.226.0/24 we are having an issue
on the A.
The B took the switching.
2013 Feb 10 20:49:02 sw Feb 10 20:49:02 %KERN-0-SYSTEM_MSG: Disabling IRQ #16 - kernel
Date:  Monday, 11 February 2013, 01:01AM
Reason for closing:  Done
Comment by OVH - Monday, 11 February 2013, 00:40AM

The A is back:
Reason: Reset Requested due to Fatal System Error
System version: 5.2(1)N1(2a)
Service: ISSU failure: 0x4093003B

sw.176.31.226.248(config)# inter po 100-112
sw.176.31.226.248(config-if-range)# shutdown


Comment by OVH - Monday, 11 February 2013, 00:41AM

We are going to downgrade it to the previous version
to recover the initial update process and avoid the breakdown.


Comment by OVH - Monday, 11 February 2013, 00:43AM

We have cut the 2x10G between vss-8a and n5-a
the routing is done via vss-8b and n5-b


Comment by OVH - Monday, 11 February 2013, 00:43AM

Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes disruptive reset new image is incompatible


Comment by OVH - Monday, 11 February 2013, 00:45AM

the A is UP.

we are resetting the FEX, it is accepting them now (it is the same software version).


Comment by OVH - Monday, 11 February 2013, 00:46AM

All is UP as the beginning.
we are restarting the updating process.


Comment by OVH - Monday, 11 February 2013, 00:46AM

Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive reset
100 yes non-disruptive rolling
101 yes non-disruptive rolling
102 yes non-disruptive rolling
103 yes non-disruptive rolling
104 yes non-disruptive rolling
105 yes non-disruptive rolling
106 yes non-disruptive rolling
107 yes non-disruptive rolling
108 yes non-disruptive rolling
109 yes non-disruptive rolling
110 yes non-disruptive rolling
111 yes non-disruptive rolling
112 yes non-disruptive rolling


Comment by OVH - Monday, 11 February 2013, 00:46AM

Install has failed. Return code 0x40930039 (aborting due to failed upgrade).
Please identify the cause of the failure, and try 'install all' again.


Comment by OVH - Monday, 11 February 2013, 00:48AM

Alright, the A failed.

we are going to launch the update for B around midnight.


Comment by OVH - Monday, 11 February 2013, 00:48AM

Let's try:

Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive reset
100 yes non-disruptive rolling
101 yes non-disruptive rolling
102 yes non-disruptive rolling
103 yes non-disruptive rolling
104 yes non-disruptive rolling
105 yes non-disruptive rolling
106 yes non-disruptive rolling
107 yes non-disruptive rolling
108 yes non-disruptive rolling
109 yes non-disruptive rolling
110 yes non-disruptive rolling
111 yes non-disruptive rolling
112 yes non-disruptive rolling


Comment by OVH - Monday, 11 February 2013, 00:56AM

The B has failed it's mission. We have reconnected
the FEX on A which is already on its new version.

It's updating the FEX, all from the beginning.
Downtime will take 10-15 minutes :(


Comment by OVH - Monday, 11 February 2013, 00:57AM

B is back and will take in charge also
the FEX update all from the beginning.


Comment by OVH - Monday, 11 February 2013, 00:58AM

3 remaining FEX are down. The rest is already UP.


Comment by OVH - Monday, 11 February 2013, 00:59AM

2 FEX: 104 and 108 still in a software update by the B.


Comment by OVH - Monday, 11 February 2013, 00:59AM

104 update, in boot.


Comment by OVH - Monday, 11 February 2013, 00:59AM

104 UP

108 in update by the B.


Comment by OVH - Monday, 11 February 2013, 01:00AM

108 update done. in boot.


Comment by OVH - Monday, 11 February 2013, 01:01AM

108 UP.


All is UP. we do apologize for this failed update.