OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#8035 — 176.31.226.0/24
Incident Report for Network & Infrastructure
Resolved
We are updating the switches:
http://status.ovh.net/?do=details&id=4091

on 176.31.226.0/24 we are having an issue
on the A.
The B took the switching.
2013 Feb 10 20:49:02 sw Feb 10 20:49:02 %KERN-0-SYSTEM_MSG: Disabling IRQ #16 - kernel

Update(s):

Date: 2013-02-11 00:01:29 UTC
108 UP.


All is UP. we do apologize for this failed update.

Date: 2013-02-11 00:00:20 UTC
108 update done. in boot.

Date: 2013-02-10 23:59:57 UTC
104 UP

108 in update by the B.

Date: 2013-02-10 23:59:41 UTC
104 update, in boot.

Date: 2013-02-10 23:59:23 UTC
2 FEX: 104 and 108 still in a software update by the B.


Date: 2013-02-10 23:58:35 UTC
3 remaining FEX are down. The rest is already UP.


Date: 2013-02-10 23:57:39 UTC
B is back and will take in charge also
the FEX update all from the beginning.

Date: 2013-02-10 23:56:24 UTC
The B has failed it's mission. We have reconnected
the FEX on A which is already on its new version.

It's updating the FEX, all from the beginning.
Downtime will take 10-15 minutes :(


Date: 2013-02-10 23:48:58 UTC
Let's try:

Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive reset
100 yes non-disruptive rolling
101 yes non-disruptive rolling
102 yes non-disruptive rolling
103 yes non-disruptive rolling
104 yes non-disruptive rolling
105 yes non-disruptive rolling
106 yes non-disruptive rolling
107 yes non-disruptive rolling
108 yes non-disruptive rolling
109 yes non-disruptive rolling
110 yes non-disruptive rolling
111 yes non-disruptive rolling
112 yes non-disruptive rolling

Date: 2013-02-10 23:48:36 UTC
Alright, the A failed.

we are going to launch the update for B around midnight.

Date: 2013-02-10 23:46:36 UTC
Install has failed. Return code 0x40930039 (aborting due to failed upgrade).
Please identify the cause of the failure, and try 'install all' again.

Date: 2013-02-10 23:46:21 UTC
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive reset
100 yes non-disruptive rolling
101 yes non-disruptive rolling
102 yes non-disruptive rolling
103 yes non-disruptive rolling
104 yes non-disruptive rolling
105 yes non-disruptive rolling
106 yes non-disruptive rolling
107 yes non-disruptive rolling
108 yes non-disruptive rolling
109 yes non-disruptive rolling
110 yes non-disruptive rolling
111 yes non-disruptive rolling
112 yes non-disruptive rolling

Date: 2013-02-10 23:46:04 UTC
All is UP as the beginning.
we are restarting the updating process.


Date: 2013-02-10 23:45:05 UTC
the A is UP.

we are resetting the FEX, it is accepting them now (it is the same software version).


Date: 2013-02-10 23:43:21 UTC
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes disruptive reset new image is incompatible

Date: 2013-02-10 23:43:09 UTC
We have cut the 2x10G between vss-8a and n5-a
the routing is done via vss-8b and n5-b


Date: 2013-02-10 23:41:59 UTC
We are going to downgrade it to the previous version
to recover the initial update process and avoid the breakdown.


Date: 2013-02-10 23:40:48 UTC
The A is back:
Reason: Reset Requested due to Fatal System Error
System version: 5.2(1)N1(2a)
Service: ISSU failure: 0x4093003B

sw.176.31.226.248(config)# inter po 100-112
sw.176.31.226.248(config-if-range)# shutdown
Posted Feb 10, 2013 - 23:40 UTC