OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#7524 — Vlan corruption
Scheduled Maintenance Report for Network & Infrastructure
Completed
Following to http://status.ovh.net/?do=details&id=3534
we have had a vlan corruption on some N5
switch. Cisco worked on the bug
CSCuc37057 and a new version was released:
5.2.1.N1.2a

We will update a switch pdc1-17-n5
to validate the reparation of the vlan status.


Update(s):

Date: 2012-11-23 00:12:44 UTC
p19-53/54-n5: done with a cutoff in some old internal services. a fex has crashed.

p19-54-n5# sh install all status
This is the log of last installation.

Need to perform cleanup of failed upgrade.

Install has failed. Return code 0x40930039 (aborting due to failed upgrade).
Please identify the cause of the failure, and try 'install all' again.

Reason: Reset Requested due to Fatal System Error
System version: 5.2(1)N1(2a)
Service: ISSU failure: 0x4093003B

Date: 2012-11-23 00:11:33 UTC
minicloud-1a/b (kimsufi): done
Difficult, but done:
http://status.ovh.net/?do=details&id=3734

Date: 2012-11-22 15:53:50 UTC
p19-61/62: doing
p19-63/64: doing
p19-65/66: doing
p19-67/68: doing

Date: 2012-11-22 15:53:41 UTC
vrack-1a/b-n5: doing
vrack-2a/b-n5: todo

Date: 2012-11-22 13:32:58 UTC
arc-2a/b-n5: todo
omer-2a/b-n5: todo

Date: 2012-11-22 13:32:50 UTC
4 rese&aux de HG:
188.165.13.0/24: done
188.165.14.0/24: done
188.165.15.0/24: done
178.33.122.0/24: done

Date: 2012-11-22 13:32:39 UTC
arc-1a/1b-n5: doing

Nov 22 13:49:44 arc-1a-n5 %VPC-2-VPC_ISSU_START: Peer vPC switch ISSU start, locking configuration
2012 Nov 22 13:53:05 arc-1a-n5 %SATCTRL-FEX100-2-SATCTRL_IMAGE: FEX100 Image update in progress.
2012 Nov 22 13:53:15 arc-1a-n5 %SATCTRL-FEX101-2-SATCTRL_IMAGE: FEX101 Image update in progress.
2012 Nov 22 13:53:26 arc-1a-n5 %SATCTRL-FEX102-2-SATCTRL_IMAGE: FEX102 Image update in progress.
2012 Nov 22 13:53:36 arc-1a-n5 %SATCTRL-FEX103-2-SATCTRL_IMAGE: FEX103 Image update in progress.
2012 Nov 22 13:53:46 arc-1a-n5 %SATCTRL-FEX104-2-SATCTRL_IMAGE: FEX104 Image update in progress.
2012 Nov 22 13:53:55 arc-1a-n5 %SATCTRL-FEX105-2-SATCTRL_IMAGE: FEX105 Image update in progress.
2012 Nov 22 13:54:05 arc-1a-n5 %SATCTRL-FEX106-2-SATCTRL_IMAGE: FEX106 Image update in progress.
2012 Nov 22 13:54:16 arc-1a-n5 %SATCTRL-FEX107-2-SATCTRL_IMAGE: FEX107 Image update in progress.
2012 Nov 22 13:54:26 arc-1a-n5 %SATCTRL-FEX108-2-SATCTRL_IMAGE: FEX108 Image update in progress.
2012 Nov 22 13:54:36 arc-1a-n5 %SATCTRL-FEX109-2-SATCTRL_IMAGE: FEX109 Image update in progress.
2012 Nov 22 13:54:46 arc-1a-n5 %SATCTRL-FEX110-2-SATCTRL_IMAGE: FEX110 Image update in progress.
2012 Nov 22 13:54:56 arc-1a-n5 %SATCTRL-FEX111-2-SATCTRL_IMAGE: FEX111 Image update in progress.
2012 Nov 22 13:57:42 arc-1a-n5 %SATCTRL-FEX101-2-SATCTRL_IMAGE: FEX101 Image update complete. Install pending
2012 Nov 22 13:58:21 arc-1a-n5 %SATCTRL-FEX100-2-SATCTRL_IMAGE: FEX100 Image update complete. Install pending
2012 Nov 22 13:59:44 arc-1a-n5 %SATCTRL-FEX103-2-SATCTRL_IMAGE: FEX103 Image update complete. Install pending
2012 Nov 22 14:00:09 arc-1a-n5 %SATCTRL-FEX104-2-SATCTRL_IMAGE: FEX104 Image update complete. Install pending
2012 Nov 22 14:00:15 arc-1a-n5 %SATCTRL-FEX102-2-SATCTRL_IMAGE: FEX102 Image update complete. Install pending
2012 Nov 22 14:00:19 arc-1a-n5 %SATCTRL-FEX106-2-SATCTRL_IMAGE: FEX106 Image update complete. Install pending
2012 Nov 22 14:00:51 arc-1a-n5 %SATCTRL-FEX107-2-SATCTRL_IMAGE: FEX107 Image update complete. Install pending
2012 Nov 22 14:00:51 arc-1a-n5 %SATCTRL-FEX105-2-SATCTRL_IMAGE: FEX105 Image update complete. Install pending
2012 Nov 22 14:01:01 arc-1a-n5 %SATCTRL-FEX111-2-SATCTRL_IMAGE: FEX111 Image update complete. Install pending
2012 Nov 22 14:01:06 arc-1a-n5 %SATCTRL-FEX110-2-SATCTRL_IMAGE: FEX110 Image update complete. Install pending
2012 Nov 22 14:01:09 arc-1a-n5 %SATCTRL-FEX109-2-SATCTRL_IMAGE: FEX109 Image update complete. Install pending
2012 Nov 22 14:01:13 arc-1a-n5 %SATCTRL-FEX108-2-SATCTRL_IMAGE: FEX108 Image update complete. Install pending
2012 Nov 22 14:02:17 arc-1a-n5 %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-100 On-line
2012 Nov 22 14:02:18 arc-1a-n5 %PFMA-2-FEX_STATUS: Fex 100 is online
2012 Nov 22 14:03:34 arc-1a-n5 %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-101 On-line
2012 Nov 22 14:03:34 arc-1a-n5 %PFMA-2-FEX_STATUS: Fex 101 is online
2012 Nov 22 14:04:50 arc-1a-n5 %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-102 On-line
2012 Nov 22 14:04:51 arc-1a-n5 %PFMA-2-FEX_STATUS: Fex 102 is online
2012 Nov 22 14:06:07 arc-1a-n5 %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-103 On-line
2012 Nov 22 14:06:08 arc-1a-n5 %PFMA-2-FEX_STATUS: Fex 103 is online
2012 Nov 22 14:07:23 arc-1a-n5 %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-104 On-line
2012 Nov 22 14:07:24 arc-1a-n5 %PFMA-2-FEX_STATUS: Fex 104 is online
2012 Nov 22 14:08:39 arc-1a-n5 %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-105 On-line
2012 Nov 22 14:08:40 arc-1a-n5 %PFMA-2-FEX_STATUS: Fex 105 is online
2012 Nov 22 14:09:56 arc-1a-n5 %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-106 On-line
2012 Nov 22 14:09:56 arc-1a-n5 %PFMA-2-FEX_STATUS: Fex 106 is online
2012 Nov 22 14:11:11 arc-1a-n5 %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-107 On-line
2012 Nov 22 14:11:12 arc-1a-n5 %PFMA-2-FEX_STATUS: Fex 107 is online
2012 Nov 22 14:12:27 arc-1a-n5 %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-108 On-line
2012 Nov 22 14:12:28 arc-1a-n5 %PFMA-2-FEX_STATUS: Fex 108 is online
2012 Nov 22 14:13:43 arc-1a-n5 %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-109 On-line
2012 Nov 22 14:13:43 arc-1a-n5 %PFMA-2-FEX_STATUS: Fex 109 is online
2012 Nov 22 14:14:59 arc-1a-n5 %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-110 On-line
2012 Nov 22 14:14:59 arc-1a-n5 %PFMA-2-FEX_STATUS: Fex 110 is online
2012 Nov 22 14:16:14 arc-1a-n5 %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-111 On-line
2012 Nov 22 14:16:15 arc-1a-n5 %PFMA-2-FEX_STATUS: Fex 111 is online
2012 Nov 22 14:17:01 arc-1a-n5 %VPC-2-VPC_ISSU_END: Peer vPC switch ISSU end, unlocking configuration


2012 Nov 22 14:28:04 arc-1b-n5 %VPC-2-VPC_ISSU_START: Peer vPC switch ISSU start, locking configuration
2012 Nov 22 14:29:29 arc-1b-n5 %IM-5-IM_INTF_STATE: mgmt0 is DOWN in vdc 1
2012 Nov 22 14:29:35 arc-1b-n5 %IM-5-IM_INTF_STATE: mgmt0 is UP in vdc 1
2012 Nov 22 14:29:41 arc-1b-n5 %IM-5-IM_INTF_STATE: mgmt0 is DOWN in vdc 1
2012 Nov 22 14:29:47 arc-1b-n5 %IM-5-IM_INTF_STATE: mgmt0 is UP in vdc 1
2012 Nov 22 14:30:29 arc-1b-n5 %IM-5-IM_INTF_STATE: mgmt0 is DOWN in vdc 1
2012 Nov 22 14:30:38 arc-1b-n5 %IM-5-IM_INTF_STATE: mgmt0 is UP in vdc 1
2012 Nov 22 14:31:40 arc-1b-n5 %VPC-2-VPC_ISSU_END: Peer vPC switch ISSU end, unlocking configuration


Date: 2012-11-22 13:32:22 UTC
We continue with updates:
Pcc-104-n5: done
Pcc-105-n5: done
Pcc-106-n5: done
Pcc-107-n5: done
Pcc-108-n5: done
Pcc-109-n5: done
Pcc-110-n5: done
Pcc-111-n5: done
Pcc-112-n5: done
Pcc-113-n5: done


Date: 2012-10-29 10:17:00 UTC
storage-s31a/b-n5: doing

Date: 2012-10-29 10:16:52 UTC
storage-s27a/b-n5: done
omer-1a/b-n5: done

Date: 2012-10-29 09:32:42 UTC
storage-s15a/b-n5

Date: 2012-10-26 10:08:22 UTC
rps-1a/rps-1b

Date: 2012-10-26 10:08:00 UTC
pdc1-12-n5
backup-1-n5

Date: 2012-10-26 09:40:25 UTC
before (we see the status UNKNOWN

vlan id 1
vdc 1 bd_id = 1, bd_id_orig = 1, vlan_type = USER_VLAN(1)
sdb_vlan_type UNKNOWN(16777216, err 0x0), oper =down
vlan_state = active, vlan_shut_state = no-shutdown, vlan_modes = (1)ce
sdb_vlan_mode 0(unknown, err 0x0), n_access_ports 23, n_native_ports 23
mtu = 1500, internal_state = 0, cfg_flags 0, vlan_name = default

After

vlan id 1
vdc 1 bd_id = 1, bd_id_orig = 1, vlan_type = USER_VLAN(1)
sdb_vlan_type USER_VLAN(1, err 0x0), oper =down
vlan_state = active, vlan_shut_state = no-shutdown, vlan_modes = (1)ce
sdb_vlan_mode 1(ce, err 0x0), n_access_ports 23, n_native_ports 23
mtu = 1500, internal_state = 0, cfg_flags 0, vlan_name = default

Then it peroperly corrects the problem.


Date: 2012-10-26 09:39:02 UTC
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive reset



Images will be upgraded according to following table:
Module Image Running-Version New-Version Upg-Required
------ ---------- ---------------------- ---------------------- ------------
1 system 5.2(1)N1(1) 5.2(1)N1(2a) yes
1 kickstart 5.2(1)N1(1) 5.2(1)N1(2a) yes
1 bios v1.5.0(11/30/10) v1.5.0(11/30/10) no
1 power-seq v1.2 v1.2 no
Posted Oct 26, 2012 - 09:37 UTC