OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#14532 — 46.105.108.0/24 & 46.105.109.0/24
Scheduled Maintenance Report for Network & Infrastructure
Completed
One of two Nexus managing these networks lost some of the FEX (Bug).

We will upgrade the couple of Nexus to ISSU tonight to avoid the failure, if the problem occurrs on the second Nexus.

For now, some of the traffic flows through the secondary N5.

The intervention is scheduled this Friday, August 28th from 11:00 pm GMT + 2

Update(s):

Date: 2015-09-04 11:33:04 UTC
when in doubt, reboot....

# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 fex100|51C09 Online N2K-C2248TP-1GE SSI14280VP3
101 FEX0101 Online N2K-C2248TP-1GE SSI14260CR3
102 fex102|51C07 Online N2K-C2248TP-1GE SSI14270LJ4
103 fex103|51C06 Online N2K-C2248TP-1GE SSI142709EX <<<<<<<
104 fex104|51C05 Online N2K-C2248TP-1GE SSI14270LRD
106 FEX0106 Online N2K-C2232PP-10GE SSI14160433
110 FEX0110 Online N2K-C2232PP-10GE SSI14260EFJ
111 FEX0111 Online N2K-C2248TP-1GE SSI14270E6V
112 FEX0112 Online N2K-C2248TP-E-1GE SSI171301VL

The FEX is back up. The configuration is synchronised. All good.

Date: 2015-09-04 11:28:16 UTC
Deconf/reconf is not working.

We will electrically reboot FEX103.

Date: 2015-09-04 11:27:51 UTC
# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 fex100|51C09 Online N2K-C2248TP-1GE SSI14280VP3
101 FEX0101 Online N2K-C2248TP-1GE SSI14260CR3
102 fex102|51C07 Online N2K-C2248TP-1GE SSI14270LJ4
104 fex104|51C05 Online N2K-C2248TP-1GE SSI14270LRD
106 FEX0106 Online N2K-C2232PP-10GE SSI14160433
110 FEX0110 Online N2K-C2232PP-10GE SSI14260EFJ
111 FEX0111 Online N2K-C2248TP-1GE SSI14270E6V
112 FEX0112 Online N2K-C2248TP-E-1GE SSI171301VL
--- -------- Connected N2K-C2248TP-1GE SSI142709EX <<<<<<

FEX 103 is still not present, despite the port shut....

I will fully deconfigure the 103 and repush the configuration.

Date: 2015-09-04 11:27:01 UTC
We will reboot the 1st n5.

Date: 2015-09-04 11:26:48 UTC
FEX 103 disappeared from one of the 2 nexus.

# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 fex100|51C09 Online N2K-C2248TP-1GE SSI14280VP3
101 FEX0101 Online N2K-C2248TP-1GE SSI14260CR3
102 fex102|51C07 Online N2K-C2248TP-1GE SSI14270LJ4
104 fex104|51C05 Online N2K-C2248TP-1GE SSI14270LRD
106 FEX0106 Online N2K-C2232PP-10GE SSI14160433
110 FEX0110 Online N2K-C2232PP-10GE SSI14260EFJ
111 FEX0111 Online N2K-C2248TP-1GE SSI14270E6V
112 FEX0112 Online N2K-C2248TP-E-1GE SSI171301VL
--- -------- Offline N2K-C2248TP-1GE SSI142709EX

This only impacts redundancy and delivery of new servers in this rack.
We will reboot the n5 on 03/09/2015, the traffic will be redirected by the 2nd n5.

Date: 2015-08-31 14:22:31 UTC
When restarting the FEX on the primar y switch, those that had lost their config (probably from before), are causing inconsistencies in the vpc.
We have checked all the configurations and all servers are pinging.

The couple is now stable.

Date: 2015-08-31 14:20:04 UTC
The ISSU on the secondary switch went well. We are reloading the primar y switch that is in an abnormal condition (ports are inactive)

Date: 2015-08-31 14:18:47 UTC
The primary switch has been reloaded and crashed during the ISSU. It rebooted causing a small interruption on the service. The FEX fell during the upgrade. We are updating the secondary switch for the FEX to be updated and back on the primary switch.

Date: 2015-08-31 14:16:02 UTC
The intervention starts, we have lost a FEX.
Posted Aug 28, 2015 - 18:16 UTC