OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
bhs-3b-a9
Incident Report for Network & Infrastructure
Resolved
We loss access to bhs-3b-a9. We investiguate.

Update(s):

Date: 2017-06-01 12:44:37 UTC
Traffic is now stable since 13:41 GMT+2

Date: 2017-06-01 11:19:36 UTC
We see again some loss.
We investiguate.

Date: 2017-06-01 10:40:33 UTC
During the incident of this morning, nous moved some vlans by shutting them on B side to restaure the service quickly. All the traffic for these vlans passes through A side.
Here are the vlans:
290, 292, 294 to 303, 305 to 310

Date: 2017-06-01 08:30:58 UTC
Cisco talks about a possible known bug.
We will continue the investigation with Cisco.
All IP returned at 07:02 GMT+2 (servers IP) and 07:24 (IPFO)

We will upgrade the couple as soon as possible.

Date: 2017-06-01 05:52:51 UTC
Cisco investiguate about this log:

2017-06-01T06:29:10+02:00 198.27.73.96 bhs-3b-a9 3307013 ba [local7.crit] === 3307013: LC/0/1/CPU0:Jun 1 06:29:10 CEST: fib_mgr[183]: %ROUTING-FIB-2-OOR : FIB has run out of DATA_TYPE_TABLE_SET resource memory. No more route updates will be handled by FIB. Please delete routes, and then clear FIB on this node to resume normal operation.
2017-06-01T06:31:17+02:00 198.27.73.96 bhs-3b-a9 3307026 ba [local7.crit] === 3307026: LC/0/0/CPU0:Jun 1 06:31:17 CEST: fib_mgr[183]: %ROUTING-FIB-2-OOR : FIB has run out of DATA_TYPE_TABLE_SET resource memory. No more route updates will be handled by FIB. Please delete routes, and then clear FIB on this node to resume normal operation.
2017-06-01T06:41:40+02:00 198.27.73.96 bhs-3b-a9 3307127 be [local7.info] === 3307127: LC/0/1/CPU0:Jun 1 06:41:40 CEST: fib_mgr[183]: %ROUTING-FIB-6-RSRC_OK : FIB resource state has returned to normal. FIB has exited resource constrained operation and normal forwarding has been restored
2017-06-01T06:41:47+02:00 198.27.73.96 bhs-3b-a9 3307129 be [local7.info] === 3307129: LC/0/0/CPU0:Jun 1 06:41:47 CEST: fib_mgr[183]: %ROUTING-FIB-6-RSRC_OK : FIB resource state has returned to normal. FIB has exited resource constrained operation and normal forwarding has been restored

Date: 2017-06-01 05:44:19 UTC
The equipement come back and then down again.
We contact Cisco in emergency.

For now, service is stable.

We investiguate with cisco.
Posted Jun 01, 2017 - 04:16 UTC